Sample content for editorial review. Examples are illustrative.
The demo answers a narrow question
A demo proves that a useful response is possible. Production asks whether useful responses arrive consistently, at an acceptable cost, with someone responsible when things fail. Those are different questions. I would treat a polished prototype as the start of an investigation, then write down the assumptions it depends on. Which documents are available? Who checks the answer? What happens when the service is slow?
Build evidence before adding features
Start with evaluation. Collect representative requests, including ambiguous questions, missing sources, and cases where the system should decline. Keep some examples separate from prompt development. Record what a good result requires in each case. A single overall score can hide a serious failure, so inspect the failures by type and by the consequences they create for the user.
Plan for an ordinary bad day
Reliability is more than a successful model call. Put limits on retries, timeouts, tool access, and spending. Decide which failures can be retried safely and which need a human decision. A clear failure message is often more useful than a confident answer produced after the system has lost the context it needed. Make the fallback part of the product flow.
Give the outcome an owner
Trust needs visible boundaries. Show relevant sources when they help a person check the answer. Keep an audit trail that supports investigation without collecting unnecessary sensitive information. Ownership also needs a name: who reviews incidents, changes evaluation examples, and decides whether a release is ready? Without that agreement, every operational problem becomes somebody else’s problem.
Use a small release as a learning tool
I would release a bounded workflow to a small group before expanding its scope. Track corrections, abandoned tasks, review time, and the reasons people override the output. Compare those observations with the original evaluation set. The next feature should address a measured gap. A demo becomes a product through repeated evidence that the workflow helps people under real constraints, not through the number of components in its architecture.
