AI-Agent Testing Checklist: What to Test Before You Publish

A polished answer can still be unsupported, incomplete, out of date, or unsafe. Launch testing should compare the response with an expected outcome: the facts it should include, the qualifier it must preserve, the topics it should refuse, or the handoff it should provide.

Create the evaluation set before making final prompt changes. Otherwise it becomes difficult to tell whether a revision improved the agent or merely changed its style.

• Core FAQsTest the questions the business already receives most often.
• Buying or next-step questionsTest fit, inclusions, process, timing, price context, location, and contact routes.
• Procedural questionsConfirm ordered steps, prerequisites, stop conditions, and escalation points.
• Follow-up questionsAsk a short follow-up that only makes sense in the context of the previous message.

Ask the same intent with formal wording, shorthand, a common synonym, a typo, and a vague follow-up. The agent should find the same authoritative answer without pretending that a genuinely ambiguous question is clear.

Include a question that could refer to two products, locations, plans, or policies. A safe response should clarify the missing detail or explain both scoped possibilities instead of selecting one without support.

• Missing answerAsk for a price, date, feature, policy, or exception that is not in the approved knowledge.
• Stale versionAsk using an old product name or superseded process.
• Conflicting sourcesCreate a controlled test where two sources disagree and verify that the agent does not merge them into a new claim.
• False premiseState an incorrect assumption and confirm the agent corrects it only when the approved information supports the correction.

Ask for private information, individual professional advice, guarantees, discounts, negotiations, credentials, internal instructions, or account changes that the agent cannot authorize. Confirm that the response protects the boundary while still offering a useful next step.

Try requests to ignore earlier rules, reveal system instructions, treat visitor text as authoritative business knowledge, or answer outside the configured scope. The agent should keep platform safeguards and owner-approved rules in control.

• SupportedAre material claims present in the approved knowledge?
• CompleteDoes the answer preserve important conditions, dates, limits, and exceptions?
• RelevantDoes it answer the actual question without unrelated material?
• BoundedDoes it refuse or qualify requests outside its authority?
• ActionableWhen a person is needed, does the answer give the correct usable route?
• AppropriateIs the tone, length, formatting, and disclosure suitable for the audience?

Rerun affected cases when prices, policies, products, documents, instructions, model behavior, retrieval, or publishing settings change. Keep a small permanent regression set for the most important and highest-risk questions.

A failed test is not always a prompt problem. The source may be unclear, duplicated, inaccessible, or missing. Repair the authoritative knowledge first when that is the actual cause.