Reviewing AI draft replies: what to check before you approve
A practical checklist for approving, editing or rejecting an AI support draft, and when to talk to the agent first, based on how Cerberius Agents' approval flow works.
If you approve every reply your AI support agent writes, you are the last check before a customer reads it. That is a good place to be, but only if you know what to look at. Reading the draft alone is not enough, because a confident, well-written answer can still rest on a wrong assumption.
This is the checklist I would use, and how the approval flow in Cerberius Agents supports it.
What "nothing is sent until a person approves" means
By default, every reply waits for a person. Nothing reaches the requester until someone on your team approves it. The one exception is automatic replies, which are off by default and which an owner or admin can switch on per platform. Even then, replies with a proposed fix, replies where the agent has questions, follow-ups after a fix, and long replies or ones that look like they contain internal details (code, file names, logs, server names or addresses) still wait for a person. See Automatic replies and What the agent can and cannot do.
So if you leave automatic replies off, the draft in your Queue is the only thing standing between the agent and the customer. Treat it that way.
Before you read the draft, read the findings
Open the reply from Queue › Waiting for approval. The page has tabs: Conversation, Agent's findings, Talk to the agent and History. Start with Conversation so you remember what the customer actually asked, then go to Agent's findings.
Click Show details and check these fields:
- Root cause. Does it say why this happened, or does it say Not determined? If it is not determined, the reply should not sound certain.
- Confidence. The agent gives a figure from 0% to 100%. A low figure is a reason to slow down. The help is blunt about this: don't approve a reply you're unsure about.
- Category. One of bug, how to, data issue, config, account or other. If the agent filed a bug as a how-to, the reply will probably tell the customer to do something that won't help.
- Assumptions. These are things the agent took for granted. Check each one is true. This is where invented detail tends to hide, for example a plan the customer is on or a setting they have changed.
Check the evidence
Under Show details, Evidence for these findings lists what backs the findings. Each item shows where it came from (code, k8s, loki, database or ticket), the exact file, log or record, and the relevant extract.
Ask two things. Does the evidence actually support the claim in the reply? And is there any claim in the reply with no evidence behind it?
If you want to see how the agent got there, click Show on What the agent looked at. It lists every step in order, in plain words, and you can click a step to see what it found. A step that didn't work shows failed. If the agent answered without looking anything up, it says so. That last case deserves extra suspicion when the question was about your product's specific behaviour.
Check the reply text itself
The findings can be right and the wording still wrong. Before you click anything, read the reply as the customer would.
- Does it answer the question they asked, not a nearby one?
- Does it promise anything you can't keep, such as a date, a refund or a fix?
- Are any facts in it ones you would want to verify, such as timeframes or prices?
- Does it mention internal details the customer shouldn't see?
- Is the tone right for this person?
Your three options
Approve as it is
Click Approve & send. On a platform with no ticket system the button says Approve and nothing is sent. Approving records that you agree and marks the ticket done.
Edit, then approve
If the draft is mostly right, change the text in the box. You will see Edited. Your version will be sent. Click Approve & send and your version goes out, not the agent's.
Both versions are kept. On the ticket's History, the reply shows The agent's draft and What was sent, after edits. That is useful later, because a pattern of the same edit tells you what to fix at the source. You can't approve an empty reply.
Reject
Reject when the reply shouldn't be sent and you don't want the agent to try again. Click Reject, write the reason in Why reject it?, then click Reject and send nothing. The reason is kept for the record.
Nothing is sent. The ticket shows as Needs a person, because the requester is still waiting. From there you can use Write a reply to answer them yourself, or Investigate again to ask the agent for a new reply. When it is sorted, close it with Close this ticket…. If you want to close the ticket in your ticket system, use that button rather than Reject. See Closing tickets.
When to talk to the agent first
Often the right move is none of the three. If the draft is close but the findings are off, tell the agent. The Talk to the agent box sits on every ticket page. Your messages go only to the agent, and the requester never sees them.
Good reasons to use it:
- The agent has questions. Answer them there.
- A fact is wrong. For example: "Refunds take 5 to 7 business days, not 3."
- It looked in the wrong place. For example: "Check the payment logs for booking B-78 from yesterday instead."
- The tone is off. For example: "Make the reply shorter, and don't mention the database."
Be specific, as you would with a new colleague. Then wait. The page shows that the agent is working and updates by itself.
Two things can happen. If its findings change, you get a new version of the reply, and the old one is marked Replaced by a newer draft. If they stay the same, it answers you in the box and the current reply is still ready to approve.
The difference from editing is that editing fixes one reply, while talking to the agent corrects the findings the reply is built on. If the draft is wrong because of a wrong assumption, fix the assumption. See Talking to the agent.
If the customer writes again while you are reviewing, the waiting reply is withdrawn and the agent works again, so you get a fresh reply that answers both messages.
A quick decision guide
- Findings right, wording slightly off: edit and approve.
- Findings uncertain, or an assumption is wrong: talk to the agent.
- Reply shouldn't go out at all: reject, then write your own or investigate again.
- Everything checks out against the evidence: approve.
Close the loop
After you decide, click 👍 or 👎 under the findings to say whether the root cause was right. Each person gets one vote per draft, and your votes feed the Root cause confirmed right figure on the Metrics page. See Metrics and feedback.
For the full screen-by-screen detail, read Reviewing a reply.