A 30-minute weekly review to catch wrong AI support answers
A simple weekly routine for founders and support heads: use the Metrics page and ticket history to find weak AI answers, then fix the background and rules behind them.
You don't need a dashboard full of percentages to know whether your AI support agent is getting things wrong. You need a regular look at what it said, what your team changed, and why.
Here is a 30-minute routine that does that. It uses the Metrics and feedback page, ticket history, and the places where you can fix the cause. It doesn't produce a "hallucination rate", and I'd be wary of anyone who gives you one without showing how it's counted. It does show you, week by week, whether the agent is understanding your system.
What to look at first: two numbers that tell you the most
Open the metrics page. Two figures matter for this review.
- Replies sent without edits. If your team keeps rewriting the agent's drafts, the drafts are missing something.
- Root cause confirmed right. This is made up of your own votes on tickets. The help describes it as the clearest sign of whether the agent understands your system.
Read them together. High on both means the agent is doing well. Low on either tells you where to dig. Write both down each week in a plain spreadsheet or a note. Trends over a few weeks are more useful than any single reading.
One caution. Root cause confirmed right only means something if people vote. If nobody votes, the number reflects very little. Part of this routine is making sure the votes actually happen.
Minutes 0 to 5: check the metrics and note the direction
Look at the two numbers and compare them with last week's. Ask only one question: better, worse or flat?
If both are high and steady, and you rarely edit replies, the help suggests you may be ready to consider automatic replies on simple tickets. That is an optional setting and it is off by default. Treat it as a decision for later, not part of this week's review.
Minutes 5 to 20: read the tickets behind the numbers
Numbers tell you something is off. Tickets tell you what. Go to your tickets list and open a handful from the past week. The list has a search box and a status filter, and rows show the subject, ticket number, requester and status. See Tickets and history.
Pick tickets in two groups:
- Ones where a reply was edited before sending.
- Ones where you voted that the root cause was wrong, or would have if you'd had time.
On each ticket, scroll to History. It lists everything that happened, oldest first. For a reply, click Show the reply. If someone edited it, you will see The agent's draft next to What was sent, after edits. That comparison is the most useful thing in the whole review. It shows exactly what your team had to correct.
Sort each miss into a cause
Keep a short list. For each weak answer, ask which of these it was:
- The agent didn't know something about your setup. For example, it didn't know what a repository does, or which part of the system a feature lives in.
- It broke a rule you hold everywhere. Tone, wording about refunds, what never to promise.
- It misread this one ticket. One-off cases don't need a system change.
- The customer's request was unclear. Not every miss is the agent's fault.
Only the first two lead to a fix. If you change things after every one-off, you'll end up with a messy pile of instructions.
Minutes 20 to 28: fix the cause in the right place
Feedback on its own doesn't change how the agent works. The help is clear on that. You have three places to make a correction, and each suits a different kind of miss. See Metrics and feedback.
Correct it on the ticket
If a ticket is still open, use Talk to the agent to correct it there. This fixes this ticket. See Talking to the agent. Note that a closed ticket can't take new messages, so correct things before you close. See Closing tickets.
Improve the background for your system
When the agent lacked context, add it to About your system under Settings › General, and to each repository's description under Settings › Repositories. Say plainly what each repository does and what matters about the system. See General settings.
Add or tighten a rule
When the agent broke a standing preference, go to Organisation › Rules, type the rule in Organisation rules and click Save. The help recommends short, one-line rules. Changes apply from the next investigation, and investigations already running keep the rules they started with.
Keep the split clean. Facts about one system belong in that platform's About your system. Rules in Organisation › Rules are for things that apply everywhere. There is a limit of 10,000 characters, and a counter shows how much you've used. A rules box that has grown into an essay is harder to keep correct.
If you run a website chatbot
A chatbot is different. It answers only from your help sources, and nobody approves its answers before the visitor sees them. If it gets something wrong, the usual cause is the content. The help's tips apply: use your customers' words, keep one topic per page or section, put answers in text rather than images, and write out the steps. It also suggests checking the Conversations tab, because questions the bot couldn't answer show you what to write next. See Help sources.
Minutes 28 to 30: record one change and one vote habit
Write down the single change you made this week and the date. Next week, when you look at the same two numbers, you have something to connect to any movement. Make one or two changes a week, not ten, or you won't know what helped.
Then check that your team is voting. The metric is only as good as the votes behind it.
What this routine can't tell you
It won't give you an exact count of wrong answers. It samples what your team reviews and edits. That is honest, and it is usually enough to find the weak spots worth fixing. Where the agent's drafts are approved by a person before they go out, your review also catches errors before customers do.
Start this week: open the metrics page, write down the two numbers, and read five edited replies in History.