When should an AI agent hand a ticket to a human?

A plain checklist for small software teams deciding what an AI agent drafts, what a person approves and what stays manual, using Cerberius Agents' approval-first defaults.

If you run support with two or three people, you don't have time to read every reply an AI drafts. You also can't afford a wrong answer going out under your name. The question is where to put the human.

Here is a way to decide, using how Cerberius Agents handles it. I'll keep it to what the product does, and I won't quote any resolution rates, because they would only be guesses.

Start with everything waiting for a person

In Cerberius Agents, every reply waits for a person by default. Nothing reaches a requester until someone on your team approves it. That is the Off mode for automatic replies, and it is where I'd start.

It isn't wasted time. For the first few weeks you are learning what the agent gets right on your product and where it slips. You see the draft, you approve it, edit it or reject it, and you build a picture of which tickets are safe to loosen up on.

To review a draft, open the ticket from the Queue. A reply shows as Waiting for approval until you decide. If you want the detail on that screen, see Reviewing a reply.

The three levels of automation

Automatic replies live under Platform › Settings › General › Automatic replies. There are three modes:

Only an owner or admin can switch these on for a platform. Automatic replies are also sent through your ticket system's reply webhook. If a platform has no ticket system connected, nothing is sent automatically and the reply waits for a person as usual. The developer who connects your system will want Connect your ticket system.

A checklist for what a person should always see

Some tickets should never go out unseen, however well the agent usually does. Use this as your starting list.

1. The customer is angry, or money is involved. Refunds, billing disputes, cancellations and contract questions are judgement calls. They are about your relationship with the customer. Keep these manual or approval-only.

2. The answer depends on something the customer hasn't told you. If the agent has to ask questions or needs more information, the product already holds that reply for a person. That is a good rule for your own team as well: a guess dressed up as an answer is worse than a pause.

3. The reply proposes a fix or change. A reply that includes a proposed fix or change always waits for a person, even with automatic replies on. Treat it the same way yourself. A person decides whether a change to your live system goes ahead. See Approving a fix.

4. The reply is long or technical. Automatic replies also hold back replies that are long, or that look like they contain internal details such as code, file names, logs, server names or addresses. That check is a fixed check on the reply text and doesn't rely on the AI, so it works as a safety net. Even so, don't count on it as your only one. A long reply usually means the question was not simple.

5. It's a follow-up after a fix. These always wait for a person too. The customer is checking whether you did what you said, so a human should look.

6. You can't say what the correct answer is. If you wouldn't be able to point to a line in your own docs that backs the answer, a person should read it first. This is general advice, not a product rule. It is also the signal that your help content has a gap.

What is safe to loosen

Once you have reviewed enough drafts to see a pattern, you have candidates:

Be strict with the rules you set for the confident mode. The reply goes out only when every rule matches, so each rule you add makes it harder for a reply to skip review. That is what you want while you build trust.

What happens when you disagree with a draft

Say the agent drafts something you don't want sent. You have two options in the product.

Click Reject, then say why in Why reject it?. The reason is kept for the record. Choose Reject and send nothing. Nothing is sent, and the ticket shows as Needs a person on the Queue, because the requester is still waiting.

From there you can Write a reply and click Send reply, which goes to your ticket system like an approved reply. Or you can click Investigate again to ask the agent for a new one.

You can also write your own reply on any open ticket that has nothing waiting for approval, using Write a reply at the top of the ticket. So a ticket that should stay manual is easy to keep manual.

Write a proper reason when you reject. Those reasons are what you look at later to see which kinds of tickets keep going wrong. If you do a weekly review, they tell you where to tighten rules or fix your help content.

How to keep an eye on automatic replies

A reply sent by automatic replies is marked Sent automatically on the ticket, so you can tell it apart from one a person approved. Check these first in your regular review. If one is wrong, you don't have to fix the rules in a hurry. Untick Allow automatic replies and every automatic reply stops at once. Everything goes back to waiting for a person.

That switch is worth knowing about before you need it. Tell the rest of the team where it is.

A sensible order of rollout

  1. Leave automatic replies off and review every draft for a couple of weeks.
  2. Turn on Acknowledge only, with a short, honest message.
  3. Read the Sent automatically replies regularly after you try Answer automatically when confident, and start with strict rules.
  4. Keep the checklist above as your list of what stays with a person.

Nothing here needs to be permanent. If the agent starts getting something wrong, move back a step.

If you haven't set up automatic replies yet, the page to read first is Automatic replies.

All articles

Loading…