AI Customer Service Agents: What Works and What Does Not

What should an AI agent handle, and what should it not?

Give an AI agent the tickets that have one correct answer written down somewhere, and keep everything else for people. The test is simple: could a new employee answer this from your help page alone?

Three public cases worth knowing

Real cases are more useful than vendor claims, because they show what happens outside the demo.

Air Canada, 2024. Air Canada's website chatbot told a customer he could claim a bereavement fare refund within 90 days of buying his ticket. The airline's actual policy required the request before travel. CBS News reported that a Canadian tribunal ordered Air Canada to pay CAD 812 in damages and fees, rejecting the airline's argument that the chatbot was a separate legal entity responsible for its own actions. The tribunal said the chatbot was part of the airline's website. The sum was small, and the principle was not: your bot's answer is your answer.

Klarna, 2024 to 2025. Entrepreneur reported that Klarna said in February 2024 that its AI could do the work of 700 customer service agents and had handled 75% of customer chats. In May 2025, its chief executive told Bloomberg that AI support was cheaper but of lower quality and that the company was hiring human agents again. It did not abandon AI. It changed where it used it.

Verizon, 2025. Mobile World Live reported in June 2025 that Verizon re-tooled its app with a Gemini-based assistant, lengthened support call hours with real agents and began round-the-clock live chat. The AI arrived alongside more access to people. We did not check how Xfinity, T-Mobile or other carriers use AI, so look at each company's own pages for its current practice.

How do you measure whether it works?

Measure how many customers got a correct answer and did not need to come back, not how many chats the bot closed. Vendors often measure the second because it is easier.

Intercom's pricing page defines an outcome for its Fin agent as a customer confirming resolution, not asking for more help after Fin replies, or Fin completing a workflow. Under that definition a customer who gives up counts as resolved. Track these instead:

• Real resolution. A thumbs-up or a short question after the answer.
• Recontact rate. The same customer asking the same thing within three days.
• Handover rate. The share passed to a person, and how many handovers were avoidable.
• Wrong-answer rate. A weekly review of a random sample, scored by someone who knows the policy.

An illustrative example shows why this matters. Out of 1,000 chats, a bot closes 700. At USD 0.99 an outcome, that is USD 693. If 20% of those 700 customers come back with the same problem, only 560 were really resolved, and the cost per real resolution is 693 / 560 = about USD 1.24. That is still cheap. The danger lies in the 140 customers who left unhappy while the dashboard said success.

Design rules that prevent most problems

1. One source of truth. The agent answers only from documents you maintain. Tell it to say so when it does not know.
2. No policy exceptions. It cannot promise refunds, discounts or deadlines. Those are for a person.
3. A visible route to a person. In the first message, not buried. WhatsApp's own business policy asks that automated responses come with prompt, clear and direct escalation paths.
4. Disclosure. Say at the start that it is an AI. Article 50 of the EU AI Act has applied since 2 August 2026 and requires chatbot providers to ensure people know.
5. No fully automated decisions with serious effects. GDPR Article 22 gives people the right not to be subject to decisions based solely on automated processing where they significantly affect them. A refusal of a refund or service can fall near that line, so keep a person in the decision.
6. A weekly review. Read 30 conversations every week for the first two months. Nothing else finds the awkward cases.
7. Update the source when policy changes. The Air Canada case came down to a bot that was out of step with the policy page.

What happens to your human team

Your people will get the harder tickets, because the easy ones are gone. That means each ticket takes longer and needs more skill, and a team sized for the old mix will struggle. Plan time for training and for reviewing the agent's work, and do not cut the team to the saving on paper.

A 30-day pilot

1. Week 1. Export 200 past tickets. Mark which could have been answered from your documents alone. If it is under a third, an agent will not help much yet.
2. Week 2. Write answers for the top 15 questions. Load them. Test with 50 real past questions and fix every wrong answer.
3. Week 3. Launch on one channel with a visible human route. Read every conversation.
4. Week 4. Measure real resolution, recontact and handover. Decide whether to widen, change or stop.

If you want to put a bot on your website, see how to add an AI chatbot to your website. For phone, see the AI receptionist guide. For the broader question of agents versus workflows, read AI agents for business, or ask about our AI automation service.