Insurance support QA usually means a team lead listening to a handful of calls and reading a few transcripts each week. Manual review caps out around 5% of tickets, and that 5% is not random. People grade the easy cases first and skip the long, ambiguous ones, which are exactly where errors and complaints hide. In one EU engagement we moved that ceiling to 30% coverage with no added headcount, by changing two things: what the model sees, and how it decides.
The thesis is simple. The privacy control belongs in the pipeline, not in a model policy or a vendor promise. If raw customer data never reaches the model, your GDPR posture stops depending on what an LLM does or does not retain. That single decision is what made the rest of the system shippable in a regulated environment.
If raw customer data never reaches the model, your privacy posture stops depending on what the model does.
Redact before the model, not inside it
We run Microsoft Presidio as a hard gate in front of every model call. Names, policy numbers, emails, phone numbers, IBANs, and national IDs are detected and replaced with stable typed placeholders before any text leaves our boundary. The model evaluates a transcript that reads like POLICYHOLDER_1 and POLICY_NUMBER_2, never the real values.
Stable placeholders matter more than they look. If the same entity maps to the same token across a conversation, the evaluator can still reason about who said what and whether the agent verified identity correctly, without ever holding the underlying PII. You keep the analytical signal and drop the exposure.
Two operational notes from production. Tune Presidio recognizers and confidence thresholds to your own data: default entity sets miss locale-specific formats, and an over-eager recognizer that redacts the policy clause being discussed will quietly destroy the thing you are trying to grade. Treat the redaction layer as a tested component with its own recall metrics, not a one-line import.
Multi-agent evaluation against the actual policy
A single prompt asking is this a good response is a vibe check, and it drifts. Real insurance QA is judged against multi-page policy documents: coverage limits, exclusions, required disclosures, escalation rules. So we split the work. One agent retrieves and grounds the relevant policy sections for the ticket. A second checks the agent's answer against those sections for factual and coverage accuracy. A third scores procedure: identity verification, mandatory disclosures, tone, correct escalation.
Separating retrieval, factual checking, and procedure scoring keeps each judgment narrow and auditable. When a ticket is flagged, you can see which agent flagged it and against which policy clause, instead of staring at one opaque score.
Our decision rule for what reaches a human: route to a reviewer only when agents disagree, when retrieval confidence is low, or when the ticket touches a high-risk category like a claim denial or a complaint. Everything else gets an automated score with its policy citation. This is where the coverage jump comes from. Humans stop spending time confirming the obvious passes and spend it on the genuinely contested cases.
Calibrate the graders before you trust the graph
An automated QA layer is itself a model making judgments, so it needs its own quality control. We ran a calibration set: a few hundred tickets scored independently by senior human reviewers, against which the agent scores are measured for agreement before go-live and re-measured on a schedule after. Where agents and humans disagreed systematically, the fix was usually in the rubric, not the model: a scoring criterion humans applied with unstated context that had to be made explicit. Version the rubric like code, because when it changes, historical scores stop being comparable, and a compliance team that discovers that mid-audit will not enjoy the surprise.
The calibration set also settles the internal argument every such system triggers: whether the machine grades fairly. Showing agreement rates against the team's own senior reviewers, per criterion, converts the debate from opinion to a number the QA lead can own.
Why consistency beats throughput
The sixfold coverage gain is the headline, but the more durable win is consistency. A tired reviewer at 4pm grades differently than a fresh one at 9am, and two reviewers rarely agree on the same call. The automated layer applies the same rubric to every ticket, so trends are real signal rather than reviewer noise. When QA scores move, you can trust that the agents changed, not the graders.
What this buys the business
You go from auditing a biased 5% to a representative 30%, with every automated score carrying a policy citation a regulator or auditor can follow. Because redaction happens in-pipeline, the privacy story is structural and easy to defend. Reviewers are redirected onto the high-risk tickets where their judgment actually pays off, and the same architecture extends cleanly to claims handling, complaints triage, and any workflow judged against long, formal documents. A fair question is why coverage stops at 30% rather than everything. The ceiling is deliberate: automated scores are only issued where policy retrieval is confident and the agents agree, and that safe zone grows as the calibration set accumulates adjudicated cases. Pushing to full coverage on day one would mean scoring exactly the ambiguous tickets the system is least qualified to judge, which is how automated QA loses the trust it needs to be worth running. Grow the safe zone with evidence, and the ceiling rises on its own.
