In February 2022 we replaced the manual classification work of 6 FTEs with a Forethought AI model running at 89% accuracy — saving $200K in year one. Here's exactly how we did it, and what we'd do differently.
At Q4 Inc, every incoming support ticket required manual classification before it could be routed to the right team. That meant reading the ticket, determining the category, and tagging it — a task that was consuming the equivalent of six full-time employees across the support organization.
The work was repetitive, low-value, and error-prone. Misclassified tickets created downstream delays, frustrated customers, and generated rework for agents who received tickets outside their area. The problem was well understood. The solution — AI-powered triage — was available but hadn't been deployed at scale in most ops environments at that point.
We selected Forethought AI as the platform and designed two concurrent models to run from day one:
Both models were trained on historical ticket data before launch. The critical pre-launch work was data quality preparation — cleaning, standardizing, and re-categorizing historical tickets that had been manually tagged inconsistently over years. Garbage in, garbage out applies to AI more brutally than anywhere else.
We built a human-in-the-loop layer for tickets the model scored below a confidence threshold, rather than forcing a prediction. This was the right call — it kept accuracy high while giving us an ongoing dataset of edge cases to retrain against.
Both models went live simultaneously on February 3, 2022. From day one, 100% of incoming tickets were processed through the AI — no phased rollout, no shadow period running in parallel. We had enough confidence in the training data and model performance to go full coverage immediately.
First-week performance validated the approach: 89% accuracy across both models, with the confidence-threshold routing keeping the human queue manageable. The 11% of tickets routed to humans were disproportionately the genuinely ambiguous ones — which is exactly what you want.
Data quality is the real project. The AI implementation took weeks. The data cleanup that made it possible took months. Every historical ticket tagged inconsistently by a human was a potential training error. This work is invisible in vendor demos.
The org change was harder than the tech. Six people had been doing this work. Redeploying that capacity — having honest conversations, redesigning roles, managing the human side of automation — required more leadership bandwidth than the implementation itself.
Confidence thresholds matter more than accuracy targets. "89% accurate" sounds good until you think about what happens to the 11%. Designing the edge case routing thoughtfully made the system trustworthy. Ignoring it would have made it a liability.
What we'd do differently: Start the retraining pipeline earlier. We had 54,821 correct predictions to learn from in year one. A structured retraining cadence from month three would have pushed accuracy higher faster.