Enter password to view case study
CASE STUDY
How I designed the workflow that keeps an AI support agent's knowledge in sync with its human knowledge base, and measured the result.
THE PROBLEM
The company ran two knowledge bases side by side: a human-readable one that support agents used every day, and a structured one that fed an AI customer agent. They started life as separate things, maintained on separate schedules — which meant the AI agent was quietly answering customers from information that support agents had already corrected, retired, or superseded weeks earlier. Every gap was a customer getting a wrong answer, or the AI agent escalating a question a human had already answered in the knowledge base.
The fix wasn't "write more AI content." It was closing the loop — making sure that every time a human article changed, the AI-facing version changed with it, without turning every content update into a manual double-entry job for already-stretched support writers.
MY ROLE
I owned three connected pieces of this:
The workflow agents follow to keep the AI knowledge base current — what triggers a review, who approves it, what "done" looks like.
Shaping the translation logic — the rules that decide what counts as customer-relevant, and how a human-facing article gets rewritten for an AI reader.
The knowledge gaps dashboard — a regularly-refreshed view showing where the AI agent was actually failing customers, and whether content work was closing those gaps.
DESIGNING THE SOLUTION
The two knowledge bases were structured completely differently — the AI version didn't map 1:1 to article paths in the human one, so "just diff the two folders" was never going to work. I designed the process around a simple principle: every human knowledge base change gets a proposed AI counterpart, but nothing ships without a human seeing the diff first.
Concretely, that meant defining the trigger, the proposal (its own linked review, never silently merged), the approval gate (a person reviews actual before/after content, not a checkbox), and — just as important — the "do nothing" case, since not every human edit is customer-relevant. If reviewers stop trusting a "nothing to do here" verdict, they re-read everything manually anyway, and the workflow adds overhead instead of removing it.
*Illustrative data for portfolio use — figures are representative, not actual production numbers.
Shaping the translation logic
A straight copy-paste didn't work — the human knowledge base was written by agents for agents. The rules I helped define: strip everything internal (screenshots, internal tool references, "raise a task to..."), rewrite the point of view (human articles say "you can find this on Settings" — but the AI knowledge base isn't read by the customer, it's read by the AI agent, who then talks to the customer, so it becomes "the customer can find this on Settings"), respect an explicit "internal-only" marker, and structure content for retrieval — a heading that reads as a real customer question retrieves far better than a generic one.
Building the knowledge gaps dashboard
Working with a backend engineer on the data pipeline - primarily from real customer conversations and knowledge gaps, I designed a dashboard turning raw escalation logs into a ranked, regularly-updated view: which topics were driving the most human hand-offs, how that trended over time, and whether the knowledge base updates we shipped actually moved the number — each topic expanding into the specific gap, what the AI-facing content currently said, work in progress against it, and a link into deeper analysis.
Agents could then use the dashboard to view where the gap lies using a single command string to implement a gold answer update
that would give the fulfil the AI knowledge gap without having to create or write articles from scratch,
*Illustrative data for portfolio use — figures are representative, not actual production numbers.
THE RESULT
Comparing the period since this workflow launched to the present:
The AI-facing knowledge base grew substantially faster than the human one over the same period — roughly five times the rate, in both word count and article count.
The gap between the two knowledge bases narrowed measurably, translating into fewer customers getting outdated answers from the AI agent.
*Illustrative data for portfolio use — figures are representative, not actual production numbers.
WHAT I’D DO DIFFERENTLY
Getting the "no change needed" verdict trustworthy enough that reviewers stopped double-checking took a couple of iterations — I'd invest in that earlier next time. I'd also wire the dashboard directly to the propagation workflow from day one, rather than leaving "we closed this gap" and "the dashboard reflects it" as two separate steps.
In addition I would review the approval process for the entire knowledge base as over time, more signals were introduced and more reviews were needed and more notifications were added to the process which has become quite overwhelming.