I designed a workflow to keep an AI support agent’s knowledge aligned with the human support knowledge base—without asking support writers to maintain both manually.
5x
growth in AI knowledge
82%
Of agent knowledge converted to AI
The human and AI knowledge bases were maintained separately. When a support article changed, the AI agent could keep using outdated information.
INSIGHT 01
Knowledge lived in three libraries
AI Content had to be created and stored separately in two places meaning it was never in sync with updates.
Agent knowledge to be translated
Every time someone updated the agent articles, someone had to remember to translate this into bot knowledge.
Agent updates were complicated
Agents were learning how to write and edit code in order to update articles, making some fear the process as it felt far too technical.
Where the process failed
Around one agent from each circle had permissions to update their own articles. It created a blocker when there were absences. In turn, AI knowledge was never a focus, and any article updates did not get translated to AI Bot knowledge.
Agent spots gap in knowledge and reports to Circle lead.
Circle lead uses Cursor to update code of article.
Article is pushed to GitHub for Circle lead to review
Article merged
~ 8 hours
The agents already had familiarity with updating and maintaining their knowledge base, but with limited people who had access to do so meant delays in information being updated.
Every relevant human knowledge-base change should create a proposed AI counterpart. A person reviews the before-and-after content before it can be published.
Streamline the workflow
Let agents focus the human knowledge base updates, allowing anyone to feed into the changes and help them understand AI knowledge requirements.
Increase AI knowledge coverage
Provide customers with accurate information fed by our AI knowledge base, translated from human knowledge.
I designed a simple workflow that enabled agents to utilise existing tools to update their articles, translating the agent articles into AI articles and running a series of test sets to ensure that the knowledge being submitted doesn't break or contradict any existing knowledge.
Translation rules
Remove internal-only details, rewrite agent-focused language, and structure answers around questions customers might ask.
Human approval
Show reviewers the proposed changes before anything reaches the AI knowledge base.
Agent updates were complicated
Surface topics driving human hand-offs, show what the AI currently answers, and track whether updates close the gaps.
Agent uses Claude in Slack to start article changes
If agent has approval access, they can merge their own PR
Article published to agent knowledge base
Article translated into AI knowledge using LLM
AI Article reviewed & approved by Product owner
The circle lead will receive a slack notification they need to approve a PR
AI Articles will run a series of test to ensure it can answer customer queries. The results must be checked by a Product owner before it is approved and released.
Knowledge gaps dashboard
Using Claude Code, I built a fully working operational dashboard to help us identify the gaps in our knowledge - where the bot was unable to support the customer. The dashboard featured:
Customer escalated conversations categorised into product domains
Cross referencing existing agent articles, AI articles or missing articles
Easy skills to run in Claude to update or create knowledge
DASHBOARDS
INTERNAL VIEW
For agents who want to understand the knowledge gaps, where the bot was unable to help
DASHBOARDS
INTERNAL VIEW
Each product area had a focus: Translate agent knowledge, update existing knowledge or create new knowledge.
DASHBOARDS
INTERNAL VIEW
Within the gap, it highlights what is missing, what the article says (if applicable) and a Claude skill prompt to translate, create or edit the articles.
Customer impact
Our bot drew on multiple knowledge sources to answer customer questions, providing both general account information and details specific to each customer. By expanding its AI knowledge, we could offer universal support for general questions about our product.
IN APP
AI KNOWLEDGE
AI Bot response on AI knowledge about the ANNA money. Knowledge taken from AI article about general ANNA offering.
IN APP
AI Knowledge
AI Bot response on AI knowledge about FSCS protection. Knowledge taken from AI article about general ANNA offering.
04 / OUTCOMES
The AI-facing knowledge base grew 5× faster than the human knowledge base in both word volume and article count. 82% of human knowledge was converted for AI use, narrowing the gap between the two knowledge bases and reducing the chance that customers would receive outdated answers.
growth in AI knowledge
in 5 months
of human knowledge converted into AI knowledge
Connect the dashboard to the propagation workflow from day one. The dashboard and update workflow were separate, so closing a knowledge gap and reflecting that progress were two steps. Connecting them earlier would make the system more coherent and reduce hand-offs.
Revisit approvals as the system evolves. As more signals were introduced, reviews and notifications accumulated. I’d regularly reassess which signals need human attention and which can be consolidated or safely automated.