RAG over Confluence in JSM workflows. what i learned trying to make stale docs useful
My boss told me to make our Confluence content useful for IT request answering. Spent like 6 weeks trying to do it and it failed. Tbh the premise was reasonable: we have a couple thousand IT runbook articles in Confluence, the team writes them well, surely we can layer something AI on top and have it answer the dumb questions for us.
What I learned that I think is worth sharing:
First the obvious. Confluence content is f***ing messier than you think! Out-of-date pages, duplicates, half-finished drafts that nobody archived, articles written for one team that get found by another, etc. Pulling it through any AI layer requires either a serious clean-up first or some kind of dating/owner metadata so the AI can prefer fresh content. We did the cleanup, took a month.
2- a lot of the actual employee questions arent in Confluence at all. A) because my company is cheap and hasn't given everyone a Jira/Confluence seat yet, B) people just don't use it. Theyre in the threads. Slack threads, email threads, jira ticket comments. The stuff that actually solved a problem at 4am six weeks ago is in someones DM with a senior engineer, not in a wiki. So a Confluence-only RAG layer hits the easy questions and misses the gnarly ones.
If you are about to do this, do the Confluence audit first. Your future self will thank you! :)
(btw, for us specifically, my boss ultimately agreed that my salary is more valuable than trying to automate something that there are pretty good tools out there for already. There are at least like 3-4 good ones. We chose Risotto because my boss liked how snazzy it is. IT did bridge the gap between slack threads and Confluence content together. It doesnt fix is the underlying problem which is Confluence messiness. but its another layer over a problem the org still needs to deal with eventually.)