A multi-agent retrieval pipeline that cut internal search time by 54% across engineering and support, now answering more than 8,000 queries a day.
IndustryEnterprise SaaS
Duration8 weeks
EngagementEnd-to-End Ownership
CategoryAgentic RAG
My role: Architecture, retrieval engineering, evaluation, production rollout
The challenge
Engineering and support teams were spending a significant share of every day hunting through wikis, ticket history, runbooks and design docs that lived in four different systems. Keyword search returned hundreds of stale hits and no ranking that reflected whether a document was still true.
The failure mode that mattered most was not a missed answer — it was a confident answer drawn from a document that had been superseded eighteen months earlier. Any system we shipped had to know how fresh its sources were and say so.
Approach
01
Audit the corpus before touching a model
Inventoried every source system, measured document age distribution, and tagged which collections were authoritative versus historical. This drove the metadata schema more than any modelling decision did.
02
Metadata-first chunking
Chunks carry source system, owning team, last-modified date and authority tier. Retrieval filters on these before ranking, so superseded content is down-weighted rather than competing on raw semantic similarity.
03
Agentic retrieval with LangGraph
A planner agent decomposes multi-part questions, dispatches parallel retrieval calls per sub-question, and a synthesis agent composes the answer with inline citations. Single-hop questions skip the planner to keep latency down.
04
Evaluation gate before every deploy
A held-out set of real questions with human-graded answers runs on every change. Retrieval precision and citation correctness must not regress or the deploy is blocked.
Architecture
Ingest
Connectors pull from wiki, ticketing, code host and document storage on a schedule; changed documents are re-embedded incrementally rather than full re-index.
Store
Pinecone for dense vectors with metadata filters; AWS S3 holds the canonical documents and version history.
Retrieve
Hybrid dense plus keyword retrieval, metadata pre-filter, then cross-encoder rerank on the top candidates.
Reason
LangGraph planner and synthesis agents on GPT-5, with a citation validator that rejects any claim whose source chunk does not support it.
Serve
FastAPI endpoint behind the internal auth layer, with per-team access scoping inherited from the source systems.
Results
54%Reduction in internal search time
61%Better retrieval precision
8,000+Queries handled daily
4Source systems unified
100%Answers carry inline citations
What I took from it
The retrieval quality gain came from metadata discipline, not from a bigger model. Swapping the generation model changed the evaluation score by a rounding error; adding authority tiers and recency filters moved it double digits.
RAG system now processes 8,000+ queries daily with 61% better retrieval precision, cutting support workload by 65%. We consider Jillani SofTech a long-term strategic partner, not a vendor.
Sarah ChenVP Engineering, TechVentures Global, Germany
Stack
Orchestration
LangGraphLangChain
Models
GPT-5
Data
PineconeAWS S3
Serving
FastAPIAWS
Facing something similar?
Every engagement starts with a discovery call where we define the KPIs before any code is written. You leave with a plan whether or not we work together.