Armada AI

Support RAG Agent

A grounded question-answering system for Armada’s internal Customer Support team, searching runbooks, help-center content and vendor documentation.

Sole owner, from discussions with the support team and an empty repository through production integration.

Understanding the workflow

Met with the internal Customer Support team before writing code and caught up again after launch to understand their workflows, documentation search habits and pain points.

Problem

Customer Support searched manually across internal runbooks, help-center content and vendor documentation. The system needed to retrieve relevant evidence, understand organizational vocabulary and produce answers grounded in those sources.

My contribution

Designed a CRAG and controllable ReAct workflow in LangGraph, with retrieval evaluation before answer generation. Shared query reformulation and embedding generation across branches to remove redundant calls.

Built a custom BM25 index with incremental updates and removal. Combined sparse and dense retrieval over Qdrant to address organizational vocabulary that dense retrieval missed.

From question to cited answer

Workflow overview

Answer path

  1. Input

    Question

    A support question

  2. Process

    Prepare query

    Shared reformulation and embedding

  3. Process

    Retrieve

    Dense + sparse search

  4. Check

    Check evidence

    Relevance and confidence gating

  5. Output

    Cited answer

    Generation grounded in retrieved sources

Simplified answer path. Source-priority branches and corrective retries are omitted; the arrows show information flow, not elapsed time.

Evaluation

Built two levels of evaluation: retrieval-confidence and CRAG-score gating within the agent, and RAGAS-based response evaluation against curated ground truth. These assess different failure points: finding relevant evidence and producing a grounded answer.

Document processing

Replaced separate ingestion paths with a normalized Markdown intermediate representation. File formats converge on that representation before chunking and indexing.

Replaced single-pass image captioning with classification followed by type-conditioned captioning for document images, including flowcharts and screenshots.

Outcome

Reduced cost per query/run to approximately one-third of the initial graph, comparing similar workloads.

The shared graph node removes repeated reformulation and embedding work. Incremental sparse indexing and the common ingestion representation make knowledge-base changes easier to maintain.

Tools

LangGraph · Qdrant · Hybrid Search · FastAPI · Docker