Chatterbox — RAG Chatbot

A production-ready Retrieval-Augmented Generation (RAG) chatbot. Upload PDFs, text files, or web pages and ask questions about them. Answers are grounded in your documents with source attribution and confidence scoring.

Features

• Hybrid search — vector similarity + BM25 keyword search fused with Reciprocal Rank Fusion (RRF)
• HNSW index — via Pinecone (managed), ChromaDB (local, hnswlib), or Elasticsearch
• Three chunking strategies — Semantic (embedding-based breakpoints), Recursive (separator hierarchy), Document-Structure-Aware (heading/section tree)
• Cross-encoder reranking — BAAI/bge-reranker-large re-scores retrieved candidates for precision
• Self-correction (CRAG) — reformulates the query and retries retrieval when confidence is low; flags low-confidence answers to the user
• HyDE — optional Hypothetical Document Embeddings to improve recall
• Streaming responses — token-by-token SSE stream from FastAPI to the browser
• Fully configurable — swap any provider (embedder, vector store, LLM, chunker) with a single env var, zero code changes
• HuggingFace-first with OpenAI as fallback for both embeddings and chat

---

Stack

Layer	Technology
Frontend	Next.js 15 · TypeScript · Tailwind CSS
Backend	FastAPI · Python 3.11 · uvicorn
Embeddings	BAAI/bge-large-en-v1.5 (HF, default) · text-embedding-3-small (OpenAI)
Reranker	BAAI/bge-reranker-large
Chat LLM	meta-llama/Llama-3.1-8B-Instruct:novita via HF Inference API · gpt-4o-mini (OpenAI)
Vector Store	ChromaDB · Pinecone · Elasticsearch
BM25	rank-bm25 · persisted to disk with pickle

---

Quick Start

Prerequisites

• Python 3.11+
• Node.js 20+
• A HuggingFace API key (free tier works) and/or an OpenAI API key

1. Backend

cd backend

# Copy and fill in secrets
cp .env.example .env

# Install dependencies (choose one)
pip install -e .          # standard pip
# or
uv sync                   # if you use uv

# Start the API server
uvicorn app.main:app --reload --port 8000

The API will be available at http://localhost:8000.
Interactive docs: http://localhost:8000/docs

2. Frontend

cd frontend

# Copy and configure the API URL
cp .env.local.example .env.local

npm install
npm run dev

Open http://localhost:3000.

3. Docker Compose (all services including Elasticsearch)

cp backend/.env.example backend/.env
# Edit backend/.env — add your API keys

docker compose up --build

Services started:

• frontend → http://localhost:3000
• backend → http://localhost:8000
• elasticsearch → http://localhost:9200 (only needed for the Elasticsearch vector store)

---

Configuration

All behaviour is controlled by backend/.env. No code changes are required to switch providers — edit the env file and restart the server.

# ── Chunking ──────────────────────────────────────────────────
CHUNKER_TYPE=semantic          # semantic | recursive | document_structure
CHUNK_SIZE=512
CHUNK_OVERLAP=50
SEMANTIC_BREAKPOINT_PERCENTILE=95.0

# ── Embeddings ─────────────────────────────────────────────────
EMBEDDER_PROVIDER=huggingface  # huggingface | openai
EMBEDDER_MODEL=BAAI/bge-large-en-v1.5
EMBEDDING_DIM=1024

# ── Vector Store ───────────────────────────────────────────────
VECTOR_STORE_PROVIDER=chroma   # chroma | pinecone | elasticsearch

# ── LLM ────────────────────────────────────────────────────────
LLM_PROVIDER=huggingface       # huggingface | openai
LLM_MODEL=meta-llama/Llama-3.1-8B-Instruct:novita

# ── Retrieval ──────────────────────────────────────────────────
RETRIEVAL_TOP_K=20             # candidates fetched before reranking
RERANKER_TOP_K=5               # kept after reranking
HYBRID_ALPHA=0.7               # 1.0 = pure vector, 0.0 = pure BM25

# ── Self-correction ────────────────────────────────────────────
ENABLE_SELF_CORRECTION=true
RELEVANCE_THRESHOLD=0.4
MAX_CORRECTION_RETRIES=2

# ── Optional enhancements ──────────────────────────────────────
ENABLE_HYDE=false              # Hypothetical Document Embeddings

Common provider switches

# Use OpenAI for everything
EMBEDDER_PROVIDER=openai
OPENAI_API_KEY=sk-...
LLM_PROVIDER=openai

# Use Pinecone instead of ChromaDB
VECTOR_STORE_PROVIDER=pinecone
PINECONE_API_KEY=...
PINECONE_INDEX_NAME=chatterbox

# Use Elasticsearch
VECTOR_STORE_PROVIDER=elasticsearch
ELASTICSEARCH_URL=http://localhost:9200

Note: Switching embedder providers requires re-ingesting all documents because the vector spaces are incompatible. Update EMBEDDING_DIM to match the new model's output size.

---

API Reference

Method	Path	Description
POST	/chat	SSE streaming chat response
POST	/ingest/url	Ingest a web page by URL
POST	/ingest/file	Upload and ingest a file (PDF, TXT, MD, …)
GET	/ingest/status/{task_id}	Poll background ingestion task status
GET	/config	Return current active configuration
GET	/health	Health check

Chat request

POST /chat
{
  "query": "What is the main finding of the paper?",
  "history": [
    { "role": "user", "content": "..." },
    { "role": "assistant", "content": "..." }
  ],
  "namespace": "default",
  "filter": null
}

SSE event types

data: {"type": "sources", "data": [{...}, ...]}   ← retrieved documents (first)
data: {"type": "token",   "data": "Hello"}         ← streamed tokens
data: {"type": "meta",    "data": {"low_confidence": false, "reformulations": 0}}
data: {"type": "error",   "data": {"message": "..."}}

---

Project Structure

chatterbox/
├── backend/
│   ├── app/
│   │   ├── core/
│   │   │   ├── config.py           # Pydantic Settings — single source of truth
│   │   │   └── factory.py          # Component factory — maps config → implementation
│   │   ├── ingestion/
│   │   │   ├── loaders/            # PDFLoader, TextLoader, WebLoader
│   │   │   └── chunkers/           # SemanticChunker, RecursiveChunker, DocumentStructureChunker
│   │   ├── embeddings/             # HuggingFaceEmbedder, OpenAIEmbedder
│   │   ├── vector_stores/          # PineconeStore, ChromaStore, ElasticsearchStore
│   │   ├── retrieval/
│   │   │   ├── bm25_index.py       # rank-bm25 with pickle persistence
│   │   │   ├── hybrid_search.py    # RRF fusion
│   │   │   ├── reranker.py         # cross-encoder reranking
│   │   │   └── self_correction.py  # CRAG — evaluate → reformulate → retry
│   │   ├── llm/                    # HuggingFaceLLM, OpenAILLM (both streaming)
│   │   ├── pipeline/
│   │   │   ├── ingestion_pipeline.py   # load → chunk → embed → store
│   │   │   └── rag_pipeline.py         # HyDE → hybrid → rerank → CRAG → stream
│   │   └── api/
│   │       ├── routes/             # chat.py (SSE), ingest.py (background tasks)
│   │       └── schemas.py          # Pydantic request/response models
│   ├── Dockerfile
│   ├── pyproject.toml
│   └── .env.example
├── frontend/
│   └── src/
│       ├── app/                    # Next.js App Router pages + layout
│       ├── components/
│       │   ├── ChatInterface.tsx   # Main layout with collapsible sidebar
│       │   ├── MessageBubble.tsx   # Streaming message with cursor animation
│       │   ├── SourceCard.tsx      # Expandable source with relevance score
│       │   ├── FileUpload.tsx      # Drag-and-drop + progress bar + status polling
│       │   └── UrlIngestion.tsx    # URL ingest form with async polling
│       ├── hooks/
│       │   └── useStreamingChat.ts # SSE consumer, message state, abort
│       └── lib/
│           └── api.ts              # Typed fetch wrappers for all API endpoints
├── docker-compose.yml
└── README.md

---

Chunking Strategies

Semantic Chunking (default)

Splits text based on topic shifts detected by embedding similarity.

1. Split text into sentences
2. Embed every sentence with the configured model
3. Compute cosine similarity between each consecutive pair
4. Find breakpoints where similarity drops below the Nth percentile (configurable)
5. Merge sentences between breakpoints into chunks

Best for: unstructured prose, articles, reports.

Recursive Chunking

Hierarchical separator splitting: \n\n → \n → . → with configurable overlap. Fast and deterministic — no embedding cost at ingestion time.

Best for: when speed matters or when you don't have a GPU.

Document-Structure-Aware Chunking

Parses the document hierarchy (Markdown # headings, HTML <h1>–<h4> tags, PDF TOC bookmarks) and chunks at structural boundaries. Each chunk is prefixed with its full breadcrumb path:

Chapter 3 > 3.2 Methodology > Data Collection

We sampled 500 users from…

Best for: technical docs, manuals, academic papers with clear section structure.

---

Self-Correction (CRAG)

Query
  ↓
Hybrid Search (vector + BM25)
  ↓
Cross-Encoder Reranking
  ↓
Evaluate: max_score >= relevance_threshold?
  ├── YES → use results
  └── NO  → LLM reformulates query → retry (up to max_correction_retries)
              └── still low? → proceed with low_confidence flag
  ↓
Stream answer to client
  └── low_confidence? → show warning badge in UI

---

Hybrid Search (RRF)

Vector and BM25 results are merged using Reciprocal Rank Fusion without requiring score normalization:

rrf_score(doc) = alpha × 1/(60 + vector_rank) + (1-alpha) × 1/(60 + bm25_rank)

Documents appearing in both result lists receive a strong combined boost.

---

Development Tips

• Local dev with no API keys: set VECTOR_STORE_PROVIDER=chroma (no key needed) and LLM_PROVIDER=huggingface with a free HF token.
• Speed up ingestion: set CHUNKER_TYPE=recursive to skip sentence-embedding during chunking.
• Tune retrieval: increase RETRIEVAL_TOP_K for higher recall before reranking; decrease RERANKER_TOP_K to pass less context to the LLM.
• Namespaces: use the namespace field to isolate document sets per user or project without creating separate indexes.