RAG Failure Diagnostics & Architect

This skill applies a structured debugging framework to RAG systems that return confident but wrong answers. You describe your setup — the corpus, chunking strategy, embedding model, retrieval pipeline, and a failing query or question type — and the skill classifies the failure first (retrieval miss, generation hallucination, structural incompatibility with vector search) before recommending anything. It operates in three modes: DIAGNOSE for a specific bad query, ARCHITECT for choosing the right retrieval shape for a given question type, and SCHEMA for designing the institutional-memory layer that embeddings permanently discard.

A typical input: 'Our RAG system answers questions about internal engineering decisions. When users ask why a particular architecture was chosen two years ago, it retrieves plausible documents but stitches together a confident, fabricated rationale. Chunking is paragraph-level, embeddings are OpenAI text-embedding-3-small, no reranker.' The skill identifies this as a causal-chain failure — the answer requires decision provenance, not semantic proximity — and routes it to DIAGNOSE and ARCHITECT modes rather than chunk-size tuning.

The output for that input would include: Failure mode — 'Causal/provenance miss: the rationale was never stored as a traversable relationship; top-k retrieval cannot reconstruct it.' Structural cause — 'Vector search flattens decision context into token proximity; reranking cannot recover what was never indexed.' Architect recommendation — 'Decision-provenance graph with typed edges (decision → option → rationale → author → date); plain RAG is a structural mismatch for this question class.' Remediation plan — ranked steps from schema design to query routing, with effort estimates and the explicit 80% boundary where tuning alone would stop helping.