Copilot RAG Customization Patterns
RAG (Retrieval-Augmented Generation) done right for enterprise Copilot scenarios. Grounding data + hybrid retrieval + reranker + evaluation framework.
Frequently Asked Questions
What is RAG and how does it relate to Copilot?
RAG (Retrieval-Augmented Generation) is the pattern where an LLM query is augmented with retrieved documents before generation — so the LLM's answer is grounded on your data rather than only its training data. Microsoft 365 Copilot's Business Chat is a productized RAG system: user query → SharePoint/OneDrive/Outlook semantic index retrieval → augmented prompt → GPT response with citations. Custom RAG becomes relevant when you need to ground on data sources Copilot's built-in retrieval doesn't reach — external SaaS (Salesforce, Workday, ServiceNow), custom internal databases, industry-specific corpora, or specialized document formats.
When do we need custom RAG vs use built-in Copilot?
Built-in Copilot Business Chat covers the M365 corpus (SharePoint / OneDrive / Outlook / Teams / Loop / Viva Engage). Custom RAG when: (1) grounding data lives outside M365 and connector-based integration isn't sufficient. (2) The retrieval strategy needs domain-specific tuning (legal case law, medical literature, engineering specs). (3) You need control over the retrieval + reranking + prompting layer (evaluation frameworks, citation formatting, compliance-specific grounding rules). (4) You need retention + audit control that's tighter than Copilot's built-in eDiscovery integration.
What retrieval strategy works for enterprise RAG?
Hybrid retrieval — dense vector search + BM25 keyword search — with a reranker layer. Dense-only retrieval misses exact-term matches (e.g., product SKUs, contract numbers, code identifiers). Keyword-only retrieval misses semantic paraphrases. Hybrid catches both, then a cross-encoder reranker (Cohere Rerank, Voyage Rerank, or Azure AI Search's built-in semantic reranker) reorders the top-k results by true relevance. Best-in-class enterprise RAG deployments typically use dense embeddings + BM25 + reranker in that order. Skipping the reranker is the #1 quality problem in DIY custom RAG.
What is the enterprise search integration story?
Two paths. (1) Copilot Studio agents that plug into custom RAG via HTTP Custom Connectors — the agent handles the conversational surface, RAG endpoint handles retrieval + generation. Common pattern for business-user-facing scenarios. (2) Azure AI Search as the RAG backbone with Copilot Studio agents on top — Azure AI Search provides hybrid retrieval + semantic reranker natively; Copilot Studio provides the conversation. This is the pattern EPC Group deploys for enterprise use cases where Copilot's built-in retrieval doesn't reach the required data sources.
How do we evaluate RAG quality?
Four evaluation categories. (1) Retrieval quality — does the retriever return the right documents for a query? Measured as Recall@K + Mean Reciprocal Rank on a labeled test set. (2) Faithfulness — does the LLM answer stay grounded in the retrieved documents (no hallucination)? Measured via LLM-as-judge on Faithfulness score. (3) Answer relevance — does the answer address the user question? Measured via LLM-as-judge. (4) Citation accuracy — do the citations link to the actual source passages? Measured by human review + automated citation-passage overlap. Frameworks: Ragas, DeepEval, or Azure AI Evaluation. Deploying RAG without an evaluation framework is deploying a black box.
What are the common pitfalls?
Five. (1) Chunking too small (loses context) or too large (dilutes relevance). Best practice: 400-800 tokens per chunk with 15-20% overlap. (2) Ignoring the reranker — dense-only retrieval consistently underperforms hybrid + reranker on enterprise data. (3) No permission enforcement — RAG surfaces documents the requesting user should not have access to. Fix: filter retrieval results by ACL BEFORE reranking. (4) Stale index — grounding data changes but the index rebuild cadence is monthly. Fix: incremental indexing with change-tracking. (5) No evaluation framework — impossible to know if changes improve or degrade quality.
Talk to a senior architect
Email contact@epcgroup.net or call 888-381-9725.
North America's oldest continuous Microsoft Gold Partner (2000 until Microsoft retired the program in 2022) — today holding all six Microsoft Solutions Partner Designations.