RAG Query Rewriting: Resolving Follow-Up Questions for AI Chatbots Correctly

A single question like "And how long is that valid for?" is often clear to humans. They remember the previously discussed product, the location, and the intended timeframe. A knowledge search, on the other hand, initially sees only a few words. Without the appropriate conversation context, it might find nothing at all or search for the wrong topic. RAG Query Rewriting solves this problem by transforming a context-dependent follow-up question into a standalone search query before the retrieval step.

This sounds like a small intermediate step, but it frequently determines the quality of a multi-turn website chat. This guide shows how teams resolve follow-up questions, when it is better to ask for clarification, and how they prevent a rewrite from smuggling new facts, wrong permissions, or outdated context into the search.

Why Follow-Up Questions Overwhelm Knowledge Search

The initial user question is usually concrete: "Which warranty applies to Model A?" After that come short phrases like "And for the larger variant?", "Does that apply in Austria too?", or "What do I need for that?". Pronouns, omitted subjects, and references to previous answers are natural in conversation. However, as isolated search queries, they are weak.

A classic keyword, vector, or hybrid search pipeline can only evaluate what it receives as a query. Reranking improves the order of existing hits, but it does not replace the missing meaning of "that" or "for that". Query Rewriting sits ahead of it: it transforms the current question and relevant history into a searchable, standalone query.

What a Good Rewrite Must Accomplish

A successful rewritten query is complete enough for retrieval, yet stays strictly close to the user's intent. For example, "And for Austria?" can become "Which warranty conditions apply to Model A in Austria?" if Model A and warranty are clearly established in the immediately preceding dialogue. The rewrite does not answer the question yet. It serves exclusively to find suitable sources.

Current Azure architectural guidance on Conversational RAG recommends including relevant conversation history and formulating the current question prior to retrieval as a standalone query with resolved references. The separation visible there is crucial: the original user question is preserved for the subsequent response generation. This allows the system to verify whether the retrieved evidence actually fits the question asked.

Supplement, but Do Not Invent

A rewriter may adopt clearly present details: product, version, country, language, or the most recently mentioned process. However, it must not add a missing customer number, specify a guessed product variant, or turn an uncertain timeframe into a concrete date. A helpful-sounding but invented specification reliably sends the search in the wrong direction.

Permissions Stay Outside the Text Model

Tenant, logged-in user, accessible document scope, and roles are determined server-side. They do not belong in the rewrite query as freely formable assertions. The backend applies the corresponding metadata filters separately and immutably. Neither a previous chat message nor a model rewrite should unlock a broader search space.

Context Requires a Conscious Budget

Sending the complete, unfiltered chat history to the rewriter is rarely a good solution. Old topics can overshadow the current question, personal data may be unnecessarily carried forward, and long histories increase latency and costs. As a practical orientation, Microsoft guidance suggests two to five recent conversation turns and a summary of older content. This is not a universal limit, but a starting point for your own tests.

A compact context package can consist of the following components:

• the unmodified current user question,
• a few immediately relevant user and assistant turns,
• already confirmed entities such as product, transaction, or location,
• locale and time zone as technical fields,
• a short-lived, validated summary of older dialogue parts, and
• the version of the rewrite rule, knowledge index, and retrieval configuration.

Actual document permissions remain separated from this. Likewise, unneeded email addresses, order numbers, or complete answers should be removed before the rewrite. A data-frugal history also simplifies later debugging.

A Robust Six-Step Workflow

1. Check self-containment: A clear, new question like "How do I change my password?" can go directly to search. Not every message requires a model rewrite.
2. Detect references: The system flags pronouns, ellipses, comparative words, and pointers like "there", "both", or "the second option".
3. Select relevant context: Only the contributions that plausibly resolve these references are included. An intentional topic change terminates the old context.
4. Decide rewrite vs. clarification: If exactly one resolution is reliable, a standalone search query is produced. If there are multiple plausible meanings, the chatbot asks a brief clarification question.
5. Search and optionally decompose: The query runs through keyword, vector, or hybrid search. Multi-part questions can be broken down into clearly defined sub-queries.
6. Answer against the original question: The answer is generated from the retrieved sources, addresses the original wording, and explicitly states any uncertainty or missing evidence.

Microsoft's overview on Agentic Retrieval describes a related workflow: query and conversation history flow into planning, focused sub-queries are executed in parallel, and results are subsequently merged. Amazon Bedrock documentation similarly highlights planning, iterative sub-queries, and evaluating whether retrieved content suffices for an answer. Such product features can handle parts of the pipeline; however, the custom application's quality and security gates remain necessary.

Rewrite, Clarification, or Query Decomposition?

Input	Appropriate Reaction	Rationale
"And does that apply in Austria?" after an unambiguous warranty question	Formulate a standalone query	Subject and reference are clear.
"What about the other one?" after three variants were mentioned	Ask a brief clarification question	Multiple resolutions are plausible.
"Compare price, delivery time, and returns for both models"	Decompose into focused sub-queries	Multiple independent aspects require reliable search results.
"New topic: How do I reach support?"	Search without old product context	The user signals a topic switch.

Query Decomposition is therefore not the same as Query Rewriting. Rewriting makes a dependent question standalone; Decomposition splits a complex question into multiple search tasks. The Bedrock documentation on Query Decomposition shows that multiple sub-queries can improve coverage. However, each additional query requires a limit, a unified permission model, and a transparent consolidation mechanism.

Treat Rewrite Outputs Like Code

Even if the output is just text, it should conform to a strict contract. A structured object with fields such as standaloneQuery, decision, resolvedReferences, and reason is recommended. Valid decisions include SEARCH_AS_IS, REWRITE, CLARIFY, and DECOMPOSE. The backend validates length, language, and allowed fields before initiating a search.

The rewriter receives no tools and does not reply directly to the user. System instructions from the chat history, inserted document text, or prompts like "Ignore rules" remain data, not control commands. For high-risk search domains, a deterministic rule can additionally enforce that product, locale, or tenant filters never originate from free text.

Testing with a Custom Follow-Up Test Set

Quality cannot be proven through a few successful demos. Extend your existing Golden Set for response quality with real multi-turn dialogues. For each case, record the original history, current question, expected rewrite decision, permitted entities, forbidden additions, and expected sources.

• Pronouns and omitted subjects in short follow-up questions
• Corrections such as "No, I meant Model B"
• Topic changes and returning to a previous topic
• Ambiguous variants that strictly require a clarification question
• Locale, date, and time zone changes
• Unauthorized attempts to switch search scope or tenant
• Long conversation histories containing irrelevant older details
• Multi-part questions that must be decomposed and merged again

Measure independently: Does the rewrite match user intent? Does retrieval find the expected sources? Was clarification requested in cases of genuine ambiguity? Did permission filters remain unmodified? How much additional latency does the step introduce? The NIST AI RMF Core places continuous testing, measurement, and documentation across the entire AI lifecycle. For website teams, this means: only update rewrite rules, models, or context selection with regression testing and observable rollout strategies.

Compact Checklist for Website Teams

• Is the original user question preserved unchanged until response generation?
• Are only relevant and data-frugal parts of the conversation history included?
• Can the rewriter clearly choose between rewriting, asking for clarification, and decomposition?
• Does it supplement strictly confirmed entities without adding assumptions?
• Does the backend enforce locale, tenant, and permission filters independently of the rewrite?
• Does each sub-query have fixed limits on count, execution time, and cost?
• Are retrieval results evaluated against the original question?
• Does a multi-turn test set cover references, corrections, and topic shifts?

Conclusion: Resolve the Search Query First, Then Answer

RAG Query Rewriting transforms natural conversation brevity into a reliable search query. The greatest value stems not from overly creative rewrites, but from clear boundaries: adopt confirmed context, resolve uncertainty via clarification, handle permissions server-side, and continue validating the response against the original question. Start with twenty typical follow-up questions from your support logs, mark the expected decision, and test every change against these exact test cases. This makes a multi-turn chat far more reliable without search silently answering a different question.

Sources

• Microsoft Azure Architecture Center: Prompt Engineering for RAG
• Microsoft Learn: Agentic Retrieval in Azure AI Search
• Amazon Bedrock: Configure Query and Response Generation
• Amazon Bedrock: Agentic Retrieval for Knowledge Bases
• NIST AI Risk Management Framework Core