How does RAG query affect the similarity search?

Question

I have a RAG pipeline where I want to extract a piece of information called "X" In a regular RAG pipeline, there is a query entered by the user. Then, this query will be embedded, and the resulting embedding vector will be compared by some metric (cosine similarity) to other embeddings of the saved documents.

If I write the query like this: "What information does this document contain about X?". The result from the similarity search should be worse than using a query containing just "X"

My question is: why is the entered query in a question form? And if it is not in question form, will it produce better or worse results, and why?

Answer 1

The query is entered in question form because it is easy.

There are different variations of what you embed to compute the similarity search. Another variation is to let a LLM hallucinate a few answers to the question and use them to compute similarity search to get the context for the actual answer to be computed by the LLM. This also poses problems, as the hallucinated answers may not be similar to the actual one, and may be appropriate by some more general domains but not so appropriate for others. With many queries, you get more potential pieces of context to pass in the limited LLM window, so you may have then to summarize or to rank them. Also, you are making the system slower and more expensive by using the LLM at least twice.

So it's a trade off. Entering the question as-is is easy and does not complicate things further.

Normally, when creating a RAG pipeline you test many different combinations taking into account the peculiarities of your domain and choose the best performing approach.