Traditional RAG not working

[alaap001]

Hey,
Thanks for the good work.

I noticed that the traditional rag is not working as expected.

Problem statement: A lot of SOTA open source llms don't use tool support, making Agentic RAG use a limitation. However, we can use traditional RAG on them and still get some relevant context. So I tried to use Traditional RAG on gpt4o and few open source models.

Observation: Wheen setting search knowledge = True, our Agent is able to generate proper SQL queries using schema from knowledge base as expected. However when I set it to False and set add context = True, Agent is not able to add any context to prompt at all and is not able to access knowledge base.

Code for reference:

from phi.agent import Agent
from phi.model.openai import OpenAIChat
from phi.utils.pprint import pprint_run_response
from phi.knowledge.text import TextKnowledgeBase
from phi.vectordb.pgvector import PgVector
from phi.vectordb.search import SearchType
from phi.embedder.openai import OpenAIEmbedder
import json
from phi.model.ollama import Ollama
from dotenv import load_dotenv
from phi.embedder.ollama import OllamaEmbedder

load_dotenv()

knowledge_base: TextKnowledgeBase = TextKnowledgeBase(
    path="knowledge",
    
    # chunking_strategy=MarkerChunking(),
    vector_db=PgVector(
        table_name="sql_knowledge",
        db_url=url,
        # search_type=SearchType.hybrid,
        search_type=SearchType.vector,
        embedder=OpenAIEmbedder(model="text-embedding-3-small"),
        # embedder=OllamaEmbedder(model="mxbai-embed-large",dimensions=1024)
    )
)

# knowledge_base.load(recreate=True, upsert=True)

query_generator: Agent = Agent(
    name="SQL Query Generator",
    
    description="""You are a highly skilled SQL Expert. You will use available knowledge base context to get detailed context about the shema of database you're working with. 
    Your sole responsibility is to craft precise and efficient SQL query to retrieve the data required to answer the user's questions by getting schema from knowledge base.""",

    knowledge=knowledge_base,
    search_knowledge=False,
    add_context = True,
    debug_mode=True,
    # show_tool_calls=True,
    # read_chat_history = True,
    # tool_call_limit = 12,
    # model=Ollama(id="phi4:14b-q8_0"),
    # model=Ollama(id="qwen2.5:32b-instruct-q8_0"),
    # model=Ollama(id="qwen2.5:72b-instruct-q8_0"),
    model=OpenAIChat(id="gpt-4o-mini", temperature=0.2),
    # response_model=SQLQuery,
    # structured_outputs=True,
    markdown=True
)

from rich.prompt import Prompt

question = Prompt.ask(
        "[bold]Enter your SQL analysis question[/bold]\n✨",
        default="Who are my most efficient operators based on average case setup time?"
)

# question = "use knowledge base context and tell me "+ question

# question = "Who are my most efficient operators based on average case setup time?"

query_generator.print_response(question, stream=True)

for add context=True and search knowledge=False, it is not adding any context to prompt, prompt is as it is and then it is assuming column and table names from the question instead of from knowledge base.

On the other hand when search knowledge is True, it is referencing to the knowledge base and able to generate correct query.

Logs for traditional RAG:

✨ (Who are my most efficient operators based on average case setup time?):                                                                                                                                                                             
DEBUG    *********** Agent Run Start: b0ac4f9a-3e9a-499e-9d8c-b4709677a5fc ***********                                                                                                                                                                  
DEBUG    ---------- OpenAI Response Start ----------                                                                                                                                                                                                    
DEBUG    ============== system ==============                                                                                                                                                                                                           
DEBUG    ## Instructions                                                                                                                                                                                                                                
         Use markdown to format your answers.                                                                                                                                                                                                           
DEBUG    ============== user ==============                                                                                                                                                                                                             
DEBUG    Who are my most efficient operators based on average case setup time?                                                                                                                                                                          
                                                                                                                                                                                                                                                        
DEBUG    ============== assistant ==============                                                                                                                                                                                                        
DEBUG    To identify your most efficient operators based on average case setup time, you'll need to analyze the setup times recorded for each operator. Here's how you can approach this task:                                                          
                                                                                                                                                                                                                                                        
         random details...                                          

         If you share the specific data or a sample of your setup times, I can help you with more detailed analysis or guidance!
                                                                                                                        
DEBUG    **************** METRICS START ****************      
DEBUG    * Time to first token:         0.3971s               
DEBUG    * Time to generate response:   4.6745s               
DEBUG    * Tokens per second:           51.5567 tokens/s      
DEBUG    * Input tokens:                34                    
DEBUG    * Output tokens:               241                   
DEBUG    * Total tokens:                275                   
DEBUG    * Prompt tokens details:       {'audio_tokens': 0, 'cached_tokens': 0}                                             
DEBUG    * Completion tokens details:   {'audio_tokens': 0, 'reasoning_tokens': 0, 'accepted_prediction_tokens': 0, 'rejected_prediction_tokens': 0}                                                                                                    
DEBUG    **************** METRICS END ******************      
DEBUG    ---------- OpenAI Response End ----------            
DEBUG    Added 2 Messages to AgentMemory                      
DEBUG    Added AgentRun to AgentMemory                        
DEBUG    --**-- Logging Agent Run                             

Can you please test this at your end and update?

If Traditional RAG doesn't work on a simple use case such ass this then this an be a big trouble as that is the most common usecase for any Agentic application.

Thanks.

[alaap001]

FYI, I tried add_references=True and that works but nowhere in cookbooks do I see add_references = True, also what does it even do as compared to add_context? how is it different?

If add_references work, then what is add_context used for?

[manthanguptaa]

@alaap001 add_references when used adds all the relevant documents from the knowledge base to the user prompt for the model. add_context on the other hand adds the context you give in the context param of the agent in the user prompt.

[alaap001]

That's what I am saying, add_context is not doing anything. Model is not able to use it at all.
Can you please check this once, I have attached everything in question.

In the cookbook, I saw knowledge and add context being used, but that is not working

Basically this combo isn't working for RAG, which is what I saw in cookbook.

agent = Agent(
    model=OpenAIChat(id="gpt-4o"),
    knowledge=knowledge_base,
    # Enable RAG by adding context from the `knowledge` to the user prompt.
    add_context=True,
    # Set as False because Agents default to `search_knowledge=True`
    search_knowledge=False,
)

[manthanguptaa]

@alaap001 add_context=True works when you add context param to the agent config. For example

agent = Agent(
    model=OpenAIChat(id="gpt-4o"),
    knowledge=knowledge_base,
    context={"user_id": "user_id"}
    # Enable RAG by adding context from the `knowledge` to the user prompt.
    add_context=True,
    # Set as False because Agents default to `search_knowledge=True`
    search_knowledge=False,
)

[alaap001]

Thanks, I'll check this out and report soon.

And if so, can we please add this to the docs or cookbook?

https://github.com/phidatahq/phidata/blob/main/cookbook/rag/01_traditional_rag_pgvector.py

it says, Enable RAG by adding context from the knowledge to the user prompt. but this isn't happening atm.
I thought if knowledge is provided it'll automatically search for relevant stuff from knowledge base and add it as context