Back to Blog

Building a RAG Agent with LangChain

October 13, 2026 (2w ago)

Large language models are impressive, but they have two well-known weaknesses. They don’t know about your data, and when they don’t know something they sometimes make it up. Retrieval-Augmented Generation (RAG) addresses both: before the model answers, you fetch relevant passages from your own sources and hand them over as context.

When I built BONKOUNGOU AI, a web-based AI agent, RAG was the core of it. This post walks through the pattern step by step with LangChain in Python.

The big picture

A RAG system has two phases:

INDEXING (done ahead of time)
  Documents ──▶ Split into chunks ──▶ Embed ──▶ Vector store
 
QUERYING (every question)
  Question ──▶ Retrieve similar chunks ──▶ LLM answers using those chunks

An agent adds one more idea on top: the model decides when to search, and can search several times, instead of retrieving once for every question.

Step 1: Load and split your documents

Models have limited context windows, and smaller chunks give more precise matches. So documents are split into overlapping chunks first:

index.py
from langchain_community.document_loaders import WebBaseLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
 
docs = WebBaseLoader("https://example.com/docs").load()
 
splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,    # characters per chunk
    chunk_overlap=200,  # overlap keeps context across boundaries
)
chunks = splitter.split_documents(docs)

The overlap matters. Without it, a sentence that crosses a chunk boundary is cut in half, and neither half makes sense on its own.

Step 2: Embed and store

Each chunk is turned into a vector (an embedding) that captures its meaning. Chunks about similar topics end up close together in vector space.

index.py
from langchain.embeddings import init_embeddings
from langchain_core.vectorstores import InMemoryVectorStore
 
embeddings = init_embeddings("openai:text-embedding-3-small")
vector_store = InMemoryVectorStore(embeddings)
vector_store.add_documents(chunks)

An in-memory store is fine for prototyping. For production, swap in a persistent store such as pgvector, Chroma or Qdrant. The rest of the code doesn’t change.

Step 3: Turn retrieval into a tool

This step is what separates an agent from a plain RAG chain. Instead of always retrieving, we give the model a tool it can choose to call:

agent.py
from langchain.tools import tool
 
@tool
def search_docs(query: str) -> str:
    """Search the knowledge base for passages relevant to the query."""
    results = vector_store.similarity_search(query, k=4)
    return "\n\n".join(
        f"[Source: {doc.metadata.get('source', 'unknown')}]\n{doc.page_content}"
        for doc in results
    )

The docstring is important: it’s what the model reads to decide when to use the tool. Write it the way you’d brief a colleague.

Step 4: Create the agent

agent.py
from langchain.agents import create_agent
 
agent = create_agent(
    model="openai:gpt-4.1-mini",
    tools=[search_docs],
    system_prompt=(
        "You answer questions using the knowledge base. "
        "Always search before answering factual questions. "
        "If the search results don't contain the answer, say you don't know. "
        "Cite the sources you used."
    ),
)
 
response = agent.invoke(
    {"messages": [{"role": "user", "content": "How do I reset my password?"}]}
)
print(response["messages"][-1].content)

For a simple question the agent searches once and answers. For a harder one, such as “compare the old and new pricing plans”, it may search two or three times with different queries before writing its answer.

Lessons learned

1. Retrieval quality determines answer quality. If the right chunk isn’t retrieved, even the best model can’t answer well. When answers are poor, debug retrieval first by printing which chunks came back.

2. Chunk size is a real trade-off. Small chunks give precise matches but lose context. Large chunks keep context but dilute relevance. Try a few sizes on your actual data.

3. Tell the model it’s allowed to say “I don’t know.” Without that instruction in the system prompt, models tend to fill gaps with plausible-sounding guesses, which defeats the purpose of grounding.

4. Keep sources in the metadata. Citations make answers verifiable, and they help users trust the system.

5. Prompt templates are part of your code. Version them, review them and test them like any other code.

Wrap-up

RAG turns a general-purpose model into an assistant that knows your data, and the agent pattern lets it decide how to look things up. The core comes down to a few steps: split, embed, retrieve, generate.

If you’re building something similar or want to swap notes, find me on GitHub.

GitHub
LinkedIn