Both frameworks solve the same problem. Where they differ is in how much they do for you versus how much they let you configure yourself.

Same pipeline, different defaults

Every RAG system — regardless of framework — goes through the same basic stages: load the source documents, break them into chunks, convert those chunks into vectors, store the vectors, then at query time convert the user's question into a vector too, retrieve the closest-matching chunks, insert them into a prompt, and send that prompt to the LLM.

LangChain and LlamaIndex both implement all of this. The difference is how many decisions get made for you by default.

Loading and chunking documents

LangChain gives you a long list of format-specific loaders — one for plain text, one for CSVs, one for JSON, one for web pages — plus a DirectoryLoader that can point at any of them to bulk-load a folder. It's flexible, but you're often picking the right tool from a menu.

LlamaIndex's answer is SimpleDirectoryReader: one loader that natively handles most common formats (PDF, Word, PowerPoint, markdown, and more) and can recursively walk an entire directory tree without much configuration. It's less flexible in the sense that you're not choosing between a dozen options, but for typical use cases it's noticeably less setup.

Chunking looks similar on the surface — both frameworks offer a "smart" recursive splitter (LangChain's RecursiveCharacterTextSplitter, LlamaIndex's SentenceSplitter) that tries to break text at natural boundaries before falling back to harder cuts. Both also offer semantic chunking, which splits based on meaning rather than character count. LlamaIndex even provides a wrapper that lets you use any LangChain splitter inside a LlamaIndex pipeline, if you want LlamaIndex's other defaults but LangChain's specific splitting logic.

Embedding and storing vectors

This is where the philosophy difference becomes concrete. In LangChain, embedding and storing are two separate steps — you generate the vectors, then you store them, and you're expected to wire that together yourself if you want anything beyond the basic in-memory store.

In LlamaIndex, both happen in a single call: build an index from your chunks, and the embedding and storage happen together. It's a smaller amount of code to get something working, though it also means less visibility into each individual step.

LlamaIndex also automatically tracks metadata about each chunk as part of that index. LangChain can do this too, but it's typically something you set up manually, and the exact approach shifts depending on which vector database you're using underneath — a direct consequence of LangChain not having one single, unified vector store class the way LlamaIndex does.

Retrieval and prompt handling

For standard "retrieve the top few most relevant chunks" use cases, the two frameworks are functionally similar. Where they diverge is in the more advanced patterns — LangChain, for instance, has a retriever that can pull back the entire parent document a matching chunk came from, rather than just the isolated chunk, which is useful when a chunk alone lacks enough context to be useful.

The more interesting difference is in how each framework treats prompt customization. In LangChain, augmenting the prompt with retrieved context is its own distinct step, decoupled from everything else — which makes it easy to inspect or rewrite exactly what gets sent to the model. In LlamaIndex, that step is typically bundled together with response generation (or, if you're using a "query engine," bundled together with retrieval and generation). The defaults work well without much tuning, but if you want to heavily customize the actual prompt template, LangChain's more decoupled structure gives you an easier point of entry.

So which one should you use?

Neither framework is objectively better — they're optimized for different priorities:

Reach for LangChain if you want fine-grained control over each step, you're combining RAG with other things (agents, tools, complex multi-step chains), or you're pulling from a wide variety of data source types where LangChain's larger integration ecosystem helps.

Reach for LlamaIndex if you want to get a solid RAG pipeline running with less boilerplate, your use case is squarely "answer questions about documents," and you'd rather lean on sensible defaults than configure every piece by hand.

For my own PDF app, LangChain's separation between retrieval and prompt augmentation actually made it easier to swap in TF-IDF as a stand-in for real embeddings while debugging — I could isolate exactly which step was going wrong without untangling it from response generation. That's a LangChain-shaped advantage more than a universal one, though; a LlamaIndex version of the same app might have taken less code to get to a first working version.

If you're just starting out, I'd say don't overthink the choice — both frameworks are actively maintained, both cover the same fundamental RAG workflow, and skills from one transfer conceptually to the other. Pick whichever one's documentation makes more sense to you on a first read, and switch later if you hit a wall.

TL;DR: I built a small Flask app that lets you upload a PDF and ask it questions, grounded in that specific document — not the model's general training. Under the hood it's a classic RAG pipeline: extract text → split into overlapping chunks → embed each chunk as a vector → retrieve the closest-matching chunks for a question → hand those to the LLM. I tested it on a real government PDF (not a toy example) and it correctly answered a question about Medicare's risk-scoring model — but also retrieved the wrong chunk on a trickier question, a real example of why keyword-based retrieval isn't the same as true semantic search. Code, diagrams, and screenshots below if you want to build your own.

I built a small app this week that does one thing: you upload a PDF, and then you can ask it questions — real questions, answered using the actual contents of that specific document, not a generic response pulled from the model's general training.

To prove it actually works (and not just on a fake example), I pointed it at a real government document — a 4-page PDF from the Centers for Medicare & Medicaid Services on how Medicare risk-adjusts its cost and quality measures. Here it is in action:

Then I asked it something only someone who'd actually read the document would know:

That's a real, running Flask app, answering a real question, grounded in a document it had never seen before I uploaded it. No fine-tuning, no manual pre-loading of facts — just a pipeline that reads the PDF, breaks it into searchable pieces, and hands the relevant pieces to the model when a question comes in.

Here's how it works, and how you can build your own version.

Why you can't just paste the PDF into a prompt

The tempting shortcut is to extract the text and drop the whole thing into your prompt. For a short PDF, that technically works — but it breaks down fast once documents get longer:

Context window limits. Most models cap how much text they can process per call. A 40-page contract won't fit, and even when it does, you're paying for every token on every single question. Signal-to-noise. If someone asks "what's the termination clause," you don't want the model wading through 40 pages to find three relevant sentences — slower, and more likely to miss the point. Reusability. If a user asks five different questions about the same document, you don't want to reprocess the whole thing from scratch every time.

The fix is to process the document once, break it into searchable pieces, and only pull in the pieces relevant to whatever's being asked. This pattern has a name — retrieval-augmented generation (RAG) — and a "PDF Q&A" app is really just the ingestion half of a RAG pipeline with a friendly UI on top.

The pieces involved

File Upload — an endpoint that accepts a PDF and saves it somewhere the app can read it.

Text Extraction — pulling raw text out of the PDF. PDFs aren't structured like plain text files, so this step handles the parsing for you.

Chunking — splitting extracted text into smaller, overlapping pieces. The overlap matters: cut a chunk mid-idea and you risk losing context that spans the boundary between two chunks.

Embeddings — converting each chunk into a vector that captures its meaning, so you can search by meaning instead of exact keyword matches.

Vector Store — a database built to store and search those vectors efficiently. A question gets converted to a vector too, and the store returns whichever chunks are closest to it.

Retrieval + LLM — the closest-matching chunks get handed to the LLM along with the question, so it answers using the actual document instead of guessing from general knowledge.

How it fits together

The key thing to notice: ingestion happens once, when the file is uploaded. Query happens every time someone asks something. You're never re-reading the whole PDF on every question — just searching against chunks you already prepared.

Building it: a Flask PDF Q&A app

Here's the actual pipeline behind the screenshots above, using LangChain and Flask.

Step 1: Accept the upload and extract text

from flask import Flask, request, jsonify 
from langchain_community.document_loaders import PyPDFLoader import os

app = Flask(name) UPLOAD_FOLDER = "uploads" os.makedirs(UPLOAD_FOLDER, exist_ok=True)

@app.route("/upload", methods=["POST"]) def upload_pdf(): file = request.files["file"] path = os.path.join(UPLOAD_FOLDER, file.filename) file.save(path)

loader = PyPDFLoader(path)
pages = loader.load()  # one Document object per page

return process_document(pages)

PyPDFLoader handles the messy work of extracting text from the PDF's internal structure — you get back page-level text objects instead of raw bytes.

Step 2: Split the extracted text into chunks

from langchain.text_splitter import RecursiveCharacterTextSplitter

def process_document(pages):
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,      # characters per chunk
        chunk_overlap=150,    # overlap so context isn't lost at chunk boundaries
    )
    chunks = splitter.split_documents(pages)
    return embed_and_store(chunks)

A chunk_size of 1000 characters is a reasonable starting point — small enough to keep retrieval precise, large enough to preserve context within a chunk.

Step 3: Embed the chunks and store them

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS

vector_store = None  # in a real app, key this by user/session/document id

def embed_and_store(chunks):
    global vector_store
    embeddings = OpenAIEmbeddings()
    vector_store = FAISS.from_documents(chunks, embeddings)
    return jsonify({"status": "PDF processed", "chunks_created": len(chunks)})

FAISS is a simple, local vector store — good for learning and small apps. For production, you'd typically swap in a managed vector database, but the rest of the pipeline stays the same.

Step 4: Answer questions using retrieval

from langchain_openai import ChatOpenAI
from langchain.chains import RetrievalQA

@app.route("/ask", methods=["POST"])
def ask_question():
    question = request.json["question"]

    retriever = vector_store.as_retriever(search_kwargs={"k": 3})  # top 3 matching chunks
    qa_chain = RetrievalQA.from_chain_type(
        llm=ChatOpenAI(model="gpt-4o-mini"),
        retriever=retriever,
    )

    answer = qa_chain.run(question)
    return jsonify({"answer": answer})

as_retriever(search_kwargs={"k": 3}) tells the vector store to return the 3 most relevant chunks for whatever question comes in. RetrievalQA handles the rest — combining those chunks and the question into a prompt, then sending it to the LLM.

Upload a PDF once, then ask questions as many times as you want — each call retrieves fresh, relevant chunks from the same stored document.

Putting it through a real test

I didn't just want to show clean, cherry-picked output, so here's the full run — including where it stumbled.

For retrieval in this demo, I swapped in TfidfVectorizer + cosine similarity from scikit-learn instead of OpenAIEmbeddings, so the whole thing runs with no API key. Same concept — convert text to vectors, compare a question's vector to every chunk's vector — just a simpler, keyword-based version of it.

The empty app, before anything's uploaded:

After uploading the real CMS PDF, it extracted 11,149 characters and split them into 14 chunks — real numbers from the actual PdfReader + RecursiveCharacterTextSplitter run, not placeholders.

First question: "What does the CMS-HCC model use to calculate a beneficiary's risk score?"

Retrieval pulled the exact right chunk — the one explaining that the model generates a risk score from each beneficiary's expected cost of care, with separate logic for new versus continuing enrollees. Correct, grounded answer.

Second question: "How is the 30-day hospital readmission measure risk adjusted?"

This is where it got interesting. The retrieved chunk scored higher (0.526) than the first question's match — but it's the wrong chunk. It's just the list of which measures get risk adjusted, not the paragraph explaining how the readmission measure specifically works. That explanation exists elsewhere in the document but scored lower, because it doesn't repeat the exact phrase "30-day All-Cause Hospital Readmission measure" as densely.

💡

How retrieved chunk score is calculated

TfidfVectorizer turns each chunk — and the question — into a vector of word weights. Each word's weight combines how often it appears in that chunk with how rare it is across all chunks, so common words like "measure" get down-weighted and distinctive words get up-weighted. cosine_similarity then measures the angle between the question's vector and each chunk's vector, producing a score from 0 (no shared distinctive words) to 1 (near-identical word patterns).

The important part: this process has no idea what any of the words mean. It's pure vocabulary overlap. Chunk 4 won because it literally repeats "30-day," "hospital," "readmission," and "measure" — the same words in the question. The methodology chunk explains the identical concept using different words ("beneficiary age," "clinical risk factors," "specialty cohorts") and scored lower purely because the phrasing doesn't match, even though the content does.

A real embedding model (like text-embedding-3-small) doesn't have this blind spot — it maps text into a space learned from patterns across huge amounts of text, where "how is X risk-adjusted" and "risk adjustment accounts for age and clinical factors" land close together even with zero literal word overlap, because the model has learned they're related concepts, not just related words.

That's a real, visible example of a known limitation of keyword-based retrieval: TF-IDF matches on repeated exact terms, not meaning. A true semantic embedding model (like text-embedding-3-small) recognizes that "how is X risk adjusted" and a paragraph describing risk-adjustment methodology are related, even without exact keyword overlap — which is exactly why production RAG systems use embeddings instead of keyword matching.

I could have left this out and just shown the clean success. But the miss is arguably the more useful result — it's a concrete, visible reason to use real embeddings once you move past a demo, instead of just taking that advice on faith.

A few things to watch for if you build this yourself

• k is a tradeoff in retriever = vector_store.as_retriever(search_kwargs={"k": 3}). Too few retrieved chunks and the model might miss the relevant section; too many and you're back to burning tokens on irrelevant text. 3–5 is a common starting range.

• Scanned PDFs won't extract cleanly. PyPDFLoader works on text-based PDFs. For scanned documents or images of text, you'll need OCR (like pytesseract) as an extra step before chunking.

• This demo stores one document globally, which is fine for a single-user proof of concept but won't hold up for multiple users or documents at once. In practice, key each vector store by user session or document ID. Where this fits in the bigger picture

This is the same underlying pattern behind every "chat with your data" product you've probably already used. It's also just the ingestion half of a full RAG system — there's a lot more to explore in retrieval strategies, evaluating answer quality, and knowing when RAG is (and isn't) the right tool for the job.

I'll be digging into those next as I continue learning. If you build your own version of this, I'd like to hear what document you pointed it at, and what it got wrong first.

I'm writing this partly to reinforce what I learned, and partly because I think these five concepts are becoming baseline knowledge for anyone working near AI products — not just ML engineers. If you're evaluating AI tools, managing a team that builds them, or just trying to have an informed opinion in a planning meeting, this is a good place to start.

Here's what stuck with me, with small code examples for each.

1. In-Context Learning

The core insight: an LLM doesn't need to be retrained to handle a new task. You can teach it the pattern you want just by showing examples inside the prompt itself.

This is called few-shot prompting, and it's the simplest form of in-context learning:

python prompt = """ Classify the sentiment of each review as Positive, Negative, or Neutral.

Review: "This laptop exceeded my expectations." Sentiment: Positive

Review: "Arrived broken and support was unhelpful." Sentiment: Negative

Review: "It's fine, does what it says." Sentiment: Neutral

Review: "Best purchase I've made all year!" Sentiment: """

No fine-tuning, no training data pipeline — just a well-structured prompt. The model infers the pattern from the examples and applies it to the new input. This is also why prompt quality matters so much more than people expect: the examples you choose directly shape the model's output.

2. Prompt Templates

Once you're writing more than a couple of prompts, hardcoding strings gets messy fast. Prompt templates solve this the same way parameterized queries solve SQL injection risk and repetition — you separate the fixed structure from the variable inputs.

In LangChain, this looks like:

python from langchain.prompts import PromptTemplate

template = PromptTemplate( input_variables=["product", "audience"], template=( "Write a two-sentence product description for {product}, " "targeted at {audience}. Keep the tone confident and concise." ), )

prompt = template.format(product="a noise-cancelling headset", audience="remote workers")

The template is reusable across any product/audience pair. This becomes essential once your app needs to generate prompts dynamically from user input or database records, rather than from text you typed once and never touched again.

3. Chains

A chain links multiple steps together, where the output of one step becomes the input to the next. Instead of asking a model to do everything in a single giant prompt, you break the task into smaller, testable pieces.

A simple two-step chain: summarize a document, then translate the summary.

python from langchain.chains import LLMChain

summarize_chain = LLMChain(llm=llm, prompt=summarize_prompt) translate_chain = LLMChain(llm=llm, prompt=translate_prompt)

summary = summarize_chain.run(document=long_text) translated = translate_chain.run(text=summary, language="Spanish")

The benefit isn't just cleaner code — it's debuggability. If the final output is wrong, you can inspect each intermediate step to see exactly where it broke, instead of guessing what went wrong inside one massive prompt.

4. Agents

This is where things start to feel less like "calling an API" and more like "building a system." A chain follows a fixed sequence you define in advance. An agent decides, at runtime, which tool or action to use next based on the input it's given.

Conceptually:

python from langchain.agents import initialize_agent, Tool

tools = [ Tool(name="Calculator", func=calculator_tool, description="Use for math"), Tool(name="WebSearch", func=search_tool, description="Use for current events"), ]

agent = initialize_agent(tools, llm, agent="zero-shot-react-description") agent.run("What's the population of Portugal divided by the population of Iceland?")

The agent has to figure out: this requires a search (to find both populations) and a calculation (to divide them) — and in what order. That reasoning-and-tool-selection loop is the shift from "AI feature" to "AI system," and it's the foundation everything in the rest of this certificate builds on.

5. Structured Output

An LLM's natural output is free text. That's fine for a chat window, but useless if another piece of software needs to consume that output. Structured output means constraining the model to return data in a fixed schema — usually JSON — so your application can parse it reliably.

from langchain.output_parsers import StructuredOutputParser, ResponseSchema

response_schemas = [
    ResponseSchema(name="sentiment", description="Positive, Negative, or Neutral"),
    ResponseSchema(name="confidence", description="A score from 0 to 1"),
]

parser = StructuredOutputParser.from_response_schemas(response_schemas)
format_instructions = parser.get_format_instructions()

# format_instructions gets injected into the prompt, telling the model
# exactly what JSON shape to return

parsed = parser.parse(model_output)
# parsed = {"sentiment": "Positive", "confidence": 0.92}

This is the piece that makes it possible to connect an LLM into a real application — a form, a dashboard, a database — instead of just a text box.

Why this matters beyond the course

None of this requires being a machine learning engineer to understand. But as agentic AI moves from research demos into production systems, I think knowing how these pieces fit together is becoming table stakes — for evaluating vendor claims, scoping what's realistic for a project, or just not glazing over when "agent" comes up in a meeting.

I'll be posting similar breakdowns as I move through the rest of the specialization. Next up: retrieval-augmented generation (RAG) and, eventually, multi-agent systems.

If you're working through similar material or have opinions on where LangChain's abstractions help vs. get in the way, I'd like to hear them.

This post covers Course 1 (Develop Generative AI Applications: Get Started) of IBM's RAG and Agentic AI Professional Certificate on Coursera.