The Idea We've Simplified Away
Ask a language model a question about something that happened after it finished training, and it will either admit it doesn't know or, worse, make something up with total confidence. A model trained in 2023 will insist that year is still current, because its knowledge is frozen at the moment training stopped. To fix this, engineers need a way to hand the model fresh documents — current events, private company files, anything outside its original training data — at the moment it's asked a question.
The dominant technique for this is called Retrieval-Augmented Generation, or RAG. In its popular form today, it's a two-step process: search a database for documents relevant to the question, then paste those documents into the prompt (the text instruction) you send to the model. It works, and it has become the standard way to make AI systems aware of things they weren't originally trained on. But it's a simplified version of the much more elegant and powerful idea that gave RAG its name.
The 2020 paper that introduced the term "RAG" proposed something different. It wasn't about bolting a search engine onto a language model as two separate pieces. It was about building a single, unified system where the part that searches for documents (the retriever) and the part that writes the answer (the generator) learn together, as one model. The retriever was trained to find documents that were not just similar in wording to the user's question, but specifically useful for the generator to produce a better answer. This feedback loop — where the quality of the final answer actually reshapes how the system searches, over time — has been almost entirely lost in the common RAG implementations of 2025. We've traded the original's depth for convenience.
Parametric and Non-Parametric Memory
To understand the original vision for RAG, you first have to think about models as having two different kinds of memory — a distinction central to the problem RAG was designed to solve.
Parametric Memory: The Knowledge in the Weights
The first kind is parametric memory. This is all the information the language model learned during its initial, massive training phase. Facts about the world, grammatical rules, patterns of reasoning, and stylistic nuances are all encoded implicitly within the billions of parameters, or weights, of the neural network.
This memory has two key properties:
It's fast. Accessing it is just the cost of a forward pass — the standard process of feeding input through the network to get an output.
It's opaque and static. You can't easily trace a specific fact back to its source in the training data. More importantly, you can't update it without expensive retraining. If the model was trained in 2023, it will forever believe that's the current year unless you retrain it.
For a model to say "The current CEO of that company is Jane Doe," that fact must be stored in its parameters. If the CEO changes, the model's parametric memory is now wrong, and fixing it is a monumental task.
Non-Parametric Memory: The Knowledge in the Database
The second kind is non-parametric memory. This is an external, explicit knowledge source that the model can access at inference time. It could be a collection of Wikipedia articles, a company's internal documentation, a database of product specifications, or any other corpus of text.
This memory has the opposite properties:
It's slow (relatively). Accessing it requires an extra step: searching the entire database for the right piece of information before the language model can even begin its work.
It's transparent and dynamic. You know exactly where a piece of information came from (e.g., "document_34.txt"). Crucially, you can add, remove, or edit information instantly without touching the model itself. If the CEO changes, you just update the relevant document in the database.
The goal of any knowledge-intensive system is to get the best of both worlds: the fast, generalized reasoning of parametric memory combined with the dynamic, factual accuracy of non-parametric memory. This is the problem RAG was introduced to solve.
The Original Proposal: A Truly Unified System
In 2020, a paper from a team of researchers at Facebook AI Research (now Meta AI), UCL, and NYU — titled "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" ↗ — laid out a blueprint for doing just that. Their system, RAG, was not two separate boxes connected by an API call. It was conceived and built as a single, end-to-end trainable model.
The architecture has two main components, a retriever and a generator, but the magic is in how they interact.
The Retriever: A user asks a question. The retriever's job is to find a small set of documents from the non-parametric memory (the knowledge corpus) that seem relevant to the question.
The Generator: The generator, a language model, receives the original question and the documents selected by the retriever. It then uses this combined information to generate the final, comprehensive answer.
This sounds familiar, but the crucial detail is that these two components were trained together.
A Retriever That Learns What's Useful
The retriever in the original RAG paper was a dense retriever. Instead of relying on keyword matching, a dense retriever uses a neural network to map both the user's query and all the documents in the corpus into a shared high-dimensional vector space. In this space, the "distance" between vectors corresponds to semantic similarity. To find relevant documents, the retriever embeds the query and finds the document vectors that are closest to it.
But in the RAG paper, the retriever's neural network wasn't pre-trained and frozen. It was an active part of the system that learned and adapted during the training process. The goal wasn't just to find documents that were semantically similar to the query. The goal was to find documents that, when passed to the generator, would help it produce the best possible final answer.
The Power of the Joint Training Feedback Loop
This is where the concept of joint training, or end-to-end training, becomes critical. In machine learning, models learn by a process called backpropagation. After the model produces an output, a "loss function" compares that output to the correct answer to calculate an error. This error signal is then propagated backward through the network, telling each parameter how to adjust itself to produce a better result next time.
In the original RAG architecture, this error signal flowed all the way back from the final generated text.
The system generates an answer.
The loss function calculates how "wrong" the answer is.
The error signal backpropagates through the generator, telling it how to produce better text given the documents it saw.
The error signal continues to backpropagate all the way into the retriever, telling it how to select better documents in the first place.
Imagine the query is "What did the lead scientist say about the discovery?"
A simple semantic retriever might fetch a document that contains the scientist's name and the word "discovery" but only includes biographical information.
The generator, given this document, might struggle and produce a generic or evasive answer like, "The lead scientist commented on the significance of the discovery."
The loss function would register this as a poor answer.
During backpropagation, the RAG system learns a crucial lesson. The error signal tells the retriever, "That document you picked last time? It led to a bad answer. Next time you see a query like this, de-prioritize documents like that one and instead look for one that contains a direct quote."
The retriever learns not just what the query is about, but what the generator needs. It's a true partnership, where both components are optimized for the same final objective: the best possible answer.
The Great Divergence: Today's "Bolt-On" RAG
Fast forward to 2025, and the term "RAG" almost universally refers to a different, much simpler pattern. It is a multi-stage, modular system where the components are developed and operated in total isolation from one another.
The typical workflow looks like this:
Offline Pre-processing: You take a large, pre-trained embedding model off the shelf. You use it to convert your entire document corpus into vectors. You then load these vectors into a specialized vector database for efficient searching. This is a one-time, upfront task.
Online Retrieval (Stage 1): A user query comes in. You use the same off-the-shelf embedding model to convert the query into a vector. You send this vector to your database, which performs an Approximate Nearest Neighbor (ANN) search to find the "closest" document vectors. You fetch these top-k documents.
Online Generation (Stage 2): You write a prompt template that looks something like this: "Given the following documents: [insert documents here], answer this question: [insert query here]". You send this formatted text to a powerful, general-purpose LLM through an API.
This modular, "bolt-on" approach has become the industry standard for a host of practical reasons, but it comes at the cost of breaking the feedback loop that made the original RAG so compelling.
The Missing Connection
In the modern RAG pattern, the retriever and the generator are strangers.
The retriever (the combination of the embedding model and vector database) is optimized on one task: finding documents that are semantically similar to the query. It has no idea what the LLM is, what it knows, or what it finds helpful. Its job ends the moment it returns a list of documents.
The generator (the LLM API) is a black box. It receives a long string of text and does its best to follow the instructions. It has no way of telling the retriever, "These documents were confusing," or "You missed the one document that had the actual answer."
The two stages are connected by nothing more than a copy-paste operation. The intelligence is confined to each stage, with no learning happening between them.
Why Did the Industry Choose This Path?
The divergence wasn't an accident or a mistake; it was a series of rational engineering trade-offs that prioritized practicality over theoretical elegance.
Simplicity and Modularity: The two-stage approach is dramatically easier to build and maintain. You can choose the best vector database, the best embedding model, and the best generator LLM, treating them as interchangeable Lego bricks. You don't need a team of highly specialized ML engineers to manage a complex, end-to-end training pipeline. You just need engineers who can call APIs and manage data infrastructure.
Cost and Specialization: Jointly training a large-scale retriever and generator is enormously expensive and requires a tightly integrated model architecture. The modular approach outsources the most expensive parts. A third party pays the massive cost of pre-training the LLM and the embedding model; you just pay for usage. This democratized RAG, making it accessible to teams without nation-state-level compute budgets.
Flexibility: With the modular approach, you can swap components at will. If a new, better embedding model is released, you can re-index your documents and update your retrieval pipeline without touching your generator logic. If a more powerful LLM becomes available, you can redirect your API calls to the new endpoint with minimal code changes. This is nearly impossible in a deeply integrated, jointly trained system.
The "bolt-on" pattern won because it is good enough for many cases and vastly more practical from an engineering and business perspective.
What We Lost, and What We Might Regain
Simplifying RAG turned it into a practical tool that has unlocked real value across the industry. But it's worth being precise about what got left behind in that trade. The original proposal was for a system that learns to retrieve. The popular pattern we use today is a system that simply searches.
The retriever in the 2020 paper is a trainable part of the model itself, shaped by the same error signal that shapes the generator. The "retriever" in a modern RAG stack is typically a collection of separate, static software: an embedding model that was trained once, elsewhere, for a generic purpose, and a vector database that has no opinion about what the generator finds useful.
This isn't just an academic distinction — it shows up as a practical ceiling on how good these systems can get. Teams spend countless hours on "prompt engineering" and layering on extra retrieval steps like re-ranking (re-scoring the retrieved documents with a second model) and query expansion (rewriting the question to improve the search) to manually close the gap between what the retriever fetches and what the generator actually needs. Much of that manual effort is, in effect, standing in for the automatic learning process that end-to-end training was designed to provide.
The path back to that idea is becoming more realistic. Open-source models are more capable than they used to be, and the tools for fine-tuning them are more mature, which makes jointly training a retriever and generator for a specific domain a more practical option than it was in 2020. This probably isn't worth the engineering cost for a general-purpose chatbot answering arbitrary questions. But for a narrow, high-stakes system — a legal research assistant that must find the one clause that matters, a medical assistant that must surface the relevant contraindication, a financial tool that must catch the exact figure in a filing — a retriever that has actually learned what its generator needs, rather than one that only measures word similarity, could mean fewer missed documents and fewer generic, hedging answers.
The lesson from the original RAG paper isn't that the industry took a "wrong turn." It's that the map is bigger than we remember. The simple, modular RAG we use today is one stop on that map — a pragmatic and powerful one. But the original vision of a deeply integrated, learning-based system points to where the journey could go next.