Your First Instinct Is Wrong
Say you've built a chatbot to answer questions about your company's internal documents. You ask it something simple: "How many days of paid time off do new hires get?" The model was trained on public internet text, so it has no idea what your company's policies say — it apologizes and tells you it doesn't have access to private information.
The natural next move, for most developers, is to try to "teach" the model directly. You gather every HR document you can find and start researching how to fine-tune the model — that is, retrain an existing model on your own data so it absorbs company-specific facts.
This instinct is almost always wrong. Fine-tuning is usually the most expensive, slowest, and most brittle way to solve this particular problem.
There's a better way. When a large language model, or LLM — the AI system behind tools like chatbots — doesn't know something, you actually have three options. The trick is to use them in the right order, from cheapest to most expensive: prompt engineering (simply handing the model the facts it needs inside the question itself), Retrieval-Augmented Generation, or RAG (automatically fetching relevant information from a document store before asking the model anything), and, only as a last resort, fine-tuning. RAG in particular hits a sweet spot: the model gets access to outside information without any change to its internal wiring. It's cheaper, faster, and — as one team building a RAG framework put it — "more reliable since the source of information is provided with each response."
The Three Paths to Knowledge
Before diving in, let's clarify the three distinct methods for providing an LLM with information it wasn't trained on. Understanding the difference is the key to building effective and maintainable AI applications.
Prompting: This is the simplest method. You provide the necessary facts directly within the prompt, or the instruction you give the model. The context window—the space for the prompt and its response—acts as the model's short-term memory.
Retrieval-Augmented Generation (RAG): Instead of you manually finding the facts, the system does it for you. When you ask a question, the system first "retrieves" relevant information from an external knowledge base (like company documents, databases, or APIs) and then "augments" the prompt with this information before sending it to the LLM.
Fine-Tuning: This is the only method that actually changes the model itself. It involves taking a pre-trained model and continuing its training on a smaller, specific dataset. This adjusts the model's internal weights to specialize it for a particular task or style.
Think of them as a hierarchy of complexity and cost. You should always start with the simplest method that can solve your problem. In most cases, that means starting with the prompt and quickly moving to retrieval.
Step 1: Just Ask Nicely (Prompting)
The most direct way to give a model information is to put it right in the prompt. If you want it to summarize an article, you include the article's text. If you want it to answer a question about a specific customer, you provide the customer's details.
This technique, sometimes called "in-context learning," works because the model uses the entire prompt as the basis for generating its response. It doesn't have persistent memory between conversations, but it has perfect memory of the context you provide in a single turn.
When It Works
Prompting is perfect for one-off tasks where the required knowledge is small, self-contained, and readily available to the user or application at the time of the request.
Example: "Based on this customer complaint—'The delivery was three days late and the box was damaged'—draft a polite apology email that offers a 15% discount on their next order."
Here, all the necessary information is provided in the prompt. The model doesn't need external knowledge.
When It Fails
The limits of prompting become clear very quickly. As one developer on Hacker News noted, if you just keep prepending history or context, "tokens and cost explode fast."
Prompting breaks down when:
The knowledge is too large: Most models have a finite context window. You can't paste your entire company's wiki into a single prompt.
The knowledge is needed repeatedly: It's inefficient to manually find and paste the same policy document every time a user asks about it.
The user doesn't know what knowledge is needed: The whole point of a corporate oracle is that it finds the information for the user. The user just has a question, not the answer document.
When you hit these limits, it's time to automate the process of finding and adding context. It's time for retrieval.
Step 2: Give the Model a Library Card (Retrieval)
Retrieval-Augmented Generation is the most significant practical advancement in applying LLMs to real-world problems. The core idea is simple: give the LLM a search engine. Instead of expecting the model to have memorized your data, you connect it to a knowledge base and retrieve relevant facts on the fly.
This is the technique behind most modern "chat with your data" applications. Projects like RAGstack aim to make this accessible, allowing a company to deploy an open-source model like Llama 2 or Falcon-7b and connect it to internal knowledge bases like Salesforce or Confluence.
The RAG Workflow
A typical RAG system follows these steps for every user query:
Ingestion (Done once, then updated): All your source documents (PDFs, web pages, database records) are processed, broken into manageable chunks, and converted into numerical representations called "embeddings." These embeddings are stored in a specialized database.
User Query: The user asks a question, e.g., "What is our policy on international travel?"
Retrieval: The system takes the user's query, creates an embedding of it, and uses that to search the database for the most semantically similar chunks of text. This might be a paragraph from the "Employee Travel Policy" document.
Augmentation: The system constructs a new prompt. It combines the original user query with the retrieved text chunks.
Generation: This combined prompt is sent to the LLM, which now has the specific context it needs to answer the question accurately.
The final prompt might look something like this:
Context:
---
Section 4.1 - International Travel Approval:
All international business travel must be approved by the employee's direct manager and the department head at least 14 days prior to the departure date. Travel requests must be submitted through the internal T&E portal. For travel to high-risk countries, additional approval from Corporate Security is required.
---
Question: What is our policy on international travel?
Answer the question based only on the context provided.This approach is powerful because it separates knowledge from reasoning. The LLM's job is to reason, read, and synthesize. The knowledge base's job is to store facts. This separation is why RAG is often "cheaper, faster, and more reliable" than fine-tuning. It's also more trustworthy, as the system can cite the exact source documents it used to form the answer.
The Retrieval Zoo: Vectors, Graphs, and SQL
The "retrieval" step is where most of the innovation is happening. The first and most common method is vector search, but it's not the only one.
Vector Search
This is the workhorse of modern RAG. It uses embeddings to find text that is semantically similar in meaning, not just text that shares keywords. This is how a search for "rules for traveling abroad" can find a document titled "International Travel Policy."
However, this approach has its limits. As one team observed, retrieval from vector databases can be "noisy and loses structure." You might retrieve five chunks of text that all use similar words but contradict each other or lack the relational context needed to answer a complex query — a common failure mode when you're just matching on semantic similarity.
Graph Databases
To overcome the "noisy retrieval" problem, some developers are turning to graph databases. A project called Graphiti, for example, builds "temporal context graphs" for AI agents. Instead of just storing chunks of text, a graph database stores entities (like "Employee," "Policy," "Manager") and the relationships between them.
Crucially, these graphs can also be temporal, meaning they track how facts and relationships change over time. This is purpose-built for agents that need to operate on real-world data that evolves. For a query like, "Who was the project manager for Project Phoenix in May 2025?" a simple vector search might fail, but a temporal graph can provide a precise answer.
Relational Databases (SQL)
The oldest and most established database technology is also finding a new life in the AI era. As one Hacker News comment pointed out, relational databases are a highly practical way to give AI persistent memory. Instead of unstructured text, you can store key information in structured SQL tables:
A table of user preferences (
user_id,preference_key,preference_value).A table of known entities (
entity_id,entity_name,entity_type).A log of key decisions or facts from past conversations.
When the LLM needs to know something, it can be empowered to generate a SQL query to retrieve the exact data it needs from a well-defined schema. This is less "noisy" than vector search but requires that your knowledge can be neatly organized into tables and rows.
Step 3: The Last Resort (Fine-Tuning)
After trying prompting and retrieval, you may still find the model isn't performing as you wish. This is the point where developers are tempted to fine-tune, but often for the wrong reasons.
What Fine-Tuning Actually Does
Fine-tuning is not primarily for teaching an LLM new facts. While some factual knowledge is inevitably baked in during the process, it's an inefficient and unreliable way to do it. If the information changes, you have to run the entire expensive fine-tuning process again.
Fine-tuning is for teaching a model a new skill or style.
You should fine-tune when you want to change the model's behavior, not its knowledge base. It works by showing the model thousands of examples of high-quality prompts and their ideal completions. The model learns the pattern connecting the input to the output.
Good use cases for fine-tuning include:
Adopting a Persona: You want a chatbot that always responds in the voice of a 17th-century pirate or always follows a strict, formal support script.
Mastering a Format: You need the model to reliably output a specific, complex JSON structure or convert natural language into a proprietary query language.
Learning a New Capability: You want to teach the model to summarize legal contracts by extracting specific clauses (Parties, Governing Law, Liability Cap) that it wouldn't know how to identify out of the box. The creator of Screenpipe, an app for building agents from screen recordings, mentioned experimenting with "hundreds of fine-tuned GPT2 models" in the early days to get an AI to understand his notes—a process of teaching a new skill, not just feeding facts.
The True Cost of Fine-Tuning
The primary reason to treat fine-tuning as a last resort is the cost. The team behind Bloop, a code search tool, noted that there is "significant overhead (and expense) in fine-tuning the largest LLMs on private data." This is why they opted for a RAG-based approach using semantic search and chained LLM calls instead.
The costs are not just financial:
Data Curation: You need a large, high-quality dataset of prompt-completion pairs. Creating and cleaning this data is a massive, expert-driven effort.
Training Cost: While cheaper than training a model from scratch, fine-tuning still requires significant computation, especially for large models.
Catastrophic Forgetting: An improperly fine-tuned model can lose some of its general capabilities. By specializing it in one area, you can make it worse at others.
Maintenance: Your fine-tuned model is a fork. When the base model provider releases a new, improved version, you can't just upgrade. You have to re-run your entire fine-tuning process on the new base model, hoping the results are just as good.
The Right Tool for the Job: A Decision Framework
To bring this all together, here is a simple decision framework for how to give your model new knowledge or capabilities. Always start at the top and move down only when necessary.
Is the model's default behavior and style acceptable, and is the required knowledge either self-contained in the query or not needed?
Yes: You don't need anything special. Just use the base model.
Does the model need knowledge it doesn't have, and is that knowledge small enough to be included in the prompt by the user/application?
Yes: Use prompting. Put the context directly in the prompt.
Does the model need access to a large, external, or changing body of factual knowledge to answer questions?
Yes: Use Retrieval-Augmented Generation (RAG).
If your data is mostly unstructured documents (wikis, PDFs, support tickets), start with vector search.
If the relationships between data points and how they change over time are critical, explore graph databases.
If your data is already highly structured in tables, use SQL databases.
After implementing RAG, does the model still fail because it needs to adopt a specific personality, produce a complex structured output, or follow a nuanced, multi-step reasoning process it can't learn from the prompt alone?
Yes: And only now, use fine-tuning. Prepare a dataset of several hundred to a few thousand high-quality examples and specialize the model for its unique behavior.
Follow this order — prompt, retrieve, fine-tune — and the practical difference shows up immediately: updating your system's knowledge becomes as simple as adding or editing a document, rather than re-running a training job every time a policy changes. Fine-tuning still has a place, but only after prompting and retrieval have been tried and found wanting — and even then, it's solving a behavior problem (tone, output format, a specific skill), not a knowledge problem. The thing to keep watching: a retrieval system is only as good as what it retrieves, so as your knowledge base grows, expect to spend real engineering time on that step itself — choosing between vector search, graph databases, and SQL, and tuning how chunks get matched to queries. That's a smaller, cheaper problem than re-training a model, but it isn't a free one.