What an LLM doesn't know
Imagine hiring a brilliant research assistant who has read most of the internet but has been locked in a room with no news, no internet access, and no idea what's in your company's filing cabinets. That's a Large Language Model (LLM) — the AI system behind tools like ChatGPT. It knows an enormous amount in general, but nothing about your specific business, your latest reports, or anything that happened after its training ended.
If you just ask this assistant a question about your company, it will either guess, give you a generic answer, or confidently make something up.
To get useful answers, you need to change how you work with it. Instead of asking blind, you first hand the assistant the right documents to read, then ask your question. In practice, this means building a system that:
Keeps your documents organized in a searchable library. In AI systems, this library is called a "vector database" — a filing system that stores documents based on their meaning, not just keywords, so it can find "the sales report" even if you search for "quarterly revenue numbers."
When you ask a question, quickly searches that library for the handful of documents most relevant to it.
Hands those documents to the assistant along with your question, so it can read them and answer based on what's actually there — not just its fuzzy general memory.
This two-step process — search your library, then let the AI read and answer — is called Retrieval-Augmented Generation, or RAG. It's the backbone of most serious AI applications built today. Instead of a generalist who might bluff, you end up with a specialist who cites its sources and knows the specifics of your world.
How it works
Large Language Models have fundamentally changed how we build software. But moving from a simple demo to a production-ready application requires far more than wrapping an OpenAI API call in a web server. You need orchestration layers, retrieval pipelines, guardrails, and a system design that can handle the unique challenges of AI workloads. What follows is a comprehensive reference for architecting these systems.
Why LLM Applications Are Different
LLM applications break almost every assumption we have from traditional software and even conventional machine learning. Understanding these differences is the first step to designing robust systems. According to an analysis by InfraSketch, the key challenges are:
Nondeterministic Outputs: The same prompt can produce different responses across calls. Your system must be built to handle variability in the format, length, and content of the LLM's output.
High Latency: Unlike traditional services with millisecond response times, a single LLM call can take anywhere from 5 to over 60 seconds. This latency, dependent on the model and the length of the prompt and response, makes synchronous request-response patterns unworkable for complex tasks.
Token-Based Costs: You don't pay per request; you pay per input and output token. A poorly designed system that feeds unnecessary information into the model's context window can cause costs to skyrocket.
Multi-Step Reasoning: A single API call is rarely enough. Real-world problems require the LLM to use tools, retrieve information, analyze results, and iterate. This necessitates a more complex orchestration layer.
Context Window Management: Models have finite memory, or "context windows," ranging from 8,000 to over 200,000 tokens. Your architecture must intelligently decide what information to include in the prompt, how to chunk it, and when to summarize previous turns of a conversation.
These constraints demand a new set of architectural patterns, distinct from both standard web services and classic ML systems. For a deeper look at general system design principles, you can review a complete guide to system design.
The Core Pattern: Retrieval-Augmented Generation (RAG)
Retrieval-Augmented Generation (RAG) is the most widely deployed architecture pattern in production GenAI systems today. It's the dominant approach because it directly solves the LLM's biggest weakness: its static, outdated knowledge.
LLMs are trained on a massive but fixed dataset with a hard knowledge cutoff. They have no access to your private documents, internal wikis, customer data, or any real-time information. RAG solves this by creating a two-step process: retrieve, then generate. At query time, the system retrieves relevant documents from an external knowledge base and injects them into the prompt. This allows the LLM to reason over the provided context instead of trying to recall information it was never trained on.
This provides two crucial benefits:
Knowledge Currency: It solves the problem of outdated knowledge without requiring constant, expensive model retraining.
Grounding: The LLM's answer is traceable back to specific source documents, increasing reliability and allowing for verification.
A production RAG system is composed of two distinct pipelines: the Indexing Pipeline and the Retrieval Pipeline.
The Indexing Pipeline: Preparing Your Knowledge
Before you can answer questions, you must prepare your knowledge base. The indexing pipeline is an offline process that ingests, processes, and stores your data for efficient retrieval.
Load Data: The first step is to load your documents from their source. This could be anything from internal wikis and customer support tickets to legal contracts or product manuals.
Chunking: You cannot fit an entire document library into a single prompt. Documents are broken down into smaller, manageable "chunks." The chunking strategy is critical for RAG performance; chunks must be small enough to be retrieved efficiently but large enough to contain meaningful context.
Embedding: Each chunk of text is then converted into a numerical representation called an "embedding" using a specialized embedding model. This vector captures the semantic meaning of the text.
Store in a Vector Database: These embeddings (along with the original text chunks) are loaded into a vector database. This specialized database is optimized for finding vectors that are "close" to each other in high-dimensional space, enabling lightning-fast semantic search.
The Retrieval Pipeline: Answering the Question
The retrieval pipeline is the real-time, query-time process that generates an answer for the user.
User Query: The process begins when a user submits a query.
Query Embedding: The user's query is converted into an embedding using the same model from the indexing pipeline.
Semantic Search: The system uses the query embedding to search the vector database. It finds the text chunks whose embeddings are most semantically similar to the query's embedding. This is the "retrieval" step. Advanced systems may use "hybrid search," which combines this semantic search with traditional keyword search for better accuracy.
Augment Prompt: The retrieved text chunks are collected and formatted into a context block. This context, along with the original user query, is inserted into a carefully crafted prompt for the LLM.
Generate Response: The LLM receives the augmented prompt and generates a response that synthesizes the information from the retrieved chunks to directly answer the user's question. Because the answer is based on the provided documents, it is "grounded" in your specific data.
Beyond RAG: LLM Agents and Orchestration
While RAG is powerful for knowledge-intensive tasks, some problems require more than just question-answering. They require action. This is where LLM Agents come in. An LLM Agent is a system that uses an LLM not just to generate text, but to reason, plan, and use tools (like APIs) to accomplish a goal.
This introduces the need for an orchestration layer. An orchestrator manages the multi-step reasoning process. For example, if a user asks, "What were our top-selling products last quarter and who was the lead salesperson for each?" an agentic system might:
Plan: The LLM decides it needs to query the sales database and the employee directory.
Tool Use (Step 1): The orchestrator calls a
query_sales_dbtool with the appropriate date range.Tool Use (Step 2): It then calls a
get_employee_infotool using the results from the first call.Synthesize: The LLM receives the data from both tool calls and generates a final, synthesized answer.
This kind of workflow cannot be handled with a single, synchronous API call, especially given the high latency of LLM inferences. The orchestrator must manage this asynchronous, stateful process. Tools like InfraSketch ↗ are designed to help build and manage these complex, multi-step LLM workflows behind the scenes.
What this means in practice
This shift in architecture changes what's expected of developers, what businesses need to decide, and what users actually experience.
For developers: The job now goes well beyond writing a good prompt. A "GenAI Engineer" needs to understand data pipelines, how to split documents into chunks sensibly, how to combine keyword and semantic search, how vector databases work, and how to test whether the whole system is actually giving good answers. This has spawned a growing set of dedicated courses and tools, such as the AI Systems Design: RAG Pipelines and LLM Architecture ↗ course on Coursera.
For businesses: The payoff is AI applications that work with your actual, current data instead of generic knowledge — a customer support bot that knows this week's product catalog, or a legal tool that can read a specific contract. But getting there means choosing between four different approaches, each with a real trade-off:
LLM App Stores: Easy to get started, but you're building on someone else's platform and can be locked in.
LLM Agents: Can act independently and handle multi-step tasks, but there's no agreed-upon way for different agent systems to talk to each other yet.
Self-Hosted LLM Services: You keep full control over your data and system, but you also take on the work of running and maintaining it yourself.
LLM-Powered Devices: Running the AI directly on a device (like a phone) is fast and keeps data private, but you're limited by that device's processing power.
For users: The practical upshot is fewer shrugging, generic answers and fewer confident-sounding guesses. Because the system relies on grounding— basing answers on retrieved documents rather than memory alone — it can show you exactly which document or passage an answer came from. That means you can actually check the AI's homework, which builds trust in a way that a plain, unsourced answer never could.
Where this is heading
The field is moving fast, but a few clear trends point toward where LLM applications are headed: better underlying models, and — more importantly — better ways of connecting the systems built on top of them.
The models themselves keep improving quickly. A 2024 academic paper already looks ahead to 2025 releases such as GPT-5 from OpenAI, Claude Sonnet 4.5 from Anthropic, and Gemini 2.5 Flash from DeepMind. As these underlying models get more capable, the systems built around them — the RAG pipelines and agents described above — inherit those gains almost automatically.
But the harder problems now sit above the model layer, at the level of how these systems are built and connected. A recent paper sketches out a "next frontier" for LLM applications, aimed at fixing two persistent problems: today's ecosystem is fragmented (different tools and agents that can't talk to one another), and it has real security gaps. Reading between the lines, the proposal is really about making things more standardized and interoperable — closer to how the internet settled on shared protocols so different systems could work together. The paper describes a three-layer structure:
Infrastructure Layer: The base layer of models and hardware.
Protocol Layer: A shared middle layer that standardizes how systems talk to each other, exchange data, and handle security — think of it as a common language that lets different agent systems collaborate instead of working in isolated silos.
Application Layer: The top layer where the actual user-facing product lives.
This layered vision is aimed squarely at today's open problems: reducing fragmentation, tightening security, improving scalability, and giving agents a common way to communicate. In short, the next leap in AI won't just come from bigger models — it will come from the connective tissue, the shared rules and protocols, that let all these separate systems work together reliably.