Most organizations building with Large Language Models (LLMs — the AI models behind tools like chatbots and text generators) start the same way: they hire a handful of specialists, put them on a single central "AI team," and point them at the company's most ambitious AI project. It works, at first. The team ships something impressive, and leadership takes notice.
Then the requests pile up. Marketing wants an AI tool that writes copy. Customer support wants an AI chatbot. The e-commerce team wants search that understands what shoppers mean, not just the words they type. Every one of these requests lands on the desk of the same small team that built the first feature — and that team doesn't have the hands to build all of it, doesn't know each product area well enough to do it quickly, and spends its days firefighting instead of building anything reusable. Innovation, which sped up at first, grinds to a stop.
The fix is to stop treating AI as a special case that needs its own dedicated team for every project. Instead, treat it the way engineering organizations treat other shared capabilities — as a service delivered by a platform. In this setup, one central platform team builds and maintains the complicated shared plumbing that LLM features need: pipelines that feed the model relevant company data, a single point of access to the underlying AI models, and tools for tracking what it all costs. That team offers these as a reliable, self-service product to the rest of engineering. Product teams — who understand the users and own the user experience — then build features on top of that foundation without needing to become AI infrastructure experts themselves. This split of responsibilities is, in practice, the only way to scale AI development across an entire company.
The Short-Lived Success of the Central AI Team
The pattern is a familiar one. A company decides to invest in generative AI. It hires a handful of machine learning engineers and data scientists, puts them in a new "AI Skunkworks" or "AI Center of Excellence," and tasks them with building a flagship feature.
Initially, this approach is wildly successful. The team is focused, free from the legacy constraints of other departments, and has a direct line to leadership. They ship a proof-of-concept that wows everyone. The company issues a press release. The stock price gets a little bump.
Then the problems begin.
The marketing team wants an AI copy generator. The customer support team wants an AI-powered chatbot. The e-commerce team wants semantic search. All these requests land on the desk of the same small AI team, which was so effective when focused on a single problem and is now buried in tickets.
This model fails for three fundamental reasons:
Lack of Domain Context: The central AI team cannot be experts in customer support, e-commerce merchandising, and B2B marketing funnels. They are forced to spend half their time just learning the basics of the product area they're supposed to be helping, which slows them down and leads to superficial solutions.
No Scalable Foundation: Because they are project-driven, they build bespoke solutions for each request. The pipeline built for the chatbot can't be reused for semantic search. They are constantly reinventing the wheel, accumulating technical debt with every "win."
The Bottleneck Effect: With every product team dependent on them, the AI team becomes a gatekeeper for all AI innovation. Instead of enabling the organization, they become the single biggest constraint on its velocity.
This isn't a new problem. We saw the same pattern a decade ago with "DevOps teams" and "Mobile teams." The lesson we learned then is the same one we must apply now: you cannot scale a capability by concentrating it in a silo. You scale it by turning it into a platform that empowers everyone.
The Platform as an Internal Product
For years, high-performing engineering organizations have been moving towards platform engineering. As Gartner predicted, by 2026 the vast majority of large software companies would have internal platform teams, and we are now living in that reality. The conversation is no longer if you should have a platform team, but how to make it effective.
Successful platform teams operate on a simple but powerful principle: they treat their platform as a product and the company's other developers as their customers. Their goal, as described by platform engineering advocates, is to reduce the cognitive load on product developers by providing a paved road for building, shipping, and running software. This "Internal Developer Platform" (IDP) isn't just a pile of tools; it's a cohesive product with a roadmap, clear documentation, and a focus on developer experience.
According to sources like Humanitec and Scalence, a good IDP provides standardized, self-service capabilities for:
Infrastructure orchestration and application deployment
CI/CD pipelines
Observability, logging, and monitoring
Cost management and showback
Security scanning and policy enforcement
The success of a platform isn't measured by its technical sophistication, but by its adoption. As Martin Fowler notes, a platform team's effectiveness hinges on its ability to collaborate with product teams and get them to willingly use the platform. They are a force multiplier, not a mandate.
When we apply this thinking to LLMs, the path forward becomes clear. The complex, undifferentiated heavy lifting of building with AI is a perfect candidate for a platform service.
What an LLM Platform Actually Provides
An "LLM system" is not a single thing. It's a stack of capabilities. When we assign ownership to a platform team, we are giving them responsibility for providing this stack as a set of reliable, self-service APIs. Product teams become the consumers of these APIs, free to focus on the user-facing problem.
Here is what belongs in the LLM platform.
Model Access and Abstraction
No product team should be hard-coding calls to a specific model provider's API. The platform should own this layer.
What the platform owns: A central gateway service that routes requests to one or more underlying LLMs (whether commercial or open-source). This service manages API keys, handles authentication, implements retry logic and failover, and enforces rate limits. It exposes a stable, internal API, such as
/v1/chat/completions, that mirrors a standard format.What the product team does: They make a simple, authenticated API call to the internal gateway. They specify the capability they need (e.g.,
model="fast-and-cheap"ormodel="powerful-and-creative"), but they don't need to know which specific model version will serve the request. The platform team can swap out the underlying model for a newer, better, or cheaper one without any product team needing to change a single line of code.
Data Engineering for Retrieval-Augmented Generation (RAG)
Most useful LLM applications are grounded in the company's private data using RAG. The infrastructure for this is complex and a classic platform responsibility.
What the platform owns: The entire data pipeline. This includes the vector databases, the embedding models and services, and the data connectors that pull from sources like Confluence, Salesforce, or product catalogs. The platform provides a "RAG-as-a-Service" API. A product team can hit an endpoint like
POST /v1/collections/{collection_name}/documentsto index new data, and another likePOST /v1/collections/{collection_name}/queryto retrieve relevant context for a user's query.What the product team does: They own the data, not the pipeline. The e-commerce team is responsible for providing a clean, up-to-date feed of their product catalog. The support team is responsible for curating their knowledge base articles. They use the platform's self-service tools to manage their specific
collection, but they don't need to know or care what vector database is being used underneath.
Prompt Management and Versioning
Prompts are a new kind of source code. They need to be versioned, tested, and managed.
What the platform owns: A central prompt registry. This is a service where teams can store and manage their prompt templates. The platform can provide features like versioning, RBAC (so the marketing team can't edit the support team's prompts), and basic templating. It might also provide a playground UI for experimenting.
What the product team does: They write, test, and own the prompts for their specific feature. They use the platform's registry to store them. When their application calls the LLM gateway, it might pass a prompt ID (e.g.,
prompt_id="prod-support-chatbot-v3") instead of the full text, allowing the platform to fetch, fill, and execute the correct, versioned prompt.
Cost, Security, and Observability
These cross-cutting concerns are the bread and butter of platform engineering.
What the platform owns: The central tooling for monitoring and control. Every call to the LLM gateway is logged with tags identifying the calling product team. The platform team builds and maintains dashboards in Datadog or Grafana that show latency, error rates, and, most importantly, token consumption and cost, all broken down by team. They also implement universal guardrails, such as PII scrubbing and content filtering, to ensure a baseline of safety and compliance for every feature.
What the product team does: They consume this data. They are responsible for their own budget and for ensuring their feature meets its SLOs. The platform gives them the tools they need to do this without having to build their own monitoring stack.
Where Product Teams and Real Work Fit In
By delegating this infrastructure to the platform, we empower product teams to do what they do best: build great user experiences. In this model, the product team is the customer of the LLM platform. Their responsibilities are clear:
Own the Feature: They own the business problem, the user interface, and the end-to-end application logic.
Write the Prompts: They are the experts in their domain, so they are best equipped to write and refine the prompts that deliver the desired behavior.
Prepare the Data: They are responsible for the quality and freshness of the domain-specific data used in RAG.
Evaluate the Output: They perform the first line of evaluation. Does the feature work? Does it solve the user's problem? Is it better than the non-AI version?
This division of labor clarifies which roles are essential. You need Platform Engineers to build the IDP. You need Product Engineers to build the features. You need Data Engineers, often sitting within or working closely with the platform team, to build the data pipelines. And you need Product Managers, both for the external products and for the internal platform itself, to define the roadmap and ensure the work is valuable.
Notice what's missing: the standalone "AI Engineer" title. In this model, AI skills are diffused. Platform engineers learn to manage LLM infrastructure, while product engineers learn to write effective prompts. It becomes a skill, not a job title, which is how all mature technologies are eventually integrated.
The Uncomfortable Necessity of Independent Evaluation
There is one final, critical piece. A product team that builds a feature is incentivized to see it succeed. When they are also the sole judge of its quality, they have an inherent conflict of interest. They will be reluctant to delay a launch or admit that their "magical" AI feature frequently hallucinates, reveals private information, or gives dangerously wrong answers.
For this reason, a mature organization needs an independent evaluation function.
This is not the platform team, and it is not the product team. It is a small, separate team whose only job is to provide objective, adversarial testing and quality assurance for AI features before they ship. Their mandate is to say "no."
Reporting Structure: This team cannot report to a product leader or a platform leader. To be truly independent, they must report to a higher, neutral authority like the CTO, CISO, or Head of Engineering.
Responsibilities: They maintain "golden datasets" of test cases. They run systematic tests to measure accuracy, bias, robustness, and safety. They perform red-teaming exercises to find failure modes.
Deliverable: Their output is not code; it is a go/no-go recommendation backed by data. "This feature is cleared for launch," or "This feature fails 15% of safety tests and cannot be released until fixed."
This function is non-negotiable for any company operating at scale or in a regulated industry. It is the safety brake that allows the rest of the organization to move fast.
A Worked Example: AI-Powered E-Commerce Search
Let's see how this works in practice. A product team wants to replace their keyword-based site search with a new, LLM-powered semantic search.
The Product Team starts by defining the user experience. They design the UI and the flow. They decide that the core of the feature will be a prompt that takes the user's natural language query (e.g., "warm jacket for hiking that isn't too bulky") and finds relevant products. They write a v1 of this prompt template. They also work with their business analyst to provide a clean data feed of the entire product catalog.
They use the LLM Platform's self-service tools. They create a new RAG collection called
product-catalogand use the platform's API to start indexing their data feed. They store their prompt template in the platform's prompt registry, getting back an ID:ecomm-semantic-search-v1.The product engineers write the application code. A user's query comes into their backend service. The service makes two calls to the LLM Platform:
First, it calls the RAG service:
POST /v1/collections/product-catalog/querywith the user's query. The platform returns the top 5 most relevant product descriptions.Second, it calls the chat completions gateway:
POST /v1/chat/completionswith the prompt ID and the context retrieved from the RAG service. The platform handles fetching the prompt, inserting the context, and returning the LLM's final answer.
Before launch, the Independent Evaluation Team steps in. They test the feature against a curated list of 1,000 tricky queries. They find that for queries mentioning out-of-stock items, the model sometimes hallucinates and claims they are available. They issue a "no-go" report.
The Product Team iterates. They update their prompt to explicitly instruct the model to state when an item is out of stock. They push v2 of the prompt to the registry. The evaluation team re-runs the tests, which now pass. The feature is approved for launch.
Post-launch, the product team monitors the dashboards provided by the LLM Platform to track latency, errors, and, crucially, the cost of their new feature.
Notice what happened in each step: the platform team never touched a prompt, and the product team never had to think about vector databases or rate limits. Each group did the work only they were positioned to do well — the platform kept things fast and consistent, the product team shipped something users actually wanted, and the evaluation team caught a real failure (hallucinated stock availability) before it reached a customer. That division of labor, repeated across ten or fifty product teams instead of one worked example, is what scaling actually looks like.
The org chart, not the technology, is usually the thing holding a company back. If your AI efforts are stuck, the fix is rarely to hire more AI specialists — it's to decide which parts of the work belong on a shared platform and which parts belong with the people who know the customer. Get that split right, and the same handful of platform engineers can support ten product teams instead of being the bottleneck for all of them.