Self-hosting an LLM interface.
What the architecture looks like once you move an AI chat tool off someone else's servers and onto your own.
Why teams self-host in the first place
Most teams that look into self-hosting an AI chat interface are trying to solve one of three problems: they don't want prompts and documents leaving their own infrastructure, they want a fixed hosting cost instead of a per-seat SaaS bill that scales with headcount, or they need to route requests to a specific model provider (or a locally-run model) that a hosted tool doesn't support. None of these are exotic requirements, but they do change the shape of the system you're running compared to just signing up for a hosted product.
The moving parts
A self-hosted LLM interface is really three separate concerns wired together: the frontend chat UI, an application layer that manages users, conversations and permissions, and one or more model backends that actually generate responses. The application layer is usually the part people underestimate. It has to store conversation history somewhere durable, authenticate users, and decide which models a given user or group is even allowed to call — that last piece matters more once more than one person is using the same deployment.
The model backend is the part with the most variation. Some deployments route entirely to hosted APIs (calling out to a model provider over HTTPS with an API key), which is the simplest option operationally but means data still leaves your infrastructure per request. Others run inference locally with something like Ollama or vLLM on GPU hardware, which keeps everything in-house but adds real operational weight: GPU provisioning, model updates, and capacity planning that a hosted API abstracts away entirely.
Storage and state
Conversation history, uploaded documents, and any retrieval index need persistent storage that survives container restarts and deployments. It's easy to get this working in a quick local test and then discover the database or file storage was ephemeral once the first redeploy wipes it. For anything beyond a single-user experiment, plan for a real database and a separate volume for uploaded files from day one, and back both of them up on a schedule rather than assuming the container will just keep running.
Where this gets harder than a SaaS tool
The honest trade-off is operational ownership. A hosted product handles uptime, scaling, security patches, and model provider outages for you. Self-hosting means you're the one who notices when a dependency has a security advisory, when disk usage on the uploads volume creeps toward full, or when a model provider changes an API in a way that breaks your integration. None of this is unmanageable — it's the same category of work as running any other internal web service — but it's real, ongoing work that doesn't show up until after the initial setup is done.
What you get in exchange is control: the ability to point different user groups at different models, keep sensitive data inside your own network boundary, and avoid a subscription cost that grows every time you add a new employee to the tool.