Open-weight AI models now make it practical to run an assistant on infrastructure you control. Here is what the setup involves, what hardware it needs and where the trade-offs are.
Key takeaways
- Open-weight AI models can run on your own server or private cloud, so documents never leave your control.
- Answer quality depends on the model, the retrieval setup and clean source documents.
- Sign-in, permissions and audit logs are part of the build, not extras.
Why teams want private AI
Many businesses would like an assistant over their own documents but cannot send client files, contracts or health records to a public AI service. Regulators, client contracts or simple caution rule it out. Open-weight models such as Llama, Mistral and Qwen make a different setup possible: the model runs on a server you control, and nothing leaves your environment.
The pieces of a private assistant
- A model: an open-weight language model sized for the hardware and the task.
- An inference server: software such as vLLM or Ollama that loads the model and answers requests.
- Retrieval: an index of your documents so the model answers from your sources and cites them.
- An interface: a chat window, or an assistant inside a tool people already use, such as Microsoft Teams or your CRM.
- Access control and logging: single sign-on, permissions that match what each person can already see, and a record of questions and sources.
Hardware and hosting
The main constraint is GPU memory. Smaller models run on a single workstation-class GPU and handle summaries, drafting and document questions well. Larger models give better reasoning but need more memory or several GPUs. Quantized versions of a model reduce memory use with a modest loss in quality, which is often a good trade.
You do not have to buy hardware to start. A private instance in your own cloud account keeps data inside your tenant and lets you test model sizes before committing to a server in your office.
What to expect from quality
Open-weight models have improved quickly, and for answering questions over your own documents, retrieval quality usually matters more than model size. Where they still trail the largest commercial models is long, multi-step reasoning. A practical setup routes most work to the private model and, only where you approve it, sends non-sensitive tasks to a commercial model under your own account.
Security is part of the build
Running privately does not make a system secure by itself. The assistant should only retrieve documents a user is already allowed to open, logs should be kept and reviewed, and the server should be patched and monitored like any other production system. Decide before launch who owns it and what happens when someone asks a question it should not answer.
FAQ
Do we need GPUs to run a private AI model?
For fast answers with larger models, usually yes. Smaller models can run on standard servers for light internal use.
Which models can run privately?
Open-weight model families such as Llama, Mistral, Qwen and Gemma can be hosted on your own infrastructure, subject to each model license.
Is a private model as good as ChatGPT?
For focused tasks such as answering from your own documents, a well-built private setup is often good enough. The largest hosted models still lead on broad general reasoning.
A sensible first project
Choose one document set, one team and one measurable task, such as finding precedent in past matters or answering policy questions for staff. Run it with a small group for a few weeks, measure how often it finds the right source, and expand from there.
Our private AI service sets this up on your own server or cloud account. For the difference between retrieval and training, see RAG or fine-tuning. For a real example, see how an insurance agency used private document AI with human review.