Private LLM API, three ways to get one
You can get a private LLM API by running an open-weight model yourself, getting zero retention from a lab, or using an API that keeps no prompts by default.
Running your own keeps every prompt in your hands but costs hardware and upkeep.
The three routes
| Route | Who holds your prompts | What it takes | Models |
|---|---|---|---|
| Run it yourself | You | GPUs, an inference server, upkeep | Open weights small enough to run locally |
| Zero retention from a lab | The lab, for the length of the request | Approval, a sales request or a contract | That lab's models |
| An API that keeps nothing by default | The API, for the length of the request | An account | The open-weight models it serves |
1. Run an open-weight model yourself
llama.cpp and vLLM both serve an OpenAI-compatible endpoint, so your code doesn't change. Sizes are for 4-bit builds, from each model's Hugging Face card.
| Model | Parameters | Size at 4-bit | Runs on | License |
|---|---|---|---|---|
| Qwen 3.8 27B Abliterated | 27.8B | about 17 GB | one 24 GB GPU or a 32 GB Mac | Apache 2.0 |
| Qwen 3.6 35B-A3B Abliterated | 36B total, 3B active | about 22 GB | one 24 GB GPU | Apache 2.0 |
| Gemma 4 31B Heretic | 31.3B | about 19 GB | one 24 GB GPU | Apache 2.0 |
| Llama 3.1 8B Abliterated | 8B | about 5 GB | an 8 GB GPU or a laptop | Llama 3.1 Community License |
The largest open-weight models, such as Kimi K3.1 Pro at 2,780B parameters, need a rack of GPUs.
2. Get zero retention from a lab
Until a lab grants zero retention, its default applies.
| Lab | Keeping nothing takes | Default until then |
|---|---|---|
| OpenAI API | Prior approvalPlus additional requirements | Up to 30 days |
| Claude API | A sales requestCovered Models keep 30 days | Within 30 days |
| Mistral API | A written requestAt Mistral's discretion | No period published |
| xAI API | A console settingTeam-wide, not yet on for every team | 30 days |
| Gemini API | Settings, or Vertex AIVertex AI for a guarantee | 55 days |
See how to get zero data retention for each lab's steps.
3. Use an API that keeps nothing by default
This gets you large open-weight models without hardware or an approval step.
"Silk Compute does not store your prompts or model outputs."
It serves six open-weight models, Kimi K3.1 Pro among them, with a 1,048,576-token context.
silkcompute.comabliteration.ai and Featherless also say they keep none. The ranking compares all three, and the switching guide shows the code.