Self-hosting LLMs: When does it make sense?

A big turn to your own hardware?

Remember last year? The idea of a mid-sized company running its own LLM on its own or on rented infrastructure was more or less science fiction. Three obstacles stood in the way:

1. Hardware ordinary mortals could not get. Laying hands on a GPU powerful enough to run inference on large models meant either selling your soul to the cloud giants or getting enormously lucky in the hardware lottery.
2. LLM spend wasn't that big. Even if you got past the technical obstacles, average company spending on LLMs was not high enough to justify the high fixed costs and the work of managing your own hardware.
3. A gap in capability and quality. Open-weight models lagged well behind the closed-source ones, not only in reasoning but in features. Images, PDFs and advanced work with context were the exclusive domain of the paid APIs.

Today the situation is dramatically different. As was said at Sequoia Capital's AI Ascent 2026, the mood in the community is fundamentally turning. Self-hosting is moving out of the "for enthusiasts" category and into real company strategy. So we decided to try it for ourselves, from giant models like Kimi K2.6 down to the more manageable Qwen 3.6 27B.

Look at the hardware first. You no longer need a server room. On platforms like Vast.ai or RunPod.io you can rent monsters with 8x B200 for a reasonable $40 an hour or so. More interesting still for an ordinary company: machines with 4 decent GPUs and around 160 GB of VRAM total go today for as little as $3 to $4 an hour. That is capacity enough to run the 30B family comfortably, image encoders included.

The second factor is the growth in company spending. According to Menlo Ventures' analysis 2025: The State of Generative AI in the Enterprise, average company spending on LLMs grew an incredible 3.2x between 2024 and 2025, and 2026 shows no sign of slowing. Suddenly we are not spending hundreds of dollars a month on APIs but tens of thousands. And at those volumes it starts to make sense to think about running your own.

And that is before mentioning the biggest advantage of self-hosting: full control over your stack. You know exactly where your sensitive company data flows, nobody uses it for training, nobody knows your internal use cases, and you can tune and scale the infrastructure to exactly what you need.

Economic reality: it all stands or falls on idle time

How did our experiments with running our own turn out? Here is the brief tl;dr:

The deciding factor in whether self-hosting pays economically is GPU utilization, that is, idle time.

Our experiments and the hard data point to one clear conclusion: the only case where self-hosting makes purely economic sense today is large batch processing. The classic example is categorizing millions of products in an e-shop, or a large analysis of historical data. In that case you keep the rented GPU busy 100% of the time, reach absolutely zero idle time, and the moment the work is done you throw the machine away.

If you are thinking about self-hosting for developers and coding, which is to say interactive use, the economic arithmetic collapses. Because of the natural pauses in the work (the developer is thinking, writing local code, or at lunch) the GPU sits idle most of the time. With one developer, or a few, you simply have no chance of reaching the price of a public API, not even by aggressively shutting the servers down overnight. Self-hosting makes sense here only if other advantages, such as extreme security, compliance or full control over your data, outweigh the financial disadvantage.

Four questions worth answering before you start

1. Do we have a stable, continuous, high load? Can we keep the GPU at least 80% utilized most of the time?
2. Will we be running huge one-off batch jobs? Can we rent a powerful machine for a few hours, push millions of items through it, and then destroy it?
3. Is protecting data an absolute priority for us? Do we have customers or regulations (GDPR, HIPAA, confidential company know-how) that strictly forbid sending data to a third-party cloud?
4. Do we have the engineers to run the infrastructure? Can we afford to give the team the time to maintain our own LLM stack and debug the deployments?

Let's see a practical example: Qwen 3.6 27B

Through a cloud API (Alibaba Cloud)

You pay $0.6 per 1 million input tokens and $3.6 per 1 million output tokens. For an average developer task, say having an agent write a new feature, which according to real production data comes to roughly 42,000 input and 8,500 output tokens, that works out at:

0.6 x 42,000 / 1,000,000  +  3.6 x 8,500 / 1,000,000  =  $0.0558

On a self-hosted machine at $3.5/hour

When we ran this model with thinking enabled, multi-token prediction, and further KV cache optimizations, we reached a speed of 175 tokens processed per second (fully comparable with the fastest commercial APIs on the market, see Artificial Analysis). Processing the same task, 50,500 tokens in total, took roughly 4 to 5 minutes on this machine and so cost about $0.28.

The magic of parallelization

Our setup, though, handled 5 such connections in parallel without trouble. So if you can arrange for 5 agents to run on the machine at once, the price per task falls to just $0.06, which is practically identical to the API.

Where the invisible costs of self-hosting hide

Count up the pure run time, multiply it by $3.5 an hour, and you get a very tempting number. In real operation, though, the hidden costs you will not see in that table catch up with you:

Ops and constant maintenance

Who deals with the GPU that collapses at 23:00 because of a driver bug, or with VRAM overflowing? Development, deploying monitoring and logging, and endlessly debugging the configuration cost precious hours of your engineers' time, which in the end is far more expensive than the hardware itself.

Our recommendation

If you decide to scale the servers to zero, we recommend packing the model weights directly into your own Docker image. With that image cached, the cold start shortens from hours to a few minutes. Otherwise the cold start is very unpleasant.

The mad arithmetic of agentic workflows

Why are we talking about numbers as huge as 50,000 tokens for a single task? Production data from the past few months shows that agentic coding, where the code is not written by a human with autocomplete but by an autonomous agent such as Claude Code or Pi, has an entirely different economy from classic chat.

A recent April preprint, How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks (Bai et al., arXiv:2604.22750), from researchers at Michigan, Stanford, Google DeepMind and Microsoft, is the first systematic study of the phenomenon:

A gulf of orders of magnitude

Agentic tasks consume orders of magnitude more tokens than ordinary queries. A single agentic task can consume up to 3,500x more tokens than a classic single-turn query.

The inverted input/output ratio

Where in ordinary chat output tokens are 15-50% of input, with agents the ratio inverts. Because of the constant reloading of context, files and history (context stuffing), the average ratio of input to output tokens is 154:1. Agents read the whole project over and over.

Extreme stochasticity

Token consumption is highly unpredictable. Run the same task four times independently and the most expensive attempt differs from the cheapest by a factor of two on average, and by up to 30x at the extremes.

More money does not mean a better result

The success rate of agents peaks at medium cost. Extremely expensive runs are often just unproductive loops: reading the same files again and again, and failing to edit them.

Kimi and Claude vs. GPT-5

On SWE-bench tasks, Kimi-K2 and Claude Sonnet-4.5 consume on average 1.5 million tokens more than GPT-5 on the same problem. Efficiency in using tokens looks more like a behavioural trait of a particular model than a question of its size.

To give a sense of what real production costs look like, in March 2026 the proxy service RelayPlane published median token counts from real agentic runs.

RelayPlane reports a median of 12 turns for complex end-to-end tasks. The main devourer of money is the constant sending of the whole codebase at every step.

An analysis from Zylos Research (February 2026: AI Agent Cost Optimization: Token Economics and FinOps in Production) sums it up clearly: autonomous agents make 3x to 10x more LLM calls than chatbots, and an unsupervised agent on a complex software task can push hundreds of dollars through the window in no time. Which is exactly why optimization, and possibly self-hosting, is such a hot topic.

Our test: Qwen 3.6 27B in the Berkeley GSO benchmark

To check how open models really stand in the real world, outside contaminated benchmarks, we ran our self-hosted Qwen 3.6 27B through the excellent GSO benchmark (Global Software Optimization), created by Manish Shetty and his colleagues at UC Berkeley.

GSO is a hidden gem among tests: meaningful and demanding, but also new and not so well known, so there is a fair chance nobody has optimized for it (yet). The researchers took 102 real commits from popular libraries (NumPy, Pandas, PyTorch) that brought significant speedups. They built Docker containers holding the state before the commit, and the agent's task is to optimize the code. It counts as a success if the agent reaches at least 95% of the speedup the human expert achieved.

The standard benchmark setup uses the OpenHands framework and gives models a limit of 200 iterations. The current king is Claude 4.7 Opus with a success rate of 42.2%.

We wanted to know whether it makes sense to consider a smaller local model for everyday work, so we tested our self-hosted Qwen 3.6 27B with the limit cut to 100 iterations.

The result completely shocked us:

• Qwen 3.6 27B (100 iterations): reached an excellent success rate of 20.59%.
• Gemini 3.1 Pro (200 iterations): reaches roughly the same 21.6%, but needs twice as many attempts to do it.

That shows that modern, smaller models (around 30B parameters) are incredibly effective on specific, well-bounded tasks such as code optimization, and can hold their own against the cloud giants. But you have to test them yourself to know whether they suit you.

What to take away from this?

There is no simple answer, but the tide is clearly turning:

1. A hybrid approach makes the most sense. For 80% of ordinary, routine tasks (small edits, refactoring, writing tests) a smaller self-hosted model of the Qwen 3.6 27B type is entirely sufficient, and lightning fast.
2. Keep the proprietary frontier for the heavy work. For complex, creative and architectural tasks the difference between a local model and monsters like Claude 4.7 Opus is still enormous. The money saved on the routine can then be invested here without regret.
3. Starting it up is easy, optimizing it is a hard nut to crack. A capable DevOps engineer can put together a simple working API with frameworks like llama.cpp, vLLM or SGLang and hide it behind an Nginx reverse proxy in an afternoon, but that is where the simplicity ends. Tuning a production setup so that it competes on speed and reliability with the large cloud providers is extremely demanding. Correctly configuring the various quantizations, tuning KV cache parameters, setting up multi-token prediction, and above all stable and fast multimodality (processing images or PDFs without swamping VRAM under parallel requests) is an engineering discipline that has broken the teeth of many an experienced team.

We ran the whole benchmark, and the experiments with Kimi K2.6 and Qwen 3.6 27B, on our anex.sh. It exists to take the pain out of orchestrating and deploying LLMs. If you are using LLMs for any asynchronous task such as generating embeddings, text descriptions, categorizing large catalogs of products or others, try us out.