LLM terms, explained

What is LLM (Large Language Model)

It's a black box taking tokens on the input and producing tokens on the output.

Whatever you send (text, an image, audio, video) is first cut into tokens. A single token is a part of a word, image or an audio segment, it's a piece of information that is represented by a list of numbers (also called a vector, tensor or embedding).

The model multiplies and transforms these lists with its own weights billions of times and produces a different list of numbers. That list is used to get a new token that is added to the text so far, and the whole thing runs again for the next one.

Images, audio and video

A model that accepts an image needs to have a special front end for it: an encoder, which turns the image into a sequence of tokens. These tokens are then turned into lists of numbers same as with the text tokens.

The modalities a model handles are the kinds of input it has an encoder for. That is fixed in the model weights and cannot be changed, so a model with no audio encoder cannot be sent audio.

How many tokens one image becomes depends on both the model and the image. Some models like gemma4-31b encode every image to a fixed 280 tokens, no matter the size. Others like qwen3.6-27b cut the image into fixed-sized tiles, that results in high resolution images consuming more tokens. So it is recommended to always send images in a reasonable resolution that is good enough for the task but not bigger.

Audio and video go through their own encoders and are measured in duration rather than in pixels.

How LLMs generate text

In two stages, input and output:

Prefill

Reading your input. Every input token can be processed at the same time, so the graphics card can work through the whole prompt in one pass. This is why input tokens are cheap.

Decode

Writing the answer. Each output token depends on the one before it, so they come out strictly one after another. This is why a chat answer arrives word by word, and why output tokens cost more.

The context window is how many tokens the model can process before it starts generating the output. It is your input plus whatever it has written so far. It is called context window because it tells you how much context the LLM can have when giving you the output. 1M context means the LLM can process 1 million input tokens in order to give you the response, that is almost full Harry Potter book series.

Tokens per second (tps) is the decoding rate. How many output tokens per second can the LLM generate. This depends on the combination of model and hardware. Realtime APIs usually have <100 tokens per second because they need to support generating multiple responses at the same time. Batch APIs typically achieve much higher tps. E.g. at anex.sh we achieve thousands of tokens per second when processing large jobs.

Temperature and determinism

When generating the response, the LLM does not output a token directly. Instead it takes the last layer's output and turns it into a probability distribution over all possible tokens = a list of numbers where each number is a probability of one possible future token. Because they use probabilities for choosing the next token, LLMs are non-deterministic = re-running the same input can give you a different output. Temperature is what sets how different the output for the same input can be.

Temperature 0 means that the LLM will always pick the token with highest probability score. But if you increase the temperature it will allow the LLM to pick other tokens that have lower probability from time to time. So higher temperature leads to higher unpredictability (and originality) of the output.

Set temperature to 0 only for tasks where you need high repeatability and simple answers, e.g. classification. For anything else keep the default non-zero temperature.

Thinking

Some models are trained to think before answering the question. The thinking is just writing a bunch of output tokens before writing the actual answer. This helps the model to understand the problem from multiple angles.

Since thinking is a regular output, it is also generated one token at a time and is billed as output tokens. You can try switching thinking off to get the response faster and cheaper for very simple tasks like extraction, classification, tagging, reformatting and so on. But it is recommended to leave it on for anything non-trivial. The newer models are trained specifically to think about everything and turning thinking off might significantly hurt their performance.

Embeddings

Embedding is a vector representation of a piece of information = it's a list of numbers that contains everything the model knows about this token. It's created by the model when it processes the input tokens, the model transforms the embeddings as they pass through its layers from a simple token representation on input into a representation of what the token means in the current context. Which is why embeddings are sometimes called "inner / latent representations" of the input.

What makes the list useful is that similar inputs produce similar lists. Measure the distance between two embeddings and you have a measure of how alike the two inputs are, even when they share no words / pixels at all. That is called semantic similarity = the inputs (words, sentences, images ...) have similar meaning even if they look different.

It's important to remember that the embeddings you want to compare must come from the exact same model and at the same dimensions (dimensions = length of the vector). If any of these don't match then you cannot compare the embeddings between each other, the numbers from different models mean completely different things.

Any model can generate embeddings but there are specialized embedding models that are optimized to generate embeddings instead of tokens. These are recommended when building e.g. AI search.

Training and inference

Model's lifecycle is split into 2 main parts:

Training

Done once on a massive amount of data. Each model starts with random weights and through the process of seeing the training inputs and outputs it learns (= sets the model weights) in a way to reproduce the seen patterns. When it's done the model contains the knowledge and patterns it saw in the training data, nothing else.

Inference

Running the trained model on your input to get output. This does not influence the model and each run is independent of the others. Name comes from the model inferring the next token based on the previous ones.

Everything you do on a realtime or batch API is inference, and every run is independent. Each row in an input file is its own run. It sees only what that row carries, nothing carries over from the row before it, and nothing you send teaches the model anything, on this job or the next one.

Batch vs realtime

Both run the same model, the difference is how quickly you need the result.

A realtime request expects an answer now (in real time), so a card has to be sitting ready for it, and the idle time between requests is part of what each request costs. Example is a chat assistant.

A batch job is a file of independent rows that can be processed with a delay. The API provider can send the whole file to a dedicated card and process multiple rows together to keep the card busy from start to finish. This makes the batch jobs cheaper.

Anything processing a large amount of data is a perfect fit for batch API: classification and tagging of ecommerce products, transcribing audio calls, searching in videos and many others. 

Model sizes

The number in a model's name is its parameter count: qwen3.6-27b holds 27 billion of them. Parameters are the weights. So to run this model, you need to store 27 billion numbers in memory and compute roughly 27 billion multiplications for each generated token. 

There are two main LLM architectures:

• A dense model uses every parameter for every generated token, qwen3.6-27b is a dense model.
• A mixture of experts (MoE) splits most of its parameters into groups of weights called experts and routes each token to a selection of relevant experts. You can recognize a MoE model by having AxB in its name saying how many parameters get activated. For example Qwen3-Omni-30B-A3B is a MoE model with 30 billion parameters in total and about 3 billion get activated when generating a token.

A 30B MoE model needs same memory like a 30B dense one but compute closer to a 3B one. So if you have a DGX Spark that has a lot of memory but slow compute, it will be a great fit for a mixture of experts model.

Bigger models tend to be smarter. At the same time newer models are significantly smarter than older models. This year's many smaller models e.g. Qwen 3.8 27B are actually smarter than some of the largest models from last year. We suggest using the newest models as they tend to offer the best price/performance.

Quantization

Each weight in the model is a floating point number like 1.1337234, that number is stored with a high precision during training to ensure that the multiplications are calculated correctly. But we can reduce the precision during inference to save a lot of memory and lose just a little bit of accuracy. That reduction is called model quantization.

When using APIs serving open source models always ensure the provider clearly specifies exactly which model weights are being served (like this). Some providers mention only the model name which might hide that they are serving an inferior quantization with significantly lower capability.