What is inference?

Key facts

Definition	Running a trained model to generate predictions or text, without changing its weights
Why it matters	Inference is the recurring cost and latency of every AI feature
Main cost drivers	Model size, prompt and output length, request volume and hardware efficiency
Latency components	Time to first token plus generation speed across the output
Optimization levers	Smaller models, shorter prompts, streaming, caching and batching
Plugsky approach	Managed inference across 30+ models behind one OpenAI-compatible endpoint
Pricing model	Flat monthly self-serve plans with unlimited fair-use usage
Status	Hosted chat, streaming, embeddings and function calling are live

TL;DR

• Inference is model execution; training is what came before it.
• Time to first token and generation speed are separate metrics.
• Prompt length affects cost on every single request.
• Smaller models and streaming are the cheapest latency wins.
• You can also run inference locally or in your VPC instead of a shared API.

How it works, step by step

1. Measure both time to first token and total generation time for your workload.
2. Profile prompt sizes and trim context that does not change the answer.
3. Choose the smallest model that passes your evaluation for each task.
4. Enable streaming so users see progress while the rest generates.
5. Cache stable results and batch non-interactive jobs where the API allows.
6. Re-evaluate after model updates — quality per token changes over time.

Try it yourself

Open the API latency tester →

Inference versus training

Training adjusts a model's weights using large datasets; inference uses those frozen weights to produce output for your input. Inference is cheaper per request but happens constantly, so it dominates operational cost. It also has different bottlenecks: training is throughput-bound on accelerators, while interactive inference is latency-bound, where time to first token and tokens per second determine how responsive a product feels.

What drives latency and cost

• Model size: larger models produce better answers and slower, costlier tokens.
• Context length: every token you send is processed, so long prompts add cost to every call.
• Output length: generation is sequential; long answers take longer and cost more.
• Concurrency: batching raises throughput but can add queueing latency.
• Hardware: efficient serving stacks move the same model faster.

Optimize in that order: model choice, then prompt size, then serving tricks.

Common mistakes

• Sending entire documents and histories on every turn instead of retrieving what matters.
• Using a frontier model for tasks a small model handles identically.
• Measuring only average latency and missing slow tail requests.
• Ignoring time to first token — it is what users perceive as speed.
• Treating local and hosted inference as equivalent without benchmarking both on real prompts.

How Plugsky serves inference

Plugsky runs inference as a managed service across 30+ models at one OpenAI-compatible endpoint, so applications do not manage GPUs or serving stacks. Streaming and function calling are live, and self-serve plans use flat monthly pricing with unlimited fair-use usage rather than per-token billing, which makes capacity planning simpler. For teams that must keep inference in their own environment, VPC, on-prem and air-gapped deployment options are available.

Honest comparison

Option	Plugsky hosted	Self-hosted GPU	Local on-device
Ops burden	None — managed service	High — serving and scaling	Low, but hardware-bound
Model choice	30+ models via one API	Only models you can serve	Small models only
Privacy	Region and private deployment options	Fully in your infrastructure	Fully local
Scaling	Elastic	You provision GPUs	Fixed
Best for	Most production workloads	Strict control and scale	Offline and edge cases

Frequently asked questions

What is AI inference?

The process of running a trained model on new input to produce output — for example, generating a chat completion or an embedding — without updating the model's weights.

How is inference different from training?

Training uses datasets to adjust model weights over time. Inference uses the finished weights to answer requests. Training is a project; inference is an ongoing operating cost.

Why is my inference slow?

Usually one of three causes: a large model, a long prompt processed on every request, or a long generated output. Measure time to first token and generation speed separately to find the bottleneck.

Can inference run locally instead of in the cloud?

Yes. Local and self-hosted inference keeps data on your hardware but limits you to smaller models and requires GPU capacity. Hosted inference scales more easily.

Does Plugsky charge per token for inference?

Self-serve plans are flat monthly with unlimited fair-use usage — no per-token billing. See the live pricing page for current plan details.

Does streaming affect inference cost?

Streaming changes when you receive tokens, not how many are generated. It improves perceived speed without reducing total cost.