Coding agents are getting much better at solving complex engineering tasks. But they still have a basic problem: they forget.

You might spend 30 minutes showing Claude Code how your repo handles migrations. Later, you correct Cursor on the same testing convention. Then Codex runs into a similar debugging problem and has to rediscover the solution from scratch.

Each agent session creates useful knowledge, but most of that knowledge stays trapped inside individual transcripts.

And with new models like Jev making fast evaluation of agent runs dramatically cheaper, it is suddenly much more practical to ask:

Which of those runs are actually worth learning from?

Beacon, built by Asymptote Labs, puts that signal to work.

Beacon is an open-source telemetry and memory layer that captures agent activity across Claude Code, Codex, Cursor, OpenCode, Cline, and 19+ other agent harnesses.

It can use Jev to evaluate those traces, identify the highest-signal runs, and extract recurring workflows, corrections, and debugging patterns.

Those patterns can then be turned into skills that future agents can use.

Here’s the repo: github.com/Asymptote-Labs/agent-beacon

(don’t forget to star it ⭐)

The important part is that Beacon sits across the harness layer.

Your Claude Code sessions can teach Codex.
Your Cursor debugging can improve OpenCode.
A workflow solved once can become reusable everywhere.

The loop looks like this:

Run agents → capture traces → evaluate with Jev → extract knowledge → create skills → improve future runs

Now imagine that across an entire engineering organization.

One engineer spends an hour debugging an obscure infrastructure issue. Normally, the useful parts of that work disappear into their terminal history, a Slack thread, or an agent transcript nobody else ever reads.

Beacon can turn that successful trace into reusable knowledge for the next engineer, regardless of which coding agent they use.

This turns agent traces into something closer to a continuously growing knowledge base for engineering work. And because Beacon supports multiple harnesses, the knowledge is not locked inside a single AI coding tool.

You can switch agents without throwing away everything your previous agents learned.

That is the core idea:

A problem solved by one agent never needs to be learned from scratch by another.

Jev makes it cheap to determine what is worth learning from.

Beacon makes that knowledge portable across every agent you use.

Beacon is fully open source, so you can try it with the coding agents you already use.

Check out Beacon on GitHub →

(don’t forget to star it ⭐️)

More on Jev below👇

Jev, clearly explained:

TypeSafe AI released Jev on September 15, 2026. It cannot hold a conversation, write code, or generate a useful paragraph. The narrow interface is the product.

Software makes thousands of small judgments: Is this ticket urgent? Which model should handle this request? Is this shell command dangerous? Teams often send each one to a general-purpose LLM, then parse and validate the generated answer.

Jev takes a different interface. Unstructured state goes in. Typed answers and probabilities come out. The relevant question is whether some LLM calls should have been bounded decisions in the first place.

Let’s dive in to learn more!

Generation is expensive control flow

Tool calling and structured outputs removed a lot of brittle parsing, but the model underneath is still generative.

Even when the answer is one word, such as “billing”, it produces tokens sequentially.

Now put that inside an agent loop.

while not done:
    action = llm(context)
    result = run_tool(action)
    context += result

One agent run may call the model repeatedly to select tools, detect risk, judge results, and decide when to stop. These calls require judgment, not prose. Jev targets them directly.

The interface: state, typed questions, and distributions

Jev is a semantic decision engine. You send it two things:

• State, which is the text or JSON that describes the current situation.

• Questions, which are the decisions you want it to make about that state.

Every question declares its answer shape in advance. Jev supports three primitives:

• Choice picks one option from a list and returns a probability for every option.

• Score places the input on an ordered scale such as low, medium, and high.

• Noul answers a yes-or-no question by returning the probability that it is true.

Noul is TypeSafe’s name for its Boolean decision type. It returns a number between 0 and 1.

{
  "model": "jev-latest",
  "state": "The deploy failed twice and customers are seeing 500s.",
  "questions": {
    "urgent": {
      "type": "noul",
      "instructions": "Does this need attention right now?"
    },
    "owner": {
      "type": "choice",
      "instructions": "Which team should handle this?",
      "criteria": {
        "engineering": "Product failures and outages",
        "billing": "Charges, invoices, and refunds",
        "sales": "Pricing and new accounts"
      }
    }
  }
}

The response contains an urgency probability and a distribution over the three teams. The schema prevents undeclared answers, but it cannot repair a bad taxonomy. If billing and engineering overlap in the rubric, Jev must choose between ambiguous categories.

Your program keeps control:

if urgent > 0.9 and owner == "engineering":
    page_on_call()
elif confidence < 0.6:
    send_to_human_review()
else:
    add_to_queue(owner)

Code owns the branches. Jev estimates which branch matches language that cannot be reduced to exact conditions.

TypeSafe says Jev evaluates questions in parallel. That helps when the questions are independent. If one answer determines the next question, keep that dependency in code rather than evaluating unused branches.

The company reports latency of 70 to 500 milliseconds and a price of $0.042 per million input tokens, with no charge for output. Its claims of up to 200 times lower latency and 400 times lower cost come from favorable internal comparisons, not production capacity data.

Scoring fixed outcomes avoids token generation, output parsing, and some retries.

Why confidence exists in control flow

Suppose Jev routes a ticket to billing. The label tells you what won. The distribution tells you how close the race was.

{
  "choice": "billing",
  "probabilities": {
    "billing": 0.52,
    "technical": 0.46,
    "sales": 0.02
  },
  "confidence": 0.18
}

Billing won by six points. Automatic routing would be reckless.

The application needs an abstention policy in addition to a winning label:

• Act automatically when confidence is high and the consequence is small.

• Ask for confirmation or call a stronger model when confidence is middling.

• Send the case to a person or gather more information when confidence is low.

Thresholds belong in code and should reflect consequence. A dashboard label can tolerate weaker evidence than a command that deletes data.

TypeSafe trains Jev using Reinforcement Learning for Calibrated Decisions, or RLCD, so predicted confidence should track observed accuracy. That relationship must hold on your traffic. Measure it separately for each decision type and recheck it after changes to the model, questions, or input distribution.

TypeSafe also says Jev cannot hallucinate. More precisely, it cannot break the declared output schema. It can still choose the wrong valid option with high confidence. Type safety prevents malformed output, not bad judgment.

Using Jev inside an agent

Jev belongs beside an LLM. The LLM plans, writes, explains, and uses tools. Jev handles repeated decisions around that work.

1) Model routing

A simple lookup does not need the same model as an architecture review. Jev can score a request and choose the least expensive model likely to complete it.

route = jev.choice(
    state=user_request,
    options={
        "fast": "Lookups, extraction, and small local edits",
        "powerful": "Architecture, ambiguity, and high-stakes work",
    },
)

model = fast_model if route == "fast" else powerful_model

The router chooses a model. It does not answer the request.

Pre-execution tool risk classification

Before an agent runs a shell command, Jev can classify it as read-only, reversible, or destructive. Separate questions can check whether it deletes files, changes Git history, touches production, or leaves the repository.

High-confidence read-only actions can continue. Destructive or uncertain actions pause for approval. LangChain’s Jev integration implements this check in middleware.

Post-execution semantic checks

Jev can check whether an agent is repeating itself, whether an output follows policy, or whether a result needs review. It should never replace a hard test. Use it only where the rule depends on meaning.

Problems Jev can solve

A workload is ideal when its output set is closed, a reviewer can label examples quickly, and the decision occurs often enough to justify another model dependency.

1) Request classification and routing

Jev can classify intent, urgency, department, spam, and customer frustration. One request can score several properties of the same ticket, while code combines the results into the routing policy.

2) Retrieval relevance and citation support

Embeddings retrieve related text. They do not establish that a passage answers the query or supports a claim. Jev can score those conditions before generation and remove irrelevant context.

Embeddings are excellent at finding semantically related text. Jev can make the narrower decision of whether a particular passage is useful for this question.

3) Semantic checks at enforcement boundaries

Jev can screen for prompt injection, policy violations, and risky tool calls. Its scores can inform an enforcement boundary, but they should not become one. Permissions, sandboxes, allowlists, and tests must enforce exact rules.

4) Corpus-scale semantic enrichment

Jev can label documents, products, and customer messages or turn free text into features for another model. Low per-call cost can make semantic scoring practical across every row of a dataset.

5) Real-time interfaces

Jev can choose among known browser actions or actions from structured game state. It is text-only, so the environment must first be represented as text or JSON.

Failure modes and exclusion criteria

Reject Jev when the task cannot be expressed as a stable set of outcomes.

• It cannot write, summarize, generate code, or explain its reasoning.

• Do arithmetic, counting, date comparison, and exact string manipulation in code.

• If a decision needs several hidden reasoning steps, split it into smaller questions or use a reasoning model.

• It cannot extract an unknown value directly. Find candidates first, then let Jev choose among them.

• Irrelevant context can reduce accuracy. Send only the state needed for the decision.

• Closed weights, early access, and limited independent calibration data make blind trust premature.

If deterministic code already solves the problem, keep it. An if statement is faster, cheaper, and easier to test than any model.

Deployment procedure and operational controls

A cheap model becomes expensive when mistakes cause retries, reviews, or incidents. Measure expected cost per decision, including model calls, escalations, false approvals, false blocks, and recovery work.

A production rollout needs evidence at each boundary:

1. Choose one bounded, low-risk decision with clear possible answers.

2. Write the rubric before calling the model. Define what belongs in every option.

3. Collect representative examples with expected answers, including ambiguous and adversarial cases.

4. Run Jev in shadow mode beside the current workflow without letting it change behavior.

5. Plot accuracy against confidence and set thresholds from your data.

6. Automate the safest branch first. Keep a person or stronger model for uncertain cases.

7. Pin or log the model version, questions, criteria, and thresholds so you can replay changes against the same evaluation set.

Questions and criteria are program logic. Version, review, and test them.

The architectural trade

Jev gives up generation in exchange for fixed answer types, explicit uncertainty, and code-owned branching. The LLM produces a plan, explanation, or code. Jev routes the request, gates risky actions, checks the result, and escalates uncertain cases.

Generation and judgment are different workloads. Forcing both through a text-generation API adds latency and turns control flow into prompt parsing. Jev exposes semantic judgment as a separate component.

Start with one bounded decision currently handled by a slow LLM call or a brittle regex. Send the minimum state, define the answers, and log Jev’s probabilities beside the current result. Automate one low-risk branch only after the evaluation supports it.

An if statement handles explicit values. Jev estimates the meaning needed to choose among them. The application decides whether that estimate is safe to act on.

Good day!

A good voice sample lasts a few seconds. A useful voice agent has to carry that voice through follow-up questions and interruptions.

Deepgram’s Flux TTS keeps context from its earlier replies within the same streaming session.

The LLM tracks the conversation; Flux TTS carries the pacing and delivery of what the agent has already said. Deepgram reports that speech begins streaming in as low as 80 ms.

We recorded a conversation with its car dealership agent about hybrid and electric cars, then kept asking follow-up questions. The video lets you hear the voice across the exchange.

You can try Flux TTS yourself here → https://fandf.co/4r7TBnF

Thanks to Deepgram for partnering today!

Run any agent harness under one interface

When an agent product integrates one harness directly, its backend starts depending on that harness’s task format, event stream, session model, and filesystem behavior.

Unlike model routing, adding another harness is not a config/parameter change because it adds another runtime that plans work, calls tools, manages files, and decides when a task is complete.

And supporting more than one runtime is critical since harnesses are built differently.

The right choice can depend on the task, required tools, provider access, available models, cost, latency, or usage limits. Like models, a product should be able to add or replace a harness without rebuilding the feature around it.

Model routing does not solve that problem. It chooses which model handles an inference request, while the same agent loop remains in place. Harness routing chooses the agent runtime that owns the complete task.

Today, let’s talk about the engineering behind that second form of routing. We will understand why products may need different harnesses, what must be standardized across them, and how the Unified Harness Protocol defines that contract.

We will then use HarnessRouter, an open-source UHP implementation, to run the same workflow through Codex and Claude Code.

Why a product may need another harness

Harnesses make different choices about tool execution, context management, permissions, skills, and supported models. The same harness may not provide the right tools or model support for every product feature.

A product may also serve customers with different provider access. One customer may already use OpenAI models, while another is set up around Anthropic. A quota or provider outage can also make one backend unavailable.

Evaluation is another reason. Running the same task through two harnesses makes their cost, latency, tool use, and output easier to compare. The comparison is difficult when each backend requires different task and event code.

The goal is not to switch harnesses during every session but rather to keep the product from depending on one runtime.

A team can select a harness per feature, offer a supported choice to customers, or replace a backend without changing the product workflow.

A model call does not run the whole task

A model API accepts input and returns generated output. The application still runs requested tools, starts follow-up model calls, and manages files, permissions, retries, and task state.

An agent harness includes this control loop. It can inspect files, call tools, edit a project, run commands, review the output, and decide when to stop.

Codex and Claude Code can use similar models and still behave differently. Their instructions, tools, context handling, permissions, and completion rules affect the run.

Model routing vs. harness routing

Model routing happens inside the inference layer. It chooses an endpoint based on price, latency, context length, availability, or measured quality. The application or harness still owns the agent loop.

Harness routing selects the runtime that owns the task. That choice can change the tools, skills, workspace, permissions, and event stream.

A product may use both routing layers. In UHP, model chooses the model, while metadata.harness_id chooses the configured harness.

What the product must handle for each harness

The first integration often starts as a small adapter that launches a process and reads its output. A product UI also needs live progress, session continuity, file uploads, download links, and access rules.

Cancellation needs a defined path because local processes, remote jobs, and sessions stop differently. Errors also need enough structure to separate provider failures, permission denials, time limits, missing files, and cancelled tasks.

Adding Claude Code beside Codex means building and maintaining these mappings again. The product needs one task lifecycle that both integrations can follow.

The task lifecycle defined by UHP

The Unified Harness Protocol defines the HTTP contract between an application and a harness server. It covers task creation, progress, sessions, files, cancellation, and errors without defining how the harness plans.

UHP uses an API shaped like the OpenAI Responses API. A client starts work through POST /v1/responses. The request includes the task input, model, configured Harness ID, and optional execution limits.

A response records one task. A session groups related responses that share conversation state and a working directory. A configured harness combines a base harness with its model, instructions, skills, tools, MCP servers, and limits.

Live tasks use Server-Sent Events, or SSE. Each event has a type and sequence number, so the client can process the stream in order.

The events report text output, tool activity, files, and task status. The application displays them without searching raw terminal lines for each event.

UHP accepts small files in the request and larger files through uploads. Generated files become artifacts that the application can list and download. The files specification defines those operations.

Follow-up tasks use previous_response_id. The server continues the same session with its conversation, working directory, files, and configured harness.

You can still hand off work to another harness. If Codex starts a task and you want Claude Code to take over, start a new session with Claude Code and pass it the relevant context, such as a summary of the work so far, the next instructions, and any files it needs. Think of it as handing a task to another developer: the work can continue, but Claude Code is not resuming Codex’s session.

For comparison, the application starts two sessions and gives both the same PDF and prompt. Each harness receives clear input instead of hidden state created by another runtime. The session specification documents this rule.

The current UHP version is 2026-09-12. The project publishes the written specification, OpenAPI 3.1 schema, JSON Schema definitions, and runnable conformance checks.

How HarnessRouter runs a task

HarnessRouter (GitHub Repo) is a self-hosted UHP server. It receives a UHP request, reads the Harness ID, and starts the selected runtime. As the harness works, HarnessRouter records its output as UHP events, responses, sessions, files, and errors.

The Docker image contains three services.

The Console provides the browser UI. The Gateway handles the UHP API and harness configuration. The Runner starts harness processes inside session workspaces.

A named /data volume stores the database, installed harness CLIs, provider integrations, session files, and workspaces. Only the Console port is published. The Gateway and Runner listen on loopback inside the container.

The self-hosting guide lists Codex, Claude Code, Hermes, DeepSeek Harness, Gemini CLI, OpenCode, Qwen Code, Cline, Goose, and other backends.

A built-in harness can run without a custom configuration. A custom harness starts with one base harness and adds a default model, instructions, tools, skills, and optional MCP servers. HarnessRouter gives the saved configuration a Harness ID.

The product sends this ID with each new task. It does not need to launch the harness CLI or parse the harness output itself.

Running HarnessRouter locally

Community Edition requires Docker, about 4 GB of disk, and an API key for a supported model provider.

Start the container.

docker run -d --name harnessrouter \
  -p 127.0.0.1:3000:3000 \
  -v harnessrouter:/data \
  harnessrouter/harnessrouter

The 127.0.0.1 binding keeps the Console on the local machine. The named volume preserves its data across container restarts.

Follow the startup logs.

docker logs -f harnessrouter

The first run installs the enabled harness CLIs:

Wait for this message:

[harnessrouter] ready on :3000

Open http://localhost:3000.

Next, open Integrations in the left panel and select Add Integration. Choose a model provider and enter its API key:

This key authorizes model requests. It is separate from the Console password and the API key later used by the application.

Open Agent harnesses in the left panel. You’ll see 11 built-in harnesses, including Codex, Claude Code, and Hermes. Select Hermes to create a task.

Select a model, create a task, and follow the run in the Console.

Files created by the harness remain attached to that session:

Configuring the paper workflow

Create two custom harnesses in the Console.

Paper Explainer with DeepSeek Harness
Paper Explainer with Claude Code

The first configuration uses DeepSeek Harness as its base harness. The second uses Codex. Both receive the same instructions and the same paper-explainer skill.

The skill defines the output expected by the paper application.

Read the uploaded research paper and identify its main mechanism.
Build a self-contained interactive explainer in index.html.
Include one controllable simulation with a short explanation beside it.
Use Three.js only when 3D interaction improves the explanation.
Create README.md with the source, assumptions, and run instructions.
Verify both files before completing the task.

The following video depicts this setup:

Upload the same paper to two new sessions. Run the same prompt through both configurations.

Turn the uploaded paper into an interactive visual explainer.
Recreate its main mechanism as a controllable simulation.
Return index.html and README.md.

The prompt, skill, and required files stay the same. Codex and DeepSeek Harness can still choose different plans, tools, and implementations.

• The first part of the video above shows the HarnessRouter dashboard and how the application selected and ran the harness.

• The final part of the video above shows what the agent produced.

Calling a harness from the application

After the task works in the Console, open http://localhost:3000/keys. Create a HarnessRouter API key and store it in backend code.

Set the local API address and the API key

export HARNESSROUTER_BASE_URL=http://localhost:3000/api/harness
export HARNESSROUTER_API_KEY=sk-hr-...

Send a task through the Responses-compatible endpoint.

curl --fail-with-body -sS "$HARNESSROUTER_BASE_URL/v1/responses" \
  -H "Authorization: Bearer ${HARNESSROUTER_API_KEY:?}" \
  -H "content-type: application/json" \
  -d '{
    "input": "Reply with exactly: it works.",
    "metadata": {"harness_id": "codex"},
    "model": "gpt-5.4-mini",
    "stream": false
  }'

Here’s that request running against the local endpoint, with the response printed in the terminal:

This invocation uses the built-in codex identifier from the repository quickstart. A custom configuration uses the Harness ID shown in the Console.

For another new task, changing metadata.harness_id selects another configured harness.

Here’s the demonstration with Hermes:

curl --fail-with-body -sS "$HARNESSROUTER_BASE_URL/v1/responses" \
  -H "Authorization: Bearer ${HARNESSROUTER_API_KEY:?}" \
  -H "content-type: application/json" \
  -d '{
    "input": "Reply with exactly: it works.",
    "metadata": {"harness_id": "hermes"},
    "model": "gpt-5.4-mini",
    "stream": false
  }'

As depicted above, the application keeps the same endpoint and event format. The same client can also continue sessions, retrieve files, and cancel work.

Session and security rules

The Unified Harness Protocol (UHP) standardizes the task lifecycle, but it does not make the harnesses behave the same. Different runtimes can choose different plans, tools, files, and models. The protocol gives the product one way to submit the work and inspect the result.

The local deployment also has a clear isolation limit. Sessions use separate operating-system users and workspaces. They do not run in separate containers. The Runner keeps the permissions needed to create those users, while each harness process runs as its assigned session user.

Provider credentials should stay outside environments that agent tools can read. Application API keys should never reach browser code. A deployment outside loopback also needs TLS and the controls described in the UHP security specification.

The two API calls above show why this matters when an agent becomes part of a product. Users may want to run a task with Codex, Claude Code, or Hermes.

Without a shared task interface, supporting each harness means writing another integration for progress, sessions, files, and cancellation.

HarnessRouter lets the product offer that choice in its own UI.

It sends the task through UHP, selects the harness, and receives the result through the same API.

The team can test those workflows locally before putting them in front of users. As harnesses change, the product has a way to support more of them without rebuilding its task flow each time.

HarnessRouter is Apache 2.0 licensed. The repository contains the self-hosted implementation, UHP schemas, and conformance tests.

GitHub Repo: github.com/HarnessRouter/harnessrouter.

(don’t forget to star it ⭐️)

Good day!

The Nebius AI Builder Program launched last week to kickstart developers with $400+ in free credits.

With one free sign-up, developers get credits for Nebius Token Factory, Tavily, Toloka, LangSmith, and more.

Developers also get access to runnable cookbooks and training opportunities from some of the top companies in the industry, providing both the access and the education to get started building on the open AI ecosystem now.

For developers aiming to move into AI engineering or take an existing project beyond demos, the program provides a direct way to build production experience without paying upfront.

Things you’d learn:

• Building a deep research agent powered by Tavily and Token Factory.

• Build an agent skill for searching for car parts with Kimi K3.

• How to use OpenCode with Nebius Token Factory models.

The same learning path would otherwise require several platforms and separate resources.

Join the Nebius AI Builder Program for free at dev.nebius.com/builders and get $400+ in credits and discounts across the stack on day one.

Join Nebius AI Builder Program for free

Join early before it runs out.

Thanks to Nebius for partnering today.

Where does all the VRAM go during LLM inference?

Loading the model is only the first part of the GPU memory story.

Once inference starts, VRAM is divided across model weights, the KV cache, temporary activations and workspace, and runtime overhead. Only one bucket stays roughly fixed. The others change with context length, batch size, concurrency, and model architecture.

This is why “the model fits on the GPU” and “the workload fits on the GPU” are two different statements.

The diagrams below break the memory budget into four useful buckets.

1. Model weights

Weights are the mostly fixed part of inference memory. A model with 8 billion parameters needs about 16 GB in FP16 or BF16, 8 GB at 8 bits per parameter, or 4 GB at 4 bits per parameter. Quantized checkpoints also store scales, zero points, and metadata.

The allocation does not grow during generation. Its main inputs are parameter count, numerical format, and how the model is split across GPUs.

Quantization also creates room for longer contexts, more simultaneous requests, or larger batches. Speed still depends on kernel support and dequantization cost, so lower precision does not guarantee higher throughput.

2. KV cache

The KV cache is the dynamic part that catches many deployments by surprise.

Each transformer layer stores key and value tensors for earlier tokens. During decoding, attention reads them instead of recomputing the full prefix at every step.

The dense-cache approximation below includes the number of active sequences. The factor of two accounts for keys and values. Grouped-query attention reduces this cost because several query heads share fewer KV heads. KV-cache quantization reduces bytes per element, with accuracy and kernel-support tradeoffs.

Double the cached tokens and the cache roughly doubles. Double the active sequences and it roughly doubles again. A server handling long conversations can spend more VRAM on KV state than weights.

Serving engines manage this memory in blocks. Paged allocation reduces waste from variable sequence lengths, while prefix caching reuses blocks when requests share an identical prompt prefix.

Some engines reserve most remaining GPU memory for this pool during startup. That reservation can make VRAM look full before real traffic arrives, even though much of the pool is still available for future tokens.

3. Activations and workspace

Inference needs temporary memory for layer outputs, attention, matrix multiplication, sampling, and kernel scratch space. These allocations are reused across layers, so they do not accumulate per generated token like the KV cache.

Peak size depends on the inference phase. Prefill processes prompt tokens in parallel and usually creates larger temporary tensors. Decode handles one new token per active sequence, although continuous batching can raise its footprint.

Different attention and matrix multiplication kernels request different workspace sizes. Memory-efficient kernels can lower the peak, while larger batches trade memory for better GPU utilization.

The GPU still needs enough free memory for this temporary peak. A workload can fail even when steady-state allocations appear to fit.

4. Runtime overhead

The final bucket contains CUDA contexts, loaded kernels, communication libraries, graph captures, scheduler buffers, allocator bookkeeping, and serving-framework state.

Caching allocators keep freed blocks reserved for reuse. Reserved memory can therefore exceed live tensor memory, while nvidia-smi can report more usage than the framework’s tensor counters.

Fragmentation can leave enough free bytes in total but no suitable block for the next allocation. Capacity planning therefore needs headroom below the GPU’s physical limit.

The practical memory equation is:

Required VRAM ≈ weights + KV cache + peak activations and workspace + runtime overhead + safety margin

That equation explains why a model can load comfortably and still run out of memory after increasing the context window, concurrency, or batch size.

It also explains the wider performance problem. During decode, repeatedly reading weights and a growing KV cache can matter as much as raw compute capacity.

👉 Over to you: Which memory bucket has caused the most capacity problems in your inference workloads?

​MCP meets agent skills​

MCP already gave agents a standard way to connect to tools, resources, and external systems.

Now it also defines a standard way to discover and load Agent Skills directly from MCP servers.

The flow is simple:

→ connect to MCP server

→ discover available skills

→ inspect skill metadata

→ load the relevant 𝗦𝗞𝗜𝗟𝗟.𝗺𝗱 only when needed

Under the hood, Skills are served through MCP’s existing Resources primitive.

That means 𝗦𝗞𝗜𝗟𝗟.𝗺𝗱, references, scripts, examples, and other supporting files are exposed as resources that the client can read on demand.

This is especially useful for context window management.

Instead of loading every workflow instruction upfront, the agent can first discover what skills are available and pull in only the one required for the current task.

A useful mental model is:

• tools = what the agent can do

• resources = what the agent can access

• skills = how the agent should perform a reusable workflow

Previously, that workflow knowledge often lived separately in docs, repos, prompt files, or custom integrations.

Now the MCP server can expose the capability and the playbook for using it together.

So you get:

→ standardized skill discovery

→ on-demand context loading

→ cleaner distribution and versioning

→ reusable workflows that travel with the server

MCP was already the connection layer.

Skills now add a standardized way to ship reusable agent know-how on top of it.

The illustration below visually summarizes everything that we discussed so far.

Read more: ​https://github.com/modelcontextprotocol/ext-skills​

Talking about MCPs, we covered everything you need to know about MCPs in the MCP crash course.

• ​Part 1 covered MCP fundamentals, the architecture, context management, etc. →​​​​

• ​Part 2 covered core capabilities, JSON-RPC communication, etc. →​​​​

• ​Part 3 built a fully custom and local MCP client →​​​​

• ​Part 4 built a full-fledged MCP workflow using tools, resources, and prompts →​​​​

• ​​​​Part 5 taught how to integrate Sampling into MCP workflows →​​​​

• ​Part 6 covered testing, security, and sandboxing in MCP Workflows →​​​​

• ​Part 7 covered testing, security, and sandboxing in MCP Workflows →​​​​

• ​Part 8 integrated MCPs with the most widely used agentic frameworks: LangGraph, LlamaIndex, CrewAI, and PydanticAI →​​​​

• ​​​​P​​​​​​art 9 covered using LangGraph MCP workflows to build a comprehensive real-world use case →

Good day!​

• A shared space for personal AI assistants.

• ​How to fine-tune LLMs in 2026.

• ​One architectural change can cut KV cache by 8x!

A shared space for personal AI assistants

People usually use an AI assistant to work through their own files or notes. It has context about one person’s work and responds to that person’s requests.

Rowboat Spaces (open-source) extends that setup to a team conversation. Each person brings their own assistant into a shared channel, with access to their own meetings, notes, and open decisions.

A team member can ask their assistant to contribute what they know. The answer appears in the channel, attributed to that person. Other members can bring in context through their assistants in the same way.

The assistants can also work with the channel history. As a team discusses a spec, they can draft and update the shared file, with each change linked to the message that prompted it.

We created a shared space, invited a team member, and asked them for a status update. They asked their assistant to answer using their own context. We then built a spec from that conversation, versioned it, and tracked each change back to the message that prompted it.

Rowboat GitHub: https://github.com/rowboatlabs/rowboat

(don’t forget to star 🌟)

How to fine-tune LLMs in 2026

If you’re using GPT or Claude, you’re using the same model as everyone else, with the same capabilities, the same cost, and no competitive edge.

But if you take a small open-source model and fine-tune it on your specific task, it can outperform a model 100x its size, at a fraction of the cost and latency.

Devs typically associate fine-tuning with a painful setup, like curating datasets, labeling outputs, and hand-crafting reward functions.

In 2026, that’s no longer the case.

Modern fine-tuning using GRPO and RULER is redefining fine-tuning.

You can now train agents that genuinely improve through experience, without writing a single reward function or collecting a single labeled example.

Today, let’s walk through exactly how!

SFT vs. Reinforcement Fine-Tuning

In supervised fine-tuning (SFT), you collect input-output pairs and the model learns to imitate them.

The problem is that SFT teaches the model what to say, not how to succeed.

For agents that search, call APIs, and reason across multiple steps, imitation isn’t enough. You want improvement through trial and error.

Think of it this way:

• SFT = studying a textbook (memorizing answers to known questions)

• RL = on-the-job training (learning from trial, error, and feedback)

This is Reinforcement Fine-Tuning (RFT). You give the model a reward signal and let it discover the best strategies on its own.

How GRPO Works

GRPO (Group Relative Policy Optimization) is the most popular RFT algorithm today. It’s the same algorithm that powered DeepSeek-R1’s reasoning capabilities.

Essentially, instead of training a separate model to score responses, GRPO generates multiple completions and grades them relative to each other.

Here’s how it works for each prompt:

1. Sample a group: Generate N completions from the current model

2. Score each one: A reward function evaluates each attempt

3. Normalize within the group: Calculate relative advantage vs. the group average

4. Update the model: Reinforce above-average behaviors, suppress below-average ones

GRPO only needs relative rankings, not absolute scores. Whether completions score 0.3, 0.5, and 0.7 or 30, 50, and 70 doesn’t matter. Only the ordering drives learning.

ART: Agent Reinforcement Trainer

GRPO is powerful, but how do you actually apply it to a real-world agent?

ART (Agent Reinforcement Trainer) is a 100% open-source framework that brings GRPO to any Python application.

Most RL frameworks are built for simple chatbot interactions, involving one input, one output, and the job is done.

Real agents are fundamentally different. They search documents, invoke APIs, and reason across multiple steps before producing an answer.

ART is built for exactly this. It provides:

• Native support for tool calls and multi-turn conversations

• Integrations with LangGraph, CrewAI, and ADK

• Efficient GPU utilization during training

Architecture

ART splits into two parts: a Client and a Backend.

The Client is where your agent code lives. It sends inference requests to the backend and records every action into a Trajectory, the complete history of one agent run.

The Backend is where the heavy lifting happens. It runs vLLM for fast inference and Unsloth-powered GRPO for training. After each training step, a new LoRA checkpoint loads automatically into the inference server.

The full training loop

1. Client sends an inference request

2. Backend generates model outputs

3. Agent takes actions in the environment (tool calls, searches, etc.)

4. Environment returns a reward

5. Trainer updates the model via GRPO

6. A new LoRA checkpoint loads into the inference server

7. Repeat, with each cycle, the model gets a little better than before

RULER: RL without manual reward functions

Defining a good reward function has always been the hardest part of RL.

Training an email agent requires labeled correct answers. Training a code agent requires test suites. Each one is its own unique engineering project.

RULER (Relative Universal LLM-Elicited Rewards) eliminates this bottleneck entirely. It uses an LLM-as-judge to compare multiple agent trajectories and rank them, with no labeled data required.

It works because of two key insights:

• Asking an LLM “rate this 0-10” produces inconsistent results

• Asking “which of these 4 attempts best achieved the goal?” is far more reliable.

And since GRPO only needs relative scores, the absolute values don’t matter anyway.

The process is three steps:

1. Generate N trajectories for a scenario

2. Pass them to an LLM judge, which scores each from 0 to 1

3. Use those scores directly as rewards in GRPO

A practical example

We put together a fully working notebook that trains a 3B model to master how to use any MCP server through reinforcement learning using ART.

Simply provide an MCP server URL, and the notebook will:

1. Query the server’s tools

2. Generate a set of input tasks that use those tools

3. Train the model on those tasks using automatic RULER evaluation

You can find more examples to adapt and get started in the ART GitHub repo.

Here’s the GitHub Repo →

​One architectural change can cut KV cache by 8x!​

In standard multi-head attention, every query head gets its own key and value head.

So a model with 64 query heads therefore stores 64 sets of key and value vectors for every token at every layer.

Multi-query attention optimizes this. It shares one KV head across all query heads. This produces the smallest cache, although that much sharing can reduce model quality.

Grouped-query attention lies between them and the visual below explains all three techniques:

It divides the query heads into groups, with each group sharing one KV head.

In Llama 3 70B, every eight query heads share one KV head. The model stores 8 sets of KV vectors instead of the 64 that an equivalent MHA layout would need.

That makes this part of the KV cache 8x smaller. It also reduces the amount of KV data read during decoding by the same factor, assuming the remaining dimensions and precision stay unchanged.

This is part of the model architecture, so it cannot be enabled on an arbitrary MHA model with a serving flag.

And this is only one way to control KV-cache cost.

We covered GQA alongside ​11 other methods in the KV Cache Engineering article. It explains which part of the cost each method reduces and whether it requires a different model architecture or only a serving-engine change.

​Read it here →​

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

An LLM request rarely maps to one model call once retrieval and tools enter the pipeline.

A slow response could come from the embedding model, retrieval step, generation call, or an agent repeating the same tool. Measuring the latency of the outer HTTP request only tells you that the pipeline was slow.

Dynatrace has published a runnable reference application for inspecting the operations inside it.

The repository contains a Python travel advisor with basic, RAG, and agentic execution paths. It uses Amazon Bedrock for generation and embeddings, while OpenTelemetry and Traceloop record the application workflow and export its traces to Dynatrace through OTLP.

This lets you inspect the completion request as a distributed trace instead of treating the LLM pipeline as one opaque API call. The repository also includes sample dashboards and deployment files for running the application locally or on a small Kubernetes cluster.

It is a useful reference for instrumenting an LLM application before debugging it under real traffic.

Find the GitHub repo here →

(don’t forget to star 🌟)

Thanks to Dynatrace for partnering today!

Easiest way to run agent harnesses using local models

Local LLMs turn model selection into a resource allocation problem surprisingly quickly.

In practice, the model weights are only one part of the memory budget.

• Context length changes KV-cache usage.

• Quantization changes both memory and throughput.

• Available RAM or VRAM does not tell much about whether inference will actually be fast enough for an agent loop.

• And agent harnesses tend to generate long, repeated inference workloads rather than isolated prompts.

That makes “what model can my laptop run?” a harder question than it sounds.

A reliable and simple solution is actually implemented in Magnitude (open-source).

npm i -g @magnitudedev/cli
magnitude setup

It profiles the machine, benchmarks the hardware, estimates which models are practical rather than merely loadable, and recommends a set of candidates before connecting one to Claude Code, Codex, OpenCode, Pi, etc.

Once done, the actual harness setup becomes quite easy.

You can run two commands, pick the recommendation, and connect the harness.

We recoded this video that walks through the entire flow from machine profiling to a working local coding agent.

Chapters:

00:00 → Intro
00:17 → What Magnitude is
00:45 → Setup in two commands
01:29 → Why local model selection gets complicated
02:11 → Hardware profiling + model recommendations
02:59 → Choosing a harness
03:22 → Connecting an existing harness

Find the repo here → http://github.com/magnitudedev/magnitude

(don’t forget to star it ⭐️)

Good day!

An agent needs more than a model and a collection of tools. It also needs an execution layer that manages how the model, tools, and application state interact.

We have published Part 1 of a new hands-on series where we build this layer using LangChain and LangGraph.

Read Part 1 of the series here →

This chapter covers:

• The responsibilities of an agent harness

• How models, messages, prompts, and tools fit together

• How tool calls are generated, executed, and returned

• How the harness determines whether another model call is required

• How state moves through the execution lifecycle

• Where LangChain abstractions fit into the implementation

• How these components combine into a working agent

Everything is explained from scratch, with the corresponding implementation.

Start reading Part 1 here →

Why care?

The behavior of an agent is determined by more than the model.

Consider a tool that fails halfway through a run. The surrounding system must decide whether to retry it, expose the failure to the model, request human input, or stop the execution. Similar decisions apply to message history, persistent state, permissions, and interrupted runs.

These responsibilities belong to the agent harness.

This is why production agent platforms expose capabilities such as sessions, checkpoints, tracing, evaluation, human approval, and sandboxed tool execution. Each capability addresses a problem that appears when an agent moves from a single request to a stateful application.

For an AI engineer, this introduces a set of system design questions:

• Which state belongs to the current run?

• Which state should persist across conversations?

• How are tool errors represented and propagated?

• Which transitions can be model-driven?

• Which transitions should remain deterministic?

• How can an interrupted execution resume without repeating completed work?

LangChain provides interfaces for models, messages, and tools. LangGraph provides explicit state transitions and execution control. The series uses both to implement these concepts and examine the behavior of the resulting system.

Part 1 establishes the execution model. The upcoming chapters will extend it with graph-based control, persistence, memory, failure handling, and other production requirements.

Read the first implementation of the production agent harness here →

👉 Over to you: Which part of an agent harness has been the most difficult to implement?

Your agent harness should repair itself

An agent can complete a difficult task, save the procedure that worked, and reuse it later.

That is a useful form of runtime learning because the improvement lives outside the model weights and remains available across sessions.

Production failures create a different problem. The useful information is spread across the execution trace, tool results, local code, configuration, and the final outcome.

A patch may fix the incident, but the failure only becomes reusable knowledge when the system also records the expected behavior and tests it again.

This gives agent self-improvement two separate feedback paths:

• successful runs can produce reusable procedures

• failed runs can produce reviewed fixes and regression cases

Hermes implements the first path through persistent skills and offline optimization. Its Opik integration provides the trace data needed for the second.

Let’s look at where the two paths differ and how they fit together.

Agent improvement without weight updates

A model does not need to update its weights for an agent system to improve.

The agent can instead change the artifacts around the model. These include its instructions, skills, tool descriptions, retrieval policy, configuration, and evaluation cases. If a later run uses those changes, the system’s behavior has changed even though the underlying model is identical.

That distinction matters because agent failures rarely map to one bad final answer. An agent makes a sequence of decisions. It selects tools, constructs arguments, interprets results, updates its plan, and decides when to stop.

A useful learning record therefore needs more than the final output. It needs enough evidence to identify which decision failed and enough structure to alter future runs.

Procedural memory in Hermes

Hermes Agent treats skills as procedural memory. After completing a complex task, it can preserve the useful method in a SKILL(.)md file. A later task can load that skill instead of deriving the same procedure again.

The saved artifact is readable and editable. It can specify when the procedure applies, which tools to use, what sequence to follow, and which constraints must remain true. This makes the learned behavior easier to inspect than a weight update.

Hermes also manages this collection over time. Its skill curator can revise weak descriptions, merge overlap, and remove stale instructions. That maintenance is necessary because an unbounded skill directory eventually creates its own retrieval problem. The agent must still select the right procedure from everything it has saved.

Hermes has a second path for improving these artifacts through Hermes Forge. It uses GEPA, or Genetic-Pareto Prompt Evolution, to search over variants of prompts, skills, and tool descriptions.

GEPA runs outside the live agent loop. It evaluates candidate variants against a defined task set, uses execution feedback to propose new candidates, and keeps variants that improve the objective without violating constraints. This is optimization over text and configuration, not GPU fine-tuning.

These two mechanisms solve related but different problems:

• runtime skill creation preserves a procedure that worked during a task

• offline evolution compares variants against an evaluation set

Limits of the Hermes skill loop

A procedure saved after a successful run inherits the agent’s judgment about that run. The task may have completed with a brittle method, an unnecessarily expensive route, or an assumption that happened to hold once.

Automatic skill updates carry a related risk. A generated revision can overwrite a carefully written instruction with a weaker one unless the system versions changes and evaluates them before promotion.

Offline optimization has another boundary. Its result depends on the cases and objective supplied to it. A failure that only appears under production inputs will not influence the optimizer until someone captures that case and adds a useful evaluation signal.

A trace records the incident, but it does not convert the incident into changed behavior.

Production failure feedback with Opik

A production incident becomes reusable after the system records four pieces of information:

1. the first step that diverged from expected behavior

2. whether the fault came from the model, a tool, configuration, or application code

3. the change that corrects the fault without hiding another problem

4. the test that will fail if the same behavior returns

Opik (GitHub Repo) connects these records in one debugging workflow. Its Hermes plugin records each agent turn as a root trace, with separate spans for model and tool calls. Ollie can inspect the trace alongside a connected project, propose a code change, rerun the original input, and add the case to a Test Suite.

The full sequence is:

Failed trace -> diagnosis -> proposed diff -> human approval -> rerun -> regression test

Each stage produces evidence for the next one. The workflow does not require the agent to edit production code without review, and it does not treat a plausible patch as proof that the incident is resolved.

Execution traces

The opik-hermes plugin captures a root trace for each Hermes turn. Model spans include the messages, output, provider, token usage, and cost. Tool spans include the tool arguments and returned results.

This structure matters because the visible error often occurs after the actual fault. A final answer may be wrong because an earlier search returned nothing, a tool received malformed arguments, or the agent accepted an invalid intermediate result. The span tree preserves that causal order.

In the image above, we used a Hermes agent for a request about Google’s Gemma model family. Opik recorded the model and tool activity under the same turn, so the complete route was available without reconstructing it from terminal output.

Failure detection

Some failures are explicit exceptions. Others complete normally but produce a poor tool choice, an incomplete answer, excessive latency, or an unexpected cost.

Opik can surface candidate failures through error status, feedback scores, online evaluation, latency, and cost. Its alert system can apply thresholds over these signals and send the resulting event to Slack, PagerDuty, or a general webhook.

Detection only identifies runs that need investigation. It does not determine the root cause or decide which artifact should change.

Trace and source-level diagnosis

A trace explains what happened at runtime. Root-cause analysis often requires the source and configuration that produced it.

Running opik connect from the project directory gives Ollie session-scoped access to that project. Ollie can inspect relevant files, compare them with the failing span tree, and propose a Git-style diff. File writes still require explicit approval.

In my test, I asked Hermes to install pip-audit and scan the current Python environment:

Install the pip-audit tool and run a security
vulnerability scan on the current Python environment.

The run failed during environment validation. The trace showed the failure, while the connected configuration explained its source. The execute_code tool used an unsupported cloud_runner environment value.

Ollie proposed a small configuration change and explained which validation error it addressed. The diff remained pending until I approved it.

The Hermes agent's execute_code tool is
configured with environment type cloud_runner,
which is not a supported value.

Ollie performs the trace and code analysis. The developer decides whether the diagnosis and diff are valid.

Reruns and Agent Sandbox

Approval permits a change to be applied. Verification requires another execution.

Ollie can rerun the agent through opik connect using the original trace input. The updated execution returns to Opik as a new trace, which provides a direct comparison with the failing run.

After approving the environment configuration change, I reran the same pip-audit request. The first execution stopped during validation. The second completed with a supported environment configuration.

Opik’s Agent Sandbox covers broader interactive testing through the Agent Playground. The opik endpoint command runs the local agent and makes it available in the Opik UI. Prompts, model settings, and tool definitions can then be tested while each execution generates a complete trace.

The two commands have different responsibilities:

• opik connect gives Ollie access to the connected project for source inspection, approved file changes, and reruns

• opik endpoint runs the agent locally and connects it to the Agent Playground for interactive testing

They can run together when a debugging session needs both code-aware diagnosis and controlled test executions.

For the Hermes example, the initial pip-audit request failed during environment validation. After the configuration change was approved, the same request completed successfully. The second trace verified the runtime behavior of the fix instead of relying on a diff that only appeared correct.

Regression tests

The failed input becomes reusable only when the system records what should happen next time.

Opik Test Suites express those expectations as natural-language assertions. An LLM judge checks each assertion against the agent output and reports pass or fail. Execution policies can also require multiple successful runs, which helps when the agent is nondeterministic.

The following suite preserves the three Hermes tasks from the original test set. run_hermes is the adapter between an Opik test item and the Hermes CLI.

import subprocess

import opik

def run_hermes(item):
    process = subprocess.run(
        ["hermes", "chat", "-q", item["request"]],
        capture_output=True,
        text=True,
        check=False,
    )
    return {
        "output": process.stdout,
        "error": process.stderr,
        "exit_code": process.returncode,
    }


client = opik.Opik()
suite = client.get_or_create_test_suite(
    name="Hermes Agent Regression",
    project_name="Hermes Agent",
    global_assertions=[
        "The requested task completes without an execution or environment validation error.",
        "The response directly provides all information requested by the user.",
    ],
    global_execution_policy={"runs_per_item": 2, "pass_threshold": 2},
)

suite.insert(
    [
        {
            "data": {
                "request": "Check if the 'requests' package is installed and print its version."
            },
            "assertions": [
                "The response states whether requests is installed and prints its version when available."
            ],
        },
        {
            "data": {
                "request": "List the files in the current directory and identify the project."
            },
            "assertions": [
                "The response lists the current directory contents and identifies the project."
            ],
        },
        {
            "data": {
                "request": "Run Python and print the interpreter version together with the first ten installed packages."
            },
            "assertions": [
                "The response prints the Python interpreter version and the first ten installed packages."
            ],
        },
    ]
)

result = opik.run_tests(test_suite=suite, task=run_hermes)
print(f"Pass rate: {result.pass_rate:.0%}")

The suite does not teach Hermes by itself, but rather the combined Hermes and Opik workflow supplies a durable evaluation signal for prompt revisions, tool updates, configuration changes, and evolved skills.

Each candidate change can be checked against the same failure before release.

Combined Hermes and Opik workflow

Hermes and Opik change different artifacts at different points in the agent lifecycle.

Hermes preserves methods that proved useful during execution. Its offline optimizer can compare new skill, prompt, and tool-description variants against a task set.

Opik supplies production evidence. It records the execution, supports trace-aware diagnosis, keeps code changes behind approval, reruns the original input, and stores the expected behavior as a regression case.

That produces a more complete loop:

1. Hermes completes tasks and saves reusable procedures.

2. Production runs generate traces and feedback in Opik.

3. A failed trace is diagnosed against the project source.

4. A developer reviews the proposed change.

5. The original input is rerun against the updated agent.

6. The failure and expected behavior become a test case.

7. Later changes are evaluated against the accumulated suite.

The system is not autonomously learning from every production incident. It is turning production incidents into structured, reviewable inputs for improvement. That narrower claim is also the useful one.

Hermes and Opik setup

Hermes supports Linux, macOS, WSL2, and native Windows. The following flow uses the managed Linux, macOS, or WSL2 installation.

Install Hermes

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash

source ~/.bashrc  # or ~/.zshrc

hermes setup

The setup wizard configures the model provider, credentials, and enabled tools.

Install the Opik integration

Install opik-hermes in the same Python environment as Hermes:

~/.hermes/hermes-agent/venv/bin/python3 -m pip install opik-hermes

The package registers the plugin entry point and installs the Opik SDK dependency.

Enable trace collection

Add the plugin to ~/.hermes/config.yaml:

plugins:
  enabled: [opik]

For a local Opik deployment, add the endpoint and project name to ~/.hermes/.env:

# .env file

OPIK_URL_OVERRIDE=http://localhost:5173/api
OPIK_PROJECT_NAME=Hermes Agent

For Comet-hosted Opik, configure the workspace and API key instead:

# .env file

OPIK_API_KEY=...
OPIK_WORKSPACE=your-workspace
OPIK_PROJECT_NAME=Hermes Agent

Verify trace collection

Restart Hermes after enabling the plugin. Each turn should now appear in the configured Opik project.

hermes chat -q "List the files here and report how many there are."

The trace should contain one root record for the turn, model spans, and a span for each tool call.

Connect the project to Ollie

Run the bridge from the agent project directory:

opik connect --project "Hermes Agent"

The connection is opt-in and scoped to the launched session. Ollie can inspect files within the connected project and must request approval before writing changes.

Evaluation and release constraints

This workflow improves the feedback pah, but it does not remove the hard parts of agent evaluation.

• Failure selection still matters. A timeout, a bad answer, and an invalid tool call require different assertions. Converting every anomalous trace into a test will produce a noisy suite.

• LLM judges are not ground truth. Natural-language assertions are convenient for behavioral checks, but deterministic validators remain preferable for schemas, exit codes, database state, and exact tool arguments.

• A passing rerun proves one case. The patch can still affect other tasks. Broader suites and repeated execution are needed before treating it as a safe release.

• Learned artifacts need versioning. Skills, prompts, tool descriptions, configuration, and tests should remain tied to the change that introduced them. Otherwise, the team cannot explain why behavior changed or roll back a weak update.

These constraints are part of the design, not exceptions to it. Agent improvement becomes reliable when the feedback is observable, the change is reviewable, and the result is tested against the behavior that originally failed.

Run Opik locally

Opik is Apache-2.0 licensed, and the complete platform can run inside your own infrastructure. The self-hosted deployment includes the backend, web application, tracing, datasets, evaluations, prompt management, and agent optimization components.

On Linux or macOS, the local stack starts with three commands:

git clone https://github.com/comet-ml/opik
cd opik
./opik.sh

The script starts the full Opik suite. Once the services are healthy, the interface is available at http://localhost:5173. The Hermes plugin can then send model and tool spans to this local deployment using the OPIK_URL_OVERRIDE configuration from the setup above.

Check out Opik on GitHub →

(don’t forget to star 🌟)

👉 Over to you: Which production failure would be most valuable to preserve as a regression case for your agent?

Good day!

One limitation of coding agents does not get discussed enough.

They can inspect every file in your repository, but they cannot see how that code behaves after deployment. So when an endpoint slows down or errors spike, the agent suggests fixes from static code while the useful evidence sits in production traces and logs.

Dynatrace has open-sourced a repository that brings this runtime context into coding agents.

The MCP server provides access to live Dynatrace data. The included skills teach agents how to query and interpret that data, while reusable prompts define complete investigations.

For instance, its performance regression prompt compares P95 latency, error rate, and throughput before and after a deployment. It then finds the slowest trace, maps the bottleneck span back to the relevant workspace code, and recommends either a rollback or a targeted hotfix.

The skills work with Claude Code, Cursor, GitHub Copilot, OpenCode, Gemini CLI, and other compatible agents.

Find the GitHub repo here →

(don’t forget to star 🌟)

Thanks to Dynatrace for partnering today!

4 speculative decoding variants

Large-model decoding often spends one full model run producing a single token, even when the next few tokens are predictable.

Speculative decoding tries to get several useful tokens from that run instead of just one.

A cheaper path drafts a short continuation. The large model checks those tokens together, keeps the accepted prefix, and corrects the first mismatch. Draft tokens never reach the output without verification from the large model.

The main difference between speculative decoding variants is how they produce that draft. Some use a second model, while others build drafting into the target model itself.

Let’s look at the four approaches and the engineering tradeoffs behind each one.

To dive deeper into the full LLMOps lifecycle, we have covered every bit of this in the LLMOps course, starting from fundamentals to production.

Start here →

The common speculative decoding loop

Every variant has two stages: draft several tokens cheaply, then verify them with the target model in parallel.

Suppose the drafter proposes five tokens. If the target accepts the first three and rejects the fourth, the first three move to the output. The target supplies the replacement at position four, while the remaining draft is discarded.

When all five tokens are accepted, the verifier can add one more token from the same pass. The sequence advances by six tokens after one target-model run.

For greedy decoding, this means checking whether the predicted tokens match. Sampling requires an acceptance and correction rule based on both probability distributions.

The original algorithm preserves the target model’s distribution, so acceleration does not require changing the result.

1. Two-model speculative decoding

The original approach pairs a small draft model with a larger target model. The small model generates a few tokens sequentially, then the target verifies the full block at once.

This is still the first setup to test. It works with an unchanged target model, and most inference stacks already support some form of assisted generation. The original paper reported 2x to 3x acceleration on T5-XXL with identical outputs.

The draft model needs to be cheap enough to justify its work but accurate enough to earn a useful acceptance rate. A larger drafter often predicts better, yet its added latency can reduce the final speedup.

There is also a clear memory cost. The server holds another set of weights and a separate KV cache. Closely matched model families are usually easier to serve because their tokenizers and output behavior align better.

2. EAGLE drafts from hidden states

EAGLE removes the independent draft language model. It trains a lightweight module to predict the target model’s second-to-top-layer features, then converts those predicted features into candidate tokens.

Instead of asking a separately trained model to approximate the target, EAGLE drafts from the target model’s own internal representation.

Feature prediction has an ambiguity problem because different tokens can produce similar hidden states. EAGLE also feeds in the token sequence shifted forward by one position, giving the draft module enough information to make the feature prediction more precise.

The paper reported 2.7x to 3.5x latency speedups for LLaMA2-Chat 70B and roughly doubled throughput in its evaluated setup. In practice, EAGLE fits best when the checkpoint and serving engine can be managed together because its draft module is trained for a specific target.

3. Medusa uses parallel prediction heads

Medusa adds several small decoding heads to the target model. One head predicts the next token, another predicts two positions ahead, and later heads predict positions farther into the continuation.

All heads run from the same model state. This makes drafting parallel, but it also means one head cannot condition on what another head predicted. Their individual guesses do not always combine into a consistent sequence.

Medusa handles this by arranging alternatives into a candidate tree. Tree attention lets the target verify several possible continuations in one pass, and the verified branch determines the accepted prefix.

Medusa-1 freezes the backbone and trains only the added heads. Its paper reports more than 2.2x speedup without changing the backbone’s generation quality. Medusa-2 jointly tunes the heads and backbone, reporting 2.3x to 3.6x, but it requires a more involved training recipe.

Consider Medusa when a second model is undesirable and training small heads is feasible. Candidate-tree width becomes the main serving control because wider trees improve coverage while increasing verification work and temporary memory.

4. LayerSkip exits from early layers

LayerSkip uses the target model as both drafter and verifier. Early transformer layers produce draft tokens, while the remaining layers check and correct them.

This removes the second model and added prediction heads. Drafting and verification also share weights, vocabulary, activations, and part of the computation.

LayerSkip does require the right checkpoint. Its training recipe uses layer dropout and an early-exit loss so intermediate layers learn to produce useful predictions. A compatible checkpoint needs no separate drafter or added heads, but an arbitrary pretrained model will not provide reliable early exits automatically.

The paper reported speedups up to 2.16x on CNN/DailyMail summarization, 1.82x on coding, and 2.0x on TOPv2 semantic parsing. I would treat LayerSkip as a training decision made before deployment, not an optimization applied later to any model.

Choosing between the four variants

The number that matters is accepted tokens per target-model pass after accounting for drafting time and verification overhead.

• Two-model decoding is the easiest place to begin because it leaves the target untouched.

• EAGLE and Medusa reduce the cost of carrying another model but introduce target-specific training and serving code.

• LayerSkip has the fewest inference-time components, provided the checkpoint was trained for early exits.

The benchmark should reproduce the real workload. Low-temperature code generation usually produces more agreement than high-temperature writing. Heavy batching can also reduce the gain because the GPU is already doing more useful work during each decode step.

All four methods still rely on KV caching. Speculative decoding reduces the number of target-model runs, while the KV cache prevents accepted context from being recomputed during the next iteration.

To dive deeper into the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:

• Read Part 1 on fundamentals of LLMOps here →

• Read Part 2 on understanding the core building blocks of LLMs →

• Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →

• Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →

• Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →

• Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →

• Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →

• Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →

• Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →

• Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →

• Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →

• Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →

• Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →

• Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions.

👉 Over to you: Which speculative decoding approach best matches the models you can modify and the memory available in your serving stack?

Good day!

The difference comes from what the serving stack loads into GPU memory. A rank-8 LoRA adapter contains about 40 MB of task-specific weights. Merging it produces another 15.2 GB copy of the model.

A one-endpoint-per-variant layout also creates 100 separate scaling pools. Each one has its own workers, cold starts, and idle capacity.

Keeping the adapters separate changes the layout. One 15.2 GB base model and roughly 4 GB of adapters leave about 60 GB of an 80 GB GPU for KV cache.

Each request still reaches its fine-tuned variant. vLLM applies the requested adapter while every variant reuses the same base weights.

Memory is the obvious savings, but worker reuse matters more in production. A shared endpoint routes every request to the same pool, while separate endpoints can leave one model warm as another request waits for a new worker.

In this article, we will test both layouts under the same traffic. We will train the adapters on a Pod, serve them through vLLM on Runpod Serverless, and compare one shared endpoint with separate per-model endpoints.

Runpod Serverless works well for this test because flex workers scale to zero and bill per second. Its worker states and request timing let us separate GPU startup time from time spent waiting for capacity.

Four deployment types are available:

• Merged, one endpoint per variant → The adapter becomes part of a complete copy of the base model.

• Unmerged, registered at startup → One base model is shared, and every adapter loads when the worker starts.

• Unmerged, resolved at request time → The engine fetches an adapter when a request first needs it.

• Hosted per tenant → The provider manages the base model and adapters behind an API.

We will compare all four, build the shared-base layout on Runpod, and measure it against one endpoint per model. To follow along, create a Runpod account and work through the build below.

A quick note on the terminology:

• Adapter is a small set of trained matrices that store the fine-tuning updates.

• An endpoint is the API and scaling configuration. A worker is the running process attached to a GPU. So one endpoint may start several workers, and each worker loads the configured model weights.

Let’s begin!

1) Merged, one endpoint per variant

Merging adds the adapter’s changes into the base model so the serving engine no longer needs to load or apply the adapter separately.

That simplicity costs memory since each fine-tuned variant becomes another 15.2 GB copy of the model. Three variants require 45.6 GB, even though each one started as a 40 MB adapter.

Each variant also has its own endpoint and workers in this setup. When no worker is ready, the request must wait for Runpod to start a GPU worker and load the model.

RunPod measured this startup delay at 324 seconds in one test with a 32B model on two H200s, and four configuration changes reduced it to about 91 seconds without changing the model’s speed once it was running.

Separate endpoints also prevent workers from helping one another. An idle worker holding the extraction model cannot process a request for the SQL model. The SQL request starts another worker while the first GPU remains unused.

Merging still makes sense when variants need different base models, require different GPUs, or need complete isolation. When all the adapters share one base model, separate copies mostly duplicate memory and workers.

2) Unmerged, registered at startup

The base model is loaded once. Instead of merging every fine-tune into a complete model, the worker loads the smaller adapters beside it.

Each request invokes the adapter it needs. vLLM applies that adapter to the shared base model and generates the response. Extraction, SQL, and routing requests can therefore use the same GPU without loading three complete models.

Every worker loads the same base model and adapter set, so any worker can process any variant. The worker pool grows with traffic rather than the number of fine-tunes.

Adding another variant now requires about 40 MB of adapter memory instead of another 15.2 GB model.

There are two practical limits.

All adapters must come from the same base model. They also consume GPU memory, leaving less room for active requests.

3) Unmerged, resolved at request time

Registering adapters at startup (technique 2) works when the catalogue is small and rarely changes.

A larger catalogue may contain hundreds of customer adapters, while only a few receive requests at any given time.

Request-time loading keeps those adapters in storage instead of loading all of them when the worker starts.

When a request needs an adapter that is not loaded, vLLM fetches it and adds it to the running worker.

Later requests can reuse that adapter without restarting the endpoint. New adapters can also be added without another deployment.

This gives you a much larger catalogue, but the first request to an unloaded adapter takes longer. It must wait for the adapter to be fetched and copied into GPU memory.

Use request-time loading when adapters change frequently, or the full catalogue cannot fit in GPU memory. For a small and stable catalogue, loading everything at startup is simpler and avoids the first-request delay.

4) Hosted per-tenant

A provider stores the adapters and serves them behind an API, usually priced per token rather than per GPU hour. The provider controls the deployment layout and limits what you can measure or tune.

There is nothing to deploy yourself. The rest of this issue assumes you want control of the GPU.

Here are all four options we discussed, summarized in a table:

And this graph shows how the required worker count changes as the catalogue grows:

The merged setup needs more workers for every new variant, while both unmerged setups reuse the same worker pool and add workers only when request volume increases.

Building option two on RunPod

This experiment measures whether one shared endpoint starts fewer workers and causes fewer cold starts than three separate endpoints.

To test this, we will first train three adapters for extraction, SQL generation, and request routing. Then we will deploy them in two ways:

• One endpoint that loads the base model and all three adapters

• Three endpoints that each load the base model and one adapter

Both setups will receive the same requests. We will then compare how long those requests wait, how many workers start, and how much GPU time each setup uses.

We will run it on a Runpod Pod and then move the adapters to Runpod Serverless for the deployment test.

Serverless reports whether workers are running, starting, throttled, or waiting for GPU capacity. That helps us separate cold-start time from model execution.

Runpod’s vLLM worker also accepts adapter settings through environment variables, so we can test both layouts without building a custom container.

The experiment uses a 1.5B Qwen model to keep the cost low. The earlier example used a 7B model, but the shared-base mechanism remains the same.

Step 1: Keys:

Create a Runpod API key. Create a Hugging Face token with write access. Export both in your local terminal:

export RUNPOD_API_KEY=”your-key-here”

export HF_TOKEN=”your-huggingface-token”

The first key creates and manages Runpod resources. The second is passed into the training Pod so it can upload the finished adapters.

Step 2: Provision a Pod:

We used Runpod’s command-line client instead of creating the machine in the dashboard. Install it and connect it to your account:

wget -qO- cli.runpod.net | sudo bash # Install on Linux
brew install runpod/runpodctl/runpodctl # Install on macos

bash <(curl -sL cli.runpod.io)

runpodctl config --apiKey “$RUNPOD_API_KEY”

Now create one Pod from Runpod’s PyTorch template:

runpodctl pod create \
  --name lora-training \
  --template-id runpod-torch-v21 \
  --gpu-id “NVIDIA GeForce RTX 4090” \
  --volume-in-gb 30 \
  --env “{\”HF_TOKEN\”:\”$HF_TOKEN\”}”

This call starts one RTX 4090, mounts a 30 GB working volume, and makes the Hugging Face token available inside the Pod. The 1.5B model does not need a larger training GPU.

The command returns a Pod ID. Once the Pod is running, retrieve its SSH command and connect to it:

runpodctl ssh info YOUR_POD_ID

Steps 3 through 6 run in that SSH session. The working volume survives a stopped Pod, but Runpod deletes it when the Pod is terminated. We upload the adapters before terminating the machine.

Step 3: Install the training packages

The PyTorch template already includes a CUDA-compatible copy of PyTorch. Keep it. We only need the fine-tuning and dataset libraries:

pip install -U trl peft datasets accelerate huggingface_hub

This is the whole software setup. trl runs supervised fine-tuning, peft creates LoRA adapters, and datasets reads the training files.

Step 4: Create three small datasets

We want routing mistakes to be obvious.

So the extraction adapter returns JSON, the SQL adapter returns a query, and the routing adapter returns a support queue. These are implemented below:

import json

from pathlib import Path

def extract(i):
    note = f"Order ORD-{40000+i} has {i % 5 + 1} items. Ship to Lisbon by DHL."
    answer = json.dumps({”order_id”: f”ORD-{40000+i}”, “city”: “Lisbon”,
                         “quantity”: i % 5 + 1, “carrier”: “DHL”})
    return “Extract the order details. Reply with JSON only.”, note, answer

def sql(i):

    question = f”How many shipped orders have amount over {100+i}?”
    answer = f”SELECT COUNT(*) FROM orders WHERE status = ‘shipped’ AND amount > {100+i};”
    return “Answer with one SQLite query only.”, question, answer

def route(i):

    messages = ["My card was charged twice.", "Tracking has not moved.", "I cannot sign in."]
    queues = [”billing”, “shipping”, “account”]
    choice = i % 3
    ticket = f”Ticket {1000+i}: {messages[choice]}”
    return “Return the support queue only.”, ticket, queues[choice]

Next, we generate the data:

Path(”data”).mkdir(exist_ok=True)

for task, make_example in {”extract”: extract, “sql”: sql, “route”: route}.items():
    with open(f”data/{task}.jsonl”, “w”) as file:

        for i in range(600):
            system, user, answer = make_example(i)
            row = {
                “prompt”: [{”role”: “system”, “content”: system},
                           {”role”: “user”, “content”: user}],
                “completion”: [{”role”: “assistant”, “content”: answer}],
            }
            file.write(json.dumps(row) + “\n”)

Once done, you should see 600 rows in each file. TRL recognizes this prompt-and-completion format and trains on the answer rather than asking you to assemble a chat transcript by hand.

Step 5: Train one adapter per task

Next, we have the training script and the first half of it defines the shared base model and the LoRA adapter:

import argparse
import torch
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

parser = argparse.ArgumentParser()
parser.add_argument(”task”, choices=[”extract”, “sql”, “route”])
task = parser.parse_args().task
base = “Qwen/Qwen2.5-1.5B-Instruct”

adapter = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=[”q_proj”, “k_proj”, “v_proj”, “o_proj”,
                    “gate_proj”, “up_proj”, “down_proj”],
    task_type=”CAUSAL_LM”,
)

The rank, r=16, controls the size of the adapter. The target list tells PEFT which parts of the transformer may receive those small updates. The base weights remain frozen.

Next, the trainer:

training = SFTConfig(
    output_dir=f”adapters/{task}”,
    num_train_epochs=3,
    per_device_train_batch_size=8,
    gradient_accumulation_steps=2,
    learning_rate=2e-4,
    max_length=512,
    save_strategy=”no”,
    report_to=”none”,
    model_init_kwargs={”dtype”: torch.bfloat16},
)

trainer = SFTTrainer(
    model=base,
    args=training,
    train_dataset=load_dataset(”json”, data_files=f”data/{task}.jsonl”, split=”train”),
    peft_config=adapter,
)

trainer.train()

trainer.save_model(f”adapters/{task}”)

Train the three adapters one after another:

python train.py extract
python train.py sql
python train.py route

Each output directory should contain adapter_config.json and adapter_model.safetensors. At rank 16 across all seven projection modules, each adapter_model.safetensors is about 70 MB.

It does not contain another multi-gigabyte copy of Qwen. That is the first check that we kept the adapters separate.

Step 6: Upload the adapters

The Hugging Face CLI creates each repository if it does not exist and uploads the matching folder. Replace YOUR_USERNAME before running these commands:

hf upload YOUR_USERNAME/qwen15-lora-extract adapters/extract .
hf upload YOUR_USERNAME/qwen15-lora-sql adapters/sql .
hf upload YOUR_USERNAME/qwen15-lora-route adapters/route .

Open each repository and check that both adapter files are present. You can now stop or terminate the training Pod.

Step 7: Deploy the shared Serverless endpoint

Return to the local terminal from step 1. Search the Runpod Hub for the current vLLM worker:

runpodctl hub search vllm

Copy the ID of Runpod’s vLLM listing. Save it along with your Hugging Face username and the adapter list:

export VLLM_HUB_ID=”id-from-the-search-result”
export HF_USER=”your-hugging-face-username”

export LORA_MODULES=”[
  {\”name\”:\”extract\”,\”path\”:\”$HF_USER/qwen15-lora-extract\”},
  {\”name\”:\”sql\”,\”path\”:\”$HF_USER/qwen15-lora-sql\”},
  {\”name\”:\”route\”,\”path\”:\”$HF_USER/qwen15-lora-route\”}
]”

Now create the endpoint:

runpodctl serverless create \
  --hub-id “$VLLM_HUB_ID” \
  --name qwen-lora-shared \
  --gpu-id “NVIDIA GeForce RTX 4090” \
  --workers-min 0 \
  --workers-max 3 \
  --idle-timeout 5 \
  --env MODEL_NAME=Qwen/Qwen2.5-1.5B-Instruct \
  --env ENABLE_LORA=true \
  --env MAX_LORAS=3 \
  --env MAX_LORA_RANK=16 \
  --env “LORA_MODULES=$LORA_MODULES”

When you deploy from a Hub ID, the GPU type and container disk come from the Hub release.

• The --gpu-id flag overrides that default.

• ENABLE_LORA turns on adapter support.

• MAX_LORAS=3 lets requests for all three adapters share one batch.

• LORA_MODULES registers the names that later appear in each request’s model field.

The endpoint starts with no workers and can scale to three. This lets it return to zero after its five-second idle period, which we use for the cold-start test in step 10.

The command returns the new endpoint ID. Save it and inspect the deployment:

export ENDPOINT_ID=”id-from-the-create-result”

Step 8: Confirm the adapters loaded

curl -s "https://api.runpod.ai/v2/$ENDPOINT_ID/openai/v1/models" \
  -H "Authorization: Bearer $RUNPOD_API_KEY" | python -m json.tool

This request starts a worker if none is running, so expect it to take a while the first time. The response should list the base model plus extract, sql, and route. If you see only the base model, stop here. The endpoint did not load the adapters, so every later measurement would test the wrong thing.

Step 9: Send one request to each adapter

The URL stays the same. Only the model field changes:

curl -s “https://api.runpod.ai/v2/$ENDPOINT_ID/openai/v1/chat/completions” \
  -H “Authorization: Bearer $RUNPOD_API_KEY” \
  -H “Content-Type: application/json” \
  -d ‘{”model”:”sql”,”messages”:[{”role”:”user”,”content”:”How many shipped orders have amount over 500?”}],”max_tokens”:48}’

Run the same request with extract and route, using a matching prompt for each task. If the outputs have the expected formats, one endpoint is switching between three fine-tunes.

You can automate a warm latency check with a small Python script. Install openai in that local shell if you do not already have it. This measures the complete request, which is enough to catch a large switching penalty:

import os, statistics, time
from openai import OpenAI

client = OpenAI(
    api_key=os.environ[”RUNPOD_API_KEY”],
    base_url=f”https://api.runpod.ai/v2/{os.environ[’ENDPOINT_ID’]}/openai/v1”,
)

prompts = {
    “extract”: “Extract: Order ORD-40100 has 2 items. Ship to Lisbon by DHL.”,
    “sql”: “How many shipped orders have amount over 500?”,
    “route”: “Tracking has not moved.”,
}

for model, prompt in prompts.items():

    times = []
    for _ in range(5):

        started = time.perf_counter()
        client.chat.completions.create(
            model=model,
            messages=[{”role”: “user”, “content”: prompt}],
            max_tokens=48,
            temperature=0,
        )
        times.append(time.perf_counter() - started)
    print(model, f”{statistics.median(times):.2f}s median”)

In our warm tests, the adapters stayed within the base model’s normal variation. The serving layout saved memory without adding a visible request-time penalty.

The script prints one timing result for each adapter. Each number is the median time across five requests:

It does not print the generated answers. Run the script against your endpoint and replace these placeholders with the three values from your terminal. Those are the numbers to compare with the separate-endpoint test in step 11.

Step 10: Measure a cold start

Step 9 measured requests while a worker was already running. This test starts after the endpoint has returned to zero workers.

Check the endpoint from your local terminal:

curl -s “https://api.runpod.ai/v2/$ENDPOINT_ID/health” \
  -H “Authorization: Bearer $RUNPOD_API_KEY” | python -m json.tool

The workers section should show zero workers in every state, including throttled:

“workers”: {”idle”: 0, “initializing”: 0, “ready”: 0,
            “running”: 0, “throttled”: 0, “unhealthy”: 0}

Next, we run this script to measure the cold start stats:

import os, time, requests

base = f”https://api.runpod.ai/v2/{os.environ[’ENDPOINT_ID’]}”
headers = {”Authorization”: f”Bearer {os.environ[’RUNPOD_API_KEY’]}”}

payload = {”input”: {
    “openai_route”: “/v1/chat/completions”,
    “openai_input”:{
        “model”: “sql”,
        “messages”: [{”role”: “user”, “content”: “Count shipped orders over 500.”}],
        “max_tokens”: 48,
        “temperature”: 0,
    },
}}

response = requests.post(f”{base}/run”, headers=headers, json=payload, timeout=30)
response.raise_for_status()
job = response.json()

while True:
    response = requests.get(f”{base}/status/{job[’id’]}”, headers=headers, timeout=30)
    response.raise_for_status()
    result = response.json()

    if result[”status”] in {”COMPLETED”, “FAILED”, “CANCELLED”, “TIMED_OUT”}:
        break
    time.sleep(2)

if result[”status”] != “COMPLETED”:
    raise RuntimeError(result)

print(”status:”, result[”status”])
print(”delayTime:”, f”{result[’delayTime’] / 1000:.2f}s”)
print(”executionTime:”, f”{result[’executionTime’] / 1000:.2f}s”)

The terminal will print this:

status: COMPLETED
delayTime: 171.89s
executionTime: 0.56s

• delayTime covers everything before a worker picks up the job, including worker startup and any wait for GPU capacity.

• executionTime covers the request after pickup.

In this run, 99.7 percent of the measured time came before inference. The pickup delay was roughly 307 times the execution time. A model that responds in half a second can therefore take almost three minutes to reach the user after the endpoint scales to zero.

Since this test began with zero workers and one queued request, the startup path dominated unless Runpod marked the worker as throttled.

Repeat the health check and script three times. Let the endpoint return to zero workers before each run, then record both values.

Our three clean trials waited 409, 245, and 214 seconds before pickup. The median cold start was 245 seconds. Execution stayed near half a second.

That spread is uncomfortable, but useful. Most of the uncertainty came from starting the worker, not running the model.

Those numbers are also far longer than a 1.5B model on a 24 GB GPU should need, and the setup explains why. Nothing is cached: each cold start pulls the vLLM worker image, then downloads the base model and all three adapters from Hugging Face before the engine can warm up.

Runpod’s own post on the linked configuration changes cut a much larger model from 324 to 91 seconds without touching the code. Baking the model into the image or caching it on a network volume, and keeping FlashBoot enabled, would remove most of the delay we measured here. We left the default configuration in place so the two layouts stay comparable, but treat 245 seconds as an unoptimized baseline, not a property of the platform.

If Runpod reports a throttled worker, discard that trial. The request was waiting for GPU capacity, so delayTime no longer represents worker startup alone.

Step 11: Compare one endpoint with three

The shared endpoint loads all three adapters. We now create three separate endpoints that load one adapter each.

Return to the local terminal where VLLM_HUB_ID and HF_USER are already set. Define a small shell function so the common settings stay identical:

create_single_adapter_endpoint() {
  adapter=”$1”
  runpodctl serverless create \
    --hub-id “$VLLM_HUB_ID” \
    --name “qwen-lora-$adapter” \
    --gpu-id “NVIDIA GeForce RTX 4090” \
    --workers-min 0 \
    --workers-max 3 \
    --idle-timeout 5 \
    --env MODEL_NAME=Qwen/Qwen2.5-1.5B-Instruct \
    --env ENABLE_LORA=true \
    --env MAX_LORAS=1 \
    --env MAX_LORA_RANK=16 \
    --env “LORA_MODULES=[{\”name\”:\”$adapter\”,\”path\”:\”$HF_USER/qwen15-lora-$adapter\”}]”
}

Run the function once for each adapter:

create_single_adapter_endpoint extract
create_single_adapter_endpoint sql
create_single_adapter_endpoint route

Each command prints the endpoint it created. Copy the three endpoint IDs into environment variables:

export EXTRACT_ENDPOINT_ID=”id-from-extract-create-result”
export SQL_ENDPOINT_ID=”id-from-sql-create-result”
export ROUTE_ENDPOINT_ID=”id-from-route-create-result”

Confirm each endpoint registered its adapter the same way as in step 8:

for id in \
  “$EXTRACT_ENDPOINT_ID” \
  “$SQL_ENDPOINT_ID” \
  “$ROUTE_ENDPOINT_ID”
do

curl -s “https://api.runpod.ai/v2/$id/openai/v1/models” \ 
     -H “Authorization: Bearer $RUNPOD_API_KEY” | python -m json.tool

done

Every endpoint uses the same base model, GPU, worker limits, and five-second idle timeout. Only LORA_MODULES changes. MAX_LORAS is now 1 because each endpoint serves one adapter.

We now have two layouts:

• Shared layout: one endpoint, three adapters.

• Separate layout: three endpoints with 1 adapter each

Send the same five-minute stream to both layouts at one request per second. Rotate through extraction, SQL, and routing. For every job, save its delayTime. Also record the highest worker count shown by Runpod and the billed worker time for the run.

Keep the rate low enough that warm workers can keep up. If requests queue behind other requests, a long delayTime may look like a cold start even when no worker is booting.

This is what happened in our run:

Note that the separate layout used fewer billed worker-seconds, but not because its workers idled out. At one request every three seconds per endpoint, a warm worker never reaches the five-second idle timeout. The separate endpoints spent most of the five minutes cold-starting, so requests queued behind the boot and far fewer completed.

It was cheaper during this short run because it completed fewer requests and made most of them wait more than two minutes.

The shared workers stayed alive because the combined traffic kept reaching the same pool. That is the operational advantage the memory calculation misses.

Step 12: Turn the measurements into a cost estimate

Use your current Runpod GPU rate rather than copying a price from this article. Rates depend on GPU type and worker mode.

For flex workers, estimate monthly cost as:

GPU rate × (request execution hours + startup hours + idle hours)

Request execution should be similar in both layouts because the models generate the same tokens. Startup and idle time are paid once per endpoint. Three separate endpoints can therefore pay that overhead three times during the same traffic cycle.

For active workers, multiply the monthly price of one always-on worker by the number of endpoints. One shared endpoint needs one warm floor. Three separate endpoints need three if every variant must respond immediately.

Use the 245-second cold start from step 10 only as an example. Replace it with your median, then calculate busy and quiet periods separately. Startup consumes a much larger share of the bill when traffic is sporadic.

Wrapping up

A LoRA fine-tune is small until you merge it. Merging creates a complete model copy for every variant, and separate endpoints give each copy its own worker pool.

That trade is reasonable when models need different hardware or complete isolation. It wastes memory and workers when every adapter comes from the same base.

For a family of adapters on one base, my default is to keep them separate. Load one base model, attach the adapters, and let every request use the same worker pool.

The catalogue still consumes GPU memory, and all adapters must match the base, but adding a variant no longer means adding another full model and endpoint.

The experiment also gives us a better way to discuss serverless latency. Record waiting time and execution time separately. A model that runs in half a second still feels slow if the endpoint has to pull an image and download weights every time it wakes, which is why cold-start configuration deserves as much attention as the serving layout.

Runpod made that distinction visible in this test. Pods handled training, Serverless exposed the worker behaviour, and the ready-made vLLM worker let us compare the layouts without maintaining four container images.

Thanks for reading, and thanks to Runpod for partnering with us on this issue.

Production LLM apps do not receive completely new questions every time.

A customer-support assistant might receive all three of these:

• “Can I get a refund after buying the monthly plan?”

• “Is the monthly subscription refundable?”

• “Can I cancel the plan and get my money back?”

The wording is different, but the underlying question and its answer remain the same.

Yet LLM apps process every version as a new request. They assemble the prompt, send it to the model, and generate an answer that may have already been generated.

Prefix caching reduces part of these repeated calls.

When requests begin with the same system prompt or context, the model can reuse the KV states already computed for that shared prefix.

But the request still hits the LLM. The new tokens must be processed, and the complete answer must still be decoded.

So even with a prefix-cache hit, there’s another generation call involved.

To solve this, instead of only caching computation inside the model, the application can cache the generated response outside it.

When another question arrives, the system embeds it and compares it with previously answered questions. If it finds a sufficiently close match, it returns the stored response without invoking the LLM again.

A cache hit removes the input tokens, output tokens, and decoding time associated with another LLM call.

In practice, it is important to decide which questions can safely share an answer since a production setup needs well-tuned similarity thresholds, expiration policies, data isolation, and monitoring for incorrect matches.

If you want to use this, Redis already implements it as a managed service called Redis LangCache.

Try Redis LangCache for free

Under the hood, it generates embeddings, searches previous responses, and returns a matching answer before another model call occurs.

Redis also handles access scopes, custom filtering, TTL and eviction controls, and cache monitoring through Redis Cloud.

We built an interface to compare it against direct LLM inference:

For the paraphrased question in our run, direct inference took 2.232 seconds and consumed 514 input tokens plus 250 output tokens.

Redis returned the earlier response in 0.373 seconds with zero LLM input or output tokens. That was roughly 6x faster in this run.

Redis reports API cost savings of up to 90% and cache-hit responses up to 15x faster. The actual result depends on how much safe repetition exists in the workload.

You can try Redis LangCache here: https://redis.io/langcache/

Thanks to Redis for partnering today!

Why multi-turn agents need more than a task graph

Most agent frameworks represent one run as a graph. A call enters the graph, moves through its nodes, and returns an answer. That model works well for bounded tasks such as parsing a document and writing a summary.

A conversation does not follow the same lifecycle.

Every message starts another graph run, while the session must retain enough context to connect that run to the previous ones.

The same flow object can hold conversation data and execution bookkeeping, even though they need different lifetimes. Conversation history should survive across turns. Completed nodes and cached method outputs should reset.

CrewAI’s conversational flows add a layer around the graph to separate those lifetimes.

Let’s examine the failure first, then build a support agent with session continuity, routing, clean outputs, and session-level tracing.

Let’s begin!

Why repeated kickoffs replay previous output

A flow keeps two kinds of information during a run:

• Application state holds data such as the current message.

• Execution state records which nodes finished and what they returned.

Those records can share a lifecycle for a one-shot task because both become irrelevant when the run ends.

A conversation needs different behavior. Message history and domain data must survive for the next turn, while completed nodes and cached outputs must reset.

If both remain attached to the same flow instance, the second call may find its nodes already marked complete. It can then return the previous output without processing the new message.

A conversational layer separates these. It preserves session data, clears per-run execution records, and places the latest message into the new run.

State persistence across repeated kickoffs

A task run has a defined end. A conversation stays open for another message, so the second turn must begin a new run without losing the session around it.

Persisting the first run seems like the direct fix. It also creates the bug. Consider this small CrewAI flow:

• @start() marks the step that opens the run.

• @listen(x) marks a step that runs after the step x finishes.

• @persist() on the class saves a snapshot of state, so a later run with the same session ID can pick it up.

from crewai.flow import Flow, listen, start, persist

@persist()
class SupportFlow(Flow[ChatState]):

    @start()
    def read_message(self):
        return self.state.message

    @listen(read_message)
    def answer(self, message):
        return f"answering: {message}"

flow = SupportFlow()
session_id = "customer-4471"

first = flow.kickoff(inputs={"id": session_id, "message": "Where is order 4471?"})
second = flow.kickoff(inputs={"id": session_id, "message": "Has that arrived yet?"})

The two calls share a session ID but contain different questions. The expected output is:

answering: Where is order 4471?
answering: Has that arrived yet?

The actual output repeats the first answer:

answering: Where is order 4471?
answering: Where is order 4471?

What survives into the second turn

The answer step never runs during the second call. The same Python instance still holds the set of completed methods and their outputs.

When the graph checks that record, it treats both steps as finished and returns the earlier result.

A second leak sits underneath the first:

• kickoff reads inputs={"id": ...} as a checkpoint restore, so the first run’s snapshot comes back and brings the old message with it.

• When you clear the completed-method record without addressing that, the answer runs, reads state, and finds the previous turn’s message sitting there.

The completed-method record leaks execution progress. The restored snapshot leaks data. Both come from a finished run but remain active during the next one.

Class-level @persist() adds another edge case. It writes a snapshot after every method, so the latest row can represent the middle of a run rather than its final state. A later turn may restore that incomplete snapshot and miss updates made by the handler.

Run state, session state, and memory

1. Run state tracks the nodes and method outputs completed during one execution. It should disappear when that run finishes.

2. Conversation history stores messages exchanged during one session. It must persist across graph runs.

3. Memory stores facts that may outlive a conversation and appear in later sessions. Its retention policy is different from both.

The broken flow keeps run state alive on the instance while restoring conversational data from a snapshot. Without separate lifecycles, each mechanism can carry stale data into the next turn.

Requirements for multi-turn agent execution

A conversational flow must connect graph runs without carrying one run’s execution into the next. That requires four pieces.

1. Session identifies the conversation and restores the history needed by each new run.

2. Router sends the message to the appropriate execution path. A database lookup, research request, and clarification need different amounts of work.

3. Output layer separates internal agent work from the visible conversation. Tool calls and intermediate results remain traceable without filling the history.

4. Session trace connects otherwise independent graph runs into one interaction.

The session gives an otherwise independent run its conversational context.

Session persistence across runs

Every message carries a session ID. Messages from different customers resolve to different histories, even when the same deployment handles both.

Each turn follows the same cycle:

restore the conversation → process the new message → save the updated session

Restoration connects the new run to earlier turns. Without it, a follow-up reaches the model without the context that gives the message meaning.

Three kinds of data belong to the session rather than one graph run:

1. Conversation history, so the next run knows what was already said.

2. Domain data produced by earlier turns, so a finding from turn two remains available at turn six.

3. One-time setup that belongs to the session, such as permissions, cache warming, or loading customer records.

Execution progress is absent from that list. Carrying it forward caused the agent to replay its first answer.

Session storage remains an application decision. In-process storage stops being reliable once multiple workers handle requests because consecutive turns can land on different processes.

Session continuity tells a run which conversation it belongs to. Routing decides what the latest message needs.

Intent routing per turn

One agent can hold every tool, but that design makes the model decide when to search during the run. Weak results can trigger more tool calls and more model round trips.

A router makes the first decision before the expensive path begins and sends the message to a narrower handler.

• An order question goes to a lookup handler with database access.

• A question about a carrier delay goes to a research handler with web access.

• A clarification goes to a handler that reads the history and calls nothing else.

Clarifications and follow-ups often need history but no new research. Sending them through the same research loop adds work without adding evidence.

Routing cost and fallback behavior

The router adds a model call to every turn, so its value depends on the execution paths it avoids. It makes sense when costly routes are uncommon. If every message requires research, the router becomes pure overhead.

An LLM also should not be the only routing mechanism. Deterministic rules are cheaper and easier to test when the intent is explicit. The router still needs defaults for failed calls and invalid route labels.

Traditional systems made a similar decision with intent classifiers. An LLM makes the route catalog easier to extend, but the cost and failure modes remain.

Separating agent outputs from message history

A research run produces more than its final answer. It may contain queries, tool results, failed branches, and intermediate findings. The next conversational turn rarely needs all of that material.

Writing the full execution record into message history forces later turns to process it again. The run therefore needs two destinations:

1. Intermediate work stays in the execution record and trace.

2. The final answer enters the conversation history read by the next turn.

The model receives the user-visible exchange, while engineers can still inspect the full execution when a run fails.

Excluding scratch work slows history growth but does not stop it. Long sessions still need a policy, such as a recent-turn window, a token budget, or a rolling summary that compresses older turns while preserving recent messages.

That summary belongs to conversation history. Durable facts extracted for later sessions belong to memory. Once the records are separate, tracing must reconnect them for debugging.

Session-level tracing

A per-run trace can report success even when the conversation drifts. One turn may answer a slightly different question, and later turns can build on that answer without raising an exception.

A session-level trace keeps routing decisions, handlers, tool calls, and replies in order across turns. It shows whether the interaction progressed correctly, not merely whether each graph run completed.

A deferred trace must eventually close. Otherwise, the batch may never export the conversation it contains. The application therefore needs a finalization path for explicit exits, timeouts, dropped connections, and abandoned sessions.

The implementation now has a clear contract: create a fresh run for every message, preserve the session around those runs, and close the trace when the session ends.

Implementing the conversational runtime

CrewAI implements that contract with conversational flows. Instead of calling the same flow directly for every message, handle_turn() wraps each kickoff with session restoration, per-turn execution reset, message handling, routing, and deferred tracing.

This API is experimental in CrewAI v1.15.6 and lives under crewai.experimental, so pin that version if you use the code below.

The example is an e-commerce support assistant with one session and three routes that perform different amounts of work.

Session state and turn continuity

A conversational flow receives a message and session ID. The ID selects the conversation, while the state holds the history and domain fields needed by later turns.

from uuid import uuid4
from crewai.experimental.conversational import ConversationState

class SupportState(ConversationState):
    last_order_id: str | None = None
    ...

session_id = str(uuid4())
flow = SupportFlow(

# The second message has no order number in it, so it resolves
# against what the first turn left in state.
flow.handle_turn("Where is order 4471?", session_id=session_id)
flow.handle_turn("Has that arrived yet?", session_id=session_id)

ConversationState already contains message history and the last routing decision. The subclass adds domain fields required by the handlers.

The second message contains no order number. Its handler falls back to last_order_id, which the first turn stored in the session.

Route handlers

The router reads the current message and selects a handler. Each capability is an ordinary flow listener with a narrow tool set and execution path.

from crewai.experimental.conversational import RouterConfig
from crewai.flow import listen

router_config = RouterConfig(
    prompt=(
        "You route messages for an e-commerce support assistant. "
        "Pick the route that matches what the customer is asking for right now."
    ),
    llm=ROUTER_LLM,
    default_intent="converse",   # router call failed, or no LLM configured
    fallback_intent="converse",  # router returned a label that does not exist
)

# Each label becomes a route. Each docstring line becomes its description,
# which is the text the router LLM reads when deciding where a message goes.
@listen("ORDER_LOOKUP")
def handle_order_lookup(self) -> str:
    """Status, tracking, or delivery date for an order."""
    ...

CrewAI includes three built-in routes:

1. converse for ordinary chat and follow-ups,

2. end for goodbyes,

3. an optional answer_from_history, and you can override any of them by defining a handler with the same name.

The configuration prompt supplies domain context and business rules. CrewAI builds the route catalog from handler docstrings, which keeps route definitions tied to executable listeners.

Output visibility

The research route runs a two-agent crew:

1. The researcher queries live sources and records its findings.

2. The summarizer converts those findings into a customer-facing answer.

from crewai import Crew, Process

# Sequential: the researcher runs first, the summarizer reads its output.
result = Crew(
    agents=[researcher, summarizer],
    tasks=[research_task, summarize_task],
    process=Process.sequential,
).kickoff()

# The researcher's raw findings: every search, dead end, and retry.
self.append_agent_result(
    "researcher", result.tasks_output[0].raw, visibility="private"
)

# The summarizer's two or three sentences.
# This is the only part the next turn reads.
self.append_assistant_message(result.raw)

Both outputs come from the same crew run but serve different consumers.

• append_agent_result(..., visibility="private") writes a structured event to state.events and a thread under state.agent_threads.

• append_assistant_message() puts the concise reply in the canonical conversation history.

Deferred trace finalization

Conversational flows can keep one trace batch open across turns. The resulting trace contains each routing decision, handler, tool call, and reply in session order.

from crewai import Flow
from crewai.experimental.conversational import ConversationConfig

@ConversationConfig(
    llm=CONVERSATION_LLM,
    router=router_config,
    # Hold one trace batch open across turns instead of closing per kickoff.
    defer_trace_finalization=True,
)
class SupportFlow(Flow[SupportState]):
    conversational = True
    ...

flow = SupportFlow()

try:
    for message in conversation:
        # Nested crews append to the open batch rather than closing it
        # So their tool calls stay inside the session.
        reply = flow.handle_turn(message, session_id=session_id)
finally:
    flow.finalize_session_traces()

defer_trace_finalization defaults to True, so the application must close the batch when the session ends. The finally block guarantees that explicit loop exits still export the trace.

Conversational flows do not turn a chat into one endless graph run. They preserve the session while resetting and executing the graph once per message.

End-to-end support flow

The complete turn lifecycle now looks like this:

1. The turn opens with a message and session ID.

2. Per-turn execution tracking resets, so the graph runs instead of replaying cached outputs.

3. The session restores conversation history and persisted domain fields.

4. The router reads the new message and selects a route.

5. The handler records private work and produces a visible reply.

6. The reply enters conversation history while the session trace remains open.

Model and service dependencies

The order service wraps an SQLite database behind an order-management API. Return policies live in documents queried through retrieval.

The flow uses two models for two different workloads. A small routing model classifies short messages, while the conversation model handles the agent work.

from crewai import LLM

ROUTER_LLM = LLM(model="gpt-4o-mini")
CONVERSATION_LLM = LLM(model="gpt-4o")

This keeps classification on the cheaper model and reserves the larger model for routes that need it.

from crewai import Flow
from crewai.experimental.conversational import ConversationConfig
from crewai.flow import listen

# Models, router config, services, and crews are set up above.
@ConversationConfig(llm=CONVERSATION_LLM, router=router_config)
class SupportFlow(Flow[SupportState]):
    conversational = True

    @listen("ORDER_LOOKUP")
    def handle_order_lookup(self) -> str:
        """Order status, tracking, or delivery date."""
        msg = self.state.current_user_message
        order_id = extract_order_id(msg) or self.state.last_order_id
        self.state.last_order_id = order_id
        self.append_assistant_message(reply)
        return reply

    @listen("RETURN_POLICY")
    def handle_return_policy(self) -> str:
        """Returns, refunds, and damaged items."""
        reply = return_agent.kickoff(self.state.current_user_message).raw
        self.append_assistant_message(reply)
        return reply

    @listen("RESEARCH")
    def handle_research(self) -> str:
        """Live carrier delays and outside conditions."""
        question = self.state.current_user_message
        result = research_crew.kickoff(inputs={"question": question})
        findings = result.tasks_output[0].raw
        self.append_agent_result("researcher", findings, visibility="private")
        self.append_assistant_message(result.raw)
        return reply

(Full code is provided below)

Exposing the flow through an API

The earlier loop runs in a terminal. The AG-UI endpoint exposes the same flow to a chat frontend by enabling conversational mode.

from ag_ui_crewai.endpoint import add_crewai_flow_fastapi_endpoint
from fastapi import FastAPI
from support_flow import SupportFlow

app = FastAPI(title="Support Flow Agent Server")

flow = SupportFlow()
add_crewai_flow_fastapi_endpoint(
    app=app,
    flow=flow,
    path="/conversation",
    conversational=True,
)

Running a multi-turn conversation

Each message below exercises a different part of the flow. Every handle_turn() call creates a separate graph run under the same session ID.

conversation = [
    "Where is order 4471?",                      # plain lookup, no agent
    "Has that arrived yet?",                     # no order number in the text
    "Can I return it if it shows up damaged?",   # needs the policy for that category
    "Is Bluedart running late this week?",       # needs live external information
    "Sorry, what was the order number again?",   # already answered on turn one
]

session_id = str(uuid4())
flow = SupportFlow()

try:
    for message in conversation:
        # A chat UI makes these calls across HTTP requests instead of loop
        reply = flow.handle_turn(message, session_id=session_id)
        print(f"customer: {message}")
        print(f"  route: {flow.state.last_intent}")
        print(f"  agent: {reply}\n")
finally:
    flow.finalize_session_traces()

Production properties

Together, these pieces keep each graph run independent without breaking the conversation around it:

• Multi-turn continuity. Conversation data survives across runs, while the graph’s completed-method record resets before each turn.

• Controlled execution. A session containing fifteen clarifications and two research questions can run the research path twice rather than seventeen times.

• Clean context. Later turns receive the visible conversation instead of the research log behind it. History still grows, but intermediate tool activity no longer accelerates that growth.

• Session-level debugging. One trace follows routing, execution, and replies across turns, matching the interaction the customer experienced.

An ordinary flow decides what one run should execute. A conversational flow also identifies the session, restores the right history, resets execution progress, and preserves the result for the next turn.

CrewAI handles that lifecycle. The application still decides where sessions live, what each handler does, how long history can grow, and when an inactive session should close.

• Read more about CrewAI conversational flows →

• Explore CrewAI on GitHub →

• Full demo code in this GitHub repository →

Good day!

The modern agent stack has tools, memory, context, and orchestration. What it still lacks is a runtime system of record.

Recent incidents across OpenAI, Anthropic, and xAI point to the same gap.

In July 2026, an autonomous agent running in an OpenAI cybersecurity evaluation carried out a 4.5-day intrusion against Hugging Face’s production infrastructure. Reconstructing the attack required a second LLM pipeline because the payloads had been packed specifically to evade traditional log scanners.

Anthropic reached a similar conclusion after Claude models accessed real production systems during cybersecurity evaluations: real-time monitoring of agent transcripts and behavior could have changed the outcome.

And Adversa AI found xAI’s Grok Build exfiltrating user names, locations, and full chat histories through encoded payloads that safety scanners could not interpret, but the model runtime could still execute.

Across three of the world’s leading AI labs, the pattern is the same: security systems can inspect what goes into an agent and collect logs after the fact, but they have limited visibility into what the agent actually does while it is running.

That missing runtime layer is what Agent Beacon (GitHub repo) is built to capture.

Agent Beacon GitHub Repo

Beacon is a 100% open-source telemetry layer for AI agents. It records runtime activity across the agent harnesses engineering teams already use and normalizes it into a common schema, creating a system of record for agent behavior.

Beacon turns agent activity into a live, structured runtime record. Every tool call, shell command, file change, and approval decision becomes a structured event as it happens.

That changes the security model. Instead of piecing together an agent session after something goes wrong, you have a live record of what the agent is doing while it is still running.

You can answer questions like:

• Which agent executed this command?

• What prompt and session caused it?

• Was the action approved?

• What files did it read or modify?

• What happened immediately before and after?

• What else did the agent do in the same session?

And because Beacon normalizes this activity across harnesses, you can ask those questions the same way everywhere.

One runtime record. Across every agent.

Let’s install Beacon and see it in action.

Setup

Beacon runs locally as an endpoint agent on your machine. Install it with Homebrew:

brew tap asymptote-labs/tap
brew install beacon

Then start the endpoint:

beacon endpoint install

Beacon registers itself as a background service and configures supported runtimes to send telemetry to the local collector at 127.0.0.1:4317. By default, runtime events are written locally to ~/.beacon/runtime.jsonl.

As your agent works, its activity is recorded there in real time. To confirm the endpoint is running:

beacon endpoint status

That’s it. Once the endpoint is live, Beacon starts capturing runtime activity from supported agent harnesses automatically.

Capturing agent runtime activity

Beacon turns fragmented agent activity into one security-ready event stream.

It integrates with 23+ agent harnesses using native hooks, plugins, and OpenTelemetry, without requiring changes to your agent code.

The problem is that every harness represents agent activity differently. The same underlying action might appear as a tool invocation in one system, a shell execution in another, or be buried inside an OpenTelemetry event.

Beacon normalizes those signals into a single schema. That means your investigations and detection rules can operate on the same fields everywhere, instead of maintaining separate logic for every agent runtime.

For example, a file modification might look like this:

{
  "timestamp": "2026-09-02T07:38:15.469000000Z",
  "event": {
    "action": "file.modified",
    "fidelity": "observed"
  },
  "harness": {
    "name": "claude_code",
    "collection_method": "otlp"
  },
  "tool": {
    "name": "Edit",
    "path": "buggy_processor.py"
  },
  "session": {
    "id": "678c9516-5eaf-4c6a-abfb-5313aa57153d"
  }
}

Two fields are important here:

1. event.action tells you what actually happened. Beacon maps harness-specific telemetry into normalized actions such as tool.invoked, command.executed, and file.modified. So your security logic doesn’t need to understand how each individual harness represents an action. It can simply ask: did a command execute? was a file modified? was a tool invoked?

2. event.fidelity tells you how Beacon knows. An observed event means the runtime reported the action directly through a structured integration. An inferred event means Beacon derived it from less-direct evidence, such as a log pattern. That distinction matters once telemetry becomes detection logic.

A rule can require not just that an approval appears to have happened, but that the runtime itself reported it:

event.action == "approval.allowed"
event.fidelity == "observed"

This is what turns Beacon from another source of agent logs into a runtime security layer: one normalized event model, with provenance, that you can investigate and write detections against across agent harnesses.

A live Claude Code session

To see this in practice, we gave Claude Code a simple debugging task: fix a Python script that crashed when it encountered an empty list.

Claude solved the problem in three actions. Beacon recorded all three with timestamps and a shared session ID.

Nothing about this session was suspicious. That’s what makes it useful.

Runtime telemetry only becomes valuable if it is captured continuously, not activated after an incident begins. Whether an agent is fixing a trivial bug or touching production infrastructure, the execution trail should already exist.

That mattered in the Hugging Face intrusion, where thousands of individual actions had to be reconstructed after the fact. Many of those actions were unremarkable in isolation. The value was in seeing how they connected across the full session.

Inspecting sessions in real time

Beacon also includes a local dashboard for inspecting agent activity as it happens.

beacon dashboard

Then open:

http://localhost:4321

The dashboard streams runtime events in real time and lets you trace activity by timestamp and session. Everything stays on your machine. There is no Beacon backend and no account required.

Once that execution trail exists, you can do more than inspect it. You can detect against it.

Running detection rules

Beacon includes a local scanner that evaluates detection rules against the same normalized telemetry.

beacon scan

The scan runs locally with no network access required.

Rules are written in YAML and match against Beacon’s normalized event schema. For example, this rule detects an agent executing a command that references the DEBUG environment variable:

id: debug-env-access
version: 1
title: Agent read DEBUG environment variable
severity: medium
status: experimental
posture: detect
match: >
  e.event.action == "command.executed" &&
  e.command.command.contains("DEBUG")
emit:
  reason: "Agent executed a command reading the DEBUG environment variable"

We then asked Claude Code to read the DEBUG environment variable.

The rule fired when the command executed.

That is the important shift: detections can operate on the agent’s runtime behavior itself, rather than trying to infer what happened later from generated text or incomplete logs.

During the Hugging Face intrusion, for example, one of the observed commands was: 𝗰𝗮𝘁 /𝗽𝗿𝗼𝗰/𝘀𝗲𝗹𝗳/𝗲𝗻𝘃𝗶𝗿𝗼𝗻

With runtime telemetry in place, behavior like that can become a detection signal at execution time, while the session is still unfolding.

You can write your own rules against the same schema and add them with:

beacon rules add

But Beacon does not require security teams to replace the infrastructure they already use. The same runtime events can be sent downstream.

Sending events to your existing security stack

By default, Beacon writes events locally to ~/.beacon/runtime.jsonl.

Teams can also forward those normalized events into their existing security and observability systems.

For example, Splunk:

beacon endpoint install \
  --splunk-hec-url https://splunk.company.com:8088 \
  --splunk-hec-token your-token

Beacon also supports integrations and configuration packs for systems including Datadog, Elastic, Microsoft Sentinel, CrowdStrike, and others. Events can also be routed to cloud storage through Vector.

The destination is less important than the abstraction underneath it.

A shell command executed through one agent harness and the same command executed through another can be represented using the same event schema before it reaches your SIEM, data lake, or detection pipeline.

That gives security teams a common data layer for agent activity instead of requiring a new collection and detection model for every harness that enters the enterprise.

Why the runtime record matters

This brings us back to the problem we started with.

The Hugging Face forensic team ultimately reconstructed roughly 17,600 agent actions using another LLM pipeline. The attacker had deliberately packed its payloads in ways that made downstream inspection difficult.

By the time investigators had the full picture, they were reconstructing execution from artifacts left behind.

A runtime system of record changes that.

With Beacon running, a shell command, file modification, tool invocation, or approval decision can become a structured event at the moment it happens.

That event already has the context needed to investigate it: when it happened, which session it belonged to, which harness produced it, and what action occurred.

The execution trail exists before the incident response begins.

That is the missing layer in today’s agent stack: a consistent runtime record of what agents actually did.

Agent Beacon is 100% open source, MIT licensed, and local by default. Nothing leaves your machine unless you explicitly configure a destination.

Check out Agent Beacon on GitHub →

(don’t forget to star 🌟)

Good day!

• ​Agent hackers to test your AI apps!

• Momentum in ML, explained visually and intuitively!

• Build a Reasoning LLM using GRPO.

Agent hackers to test your AI apps!​

Pentesting firms don’t want you to see this.

An open-source AI agent​ just replicated their $50k service.

Here’s why this matters right now.

Teams are shipping faster than ever. AI writes the code, CI catches build failures, tests catch regressions, and observability catches outages.

But one more key question to ask is: What can an attacker do with this, right now?

It’s important to answer this question because several real-world examples make this hard to ignore:

• Moltbook exposed 1.5M auth tokens. The owner hadn’t written a single line of code.

• Tea App leaked 72,000 government IDs. The database was just open, no sophisticated hack needed.

• A researcher took control of a journalist’s computer through her own vibe-coded game, without a single click.

The code ran fine in all three cases, tests passed, and nothing raised a flag.

Because the bottleneck is no longer writing code, it’s understanding what that code actually exposes once it’s live. PR reviews miss auth edge cases, unit tests don’t probe broken access control, staging environments don’t simulate adversarial behavior, and business logic flaws look completely fine until someone decides to break them on purpose.

An automated approach is actually implemented in ​Strix, a recently trending open-source framework (61k+ stars) for an AI pentesting agent.

It reviews any running app the way an attacker would:

• Crawls the app and maps every exposed route and flow

• Probes abuse paths dynamically, not just at build time

• Returns findings with proofs-of-concept and suggested fixes

It is benchmarked against 200 real companies and open-source repos, and it found 600+ verified vulnerabilities, including assigned CVEs.

It’s designed to fit into how modern teams already work: run it before a release, after major changes, or continuously as the app evolves.

Strix GitHub repo

​You can find the GitHub repo here →​ (don’t forget to star it)

Momentum in ML, explained visually and intuitively!

As we progress towards building larger and larger models, every bit of possible optimization becomes crucial.

And there are various ways to speed up model training, like:

• Leverage distributed training with PySpark MLlib (covered here)

• Use better Hyperparameter Optimization, like Bayesian Optimization, which we discussed here →

• Or use other optimization techniques listed below (covered here with code)

Momentum is another reliable and effective technique.

Here’s an intuitive guide that explains its effectiveness.

Issues with Gradient Descent

In gradient descent, every parameter update solely depends on the current gradient.

This results in many unwanted oscillations during the optimization process.

To understand better, imagine this is the loss function contour plot with the optimal location marked:

The parameter update trajectory is depicted below:

Notice two things here:

1. It unnecessarily oscillates vertically.

2. It ends up at the non-optimal solution after some epochs.

Ideally, the update process must have taken longer steps in the horizontal direction and smaller vertical steps because a movement in this direction is unnecessary:

Solution: Momentum

Momentum modifies the update rule of gradient descent by also considering a moving average of past gradients:

This handles the unnecessary vertical oscillations observed above.

More specifically, by using a moving average of past gradients, the oscillations in the vertical direction cancel out, and those in the horizontal direction push the parameters to the optimal point faster:

This smoothens the optimization trajectory and reduces unnecessary oscillations in parameter updates, as depicted below:

This is how Momentum works.

Of course, Momentum introduces another hyperparameter (Momentum rate) in the model, which should be tuned like any other hyperparameter:

• Setting a large Momentum rate will significantly expedite the gradient update in the horizontal direction, leading to overshooting the minima:

• Setting a small Momentum rate will slow down the optimal gradient update, defeating the whole purpose of Momentum.

👉 Over to you: What are some other reliable ways to speed up machine learning model training?

We covered 15 different techniques here: 15 Ways to Optimize Neural Network Training (With Implementation).

If you want to have a more hands-on experience with Momentum, check out this tool: Momentum Tool.

​Build a Reasoning LLM using GRPO​

Group Relative Policy Optimization is a reinforcement learning method that fine-tunes LLMs for math and reasoning tasks using deterministic reward functions, eliminating the need for labeled data.

Here’s a brief overview of GRPO:

• Start with a dataset and add a reasoning-focused system prompt (e.g., “Think step by step…”).

• The LLM generates multiple candidate responses using a sampling engine.

• Each response is assigned rewards, which are aggregated to produce a score for every generated response.

• A GRPO loss function uses these rewards to calculate gradients, backpropagation updates the LLM, and the model improves its reasoning ability over time.

Let’s dive into the code to see how we can use GRPO to turn any model into a reasoning powerhouse without any labeled data or human intervention.

We’ll use:

• UnslothAI for efficient fine-tuning.

• HuggingFace TRL to apply GRPO.

The code is available here: ​Build a reasoning LLM from scratch using GRPO​. You can run it without any installations by reproducing our environment below:

Let’s begin!

Load the model

We start by loading Qwen3-4B-Base and its tokenizer using Unsloth.

You can use any other open-weight LLM here.

Define LoRA config

We’ll use LoRA to avoid fine-tuning the entire model weights. In this code, we use Unsloth’s PEFT by specifying:

• The model

• LoRA low-rank (r)

• Modules for fine-tuning, etc.

Create the dataset

We load the Open R1 Math dataset (a math problem dataset) and format it for reasoning.

Each sample includes:

• A system prompt enforcing structured reasoning

• A question from the dataset

• The answer in the required format

Define reward functions

In GRPO, we use deterministic functions to validate the response and assign a reward. No manual labelling required!

The reward functions:

• Match format exactly

• Match format approximately

• Check the answer

• Check numbers

Use GRPO and start training

Now that we have the dataset and reward functions ready, it’s time to apply GRPO.

HuggingFace TRL provides everything we described in the GRPO diagram, out of the box, in the form of the GRPOConfig and GRPOTrainer.

Comparison

We can see how GRPO turned a base model into a reasoning powerhouse:

Before we conclude, let’s address an important question:

When should you use reinforcement fine-tuning (RFT) versus supervised fine-tuning (SFT)?

We created this diagram to provide an answer:

Finally, we’ll leave you with an overview of the GRPO process.

Let us know what other techniques you have used for fine-tuning LLMs.

The code is available here: ​Build a reasoning LLM from scratch using GRPO​. You can run it without any installations by reproducing our environment below:

Good day!

When an agent run fails, you usually can’t reproduce it. The model can sample differently, the tool can return something else, and more. All you are left with is whatever you recorded while it was running.

Dashboards don’t help much here, because you built them around the failures you already knew about. Once you’re running LLM calls, retries, and tool chains inside a single request, the number of ways it can go wrong is far larger than the number of charts you can maintain.

What works instead is recording enough context on each request that you can slice it afterwards on a field you never planned for, like which model version handled it, or which customer, or which tools it called.

Honeycomb is running a six-session live masterclass on exactly this, taught by Liz Fong-Jones, who co-wrote the O’Reilly book Observability Engineering. It’s free, and three sessions are still ahead:

• Sep 16 → Decide what counts as too slow, and what you do when you hit it

• Sep 30 → Cut the bill without losing the data you need at 3am

• Oct 14 → Do the same for mobile apps, data pipelines, and ML systems

Each session covers one chapter and turns it into something you can use that same week in production. The first two sessions covered instrumenting with OpenTelemetry and catching a bad deploy before users find it. You can still register to get the recordings and start now without being behind.

Live sessions include Q&A with Liz. Recordings and hands-on labs will also be made available after each one.

Register here for free now so that you don’t miss out →

Learn AI observability for free

Thanks to Honeycomb for partnering today!

LLM routing can cost more than not routing

A single coding agent session can analyze a codebase, write new functions, fix bugs from test output, explain methods, and search documentation.

Those tasks are not equally difficult. But when the agent uses one hardcoded model, they all bill at the same rate.

A docstring lookup and a distributed systems refactor should not cost the same.

Routing is the obvious answer where you send easy work to a cheaper model and reserve the expensive one for requests that need it.

That works for isolated requests. Inside an agent loop, though, a naive router can cost more than using one model throughout the session.

Routing works in theory but not as much in practice

The basic rule is that you send each request to the cheapest model that can handle it well.

Classification, extraction, formatting, and short summaries often produce comparable results on models that cost 10 times less. The frontier model stays available for harder work.

If 70% of requests go to a model priced at $0.10 per million tokens and the remaining 30% go to one priced at $3, the blended rate is $0.97 per million instead of $3.

A 2026 arXiv survey on dynamic routing found that a well-designed router can even outperform the best individual model by sending tasks to models with different strengths.

So while routing works, teams still hardcode one model because they do not want to build and maintain the routing layer around it.

Four problems with DIY routing

These are four typical problems when building a production router:

1. You pay for two inference calls. There’s a small LLM in front of every request that acts as a classifier, so you first pay for classification and then for the actual response. The savings survive only when the classifier costs far less than the gap between model tiers and routes accurately enough to avoid expensive mistakes.

2. General models are mediocre routers. A request such as “fix this” or “make it faster” contains almost no useful signal by itself. The intent lives in the conversation history. General-purpose models were not trained to compare that history against a fixed set of route definitions, and coding requests are particularly easy to misread.

3. The routing logic is never really finished. It may work well when you first ship it, but it was written around the models, task definitions, and provider prices available at that point. Those assumptions keep changing while the routing code stays the same. Eventually, easy requests start reaching expensive models, or difficult ones get sent to models that cannot handle them.

4. Model switching invalidates cache reuse. This failure is specific to multi-turn workloads. Providers cache the attention state for repeated prompt prefixes. If the same token sequence reaches the same model, the provider can reuse that state and bill those input tokens at a lower rate. But if you switch models mid-session, the new model must process the conversation again at full price.

Routing at the infrastructure level

These problems have the same cause. When routing lives in your application, you own the classifier, decision logic, maintenance, and cache behavior.

DigitalOcean’s Inference Router moves that work into the infrastructure. You describe the tasks and choose which models may handle each one. The platform makes the routing decision for every request.

The router uses Plano, an open-source AI proxy. Each decision has two phases.

Phase 1: Resolving intent

A small language model reads the conversation, compares it with your task descriptions, and returns a routing decision.

That sounds like the double-inference problem from earlier. The difference is the classifier.

Katanemo’s first routing model, Arch-Router, had 1.5 billion parameters and was fine-tuned for one job, i.e., read a conversation, compare it with route descriptions, and emit JSON.

On that task, the 1.5B model beat Claude 3.7 Sonnet on accuracy and ran 28 times faster.

This is the difference between prompting a general model to classify and training a small model for classification. The router does not need prose generation, tool use, or broad multi-step reasoning. Its job is narrow enough for a much smaller model.

The classifier runs inside the proxy, so it does not appear as a second API call on your bill. The tradeoff is roughly 200 milliseconds of added latency.

The production model today is Plano-Orchestrator. It was trained on harder conversational cases, including ambiguous follow-ups, topic changes during a conversation, and messages that should not be routed.

It scores slightly above GPT-5.1 and Claude Sonnet 4.5 on overall routing accuracy. Its largest margin is on coding requests.

A model trained on that pattern can resolve the intent better than a general model prompted to classify it.

Phase 2: Ranking the pool

Once the router knows the task, it narrows the choice to a pool of up to three models. It then has to pick one.

You could define a fixed order and always choose the first available model. The problem is that provider latency can vary by 2 to 3 times over the course of a day.

The fastest model at 2 a.m. may be the slowest at 2 p.m.

With this, prices change, latency drifts and rate limits affect availability.

The ranking engine reads cost data from DigitalOcean’s pricing API and latency data from Prometheus. It then sorts the candidate models according to the policy you chose:

• Cost Efficiency sorts by token price.

• Speed Optimization sorts by time to first token.

• Manual Ranking keeps the order you specified.

• Optimal uses DigitalOcean’s benchmarked ordering.

A background process refreshes the metrics and stores them in memory, so the router does little work at request time.

Each router also has a fallback list. If the selected model is down, rate-limited, or unavailable, the router tries the next candidate under the same policy before moving to the configured fallbacks.

Model affinity

An agent resends its system prompt, tool definitions, files, and previous steps on every turn.

Prefix caching makes this repeated input much cheaper, but the cache belongs to the model that processed it.

If turn 3 goes to Model A and turn 4 goes to Model B, Model B must process the entire conversation again at the full input rate. This is why the router should choose a model once and keep the rest of the session on it.

Three problems follow:

• In a 15-turn loop where 90% of the input is a repeated prefix, model affinity can reduce input cost by 45% to 80%. Switching models removes those cache hits.

• Models differ in output style and tool-call formatting. A mid-session switch can break the agent’s parser.

• The next model may interpret the existing reasoning and instructions differently.

Session pinning solves this. Essentially, the router chooses a model for the first request, then sends every later request with the same session ID to that model.

DigitalOcean exposes this through the X-Model-Affinity header:

curl --location 'https://inference.do-ai.run/v1/chat/completions' \
  -H "Authorization: Bearer $MODEL_ACCESS_KEY" \
  -H "X-Model-Affinity: session-001" \
  -d '{
    "model": "router:test-router",
    "messages": [
      {"role": "user", "content": "Write a Python function for binary search"}
    ]
  }'

A second call with the same affinity ID skips routing, uses the same model, and returns "pinned": true.

You pay for one routing decision, then keep the cache savings for the rest of the session.

Without session pinning, routing an agent loop can cost more than keeping the entire session on one model.

Building an Inference Router

Setting up a router takes about five minutes.

DigitalOcean provides preset routers for Software Engineering, General, Writing, and Knowledge Base & Document Intelligence workloads.

Start with a name and a description.

The name is how you reference the router later.

The description becomes part of the routing prompt. The model reads it when deciding where a request belongs, so vague wording leads to vague matches.

Next, add your tasks.

A router contains tasks and a fallback list. Each task pairs a description with the pool of models allowed to handle it.

You can use DigitalOcean’s preset tasks or write your own.

The Coding & Development preset includes Bug Fixing, Code Generation, Performance Optimization, and System Architecture & Design. Other presets cover work such as summarization, extraction, translation, classification, long-document Q&A, and RAG evaluation.

Alternatively, you can also define a custom task.

For a custom task, you provide the name, routing description, model pool, and prioritization policy. The available policies are Cost Efficiency, Speed Optimization, and Manual Ranking.

The routing model matches requests against these descriptions, so specificity matters. Concrete task names and nouns work better than broad descriptions.

Next, pick your models.

The model picker displays the current price per token. DeepSeek V4 Flash costs $0.08 per million input tokens, while Claude Opus 5 costs $5.

Finally, add fallback models and create the router.

Fallbacks handle requests that do not match a task. The router tries them in the order you set.

Test the router

The Playground runs your router and a single model side by side. You can compare them with your own prompts instead of relying only on benchmarks.

We tested both with a difficult request: design a transactional outbox pattern for PostgreSQL and Kafka, including the schema, polling publisher logic, and idempotency handling.

Claude Opus 5 ran directly on the left. My custom coding-router ran on the right.

The router matched the request to System Architecture & Design and sent it to GLM-5.2. Both responses were correct and complete.

The routed response was 94% cheaper and 77% faster to first byte.

Once real traffic starts flowing, two other tools become useful.

The Analyze tab reports model match rate and fallback rate. If the fallback rate is high, your task descriptions probably need to be more specific.

Router Evaluation tests the configuration against an uploaded dataset. It uses an LLM judge to score completeness and correctness, giving you a way to catch bad routing changes before they reach production.

Next steps

Token prices keep falling, but agents are consuming tokens even faster. Gartner estimates that agentic workflows use 5 to 30 times more tokens per task than a standard chat interaction.

Spending caps control that cost by limiting how much people can use the tools. Routing attacks the waste instead. Easy requests go to cheaper models, while difficult work still reaches the models capable of handling it.

DigitalOcean has taken on the parts that usually make teams abandon an in-house router:

• A small model trained specifically for intent resolution

• Rankings that update with current prices and latency

• Session pinning that keeps an agent on one model long enough to reuse its cache

The question is no longer whether routing works but rather how much of your traffic is still going to a model that costs more than the request requires.

Try DigitalOcean Inference Router here →

Good day, and thanks to DigitalOcean for partnering today!

Model weights stay roughly fixed during inference. The KV cache does not. It grows with every token in the sequence, and by default, every active sequence needs its own cache.

For Llama 3.1 70B, one 128K-token sequence requires about 40 GB of BF16 KV cache. Four full-length sequences would add another 160 GB, before accounting for model weights and serving overhead.

That creates two separate costs. The cache occupies GPU memory, and the attention layers must repeatedly read it while generating new tokens.

The techniques described as KV cache optimizations address different parts of this problem. Some change what the model stores. Some retain fewer tokens or use fewer bits. Others leave the logical cache intact and improve how the serving engine reads, allocates, shares, or moves it.

This article organizes twelve techniques by what they reduce, when they can be applied, and what trade-off remains. Each section includes a small runnable example that explains the main idea.

To dive deeper into the full LLMOps lifecycle, we have covered every bit of this in the LLMOps course, starting from fundamentals to production.

Start here →

The cache formula

Each attention layer stores one key and one value for every retained token.

The raw cache size for one sequence is:

The leading 2 counts keys and values. BF16 uses two bytes per value, while FP8 uses one.

That formula also gives us a clean way to group the techniques:

• GQA and MQA reduce the number of KV heads.

• Cross-layer attention reduces the number of unique cached layers.

• Sliding windows and eviction reduce the number of retained tokens.

• MLA reduces the width of the stored representation.

• Quantization reduces the bytes used by each value.

• Hybrid recurrent layers replace a growing cache with fixed-size state.

• Paging and prefix reuse reduce allocation waste and duplicate blocks.

Sparse attention needs a separate line. It may read fewer tokens without removing them from memory, so it can cut attention work while leaving cache capacity unchanged.

This distinction matters when the GPU is full. A faster attention kernel doesn’t create room for another sequence unless it also stores fewer bytes.

Technique map

The visual below puts all 12 techniques in one place, showing what each one changes and whether its main saving comes from fewer heads, layers, tokens, dimensions, bits, or duplicated memory.

The table above represents what each technique tries to tackle.

The rest of the article works through the full table in order.

1. GQA and MQA cache fewer heads

The cache formula above contains a term for KV heads. Reducing that term is one of the largest architectural savings available.

Inside each transformer layer, every token produces three representations:

• The query represents what the current token needs.

• The key describes what each earlier token can match.

• The value carries the information returned when that match receives attention.

During generation, the current token’s query is compared with cached keys.

Those comparisons produce attention weights, which combine the cached values. The layer keeps keys and values for later decoding steps. It does not keep the queries.

Attention splits these representations into smaller parts called heads.

A query head is one slice of the token’s query representation. It is not the user’s prompt or a separate model request. Different heads can learn to focus on different relationships in the same sequence.

• Standard multi-head attention, or MHA, gives each query head a matching key head and value head. A layer with eight query heads therefore produces eight key heads and eight value heads. Every attention layer performs this projection independently and keeps its own KV cache.

• Multi-query attention, or MQA, keeps all eight query heads but produces only one key head and one value head. Every query head reads from that shared KV pair. The sharing does not cross transformer layers.

• Grouped-query attention, or GQA, sits between these two layouts. It divides the query heads into groups and assigns one KV pair to each group. With eight query heads and two KV heads, four query heads share each KV pair.

All three methods operate inside each attention layer. They differ only in how many KV heads that layer stores. Cross-layer attention, covered next, is a separate idea that shares cached information between layers.

The GQA paper described it as an intermediate point between MHA and MQA.

The authors converted existing MHA checkpoints and used 5 percent of the original pretraining compute for recovery training. Their uptrained GQA models stayed close to MHA quality while approaching MQA speed.

The following example uses eight query heads. It shows which KV head each query head reads. It also counts the cached scalars for six tokens:

query_heads = 8
tokens = 6
head_size = 8

for name, kv_heads in [("MHA", 8), ("GQA", 2), ("MQA", 1)]:
    queries_per_kv_head = query_heads // kv_heads
    mapping = [head // queries_per_kv_head for head in range(query_heads)]
    cached_scalars = 2 * kv_heads * tokens * head_size

    print(name, "mapping:", mapping)
    print(name, "cached scalars:", cached_scalars)

# Output:
"""
MHA mapping: [0, 1, 2, 3, 4, 5, 6, 7]
MHA cached scalars: 768
GQA mapping: [0, 0, 0, 0, 1, 1, 1, 1]
GQA cached scalars: 192
MQA mapping: [0, 0, 0, 0, 0, 0, 0, 0]
MQA cached scalars: 96
"""

The mapping lists one KV head number for every query head. MHA uses a different KV head each time. GQA shares KV head 0 across four queries, then does the same with KV head 1. MQA maps every query head to KV head 0.

The cache count includes both keys and values. It then multiplies by six tokens and eight scalars per head.

Llama 3.1 70B uses 64 query heads and 8 KV heads. At 128K context, that produces a 40 GB BF16 cache instead of the 320 GB required by the same shape with 64 KV heads.

This code proves the storage ratio. It doesn’t test quality, because head sharing changes what the model learns and must be evaluated after training.

2. Cross-layer attention

GQA shares keys and values across heads within one layer. Cross-layer attention shares them across adjacent layers.

So a sharing factor of two implies two attention layers use one set of cached keys and values. The model now stores half as many unique layer caches.

The Cross-Layer Attention (CLA) paper trained 1B and 3B models and reported another 2x KV cache reduction while keeping accuracy close to the MQA baseline.

The next script represents two layers sharing one cache allocation:

tokens = 32
kv_heads = 2
head_size = 8

one_layer_cache = 2 * tokens * kv_heads * head_size
shared_cache = {"stored_scalars": one_layer_cache}
layer_caches = [shared_cache, shared_cache]

separate_total = 2 * one_layer_cache
shared_total = shared_cache["stored_scalars"]

print("layers point to same cache:", layer_caches[0] is layer_caches[1])
print("two separate layer caches:", separate_total, "scalars")
print("one shared layer cache:", shared_total, "scalars")

# Output:
"""
layers point to same cache: True
two separate layer caches: 2048 scalars
one shared layer cache: 1024 scalars
"""

One layer would store 1,024 scalars here. The leading 2 counts keys and values. Two independent layers would therefore store 2,048 scalars.

Both items in layer_caches point to the same Python object. The physical storage remains 1,024 scalars even though two logical layers use it.

Cross-layer sharing compounds GQA because it reduces a different term. GQA stores fewer KV heads inside each layer. CLA stores fewer distinct layer caches across the model.

A standard checkpoint learns separate KV projections for every layer. Each layer expects the cache created by its own weights. Redirecting it to another layer’s cache changes the model’s computation.

A CLA model trains with the sharing pattern already present. Training lets each dependent layer adapt to the shared cache. The serving engine must then reproduce the same ownership pattern.

The checkpoint and engine must agree on which layer owns each cache. This is why CLA cannot be enabled as a generic runtime flag.

3. Sliding windows

CLA reduces the number of stored layer caches. Sliding-window attention reduces the tokens stored inside selected layers.

In full self-attention, each new token can attend to every earlier token. Each attention layer must therefore keep keys and values for the full sequence. As the sequence grows, every layer’s KV cache grows with it.

Sliding-window attention limits a layer to recent tokens. Suppose the window contains 1,024 positions. That layer stores KV entries for only the latest 1,024 tokens. Each new entry removes the oldest one after the window fills. Its cache then stays fixed at 1,024 positions.

We call this a local layer because it sees nearby context. A global layer can still attend to every earlier token. Its cache keeps growing until the sequence ends.

Many models mix both layer types. Local layers provide most of the memory saving. Occasional global layers preserve access to distant parts of the prompt. The total cache still grows, but much slower than full attention everywhere.

For instance, Gemma 3 uses a repeating pattern of five local layers followed by one global layer. Its local window is 1,024 tokens, while the global layer supports the full 128K context:

The following example tracks the positions retained by a four-token window:

window_size = 4
retained_positions = []

for position in range(10):
    retained_positions.append(position)

    if len(retained_positions) > window_size:
        retained_positions.pop(0)

    print(f"after token {position}: {retained_positions}")

print("maximum retained positions:", window_size)

# Output:
"""
after token 0: [0]
after token 1: [0, 1]
after token 2: [0, 1, 2]
after token 3: [0, 1, 2, 3]
after token 4: [1, 2, 3, 4]
after token 5: [2, 3, 4, 5]
after token 6: [3, 4, 5, 6]
after token 7: [4, 5, 6, 7]
after token 8: [5, 6, 7, 8]
after token 9: [6, 7, 8, 9]
maximum retained positions: 4
"""

The list fills with positions 0 through 3. When position 4 arrives, the list becomes too long. Removing its first item drops position 0.

The same step repeats for every later token. Production engines usually use a ring buffer, which overwrites old slots without moving the remaining entries.

The first four tokens fill the available slots. Token 4 then removes token 0, so the window shifts to positions 1 through 4. Every later token shifts that range forward by one. After token 9, only positions 6 through 9 remain. The allocation stays fixed at four slots throughout.

The ratio also changes with context length. Below 1,024 tokens, both layouts retain the same number of positions. The saving appears only after the sequence exceeds the local window.

4. Multi-head latent attention

Multi-head latent attention, or MLA, doesn’t cache a complete key and value for every head. It compresses the hidden state into a smaller latent representation and stores that instead.

During decode, the model uses learned projections to recover the information needed by attention.

Implementations can absorb parts of those projections into the query and output paths, which avoids reconstructing every full KV tensor in memory.

DeepSeek-V2 introduced this design. The paper reported a 93.3 percent KV cache reduction compared with DeepSeek 67B and a 5.76x increase in maximum generation throughput:

An actual MLA layer uses learned matrices to create and use the latent representation. Those matrix operations hide the basic memory difference in a small example.

The following example counts storage for one token. The dimensions are intentionally small. They make the difference between both layouts visible.

kv_heads = 4
head_size = 8

standard_keys = kv_heads * head_size
standard_values = kv_heads * head_size
standard_total = standard_keys + standard_values

mla_latent = 8
mla_rotary_key = 4
mla_total = mla_latent + mla_rotary_key

print("standard cache:", standard_keys, "key +", standard_values, "value")
print("MLA cache:", mla_latent, "latent +", mla_rotary_key, "rotary key")
print("stored scalars per token:", standard_total, "->", mla_total)
print(f"example reduction: {standard_total / mla_total:.2f}x")

# Output:
"""
standard cache: 32 key + 32 value
MLA cache: 8 latent + 4 rotary key
stored scalars per token: 64 -> 12
example reduction: 5.33x
"""

Four KV heads with eight scalars each produce 32 key scalars. The values require another 32. Standard attention therefore stores 64 scalars per token.

The illustrative MLA layout stores eight latent scalars and four positional scalars. Its total is 12. Dividing 64 by 12 gives the example ratio.

The 5.33x figure belongs only to this example. Real MLA models choose different latent and positional dimensions.

MLA is not an inference flag for a standard checkpoint. Its projection weights and cache layout belong to the trained model. Converting another model still requires training.

5. Replace growing cache with fixed state using Hybrid models

Mamba and recurrent linear-attention layers don’t store one key and value for every old token. They update a fixed-size state as the sequence grows.

Pure recurrent models keep memory nearly constant with context, but full attention remains better at exact retrieval in many settings. Recent models combine both.

Qwen3-Next has 48 layers arranged as 12 repeated blocks. Each block contains three Gated DeltaNet layers and one full-attention layer. Only the 12 full-attention layers create a standard growing KV cache.

The model card reports 16 query heads, 2 KV heads, and a head dimension of 256 for those attention layers.

The next example compares their memory growth. The attention layer stores one 64-value key and value per token. The recurrent layer keeps one fixed 64-by-64 state.

width = 64
bytes_per_value = 4

attention_values_per_token = 2 * width
recurrent_state_values = width * width

for tokens in [128, 4096]:
    attention_kb = tokens * attention_values_per_token * bytes_per_value // 1024
    recurrent_kb = recurrent_state_values * bytes_per_value // 1024

    print(f"{tokens} tokens")
    print("  attention KV:", attention_kb, "KB")
    print("  recurrent state:", recurrent_kb, "KB")

# Output:
"""
128 tokens
  attention KV: 64 KB
  recurrent state: 16 KB
4096 tokens
  attention KV: 2048 KB
  recurrent state: 16 KB
"""

The attention calculation multiplies storage per token by sequence length. The recurrent calculation does not use tokens at all. Its state remains the same size in both cases.

The example tracks memory only. It does not implement Mamba or Gated DeltaNet. Those architectures use learned update rules to decide what the fixed state retains.

The attention cache grows 32x between the two sequence lengths. The recurrent state remains 16 KB. Hybrid models pay linear cache growth only in their full-attention layers.

For the Qwen3-Next attention shape, 12 full-attention layers require 3 GB of BF16 KV at 128K context.

Making all 48 layers full attention would raise that growing portion to 12 GB.

Jamba shows the same design choice with a different ratio. It interleaves one attention layer with seven Mamba layers. At 256K context, the Jamba paper reports a 4 GB KV cache, compared with 32 GB for Mixtral.

This layout belongs to the trained model. A serving engine cannot replace arbitrary attention layers with recurrent ones at runtime.

6. Compressed sparse attention

Full attention does two expensive things. It stores a separate KV entry for every earlier token, then reads every stored entry when generating the next token.

Sparse attention changes only the second part.

It reads fewer entries for each new token, but it may still keep the complete cache in memory. That can make generation faster without making the resident KV cache smaller.

Compressed sparse attention focuses on optimizing both parts.

• First, it combines several neighbouring token entries into one compressed entry. It stores that shorter sequence instead of one long-range entry per token.

• Then, for each new token, a selector chooses only the compressed entries that appear most relevant. Attention reads that selected subset.

DeepSeek V4 uses this design.

If the compression factor is m, every m token entries become one stored entry. So a context of n tokens produces roughly n / m compressed entries.

The model then selects k of those entries for attention. A small sliding window preserves finer detail for the most recent tokens.

At one million tokens, DeepSeek reports that V4-Pro uses 10 percent of the KV cache and 27 percent of the single-token inference FLOPs required by DeepSeek V3.2. V4-Flash goes further at 7 percent of the cache and 10 percent of the FLOPs

These are whole-model results. They include V4’s hybrid attention layout and precision choices, so they should not be read as the isolated effect of compression alone.

The next block isolates the two reductions. Compressing 32 long-range entries in groups of four leaves eight stored entries. If attention selects two of those eight, it reads only two compressed entries for the current step.

full_token_entries = 32
tokens_per_group = 4
selected_entries = 2

compressed_entries = full_token_entries // tokens_per_group

print("full token entries:", full_token_entries)
print("compressed entries stored:", compressed_entries)
print("compressed entries read:", selected_entries)
print("storage reduction:", f"{full_token_entries / compressed_entries:.0f}x")

# Output:
"""
full token entries: 32
compressed entries stored: 8
compressed entries read: 2
storage reduction: 4x
"""

Dividing 32 tokens into groups of four creates eight compressed entries. That is the storage reduction. Selecting two of those eight entries is the separate attention-work reduction.

The cache now holds eight long-range entries instead of 32. Attention reads two of them. A small local window separately preserves recent token detail.

This technique has to be built into the model and training process. A serving engine cannot convert an existing full-attention model into compressed sparse attention by changing a cache setting.

7. Query-aware sparse reads

The previous technique reduced both storage and attention work.

Sometimes, changing the model architecture is not an option. We may still want to reduce how much cache each decode step reads.

Recall what happens while the model generates a token. The new token produces a query, which acts like a search signal. Full attention compares it with every stored key. A long prompt therefore forces the GPU to load a large cache repeatedly.

Quest is a technique that reduces those reads without deleting any KV entries.

• It divides the cache into pages.

• A page is a small group of neighbouring token entries.

• Each page also stores the minimum and maximum key values found inside it.

For every new query, Quest scores these short page summaries first. It then loads only the pages with the highest scores. The full cache remains stored because a page ignored now may matter later.

The Quest paper reports up to 7.03x lower self-attention latency. Its complete inference system reached up to 2.23x speedup.

The example below uses four pages. Each page contains two keys. A key has only two numbers here, which keeps the scoring visible.

import numpy as np

pages = np.array([
    [[1, 0], [2, 1]],
    [[5, 1], [4, 2]],
    [[0, 6], [1, 5]],
    [[-2, -1], [-1, 0]],
])
query = np.array([1.0, 0.5])

page_min = pages.min(axis=1)
page_max = pages.max(axis=1)
page_scores = np.maximum(query * page_min, query * page_max).sum(axis=1)
selected_pages = np.argsort(page_scores)[-2:]

for page, score in enumerate(page_scores):
    print(f"page {page} score: {score:.1f}")

print("selected pages:", np.sort(selected_pages).tolist())
print("KV entries read:", len(selected_pages) * 2, "of", pages.size // 2)
print("KV entries still stored:", pages.size // 2)

# Output:
"""
page 0 score: 2.5
page 1 score: 6.0
page 2 score: 4.0
page 3 score: -1.0
selected pages: [1, 2]
KV entries read: 4 of 8
KV entries still stored: 8
"""

The pages array contains eight keys in total. The next two lines create one minimum and one maximum summary per page. Quest uses similar summaries to estimate a page’s best possible attention score.

The query prefers large values in both key dimensions. Pages 1 and 2 receive the highest scores, so the example reads their four keys. It leaves the other four keys untouched during this step.

If you notice the last line, the step reads four entries, but all eight remain resident. Quest reduces cache traffic and attention work. It does not reduce the memory needed to hold the full cache.

In practice, Quest uses many-dimensional keys and dedicated GPU code. It also repeats the selection for every query.

8. Quantization

Quest keeps every KV entry and skips some reads. Quantization takes another route. It keeps every entry but stores each number with fewer bits.

Keys and values normally contain floating-point numbers. BF16 stores each number with 16 bits. FP8 uses 8 bits, so its raw payload is roughly half as large. A 4-bit format cuts the payload to 25%.

Quantization maps many original numbers onto a smaller set of allowed values. A scale controls the gap between those values. The engine stores the small integer code and enough scale information to interpret it later.

The choice of scale affects accuracy. One extreme value can make a shared scale too wide. Most ordinary values then get rounded too aggressively.

KIVI handles keys and values differently. It groups key values by channel, which means the same feature across tokens. It groups value-cache numbers by token. The KIVI paper reports 2.6x lower peak memory and up to 4x larger batches with its 2-bit cache.

The small example below shows the basic mapping. It converts six BF16-style values into signed 4-bit codes. It then converts those codes back into approximate values.

import numpy as np

values = np.array([-1.0, -0.55, -0.10, 0.25, 0.60, 1.0])
bits = 4
largest_code = 2 ** (bits - 1) - 1
scale = np.max(np.abs(values)) / largest_code

codes = np.rint(values / scale)
codes = np.clip(codes, -largest_code, largest_code).astype(np.int8)
restored = codes * scale

print("original values:", values.tolist())
print("4-bit codes:", codes.tolist())
print("restored values:", np.round(restored, 3).tolist())
print("raw payload:", len(values) * 16, "bits ->", len(values) * 4, "bits")

# Output:
"""
original values: [-1.0, -0.55, -0.1, 0.25, 0.6, 1.0]
4-bit codes: [-7, -4, -1, 2, 4, 7]
restored values: [-1.0, -0.571, -0.143, 0.286, 0.571, 1.0]
raw payload: 96 bits -> 24 bits
"""

Signed 4-bit storage provides codes from -7 through 7 in this example. Dividing by scale maps each original value to a code. Multiplying by the same scale produces the approximate value used during attention.

As you may have noticed, the restored values are close, but not identical. That rounding difference is quantization error. The final line counts only the six stored values. A real cache also stores scale metadata.

The example uses one scale for all six values. KIVI uses smaller groups because each group can fit its own numerical range. This usually reduces rounding error, though it requires more scale metadata.

If you use vLLM, it exposes FP8 cache storage directly:

MODEL_ID=Qwen/Qwen2.5-7B-Instruct
vllm serve "$MODEL_ID" --kv-cache-dtype fp8

The first line defines the model name. The second starts a vLLM server and asks it to store the KV cache in FP8. This command keeps running because it launches the inference server.

Recent vLLM versions can also leave selected layers in their native type. This is useful when a sliding-window layer proves more sensitive:

MODEL_ID=google/gemma-3-4b-it
vllm serve "$MODEL_ID" \
  --kv-cache-dtype fp8 \
  --kv-cache-dtype-skip-layers sliding_window

This version leaves sliding-window layers in their original format. It quantizes the remaining cache layers. The option helps when those local-attention layers lose too much accuracy under FP8.

Quantization preserves every token position. The next technique reduces memory by removing positions instead.

9. Eviction

Quantization keeps all token positions. That may still be too large for a long context. Eviction sets a fixed position budget and discards entries outside it.

The difficult question is which positions should not be evicted. Recent tokens often matter because they contain the current conversation. Older tokens may still carry instructions, facts, or tool results needed later.

H2O is a technique that keeps recent positions and older heavy hitters. A heavy hitter is a token that has accumulated high attention during earlier steps. Once H2O evicts another token, its KV entry is no longer available.

Other methods estimate importance differently. SnapKV studies attention near the end of the input prompt. It uses that observation window to choose prompt positions before generation.

PyramidKV gives lower layers larger budgets and higher layers smaller ones, based on the attention patterns reported by its authors.

SnapKV reports 8.2x higher memory efficiency and 3.6x faster generation with 16K-token inputs. PyramidKV reports matching full-cache performance while retaining 12 percent of the cache in its LongBench experiments.

The following example takes a snapshot after ten tokens. Their scores represent attention accumulated during earlier decode steps. The cache can retain only six positions, including the latest two.

scores = [9.0, 0.2, 4.0, 0.1, 0.3, 6.0, 0.2, 0.4, 0.1, 0.2]
budget = 6
recent_count = 2

positions = list(range(len(scores)))
recent = positions[-recent_count:]
older = positions[:-recent_count]

heavy_hitter_count = budget - recent_count
heavy_hitters = sorted(
    older,
    key=lambda position: scores[position],
    reverse=True,
)[:heavy_hitter_count]

retained = sorted(heavy_hitters + recent)

print("heavy hitters:", sorted(heavy_hitters))
print("recent positions:", recent)
print("retained positions:", retained)
print("cache entries:", len(retained), "of", len(scores))

# Output:
"""
heavy hitters: [0, 2, 5, 7]
recent positions: [8, 9]
retained positions: [0, 2, 5, 7, 8, 9]
cache entries: 6 of 10
"""

Positions 8 and 9 are retained because they are the newest. Four slots remain. The sort selects positions 0, 2, 5, and 7 because they have the largest scores among older tokens.

The union of those two groups produces the final six-position cache. Positions 1, 3, 4, and 6 are discarded.

Real H2O updates its heavy-hitter statistics as generation proceeds. This smaller example starts with the accumulated scores so the selection rule remains visible.

That said, eviction needs a cautious evaluation. Importance changes across turns. A tool result that receives little attention now may become necessary after several more calls. System instructions and delimiters can also be disproportionately important.

Eviction evaluations should therefore use complete agent traces and delayed references. They should also include structured prompts. A good Needle-in-a-Haystack score does not cover those cases.

10. Paging

So far, we have looked at what one request stores or reads. Paging solves a different problem: how the serving engine allocates cache memory across many requests.

A simple allocator may reserve one continuous memory region per request. It often reserves enough space for the request’s maximum length. Shorter requests leave part of that region unused. Free space also gets split into gaps as requests finish.

PagedAttention divides GPU cache memory into equal-sized blocks. Each block holds KV entries for a fixed number of tokens. A request can use blocks found anywhere in the shared pool, so its blocks do not need to lie next to each other.

When a request finishes, the allocator returns its blocks to the pool. Another request can reuse them immediately. The PagedAttention paper reports near-zero KV cache waste and 2x to 4x higher throughput than the systems tested.

A fixed pool can hide the effect of quantization. The engine may reserve the same 8 GB before and after enabling FP8. Smaller entries increase the number of tokens that fit inside that pool. The allocated memory shown by nvidia-smi may remain unchanged.

The example below begins with six free blocks. Request A takes three. Request B takes two. A then finishes and returns its blocks before request C arrives.

free_blocks = [0, 1, 2, 3, 4, 5]

request_a = free_blocks[:3]
free_blocks = free_blocks[3:]

request_b = free_blocks[:2]
free_blocks = free_blocks[2:]

free_blocks.extend(request_a)
free_after_a = free_blocks.copy()
request_c = free_blocks[:2]
free_blocks = free_blocks[2:]

print("request A blocks:", request_a)
print("request B blocks:", request_b)
print("free after A finishes:", free_after_a)
print("request C blocks:", request_c)

# Output:
"""
request A blocks: [0, 1, 2]
request B blocks: [3, 4]
free after A finishes: [5, 0, 1, 2]
request C blocks: [5, 0]
"""

After A finishes, the free pool contains block 5 and A’s returned blocks. Request C takes blocks 5 and 0. Those block numbers are not adjacent, but the block table records their order for C.

This is the central paging idea. The allocator reuses available blocks instead of searching for one large continuous region.

vLLM also tracks which request owns each block. Its attention code can read the non-adjacent blocks in logical sequence order. The short example isolates allocation and reuse.

Quantization still changes the capacity inside this pool. An 8 GB pool holds 26,214 tokens at 320 KB per token, compared with 52,428 tokens at 160 KB. Total allocated GPU memory remains 8 GB in both cases.

11. Prefix reuse

Paging reuses blocks after a request finishes. Prefix caching can share blocks while several requests are still active. It applies when those requests begin with exactly the same tokens.

This happens often with system prompts and tool definitions. Without prefix caching, the engine computes and stores another KV copy for every request. The repeated prefix consumes memory and repeats the prefill work that created it.

Automatic prefix caching stores each completed prefix block under a lookup key. When another request has the same token prefix, it points to the existing KV block. New computation begins only after the requests diverge.

vLLM builds each lookup key from the current block and its preceding blocks. It also includes details such as adapter identifiers when they affect compatibility. The vLLM prefix-caching design documents this structure.

The example below uses four-token blocks. Each lookup key contains the entire prefix up to that block. This is a readable stand-in for vLLM’s chained hashes.

def prefix_keys(tokens, block_size):
    block_ends = range(block_size, len(tokens) + 1, block_size)
    return [tuple(tokens[:end]) for end in block_ends]

first = [10, 11, 12, 13, 20, 21, 22, 23, 30, 31, 32, 33]
second = [10, 11, 12, 13, 20, 21, 22, 23, 40, 41, 42, 43]

cache = set()
first_keys = prefix_keys(first, block_size=4)
first_hits = sum(key in cache for key in first_keys)
cache.update(first_keys)

second_keys = prefix_keys(second, block_size=4)
second_hits = sum(key in cache for key in second_keys)

print("first request reused:", first_hits, "of", len(first_keys), "blocks")
print("second request reused:", second_hits, "of", len(second_keys), "blocks")
print("second request computes:", len(second_keys) - second_hits, "new block")

# Output:
"""
first request reused: 0 of 3 blocks
second request reused: 2 of 3 blocks
second request computes: 1 new block
"""

The first request has no cache entries to reuse. It adds keys for its first four, eight, and twelve tokens. The second request has the same first eight tokens, so its first two keys match.

The final four tokens differ. That changes the last prefix key, so the second request computes one new block.

Using the full prefix prevents a false match. Two requests cannot reuse a block merely because four tokens match somewhere in the middle. Their preceding context must match as well.

Prefix reuse depends on exact tokenized prefixes. A timestamp in the system prompt, reordered JSON keys in a tool schema, or different chat-template whitespace can break the match.

Stable prompt templates therefore improve the hit rate. The model and cache format can remain unchanged while duplicate storage drops.

12. Cache Offloading from GPU

Prefix reuse helps when another request begins with tokens the engine has already processed. Offloading handles a different problem. It moves KV blocks that are not currently needed from GPU memory to a larger but slower memory tier, usually CPU memory.

Those blocks may belong to a sequence paused by the scheduler or to a cached prefix retained for reuse. When the engine needs them again, it copies them back to the GPU.

There is one important limitation. A new OpenAI-compatible API request does not automatically resume the KV cache from an earlier request. Reuse happens only if the engine still tracks the sequence, recognizes an exact prefix-cache match, or restores the cache through an external storage system.

Offloading frees GPU memory without deleting the stored KV data. It does not reduce the total number of bytes held across GPU and CPU memory. The trade-off appears when the engine must transfer those blocks back before computation can continue.

The example below gives the GPU room for two sessions. Adding session C moves the oldest session, A, to CPU memory. Resuming A brings it back and moves B out.

gpu_cache = ["cache-a", "cache-b"]
cpu_cache = []

# Cache C needs GPU space, so cache A moves to CPU memory.
cpu_cache.append(gpu_cache.pop(0))
gpu_cache.append("cache-c")

print("GPU after cache C arrives:", gpu_cache)
print("CPU after cache C arrives:", cpu_cache)

# Cache A is needed again, so it returns to the GPU.
cpu_cache.remove("cache-a")
cpu_cache.append(gpu_cache.pop(0))
gpu_cache.append("cache-a")

print("GPU after cache A returns:", gpu_cache)
print("CPU after cache A returns:", cpu_cache)

# Output:
"""
GPU after cache C arrives: ['cache-b', 'cache-c']
CPU after cache C arrives: ['cache-a']
GPU after cache A returns: ['cache-c', 'cache-a']
CPU after cache A returns: ['cache-b']
"""

The first move leaves caches B and C on the GPU while cache A remains available in CPU memory. When cache A is needed again, the example brings it back and moves cache B out.

The lists contain cache names instead of tensors. A real engine performs this operation on KV blocks and tracks which request or reusable prefix owns each block.

The example assumes that the engine has retained enough information to identify cache A later. Sending an unrelated API request with the same label would not restore it automatically.

vLLM can enable a 16 GB CPU offloading buffer directly:

MODEL_ID=Qwen/Qwen2.5-7B-Instruct

vllm serve "$MODEL_ID" \
  --kv-offloading-size 16 \
  --kv-offloading-backend native

The first option sets the CPU buffer size in GB. The second selects vLLM’s native CPU backend. Like the earlier serving command, this process keeps running after startup.

Offloading trades GPU capacity for transfer time. Measure resume latency and CPU bandwidth before enabling a large offload tier.

How to use them together

Some of these methods compose because they reduce different terms. Architecture-level methods still depend on how the checkpoint was trained.

For instance, take the 40 GB GQA cache from the first example.

• CLA2 can cut the unique layers in half, producing 20 GB.

• FP8 can halve the bytes per value, producing roughly 10 GB.

• A 50 percent token budget would reduce the resident cache to roughly 5 GB.

That multiplication is useful for capacity planning, but it hides model quality and kernel support.

• CLA requires training.

• FP8 introduces quantization error.

• Eviction discards context.

• And the serving engine must also support the resulting layout.

These are some situations you may face, along with solutions:

• When choosing a model → Inspect KV head count, local and global layer ratios, latent-attention dimensions, and recurrent layers. These determine the starting cache before deployment.

• When serving an existing model → Test FP8 first, then standardize prefixes. Both preserve the token set. Add eviction only after workload-specific quality tests.

• When memory appears unchanged → Inspect cache capacity and block counts. A fixed pool can absorb the savings while total allocated GPU memory stays flat.

• When latency is the problem → Measure cache reads and attention time. Quest-style sparse reads may help even when they don’t free capacity.

• When inactive sessions occupy the GPU → Test offloading and scheduling. Moving cold blocks can matter more than compressing the active ones further.

KV cache reduction is easier to reason about once each technique has a specific target.

This table summarizes all of the techniques we cover, what they alter, the takeaway from it, and what part they don’t handle:

Start with the constraint you need to change, then pick the technique that handles it.

To dive deeper into the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:

• Read Part 1 on fundamentals of LLMOps here →

• Read Part 2 on understanding the core building blocks of LLMs →

• Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →

• Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →

• Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →

• Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →

• Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →

• Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →

• Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →

• Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →

• Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →

• Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →

• Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →

• Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions.

👉 Over to you: how do you reduce KV cache in production?

Good day!

Just run:

npm i -g @magnitudedev/cli
magnitude setup

It profiles your machine and ranks the models across:

• Speed

• Accuracy

• Intelligence

• Memory required

Finally, you can choose your favorite harness (Pi, OpenCode, Claude Code, Codex, etc.) to run with it.

Here’s the GitHub repo →

(don’t forget to star 🌟)

5 embedding compression techniques

Ten million 1,536-dimensional embeddings will occupy:

• 62 GB in float32

• 15 GB in int8

• 2 GB as packed bits

This only covers the raw vector payload, and an in-memory system also needs space for the ANN index, metadata, and allocator overhead.

Embedding compression works along two axes:

• the number of dimensions stored

• the number of bits used for each dimension

The five techniques in the visual reduce different parts of the payload.

1) PCA applies a post-training transform:

It learns the directions with the most variance from a representative sample, then projects existing embeddings into a smaller space.

It works with any embedding model, although the projection must be fitted and applied consistently to indexed vectors and queries.

2) MRL alters the training objective:

The model is trained so that selected prefixes (say, the first <n> dimensions) of the embedding remain useful independently.

You can therefore truncate an MRL embedding at inference without training a separate model for every target dimension.

OpenAI reports that text-embedding-3-large at 256 dimensions still outperforms the 1,536-dimensional text-embedding-ada-002 on MTEB.

3-4) Scalar and binary quantization keep the dimension count fixed and reduce the representation used for each value.

Scalar quantization typically maps float32 values to int8, giving a 4x reduction in the raw vector payload.

The scale and offset add a small amount of metadata.

Binary quantization keeps one bit per dimension, which gives a 32x reduction.

Similarity search can then use XOR and a population count instead of floating-point distance calculations.

5) Product Quantization encodes subvectors as centroid IDs:

It splits a vector into subvectors and replaces each subvector with the ID of its nearest centroid.

Query distances are then approximated through centroid lookup tables.

In all of these setups, the compressed representation does not need to produce the final ranking.

Usually, you retrieve extra candidates from the compressed index, then recompute similarity using higher-precision document embeddings.

This is especially useful with binary quantization. One bit can preserve enough coarse structure but loses magnitude information.

Rescoring improves the ordering, but it cannot recover an item missed by the initial compressed-retrieval stage.

Also, these methods can work together. An MRL embedding can be truncated first and quantized afterward, reducing both dimension and precision.

All of these techniques reduce the memory and compute a retriever uses once the index holds millions of vectors.

But it is a small part of building a retriever that works in production, which is what the RAG Systems course goes through. Here is the full series in order.

• Part 1 covers the foundations of a RAG system, including vector search, chunking, and embeddings →

• Part 2 shows how to evaluate a RAG system and which metrics tell you whether it actually works →

• Part 3 works on making retrieval faster and lighter on memory →

• Part 4 extends RAG to multimodal data like images and tables, not just text →

• Part 5 explains CLIP embeddings and multimodal prompting that put text and images in one space →

• Part 6 builds a full multimodal RAG system on real-world data →

• Part 7 brings in Graph RAG to use the relationships between entities during retrieval →

• Part 8 introduces late interaction with ColBERT for token-level matching between the query and the document →

• Part 9 builds vision-driven retrieval with ColPali that searches over document images directly →

• Part 10 walks through the first set of 16 practical techniques for production RAG →

• Part 11 continues with the remaining techniques from that set →

• Part 12 explains why RAG latency is really a prefill problem, the model reading the chunks rather than finding them →

• Part 13 starts the preloading arc by loading a corpus into cache before any query arrives →

• Part 14 compresses that cache and covers why most published methods still need a query first →

• Part 15 closes the preloading arc with the training-based methods and what running them in production takes →

Good day!

Agents can build a beautiful frontend in minutes, set up API routes, and lay out the component architecture. But the moment it needs to enable auth or configure a database, it completely falls apart.

The reason is that every backend platform today (Firebase, Supabase, AWS) was designed for humans clicking through dashboards. When agents try to interact with these platforms through MCP servers, they get fragmented context like table names without schema details or auth endpoints without security configs. So agents end up guessing, hallucinating, and generating broken code.

​​InsForge​​ fixes this at the infrastructure level rather than the tooling level. It introduces a semantic layer where every backend primitive (auth, database, storage, AI features) is exposed as structured, machine-readable capabilities with metadata, constraints, and documentation baked in.

InsForge GitHub Repo

Primitives are also aware of each other, so auth knows about database permissions and storage understands access policies.

Because agents get a complete, structured context instead of inferring what’s missing, InsForge delivers:

• Roughly 2x more accuracy than Supabase MCP

• 1.6x faster task completion

• 30% better token efficiency

To test this out, we built a full ChatGPT clone with auth, database, storage, and AI integration, built entirely with Claude Code using InsForge as the backend. No manual configuration was needed, not because of any magic, but because the agent could reason about the entire backend as one coherent system.

InsForge works with any AI coding agent, including Cursor, Claude Code, Windsurf, and Codex. You can use all the primitives together or just pick what you need, like database only or auth only.

It’s fully open-source under Apache 2.0.

​​Find the GitHub repo here →​​ (don’t forget to star it ⭐️)

Attention Mechanisms in LLMs, clearly explained

Every model card advertises its attention mechanism.