• Your agents are doing web search wrong.

• 6 LLM deployment formats in production.

• ​Prevent data leakage in ML pipelines​.

Your agents are doing web search wrong

The agent isn't searching badly. It gets handed a table of contents and is left to go read the web itself.

A search call returns links and thirty-word snippets, so the agent fetches each page, strips the markup, and cleans the text before any reasoning starts, and that work repeats on every hop of the loop.

The same question runs above 28,000 tokens through a three-hop search loop and under 7,000 through an owned index that already holds the processed page.

Seltz is that index. It crawls and processes pages ahead of the query, so one call returns a finished document across its people, news, and wiki scopes.

Try Seltz here →

Get Seltz API key

Thanks to Seltz for partnering today!

6 LLM deployment formats in production

Google and Anthropic agree on one thing about LLM inference:

The faster a model runs, the fewer machines can run it.

This sounds counterintuitive, but a model file gets faster by locking in hardware-specific choices into the file.

And those choices only fit the hardware they were made for.

That is why they build the same model more than once, one version per chip or runtime it has to run on.

The visual below covers the six formats used to run LLMs in production:

1-2) Pickle and safetensors hold raw weights. Both run anywhere Python runs, so neither is faster than the framework around it.

A .pt file is a list of instructions that torch.load executes and loading a checkpoint can also run a payload (possibly malicious).

Safetensors holds plain numbers instead, laid out the way memory needs them, so there is nothing to rebuild when the file opens.

Both still need Python, the model code, and a tokenizer alongside.

3) GGUF is llama.cpp’s format, built so a model can run on a machine with no ML framework installed.

To achieve this, weights, tokenizer, and chat template ship in one file, and llama.cpp or Ollama runs it directly.

4) ONNX is an interchange format, built so a model trained in one framework can run somewhere that framework was never installed.

It decides the operations and the order they run in, but leaves the hardware open, so the runtime picks CPU, CUDA, or an NPU when the file loads.

5) MLX is Apple’s framework for its own silicon, where the CPU and GPU share a single pool of memory.

It decides where the weights live, so they are never copied between the two, and that advantage does not help on any other machine.

6) TensorRT is NVIDIA’s compiler, which turns a model into machine code for one specific GPU.

Everything is settled at build time, down to instructions tested on the exact card it compiles for, which is why the file will not load on a different GPU architecture.

The safe default is to stop as high on the chart as the latency budget allows, because every row down trades speed with the types of hardware it can run on.

To dive deeper into the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:

• Read Part 1 on fundamentals of LLMOps here →

• Read Part 2 on understanding the core building blocks of LLMs →

• Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →

• Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →

• Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →

• Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →

• Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →

• Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →

• Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →

• Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →

• Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →

• Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →

• Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →

• Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions.

​Prevent data leakage in ML pipelines​

​Data leakage​ happens when your model accidentally gets access to information during training that it won’t have during inference.

And the scary part is that it can be incredibly subtle.

Let us walk you through the most common ways it sneaks in.

​To specifically explore data leakage in greater depth with code, we covered it in Part 6 of the MLOps course here →​

Why is data leakage dangerous?

A leaked feature can make a model seem extremely accurate during training/validation, because it’s indirectly using the answer!

But when deployed, that info isn’t available, so the model fails unexpectedly.

Leakage can be subtle and is often discovered only after deployment, when the model behaves too well on historical data but poorly on new data.

Common causes of data leakage

Train/test contamination

This is the most straightforward but pernicious case. Training data somehow bleeds into the test set. For example, random shuffling might break temporal integrity in time-series data, accidentally leaking future information into the model’s eyes.

Leaking through preprocessing

Do not scale or transform using the combined statistics of your dataset. This subtle form of leakage creeps in when the test data informs transformations like scaling.

For example, scaling features to 0-1 using the min and max of the entire dataset (including test) leaks knowledge of the test distribution into the train.

To prevent this type of leakage, always fit preprocessing only on the training set, then apply it to the validation/test sets.

Using target-derived features

This is a common mistake in feature engineering. For instance, say you’re predicting whether a user will churn next month, and you accidentally include a feature like “number of logins in next month,” which obviously references the outcome.

Sometimes this happens less obviously: e.g., you include a summary that was computed including the target period.

How to prevent this? Think carefully. Any feature that wouldn’t be available at prediction time (or that uses information from the future relative to prediction) is a leakage risk.

How to prevent leakage

Holdout validation

Always assess your models on a truly independent dataset. Steep drops in performance from training to evaluation signal potential leakage.

Feature importance analysis

Calculating feature importance reveals suspicious features masquerading as informative only due to leakage. If a particular feature is overwhelmingly important, inspect it; could it be leaking info?

Null model testing

In deliberately nonsensical setups (e.g., predicting 2019 with a 2020 model), leakage causes even ill-formed models to perform inexplicably well.

A leak-safe approach

1. Perform train/validation/test splits before peeking or processing.

2. Use stats from only the training data for things like scaling and encoding.

3. In time-dependent data, keep the chronological order for validation.

Understanding leaks and vigilantly safeguarding against them ensures your ML pipeline runs smoothly and reliably in the real world.

If you want to learn more about these real-world ML practices and start your journey with MLOps, we have already covered MLOps from an engineering perspective in our 18-part crash course.

It covers foundations, ML system lifecycle, reproducibility, versioning, data and pipeline engineering, model compression, deployment, Docker and Kubernetes, cloud fundamentals, virtualization, a deep dive into AWS EKS, monitoring, and CI/CD in production.

​Start with MLOps Part 1 here →​

​To specifically explore data leakage in greater depth with code, we covered it in Part 6 of the MLOps course here →

Good day!​

You have 80,000 agent trajectories from production. You need to find top 100 worth reviewing to improve your agent.

No LLM allowed to evaluate trajectories. How will you do this?

Let’s look at some approaches.

The simplest solution one could start with is random sampling. Pick 100 random trajectories and review.

But most production agents handle routine requests just fine, so you end up wasting a big chunk of your annotation budget.

Another approach can filter for longer conversations since 10+ user messages means more complexity.

But longer conversations skew heavily toward outright failures. You’ll surface obvious breakdowns but miss subtle issues hiding in conversations where the agent technically succeeded.

A ​recent paper from DigitalOcean takes a new approach.

It computes lightweight behavioral signals directly from the trajectory data using deterministic rules.

The signals fall into three groups:

1) Interaction signals:

• If a user rephrases the request or corrects the agent, that’s misalignment.

• Agent repeating itself is stagnation.

• User abandoning the agent is disengagement.

• User confirming something worked is satisfaction.

All are detected through normalized phrase matching and similarity checks.

2) Execution signals:

• A tool call that doesn’t advance the task is a failure signal.

• Repeated calls with identical or drifting inputs indicate a loop.

These are straightforward to extract from execution logs.

3) Environment signals, like rate limits, context overflow, and API errors.

• Useful to diagnose but not for training since they reflect system constraints, not agent decisions.

Each trajectory gets scored based on which signals fire, and you sample the highest-signal ones for review.

On τ-bench, they compared all three approaches on 100 trajectories:

• Random sampling hit a 54% informativeness rate.

• The length-based heuristic reached 74%.

• Signal-based sampling reached 82%.

This means roughly 4 out of every 5 trajectories are genuinely useful to improve the agent.

In fact, among conversations where the agent completed the task correctly, signal sampling still identified useful patterns in 66.7% of cases vs. 41.3% for random.

These are the subtle issues like policy violations, inefficient tool use, and unnecessary steps that don’t break the task but still matter for optimization.

The whole framework runs without any LLM overhead and can sit always-on in a production pipeline.

If you want to see this in practice, this signal-based approach is already integrated into ​Plano​, an open-source AI-native proxy that handles routing, orchestration, guardrails, and observability in one place.

​Here’s the Plano GitHub repo →​

​Here’s the paper on arxiv →​

👉 Over to you: What is your approach to solve this?

Serverless vs on-prem vs edge deployment

Serverless, on-prem, and edge deployment are three different answers to the same question of where your model actually runs, and each one costs you something different.

The visual below depicts them:

• Serverless means you hand your model to a provider that only turns on a machine when a request arrives. Between requests the container shuts down, so you pay nothing while idle. The weights get unloaded along with it, so the next call has to pull gigabytes back into memory before it can answer, which adds up to 90 seconds. A reranker sitting in the middle of a search cannot spend that long waking up, so you keep instances warm, and you are back to paying for idle GPUs.

• On-premise means you rent or own the GPU yourself and keep the server running inside your own network. Nothing leaves your walls, there is no cold start because the engine already holds the weights, and you pay a flat hourly rate instead of a per-token meter. The card runs whether traffic arrives or not.

• Edge means the model runs on the user’s own device, on its NPU or CPU, through a small local runtime. The model has to be shrunk down to fit, so it works offline with nothing ever leaving the hardware, but it handles one narrow task rather than a full pipeline.

On-premise is where serious teams land, and it only pays off if several models share one card.

In reality, this rarely happens because vLLM pre-allocates 90% of the GPU at startup and ignores every other vLLM instance on it, while HuggingFace TEI (Text Embeddings Inference) takes one model-id per process. Neither one knows the other is there.

So four small models end up on four separate cards. You switched to small models to spend less, and you are now renting four GPUs to run what one could hold.

The serving engine box in on-prem decides whether on-prem is cheap or just differently expensive.

The card is not the constraint. Instead, the missing piece is one server that holds all the models and shifts memory between them as traffic moves, instead of four servers each convinced they own the whole GPU.

The Superlinked Inference Engine (SIE) is the open-source project built for that slot. One server handles embeddings, reranking, extraction, and generation, loads a model the first time a request needs it, and drops the least recently used one when memory runs short.

GitHub repo: https://github.com/superlinked/sie

(don’t forget to star 🌟)

We also wrote a detailed breakdown on what SIE is and how to get started with it.

Read it here →

​CPU vs. GPU vs. TPU vs. NPU vs. LPU​

5 hardware architectures power AI today.

Each one makes a fundamentally different tradeoff between flexibility, parallelism, and memory access.

The visual below maps the internal architecture of all five side by side:

CPU

It is built for general-purpose computing. A few powerful cores handle complex logic, branching, and system-level tasks.

It has deep cache hierarchies and off-chip main memory (DRAM). It’s great for operating systems, databases, and decision-heavy code, but not that great for repetitive math like matrix multiplications.

GPU

Instead of a few powerful cores, GPUs spread work across thousands of smaller cores that all execute the same instruction on different data.

This is why GPUs dominate AI training. The parallelism maps directly to the kind of math neural networks need.

TPU

They go one step further with specialization.

The core compute unit is a grid of multiply-accumulate (MAC) units where data flows through in a wave pattern.

Weights enter from one side, activations from the other, and partial results propagate without going back to memory each time.

The entire execution is compiler-controlled, not hardware-scheduled. Google designed TPUs specifically for neural network workloads.

NPU

This is an edge-optimized variant.

The architecture is built around a Neural Compute Engine packed with MAC arrays and on-chip SRAM, but instead of high-bandwidth memory (HBM), NPUs use low-power system memory

The design goal is to run inference at single-digit watt power budgets, like smartphones, wearables, and IoT devices.

Apple Neural Engine and Intel’s NPU follow this pattern.

LPU (Language Processing Unit)

This is the newest entrant, by Groq.

The architecture removes off-chip memory from the critical path entirely. All weight storage lives in on-chip SRAM.

Execution is fully deterministic and compiler-scheduled, which means zero cache misses and zero runtime scheduling overhead.

The tradeoff is that it provides limited memory per chip, which means you need hundreds of chips linked together to serve a single large model. But the latency advantage is real.

AI compute has evolved from general-purpose flexibility (CPU) to extreme specialization (LPU). Each step trades some level of generality for efficiency.

The visual below maps the internal architecture of all five side by side, and it was inspired by ByteByteGo’s post on CPU vs GPU vs TPU. We expanded it to include two more architectures that are becoming central to AI inference today.

That said, if you want to get hands-on with actual GPU programming using CUDA, learn about how CUDA operates GPU’s threads, blocks, grids (with visuals), etc., we covered it here: ​​​​Implementing (Massively) Parallelized CUDA Programs From Scratch Using CUDA Programming.

👉 Over to you: Which of these 5 have you actually worked with or deployed on?

Good day!

• Agent memory and state are not the same thing!

• Graph engineering, clearly explained.

• ​The anatomy of diffusion LLMs.

Agent memory and state are not the same thing!

If an agent forgets something it has already learned, that’s a memory problem. If it forgets where it was in the middle of a task and starts over, that’s a state problem.

We once killed one of our agents mid-task to test something else, and it started over like the execution so far never happened.

That’s when it clicked that we’d been treating two different problems as one.

State is tied to the current run as to what task the agent is working on and what it’s already found.

None of that exists unless something writes it down.

The fix was to add a checkpoint after every completed step that records the agent’s progress, so if the process dies, it resumes from that exact point instead of starting from scratch.

Memory is a different thing entirely. It’s what survives across runs as facts, lessons, and findings that are worth retaining.

At first, we had one shared memory for all our agents and assumed that was enough. But it wasn’t until our agents started reading each other’s findings and treating them as their own.

That’s why giving each agent its own memory scope is important with memory = memory.scope(”/agent”).

Once we separated state from memory, everything became much easier to reason about.

Here’s what the final Agent Harness looks like:

• separate memory from state as two different problems

• scope memory per agent when findings shouldn’t be shared

• write a checkpoint after every completed task

• resume interrupted runs from the last checkpoint

• fork a checkpoint into a new branch without redoing previous work

This is the harness baseline to build any agent. It’s enough for general-purpose workflows, but coding agents and long-running systems need other layers along.

We wrote an article that builds that next layer from scratch. It walks through planning, the agent loop, subagents, sandboxing, memory, and checkpointing, one layer at a time.

The whole thing is built with CrewAI, a 100% open-source framework.

You can read the article here →

Graph engineering clearly explained

The moment you have several loops that need to work together, you have a coordination problem, and graphs are how engineers have always described coordination.

That’s the whole idea behind graph engineering that Peter Steinberger mentioned a few days back.

Today, let’s understand what exactly graph engineering is!

First, the graph itself

A graph is three things.

• Nodes are units of work, whether that’s an agent, a plain model call, a deterministic function, a tool, or a human approving something.

• Edges decide what runs next, either in sequence, in parallel, or conditionally based on what the last node produced.

• State is a shared object that flows along the edges. Every node reads from it and writes to it.

graph.add_node("research", research_agent)
graph.add_node("write", writer_agent)
graph.add_node("review", reviewer_agent)

graph.add_edge("research", "write")
graph.add_edge("write", "review")

graph.add_conditional_edge("review",
    lambda state: "done" if state.approved else "write")

This is the starter graph almost every example uses. A researcher gathers material, a writer drafts, and a reviewer judges. If the review passes, the run ends. If it fails, an edge sends the draft back to the writer.

There are three nodes and four edges, and one of those edges forms a loop.

But here’s what reframes everything. A single agent loop is just a one-node graph with an edge pointing back to itself.

Graphs don’t replace loops but rather connect and govern them.

The stack kept growing

The center of gravity in AI keeps drifting away from the model, and each shift picked up a name.

• Prompt engineering → The words you send.

• Context engineering → Everything the model sees, not just your instructions.

• Harness engineering → The code around the model that runs tools, tracks state, and handles errors.

• Loop engineering → The autonomous cycle that drives one agent toward a goal.

• Graph engineering → The coordination layer across many loops, covering what runs when, in what order, and who checks whom.

Each layer wraps the one before it:

A graph is made of loops, each loop needs a good harness, each harness call is a context problem, and every context contains prompts.

If you skip a lower layer, the graph just fails in a more elaborate way.

Of course, none of this is new technology.

LangGraph shipped this exact model involving nodes and edges over shared state back in January 2024.

Microsoft’s AutoGen has GraphFlow, and Google built ADK 2.0’s entire workflow runtime on the same idea.

So while the name is new, the practice isn’t. In fact, the discipline isn’t inventing graphs but rather knowing when to use one, and how to keep it from rotting.

That part has four hard problems.

Hard part 1: knowing when a node deserves to exist

The most common failure is turning “summarize this PDF” into a five-node graph with a fetcher, a chunker, a summarizer, a reviewer, and a formatter.

A node earns its place only if it represents a real specialty, meaning a different model, a different toolset, or a genuinely separate role like a read-only reviewer. Steps you could inline into an existing loop are not nodes.

A useful filter is that if you can’t draw the graph on a napkin, it’s too complex. And if collapsing two nodes into one loses nothing, they were never two nodes.

Hard part 2: keeping shared state clean

In a loop, the failure mode is context rot.

In a graph, the same problem moves into a shared state.

Every node writes to the state object, so an uninformed write in node two will become a confident input for node five.

Nobody notices until the output is wrong, and by then, the bad data has flowed through half the system.

The solutions are simple and boring, yet effective:

• Give the state a typed schema.

• Decide explicitly which nodes may write to which fields.

• Checkpoint state between nodes so you can replay a run and see exactly where it went bad.

One caution applies to replays though.

Nodes after a checkpoint execute again, so any node with external side effects, like sending an email or creating a record, must be safe to run twice.

Hard part 3: routing you can trust

An edge is a decision, and the question is who makes it.

If a model decides the route, you get flexibility and instability together.

The same state can take different paths on different runs, which makes debugging miserable.

Google’s design rule for ADK 2.0 is the cleanest position in the discourse. Deterministic code should control predictable routing, and models should only handle the steps that need actual judgment.

Route with code wherever the condition is checkable, and spend model calls only where interpretation is genuinely required.

Hard part 4: agents agreeing with each other

Loop engineering’s sharpest rule was to never let an agent grade its own homework.

Graphs raise these stakes.

If you have 20 agents built on the same base model and all are reading the same flawed context, then they will happily agree with each other, and models measurably prefer their own outputs.

You can fix this with a reviewer node that runs on a different model, give it fresh context instead of the full conversation, and anchor its verdict to evidence the graph can’t fabricate, like tests that actually ran or code that actually compiled.

Note: Cognition landed in the same place after a year of running Devin, their coding agent. Their working setup lets several agents read the work and weigh in, but only one agent is ever allowed to change anything.

Reading is safe to do in parallel, because a bad opinion costs you nothing until someone acts on it. Writing is where the damage happens, so you keep it in one place where you can see it.

Where the graph is an overkill

Most of the time.

Anthropic’s own numbers suggest this.

A single agent burns roughly 4x the tokens of a chat interaction, and multi-agent systems burn roughly 15x.

Every node you add multiplies that.

Anthropic’s multi-agent research system outperformed a single Opus agent by 90.2% on their internal research eval, because research fans out into independent searches naturally.

But their standing advice from Building Effective Agents hasn’t changed. Find the simplest solution possible, and only add complexity when the task demands it.

Even LangGraph’s own guidance says that if your agent is a straightforward loop with tools, LangGraph is overkill.

The decision rule is pretty straightforward: Reach for a graph when the work splits into genuine specialties, needs parallel fan-out and join, needs different models per step, or needs failure isolation and auditable routing. Otherwise, stay in the loop.

Where to start

You don’t need an org chart of agents on day one. Build up to it.

• Master a single loop first, with brakes, a real completion check, and a critic. A graph of weak loops is just distributed failure.

• Draw the graph on paper before writing code, and challenge every node to justify its existence.

• Define the state schema and write access up front. State drift is the main way graphs rot.

• Make the reviewer node a different model with fresh context, and anchor it to external evidence.

• Put budget caps on every node. A graph has many loops spending tokens in parallel, and a weak verifier now burns money concurrently.

Key takeaway

Graph engineering isn’t a new discipline replacing loop engineering. It’s the name that stuck for a decision every agent builder eventually faces. When one loop stops being enough, coordination becomes the engineering.

The word may not survive the year. The design question will.

Here’s a summary of key takeaways in graph engineering.

​The anatomy of diffusion LLMs​

We recently published a deep dive that covers one of the most important architectural shifts happening in language modeling right now: diffusion LLMs.

​Read the full Part 1 deep dive here →​

​Read the full Part 2 deep dive here →​

It builds a complete understanding from first principles:

• how autoregressive generation is structurally memory-bandwidth bound)

• why Gaussian noise can’t work on discrete tokens

• how masked diffusion solves this with an ELBO-derived training objective

• the math behind the forward and reverse processes

• unmasking strategies

• block diffusion for KV cache compatibility

• a detailed engineering comparison between the two paradigms.

• the training techniques that scaled dLLMs from 8B to 100B parameters (including converting pre-trained autoregressive models like LLaMA into diffusion models via attention mask annealing)

• the inference acceleration stack (block-wise KV caching with Fast-dLLM, confidence-aware parallel decoding, token editing with LLaDA 2.1),

• production serving with SGLang

• hands-on code for running Dream 7B and serving LLaDA 2.0,

• and a decision framework for when dLLMs actually make sense over autoregressive models.

​Read the full Part 1 deep dive here →​

​Read the full Part 2 deep dive here →​

Why care?

Every production LLM today, GPT-4, Claude, Gemini, LLaMA, generates text the same way: one token at a time, left to right.

Each token requires loading the full model weights through GPU memory, performing a tiny computation, and then loading all the weights again for the next token. On an A100, this means roughly 1 FLOP per byte of data moved, while the GPU is designed for 100+ FLOPs per byte.

​Diffusion LLMs​ take a completely different approach. They start with a fully masked sequence and iteratively unmask all tokens in parallel, using bidirectional attention at every step. This shifts inference from memory-bandwidth bound to compute-bound, which is exactly where modern GPUs are efficient.

The results are catching up fast. Block diffusion (BD3-LM) is within 0.5 perplexity points of autoregressive on LM1B. LLaDA at 8B parameters matches LLaMA 3 on MMLU and exceeds it on TruthfulQA and HumanEval. And models like Dream 7B are already being served in production with SGLang.

Understanding how it works at a mathematical level, from the forward masking process to the ELBO objective to block-level KV caching, is going to be increasingly valuable as these models scale.

​You can read Part 1 here →​

​You can read Part 2 here →​

👉 Over to you: Do you think the future of LLM generation is pure diffusion, pure autoregressive, or some hybrid of the two?

Good day!

Kimi K3 leans on a new mechanism called delta attention that does not keep a growing KV cache.

That is how it holds a million tokens of context without the memory blowing up.

Before we can understand delta attention, we need to understand attention itself.

It is a lookup. Every token stores a key, which works like an address, and a value, which is the content at that address.

To build its output, a token sends out a query, matches it against every key in the sequence, and pulls back a blend of the values whose keys matched.

Standard attention keeps every one of those key and value pairs as a list, one entry per token.

That list is the KV cache, and it grows with the sequence, so each new token has to scan the whole thing to build its output.

Delta attention keeps the lookup process but throws away the list.

The entire past collapses into one fixed-size matrix that still behaves like the lookup table.

When you hand it a key, it returns the value vector, blended from everything the past tokens wrote, and it’s weighted by how closely each stored key matches the one you handed in.

Writing is a constrained operation. A fixed matrix has no free slot to append to, so every write lands in a direction that earlier writes already occupy and merges the two associations together.

The delta rule handles this in two moves per token, depicted below:

→ It reads before it writes. It hands the matrix the new token’s key and sees what value the memory currently returns, its existing guess for that address.

→ It writes the difference, not the value. It compares that guess against the value it actually wants stored and writes only the gap. That gap is the delta, and it corrects the old association instead of stacking a new one on top.

The matrix also lets old entries fade over time, so a fixed size can keep absorbing a long sequence without filling up.

Standard attention remembers by keeping everything and pays a quadratic cost to re-scan it.

Delta attention remembers by rewriting one matrix and pays a linear cost.

Of course, a compressed matrix cannot store every token exactly, so recall of any single token becomes approximate. That is why production models interleave the two, i.e., a few full-attention layers for exact lookup and the rest running linear.

Read more in the official announcement blog here →

11 LLM evaluation methods

A model can answer a question correctly and still score close to zero on BLEU.

That happens whenever the model says the right thing in different words than the reference. BLEU compares n-gram overlap, so a clean paraphrase reads as a total miss.

This one failure is why LLM evaluation is fragmented into several methods, depicted below:

Each method encodes a different assumption about what a correct output looks like, and different metrics are useful in different use cases.

Let’s understand the 11 must-know LLM evaluation metrics below

1) BLEU

Splits output and reference into n-grams and measures how much of the output is supported by the reference, with a brevity penalty so short outputs cannot win by omission.

It works for translation and constrained generation where the reference wording is close to the only acceptable wording. Report corpus-level BLEU instead of averaging sentence scores, and supply multiple references when more than one phrasing is correct.

2) ROUGE

Flips the direction of BLUE and measures recall, meaning how much of the reference is covered by the output.

It is the default for summarization and extraction, where missing content hurts more than extra content. ROUGE-L uses longest common subsequence and tolerates reordering better than ROUGE-2. Recall alone rewards verbosity, so it needs a precision-side metric next to it.

3) BERTScore

Embeds both texts, matches each output token to its most similar reference token, and reports precision, recall, and F1 over those similarities.

It handles the paraphrase case that causes problems in BLEU and ROUGE. Absolute values lie within a narrow high band, so scores are only meaningful when compared across systems, not read as accuracy. Rescaling the baseline helps, and the embedding model has to match the domain and language.

4) G-Eval

It hands a task description and criteria to a judge model to generate chain-of-thought evaluation steps, then scores against those steps with the result weighted by token probabilities.

It fits subjective criteria with no reference answer, like tone, instruction following, or domain correctness. Criteria written as a concrete checklist score far more consistently than criteria written as adjectives.

Ideally, it is recommended to run the judge at a low temperature and calibrate against a small human-labeled set before trusting it.

5) LLM-as-Judge

The pairwise form feeds two or more outputs to a judge with a rubric, the judge picks the better one:

It is the practical way to compare two models, two prompts, or two retrieval configs when absolute scores drift between runs.

Make sure to randomize which output appears first, because judges might favor a position. You also need to control for length, since judges reliably prefer longer answers regardless of quality.

6) Human eval

Annotators score outputs across defined dimensions, and the aggregate becomes the reference that every automated metric gets calibrated against.

It belongs in two places, building the calibration set that validates a judge, and the final gate before release.

You must define the rubric before annotation starts and measure inter-annotator agreement. Low agreement means the rubric is ambiguous, not that the annotators are wrong.

7) LLM juries

Runs several judges independently over the same output and averages their scores into one verdict.

It exists because a single judge carries a stable bias, including a preference for outputs from its own model family.

Use different model families rather than the same model three times, since correlated judges average away noise but not bias. Several small models in a jury often beat one large judge at a lower cost.

8) DAG

Scores through a decision tree where each node asks one narrow question and the answer routes to the next node, with the final leaf carrying the score.

It fits rubrics with hard requirements and ordering, like format compliance, required sections, or a mandatory disclaimer.

Because branching is deterministic, the same output always gets the same score. Always put cheap deterministic checks near the root so failing outputs exit before any model call.

9) Trajectory accuracy

Captures the agent’s full sequence of thoughts, tool calls and observations, then scores that path against the expected one.

It matters because an agent can reach the right answer through the wrong path, burning tokens on redundant calls or touching a tool it should not have.

Score path and outcome separately, otherwise a lucky correct answer hides a broken execution path. This needs tracing in place first, since there is no trajectory to score without it.

10) Multi-turn eval

Treats the whole conversation as the unit and scores role adherence, knowledge retention across turns, and coherence.

Per-turn scoring misses the failures that only appear over time, like contradicting an earlier answer or dropping a constraint set five turns ago.

Run it on real conversation logs rather than synthetic two-turn exchanges, because rule retention failures show up at depth.

11) Safety eval

Runs bias, toxicity and PII classifiers in parallel over the output and flags violations instead of producing a quality score.

It belongs as a gate, not as a term inside an average. A single PII leak matters regardless of how good the mean quality score looks, which is how folding safety into an aggregate ends up shipping violations.

To use them in practice, most of these already ship as built-in metrics in Comet Opik, an open-source LLM evaluation and observability platform, including BLEU, ROUGE, BERTScore, G-Eval, LLM juries, trajectory accuracy, conversation-level metrics, and moderation, all running over traced production data.

Here’s the repo → https://github.com/comet-ml/opik

(don’t forget to star it ⭐️)

Also, to dive deeper in the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to productions:

• Read Part 1 on fundamentals of LLMOps here →

• Read Part 2 on understanding the core building blocks of LLMs →

• Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →

• Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →

• Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →

• Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →

• Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →

• Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →

• Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →

• Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →

• Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →

• Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →

• Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →

• Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions.

👉 Over to you: What else would you add to the master tree?

Good day!

Long-running agent flows fail mid-run, and in most frameworks, that means restarting from zero and burning the tokens again.

​CrewAI just shipped checkpointing in v1.14. Every flow method becomes a recovery point, written automatically when an event you specify fires (e.g., method_execution_finished).

You can resume in one line and fork from any saved state into a new branch with full lineage tracking. An async TUI lets you browse checkpoints, inspect events, and resume or fork from the UI.

The pipelines become resumable, inspectable, and branchable processes, with zero extra infra.

​You can find the CrewAI repo here →​

5 techniques to optimize LLMs in production

A tricky LLM interview question:

You’re serving an LLM in FP16 on an A100.

Token generation is quite slow, so you upgrade to an H100, which has over 3x the FP16 compute.

But upgrading the GPU didn’t improve the token generation speed much.

Why did this happen?

LLM token generation is a memory bandwidth problem disguised as a compute problem.

An A100 can do about 300 trillion FP16 operations per second. But it can only move ~2 TB per second between HBM and its compute units.

During generation, every new token requires reading the model weights and the entire KV cache from memory. While the computation process finishes almost instantly, the chip still has to spend time waiting for data.

This is why the highest-impact serving optimizations all try to tackle the same two things:

• how much data moves

• and how often the GPU sits idle

The visual below covers five of them.

1) Flash attention:

Standard attention writes the full attention matrix in HBM, and it grows quadratically with sequence length.

Flash Attention splits Q, K, and V into tiles, computes attention inside on-chip SRAM, and writes only the final output back.

The output is mathematically identical, but the quadratic intermediate never exists in memory, so long sequences become relatively manageable.

2) Paged attention:

Naive serving reserves one contiguous memory chunk per request, sized for the maximum possible sequence length.

vLLM paper measured 60-80% of KV cache memory wasted this way.

Paged attention is inspired by virtual memory from OS concepts.

The cache stays in small fixed-size blocks mapped through a block table, allocated as generation proceeds and freed on completion, which reduces waste to under 4%.

3) Continuous batching:

Static batching waits for the slowest request in a batch before admitting anything new. So one long generation keeps every finished slot idle until it completes.

Continuous batching makes scheduling decisions after every decode step instead of after every batch.

When a request finishes, a queued request takes its slot at the next decode step, and the GPU stays saturated.

4) Speculative decoding:

A small draft model proposes K tokens cheaply, then the large target model verifies all K in a single forward pass, accepting matches and correcting the first mismatch.

The rejection sampling scheme guarantees an output distribution identical to running the large model alone, so the speedup costs zero quality.

5) Kernel fusion:

Every GPU operation runs as its own kernel by default, and each one writes its result to HBM for the next kernel to read back.

Kernel fusion merges LayerNorm, MatMul, and activations into one kernel that keeps intermediates in registers and SRAM.

Compilers like TensorRT-LLM and torch.compile do this automatically from the model graph.

That said, all five techniques above are built for one model owning the whole GPU.

Paged attention and continuous batching fit more concurrent requests when more memory is available for KV cache blocks.

So vLLM pre-allocates nearly all GPU memory at startup and turns it into one big block pool.

But production stacks rarely run just one model.

A typical pipeline also runs several small specialized models like embedders, rerankers, and extractors alongside the LLM.

Serving each one the standard way requires a separate vLLM or TEI process, and every process claims its memory upfront with no visibility into the others.

So a 600M-parameter embedder reserves a GPU it will never fully use, and models that could comfortably share one card end up spread across many.

This is also why moving to small yet better models does not lead to substantial cost savings alone.

Superlinked’s open-source inference engine (SIE) solves this by running all of those models behind one API with shared GPU memory and on-demand model loading, so the card is actually shared instead of claimed.

GitHub repo → https://github.com/superlinked/sie

(don’t forget to star it ⭐ )

We wrote a full breakdown of this small-model serving problem and how SIE solves it.

It covers why switching to small specialized models doesn’t reduce inference costs unless those models share GPUs, and why vLLM’s design makes that impossible.

You can read it here →

Good day!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

“You are still responsible for your software, just as before. You are not allowed to introduce vulnerabilities because of vibe coding. “

He said it while drawing the line between vibe coding and agentic engineering. Agents write more of the code now, but none of that takes the responsibility off you.

The assumption underneath that is that a careful enough reader catches the problem. But some failures don’t show up in anything there is to read.

For instance, a common fear with a RAG agent is that it could hallucinate when a question asks something outside its corpus.

But such cases are actually well handled by any competent model now. If nothing in the retrieved context looks relevant, there’s no material to build an answer on.

Instead, the majority of failures originate when the retrieved context has partial coverage.

The retrieval pipeline returns context that’s topically correct but doesn’t cover the full question, and the model completes the remainder from parametric knowledge.

There are no token-level labels in the output to tell what was generated using retrieved context and what came from weights.

Both are streamed the same way.

Detecting this for production-grade apps needs a metric written for it, one that’s also aligned with principles of agentic engineering.

And the solution is actually implemented in the eval skill that comes with Google’s Agents CLI.

We described the concern to Claude Code in plain English. It read the agent’s code, came back with a plan I approved.

It then reported that no built-in metric isolates the behaviour and wrote a custom rubric called corpus_abstention.

It assigned a single categorical verdict per case rather than aggregating everything into one score, since the built-in raters regenerate their rubrics each run and leave no stable number to trend.

• GROUNDED_ANSWER

• CORRECT_ABSTENTION

• UNGROUNDED_ANSWER (answered entirely from outside knowledge)

• MIXED_LEAKAGE (grounded, but slips in one unsupported claim)

• WRONG_ABSTENTION (refused something the docs actually covered)

After this, it automatically generated 33 scenarios partitioned by where the failure could occur, like:

• in-corpus

• off-domain

• out-of-corpus but plausibly answerable

• boundary cases where the topic is covered, but a specific detail isn’t.

The baseline score was 19 of 33.

• Off-domain passed 3 of 3, as expected.

• But 6 of 15 in-corpus cases retrieved the right document, cited it correctly, answered accurately, and added a claim the source never made.

The root cause was one line in the agent’s instruction: “If you already know the answer to a simple question and no document lookup is needed, you may respond directly without citations.”

The eval skill helped flag this, and then Claude removed it and forced retrieval on every question.

This took the suite to 30 of 33, and ungrounded answers went from 6 to 0. The full recording of our run is below, and we worked with the Google Cloud team on this.

Here’s the Agents CLI GitHub repo → https://github.com/google/agents-cli

(don’t forget to star it ⭐ )

Are we ready for an agent-native memory system?

First, what does that even mean?

It’s memory built for an agent that keeps running, not a store you bolt on at the end. It has to hold facts across sessions, update them when they change, drop the stale ones, and hand back the right piece when the agent asks. A real data system, not a pile of saved chunks.

A new study checked whether anything out there actually does this, and what decides the answer surprised me.

It takes 12 agent memory systems and splits each into four stages, covering how memory is represented, extracted, retrieved, and maintained. Then it checks which stage decides whether the agent stays correct.

Turns out it’s write time, not query time.

The systems that stay reliable under fact updates and cross-session reasoning are the ones that add structure as the memory is written. The graph-based methods lead right there.

The flat stores fail the same way every time. An LLM pulls out entities and relationships with no schema, and the graph fills up with generic nodes that nothing downstream can filter.

A few things follow from this.

• Structure has to be set at extraction. Once the graph is built from untyped nodes, no query can recover a distinction the extractor never made.

• Updates return stale facts. Append-only and freeform stores keep handing back old versions, which the paper calls hallucinations of the past.

• A bigger model does not save it. Swapping the backbone moves the scores but rarely changes which pipeline wins, so the memory’s behavior is set before generation runs.

The paper also weighs the cost. The heavily structured systems pay for it, sometimes orders of magnitude more time per query, and the win only holds when the upkeep stays local instead of rewriting the whole store on every change.

So the takeaway is less about picking a winner and more about where the structure lives. Decide it at write time, keep the maintenance cheap, and retrieval has something real to rank.

Retrieval only ranks what extraction chose to keep.

Link to the paper: https://arxiv.org/pdf/2606.24775

If you’re into agent memory systems, we wrote a full article on exactly this.

It walks through defining the schema as a Pydantic ontology in Zep, with custom entity and edge types, source and target constraints, and the extraction pipeline that turns raw conversations into typed, queryable memory.

This memory system is 100% open-source: https://github.com/getzep/graphiti

You can read it here →

Good day!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Every dependency in a modern codebase shows up in a manifest, except the AI ones.

Open-source models, LLM APIs, MCP servers, and agents enter production through the same pull requests as ordinary packages, but no lockfile lists them and no dependency scanner sees them.

Checkmarx found that 70% of teams expect AI components in production, while 43% have zero governance over them.

Checkmarx AI Inventory (part of Checkmarx One) catalogs every model, SDK, and MCP server deterministically, traces each one to the exact file and line, and exports an AI-BOM for compliance audits.

Gartner named Checkmarx a Leader in its inaugural Magic Quadrant for Software Supply Chain Security.

Learn more about Checkmarx AI Inventory here →

Thanks to Checkmarx for partnering today!

5 LLM Quantization Techniques

A 70B parameter model in FP16 requires 140GB of memory for weights alone.

That exceeds any single GPU available today:

• An H100 carries 80GB

• An RTX 4090 carries 24GB

So serving the model in half precision means multi-GPU setups and tensor parallelism before the first token is generated.

Quantization reduces this footprint by storing weights in lower bits.

The standard scheme maps each weight from FP16 onto a small integer grid: compute the dynamic range of the tensor, divide it into 2^b levels (16 levels at 4-bit), round each weight to the nearest level, and store a scale factor to dequantize during inference.

At 4-bit, the weight memory drops 4x. The 140GB model fits in 35GB, within the range of a single 40GB or 48GB card.

That said, if rounding were lossless in practice, one method would suffice, and the diagram below would have had one row:

But it has five because weights and activations in large transformers are not uniformly important.

The LLM.int8() paper quantified this. Beyond roughly 6.7B parameters, every transformer layer develops a handful of hidden dimensions carrying values 20x to 100x larger than the rest.

These dimensions make up about 0.1% of the model’s features, but zeroing them out nearly destroys the model’s ability to predict text.

These outliers are also what break naive rounding. The quantization grid spans the range of the tensor, so a single activation of magnitude 60 among values near 1 inflates the scale factor and collapses the remaining values into a few levels, destroying their precision.

Each method in the visual shown earlier solves this saliency problem differently.

1) RTN (round to nearest) does not handle it at all. It applies the scale-and-round procedure directly with no calibration data, which makes it the cheapest baseline and the weakest at low bit widths, where accumulated rounding error is unrecoverable.

2) GPTQ corrects rounding damage as it happens. It quantizes the weights of a layer a few at a time, measures how much error the rounding just introduced, and nudges the not-yet-quantized weights to cancel out that error before moving on.

The adjustment is not uniform. GPTQ uses statistics from calibration data to work out which remaining weights can absorb the error with the least effect on the layer’s output, which is what separates it from blind rounding.

This runs fast enough to matter in practice. A 175B model quantizes to 4-bit in around four GPU hours. The known weakness is that fitting the calibration data this closely can tune the model to it, and the AWQ authors showed this hurts accuracy on inputs that look different from the calibration set.

3) AWQ protects the important weights before rounding instead of repairing damage afterward. It runs calibration samples through the model and watches which weights get multiplied by the largest incoming values, because a rounding error on those weights gets amplified the most in the layer’s output.

Only around 1% of weight channels turn out to matter this much. Rather than storing them at higher precision, AWQ multiplies them by a scaling factor before rounding, so they spread across more levels of the grid and lose less detail, and the math is rearranged so a matching inverse scale elsewhere keeps the layer’s output unchanged.

Every weight still ends up in plain INT4, with no special-case handling at inference time. AWQ also depends far less on which calibration samples were used than GPTQ does, which is a big part of why it became the standard choice in serving engines like vLLM.

4) LLM.int8() isolates the outliers instead of fighting them. When the model is loaded, it finds the dimensions carrying extreme values and splits every matrix multiplication into two:

• The outliers run in full FP16

• The other 99.9% of values run in INT8, and the two results are added back together.

Nothing needs to be computed ahead of time. That is why it is also available as the load_in_8bit flag in bitsandbytes.

The cost is speed. Splitting and merging every matrix multiplication is slower than running one optimized 4-bit kernel, so this path is common for local development and QLoRA fine-tuning, not for serving traffic at scale.

5) QAT modifies training rather than the quantization procedure. During a short fine-tuning run, every forward pass rounds the weights to INT4 before using them, so the model experiences the exact damage that quantization will cause, and its predictions are scored under that damage.

The weight updates still happen in full precision behind the scenes, since rounding itself cannot be trained through directly. Over the fine-tuning run, the weights drift toward values that keep the loss low even after rounding, so when the real quantization is applied at the end, there is almost nothing left to break.

Google produced its Gemma 3 QAT checkpoints with roughly 5,000 fine-tuning steps guided by the original model’s outputs, and the int4 versions hold close to full-precision quality at about 3x lower memory.

All five methods described above produce the same artifact, i.e., a model at a fraction of its trained precision.

They differ in where the outlier problem gets addressed, at rounding time, after rounding, before rounding, at inference, or during training itself.

To dive deeper, we have already written a full deep dive on Quantization, specifically, which covers several of these methods with their simplified mathematics.

Learn how Quantization optimizes LLMs to run them on tiny hardware here →

Good day!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Part 13 of the Reinforcement Learning course is available now.

Here, we look at what actual AI labs and companies are actually doing with everything we learned across twelve chapters.

Read Part 13 of the RL course here →

RL Nanodegree Part 13

It covers four case studies, each showing a different role RL plays in the industry:

• Cursor’s real-time RL loop that ships improved checkpoints every five hours

• How frontier labs turned verifiable rewards into reasoning and agentic capabilities

• Scale AI’s enterprise results, where a 4B model beat GPT-5 on domain-specific tasks

• The emerging market where RL environments are built, priced, and sold as products

Read Part 13 of the RL course here →

RL Nanodegree Part 13

Why care?

Across all twelve chapters of this series, we built the full mechanics of RL piece by piece.

We covered bandits, value functions, Bellman equations, policy gradients, PPO, GRPO, judges, trajectories, and environments.

One real test of understanding RL is whether you can read a company’s blog post about their training pipeline and immediately identify the environment, the reward source, the trajectory structure, and the failure modes they are likely facing.

That is what this chapter exercises.

We cover stories of very different companies, each using RL in a way that maps directly onto concepts from this series.

The vocabulary you built is the same vocabulary these teams operate in, like reward hacking, credit assignment, environment design, and the tension between learned and verifiable rewards.

This is the final chapter, and this is what we have covered in the full course:

• Part 1: Foundations of Reinforcement Learning →

• Part 2: MDPs and Value Functions →

• Part 3: Bellman Equations and Dynamic Programming →

• Part 4: Model-Free Learning →

• Part 5: Function Approximation →

• Part 6: Deep RL and DQN →

• Part 7: Policy Gradients and Actor-Critic →

• Part 8: Proximal Policy Optimization →

• Part 9: RLHF →

• Part 10: Verifiable Rewards and GRPO →

• Part 11: Reward signal problem for Agents →

• Part 12: Environments, trajectories, and the training Loop →

• Part 13: How AI teams use RL in production →

👉 Over to you: What topics would you like us to cover in this RL series?

LLM Routing vs Mixture of Agents

When you’re building a system that needs to handle diverse queries, two architectural patterns are considered.

One picks the best model for each task, and the other gets multiple models to collaborate on the same task.

Both involve multiple LLMs, but what they do with those models is completely different.

The visual below explains how each one works:

In LLM routing, an intent classifier reads the prompt, and a cost ranker weighs latency, token price, and capability, so only one model gets the call while everything else stays idle.

• The prompt is classified by domain and action.

• A cost ranker picks the best model given latency and price constraints.

• One model responds, and the rest never see the query.

In Mixture of Agents, a set of consulting models each analyze the query independently and hands their output to a single aggregator, which uses every perspective to write the final answer and make the tool calls.

The consulting models never see each other’s analysis, only the aggregator does.

On HermesBench, Nous Research reports a MoA preset scored higher than either model alone by roughly six points and eight to eleven percent, and no individual model in that setup reached the same result on its own.

LLM routing and MoA are not competing approaches but rather two different tools for two different problems.

Routing is about picking the right model for the task, while MoA is about combining multiple models to exceed what any one can do alone.

We wrote a full deep dive on LLM routing with Plano, including a working demo, in a recent article.

Here’s the Plano GitHub repo: https://github.com/katanemo/plano (Don’t forget to star it ⭐️)

And here’s the article we wrote →

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Anyone can write that loop in thirty lines. But a bare version of it falls apart on a real codebase, reading the wrong files, losing the goal midway, and filling its context with output it no longer needs, while Claude Code finishes the same task cleanly.

The easy conclusion is that Anthropic simply has a better model. The actual difference is the harness, the ordinary code wrapped around the model that handles planning, tool execution, memory, and safety, while the model only decides the next step.

Here’s what a fully harnessed agent looks like when you draw it out:

• Memory feeds the model its working context plus facts it has learned across sessions.

• Skills encode how the agent should operate, meaning the procedures, constraints, and heuristics it follows.

• Protocols connect the agent to users, tools, and other agents.

• The harness core ties it all together with sub-agent orchestration, a sandbox, an evaluator, an approval loop, observability, and context compression.

Anthropic describes this split as the brain and the hands. The model is the brain that picks each action, and the harness is the hands that carry it out and keep the run on track.

So the gap between your agent and Claude Code isn’t the model, it’s the machinery around the model.

Claude Code is one of the most capable harnesses in production today, and it’s built from a surprisingly small set of layers in that illustration. To see how much of that machinery you’d have to build yourself, I rebuilt it in CrewAI, an open-source framework for orchestrating agents.

More of it maps onto built-in features than I expected, and the part that doesn’t is where the real engineering lives.

Let’s build it layer by layer, starting with the core loop and stacking planning, subagents, sandboxing, and memory on top. At each step, we’ll mark where the framework ends and where your job begins.

How Claude Code's harness works

At the center of Claude Code is a plain agent loop. You send it a message, the model decides what to do next, and it either responds directly or requests a tool. If it requests one, the tool runs, the result goes back into the conversation, and the model decides again.

This repeats until the model returns a final answer with no further tool calls.

Inside that loop, the model reads files, edits code, runs shell commands, and executes tests. These aren’t separate modes. They’re just different tool calls inside the same loop.

The loop alone isn’t enough for a reliable coding agent, though. Claude Code adds planning, file tools, subagents, memory, and a permission and sandbox system around it. These layers don’t replace the loop, they make it safe and reliable enough for real work.

That’s the architecture we’ll rebuild, the core loop first, then each layer on top, mapping every layer to the CrewAI feature that handles it.

The core agent loop

The loop runs the same sequence until the task is done:

1. Ask the model to perform the task.

2. The model responds directly or requests one or more tools.

3. If tools are requested, run them and return the results to the model.

4. Repeat with the updated conversation.

5. When the model responds without requesting any tools, the task is complete.

while True:
    reply = model(messages, tools)
    calls = [b for b in reply if b.type == "tool_use"]
    if not calls:            # plain text, no tool call: the job is done
        return reply.text
    messages += [reply, run_all(calls)]

Each tool call completes one step, gives the model new information, and feeds into the next decision. A simple question might finish in one iteration, while fixing a complex bug or refactoring a large codebase can take dozens of iterations before the model has enough to produce a final answer.

CrewAI provides this execution loop automatically as soon as you create an agent. You don’t implement the while loop yourself, you define the agent and assign it a task.

Building the first agent

Let’s create a simple Bug Fixer agent.

from crewai import LLM, Agent, Crew, Task

bug_fixer = Agent(
    role="Bug Fixer",
    goal="Find and describe the fix for the reported bug in the codebase.",
    backstory="You read directories and files to build an accurate picture of the code.",
    llm="claude-sonnet-4-6",
)

task = Task(
    description="Find the fix for {objective}.",
    expected_output="A short description of the fix and which file it belongs in.",
)

result = Crew(agents=[bug_fixer], tasks=[task]).kickoff(
    inputs={"objective": "the overdraft bug in account.py"}
)

Three concepts to understand here:

• An Agent defines who does the work, through its role, goal, LLM, and tools.

• A Task describes the assignment.

• A Crew brings agents and tasks together. Calling kickoff() runs the same execution loop described above, regardless of whether the underlying model is Anthropic, OpenAI, Google, or something else.

Giving the agent tools

Tools are what let a model that only generates text actually work on a codebase. They read files, write them, run shell commands, and call external APIs.

CrewAI ships file system tools out of the box:

• FileReadTool reads files.

• DirectoryReadTool lists directories.

• FileWriterTool writes files.

from crewai_tools import DirectoryReadTool, FileReadTool, FileWriterTool

read_file = FileReadTool()
write_file = FileWriterTool()
list_dir = DirectoryReadTool()

filesystem_tools = [read_file, write_file, list_dir]

These also double as external memory. Instead of holding a large search result in the model’s context window, the agent can write it to a file, keep only the filename, and read it back when needed.

This keeps the context window smaller and the model more focused.

Built-in tools cover common workflows only. For anything more specific, you expose a Python function as a tool with the @tool decorator.

The docstring acts as the instruction manual, telling the model what the tool does, when to use it, and what it expects as input.

from crewai.tools import tool
import subprocess

@tool("run_tests")
def run_tests(path: str = "tests/") -> str:
    """Run the pytest suite at the given path and return the result."""
    result = subprocess.run(
        ["pytest", path, "-q"], capture_output=True, text=True, timeout=120
    )
    output = result.stdout + result.stderr
    return output[-4000:] if len(output) > 4000 else output

Adding planning capability

As tasks get more complex, a plain execution loop starts losing track of the original goal. After enough tool calls, file reads, and intermediate results, the context fills up and the objective gets crowded out by everything that came after it.

This slow degradation is what people call context rot.

Planning addresses it directly. The agent builds a step-by-step plan before doing any work and keeps that plan in context throughout execution.

The plan doesn’t do the work. It’s a roadmap that keeps the model connected to the original goal, which is the same job Claude Code’s to-do list does.

CrewAI adds this at the crew level with planning=True. It generates a plan before execution and keeps it available as the task progresses.

from crewai import Crew, LLM

crew = Crew(
    agents=self.agents,
    tasks=self.tasks,
    planning=True,
    planning_llm=LLM(model="gpt-4o-mini"),
)

Note: By default, CrewAI uses gpt-4o-mini for planning, and you can swap in any LLM you prefer for that step.

Individual agents can also reason about their own work with reasoning=True:

from crewai import Agent

bug_fixer = Agent(
    role="Bug Fixer",
    goal="Find and describe the fix for the reported bug in the codebase.",
    backstory="You read directories and files to build an accurate picture of the code.",
    tools=[FileReadTool()],
    reasoning=True,
    max_reasoning_attempts=3  # Optional: Set a maximum number of reasoning attempts
)

Planning and reasoning solve different problems. Planning builds a high-level roadmap for the overall task, while reasoning gives one agent time to think through its own approach before acting.

When reasoning is enabled, the agent:

1. Reflects on the task and drafts an execution plan.

2. Evaluates whether the plan is ready.

3. Refines the plan if necessary, until it’s satisfied or hits max_reasoning_attempts.

4. Injects the finalized reasoning plan into the task before execution.

Together, they keep the agent anchored on long-running tasks and reduce drift from the original goal.

Subagents to maintain main agent’s context

Planning keeps the agent focused, but it doesn’t reduce how much information the model has to hold. On a large codebase, even a well-planned task can exceed a single context window.

Finding one bug may require reading dozens of files, and the main agent doesn’t need to keep all of them in memory.

Subagents solve this through delegation. The main agent hands a specific task to a helper agent, which works in its own context and returns a short summary. The main agent sees the conclusion, not the intermediate steps.

CrewAI supports this through hierarchical workflows, where a manager agent delegates to specialist agents and combines their results.

In our earlier setup, one Bug Fixer agent did all the heavy lifting. Let’s split the work across a manager and three specialists:

• Codebase Explorer explores the code and maps the repository.

• Software Engineer implements the requested change.

• Test Runner runs the tests in the sandbox and reports pass or fail.

• Engineering Lead oversees the three specialists.

from crewai import Crew, Agent, Task, Process

explorer = Agent(
    role="Codebase Explorer",
    goal="Map the repository and surface the files relevant to the task.",
    backstory="You read directories and files to build a picture of the code.",
    tools=[read_file, list_dir],
    llm=llm,
)  # Same for other two specialist agents

manager = Agent(
    role="Engineering Lead",
    goal="Break the request into steps and delegate each to the right specialist.",
    backstory="You decide who does what, review tests, finish once change is done.",
    llm=llm,
    allow_delegation=True,
)

crew = Crew(
    agents=[explorer, coder, tester],
    tasks=[task],
    manager_agent=manager,
    process=Process.hierarchical,
)

One thing to note is that allow_delegation is disabled by default, so it must be explicitly enabled on the manager.

Safety comes from a sandbox and approvals, not from prompts

An agent with shell access can run a destructive command, and telling the model not to do something isn’t a safeguard.

Real protection comes from two layers:

1. A permission system that requires approval for sensitive actions.

2. A sandbox that isolates execution, so even approved commands can’t touch the host machine.

Anthropic uses the same approach. Moving code execution into a sandbox cuts down how often a user needs to approve actions while still protecting the host system.

Sandboxing in CrewAI

Executing code inside a sandbox rather than on the host machine applies that second layer. In this setup, code runs inside E2B, which spins up a fresh VM per session and destroys it afterward.

Shell commands and Python run entirely inside that isolated environment.

from crewai_tools import E2BExecTool, E2BPythonTool
sandbox_tools = [E2BExecTool(), E2BPythonTool()]  # run tests / run code

Human-in-the-loop

Setting human_input=True on a Task pauses the crew after it generates an answer. You review the output, then approve it or send it back for another iteration.

When execution reaches that task, CrewAI waits for your feedback through standard input.

from crewai import Task

task = Task(
    description=(
        "In the working directory ./workspace, {objective}. "
        "Explore the code first, make the change, then run the tests and report."
    ),
    expected_output="A summary of the files changed and the final test output.",
    human_input=True,
)

If your crew runs behind a web app or chat interface rather than a terminal, CrewAI’s webhook-based human-in-the-loop system handles the same review step.

Memory and checkpointing

By default, an agent forgets everything once a run ends. So if you come back tomorrow to fix another bug in the same project, and it starts from zero.

Two mechanisms let an agent carry information across runs, and each serves a different purpose:

• Checkpointing saves the agent’s state during a run, so it can resume after an interruption or continue from the same point along a different path.

• Persistent memory stores facts across separate conversations, including project preferences like “always format the final code before finishing.”

Adding memory

CrewAI provides a unified Memory interface rather than separate short-term, long-term, entity, and external memory types. When saving, it uses an LLM to identify important details, organize them, and make them retrievable later.

Setting memory=True on the crew gives it memory across runs. After each task, CrewAI extracts useful facts from the output and stores them, and in future runs, it retrieves relevant memories and adds them to the task prompt.

from crewai import Crew

crew = Crew(
    agents=[explorer, coder, tester],
    tasks=[task],
    memory=True,
)

All agents in a crew share its memory unless an agent is given its own.

Adding Checkpoints

A checkpoint is a snapshot of an agent’s progress, including its configuration, task state, memory, intermediate results, inputs, and execution history.

By default, CrewAI creates a checkpoint whenever a task finishes, allowing the workflow to resume from that point if it’s interrupted.

Checkpoints can live in one of two built-in stores:

• JsonProvider saves each checkpoint as a separate JSON file, which is easy to read and inspect manually.

• SqliteProvider stores all checkpoints in a single SQLite database, which holds up better under frequent checkpointing and larger workloads.

from crewai import Crew

crew = Crew(
    agents=[explorer, coder, tester],
    tasks=[task],
    checkpoint=True,
)

Crew, Flow, and Agent all accept a checkpoint argument, and children inherit from their parent unless they set their own value.

All six layers on one bug-fix task

Here’s the full harness on one task, with the execution loop, tools, planning, subagents, sandboxing, and memory working together:

from crewai import Agent, Crew, LLM, Process, Task
from crewai.tools import tool
from crewai_tools import (DirectoryReadTool, FileReadTool, FileWriterTool, 
E2BExecTool, E2BPythonTool)

llm = LLM(model="anthropic/claude-sonnet-4.6")

list_dir = DirectoryReadTool(directory="./workspace")
filesystem_tools = [FileReadTool(), FileWriterTool(), list_dir]
sandbox_tools = [exec_tool, E2BPythonTool()]

@tool("run_tests")
def run_tests(path: str = "tests/") -> str:
    """Sync ./workspace into the sandbox, then run pytest there."""
    return E2BExecTool().run(command=sync_and_test_command(path))

explorer = Agent(role="Codebase Explorer", goal="Map repo, surface relevant files.",
    tools=[read_file, list_dir], llm=llm)
coder = Agent(role="Software Engineer", goal="Implement requested change.",
    tools=filesystem_tools, reasoning=True, llm=llm)
tester = Agent(role="Test Runner", goal="Run tests in sandbox, report pass/fail.",
    tools=sandbox_tools + [read_file] + [run_tests], llm=llm)
manager = Agent(role="Engineering Lead", goal="Delegate steps, finish once tests pass.",
    allow_delegation=True, llm=llm)

task = Task(
    description="In ./workspace, {objective}. Explore, edit, test, report.",
    expected_output="Summary of changes and test output.", human_input=True,
)
crew = Crew(
    agents=[explorer, coder, tester], tasks=[task],
    manager_agent=manager, process=Process.hierarchical,
    planning=True, memory=True, checkpoint=True,
)
result = crew.kickoff(inputs={"objective": "fix failing tests in account.py"})

Agent harnesses are easiest to evaluate when success can be checked automatically. A test suite gives the agent a concrete objective, so it can plan, edit, test, and repeat until everything passes.

So this was tested against a small codebase, a BankAccount class with two real bugs and five tests, three of which failed. The rule was to fix only the implementation, not the tests.

This mirrors how Anthropic evaluates coding agents internally. One published example has Claude rebuilding a clone of the claude.ai interface against a large suite of failing tests.

Here, the harness took the project from 3 failing and 2 passing to all 5 passing, with the implementation-only rule closing off the shortcut of editing or removing the failing tests.

Beyond prompts, environment, and tool choice

Some parts of the system aren’t things the framework builds for you:

• The prompts. Each agent’s behavior comes from its role, goal, and backstory. Getting those right takes testing and iteration, and no configuration flag substitutes for it.

• The execution environment. The sandbox, whether E2B or a self-managed VM, has to be set up and wired in.

• Tool selection. Which tools each agent gets, and which agent should have access to what, is a design decision the framework doesn’t make.

There’s also a cost to the harness itself. Planning, subagents, and looping all add API calls, so a complex agent setup can end up more expensive than a task a single model call would have solved directly.

And there’s a longer-term limitation worth keeping in mind. As models improve, some of the scaffolding stops being necessary, because some of what gets built into a harness today is a workaround for today’s model limits rather than a permanent requirement.

Anthropic originally used context resets to keep Claude Sonnet 4.5 from ending tasks too early, and they weren’t needed anymore with the more capable Claude Opus 4.5.

That’s the whole finding. A coding agent’s capability lives mostly in the harness, and an orchestration framework hands you more of that harness than you’d guess.

The loop, planning, delegation, sandboxing, and memory all arrive as configuration, while the prompts, the execution environment, and the tool choices remain yours.

If you want to run this against your own codebase, the CrewAI docs cover every feature used here, and the framework is fully open source.

Check out CrewAI Docs →

Check out CrewAI GitHub (100% Open Source) →

Find the code for this project in this repo →

Good day!

When Claude Opus 4.8 shipped on May 28, it was the most capable model available to the public.

Yesterday, Moonshot AI released Kimi K3, and it beats Opus 4.8 at max effort on four of five major benchmarks: BrowseComp, DeepSWE, GPQA Diamond, FrontierSWE, and HLE with tools.

And this is not just an Opus story. K3 matches Claude Fable 5 and GPT-5.6 Sol on several benchmarks and beats them on some, while its API is priced at roughly a third of Fable 5’s.

K3 is open weights. Moonshot says it will ship all 2.8 trillion parameters on July 27, which would make it the largest open model released so far.

Every closed frontier model is rented intelligence, accessed through an API the provider can reprice, deprecate, or shut off.

Last month proved why rented intelligence comes with risks. A US export directive forced Anthropic to take Fable 5 offline globally for 18 days.

Weights on a disk don’t have that failure mode since they run on private infrastructure, fine-tune on private data, and cannot be switched off by anyone.

Do note that the weights aren’t public yet, so until July 27, this is an API model like any other.

Here’s the official release →

RLHF vs. DPO vs. GRPO in RL

RLHF, DPO and GRPO are often grouped together as variations of the same algorithm.

But they take completely different approaches to teaching a model to behave the way you want, and those differences show up in the training setup, the data you need, and what it costs you at scale.

The visual below summarizes how they work:

Let’s understand this in detail today!

To dive deeper into these topics (with hands-on implementation and understand actual production tradeoffs), we have already covered every bit of it in the Reinforcement Learning Nanodegree:

• Part 1: Foundations of Reinforcement Learning →

• Part 2: MDPs and Value Functions →

• Part 3: Bellman Equations and Dynamic Programming →

• Part 4: Model-Free Learning →

• Part 5: Function Approximation →

• Part 6: Deep RL and DQN →

• Part 7: Policy Gradients and Actor-Critic →

• Part 8: Proximal Policy Optimization →

• Part 9: RLHF →

• Part 10: Verifiable Rewards and GRPO →

• Part 11: Reward signal problem for Agents →

• Part 12: Environments, trajectories, and the training Loop →

RLHF (Reinforcement Learning from Human Feedback) is the simplest formulation, which also went into building ChatGPT.

You sample prompts, run them through the policy, and score the outputs with a reward model. Then you compute advantages, apply a KL penalty to keep the policy close to the reference, and update via PPO.

This requires four live models running simultaneously:

• The policy you’re training

• A frozen reference copy

• A reward model

• And a critic.

This means running four sets of forward passes per training step.

At production scale, this is expensive to run and hard to stabilize. And the reward model needs its own human-preference training before any of this can even begin.

DPO (Direct Preference Optimization) provides a better solution.

Instead of training a separate reward model, DPO derives the reward signal implicitly from the policy itself.

It works from preference pairs, a winner and a loser response to the same prompt. It computes log-probability ratios between the current policy and a frozen reference, and uses that ratio as the implicit reward.

This removes the explicit reward model and the critic entirely, bringing the pipeline down to just policy and frozen reference.

The tradeoff is that you need labeled preference pairs upfront. And since there’s no online exploration, DPO can be brittle if the preference data doesn’t cover the failure modes you care about.

GRPO (Group Relative Policy Optimization), introduced by DeepSeek in 2024, takes a different path.

It keeps the RL framing but eliminates the critic by replacing the value baseline with group statistics.

For each prompt, the model generates a group of outputs, and a verifier scores each one. Advantages are then computed from the mean and standard deviation of those scores within the group.

The critic is a key bottleneck in RLHF since it’s slow to converge, adds instability, and requires a regression head on top of the policy.

GRPO removes all of that by using the group itself as the baseline, keeping the RL loop intact but at lower overhead than PPO.

Here’s a useful way to understand all three at once.

• RLHF uses external scores and an internal critic to measure advantage.

• DPO encodes preference implicitly and skips both.

• GRPO generates its own baseline from a group and keeps the RL loop without the critic.

All three produce aligned models, but they differ on how much machinery the alignment requires.

To dive deeper into these topics (with hands-on implementation and understanding actual production tradeoffs), we have already covered every bit of it in the Reinforcement Learning Nanodegree. Start here →

​Paged Attention in LLMs​

When you’re serving LLMs at scale, memory becomes your bottleneck before compute does.

The problem isn’t a lack of GPU memory; it’s how inefficiently we use it.

Let’s understand why this happens and how Paged Attention solves it elegantly!

KV Cache

During LLM inference, the model stores the key and value vectors of all previously generated tokens in memory. This is the KV cache (​we covered KV caching in detail here​).

Simply put, instead of recomputing attention over all previous tokens every time a new token is generated, the model just looks up the cached values and computes attention with them.

This is crucial for efficiency. Without it, generating a 1000-token response would require recomputing attention for all previous tokens at each step. That’s O(n²) operations instead of O(n) and the speed difference is evident below:

But there’s a problem.

The memory inefficiency problem

Traditional KV cache implementations pre-allocate large, contiguous memory blocks for each request.

Here’s why this is wasteful:

Imagine serving 100 concurrent users. You don’t know how long each response will be, so you pre-allocate space for the maximum possible length, say, 2048 tokens per request. But the average response turns out to be only 200 tokens.

You’ve just reserved 10x the memory you actually need.

And it gets worse. Each request gets its own isolated block. So even if 80 of those 100 users share the same system prompt, you’re storing 80 duplicate copies of the same KV cache entries. None of that memory can be shared or reclaimed.

As a result, you’re often using only 20-30% of your allocated GPU memory effectively. The rest is reserved but unused, and because it’s scattered across fragmented contiguous blocks, you can’t even fit new requests into the gaps.

This is the core inefficiency that kills throughput.

Paged Attention

Paged Attention solves this by borrowing the concept of virtual paging memory from operating systems.

If you’ve studied OS fundamentals, you know that programs don’t get one big contiguous chunk of RAM. Instead, the OS breaks memory into small, fixed-size pages, scatters them anywhere in physical memory, and uses a page table to map each program’s virtual addresses to physical locations.

Paged Attention does the same thing for the KV cache. Here’s how:

Block-level allocation: Instead of reserving one large contiguous block per request, the KV cache is divided into small, fixed-size blocks (typically 16 tokens each). These blocks can live anywhere in GPU memory and don’t necessarily need to be next to each other.

Block table (page table): Each request maintains a block table which is a simple mapping from logical block index to physical block location in GPU memory. When the model needs the KV cache for token 33, it looks up which physical block holds it, fetches it, and computes attention. The LLM doesn’t care where the block is physically sitting.

Shared prefixes: Here’s where it gets really clever. Multiple requests that share the same system prompt don’t need separate copies of the KV cache for that shared prefix. Their block tables simply point to the same physical blocks. Only when their responses diverge do new blocks get allocated for each request individually.

In a production setting, where every request typically shares a system prompt, the memory savings via shared prefix become massive.

Impact

In the original Paged Attention paper, the authors measured that existing systems only utilized 20-38% of the allocated KV cache memory effectively. The rest was wasted due to fragmentation and over-reservation.

Paged Attention can achieve:

• 2-4x higher throughput compared to state-of-the-art systems at the same level of latency

• Near-zero memory waste, as it only allocates memory for what’s actually used

Better batching leads to more concurrent requests on the same GPU hardware

This is why vLLM, which implements Paged Attention as its core algorithm, has become the go-to inference engine for production deployments. Other frameworks like TensorRT-LLM and SGLang have also adopted similar paging mechanisms for faster inference.

This optimization becomes even more crucial when running inference servers at scale, especially with variable request patterns, shared prompts, and cost-per-request constraints.

👉 Over to you: What other OS concepts do you think could be applied to optimize LLM inference?

​To dive deeper into how LLM systems are deployed, start with the practical and hands-on LLMOps course here →

Good day!​

Coding agents collapsed the cost of writing code, but validation capacity stayed as is. Tests still encode only the failure modes someone anticipated, and staging still cannot reproduce production traffic.

AI-generated code makes this worse because it fails on unknown-unknowns. Nobody formed a mental model while writing it, so nobody can predict which inputs will break it.

The same holds for prompt changes, which can pass every offline eval and still degrade for one user segment because eval sets rarely match production traffic.

That leaves production as the only environment where this code gets validated, and it demands telemetry that answers questions nobody predicted.

Dashboards only answer the questions someone thought to set up in advance. Catching an unknown failure means keeping raw, high-cardinality events and slicing them by user ID, prompt version, or any other attribute after the fact.”

This is also why Observability Engineering, the 2022 O’Reilly book that became the standard reference on the topic, just got a near-complete rewrite.

The authors rebuilt it around this exact shift, with 27 net-new chapters on instrumenting LLM apps, feeding production telemetry back into evals, and debugging with agentic AI.

Honeycomb partnered with us on this issue and has made the early-release chapters free to download →

Download book for free!

Grab the free copy today to learn about observability engineering!

Agents need a new kind of web search

Andrej Karpathy once said LLMs are a bit like a coworker with anterograde amnesia. That’s the condition where you keep your old memories but can’t form new ones.

An LLM lives with that condition in two ways. It doesn’t remember you across conversations, and it never learns anything that happened after its training ended.

Memory features exist to patch the first gap. This piece is about the second one, because agents mostly get hired to deal with the present.

Markets move, people change roles, prices update, news breaks. Web search is how the model sees any of it, and how well it works ends up deciding how useful the whole agent is.

So you bolt a search tool onto your agent and assume it can now read the web. Then you check the first few traces, and the result is disappointing.

Nearly all the tokens you paid for went into raw page text the agent had to dig through. Only a thin slice went into answering your question.

The reason is what a search call actually returns. A typical search API hands back links and thirty-word snippets, nothing more.

So the agent isn’t reading the web, it’s reading a table of contents. Fetching the actual pages becomes its job, which means pulling the HTML, stripping the markup, and digging out the usable text before any real work starts.

On a single query, you barely notice the overhead. In a task that takes several searches, like a research run or a briefing pipeline, you pay it again on every single one.

The problem isn’t that agents search badly. It’s that each search call returns pointers to content instead of the content itself, and the agent pays to close that gap on every call.

We ran the same question through three retrieval setups and counted tokens. The most expensive setup costs roughly 4x the cheapest retrieval option.

This piece walks through those numbers and shows what a search response should look like when an agent is the one reading it. It also covers a class of questions that only full documents can answer.

Let’s start with where the cost actually comes from.

Every loop iteration pays a retrieval cost.

That repeated fetch-and-clean work has a name. Call it the retrieval tax, the tokens an agent burns preparing content before it can reason about your problem.

One looping agent is enough to feel it. A research agent searches, reads what came back, decides what to look up next, and searches again, and every pass pays the tax on top of the last one.

Here’s what that costs on a single question. We picked “What was Y2K?” on purpose, because the model already knows the answer from training.

We answered it three ways, straight from memory, through a web search loop, and through an owned index (a search service that crawls and cleans pages ahead of time and stores its own processed copy of the web), counting the total tokens billed each time.

Picking a question the model already knows is what keeps the experiment clean. The thinking cost is identical in all three runs, so any tokens above it come purely from the retrieval.

• From memory, the whole thing took about 600 tokens. That’s the baseline, the cost of just answering with zero retrieval.

• An owned index is a search service that has already crawled and cleaned the pages, so one call returns the full document. Going through one took roughly 6,900 tokens.

• A single web search hop cost about 3,750 tokens. But snippets are often too thin to answer from, so the agent refetches and refines, and three hops climbed to roughly 28,700, more than 4x the owned index.

The gap between the loop and the index is the tax made visible. Every hop re-billed the entire growing context window, so by the third pass, the agent was mostly paying to carry pages it had already read.

The owned index skipped all of that. The full document was already there when the query arrived.

What makes that possible is what comes back from the search call itself, so let’s look at that next.

Good retrieval returns documents, not directions

The fix isn’t a smarter agent or a better prompt. The fix is what comes back from the search call.

• A SERP, the plain list of results a search API returns, hands you URLs and asks the agent to go read the web itself.

• A scraper-first tool gets you closer by returning raw HTML or markdown, but the agent still has to clean it before it can use anything.

• An owned index does all of that work before the query ever arrives, so what comes back is already readable.

Here’s the same lookup across all three. We’ll search for Christoph Molnar, the author of the Interpretable Machine Learning book.

1) SERP

# query
results = search("Christoph Molnar interpretable ML")

# response
[
  {"title": "Interpretable Machine Learning", "url": "christophmolnar.com/books/...", "snippet": "A guide for making black box models explainable..."},
  {"title": "Christoph Molnar - Google Scholar", "url": "scholar.google.com/...", "snippet": "Cited by 12,847..."},
  # 5 more results across different domains
]

The agent got a list of tabs, not an answer.

2) Neural search

# query
results = neural_search("Christoph Molnar", type="person")

# response
[
  {"highlight": "Christoph Molnar is a statistician and machine learning interpretability researcher..."},
  {"highlight": "Author of Interpretable Machine Learning, an open access book with 400k+ readers..."},
  # one stray paragraph about a different researcher
]

It found the right person, but as highlight fragments. Getting the full record still needs another call.

3) Owned index

# query
result = client.search("Christoph Molnar", scope="people")

# response
{
  "name": "Christoph Molnar",
  "roles": [
    {"title": "Research Scientist", "org": "Mindful Modeler", "start": "2021"},
    {"title": "Author", "work": "Interpretable Machine Learning", "editions": 3}
  ],
  "education": [{"degree": "PhD Statistics", "institution": "LMU Munich"}],
  "websites": ["christophmolnar.com", "github.com/christophM"],
  "languages": ["German", "English"]
}

One call returns the complete record, and the agent goes straight to reasoning.

Seltz is a web index built exactly for this. It crawls the web ahead of time, processes every page into a structured document, and returns finished content through a single API call.

It comes with three scopes, and each one returns a different kind of finished document depending on what your agent needs.

• The People Scope returns full structured profiles, with every role and its dates, education, websites, and the organizations someone has worked at.

• The News Scope returns complete article text, date-windowed, so you can pull only what was published in a specific period.

• The Wiki Scope returns clean Wikipedia documents, useful for building context around companies, industries, or topics before going deeper.

One caveat is worth knowing before you build on the people scope. It works best at the director and regional-leader layer.

For the most senior executives, start with open web search and use Seltz to enrich everyone below them.

Some answers only exist across records

Full documents unlock something that goes beyond cutting the retrieval tax.

Some questions don’t live in any single search result. The answer only exists once you read across multiple records and find where they intersect.

Neither a SERP nor a neural search tool can do this on the first pass. They don’t hold enough of each record to join them, so the work lands back on the agent.

Consider a question a GTM team actually runs. You want every target account that hired someone into a data or AI leadership role in the last quarter.

Answering it needs two things together. Full role histories to find who started a new position and when, plus recent news to confirm the timing and surface the trigger event.

You query the people scope to find who moved into a relevant role recently, then cross-reference the news scope for the trigger event.

What comes out is a ranked list of warm accounts with enough context to write a relevant first outreach line for each one.

That answer doesn’t exist anywhere on the web as a page. It only exists once you join the records, and the join only works when you hold the full records to begin with.

Chain discovery and depth, and every iteration gets cheaper

Seltz isn’t a replacement for open web search. It’s the right tool for a specific class of questions.

Open web search handles discovery well. Finding who currently holds a title, confirming something launched last week, reading a page that went live this morning.

Once you know what you’re looking for, Seltz returns the complete record in one hop rather than a headline and a link.

Many pipelines use the same search tool for every query, regardless of what the question needs. Discovery queries go through the same call as deep-dive lookups, and the loop pays the retrieval tax on both.

Chain the two instead, and each loop iteration gets the right retrieval shape for what it’s actually trying to do.

That’s the core argument here. The cost of an agent loop is set less by the model and more by what each search call hands back, and returning finished documents is how you stop paying for the same pages twice.

If you’re building loops that hit these query shapes, you should give Seltz a try.

Get started here →

Stay tuned as we dig more into efficient web search for Agents and share what we learn along the way.

Thanks to Seltz for working with us on this article.

RAG, Agentic RAG, and AI Memory​

RAG was never the end goal.

Memory in AI agents is where everything is heading.

Let’s break down this evolution in the simplest way possible.

RAG (2020-2023):

• Retrieve info once, generate response

• No decision-making, just fetch and answer

• Problem: Often retrieves irrelevant context

Agentic RAG:

• Agent decides *if* retrieval is needed

• Agent picks *which* source to query

• Agent validates *if* results are useful

• Problem: Still read-only, can’t learn from interactions

AI Memory:

• Reads AND writes to external knowledge

• Learns from past conversations

• Remembers user preferences, past context

• Enables true personalization

The mental model is simple:

• RAG: read-only, one-shot

• Agentic RAG: read-only via tool calls

• Agent Memory: read-write via tool calls

Here’s what makes agent memory powerful:

The agent can now “remember” things, like user preferences, past conversations, and important dates. All stored and retrievable for future interactions.

This unlocks something bigger: continual learning.

Instead of being frozen at training time, agents can now accumulate knowledge from every interaction. They improve over time without retraining.

Memory is the bridge between static models and truly adaptive AI systems.

But it’s not as easy as it sounds.

Memory introduces new challenges RAG never had: memory corruption, deciding what to forget, and managing multiple memory types (procedural, episodic, and semantic).

Solving these problems from scratch is hard. If you want to give your agents human-like memory, check out ​Graphiti​, an open-source framework for building real-time knowledge graphs.

​You can find the GitHub repo here →​

Good day!

The move is correct, but using SLMs alone doesn’t imply savings since those models still have to be served. And serving them without wasting the GPU underneath is a separate problem to be solved.

Today, let’s understand why that is the case, what serving many small models well actually takes, and how the SIE open-source inference engine solves it end-to-end.

Let’s begin!

Workflow of a traditional AI pipeline

A production AI system is usually a stack of individual tasks, and each task is handled by the model that happens to be best at it.

For instance, a single request might:

• parse a document or an image into clean text

• embed that text into vectors for retrieval

• rerank the retrieved results by relevance

• extract the specific fields, entities, or labels that matter

• and check the output against a safety policy before sending it to user

Until recently, the default instinct was to use the largest general-purpose model you could and make it do as much of this as possible, paying frontier rates for every call.

That reflex is now reversing toward using small models fine-tuned for each task, which can run the same step at a fraction of the cost since most tasks don’t always need frontier reasoning.

But the catch teams usually miss is that a cheaper model does not imply a cheaper system.

This is because the cost of a model isn't only what it charges per call. It's also the hardware you keep running to answer that call.

A pipeline that replaced one large model with multiple small ones now has multiple models to serve, each occupying a GPU of its own.

Why don’t we fit multiple models on one GPU?

The obvious objection is that we should just put them all on one card.

This will help because no GPU provider bills for the work you do on them. They do not meter FLOPs.

What you rent is a whole GPU for a span of wall-clock time, and the meter runs whether the card is busy or sitting idle.

So every GPU you hold adds another term to the full bill:

Four models on four cards means four terms in that sum. If one card can carry all four models’ traffic, three of those terms disappear. Nothing about the work has changed, and yet the bill drops by 75%.

The objection holds up on the memory front too. A small embedding model, a cross-encoder reranker, and an entity extractor together take a few gigabytes.

An L4 has 24GB, so they can fit with room to spare, and nothing in the hardware stops you.

The reason almost nobody does it is that every layer of the stack, from the serving framework down to the driver, was built around one model owning one GPU.

vLLM and TEI are two common servers.

• TEI is Hugging Face’s embedding and reranking server and takes a single --model-id at launch.

• vLLM’s engine is built around one model as well.

So putting four models on one card requires running four separate server processes on it. Nothing in either tool coordinates those processes, and you must own that job.

In fact, each process claims its memory upfront and cannot see its neighbours.

vLLM’s --gpu-memory-utilization defaults to 0.9. The setting pre-allocates that fraction of the card at startup rather than adjusting it as per demand.

vLLM’s own documentation says this is a per-instance limit that ignores any other vLLM instance on the same GPU, and that running two means setting the value to 0.5 for each of them, by hand.

So the memory split is a constant you compute yourself, before any traffic arrives.

It must cover each model’s peak, because activation memory grows with batch size and sequence length.

Four worst-case slices pinned onto one card is the same over-provisioning you were trying to escape, and now they have been moved inside a single GPU.

Even if you size them optimally and two models spike together later, they will take the whole card down.

For instance, if the extraction model hits an unusually long document and asks for more memory than its slice allows, it does not just fail on its own. Instead, it takes OCR, embedding, and reranking with it.

This explains why one model per GPU has not been a hardware requirement but a path of least resistance through a stack where every tool was built to serve one model well, and none of them was built to share.

Sharing needs a different kind of server. One server that holds all four models, instead of four servers holding one each.

• It owns the card’s memory, so nothing has to be split by hand.

• It sees every request, so it can decide what runs first.

• And it loads a model when a request needs it, then drops it when memory runs short, so a model nobody is calling is not sitting on memory it does not need.

You do not get that by running four servers side by side. This requires an entirely different stack.

The three ways teams try to avoid this

At this point, one common reaction teams follow is to not to run GPUs the naive way. There are three common escape routes, and each one trades the problem for a different one.

• The first is a managed API. Invoke OpenAI or Cohere to avoid thinking about GPUs at all.

  

  • It is a good way to start, but catches fast when you have got scale.

  • The bill increases in a straight line with usage while the work stays routine.

  • You also run whatever the provider hosts, so the model you fine-tuned for your own task usually cannot go there, and your prompts and documents leave your environment on every call.

• The second is serverless GPUs, where you deploy your own model and pay only while it runs.

  

  • This fixes the idle-time problem on paper.

  • The problem is the cold start. A model that has scaled to zero has to pull gigabytes of weights into memory before it can answer, which adds seconds to the first request, and a reranker in the middle of a search cannot spend seconds waking up.

  • You avoid that by keeping instances warm, and now you are paying for idle GPUs again, which is the exact thing we came here to escape.

• The third is the one that is actually right for cost and for control. You self-host the models on GPUs you rent, in your own environment, where the data stays inside your walls, and the bill is a flat hourly rate rather than a per-token meter.

  

  • This is the path most serious teams end up on.

  • So you self-host, which requires you to reach for the standard serving tools. And that is where the infrastructure problem gets specific. Let’s understand more problems about them below.

Problems with standard serving tools in a multi-model pipeline

The tools devs use here were built for a different job than the one you have, and it shows up in two ways.

Start with the fact that each tool serves one shape of model.

• vLLM is built to serve LLMs.

• TEI is built to serve embedding and reranking models.

• Neither one does OCR, vision, or document parsing.

This is not a gap in the models themselves. The models for every one of these tasks already exist and are excellent, like docling and PaddleOCR for documents, SigLIP and Florence for vision, GLiNER for extraction.

What does not exist is a single server that runs all of them.

So when the agent needs to parse a document and embed it and rerank and check a policy, you need vLLM for one part, TEI for another, and some FastAPI wrapper somebody wrote for the rest.

This gives you four servers + four APIs + four deployments, all behind one agent.

Now, notice what this setup does to the GPU by going back to the reason each server holds a whole card.

vLLM will occupy 90% of the GPU at startup. TEI will grab its part. The OCR wrapper also needs something.

Each was written on the assumption that it owns the card, and none of them will give memory back to the others, because none of them knows the others exist.

There is no shared queue across them and no way to say that the live search matters more than the background indexing job. We already walked through what that looks like on one card, and it does not get better when the processes come from four different tools instead of four copies of one.

You moved to small models to save money.

But the discussion so far proves that the savings will only show up if you can put several of them on one GPU.

The standard tools cannot make that happen since they force one model per server, and one server per card, which puts you right back on one GPU per model.

The fragmentation in tooling re-creates the exact waste we were trying to avoid.

What should the serving stack for small models look like

Let’s step back from the tools and understand the things we need the ideal tooling to do.

• It should run every kind of model an agent needs, embeddings, reranking, OCR, vision, extraction, and generation, behind one API. That is the only way to get off four servers and onto one.

• It should fill the GPU instead of padding it. And that turns out to require more than a good scheduler, because to pack requests of different lengths into one pass without wasting compute, the engine has to control the batching and the attention path for every architecture it serves.

• It should load and evict models as traffic moves, keeping the busy ones loaded and letting the idle ones move out, so that a model nobody is calling is not holding memory the way a cold serverless worker or a padded vLLM instance does.

• It should ship the production layer with routing, autoscaling, monitoring, and GPU pools. This is important since a bare runtime like vLLM is only the engine. On its own, it cannot spread load across replicas, add or remove GPUs as traffic changes, or place models across a mixed fleet of cheap and large GPU cards. Today, devs need to write all of that themselves, which explains why self-hosting is more work than it sounds.

• And adding a new model should be a config change, not a redeployment.

All this is hard with months of engineering since different model families do not run the same way underneath.

• A BERT and a Qwen handle positions and attention differently.

• A ColBERT returns a vector for every token.

• A reranker returns a single score and no vector at all.

Building one engine that can hold all of those shapes, and pack any of them into a full batch, is critical work, and it is the reason this did not already exist as an open-source package.

Open-source solution: Superlinked Inference Engine

The solution to all of this is actually implemented in the open-source Superlinked Inference Engine (SIE):

It runs as one cluster inside your own cloud, and it’s built for exactly the kind of pipeline we have been describing, i.e., several small models of different kinds running back to back on shared GPUs.

• It runs every kind of model through one API with four calls, so document parsing, vision, content-safety checks, and open-model generation all run on the same server:

  • encode turns text or images into vectors

  • score reranks a query against a set of documents (returns a number)

  • extract pulls structured fields, entities, or markdown out of a raw document (return span of text)

  • and generate runs an open LLM on a prompt (returns text and token usage)

  • TEI can hand back vectors and relevance scores, but one TEI process serves exactly one model, so embedding plus reranking already means two separate deployments. And TEI does not cover extraction, OCR, vision, or generation, so each of those still needs a server of its own.

  • A single SIE cluster can serve all four shapes in one server.

• It packs those requests without wasting space.

  

  • Even inside a full batch, there is a second kind of waste. Requests come in different lengths, and the simple way to run them together is to pad the short ones out to match the longest, then process the padding as if it were real.

  • SIE batches by compute cost instead of item count. It groups requests of similar length and pads only to the longest item in that batch, never to a global maximum, so almost none of the card’s time goes to padding.

  • One key difference is where the batch gets formed. Standard stacks route each request to a worker first, and every worker batches only the slice of traffic it happens to receive, so queues run unevenly, and mixed sizes pack poorly.

  • SIE inverts this order. The gateway publishes all work into one shared queue, and each worker pulls from that pool and forms a full batch before touching the GPU.

• It loads and evicts models based on traffic:

  

  • A model loads into GPU memory the first time a request needs it, and the least recently used one is dropped when memory runs short, the same idea as a browser cache.

  • This is also the concrete answer to the reserved-but-idle problem. Several public vLLM issues report that models reserve about 80% percent of a card’s memory on load, while utilization during embedding stays under forty percent.

  • Optimized load-and-eviction strategies let you run several models in the whole pipeline.

• It comes with the production layer already built.

  • A gateway sits in front and routes each request to a worker, and when there is a burst, it accepts the request and holds it while more workers start up.

  • The workers scale up under load and back down to zero when traffic drops, so you are not paying for idle GPUs.

  • It spreads work across whatever cards you give it and comes with the dashboards to watch it and the Terraform to deploy it on AWS or GCP, so the same setup runs on your laptop and on a full cluster.

  • It can also run fully offline, with the model weights downloaded ahead of time, for environments where data cannot leave.

• Every model is tuned and ready-to-use.

  

  • With vLLM or Triton, each model has to be tuned by hand, meaning the batch size, memory fraction, precision, maximum sequence length, and attention implementation all set for that specific architecture on that specific GPU.

  • In SIE, every one of the 85+ models in the catalog carries a config file with those values already filled in, so you reference a model by name, and the engine loads it with settings known to work in production.

  • Adding or swapping just requires pointing to a different config rather than starting the tuning over.

Implementation

Below, let’s look at the pipeline that involves three calls to one server.

We'll build a small retrieval pipeline that embeds some text, reranks it, pulls named entity fields out of the context, and then generates the final answer, all through a single SIE server.

This will involve four different model architectures, all hitting one endpoint. The first three calls run on CPU, so you can follow along on a laptop.

The generation step is the one exception, since it runs on SGLang underneath, which needs a GPU.

Start by installing and starting the server. The local extra pulls the CPU runtime, and serve command starts the server on port 8080.

pip install "sie-server[local]"
sie-server serve

It is ready when the health check answers.

curl http://localhost:8080/readyz     # ok

Now, instantiate the client and point it at the running server:

from sie_sdk import SIEClient
from sie_sdk.types import Item

client = SIEClient("http://localhost:8080")

query = "Who issued the invoice and what is the amount due?"
passages = [
    "Invoice INV-4471 was issued by Northwind Logistics on 14 March 2026.",
    "The total amount due is $18,240, payable within 30 days.",
    "The cafeteria menu changes every Monday and Thursday.",
]

Every call in the rest of the pipeline goes through this one object and this one endpoint, no matter which model it hits.

First we invoke encode.

This turns each passage into a dense vector using an embedding model.

docs = client.encode(
    "sentence-transformers/all-MiniLM-L6-v2",
    [Item(text=p) for p in passages],
)

print(docs[0]["dense"].shape)
# (384,)

Passing a list gives you back a list of results, one per passage, and each carries its vector under the dense key.

Next, we invoke score that uses a cross-encoder to read the query against each passage together and returns a relevance score for each one, a different kind of output than the vectors above:

from pprint import pprint

scores = client.score(
    "cross-encoder/ms-marco-MiniLM-L-6-v2",
    Item(text=query),
    [Item(id=str(i), text=p) for i, p in enumerate(passages)],
)
pprint(scores["scores"])
# [{'item_id': '0', 'rank': 0, 'score': 3.009},
#  {'item_id': '1', 'rank': 1, 'score': -1.061},
#  {'item_id': '2', 'rank': 2, 'score': -11.425}]

The scores are cross-encoder logits, so the absolute numbers do not mean much on their own. Instead, the order is important. The invoice line scores above the payment terms, and the cafeteria has the lowest score, as expected.

The results come back already ranked, so the top passage is the best match:

best_id = scores["scores"][0]["item_id"]
best = passages[int(best_id)]
# 'Invoice INV-4471 was issued by Northwind Logistics on 14 March 2026.'

Finally, we invoke extract which runs an extraction model to generate the fields we ask for, returning each as a labelled span with a confidence score.

result = client.extract(
    "urchade/gliner_multi-v2.1",
    Item(text=best),
    labels=["organization", "date", "amount"],
)
print(result["entities"])

[
  {
    'text': 'Northwind Logistics', 'label': 'organization', 
    'score': 0.979, 'start': 31, 'end': 50, 'bbox': None
  },
  {
    'text': '14 March 2026', 'label': 'date',
    'score': 0.951, 'start': 54, 'end': 67, 'bbox': None
  }
]

This is a third architecture that has a different output shape, but is invoked via the same client and the same server.

Finally, we invoke generate to compose the answer from the top-ranked context. This call needs the GPU build of the server, since generation runs on SGLang, which has no CPU path. Everything else about the call is identical.

context = " ".join(
    passages[int(s["item_id"])] for s in scores["scores"][:2]
)

answer = client.generate(
    "Qwen/Qwen3-0.6B",
    f"Answer in one sentence using only this context.\n\n"
    f"Context: {context}\n\nQuestion: {query}",
    max_new_tokens=64,
)

print(answer["text"])
# 'The invoice was issued by Northwind Logistics, and the amount due is $18,240.'

print(answer["usage"])
# {'prompt_tokens': 61, 'completion_tokens': 17, 'total_tokens': 78}

This is a fourth output shape, free text with token usage, produced by a generative model, and it comes back through the same client as the vectors, the scores, and the spans.

That is the whole pipeline involving four model families with four different output types, running under one server and client, with nothing to deploy beyond the single process started at the top.

Each model is pulled from Hugging Face on its first call and stays active for the next request.

Using small, specialized models for the narrow tasks is the right approach, and it is not really controversial. They are accurate enough on the task they are trained for, and they keep your data in your own environment.

But switching to small models does not make inference cheaper by itself. It moves the cost from a per-token bill to the GPUs you rent, and if you put each model on its own GPU, most of that hardware sits idle, and you are paying for it.

The saving only appears when the models share GPUs, and that requires a single engine that can run all of them, because the moment you serve each model with a different tool, each one takes a GPU of its own again.

SIE (open-source) already implements that, and it lets you run the different models an agent needs on one shared cluster, keeps the GPUs busy instead of idle, and includes the routing and autoscaling you would otherwise have to build.

It also plugs into the retrieval stack you already run, with integrations for Chroma, Qdrant, Weaviate, and LanceDB.

You can find the repo here: github.com/superlinked/sie

(don’t forget to star it ⭐️)

👉 Over to you, how many separate models does a single request in your stack call, and how many of them are holding a dedicated GPU right now?

Good day!

We built a Hugging Face fine-tuning studio that lets you fine-tune any LLM directly from Claude:

The app connects to the HF Hub for model and dataset search. It handles chat template formatting for the training data, and lets you configure LoRA rank, quantization, batch size, and learning rate directly from Claude.

Training runs on HF’s GPU infra via AutoTrain.

Once training finishes, you can also chat with your fine-tuned model (or any other LLM on HF) directly from Claude.

The studio’s built with the mcp-use SDK, an open-source full-stack framework to build MCP Apps for Agents.

In mcp-use, any MCP tool can be associated with a UI.

You define a tool handler, create a React component, and the mcp-use framework handles the tool registration, prop mapping between server and widget, bundling, and hot reload during development.

The widgets follow the MCP Apps standard, inspired by OpenAI’s Apps SDK.

You can find the mcp-use GitHub repo here →

And you can find the code for this fine-tuning studio here →

The four types of agent loops

Loop engineering keeps getting talked about as one thing, when it’s actually a choice between four structures, each fitting a different kind of task.

It means designing the system that steers the agent, instead of steering it by hand, move by move.

That system always answers two questions: what starts a run, and what decides the work is done.

In a hand-run session, the human answers both, every single time. Each loop type moves more of that into the system.

Here’s each type, what triggers it, and when to reach for it.

#1) Turn-based loops, triggered by a user prompt.

The agent gathers context, acts, and checks its work inside a single turn. Then a human reviews the output and writes the next prompt.

Use this when requirements are still forming and every output changes what the next prompt should ask for.

#2) Goal-based loops, triggered by a /goal command that carries success criteria and a budget, like “get the homepage Lighthouse score to 90, stop after 5 tries.”

When the agent tries to stop, an evaluator model checks whether the goal is met, and it sends it back to work.

Use this when the outcome is measurable but the path there doesn’t need human attention.

#3) Time-based loops, triggered by a clock.

An interval fires, the agent runs a fixed prompt like “check the PR, fix CI,” then waits for the next tick. The /loop command runs on the local machine, and /schedule moves it to the cloud so it survives a closed laptop.

Use this for recurring work where the task is known in advance and only the timing repeats.

#4) Proactive loops, triggered by an event or schedule with no human present.

A routine watches a channel and spawns a workflow when something needs handling. That workflow runs a triage agent, a fix agent, and a reviewer who adversarially judges the work before the task closes.

Use this for standing responsibilities where nobody can predict what will come in, only that something will.

Each type hands off one more job than the last. Turn-based keeps both jobs with the human, goal-based automates the checking, time-based automates the trigger, and proactive automates both while deciding the workflow shape at runtime.

So the mapping question isn’t which loop is most advanced. It’s whether the task is exploratory, measurable, recurring, or standing.

The more you hand off, the less you monitor yourself.

We wrote a full breakdown on loop engineering.

You can read it here →

NVIDIA researchers built a new transformer variant

One small change to the layers made:

• decoding 1.7x faster

• long-reasoning accuracy up 6.5 points

In a typical transformer architecture, every attention layer computes Q, K, and V.

NVIDIA’s tweak adds a fourth projection, which predicts what the next layer will need.

To understand why they did this, let’s first see what happens in a Transformer architecture during inference right now.

Sparse attention was an attempt to handle long-context inference. Instead of attending to every cached token, modern designs score the KV cache in blocks, keep the top-k, and attend only to those.

That cuts attention compute and bandwidth, but this still leaves us with two problems.

> First, the KV cache still grows with every generated token.

At 100K+ context, it no longer fits in GPU memory and gets offloaded to CPU RAM.

Now every layer must first copy its selected KV blocks from CPU memory back to the GPU. That copy is slow, the GPU sits idle while it waits, and the stall repeats at every layer of every decode step.

> Second, the selection step itself is not free.

Standard selectors score every candidate block with every query head in a GQA group (grouped-query attention, where several query heads share one KV head), then softmax each head’s scores and sum them across the group.

During decode, the sparse attention itself is cheap because there is only one query token.

But the expensive part is deciding which blocks to attend to, and that cost keeps growing with context length.

Both problems trace back to the same design in today’s sparse attention methods, i.e., the attention query drives the block selection.

Selection needs the query vector Q, and Q only exists once its layer is already running. By then, it’s too late to fetch anything early.

The query also drags its multi-head layout into selection, so all that scoring computation runs just to make one top-k decision.

A recent paper from NVIDIA and MIT called SparDA breaks this coupling with one architectural change.

Each layer now emits four projections instead of three:

↳ Q, K, V, and a Forecast.

The Forecast from layer L predicts which KV blocks layer L+1 will need.

Layer L+1’s own query performs the sparse attention over those selected blocks.

This one change fixes both problems.

Since the next layer’s block set is known while the current layer is still computing, the runtime fetches those blocks from CPU memory on a separate CUDA stream.

The copy overlaps with the current layer’s compute, so the GPU no longer waits for it.

And since the Forecast is separate from the attention query, it doesn’t need one score per query head.

SparDA uses one Forecast head per GQA group, which removes the per-query-head scoring loop and skips the softmax step entirely.

DeepSeek did something similar in DSA, where a small indexer picks important tokens instead of the query doing it.

SparDA applies the same idea to blocks and adds the prefetch angle that DSA doesn’t touch.

The cost of the change is small.

The Forecast adds just 33.5M parameters on an 8B model (0.41%), and only those projections are trained, using a KL loss that matches the original selector’s block distribution.

On MiniCPM4.1-8B and NOSA-8B, accuracy matches or beats the sparse baseline, with NOSA-8B gaining +6.5 on long reasoning.

Prefill runs up to 1.25x faster and decode up to 1.7x faster than the sparse offload baseline.

There’s one more benefit.

Because prefetch hides the offload cost, most of the KV cache can live in CPU RAM, and the freed GPU memory fits much bigger batches, pushing decode throughput up to 5.3x over the non-offload sparse baseline.

That said, this lookahead will only pay off during decode with CPU offload. During prefill, all keys already live on the GPU, so the gain there comes purely from the cheaper selection.

Here’s the paper: ​https://arxiv.org/abs/2606.04511​

We wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching.

​Read it here →​

Good day!

Most GPU clouds sell you a VM and hope the abstraction holds. Lightning AI just shipped something that inverts that.

Their AI Cloud is based on the idea that instead of leasing hardware from someone else and building software on top, own the entire stack: GPUs, datacenter fabric, hypervisor, and scheduler as one system.

Why that matters mechanically:

• Topology stops being a guess. Because Lightning controls the hypervisor, the guest sees the real hardware topology instead of a flattened abstraction. NCCL and PyTorch work correctly without manual tuning.

• Multi-node placement becomes deterministic. Because they own the datacenter, the scheduler knows the physical switch fabric in advance and will land jobs on well-connected machines instead of scattered leased capacity.

• Provision has no leased layer sitting between a request and the hardware, so there’s one operator accountable for the machine actually showing up.

The AI Cloud is now available as guaranteed on-demand or spot capacity, on the same platform behind PyTorch Lightning.

Infrastructure ownership determines whether your placement, your performance, and your provisioning are things you can actually reason about, whereas when you rent your infra, you’re hoping someone (or a series of vendors) get it right in their cloud.

Request access to the Lightning Cloud now →

Thanks to Lightning for partnering today!

Agentic RL: Environments, Trajectories, and the Training Loop

Part 12 of the Reinforcement Learning course is available now.

The previous chapters covered GRPO, verifiable rewards, and LLM-as-a-judge. This one extends all of that into a working training loop for an agent that takes multi-step actions with tools.

Read Part 12 of the RL course here →

RL Nanodegree Part 12

It covers:

• What an RL environment means for an LLM agent

• Trajectories as the unit of training

• The environment spectrum from state-changing to static worlds

• Outcome vs. process-informed scoring

• RULER as a packaged judge with prefix deduplication

• The credit assignment problem in long-horizon episodes

• Hands-on: training a SQL agent with ART and RULER on free Colab

Read Part 12 of the RL course here →

RL Nanodegree Part 12

Why care?

Right now, most agents in production are shaped entirely through prompt engineering. The system prompt tells the agent what tools it has, how to reason, and when to stop.

All the intelligence comes from the base model’s pre-existing capabilities and whatever structure the prompt imposes.

That works up to a point, but they cannot reward the behavior you want and penalize the behavior you do not want, across thousands of rollouts, the way RL can.

This chapter covers the complete RL loop.

It defines what an RL environment looks like when the agent is an LLM with tools, formalizes the multi-step trajectory as the training object, introduces RULER as the scoring layer, and runs the full rollout-score-update cycle end to end.

The hands-on section trains a 3B parameter model to answer database questions using only tool calls, on a free Colab GPU, with no hand-written reward function beyond a one-line correctness bonus.

Here’s what we have covered so far:

• Part 1: Foundations of Reinforcement Learning →

• Part 2: MDPs and Value Functions →

• Part 3: Bellman Equations and Dynamic Programming →

• Part 4: Model-Free Learning →

• Part 5: Function Approximation →

• Part 6: Deep RL and DQN →

• Part 7: Policy Gradients and Actor-Critic →

• Part 8: Proximal Policy Optimization →

• Part 9: RLHF →

• Part 10: Verifiable Rewards and GRPO →

• Part 11: Reward signal problem for Agents →

• Part 12: Environments, trajectories, and the training Loop →

👉 Over to you: What topics would you like us to cover in this RL series?

​Skill bundles in Hermes agent​

We found skill bundles to be the most underrated feature in Hermes.

Real workflows need clusters of skills together, not one at a time.

For example, writing code might need a code review skill, a testing skill, and a PR workflow skill.

Every time, you’re loading the same group manually with three slash commands in sequence, or asking the agent in natural language and hoping it picks the right ones.

A skill bundle fixes this with a single YAML file that groups multiple skills under one slash command.

When you invoke it, every listed skill loads at once, plus any custom instruction you’ve baked in.

The image below shows the anatomy of a bundle file and how it expands at invocation with one command that has a shared instruction baked in.

But the design choices underneath are what make it practically useful:

1) Missing skills are skipped, and it’s not fatal.

→ If one skill in the bundle is uninstalled, the rest still load. You don’t lose the whole workflow because of one missing piece.

2) Bundles beat skills on name collisions

→ If a skill and a bundle share a name, the bundle wins since you opted into it.

3) It works everywhere

→ CLI, TUI, dashboard, Telegram, Discord, Slack. one YAML definition, all platforms.

4) No cache invalidation

→ Bundles generate a fresh user message at invocation, the same as individual skill loading, and at no performance cost.

Once your bundles are stable, the natural next step is team sharing.

Bundle YAMLs are just file so you can just put them in a Git repo and have each team member symlink into the ~/.hermes/skill-bundles/ folder.

If you update the repo, everyone gets the update.

This is what turns a personal agent into something a small team can standardize around.

We wrote ​a full deep dive covering Hermes agent’s self-evolving skills, three-tier memory, GEPA optimization, and setting up multiple specialized agents.

​Read it here →​

Good day!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Transformer Lab is an open-source ML platform that orchestrates GPUs across any cloud and runs any training or eval workflow you define:

• supports LoRA, QLoRA, DPO, ORPO, SIMPO

• use it from a GUI, CLI, or agent skill.

• works with MLX, vLLM, Ollama, and HF Transformers

• one-click convert between HF, GGUF, and MLX

• diffusion training and inference built in

• LLM-as-a-judge evals plus the EleutherAI harness

• submits jobs to Slurm and SkyPilot from the same UI

• runs on Apple Silicon, NVIDIA, and AMD

The same interface a solo researcher uses on a MacBook also drives a 64-GPU cluster. Nothing else does that today.

​Find the GitHub repo here →​ (don’t forget to star it ⭐️)

Transformer Lab GitHub Repo

How LLM routing actually works in production

A tricky LLM interview question:

Your agent runs everything on a frontier LLM, so you add a routing layer that sends several easy calls to a 15x cheaper LLM.

The router is working as expected, but the bill is still the same as before.

Where did the savings go?

On a high-level, model routing works as follows:

• Add a classifier in front of the agent

• Send easy prompts to the cheap model

• Send hard ones to an expensive model

While this looks simple, under the hood, a lot more engineering must go into this layer before you can actually realize any savings.

And the reason is in how agents consume tokens.

A typical agent task is never one LLM call. Instead, one prompt fires several LLM calls in sequence, like planning, tool use, and analyzing results.

Each call sends the accumulated context back through the model, so the input grows every step.

To avoid redundantly processing the same tokens on every call, providers built prompt caching.

Essentially, the input a model has already seen costs about 90% less than fresh input.

The problem is that this cache is specific to a model.

So every time the router switches models mid-task, the accumulated warm cache gets thrown away, and the full context is re-billed at cold rates.

This means a cache-hot expensive model can still cost less than a cache-cold cheap model.

Production routers solve this by making the routing decision once per task instead of once per call.

The first call routes normally, and the chosen model gets pinned to a session ID. Every call after that goes to the same model, so the cache stays warm and the context stays coherent.

When the task changes, the pin resets, and routing starts fresh.

The savings don’t disappear, but rather they move one level up.

Each new task still gets routed by difficulty, so easy tasks land on a cheap model and hard ones on an expensive model.

The pin only stops switching inside a task, which is exactly where switching costs more than it saves.

This is called model affinity, and it’s the last layer of a 4-stage production routing layer that runs on every request.

A) Guardrail filter

The prompt goes through a safety check. Jailbreaks and unsafe inputs get caught before they reach the router, and every call is logged and traced from this point.

B) Router model

A small routing model reads the prompt, infers the domain and action of the request, and matches it against candidate models.

The model has to be tiny, because a routing decision that costs as much as the call it’s routing saves nothing.

C) Selection policy

A cost policy picks the winner from the candidates, preferring the cheapest or the fastest, depending on the config.

Prices are picked from pricing catalogs, so the choice adapts when providers reprice, and the runner-up stays on standby as the fallback if the winner fails.

D) Model affinity

After selection, the winning model gets pinned to the session ID. Every later call in the task goes to it, so the cache never resets and the context never splits.

To see this in practice, the exact pipeline is already implemented in Plano, an open-source proxy that runs locally between your agent and your model providers.

Here’s the GitHub repo →

The config for all four layers is defined in one YAML config file, and swapping any of it never touches agent code.

The routing model in stage 2 uses Arch-Router, a 1.5B model available on HF. It’s trained on human preference data, i.e., what developers actually pick for each task type.

So routing reflects real-world preference instead of benchmark rank, and you define those preferences in plain English in the config.

It also implements an observability console with a per-request cost column, so you can see exactly which model answered each request and what it cost.

Here’s the repo: https://github.com/katanemo/plano

(don’t forget to star it ⭐ )

We put this exact pipeline in front of my Hermes agent, and the usage dropped 2x, without changing a line of agent code.

Here’s the full article we wrote →

Good day!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

MongoDB put together an AI Skill Badges program on MongoDB University with hands-on learning tracks for developers building production-grade AI applications.

Three standout AI Skill Badges you should start with:

1. Auto-embedding AI Models: Learn how to generate vector embeddings directly inside MongoDB without a separate embedding pipeline.

2. Agentic Memory with MongoDB: Design short and long-term memory for AI agents using MongoDB, LangGraph, vector search, and Voyage AI.

3. Voyage AI for Semantic Search and RAG: Build a two-step retrieval pipeline using Voyage AI and MongoDB’s auto-embedding to improve relevance while controlling latency and cost.

Beyond the three, MongoDB University also covers AI Skill Badges like:

↳ AI Data Strategy with MongoDB

↳ Building an App with Code Agents and MongoDB

↳ Vector Search Fundamentals

↳ RAG with MongoDB

↳ AI Agents with MongoDB

Every badge focuses on building working systems, not walkthroughs of abstract concepts.

Start learning for free here →

Thanks to MongoDB for making this public and partnering with us today.

RAG & Fine-tuning in LLMs

If you’re building real-world LLM apps, you can rarely use a model out of the box without adjustments.

Devs typically treat RAG and fine-tuning as interchangeable options, but in reality, they are not.

RAG and fine-tuning solve fundamentally different problems. One controls what the model knows at runtime. The other changes how the model behaves by default.

This visual breaks it down for you:

For RAG, look at the top half of the visual.

RAG operates at inference time. When a user sends a query, the retriever searches your knowledge base (PDFs, vector DBs, APIs, documents), pulls relevant context, and passes it to the LLM along with the query. The model weights never change. You’re giving the LLM a “cheat sheet” at runtime.

Fine-tuning is different. To understand, look at the bottom half of the visual.

It happens offline, before deployment. You train the model on domain-specific data, and the weights actually update. The model now behaves differently by default.

Fine-tuning is for changing how the model behaves. Its tone, vocabulary, response structure, or specialized reasoning patterns.

Two questions guide which one you need:

• How much external knowledge does your task require?

• How much behavioral adaptation do you need?

If you need the model to reference specific documents, product catalogs, or anything that updates frequently, that’s mostly a RAG territory.

If you need the model to adopt internal vocabulary, match a specific writing style, or follow domain-specific reasoning patterns, that’s mostly a fine-tuning territory.

For instance, an LLM might struggle to summarize company meeting transcripts because speakers use internal jargon the model has never seen. Fine-tuning fixes this.

That said, in production systems, you might often need both. A customer support bot might need to pull answers from documentation (RAG) while responding in your brand’s voice (fine-tuning).

The simple takeaway:

• RAG → What should the model know?

• Fine-tuning → How should the model behave

They’re not competing. They’re complementary layers in an LLM stack.

On a side note, we started a beginner-friendly crash course on RAGs recently with implementations, which covers:

• ​​RAG fundamentals​​​​​

• ​RAG evaluation​

• ​​​​​RAG optimization​​​​​

• ​Multimodal RAG​

• ​​​​​Graph RAG​​

• ​​Multivector retrieval using ColBERT​

• ​​​​​RAG over complex real-world docs ft. ColPali

Also, we have covered several LLM fine-tuning with code techniques below:

• Understanding LoRA-derived Techniques for Optimal LLM Fine-tuning.

• Implementing LoRA From Scratch for Fine-tuning LLMs.

• Implementing DoRA (an Improved LoRA) from Scratch.

​ANN search using inverted file index​

kNN performs an exhaustive search, which is inefficient at scale!

Approximate nearest neighbor search algorithms solve this.

The core idea is to narrow down the search space using indexing techniques, which improves the run-time performance.

Inverted file index (IFV) is one of the simplest and intuitive techniques to do this.

Steps for indexing:

• Partition the given data using techniques like k-means.

• Each partition gets a centroid, and each data point gets associated with only one partition.

• A map holds all the data points that belong to a centroid’s partition.

Indexing done!

Here’s how we search.

• First, find the closest centroid to the query:

• Next, find the nearest neighbor among only those data points that belong to the closest centroid’s partition:

This drastically reduces the run-time.

Consider:

• There are N data points.

• Each data point is D-dimensional.

• We create K partitions.

• Lastly, for simplicity, let’s assume that each partition gets equal data points.

In kNN, the query data point is matched to all N data points, which makes the time complexity → O(ND).

In IFV, however, there are two steps:

1. Match to all centroids → O(KD).

2. Find the nearest neighbor in the nearest partition → O(ND/K).

The final time complexity comes out to be the following:

…which is significantly lower than that of kNN.

Assume → N=10M and k=100:

• The search complexity of kNN will be proportional to 10M.

• With IFV, the search complexity will be proportional to 100 + 100k = 100100, which is nearly 100 times faster.

That said, ANN is not always accurate.

If some data points are actually close to the query data point but not in the same partition, they may still get missed:

We willingly accept such trade-offs to reduce latency.

In the deep dives on vector databases (open access), we discussed 4 such techniques, along with an entirely beginner-friendly and thorough discussion on Vector Databases.

Check it out here if you haven’t already: ​A Beginner-friendly and Comprehensive Deep Dive on Vector Databases.

Good day!

The voice agent stack is one of the few areas in AI where the demand is enormous, and the worked examples barely exist.

Speechmatics Academy has open-sourced a collection of runnable examples across batch, real-time, voice, and TTS, each standalone enough to clone a single folder and have a working pipeline in minutes.

Speechmatics Academy GitHub Repo

The integrations have complete loops with LiveKit, Pipecat, Twilio, and VAPI, covering WebRTC capture, turn detection, speaker focus, interruption handling, and function calling.

The use cases it teaches cover production territory, including SRT captioning, call-center topic detection, and HIPAA-friendly medical microbatching with Silero VAD chunking.

While everyone debates voice as the next interface, this is the repo that shows how to actually build for it.

You can find the GitHub repo here → (don’t forget to star it ⭐️)

Thanks to Speechmatics for partnering today!

[Hands-on] How to build an RL environment

Andrej Karpathy summarized the entire history of LLM training in three nouns:

• text

• conversations

• and environments

Pretraining ran on internet text, supervised fine-tuning ran on curated conversations, and the current era of reinforcement learning runs on environments.

OpenAI’s o1 proved that framing by training on math and coding problems with verifiable answers, and DeepSeek-R1 published the recipe openly.

The industry now treats environments as the scarce resource. Anthropic reportedly discussed spending over $1 billion on them in a single year.

Meanwhile, someone is giving them away for free through a Hugging Face-style hub hosting 2,500+ open-source environments.

And today, we’ll use their framework to build one of our own from the ground up.

Before we build anything, let’s understand what an environment actually is. Every RL setup is a loop with four moving parts, and they map onto anything the model interacts with:

• State → what the model sees right now

• Action → the choice it makes from that state

• Reward → a number that says how good that choice was

• Environment → the thing that holds the state, accepts the action, and returns the reward

The hard part of this loop was always the reward, which traditionally required training a separate model on human preferences.

DeepSeek-R1 replaced that entire model with a plain Python function through GRPO (Group Relative Policy Optimization). Score several answers to the same prompt and push the model toward the ones that beat the group average.

Here is an illustration of how GRPO works:

That leaves the environment as the barrier still standing. A reward function can score a final state, but something has to present the task, accept the model’s actions, enforce the rules, and produce that final state in the first place.

Frontier labs guard this part as a proprietary asset, which is why few practitioners have ever seen a working environment from the inside.

That team giving environments away is Prime Intellect, and their library, Verifiers, is the framework we’ll build with (100% open-source).

The design is model-agnostic, the rewards are fully verifiable, and the same skeleton works for any turn-based task you want to train on.

How to read this issue

The goal here is simple. We’re building a complete RL environment, and we’re covering it in a way that stays easy to digest even if this is your first one.

Every idea comes with the code that implements it, kept short enough to read in the flow of the article. The complete, runnable version is shared at the end.

And the game we’ll build around is just the working example. By the time you finish, you’ll be able to take this exact structure and adapt it to your own use case.

The Othello RL env: Why a board game

Othello is a two-player game on an 8x8 grid. You place a disc so it traps opponent discs in a straight line, every trapped disc flips to your color, and whoever owns more discs when the board fills up wins.

That flipping rule is why a single move can swing the score hard, and why corners matter so much (a corner disc can never be flipped back).

Every RL part has a home here. The board is the state, the disc placement is the action, the final result feeds the reward, and the game engine that validates moves, flips discs, and plays the other side is the environment.

The LLM plays Black and a built-in engine plays White. After each Black move, White responds, the updated board goes back to the model, and this repeats until the game ends.

Tech stack

Three tools make this work, each handling one layer:

• Verifiers: the RL framework. It defines the environment, runs the turn loop, and handles evals.

• Lightning AI: an OpenAI-compatible inference API, so the same code calls hosted models like Claude or DeepSeek without provider-specific rewrites.

• vLLM: serves open-weight models locally behind that same OpenAI-compatible endpoint.

The shared interface is what makes the environment model-agnostic. A local Ministral-3B and a hosted GPT-4.1 swap in one line, and you change the model name, nothing else.

The game loop

Here’s what the model sees on each turn:

The board, the score, and the list of valid moves are the entire state. Everything the model knows about the game comes from this text.

It must respond with a <think> section followed by a <move> tag, which forces it to reason through the position before committing to a move.

The environment validates the move against the valid list, applies it to the board, lets White respond, and sends the updated board back. An invalid move gets an error message and a retry, with a penalty applied to the reward.

All of that lives in one method. OthelloEnv inherits from MultiTurnEnv in Verifiers, which handles the loop, turn tracking, and termination, and the base class calls your env_response every time the model sends a move:

class OthelloEnv(MultiTurnEnv):
    def env_response(self, model_output, state):
        move = parse_move(model_output)

        if not is_valid(move, state.board):
            state.penalty += INVALID_MOVE_PENALTY
            return error_message(move), state          # same turn, retry

        state.board = apply_move(state.board, move, player="black")

        if not game_over(state.board):
            white_move = opponent_engine(state.board, state.difficulty)
            state.board = apply_move(state.board, white_move, player="white")

        return render_board(state.board), state

This is the shape of the logic; the real version also handles board parsing and edge cases that the snippet skips. I’ll share the full working code later once we understand the setup end-to-end.

The built-in opponent engine

The built-in engine plays white and has two modes:

• Random picks any legal move. Useful for early training, since the model can win just by making reasonable decisions.

• Minimax simulates future moves, scores each resulting position by corner control, board position, and available moves, and picks the move with the best worst-case outcome. Depth-3 means it looks three moves ahead, enough to set traps and avoid obvious blunders.

A randomness parameter controls how often White ignores its strategy and picks randomly instead. Lower randomness means more consistent, punishing play.

White’s moves are generated from the current board state and a fixed game seed, so the same position always produces the same response. Determinism lets you compare different models under identical conditions.

Here’s how the code for the same looks like:

def opponent_engine(board, randomness, depth):
    if random.random() < randomness:               # random mode
        return random.choice(legal_moves(board))
    best_move = None                                 # minimax mode: try every move,
    best_score = float("-inf")                       # keep the one with the best score
    for move in legal_moves(board):
        score = minimax(apply_move(board, move), depth - 1, my_turn=False)
        if score > best_score:
            best_move, best_score = move, score
    return best_move

The reward functions

At game end, four signals combine into a single reward:

Each one is a function that reads something off the final state and returns a number, and the win signal is the simplest:

def win_reward_func(state):
    result = state.get("result")
    if result == "black":   # the model's color
        return 1.0
    if result == "draw":
        return 0.5
    return 0.0              # loss, or game never finished

The other three follow the same shape, and all four combine into one score:

def total_reward(state):
    return (
        win_loss_score(state)
        + piece_advantage(state)
        + format_compliance(state)
        - invalid_move_penalty(state)
    )

The reason for four signals instead of a single win/loss bit is resolution. Early in training, most games are losses, and to a pure win/loss reward, they all look identical, so the model has nothing to optimize.

• Piece advantage separates a close loss from a blowout, giving the model a gradient before it starts winning.

• Format compliance carries a low weight, so clean formatting never outweighs good play.

• The invalid move penalty is capped, so one broken game can’t drown out everything the model did right.

Every score comes directly from the game state and rules, with no judge model or LLM evaluator involved, so the reward is fully deterministic and reproducible.

Wiring it together

The environment generates games across different starting positions and opponent difficulties, the model plays each one through the loop above, and at game end the four rewards are computed and combined. During evaluation, rewards, token usage, and turn counts are aggregated across all games into a results table.

One function wires everything up, and it’s what the prime eval command calls behind the scenes:

def load_environment(min_random_move_prob, max_random_move_prob, parse_think):
    dataset = generate_games(min_random_move_prob, max_random_move_prob)
    parser = XMLParser(fields=["think", "move"])
    rubric = Rubric(funcs=[...], weights=[1.0, 1.0, 0.2, 1.0])
    return OthelloEnv(dataset, parser, rubric)

What the numbers show

Running an evaluation is one command, with the opponent settings passed as arguments:

prime eval run othello -m openai/gpt-4.1 -n 100 \
  -a '{"min_random_move_prob": 0.0, "max_random_move_prob": 0.0, "minimax_depth": 3}'

Swap the model name to test something else, whether it’s hosted through Lightning AI or running on a local vLLM server. Here’s how two models fared against two opponents:

From evaluation to training

Evaluation tells you where the model struggles, and training fixes it in three stages. The same environment supports all of them:

Data generation: have your strongest model play a few hundred games and save the results. The same eval command writes straight to a dataset:

prime eval run othello -m openai/gpt-4.1 -n 200 \
  --save-to-hf-hub --hf-hub-dataset-name your-username/othello-data

Filter it down to wins and draws before training on it, so you’re not teaching the model your strongest player’s mistakes alongside its format.

Supervised fine-tuning: teach format and valid moves first. The Ministral-3B turn counts make the point directly, since unreliable formatting and illegal moves are noise RL can’t train through.

RL training: this is where strategy improves. Multiple games are played from the same starting position, each is scored with the same reward functions from evaluation, and the model is updated toward the higher-scoring rollouts.

Stripped to its core, that’s the GRPO loop from the top of the article:

for prompt in batch:
    rollouts = [play_game(model, prompt) for _ in range(group_size)]
    rewards = [total_reward(r.final_state) for r in rollouts]
    advantage = rewards - mean(rewards)     # relative to the group
    update_model(model, rollouts, advantage)

Each rollout is scored against the average of its own group, so a move that wins 6 of 10 games from a given position is rewarded more than the weaker attempts beside it.

Adapting this to your own task

Once you remove the Othello specifics and the same MultiTurnEnv gives you a skeleton that fits any turn-based task:

class TaskEnv(MultiTurnEnv):
    def env_response(self, model_output, state):
        action = parse_action(model_output)          # your task's syntax
        if not is_valid(action, state):
            return error_message(action), penalize(state)
        state = apply_action(state, action)          # your task's rules
        if not task_complete(state):
            state = environment_step(state)          # tool, API call, or opponent
        return render_state(state), state            # your task's display

This isn’t specific to games. A coding agent swaps apply_action for running a test suite against generated code, a support agent swaps it for checking whether a tool call retrieved the right record, and a research task swaps it for verifying a claim against a cited source.

Adapting it comes down to four swaps:

• Task logic: your domain’s rules for what counts as a valid action and how the state changes

• Response engine: whatever the model reacts to, from a rule-based simulator to a live API to another model

• Reward functions: keep the pattern (outcome signal, partial credit, format, penalty) and replace the domain logic

• State rendering: whatever the model needs to see, whether that’s a file diff, a conversation transcript, or a tool’s response

The structure underneath stays the same regardless of domain: parse, validate, apply, respond, score. The rubric is the design, and if you get the components right, the training signal takes care of itself.

Try it yourself

All the code, setup instructions, and ready-to-use GPUs to reproduce these results are in the Lightning AI Studio template:

​Build a Custom RL Environment →​

​Check out Lightning AI Inference →​

Good day!

Karpathy defined agentic engineering as the discipline that separates production agent work from vibe coding. The core skills he listed were spec design, eval loops, and security oversight.

The problem is that production agents don’t run on the agent code alone. They need a model provider, a retrieval layer, an eval system, a deployment target, and observability.

And normally, these are scattered across different dashboards, config files, and tools, each with its own setup and learning curve.

The solution to production-grade Agentic Engineering is now actually implemented in Google’s Agents CLI.

It covers the entire lifecycle in a single interface, covering scaffolding, evaluating, and deploying ADK agents.

The setup command injects 7 ADK-specific skills into a coding agent’s context, which lets it handle scaffolding, evals, deployment, and enterprise registration through natural language.

We mapped the whole thing end-to-end in the diagram below. Here is what each layer is doing:

• Agent Orchestration defines the agent using the ADK, and the A2A protocol handles coordination when one agent needs to invoke another. ADK is model-agnostic, so the models cover Gemini, Gemma, and third-party models like Claude, via Model Garden.

• The data layer handles retrieval. Vector Store stores the embeddings for RAG, and the ingestion pipeline loads and chunks docs into it. Monitoring and storage happen alongside it.

• The evaluation layer runs the agent against test scenarios with LLM-as judge scoring. It runs before deployment.

• For deployment, you can choose between Agent Runtime, Cloud Run, etc., depending on how much control you need. Agents CLI configures across all targets.

• Finally, teams that need staging and production environments, the IaC and CI/CD layer handles infrastructure as code and deployment pipelines.

Every one of these layers is a service that a dev would otherwise configure manually.

But Google’s Agents CLI ties them together to make the full agentic engineering lifecycle practical.

Here’s the Agents CLI GitHub repo: https://github.com/google/agents-cli

(don’t forget to star it ⭐ )

Thanks to Google Cloud for partnering today!

Build your own 100% local AI second brain

Andrej Karpathy said:

“There’s room for an incredible new product in the second brain space”

And after that, everyone is suddenly building a second brain.

Karpathy’s LLM wiki pattern went viral, and many people are now hand-wiring Obsidian to Claude Code so an agent maintains their notes for them.

The idea is simple, wherein you stop making the AI re-read raw notes on every question. Let it build a wiki that compounds.

As Karpathy put it, “LLMs don’t get bored, they don’t forget to update a cross-reference (backlinks), and can touch 15 files in one pass.”

But if you start doing it manually, it becomes a project in itself. You wire up the vault, the agents, the schedules, the integrations, etc,

So we sat down with Arjun, who actually built the open-source version of this, and we broke down what it looks like when the whole thing already works out of the box:

It just crossed 15K stars on GitHub.

It’s like Claude’s desktop app but open-source, with two things layered on top:

• A work brain: background agents index your emails, meetings, and notes into a living knowledge graph that updates itself as you work.

• Work surfaces: chat is not the best interface for real work, so you get an email client, a meeting note taker, a browser, and a code mode where you and the AI actually collaborate.

You can bring your existing Obsidian vault, connect Slack, X, and Fireflies, and let it run your day.

Here’s the full breakdown of what we covered in this session:

• 00:00: Intro

• 01:08: What is Roboat (an open source AI co-worker)

• 02:42: The second brain (a knowledge graph of your work)

• 04:01: Bringing your existing Obsidian vault in

• 04:46: Work surfaces

• 05:29: Meetings and automatic note-taking

• 06:53: Connecting Slack, X, and other sources

• 07:55: Background agents that run your day

• 09:24: Code mode (Claude Code and Codex)

• 10:18: Demo: from an email to written code

• 14:28: Guardrails: approvals and agent workspaces

• 17:15: Scheduling agents on a cron

• 18:52: The browser work surface (browser use)

• 20:42: Wrapping up: automating your whole day

• 22:44: Outro

Checkout Rowboat’s GitHub repo: https://github.com/rowboatlabs/rowboat

(don’t forget to star it 🌟)

We recently also wrote an article on the same idea, and we highly recommend reading it as well.

You can read it here →

Good day!

Code review assumes the author understood the codebase. An agent doesn’t carry that assumption.

It writes code that compiles and clears the obvious checks, enough to land a commit.

What’s actually wrong with it surfaces in CI later, after you’re already building on top of it.

Gitar, an AI-native code review and fix offering, now part of Sonar, is built for finding and fixing the issues in the pull request itself.

It reads the change with codebase context, not just the diff, catching bugs a syntax scan would miss.

When it finds something, it writes a patch and runs it against your CI. It doesn’t mark the job done until the build passes.

Sonar calls the full loop the Agent Centric Development Cycle, or AC/DC.

The agent writes, the Sonar Vortex engine verifies inside that same session, and Gitar fixes what’s wrong at the PR stage.

No human has to act on a comment first.

Teams running this combination see 44% fewer outages tied to AI-generated code.

Token usage drops too, by up to 36%, since agents spend less time re-parsing a codebase that isn’t piling up mess.

Start with AI-native code validation with Gitar from Sonar here →

Use AI-native code validation today