If you’re using GPT or Claude, you’re using the same model as everyone else, with the same capabilities, the same cost, and no competitive edge.

But if you take a small open-source model and fine-tune it on your specific task, it can outperform a model 100x its size, at a fraction of the cost and latency.

Devs typically associate fine-tuning with a painful setup, like curating datasets, labeling outputs, and hand-crafting reward functions.

In 2026, that’s no longer the case.

Modern fine-tuning using GRPO and RULER is redefining fine-tuning.

You can now train agents that genuinely improve through experience, without writing a single reward function or collecting a single labeled example.

Today, let’s walk through exactly how!

SFT vs. Reinforcement Fine-Tuning

In supervised fine-tuning (SFT), you collect input-output pairs and the model learns to imitate them.

The problem is that SFT teaches the model what to say, not how to succeed.

For agents that search, call APIs, and reason across multiple steps, imitation isn’t enough. You want improvement through trial and error.

Think of it this way:

• SFT = studying a textbook (memorizing answers to known questions)

• RL = on-the-job training (learning from trial, error, and feedback)

This is Reinforcement Fine-Tuning (RFT). You give the model a reward signal and let it discover the best strategies on its own.

How GRPO Works

GRPO (Group Relative Policy Optimization) is the most popular RFT algorithm today. It’s the same algorithm that powered DeepSeek-R1’s reasoning capabilities.

Essentially, instead of training a separate model to score responses, GRPO generates multiple completions and grades them relative to each other.

Here’s how it works for each prompt:

1. Sample a group: Generate N completions from the current model

2. Score each one: A reward function evaluates each attempt

3. Normalize within the group: Calculate relative advantage vs. the group average

4. Update the model: Reinforce above-average behaviors, suppress below-average ones

GRPO only needs relative rankings, not absolute scores. Whether completions score 0.3, 0.5, and 0.7 or 30, 50, and 70 doesn’t matter. Only the ordering drives learning.

ART: Agent Reinforcement Trainer

GRPO is powerful, but how do you actually apply it to a real-world agent?

ART (Agent Reinforcement Trainer) is a 100% open-source framework that brings GRPO to any Python application.

Most RL frameworks are built for simple chatbot interactions, involving one input, one output, and the job is done.

Real agents are fundamentally different. They search documents, invoke APIs, and reason across multiple steps before producing an answer.

ART is built for exactly this. It provides:

• Native support for tool calls and multi-turn conversations

• Integrations with LangGraph, CrewAI, and ADK

• Efficient GPU utilization during training

Architecture

ART splits into two parts: a Client and a Backend.

The Client is where your agent code lives. It sends inference requests to the backend and records every action into a Trajectory, the complete history of one agent run.

The Backend is where the heavy lifting happens. It runs vLLM for fast inference and Unsloth-powered GRPO for training. After each training step, a new LoRA checkpoint loads automatically into the inference server.

The full training loop

1. Client sends an inference request

2. Backend generates model outputs

3. Agent takes actions in the environment (tool calls, searches, etc.)

4. Environment returns a reward

5. Trainer updates the model via GRPO

6. A new LoRA checkpoint loads into the inference server

7. Repeat, with each cycle, the model gets a little better than before

RULER: RL without manual reward functions

Defining a good reward function has always been the hardest part of RL.

Training an email agent requires labeled correct answers. Training a code agent requires test suites. Each one is its own unique engineering project.

RULER (Relative Universal LLM-Elicited Rewards) eliminates this bottleneck entirely. It uses an LLM-as-judge to compare multiple agent trajectories and rank them, with no labeled data required.

It works because of two key insights:

• Asking an LLM “rate this 0-10” produces inconsistent results

• Asking “which of these 4 attempts best achieved the goal?” is far more reliable.

And since GRPO only needs relative scores, the absolute values don’t matter anyway.

The process is three steps:

1. Generate N trajectories for a scenario

2. Pass them to an LLM judge, which scores each from 0 to 1

3. Use those scores directly as rewards in GRPO

A practical example

We put together a fully working notebook that trains a 3B model to master how to use any MCP server through reinforcement learning using ART.

Simply provide an MCP server URL, and the notebook will:

1. Query the server’s tools

2. Generate a set of input tasks that use those tools

3. Train the model on those tasks using automatic RULER evaluation

You can find more examples to adapt and get started in the ART GitHub repo.

Here’s the GitHub Repo →

12 must-use features in Claude Code​

• CLAUDE .md is your project’s memory. It stores your stack details, conventions, and rules so Claude loads them at every session start.

• Permissions let you whitelist or block tools like Bash per session. If you’re working on anything production-facing, this is non-negotiable.

• Plan Mode makes Claude draft a step-by-step plan before touching any code. You get to approve or reject before anything runs.

• Rules let you set project-wide behavioral guardrails with specific dos and don’ts beyond what CLAUDE(.)md covers.

• Skills are reusable instruction files you store in .claude/skills/. Write them once and Claude follows them automatically every time.

• Hooks fire shell scripts on events like PreToolUse and PostToolUse, which makes them perfect for auto-linting or triggering tests.

• MCP connects Claude to databases, APIs, and services. This is how you give it real-world access beyond your codebase.

• Plugins let you add Docker, pytest, and VS Code extensions without writing any integration code.

• Slash Commands store workflow shortcuts in .claude/commands/ so you can trigger complex flows with a single keystroke.

• Subagents spawn parallel Claude instances that divide and conquer multi-step workflows simultaneously.

• Voice Mode lets you talk to Claude hands-free, which is great for quick queries while your hands are on the keyboard.

• Rewind lets you step back to any checkpoint in your session and restart cleanly from that point.

We covered the anatomy of the .claude folder in a recent issue.

​Read it here →​

👉 Over to you: Which features do you use the most in CC?

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Most RAG cost optimization happens at the model layer, like smaller models, fewer calls, and batching.

The retrieval payload itself rarely gets measured.

A typical setup retrieves 5 chunks per query, each around 300 tokens. That’s 1,500 input tokens before the LLM writes a single word, and at scale, that compounds.

But the bigger problem is accuracy. Enterprise documents repeat the same facts across multiple file versions.

When retrieved chunks say slightly different versions of the same thing, the LLM blends them. The answer sounds confident and is wrong in ways that are hard to catch.

Blockify (GitHub repo) sits between your raw docs and vector store.

Instead of splitting text into raw chunks, it uses a fine-tuned LLM to generate small, structured knowledge units called IdeaBlocks, where each one is built around one question and one validated answer. Average size: 98 tokens.

It runs on Intel Xeon CPUs, so no GPU server is needed to get started.

On a published benchmark, the IdeaBlock index outperformed raw chunked indexing by 13.55% on vector accuracy, using the same source documents and embedding model.

The token count dropped 3.09X as a direct result of the smaller unit size.

The cost drops because the quality improved, not separately from it.

You can find the Blockify GitHub repo here →

72 techniques to optimize LLMs in production

On an H100 running Llama 70B, a single inference request hits 92% GPU compute utilization during prefill, then drops to 28% during decode on the same hardware a moment later. The workload changed, not the GPU.

For context:

• Prefill processes the entire prompt in parallel and saturates tensor cores.

• Decode generates one token at a time and reads the full KV cache from HBM at every step, which makes it memory-bandwidth bound.

This asymmetry is why a single optimization never gets you very far, and why LLM inference prices have still fallen roughly 10x per year, with GPT-4-level performance going from $20 per million tokens in late 2022 to around $0.40 today.

Most of that drop came from the serving stack, and we put together this visual, which lists the techniques that go into optimizing LLMs in production.

Every technique in the grid above is a response to one of three bottlenecks: prefill compute, decode memory bandwidth, or the cost of everything that wraps the model.

Stacking enough of these techniques closes the 5-8x cost-efficiency gap between optimized vLLM or TensorRT-LLM deployments and naive FP16 inference.

Today, let’s walk through the nine layers, what each one actually solves, and how they stack up in a real production deployment.

We covered a lot more in the LLMOps course with implementations and engineering logic.

You can start reading it here →

1. Model compression

Model weights live in GPU memory all the time.

A 70B model in FP16 is 140GB before you load a single token of context. Compression attacks this usage directly.

• INT8 halves the memory vs FP16.

• INT4 cuts it 4x.

• FP8 gives you native tensor core support on Hopper and Blackwell, which means compression plus speedup.

GPTQ, AWQ, and SmoothQuant are the three main algorithms here.

• GPTQ uses Hessian-based second-order information

• AWQ preserves salient weights based on activation magnitudes,

• SmoothQuant handles both weights and activations at W8A8.

Distillation and pruning attack the parameter count itself rather than the bits per parameter.

Multi-LoRA serving is the escape hatch for multi-tenant deployments, where you keep one base model in memory and hot-swap small adapter weights per request.

We covered this specific pillar in

• Part 9 of MLOps course →

• Part 10 of MLOps course →

• Part 12 of LLOps course →

2. Attention and architecture

Standard attention is O(N²). At 128K context, this will have 16 billion computations, which is why naive attention is infeasible at long context even on H100-class hardware.

FlashAttention reorders the attention math to be IO-aware, avoiding materializing the full N×N matrix.

PagedAttention applies OS-style virtual memory to the KV cache, eliminating fragmentation.

MQA, GQA, and MLA attack the number of KV heads.

MQA shares one KV head across all queries, GQA groups them, MLA compresses keys and values into a low-rank latent. DeepSeek-V2 reported a 93.3% KV cache reduction from MLA alone.

Sliding window attention restricts each token to a local window. MoE activates only a subset of experts per token. These are architectural choices driven entirely by serving economics.

We covered this specific pillar in:

• Part 3 of LLMOps course →

• Part 13 of LLMOps course →

3. Decoding

Decode is memory-bound because every new token requires a full pass over the weights and KV cache.

• Speculative decoding sidesteps this by generating a draft with a cheap model, then verifying in parallel with the main model.

• Medusa attaches extra prediction heads to the model itself, so the same model can draft its own candidate tokens without needing a separate smaller model.

• EAGLE improves on this by predicting at the hidden-state level rather than the token level, which gives higher draft accuracy and better speedups.

• Lookahead decoding skips the draft model entirely. It generates and verifies multiple tokens in parallel from the main model alone.

• Prompt lookup decoding copies spans directly from the input prompt, which is surprisingly effective for tasks with heavy prompt-output overlap like summarization or code edits.

• Constrained decoding enforces grammars at the token level, which is how providers guarantee valid JSON.

• Multi-token prediction trains the model to emit several tokens per forward pass.

We covered this specific pillar in:

• Part 4 of LLMOps course →

• Part 13 of LLMOps course →

4. KV cache

The KV cache grows linearly with context length, and for long conversations, it dominates memory (learn KV caching here)

A 70B model with 4K context per request already consumes several gigabytes of KV just for a modest batch size.

• Prefix caching reuses KV across requests sharing the same prefix, which is why system prompts and few-shot examples are effectively free after the first request.

• KV offload tiers cold cache entries to CPU RAM or NVMe.

• KV cache quantization compresses the cache itself, separate from the weights.

• Token eviction methods like H2O and SnapKV drop low-attention tokens from the cache. SnapKV reports 92% KV compression at a 1024-token budget with a 3.6x decode speedup.

• Attention sinks, from the StreamingLLM paper, keep the first few tokens permanently in the cache to prevent long-context generation from going incoherent past the cache limit.

• Chunked prefill splits long prompts into smaller pieces so decode steps can interleave with prefill work.

We covered this specific pillar in:

• Part 13 of LLMOps course →

5. Batching and scheduling

LLM inference is memory-bandwidth bound during decode, which means the GPU is usually starved. Batching more requests together amortizes memory reads across more useful work.

• Continuous batching does this at the iteration level. As soon as one request finishes generating, a new one takes its slot mid-flight.

• Dynamic batching waits for a short window to group arriving requests. Batching 32 requests together cuts per-token cost roughly 85% with minor latency impact.

• Prefill-decode disaggregation splits the two phases onto separate GPU pools. Perplexity, Meta, and Mistral run this in production because co-locating prefill and decode on the same GPU means decode requests freeze every time a new prefill enters the batch.

• SLO-aware scheduling prioritizes interactive traffic over background jobs.

• Spot GPU scheduling runs preemptible workloads on cheap capacity.

• Request deduplication merges identical in-flight queries.

We covered this specific pillar in:

• Part 13 of LLMOps course →

• Part 14 of LLMOps course →

• Part 15 of MLOps course →

6. Parallelism and kernels

• Tensor parallelism splits weight matrices across GPUs.

• Pipeline parallelism splits layers.

• Expert parallelism shards MoE experts across devices.

• Sequence parallelism splits along the token dimension.

• CUDA graphs reduce kernel launch overhead, which matters because decode launches thousands of tiny kernels per second.

• Kernel fusion combines multiple operations into one launch.

• Torch compile produces fused kernels automatically via graph-level compilation.

We covered this specific pillar in:

• Part 13 of LLMOps course →

7. Application caching

The cheapest inference is the one you skip.

• Prompt caching reuses the KV state of static prefixes across calls. Anthropic reports up to 90% cost reduction and 85% latency reduction for long cached prompts.

• Semantic caching matches queries by embedding similarity rather than exact string match, which handles paraphrases.

• Exact-match caching is the hash-based baseline.

• Response caching stores completed outputs.

• Embedding deflection routes simple queries to a vector search without ever calling the LLM.

• Batch API endpoints run async jobs at roughly half the per-token price for non-realtime workloads

We covered this specific pillar in:

• Part 13 of LLMOps course →

• Part 14 of LLMOps course →

8. Input/output shaping

Output tokens cost 3-10x more than input tokens across every major provider.

Claude Sonnet 4 is $3 per million input versus $15 per million output, so trimming either side of the call translates directly into margin.

• Prompt compression with tools like LLMLingua achieves up to 20x compression with minimal quality loss.

• Context pruning drops irrelevant retrieved chunks before they reach the model.

• System prompt optimization trims static prefixes that bloat every request.

• Response length caps, structured output modes, and few-shot pruning all attack output volume.

• Context distillation summarizes long histories into a shorter state.

• RAG over long context is often cheaper than stuffing everything into the window. Retrieval keeps the prefill bill bounded.

We covered this specific pillar in:

• Part 5 of LLMOps course →

• Part 6 of LLMOps course →

• Part 7 of LLMOps course →

• Part 8 of LLMOps course →

9. Routing and cost

Not every query needs a frontier model.

• Model routing picks a smaller model when a smaller model suffices.

• Model cascading runs a cheap model first and escalates to a larger one only when confidence is low. Advisor strategy is somewhat similar to this:

• Classifier routing learns which queries go where.

• Multi-provider failover routes across APIs for reliability and cost.

• QoS tiers separate fast-and-cheap traffic from slow-and-high-quality.

• Task-specific fine-tuning lets a 7B model match a 70B model on a narrow domain.

• Function calling offloads deterministic logic to tools so the model doesn’t spend tokens computing what code could.

We covered this specific pillar in:

• Part 12 of LLMOps course →

• Part 14 of LLMOps course →

Putting it together

A serious production stack touches most of these.

A reasonable setup for a general-purpose API might run FP8 weights, GQA-based attention with FlashAttention kernels, PagedAttention for KV, continuous batching with prefill-decode disaggregation, prefix caching for system prompts, semantic caching at the application layer, prompt compression for long retrieved contexts, and model routing to send trivial queries to a small model.

The gap between this stack and a naive FP16 deployment with static batching is 5-8x on cost-per-token, and each technique alone moves the number only a small amount, which is exactly why the compounding across all nine layers is what defines a serious production setup.

We covered a lot more in the LLMOps course with implementations and engineering logic.

You can start reading it here →

Thanks for reading!

AI agents write code without knowing your dependency graph, quality profiles, or security rules. So when something goes wrong, CI catches it minutes later.

SonarQube Agentic Analysis moves that verification into the agent's inner loop.

During a regular CI run, SonarQube stores full project context, like dependencies, compiled artifacts, type information, and build configuration.

When the agent writes a file, it invokes SonarQube Agentic Analysis mid-workflow.

The engine restores that cached context, applies your team's quality profiles and security rules, and runs the same analysis your pipeline uses. Same precision as a full CI scan, in seconds.

The agent generates, verifies, fixes, re-verifies, and commits. PRs that pass quality gates the first time, without the back and forth.

Compatible with Claude Code, Cursor, Codex, Gemini CLI, and VS Code with Copilot. Direct API available for automated pipelines.

Core analysis is free during the beta period with a SonarQube Cloud Teams or Enterprise plan.

Get started with SonarQube Agentic Analysis here →

Thanks to Sonar for partnering today!

Evolution of Agent Landscape From 2022-26

The biggest shift in AI agents hasn’t been about making models smarter.

They do have their part but it has been more about making the environment around them smarter.

Here’s how agent engineering evolved in just 4 years, across three distinct phases:

Phase 1: weights (2022)

Everything was about the model itself. Bigger models, more data, better training. Scaling laws suggested that progress will come from more parameters.

RLHF and fine-tuning shaped behavior in this phase.

If you wanted a better agent, you trained a better model. This worked great for single-turn tasks.

But it hit a wall fast. Updating one fact meant retraining. Auditing behavior was nearly impossible. And personalization across millions of users from one frozen set of weights didn’t happen.

Phase 2: context (2023-2024)

A key realization that happened in this phase was that you don’t always need to change the model.

You can change what the model sees.

Prompt engineering, few-shot examples, chain-of-thought, and RAG led the way here.

Suddenly, the same frozen model could behave completely differently based on what you put in front of it.

Developers stopped fine-tuning and started iterating on prompts and retrieval pipelines instead. It was cheaper, faster, and surprisingly effective.

But context windows are finite. Long prompts get noisy. Models attend unevenly (the “lost in the middle” problem is real).

And every new session starts fresh with zero memory of what happened before.

Context made agents flexible. It didn’t make them reliable.

Phase 3: Harness engineering (2025-2026)

This is where we are now.

The question changed from “what should we tell the model?” to “what environment should the model operate in?”

The model is no longer the sole location of intelligence. It sits inside a harness that includes persistent memory, reusable skills, standardized protocols (like MCP and A2A), execution sandboxes, approval gates, and observability layers.

The model stays the same. What changes is the task it’s being asked to solve.

An example could be a coding agent asked to implement a feature, run tests, and open a PR.

Without a harness, the model must keep repo structure, project conventions, workflow state, and tool interactions all inside a fragile prompt.

With a harness, persistent memory supplies context, skill files encode conventions, protocolized interfaces enforce correct schemas, and the runtime sequences steps and handles failures.

So you have the same model but completely different reliability.

The pattern across all three phases is simple:

• weights encoded knowledge in parameters (fast but rigid)

• context staged knowledge in prompts (flexible but ephemeral)

• harnesses externalized knowledge into persistent infrastructure (reliable and governable)

Each phase didn’t replace the previous one but rather built on top of what existed.

Weights still matter and so does context engineering. But the center of gravity has moved outward.

The most consequential improvements in agent reliability today rarely come from changing the base model.

They come from better memory retrieval, sharper skill loading, tighter execution governance, and smarter context budget management.

Building better agents increasingly means building better environments for models to operate in.

There’s a great paper on this titled Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering.

You can read it here →

We also published this deep dive (article) on agent harness engineering, covering the orchestration loop, tools, memory, context management, and everything else that transforms a stateless LLM into a capable agent.

You can read the Agent Harness article here →

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

To understand why it matters, you need to see where it sits.

Level 1: Prompt → Response

Each call is stateless. The model can use tools/APIs within a single request but nothing persists. Most production LLM apps are sophisticated Level 1 wrappers.

Level 2: Interactive assistant

The platform handles persistence for you with memory, tools, files, connectors. ChatGPT and Claude live here. These are capable, but entirely reactive.

Level 3: Delegated execution

You define the goal and the agent owns the execution. Claude Code, Codex, and deep research operate here. Your task keeps running when you walk away, but the agent won’t start new work on its own.

Level 4: Autonomous scheduled operation

The agent runs on its own clock using cron, webhooks, or event triggers with persistent state across runs. OpenClaw with heartbeat, n8n with AI nodes, or the custom stacks devs stitch together.

Level 5: Self-building systems

Tools like Lovable and Bolt already go from prompt to deployed app. But the output is a web app that sits there until someone interacts with it.

Level 5 is different.

You can say “monitor my competitors’ blogs, store new posts in a table, and Slack me when they launch a product” and then the agent creates the database schema, connects the integrations, sets the schedule, and deploys a workflow that runs every morning on its own. No one needs to be present.

The workflow it just created runs on a schedule, maintains persistent state, and acts without human initiation.

Those are the exact characteristics of Level 4. So the output of a Level 5 agent is itself a Level 4 agent.

If you want to see this in practice, Sim (GitHub repo) shipped Mothership as an early implementation of this.

You can describe what you need, and it creates tables, wires workflow blocks, configures integrations, and sets the schedule.

It’s fully open-source (27k+ GitHub stars) so you can easily self-host it and see the full implementation on GitHub.

Here’s the GitHub repo →

Google solved an old RNN problem

A new paper from Google Research introduces “Memory Caching,” and the idea is quite simple.

Here’s the problem it solves:

Modern RNNs compress the entire input into a single fixed-size memory state. As sequences get longer, old information gets overwritten. That’s why they still struggle with recall-heavy tasks compared to Transformers.

Memory Caching addresses this by splitting the sequence into segments and saving the RNN’s memory state at the end of each segment.

When generating output, each token looks back at all these saved checkpoints, not just the current memory.

The complexity trade-off is elegant:

• Standard RNNs: O(L)

• Transformers: O(L²)

• Memory Caching: O(NL), where N = number of segments

You control the trade-off by choosing how many segments to cache. The model smoothly interpolates between RNN-like efficiency and Transformer-like recall.

The paper proposes four ways to use these cached memories:

1. Residual Memory: just sum all cached states (simplest)

2. Gated Residual Memory (GRM): input-dependent gates that weigh each segment’s relevance to the current token

3. Memory Soup: interpolates the actual parameters of cached memories into a custom per-token network

4. Sparse Selective Caching (SSC): MoE-style routing that picks only the most relevant segments

Gated Residual Memory (GRM) consistently performs best across tasks.

Under simplifying assumptions, hybrid architectures that interleave RNN and attention layers can be viewed as a special case of Memory Caching. This gives clean intuition for why hybrid models work. They’re implicitly caching memory states.

On recall-heavy tasks, Memory Caching significantly closes the gap between RNNs and Transformers. When applied to already strong models like Titans, it pushes them even further ahead on language understanding benchmarks.

Transformers still lead on the hardest retrieval tasks like UUID lookup at long contexts. But the direction is clear that you don’t need to choose between fixed memory and quadratic attention. There’s a useful middle ground now.

All experiments are at an academic scale (up to 1.3B params). Whether these gains hold at the frontier scale remains open.

This comes from the same team behind Titans and MIRAS, so it’s part of a larger research program on memory-augmented sequence models.

You can read the paper here →

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

You have 80,000 agent trajectories from production. You need to find top 100 worth reviewing to improve your agent.

No LLM allowed to evaluate trajectories. How will you do this?

Let’s look at some approaches.

The simplest solution one could start with is random sampling. Pick 100 random trajectories and review.

But most production agents handle routine requests just fine, so you end up wasting a big chunk of your annotation budget.

Another approach can filter for longer conversations since 10+ user messages means more complexity.

But longer conversations skew heavily toward outright failures. You’ll surface obvious breakdowns but miss subtle issues hiding in conversations where the agent technically succeeded.

A recent paper from DigitalOcean takes a new approach.

It computes lightweight behavioral signals directly from the trajectory data using deterministic rules.

The signals fall into three groups:

1) Interaction signals:

• If a user rephrases the request or corrects the agent, that’s misalignment.

• Agent repeating itself is stagnation.

• User abandoning the agent is disengagement.

• User confirming something worked is satisfaction.

All are detected through normalized phrase matching and similarity checks.

2) Execution signals:

• A tool call that doesn’t advance the task is a failure signal.

• Repeated calls with identical or drifting inputs indicate a loop.

These are straightforward to extract from execution logs.

3) Environment signals, like rate limits, context overflow, and API errors.

• Useful to diagnose but not for training since they reflect system constraints, not agent decisions.

Each trajectory gets scored based on which signals fire, and you sample the highest-signal ones for review.

On τ-bench, they compared all three approaches on 100 trajectories:

• Random sampling hit a 54% informativeness rate.

• The length-based heuristic reached 74%.

• Signal-based sampling reached 82%.

This means roughly 4 out of every 5 trajectories are genuinely useful to improve the agent.

In fact, among conversations where the agent completed the task correctly, signal sampling still identified useful patterns in 66.7% of cases vs. 41.3% for random.

These are the subtle issues like policy violations, inefficient tool use, and unnecessary steps that don’t break the task but still matter for optimization.

The whole framework runs without any LLM overhead and can sit always-on in a production pipeline.

If you want to see this in practice, this signal-based approach is already integrated into Plano, an open-source AI-native proxy that handles routing, orchestration, guardrails, and observability in one place.

Here’s the Plano GitHub repo →

Here’s the paper on arxiv →

👉 Over to you: What is your approach to solve this?

10 Must-use Slash Commands in Claude Code

Setting up shell aliases is such a natural part of working in a terminal that most developers do it almost reflexively. If you run a command often enough, you alias it.

With Claude Code prompts, though, devs typically skip this step entirely and keep retyping the same 10-15 line instructions from memory, like their code review checklist, test gen constraints, pre-commit scan...and all this session after session.

The real cost isn’t just the repetition you do as a dev, but the prompt drift.

Every time you retype a prompt from memory, the wording shifts slightly. For instance, you might forget a constraint or phrase the expected output format differently.

With shell commands, this doesn’t matter because they’re deterministic, but with an LLM, slightly different phrasing may produce noticeably different output.

Claude Code’s custom commands fix both problems.

You can save a markdown file in .claude/commands/, and it becomes a slash command you can invoke with identical instructions every time.

The prompts are version-controlled through Git, so your whole team runs the same commands, and when someone improves a prompt, everyone gets the update on their next pull.

This is the same pattern Boris Cherny described in his thread on Claude Code workflows, where his every repeated workflow becomes a command, checked into Git, and shared with the team:

Let’s walk through how to set them up, then the 10 commands that have been most useful in my workflow. I’ll demo each one on a real ML inference service (FastAPI, scikit-learn, Alembic) so you can see the actual output, with full prompt templates you can drop into your own project.

How custom commands work

A custom command is a Markdown file inside a .claude/commands/ directory. The filename becomes the command name.

# Project-scoped (shared via Git, shows as "(project)" in autocomplete):
your-repo/.claude/commands/preflight.md  →  /preflight

# User-scoped (personal, works in all projects):
~/.claude/commands/orient.md  →  /orient

# Subdirectories create prefixed commands:
.claude/commands/db/migrate.md  →  /db:migrate

The file content is the prompt that gets sent to Claude when you run the command. You can use $ARGUMENTS as a placeholder for anything typed after the command name.

For instance, running “/dissect src/auth/session.ts” substitutes $ARGUMENTS with “src/auth/session.ts“.

You can also inject dynamic context using shell commands with the !command syntax:

## Current state
- Branch: !`git branch --show-current`

- Staged changes: !`git diff --cached --stat`

- Last 3 commits: !`git log --oneline -3`

Claude runs those shell commands before processing the prompt, so the context is always fresh.

Lastly, an optional YAML frontmatter at the top of the file lets you pre-approve tools (so Claude doesn’t ask for permission on every git call), set a model override, or add a description:

---
description: Pre-commit check for debug artifacts and code smells
allowed-tools: Bash(git *), Bash(grep *), Read, Glob
---

That’s the entire system, which includes a markdown file, an optional YAML header, and $ARGUMENTS for dynamic input.

Below are the 10 commands we’ve found most useful in practice:

The newsletter ahead is a bit too long to share over email due to size constraints.

We have shared the full setup guide, with usage videos and prompts here →

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

And the “memory” you feel when chatting with ChatGPT is an illusion created by re-sending the entire conversation history with every request.

That trick works for casual chat. It falls apart the moment you try to build a real agent.

Here are 7 failure modes show up the instant you skip memory:

1. Context amnesia: the agent asks for information you already gave it

2. Zero personalization: every interaction feels generic

3. Multi-step task failure: intermediate state silently drops mid-task

4. Repeated mistakes: no episodic recall means the same errors, forever

5. No knowledge accumulation: every session starts from scratch

6. Hallucination from gaps: when context overflows, the model invents

7. Identity collapse: no continuity, no trust

The obvious response is “throw more context at it.” That’s why 128K and 200K token windows feel like they should solve everything.

They don’t.

Accuracy drops over 30% when relevant information sits in the middle of a long context. This is the well-documented effect.

Context is a shared budget. Details like the system prompts, retrieved docs, conversation history, and output…all fight for the same tokens.

Even at 100K tokens, the absence of persistence, prioritization, and salience makes raw context length insufficient.

Memory isn’t about cramming more text into the prompt. It’s about structuring what the agent remembers so it can find what matters.

The cognitive science frame that actually helps

Lilian Weng’s 2023 formulation has become the default framework here.

Agent = LLM + Memory + Planning + Tool Use.

The four co-equal pillars.

Her taxonomy borrows from cognitive science, where human memory splits into three systems:

• Sensory memory captures raw perceptual input and holds it for a fraction of a second. Only the portions you pay attention to get passed forward.

• Working memory is where active thinking happens. It holds roughly 7±2 items at a time (Miller’s 1956 finding). Lose focus, and the contents disappear.

• Long-term memory is durable storage with no practical capacity limit. Retrieval is the bottleneck: you can store millions of things and still fail to recall the one you need.

Each maps directly to a component in modern agent architectures:

Long-term memory itself splits further:

• Episodic: specific past events (”on Tuesday, the PostgreSQL cluster went down”)

• Semantic: facts and concepts (”PostgreSQL is a relational database”)

• Procedural: skills and workflows (”when a user asks for a refund, first check the purchase date”)

The bridge between episodic and semantic is memory consolidation: repeated specific events distilling into general knowledge.

An agent that notices “users consistently prefer executive summaries” across dozens of interactions should turn that into a reusable rule. Without consolidation, your agent replays individual events rather than learning from them.

The minimal agent, and what breaks first

If you strip away the frameworks, an agent is a loop which goes like: perceive, think, and act.

If you tell it “I have 4 apples,” then ask “I ate one, how many left?” and it has no idea what apples you’re talking about. Each call exists in isolation.

Layer 1: The Python list

The first fix everyone reaches for is maintaining the interaction in a messages list:

Multi-turn works now. The apples question gets answered correctly because the full conversation re-ships with every call.

Two problems show up fast:

• The list grows unbounded. Around turn 200, you hit the context ceiling and the oldest messages silently drop. The user’s name from turn 1 disappears long before yesterday’s throwaway joke. No prioritization, just strict chronological order.

• Everything lives in RAM. The moment the Python process ends, your agent has no idea who you are.

Layer 2: Markdown files for persistence

The next move is writing memory to disk.

Markdown is a natural fit since they are human-readable, Git-friendly, and the agent can read it back as plain text. Claude Code uses exactly this pattern with CLAUDE.md and MEMORY.md files:

Persistence is solved because if you restart the script, and the conversation is still on disk. You could also maintain a separate facts file that the agent extracts over time:

You can open the file in any editor, see exactly what the agent knows, and fix it by hand. Genuinely useful for prototyping.

With 4 facts, this works perfectly. Load the entire file into context and the LLM handles any question about Sarah, her company, or her industry.

Now fast-forward three months. Your agent has 2,000 extracted facts and 200 conversation logs. That’s 500K+ tokens of markdown on disk, and your context window is 128K.

You can no longer load everything. You need to selectively retrieve only the facts relevant to the current query. With flat files, your only option is keyword search:

At small scale, markdown files work. At real scale, they force keyword retrieval, and keywords can’t handle synonyms, paraphrases, or connections across facts.

The information is on disk. But you can’t load all of it, and keyword search is too brittle to find the right pieces.

OpenClaw, for instance, stores memory as markdown checkpoint files, and over weeks of daily use, earlier facts quietly slip away as context accumulates and gets compacted. The storage is there but the retrieval isn’t.

Layer 3: Vector search

Next step is to chunk the markdown, embed them, and search by cosine similarity, which solves the synonym problem.

But then you face a new problem. Consider these three facts in your vector DB:

User asks: “Was Alice’s project affected by Tuesday’s outage?”

The query mentions Alice and Tuesday’s outage, so vector search ranks the first and third facts high.

But the critical bridge, “Project Atlas uses PostgreSQL,” mentions neither Alice nor Tuesday. It’s the connecting piece, and it’s the one that won’t surface.

Each fact is an isolated point in embedding space. The connective tissue linking them is invisible to vectors.

This isn’t an edge case but rather the normal shape of real-world questions.

Business knowledge is inherently relational and any question that crosses two or more hops exceeds what flat vector retrieval can answer.

The capability matrix

Each layer fixes the previous pain but reveals a deeper one:

You need persistence, semantic understanding, and relational reasoning in a single memory layer.

Building this yourself means gluing together a vector database, a graph database, a relational store, an entity extractor, a deduplication pipeline, and an edge-weighting system.

That’s weeks of infrastructure work before you write a single line of agent logic.

Cognee as the memory layer

Cognee is an open-source knowledge engine built for agent memory. It combines vector search with knowledge graphs and a relational provenance layer into a single system.

The entire API surface is four async calls:

Under the hood, these four calls encapsulate a three-store architecture.

Each store captures a dimension of knowledge the others can’t:

• Relational store → provenance: where data came from, when it was ingested, who has access

• Vector store → semantics: what content means, what it’s similar to

• Graph store → relationships: how entities connect, what causes what, who reports to whom

If you flatten any of these, you’ll lose information that matters for retrieval accuracy.

What cognify actually does?

cognee.cognify() runs a multi-stage pipeline that converts raw text into structured, interconnected knowledge:

1. Document classification by type and domain

2. Permission checking for multi-tenant access control

3. Chunk extraction that respects paragraph structure (not fixed-size cuts)

4. Entity and relationship extraction via LLM, with automatic deduplication through content hashing

5. Summary generation for efficient retrieval

6. Dual indexing into the vector store (embeddings) and graph store (edges)

The deduplication step matters more than it sounds. If the same entity shows up across 50 documents, Cognee merges it into a single graph node with 50 inbound edges.

Your agent no longer sees “Alice” as 50 different strangers. And the pipeline is incremental by default so only new or updated files get reprocessed.

Every graph node has a corresponding embedding. This dual representation is the core trick since it allows you to enter through vectors (find semantically similar content) and exit through the graph (follow relationships to connected entities), or the reverse.

That’s what makes multi-hop queries work without sacrificing semantic search.

Memify: memory that learns

memify() is another interesting practical detail, which runs an RL-inspired optimization pass over the graph:

• Strengthening useful paths that led to good retrieval

• Pruning stale nodes that haven’t been touched

• Auto-tuning edge weights based on real usage

• Adding derived facts by identifying implicit relationships

A customer support agent’s graph naturally strengthens paths through product docs and refund policies while letting rarely-queried HR edges decay. The graph develops its own sense of relevance over time.

Fourteen retrieval modes

Cognee ships 14 search modes but these are the most useful ones:

Building a real agent with Cognee memory

Here’s the complete pattern wiring Cognee into the perceive-think-act loop:

The memory cycle follows: ingest, extract, store, retrieve, respond, store again.

Each turn enriches the knowledge graph, and incremental processing means you only pay to index new content.

Session memory handles pronoun resolution automatically:

Multi-tenancy is built in at the graph level with per-dataset permissions (read, write, delete, share).

Takeaway

If you’re building an agent today, the real starting question is: “what does my agent need to remember, and what kind of questions will it answer?”

If your queries only need similarity search (”find conversations like this one”), vector-only memory works.

The moment queries cross entity boundaries (”Was Alice’s project affected by Tuesday’s outage?”), you need graph traversal.

You can wire together separate vector, graph, and relational stores yourself. Teams that go this route typically burn weeks on infrastructure for a memory layer that still doesn’t learn from its own usage.

Cognee collapses that into four API calls. Embedded defaults get you running in minutes. Swappable backends (Postgres, Qdrant, Neo4j) take you to production without changing your agent code.

Intelligence requires structure, not just storage. The three storage paradigms (relational, vector, graph) aren’t competing options. They’re complementary layers of the same memory system.

Check out Cognee on GitHub →

Thanks for reading!

Read the full Part 1 deep dive here →

Diffusion LLMs Part 1

It builds a complete understanding from first principles:

• how autoregressive generation is structurally memory-bandwidth bound)

• why Gaussian noise can’t work on discrete tokens

• how masked diffusion solves this with an ELBO-derived training objective

• the math behind the forward and reverse processes

• unmasking strategies

• block diffusion for KV cache compatibility

• and a detailed engineering comparison between the two paradigms.

Read the full Part 1 deep dive here →

Why care?

Every production LLM today, GPT-4, Claude, Gemini, LLaMA, generates text the same way: one token at a time, left to right.

Each token requires loading the full model weights through GPU memory, performing a tiny computation, and then loading all the weights again for the next token. On an A100, this means roughly 1 FLOP per byte of data moved, while the GPU is designed for 100+ FLOPs per byte.

Diffusion LLMs take a completely different approach. They start with a fully masked sequence and iteratively unmask all tokens in parallel, using bidirectional attention at every step. This shifts inference from memory-bandwidth bound to compute-bound, which is exactly where modern GPUs are efficient.

The results are catching up fast. Block diffusion (BD3-LM) is within 0.5 perplexity points of autoregressive on LM1B. LLaDA at 8B parameters matches LLaMA 3 on MMLU and exceeds it on TruthfulQA and HumanEval. And models like Dream 7B are already being served in production with SGLang.

Understanding how it works at a mathematical level, from the forward masking process to the ELBO objective to block-level KV caching, is going to be increasingly valuable as these models scale.

You can read the Part 1 here →

👉 Over to you: Do you think the future of LLM generation is pure diffusion, pure autoregressive, or some hybrid of the two?

Thanks for reading!

Unsloth Studio is a local, browser-based GUI for fine-tuning LLMs without writing any code.

It wraps the training pipeline in a clean interface that handles model loading, dataset formatting, hyperparameter configuration, and live training monitoring.

The process to fine-tune the latest Gemma 4 is simple:

1. Open the Unsloth Colab notebook (available here).

2. Pick your model and dataset

3. Hit start training

You can find the notebook here →

Advisor strategy in LLMs to optimize token costs

Yesterday, Anthropic shipped an “advisor tool” in the Claude API that lets Sonnet or Haiku consult Opus mid-task, only when the executor needs help.

The benefit is that you get near Opus-level intelligence on the hard decisions while paying Sonnet or Haiku rates for everything else. So frontier reasoning only kicks in when it’s actually needed, not on every token.

Back in February, UC Berkeley published a paper called “Advisor Models” that trains a small 7B model with RL to generate per-instance advice for a frozen black-box model.

The paper’s approach was to take Qwen2.5 7B, train it with GRPO to generate natural language advice, and inject that advice into the prompt of a black-box model.

The black-box model never changes, and the advisor learns what to say to make it perform better.

To test it, they found that GPT-5 scored 31.2% on a tax-filing benchmark. But adding the trained advisor took that to 53.6%.

Moreover, on SWE agent tasks, a trained advisor cuts Gemini 3 Pro’s steps from 31.7 to 26.3 while keeping the same resolve rate.

Anthropic’s advisor tool takes a different path to the same idea. Sonnet runs as the executor to handle tools and iteration.

When it hits something it can’t resolve, it consults Opus, gets a plan or correction, and continues.

Sonnet with Opus as advisor gained 2.7 points on SWE-bench Multilingual over Sonnet alone, while costing 11.9% less per task.

Haiku with Opus scored 41.2% on BrowseComp. Haiku alone scored 19.7%.

Implementation-wise, it’s a one-line API change. The advisor tokens bill at Opus rates, and the advisor typically generates only 400-700 tokens per call.

response = client.messages.create(
    model="claude-sonnet-4-6",  # executor
    tools=[
        {
            "type": "advisor_20260301",
            "name": "advisor",
            "model": "claude-opus-4-6",
            "max_uses": 3,
        },
        # ... your other tools
    ],
    messages=[...]
)

So the combined cost stays well below running Opus end-to-end.

Both approaches point to the same thing that you don’t need the most powerful model on every token.

You need it at the right moments, for the right inputs.

Here’s the paper by UC Berkeley →

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

A key limitation of coding agents today is that they can’t fetch live web data on their own.

And platforms like LinkedIn, X, and Reddit (where a ton of developer discussions happen) are notoriously hard to scrape due to bot detection, CAPTCHAs, and JavaScript rendering.

Bright Data open-sourced a skills plugin (brightdata/skills), which provides live web access directly into Claude Code, Cursor, Windsurf, and 40+ other coding agents, with automatic handling of all those access barriers.

You can use it to scrape any webpage as clean markdown, run Google searches that return structured JSON, and extract structured data from 40+ platforms, including Amazon, LinkedIn, YouTube, TikTok, and Reddit.

Here’s the GitHub repo →

Thanks to Bright Data for partering today!

How to build Agents that don’t fail in production

Here are some damages caused by AI in production:

• Replit’s Agent wiped out a production DB.

• Zillow lost $304M due to its home-buying AI.

• iTutor paid $365k when AI auto-rejected old applicants.

Today, let’s learn how to build Agents that don’t fail in production (with code).

One primary challenge with customer-facing AI agents is that they either escalate the issue too quickly to a human agent or confidently mislead the user.

Also, it’s not just the frequency of mistakes, but their severity, especially when even the worst 0.001% could matter.

Talking specifically about user-facing use cases, placing control guidelines and embedding business logic into instruction-following Agents is helpful.

Let’s use Parlant (open-source with 18k stars) to build a compliant conversational finance agent that processes and approves loans.

Parlant is a framework to build customer-facing agents that behave exactly as instructed.

1️⃣ Create Agent

We start by defining a loan-approval Agent.

Next, we declare some domain-specific terms the Agent may need to know to answer confidently.

2️⃣ Journey

Parlant introduces the idea of Journeys. They inform the Agent about the multi-step conversational flow that helps it guide the user through the conversation as intended.

Check this loan approval journey:

This provides the following Journey:

• Determine the type of loan the user wants

• Collect loan and income-related details

• Call the tool to check eligibility

• End if not eligible, else ask them to upload docs

• Call the tool to process the uploaded docs

• End if invalid docs, else loan approved

3️⃣ Guidelines

Guidelines in Parlant tell the agent how to approach specific situations through condition-action pairs.

This provides more control over the agent’s behaviour.

This reduces misalignments by ensuring the Agent’s behavior aligns with the business needs.

We can do more things to embed more control, but this simple setup itself gives a powerful instruction-following Agent.

In the video below, no matter what we try to get a loan approved, the Agent refuses to do so.

You can also inspect the exact reasoning.

This was just the primer on what you actually do with Parlant. We are still exploring and will cover more on building extensive, production-grade, and user-facing AI systems that don’t fail.

Building Agents is about engineering “behavior” at scale. So you cannot vibe-prompt an Agent and expect it to work.

Parlant gives the structure to build Agents that behave exactly as instructed.

In the meantime, here’s the Parlant GitHub repo →

​The code for today’s issue is available in this GitHub repo →​

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Karpathy’s LLM Wiki compiles raw sources into a persistent MD wiki with backlinks and cross-references.

The LLM reads papers, extracts concepts, writes encyclopedia-style articles, and maintains an index. The knowledge is compiled once and kept current, so the LLM never re-derives context from scratch at query time.

This works because research is mostly about concepts and their relationships, which are relatively stable.

But this pattern breaks when you apply it to actual work, where context evolves across conversations constantly, like deadlines, plans, meetings, etc.

A compiled wiki would have a page about the project, but it wouldn’t track ground truth effectively.

Tracking this requires a different data structure altogether, which is not a wiki of summaries, but a knowledge graph of typed entities where people, decisions, commitments, and deadlines are separate nodes linked across conversations.

​Rowboat is an open-source implementation of exactly this, built on top of the same Markdown-and-Obsidian foundation that Karpathy uses, but extended into a work context.

The way it works is that it ingests conversations from Gmail, Granola, and Fireflies, and instead of writing a summary page per topic, it extracts each decision, commitment, and deadline as its own MD file with backlinks to the people and projects involved.

That’s structurally different from a wiki, because a wiki page about “Project X” gives you a summary of what was discussed.

A knowledge graph gives you every decision made, who made it, what was promised, when it was promised, and whether anything has shifted since.

It also runs background agents on a schedule, so something like a daily briefing gets assembled automatically from whatever shifted in your graph overnight. You control what runs and what gets written back into the vault.

You bring your own model through Ollama, LM Studio, or any hosted API, and everything is stored as plain Markdown you can open in Obsidian, edit, or delete.

​You can find the GitHub repo here →

Rowboat GitHub Repo

We are working on a hands-on demo for this and will share that in the coming week!

TL;DR: Karpathy’s LLM Wiki compiles research into a persistent Markdown wiki. It works well for concepts and their relationships, but breaks down for real work where the context evolves over time. Rowboat builds a knowledge graph instead of a wiki, extracts typed entities with backlinks, and runs background agents that act on that accumulated context. Open-source, local-first, bring your own model.Karpathy nailed the foundation. The next layer is here.

16 AI Agent Skills for AI Engineers

Claude Code’s .claude/ skills system lets you package reusable instructions, workflows, and tool configurations into portable folders that any agent session can pick up. The ecosystem around this has grown fast.

Here are 16 powerful Agent skills for AI engineers:

• ​Superpowers: A structured dev workflow that forces Claude to brainstorm, plan, and test before writing any code. Useful when you want rigor over speed.

• ​InsForge: Semantic backend layer that exposes auth, database, storage, and functions through one agent-friendly API. Think of it as a unified backend for agents.

• ​Bright Data Skills: Teaches Claude to orchestrate 60+ MCP tools for web scraping and structured data extraction. Handles the messy parts of live web access.

• ​Context7: MCP server that feeds live, version-specific library docs directly into Claude’s context. No more hallucinated APIs from outdated training data.

• ​Claude-Mem: Persistent memory plugin that auto-captures sessions and reinjects relevant context into future ones. Solves the “Claude forgot everything” problem between sessions.

• ​Everything Claude Code: Curated skills and rules collection with smart token-saving compaction at logical breakpoints. A good starting point if you’re building your own .claude/ setup.

• ​Planning with Files: Persistent markdown files for planning, progress tracking, and knowledge storage across sessions. Simple approach, surprisingly effective for multi-session projects.

• ​Sentry Security Review: Security review skill built on 15 years of real Sentry patches and Django ORM pitfalls. Catches the kind of bugs that only show up in production.

• ​Frontend Design: Official Anthropic skill for distinctive, non-generic UI output with bold design choices. Ships with Claude Code and pushes past the default “looks like every other AI-generated UI” problem.

• ​Web Quality Skills: Lighthouse and Core Web Vitals optimization for performance, accessibility, and SEO. Bakes web quality checks directly into the agent loop.

• ​n8n-MCP: MCP server with docs and schemas for all 1,396 n8n automation nodes. If you’re building automations with n8n, this gives Claude full visibility into the node catalog.

• ​Claude-Reflect: Captures your repeated corrections and turns them into reusable commands with human review. The agent learns your preferences over time instead of making the same mistakes.

• ​cc-DevOps Skills: Generator and validator loops for Terraform, Kubernetes, Docker, and CI/CD configs. Generates infra code, then validates it before you apply.

• ​Agent Sandbox: Isolated E2B cloud sandboxes for building, hosting, and testing apps without touching local files. Good for when you want the agent to experiment freely without risk.

• ​Agile Workflow: Full agile delivery pipeline with multi-model parallel review via Codex and Gemini agents. Brings structured software delivery practices into the agent workflow.

• ​Claude Code Plugins+: Plugin directory with a CLI package manager for searching and installing niche skills. Think npm but for Claude Code skills.

The .claude/ skills folder is becoming the package manager layer for agent behavior. Each of these skills is a self-contained instruction set that shapes how Claude approaches a specific type of work.

The interesting thing to note here is that skills aren’t just prompts. They combine instructions, file templates, tool configurations, and validation loops into composable units. The best ones encode real practitioner knowledge (like Sentry’s 15 years of security patches) into something an agent can apply consistently.

👉 Over to you: Which skills are you using with Claude Code, and have you built any custom ones for your workflow?

Thanks for reading!

InsForge (open-source) solves the most frustrating bottleneck in AI-assisted development: backend configuration.

InsForge GitHub Repo

Agents can build a beautiful frontend in minutes, set up API routes, and lay out the component architecture. But the moment it needs to enable auth or configure a database, it completely falls apart.

The reason is that every backend platform today (Firebase, Supabase, AWS) was designed for humans clicking through dashboards. When agents try to interact with these platforms through MCP servers, they get fragmented context like table names without schema details or auth endpoints without security configs. So agents end up guessing, hallucinating, and generating broken code.

​InsForge​ fixes this at the infrastructure level rather than the tooling level. It introduces a semantic layer where every backend primitive (auth, database, storage, AI features) is exposed as structured, machine-readable capabilities with metadata, constraints, and documentation baked in.

Primitives are also aware of each other, so auth knows about database permissions and storage understands access policies.

Because agents get a complete, structured context instead of inferring what’s missing, InsForge delivers:

• Roughly 2x more accuracy than Supabase MCP

• 1.6x faster task completion

• 30% better token efficiency

To test this out, we built a full ChatGPT clone with auth, database, storage, and AI integration, built entirely with Claude Code using InsForge as the backend. No manual configuration was needed, not because of any magic, but because the agent could reason about the entire backend as one coherent system.

InsForge works with any AI coding agent, including Cursor, Claude Code, Windsurf, and Codex. You can use all the primitives together or just pick what you need, like database only or auth only.

It’s fully open-source under Apache 2.0.

​Find GitHub repo here →​ (don’t forget to star it ⭐️)

Six Key Metrics for AI Agent Evaluation

An agent that completes a task in 3 tool calls and one that takes 9 calls (retrying, backtracking, calling the same API twice) can both score 1.0 on task completion.

However, end-to-end scoring won’t flag this difference, but your token bill and latency will.

Evaluating agents properly means going deeper than the final output. You need to check if the agent planned well, followed its plan, called the right tools with the right arguments, and did it all without wasted steps.

Today, let’s look at how you can do end-to-end Agent evaluation in a few lines of code using the open-source DeepEval evaluation framework (14k+ stars).

It ships six agentic metrics that cover all of this, plus a conversation simulator that auto-generates multi-turn test cases from scenario definitions.

Two layers of agent evaluation

DeepEval’s six metrics operate at two levels, based on what part of the agent’s execution they inspect:

Full-trace metrics (read the entire agent execution via @observe tracing):

• PlanQualityMetric evaluates whether the agent’s generated plan is logical, complete, and efficient for the task.

• PlanAdherenceMetric compares the plan against actual execution to check if the agent followed its own strategy or deviated mid-run.

• TaskCompletionMetric scores whether the agent accomplished the user’s task based on the full trace.

• StepEfficiencyMetric penalizes unnecessary or redundant steps even if the task was completed.

Component-level metrics (zoom into tool calls at a specific @observe span):

• ToolCorrectnessMetric compares tools_called against expected_tools to verify the agent picked the right tools.

• ArgumentCorrectnessMetric validates that the input parameters passed to each tool call were correct for the task.

Using them together is important because an agent can score 1.0 on TaskCompletion but 0.4 on StepEfficiency. If it called the same API three times to get a result, it should have cached. You’d never catch that with a single pass/fail metric.

Evaluating planning and execution from traces

Before running any metrics, we need an application to evaluate. Here’s a minimal travel booking agent with three functions:

generate_plan handles reasoning (what steps to take), execute_plan handles action (calling tools to complete the task), and travel_agent orchestrates both. This is a stand-in for any agent you’d build with OpenAI, LangGraph, or CrewAI.

To evaluate this agent with DeepEval, we need to make its execution visible. The @observe decorator does this without changing any logic:

The type parameter tells DeepEval the role each function plays: "agent" for the orchestrator, "llm" for reasoning, "tool" for action. Full-trace metrics read the entire trace starting from the "agent" span. Component-level metrics zoom into a specific span like "tool" or "llm".

Next, define a dataset of goldens (test inputs) to evaluate against:

Each Golden is one test scenario. In practice, you’d have 20-50+ goldens covering edge cases (wrong dates, ambiguous destinations, cancellations).

Now pass the four full-trace metrics to evals_iterator(), run the agent inside the loop, and DeepEval reads the execution trace to score each one:

Here’s what we got on our dummy agent:

The scores tell a clear debugging story. PlanQuality flagged that the steps (”search”, “compare”, “book”) lacked operational detail. PlanAdherence confirmed the agent didn’t execute those steps in the trace. StepEfficiency caught that the separate planning phase was redundant. TaskCompletion still passed because the final output was correct, which is exactly why you can’t rely on it alone.

Evaluating tool calls at the component level

Full-trace metrics (used above) tell you whether the overall execution worked.

Component-level metrics zoom into a specific span (typically the LLM call that decides which tools to invoke) and evaluate tool selection and argument quality independently.

The two metrics here work differently under the hood.

• ToolCorrectnessMetric is reference-based. It compares actual tools called to the expected tool calls.

• ArgumentCorrectnessMetric is referenceless. It uses an LLM judge to evaluate whether the arguments make sense given the input, without needing expected values.

You can attach both metrics to a specific component inside a traced agent using @observe(metrics=[...]):

Each metric pulls the fields it needs from the same test case.

Simulating conversations for agent testing

Manually writing multi-turn test cases doesn’t scale. You can use DeepEval’s ConversationSimulator generates realistic conversations from scenario definitions:

Each ConversationalGolden defines a scenario (what the user wants), an expected outcome (what should happen), and an optional user description (persona for the simulated user).

The simulator plays the user role and interacts with your agent for the specified number of turns, stopping early if the expected outcome is reached. The output is a list of ConversationalTestCase objects with fully populated turns, like below:

Running evals on simulated conversations

Once you have simulated test cases, evaluate them with conversational metrics:

• ConversationCompletenessMetric extracts every user intention from the conversation and checks if each one was satisfied by the agent.

• TurnRelevancyMetric scores each individual assistant response for relevance to the user’s most recent message.

A conversation can score high on completeness (all intentions met) but low on relevancy if the agent went off-topic in intermediate turns before eventually getting to the answer.

Here’s the output of one of the simulated conversational runs:

Putting it together

The full workflow for agent evaluation in DeepEval:

1. Define scenarios as ConversationalGolden objects (scenario, expected outcome, user profile).

2. Use ConversationSimulator to generate realistic multi-turn test cases against your agent.

3. Apply full-trace metrics (PlanQualityMetric, PlanAdherenceMetric, TaskCompletionMetric, StepEfficiencyMetric) via evals_iterator to evaluate the overall execution.

4. Apply component-level metrics (ToolCorrectnessMetric, ArgumentCorrectnessMetric) via @observe(metrics=[...]) on the LLM component to evaluate tool-calling.

5. Apply conversational metrics (ConversationCompletenessMetric, TurnRelevancyMetric) on simulated multi-turn test cases.

6. Run evaluate() and get scored results with reasons for every metric.

Once the scenarios are defined, you can simulate hundreds of conversations and re-run them every time you change a prompt, swap a model, or update a tool.

The metrics break down exactly where things degrade, like plan quality can drop from 0.9 to 0.7 after a prompt change, or tool correctness may fall to 0.5 when the agent needs to chain three API calls.

Here’s the DeepEval GitHub repo →

Agent evaluation guide in the docs →

Conversation Simulator docs →

👉 Over to you: how are you evaluating your agents today, and are you testing the reasoning and action layers separately, or just looking at the final output?

Thanks for reading!

Before foundation models, building an AI feature involved collecting and labeling training data, training a custom model from scratch, and only then integrating it into a product. This took months and a massive compute investment before teams could even test whether users wanted the feature.

Foundation models removed that bottleneck because they come pre-trained and accessible via API. Teams can now call GPT-4 or Claude with zero-shot or few-shot prompts, ship an MVP in days, validate user demand first, and only then invest in curating data for RAG or fine-tuning.

But for agentic systems, there’s a missing layer.

Agent design needs to come right after defining the product, because the agent’s capabilities, workflows, and memory requirements are what determine what knowledge it needs and which model providers make sense downstream.

MongoDB published a detailed breakdown of the Canvas Framework built around this exact sequence. It uses two planning canvases.

• The POC canvas has 8 squares covering product validation, agent design (capabilities, autonomy boundaries, memory requirements), data requirements (knowledge sources, update frequency, feedback loops), and model integration (provider selection, prompt strategy, cost validation)

• The production canvas adds 11 squares for scaling, including fault tolerance, multi-agent coordination, unified data architecture across application storage, vector search, and agent memory, plus security hardening and governance.

You can read the full breakdown here →

Thanks to MongoDB for partnering today!

The Anatomy of an Agent Harness

A ReAct loop, a couple of tools, and a well-written system prompt can get surprisingly far in a demo.

But the moment the task requires 10+ steps, things fall apart like the model forgets what it did three steps ago, tool calls fail silently, and the context window fills up with garbage.

The problem isn't the model. It's everything around the model.

LangChain proved this when they changed only the infrastructure wrapping their LLM (same model, same weights) and jumped from outside the top 30 to rank 5 on TerminalBench 2.0.

A separate research project hit a 76.4% pass rate by having an LLM optimize the infrastructure itself, surpassing hand-designed systems.

That infrastructure has a name now: the agent harness.

What is Agent Harness?

The term was formalized in early 2026, but the concept existed long before.

The harness is the complete software infrastructure wrapping an LLM, including the orchestration loop, tools, memory, context management, state persistence, error handling, and guardrails.

Anthropic’s Claude Code documentation puts it simply: the SDK is “the agent harness that powers Claude Code.“

We really liked the canonical formula, from LangChain’s Vivek Trivedy: “If you’re not the model, you’re the harness.”

To put it another way, the “agent” is the emergent behavior: the goal-directed, tool-using, self-correcting entity the user interacts with. The harness is the machinery producing that behavior. When someone says “I built an agent,” they mean they built a harness and pointed it at a model.

Beren Millidge made this analogy precise in his 2023 essay:

• A raw LLM is a CPU with no RAM, no disk, and no I/O.

• The context window serves as RAM (fast but limited).

• External databases function as disk storage (large but slow).

• Tool integrations act as device drivers.

The harness is the operating system.

Three levels of engineering

Three concentric levels of engineering surround the model:

• Prompt engineering crafts the instructions the model receives.

• Context engineering manages what the model sees and when.

• Harness engineering encompasses both, plus the entire application infrastructure: tool orchestration, state persistence, error recovery, verification loops, safety enforcement, and lifecycle management.

The harness is not a wrapper around a prompt. It is the complete system that makes autonomous agent behavior possible.

The 11 components of a production Harness

Synthesizing across Anthropic, OpenAI, LangChain, and the broader practitioner community, a production agent harness has eleven distinct components. Let’s walk through each one.

1. The Orchestration Loop

This is the heartbeat. It implements the Thought-Action-Observation (TAO) cycle, also called the ReAct loop. The loop runs: assemble prompt, call LLM, parse output, execute any tool calls, feed results back, repeat until done.

Mechanically, it’s often just a while loop. The complexity lives in everything the loop manages, not the loop itself. Anthropic describes their runtime as a “dumb loop” where all intelligence lives in the model. The harness just manages turns.

2. Tools

Tools are the agent’s hands. They’re defined as schemas (name, description, parameter types) injected into the LLM’s context so the model knows what’s available. The tool layer handles registration, schema validation, argument extraction, sandboxed execution, result capture, and formatting results back into LLM-readable observations.

Claude Code provides tools across six categories: file operations, search, execution, web access, code intelligence, and subagent spawning. OpenAI’s Agents SDK supports function tools (via function_tool), hosted tools (WebSearch, CodeInterpreter, FileSearch), and MCP server tools.

3. Memory

Memory operates at multiple timescales. Short-term memory is the conversation history within a single session. Long-term memory persists across sessions: Anthropic uses CLAUDE.md project files and auto-generated MEMORY.md files; LangGraph uses namespace-organized JSON Stores; OpenAI supports Sessions backed by SQLite or Redis.

Claude Code implements a three-tier hierarchy: a lightweight index (~150 characters per entry, always loaded), detailed topic files pulled in on demand, and raw transcripts accessed via search only.

4. Context management

This is where many agents fail silently. The core problem is context rot: model performance degrades 30%+ when key content falls in mid-window positions.

Even million-token windows suffer from instruction-following degradation as context grows.

Production strategies include:

• Compaction: summarizing conversation history when approaching limits (Claude Code preserves architectural decisions and unresolved bugs while discarding redundant tool outputs)

• Observation masking: JetBrains’ Junie hides old tool outputs while keeping tool calls visible

• Just-in-time retrieval: maintaining lightweight identifiers and loading data dynamically (Claude Code uses grep, glob, head, tail rather than loading full files)

• Sub-agent delegation: each subagent explores extensively but returns only 1,000 to 2,000 token condensed summaries

Anthropic’s context engineering guide states the goal: find the smallest possible set of high-signal tokens that maximize likelihood of the desired outcome.

5. Prompt construction

This assembles what the model actually sees at each step. It’s hierarchical with system prompt, tool definitions, memory files, conversation history, and the current user message.

OpenAI’s Codex uses a strict priority stack: server-controlled system message (highest priority), tool definitions, developer instructions, user instructions (cascading AGENTS.md files, 32 KiB limit), then conversation history.

6. Output parsing

Modern harnesses rely on native tool calling, where the model returns structured tool_calls objects rather than free-text that must be parsed.

The harness checks if there are any tool calls? If yes, it executes them and loops. If not, it gives the final answer.

For structured outputs, both OpenAI and LangChain support schema-constrained responses via Pydantic models.

Legacy approaches like RetryWithErrorOutputParser (which feeds the original prompt, the failed completion, and the parsing error back to the model) remain available for edge cases.

7. State management

LangGraph models state as typed dictionaries flowing through graph nodes, with reducers merging updates.

Checkpointing happens at super-step boundaries, enabling resumption after interruptions and time-travel debugging.

OpenAI offers four mutually exclusive strategies: application memory, SDK sessions, server-side Conversations API, or lightweight previous_response_id chaining. Claude Code takes a different approach: git commits as checkpoints and progress files as structured scratchpads.

8. Error handling

Here’s why this matters: a 10-step process with 99% per-step success still has only ~90.4% end-to-end success due to compounding.

LangGraph distinguishes four error types: transient (retry with backoff), LLM-recoverable (return error as ToolMessage so the model can adjust), user-fixable (interrupt for human input), and unexpected (bubble up for debugging). Anthropic catches failures within tool handlers and returns them as error results to keep the loop running. Stripe’s production harness caps retry attempts at two.

9. Guardrails and safety

OpenAI’s SDK implements three levels: input guardrails (run on the first agent), output guardrails (run on the final output), and tool guardrails (run on every tool invocation).

A “tripwire” mechanism halts the agent immediately when triggered.

Anthropic separates permission enforcement from model reasoning architecturally. The model decides what to attempt; the tool system decides what’s allowed. Claude Code gates ~40 discrete tool capabilities independently, with three stages: trust establishment at project load, permission check before each tool call, and explicit user confirmation for high-risk operations.

10. Verification loops

This is what separates toy demos from production agents. Anthropic recommends three approaches: rules-based feedback (tests, linters, type checkers), visual feedback (screenshots via Playwright for UI tasks), and LLM-as-judge (a separate subagent evaluates output).

Boris Cherny, creator of Claude Code, noted that giving the model a way to verify its work improves quality by 2 to 3x.

11. Subagent orchestration

Claude Code supports three execution models: Fork (byte-identical copy of parent context), Teammate (separate terminal pane with file-based mailbox communication), and Worktree (own git worktree, isolated branch per agent).

OpenAI’s SDK supports agents-as-tools (specialist handles bounded subtask) and handoffs (specialist takes full control). LangGraph implements subagents as nested state graphs.

A step-by-step walkthrough

Now that you know the components, let’s trace how they work together in a single cycle.

• Step 1 (Prompt Assembly): The harness constructs the full input: system prompt + tool schemas + memory files + conversation history + current user message. Important context is positioned at the beginning and end of the prompt (the “Lost in the Middle” finding).

• Step 2 (LLM Inference): The assembled prompt goes to the model API. The model generates output tokens: text, tool call requests, or both.

• Step 3 (Output Classification): If the model produced text with no tool calls, the loop ends. If it requested tool calls, proceed to execution. If a handoff was requested, update the current agent and restart.

• Step 4 (Tool Execution): For each tool call, the harness validates arguments, checks permissions, executes in a sandboxed environment, and captures results. Read-only operations can run concurrently; mutating operations run serially.

• Step 5 (Result Packaging): Tool results are formatted as LLM-readable messages. Errors are caught and returned as error results so the model can self-correct.

• Step 6 (Context Update): Results are appended to the conversation history. If approaching the context window limit, the harness triggers compaction.

• Step 7 (Loop): Return to Step 1. Repeat until termination.

Termination conditions are layered: the model produces a response with no tool calls, the maximum turn limit is exceeded, the token budget is exhausted, a guardrail tripwire fires, the user interrupts, or a safety refusal is returned. A simple question might take 1 to 2 turns. A complex refactoring task can chain dozens of tool calls across many turns.

For long-running tasks spanning multiple context windows, Anthropic developed a two-phase “Ralph Loop” pattern.

It uses an Initializer Agent that sets up the environment (init script, progress file, feature list, initial git commit), then a Coding Agent in every subsequent session reads git logs and progress files to orient itself, picks the highest-priority incomplete feature, works on it, commits, and writes summaries.

The filesystem provides continuity across context windows.

How frameworks implement the pattern

Anthropic’s Claude Agent SDK exposes the harness through a single query() function that creates the agentic loop and returns an async iterator streaming messages.

The runtime is a “dumb loop.” All intelligence lives in the model. Claude Code uses a Gather-Act-Verify cycle: gather context (search files, read code), take action (edit files, run commands), verify results (run tests, check output), repeat.

OpenAI’s Agents SDK implements the harness through the Runner class with three modes: async, sync, and streamed.

The SDK is “code-first”: workflow logic is expressed in native Python rather than graph DSLs. The Codex harness extends this with a three-layer architecture: Codex Core (agent code + runtime), App Server (bidirectional JSON-RPC API), and client surfaces (CLI, VS Code, web app). All surfaces share the same harness, which is why “Codex models feel better on Codex surfaces than a generic chat window.”

LangGraph models the harness as an explicit state graph. Two nodes (llm_call and tool_node) connected by a conditional edge: if tool calls present, route to tool_node; if absent, route to END.

LangGraph evolved from LangChain’s AgentExecutor, which was deprecated in v0.2 because it was hard to extend and lacked multi-agent support. LangChain’s Deep Agents explicitly use the term “agent harness”: built-in tools, planning (write_todos tool), file systems for context management, subagent spawning, and persistent memory.

CrewAI implements a role-based multi-agent architecture: Agent (the harness around the LLM, defined by role, goal, backstory, and tools), Task (the unit of work), and Crew (the collection of agents). CrewAI’s Flows layer adds a “deterministic backbone with intelligence where it matters,” managing routing and validation while Crews handle autonomous collaboration.

The scaffolding metaphor

Construction scaffolding is a temporary infrastructure that enables workers to build a structure they couldn’t reach otherwise. It doesn’t do the construction. But without it, workers can’t reach the upper floors.

The key insight is that scaffolding is removed when the building is complete. As models improve, harness complexity should decrease. Manus was rebuilt five times in six months, each rewrite removing complexity. Complex tool definitions became general shell execution. “Management agents” became simple structured handoffs.

This points to the co-evolution principle where models are now post-trained with specific harnesses in the loop. Claude Code’s model learned to use the specific harness it was trained with. Changing tool implementations can degrade performance because of this tight coupling.

The future-proofing test for harness design states that if performance scales up with more powerful models without adding harness complexity, the design is sound.

Seven decisions for Harness definitions

Every harness architect faces seven choices:

1. Single-agent vs. multi-agent. Both Anthropic and OpenAI ask to maximize a single agent first. Multi-agent systems add overhead (extra LLM calls for routing, context loss during handoffs). Split only when tool overload exceeds ~10 overlapping tools or clearly separate task domains exist.

2. ReAct vs. plan-and-execute. ReAct interleaves reasoning and action at every step (flexible but higher per-step cost). Plan-and-execute separates planning from execution. LLMCompiler reports a 3.6x speedup over sequential ReAct.

3. Context window management strategy. Five production approaches include time-based clearing, conversation summarization, observation masking, structured note-taking, and sub-agent delegation. ACON research showed 26 to 54% token reduction while preserving 95%+ accuracy by prioritizing reasoning traces over raw tool outputs.

4. Verification loop design. Computational verification (tests, linters) provides deterministic ground truth. Inferential verification (LLM-as-judge) catches semantic issues but adds latency. Martin Fowler’s Thoughtworks team frames this as guides (feedforward, steer before action) versus sensors (feedback, observe after action).

5. Permission and safety architecture. Permissive (fast but risky, auto-approve most actions) versus restrictive (safe but slow, require approval for each action). The choice depends on the deployment context.

6. Tool scoping strategy. More tools often mean worse performance. Vercel removed 80% of tools from v0 and got better results. Claude Code achieves 95% context reduction via lazy loading. The principle: expose the minimum tool set needed for the current step.

7. Harness thickness. How much logic lives in the harness versus the model. Anthropic bets on thin harnesses and model improvement. Graph-based frameworks bet on explicit control. Anthropic regularly deletes planning steps from Claude Code’s harness as new model versions internalize that capability.

The harness is the product

Two products using identical models can have wildly different performance based solely on harness design. The TerminalBench evidence is clear that changing only the harness moved agents by 20+ ranking positions.

The harness is not a solved problem or a commodity layer. It’s where the hard engineering lives like managing context as a scarce resource, designing verification loops that catch failures before they compound, building memory systems that provide continuity without hallucination, and making architectural bets about how much scaffolding to build versus how much to leave to the model.

The field is moving toward thinner harnesses as models improve. But the harness itself isn’t going away. Even the most capable model needs something to manage its context window, execute its tool calls, persist its state, and verify its work.

The next time your agent fails, don’t blame the model but rather look at the harness.

Thanks for reading!

We have our final chapter now, and it is one of the most valuable ones.

Read the final chapter of the MLOps/LLMOps course here →

MLOps/LLMOps case studies

This chapter is different from the rest. It shows you how companies like Booking.com, Uber, Stripe, Doordash, and many across big tech, fintech, banking, and e-commerce actually think about ML and AI systems in production.

These are real case studies with real constraints, real failures, and the decisions that shaped how these systems were built.

One example: Booking.com deployed 150+ production models and found that improving model accuracy often did not improve business outcomes at all.

The reasons why are worth understanding deeply to better approach ML projects.

Read the final chapter of the MLOps/LLMOps course here →

Why care?

When an ML system breaks in production, it is rarely due to the model architecture. Instead, it’s a silent distribution shift, stale embeddings in the feature store, label leakage the eval pipeline did not catch, or KV caches sized for 512 tokens when production prompts routinely hit 4,000+.

The interesting engineering lives in these operational layers when building reproducible pipelines, data versioning, CI/CD for model deployment, drift monitoring with Evidently and Prometheus, context engineering for LLMs, inference optimization via PagedAttention and continuous batching, serving topology decisions that directly shape cost and latency at scale.

MLOps and LLMOps are the disciplines that bring structure to all of this.

They take the entire surface area around a model, from how training data is tracked and validated, to how inference is optimized and served, to how evaluation catches regressions before users do, and turn it into something repeatable, observable, and maintainable.

The MLOps course (18 parts) covers the full lifecycle of traditional ML in production: reproducibility and versioning with W&B, data and pipeline engineering including sampling, feature stores, and distributed processing, model development and optimization through hyperparameter tuning, pruning, compression, and quantization, deployment via containerization, Kubernetes, AWS, and EKS, monitoring and observability with Evidently, Prometheus, and Grafana, and CI/CD workflows.

The LLMOps course (14 parts) transitions to the new set of challenges that come with foundation models: tokenization, embeddings, and attention internals, context engineering and prompt management, evaluation of open-ended generations including multi-turn and tool use, fine-tuning with LoRA, QLoRA, RLHF, DPO, and GRPO, inference optimization covering KV caching, PagedAttention, FlashAttention, and speculative decoding, and LLM serving concepts including self-hosted vs. API-based access and deployment topology.

You can start reading them here:

• MLOps course (18 parts)

• LLMOps course (14 parts)

Thanks for reading!

MindsDB just open-sourced Anton, an autonomous BI agent that turns plain-language questions into full dashboards.

You ask something like “Show me NVIDIA’s profit margins,” and Anton handles everything: figuring out the right data source, writing and executing the code, computing the metrics, and generating interactive visualizations.

Under the hood, it runs a sandboxed Python scratchpad that self-corrects on errors, and a memory system that learns your conventions and analysis patterns across sessions. It connects to Postgres, Snowflake, Salesforce, BigQuery, and more out of the box.

Check out Anton’s GitHub repo here →

A Memory-efficient Technique to Train Large Models

Activation checkpointing is one technique that’s common to the training procedure of almost all popular large models, GPTs, LLaMAs, etc.

In a gist, it’s super helpful to reduce the memory overhead of large neural networks.

Let’s understand this in more detail.

On a side note, while activation checkpointing is one way, we covered 15 techniques to optimize neural network training here: 15 Ways to Optimize Neural Network Training (With Implementation).

How does Activation checkpointing work?

Activation checkpointing is based on two key observations on how neural networks work:

1. The activations of a specific layer can be solely computed using the activations of the previous layer. For instance, in the image below, “Layer B” activations can be computed from “Layer A” activations only:

2. Updating the weights of a layer only depends on two things:

   1. The activations of that layer.

   2. The gradients computed in the next (right) layer (or rather, the running gradients).

      

Activation checkpointing exploits these two observations to optimize memory utilization.

Here’s how it works:

• Step 1) Divide the network into segments before the forward pass:

• Step 2) During forward pass, store the activations of the first layer only in each segment. Discard the rest when they have been used to compute the activations of their subsequent layer.

Step 3) Now comes backpropagation. To update the weights of a layer, we need its activations. Thus, we recompute those activations using the first layer in that segment.

For instance, as shown in the image below, to update the weights of the red layers, we recompute their activations using the activations of the cyan layer, which are already available in memory.

This is how Activation checkpointing works.

To summarize, the idea is that we don’t need to store all the intermediate activations in memory.

Instead, storing a few of them and recomputing the rest only when they are needed can significantly reduce the memory requirement.

Typically, activation checkpointing can reduce memory usage to sqrt(M), where M is the memory usage without activation checkpointing.

Of course, as we compute some activations twice, this does come at the cost of increased run-time, which can typically range between 15-25%.

So there’s always a tradeoff between memory and run-time.

That said, another advantage is that it allows us to use a larger batch size, which can counter the increased run-time.

To utilize this, import the necessary libraries and functions:

Next, define a neural network:

As demonstrated above, in the forward method, we use the checkpoint_sequential method to use activation checkpointing and divide the network into two segments.

Next, we can proceed with network training as we usually would.

While activation checkpointing is one way, we covered 15 techniques to optimize neural network training here: 15 Ways to Optimize Neural Network Training (With Implementation).

Types of Memory in AI Agents

Agents without memory aren’t agents at all.

We often assume LLMs remember things. They feel like humans, but the truth is that LLMs are stateless.

If you want your agent to recall anything (past chats, preferences, behaviors), you have to build memory into it.

But how to do that?

Let’s understand this step-by-step!

Agent memory comes in two scopes:

• Short-term: Handles current conversations. Maintains message history, context, and state across a session.

• Long-term: Spans multiple sessions. Remembers preferences, past actions, and user-specific facts.

But there’s more.

Just like humans, long-term memory in agents can be:

• Semantic → Stores facts and knowledge

• Episodic → Recalls past experiences or task completions

• Procedural → Learns how to do things (think: internalized prompts/instructions)

This memory isn’t just nice-to-have; it enables agents to learn from past interactions without retraining the model.

This is especially powerful for continual learning: letting agents adapt to new tasks without touching LLM weights.

All this is actually implemented in Cognee, a popular open-source knowledge engine (~15k stars) that provides a graph-based, self-improving memory system for your agents.

It uses an ECL (Extract, Cognify, Load) pipeline to turn raw data into structured knowledge graphs with embeddings and relationships, making your agent’s memory both searchable by meaning and connected by relationships.

You can see the full implementation and try it yourself.

Here’s the repo → (don’t forget to star it)

Also, in ​​​​Part 8​​​​ and ​​​​Part 9​​​​ of the Agents’ crash course, we primarily focused on 5 types of Memory for AI agents, with implementation.

And in ​​Part 15​​, ​​Part 16​​ and ​​Part 17​​, we covered practical ways to optimize the Agent’s memory in production use cases.

Thanks for reading!

Pentesting firms don’t want you to see this.

An open-source AI agent replicated their $50k service.

Here’s why this matters right now.

Teams are shipping faster than ever. AI writes the code, CI catches build failures, tests catch regressions, and observability catches outages.

But one more key question to ask is: What can an attacker do with this, right now?

It’s important to answer this question because several real-world examples make this hard to ignore:

• Moltbook exposed 1.5M auth tokens. The owner hadn’t written a single line of code.

• Tea App leaked 72,000 government IDs. The database was just open, no sophisticated hack needed.

• A researcher took control of a journalist’s computer through her own vibe-coded game, without a single click.

The code ran fine in all three cases, tests passed, and nothing raised a flag.

Because the bottleneck is no longer writing code, it’s understanding what that code actually exposes once it’s live. PR reviews miss auth edge cases, unit tests don’t probe broken access control, staging environments don’t simulate adversarial behavior, and business logic flaws look completely fine until someone decides to break them on purpose.

An automated approach is actually implemented in ​Strix, a recently trending open-source framework (23k+ stars) for an AI pentesting agent.

It reviews any running app the way an attacker would:

• Crawls the app and maps every exposed route and flow

• Probes abuse paths dynamically, not just at build time

• Returns findings with proofs-of-concept and suggested fixes

It is benchmarked against 200 real companies and open-source repos, and it found 600+ verified vulnerabilities, including assigned CVEs.

It’s designed to fit into how modern teams already work: run it before a release, after major changes, or continuously as the app evolves.

​You can find the GitHub repo here →​ (don’t forget to star it)

What are Agent Skills and How Agents Use Them?

An agent with 30 specialized workflows installed (deployment pipelines, code review checklists, document formatting rules) would need roughly ~150,000 tokens in its system prompt if you loaded everything upfront.

With Agent Skills, that drops to around 3,000 tokens at startup. The agent knows what skills exist, but loads full instructions only when the current task needs them.

Anthropic released the SKILL.md spec as an open standard in December 2025. Within months, OpenAI Codex, Google Gemini CLI, GitHub Copilot, Cursor, VS Code, JetBrains Junie, and over 30 other agent products adopted it.

You can write a skill once, use it everywhere.

Today, let’s look at how the architecture actually works!

Progressive disclosure

Skills use a three-tier loading system that keeps context costs proportional to what the agent uses, not what it has installed.

• Tier 1: Advertise (~100 tokens per skill). At startup, only the YAML frontmatter (name + description) from each SKILL.md gets injected into the system prompt. This is how the agent knows what skills are available.

• Tier 2: Load (under 5,000 tokens). When the LLM matches a user request to a skill description, it reads the full SKILL.md body: workflows, best practices, edge cases. The spec recommends keeping this under 500 lines.

• Tier 3: Deep dive (as needed). Reference files (references/style-guide.md, references/api-schema.md) and scripts (scripts/validate.py) load on-demand during execution. Scripts are executed via bash, and only the output enters context, not the script code itself.

Routing and the Skills vs. Tools distinction

Skill selection happens entirely through LLM reasoning during the model’s forward pass. There are no embeddings, classifiers or algorithmic routing.

The LLM reads the skill descriptions in the system prompt and picks the best match. This makes the description field the single most important part of any skill.

This is also why skills aren’t tools. Tools execute discrete actions and return results (read a file, call an API). Skills inject specialized instructions into the agent’s context and reshape how it approaches the task. A skill prepares the agent to solve a problem rather than solving it directly.

The format

A SKILL.md file starts with a YAML frontmatter (required name and description, optional license, compatibility, metadata, allowed-tools) followed by a Markdown body with instructions.

The skill lives in a folder that can also include scripts/, references/, and assets/ directories.

Skills are discovered from project-level directories (.claude/skills/ or .agents/skills/), personal directories (~/.claude/skills/), bundled platform skills, and plugin/marketplace sources. The .agents/skills/ path is the cross-platform convention: any compliant agent scans it.

Community adoption has been fast. There are repositories with over 1,300 contributed skills, and Google’s ADK ships with a SkillToolset class that implements the full three-tier disclosure with list_skills, load_skill, and load_skill_resource tools.

Skills + MCP

Skills and MCP are complementary, not competing. MCP provides connectivity (tools, data sources, external APIs). Skills provide procedural knowledge (workflows, best practices, domain expertise).

A skill might instruct the agent to use a specific MCP server, define how to interpret its outputs, and enforce safety checks before destructive operations. You can swap MCP servers without rewriting skills, and update skill instructions without touching MCP configs. The two layers are fully independent.

👉 Over to you: Are you using Agent Skills in your workflow yet? What’s one skill that’s made the biggest difference for you?

​MCP vs Traditional API Architecture​

Traditional APIs were built for apps to talk to servers.

You have a client (web or mobile app), that sends HTTP requests through an API gateway, and the gateway routes to different services.

This works great for applications. But AI agents aren’t apps.

Here’s the problem:

When you want an AI agent to use a tool, like querying a database, accessing files, or calling an API, you have to write custom integration code for each one. Every tool is different, and every integration is bespoke.

MCP solves this, and the visual below differentiates the architectural difference.

Instead of building custom integrations, MCP provides a universal protocol that sits between AI clients (Claude, IDEs, agents) and tools/APIs.

• One protocol to connect to any tool

• The AI doesn’t care what’s behind the server, like a database, file system, web API

• Tool providers build one MCP server, and it works with any AI client.

The visual above shows this clearly: instead of an API gateway routing traffic to individual services, MCP creates a universal layer between AI agents and backend resources.

If you don’t know MCPs, read the guidebook shared above.

And if you want to dive into core MCP engineering, we covered all these details (with implementations) in the MCP course:

• ​​Part 1 covered MCP fundamentals, the architecture, context management, etc. →​​​​​

• ​​Part 2 covered core capabilities, JSON-RPC communication, etc. →​​​​​

• ​​Part 3 built a fully custom and local MCP client →​​​​​

• ​​Part 4 built a full-fledged MCP workflow using tools, resources, and prompts →​​​​​

• ​​​​​Part 5 taught how to integrate Sampling into MCP workflows →​​​​​

• ​​Part 6 covered testing, security, and sandboxing in MCP Workflows →​​​​​

• ​​Part 7 covered testing, security, and sandboxing in MCP Workflows →​​​​​

• ​​Part 8 integrated MCPs with the most widely used agentic frameworks: LangGraph, LlamaIndex, CrewAI, and PydanticAI →​​​​​

• ​​​​​P​​​​​​art 9 covered using LangGraph MCP workflows to build a comprehensive real-world use case →​​

👉 Over to you: What is your perspective on MCP vs Traditional API?

Thanks for reading!

A common problem observed across most LLM eval setups today is that it takes too much time and effort to go from manual trace annotations to automated metrics.

Let’s say you found some failure modes by reviewing production traces. But codifying those into the right eval metrics that correlate with human reviewers is its own engineering project.

The observability built into Confident AI now handles that translation automatically.

Essentially, you can annotate traces on the platform, and it recommends the right eval metrics based on those annotations. Each metric comes with a human-alignment score, so you can validate coverage before wiring it into your pipeline.

The eval engine underneath is DeepEval (14k+ GitHub stars), already widely adopted across production LLM projects.

You can try this yourself here on the Confident AI platform →

And here’s the DeepEval GitHub repo →

How to vibe code: A developer’s playbook

Almost every developer now uses AI to write code. Very few use it well.

A randomized controlled trial found that experienced developers were 19% slower with AI coding tools. But those same developers believed they were 20% faster. That’s a nearly 40-point gap between how productive they felt and how productive they actually were.

The tools aren’t the issue. The practices around them are.

And closing that gap doesn’t take anything radical. It’s a handful of simple practices and a shift in mindset that anyone can follow.

This article covers those principles and then puts them into practice. These fundamentals apply to any AI coding tool you’ll use. For the hands-on part, we’ll use Mistral Vibe, an open-source CLI coding agent with everything you need to vibe code like a pro engineer.

Pure vibe coding vs. AI-assisted development

Before getting into practice, it’s worth understanding what you’re actually doing when you write code with AI. The industry has converged on a spectrum, and where you sit on it determines your results.

Pure vibe coding means accepting AI output without reviewing it. This is the original framing from early 2025: “forget that the code even exists.” It works for throwaway prototypes and weekend experiments.

It does not work for production.

The data makes this clear:

• 45% of AI-generated code introduces security vulnerabilities

• AI co-authored code had 2.74x higher security vulnerability rates across an analysis of 470 pull requests

The gap between “it runs” and “it’s production-ready” is enormous.

AI-assisted development sits at the other end. You use AI to accelerate implementation, but you maintain full understanding and ownership of the code. You write specs, review diffs, run tests, and can explain every line to someone else.

The AI is the typist. You are the engineer.

In traditional development, roughly 70% of your cognitive energy goes into translating ideas into syntax. In AI-assisted development, that flips. 70% goes into thinking clearly about what to build and verifying what the AI produced.

Your role doesn’t shrink. It changes. You stop being the typist and start being the architect.

Important practices

Five practices show up consistently among developers getting real, compounding value from AI-assisted development.

None of them are about writing better prompts.

1) Spec before you prompt

The single biggest mistake is prompting too early.

“Build me a task manager” produces garbage. A 15-line spec defining stack, schema, views, and auth produces a working prototype in one session.

One practitioner reported going from idea to 32 passing tests in a single session by their fifth feature, with zero debugging cycles. The difference wasn’t the model. It was the input.

A good spec has three pillars:

• Intent: What you’re building and why

• Constraints: Tech stack, architectural patterns, what NOT to do

• Acceptance criteria: Testable conditions that define “done”

You don’t need a 20-page PRD. A markdown file covering these three things is enough.

For larger features, try having the AI interview you before it writes any code. Have it probe your requirements, question edge cases, and surface tradeoffs. Once the interview is done, have it write a spec document, then start a fresh session to execute against that spec. Clean context focused entirely on implementation.

One thing that’s easy to overlook: make your architectural decisions explicitly. AI will make them for you if you don’t, and it usually picks defaults that work for demos but fail in production.

2) Context engineering > prompt engineering

This is the most underappreciated skill in AI-assisted development.

Context engineering is the practice of designing what information is available to the AI at any given moment. It matters far more than how cleverly you phrase your request.

The context window is a shared resource. Performance degrades as it fills. Three practical rules:

Start fresh sessions for new tasks. Don’t let stale context from a previous feature pollute your new implementation. Carry forward only the spec and key decisions, not the full conversation history.

Use “just in time” context retrieval. Rather than pre-loading your entire codebase, maintain lightweight references (file paths, module names) and use tools like grep to dynamically load data as needed.

Keep context files focused on things the AI can’t infer. Project conventions, naming patterns, architectural constraints, security requirements. The AI can read your code. It can’t read your team’s unwritten rules.

3) The Plan → Execute → Verify loop

Vibe coding is a conversation, not a one-shot. The developers who move fastest break the work into small, verifiable steps.

• Plan: Define the goal and constraints for this specific step. Not the whole project, just the next piece. Better yet, ask the AI to think through the plan first before writing any code. This forces the model to reason about the problem, surface edge cases, and propose an approach you can review before implementation.

• Execute: Let the AI generate code, tests, or docs.

• Verify: Review the diffs. Run the tests. Give specific, actionable feedback. “That’s wrong” is a bad prompt. “The auth middleware should read from the Authorization header, not X-Token, and return 401 on expired tokens” is a good one.

The key discipline is breaking complex tasks into atomic pieces. AI works great for the first 80% of a project but stalls on edge cases and integration. Small, focused tasks keep each interaction within the AI’s zone of competence.

4) Testing is the foundation

Automated testing is the single most important practice for production-quality AI-assisted development.

Without tests:

• Your AI agent might claim something works without having actually tested it

• Any new change could silently break an unrelated feature

• AI-generated code optimizes for plausibility, code that “looks right” but may contain subtle logic errors

Test-first development works particularly well with agents. Write (or have the AI write) the tests first. Review them. Confirm they fail. Then let the agent implement code to make them pass.

You’ve validated intent through the tests before you ever review the implementation.

5) Security and review are non-negotiable

Security is where AI-assisted development’s risks are the sharpest.

• 40% of code completion suggestions were found insecure in security-sensitive scenarios

• One platform’s missing row-level security exposed 170+ production apps

Three strategies that significantly reduce these risks:

Security-first context. Include security instructions in your project context file: “always use parameterized queries, never hardcode secrets, validate all inputs.” Research shows this significantly reduces vulnerable code generation.

Self-reflection loops. After the agent generates code, prompt it to review its own output for security vulnerabilities before you do. This catches a surprising number of issues.

Supply chain vigilance. AI models suggest packages that don’t exist on public registries (a vector for “slopsquatting” attacks), or pull in unreviewed transitive dependencies. Always verify dependencies.

The golden rule for production: don’t commit code you can’t explain to someone else. Your name is on the commit.

Anti-patterns

Three failure modes show up consistently:

The endless error loop. The AI introduces a bug, you describe the bug, and the AI “fixes” it by introducing a different bug. Stop the loop. Read the code yourself. Understand the root cause. Provide a precise description of the problem and the expected behavior.

The comprehension gap. Shipping code you don’t understand. It works today. You can’t debug it tomorrow. If you don’t understand it, don’t merge it.

Session drift. Long sessions accumulate stale context. When the AI starts losing coherence, start fresh. Carry forward the spec and decisions, not the conversation history.

Putting it into practice with Mistral Vibe

Everything above is tooling-agnostic. To walk through these workflows hands-on, we’ll use Mistral Vibe, an open-source CLI coding agent from Mistral AI.

• Open source under Apache 2.0. The code is on GitHub. You can audit exactly what the agent does, fork it, and customize the security model.

• Self-hostable. Run locally on your own hardware and your code never leaves your infrastructure. For teams where data sovereignty is a hard requirement, this matters.

• Model-agnostic. Swap providers via config.toml. Point it at OpenRouter, a local vLLM server, or any OpenAI-compatible API.

• Built-in cost controls. The --max-price and --max-turns flags hard-cap session costs. Devstral 2 also runs at roughly 7x lower per-token cost compared to frontier models, which compounds fast at scale.

Setup

Get started with a single command:

Then navigate to your project and run vibe:

Vibe automatically scans your project’s file structure and Git status. You’re now in an interactive chat with an agent that already has context about your codebase.

Agent modes: matching trust to the task

Vibe offers different modes that map directly to the practices we covered.

• Default mode requires approval for every tool execution. Every file write, every shell command gets a preview and a confirmation prompt. Start here.

• Plan mode is read-only. It can read files and search code but cannot write or execute anything. Use it to explore a codebase, create a structured plan, and surface edge cases before implementation begins.

• Accept-edits mode automatically approves file edits but still asks for shell commands. Useful for trusted refactoring workflows.

• Auto-approve mode skips all confirmations. Use for well-defined, low-risk tasks like formatting, documentation, or running linters.

Switch between them mid-session with Shift+Tab.

Plan mode deserves emphasis. Spec-driven approaches have shown 50-80% reduction in implementation time precisely because the upfront thinking eliminates entire categories of errors downstream.

[Demo] Understanding to shipping a PR

What follows is a continuous workflow on a single codebase: we understand the project, plan a new feature, implement it, verify with a subagent, and ship a PR. The same Plan → Execute → Verify loop, end to end in practice.

The demo codebase is a lightweight Express.js + SQLite task management API with JWT authentication, task CRUD, user profiles, and existing tests.

Understanding the codebase

Let’s see how Vibe understands your codebase:

Vibe explores the project structure, reads the files, understands the relationships between them, and responds with a clear breakdown. It also traces the authentication flow through @src/middleware/auth.js as prompted.

This is the step that developers skip the most, and it’s the most expensive one to skip. Every minute spent understanding the codebase saves you from fighting the AI’s output later.

This is context engineering in practice: you’re loading exactly the information you need, not dumping the entire project into a prompt. The @ file reference system makes this precise. Instead of “look at my auth code,” you say “explain @src/middleware/auth.js,” and Vibe reads exactly that file.

Planning and implementing a feature

This is where the full Plan → Execute → Verify loop becomes concrete.

Before writing the prompt, we switched to plan mode to let Vibe plan the feature thoroughly before writing a single line of code.

We ask Vibe to create a plan for adding a “delete account” feature. Vibe produces a structured plan: which files need to change, what the endpoint should look like, what edge cases to handle, and the database operations involved.

Check this out:

Once the plan looks right, we tell it to implement. Vibe patches existing files using targeted search-and-replace, showing a full preview of each diff and waiting for confirmation before writing to disk. The feature takes shape across the route file, the controller, the database migration, and the tests, each step approved individually.

Verifying with a subagent

The feature is implemented. Now we verify.

A subagent is a specialized agent instance that runs in its own isolated context window. The main agent delegates a focused task, the subagent executes independently, and only the compressed result flows back to the parent.

Why does this matter?

Context stays clean. Each subagent gets a fresh context window and reads only what it needs. The main agent’s context doesn’t get polluted with all the files the subagent explored or the dead ends it hit. Clean context means better output quality on everything that comes after.

Token efficiency. The subagent’s full exploration gets compressed into a summary before returning. You pay for the subagent’s work, but your main agent’s context window stays lean.

Here’s how we use it in Mistral Vibe:

In our example, we delegate a verification task to a subagent to review the files we just modified for bugs, inconsistencies, or missing error handling.

The subagent reads the changed files, cross-references them against the existing codebase, flags issues, implements the fix, runs the tests to confirm, and stages the changes for commit, all without us needing to intervene. The main agent then commits with a detailed message describing what the feature does and how it was verified.

Shipping with a custom skill

The code is committed. Time to push and create a PR.

Skills are reusable components that extend what Vibe can do. They’re defined as markdown files with a YAML header specifying a name, description, allowed tools, and workflow instructions.

When you mark a skill as user-invocable: true, it becomes a slash command you can trigger during any session.

Skills live in three places:

• Globally (~/.vibe/skills/) for personal skills across all projects

• Per-project (.vibe/skills/) for team skills that travel with the repo

• Custom paths in config.toml

Anything you find yourself doing more than twice becomes a skill that runs with a single command: code review checklists, migration generators, release notes writers, security audits.

Since we always need to push code and create pull requests, that’s a perfect candidate. We’ve created a ship-pr skill that analyzes the current branch, generates a PR description, pushes to origin, and creates the PR on GitHub via the gh CLI.

You can see the SKILL.md file side-by-side with the terminal. We invoke the skill and Vibe executes the full workflow, handing over the final PR link.

Branch analyzed, description written, code pushed, PR created. From a single slash command.

Other features

• Session continuation. Pick up where you left off with vibe --continue, or resume a specific session by ID.

• Configuration. Everything lives in config.toml. Custom system prompts in ~/.vibe/prompts/ encode your team’s coding standards.

• Programmatic mode. Run non-interactively with vibe --prompt “...” --max-turns 5 --max-price 1.0 --output json. Useful for CI/CD pipelines and automated code review.

• Local/offline mode. Self-host Devstral on your own GPU using vLLM or Ollama. An RTX 4090 handles the 24B model at 4-bit precision. Your code never leaves your infrastructure.

The speed AI gives you is a superpower. Overconfidence is a trap.

The developers who benefit the most bring engineering discipline to the process: the spec, the plan, the review, and the verify step. The tools amplify your judgment. They don’t replace it.

Start with the spec. Break the work into steps. Verify everything. Treat AI output with the same scrutiny you’d give a junior developer’s pull request.

That’s how you vibe code in 2026.

Check out Mistral Vibe →

Thanks for reading!

NVIDIA Nemotron 3 Super is live on Lightning AI, and you can run it today without setting up a single server.

Nemotron 3 Super is a hybrid MoE reasoning model with open weights.

That means teams can keep full control, without any vendor lock-in.

You can claim your 30M free tokens per month without any credit card →.

Thanks to Lightning AI for partnering today!

MiniMax M2.7: The self-refactoring Agent architecture

Most AI models today are deployed as static artifacts.

Devs train them, ship them, and they operate inside a fixed environment: a set of skills, tools, memory, and workflow rules called an “agent harness.”

If something is slow or brittle, a human engineer steps in and fixes the scaffold. The model itself never touches it.

MiniMax’s M2.7 treats its harness as something it can rewrite autonomously.

How M2.7 rewrites its own scaffold

Every AI agent operates inside a scaffold that defines the tools it can call, the skills it can invoke, the memory it retains, and the workflow rules it follows.

M2.7 closes the human-in-the-loop bottleneck by running a self-optimization cycle. Here, the model runs a task, analyzes where things broke, plans changes to its own scaffold (skills, memory, workflow rules), applies those changes, evaluates the results against a benchmark, and decides whether to keep or revert.

It then writes self-criticism into memory so the next iteration starts with accumulated lessons.

MiniMax ran this loop for over 100 rounds internally. Over those rounds, M2.7 discovered optimizations on its own that no human had instructed.

For instance:

• It systematically searched for optimal sampling parameters (temperature, frequency penalty, presence penalty)

• It wrote workflow-specific guidelines for itself, like automatically checking for the same bug pattern in other files after a fix.

• It added loop detection to avoid getting stuck in repetitive failure cycles.

This gave it a 30% performance improvement on internal evaluation sets, without any retraining.

The controlled test: MLE Bench Lite

MiniMax also tested this in a more controlled setting.

They ran M2.7 through 22 ML competitions from OpenAI’s MLE Bench Lite, each on a single A30 GPU.

The harness used three core components: short-term memory, self-feedback, and self-optimization.

After each iteration, the agent wrote a memory file describing what happened, performed self-criticism, and fed those insights into the next round.

With every round, the ML models M2.7 trained achieved higher medal rates. The best run earned 9 gold medals, 5 silver, and 1 bronze. The average medal rate across all three runs was 66.6%, tying with Gemini 3.1 and trailing only Opus 4.6 (75.7%) and GPT-5.4 (71.2%).

Importance of this approach

The weights in M2.7 never change during the self-optimization loop. What changes is the system around them, like better skills, better memory, and better workflow rules.

That distinction matters because it means the improvement loop can run continuously, in production, without any retraining cycle.

The broader signal is that model performance increasingly depends on the harness, not just the weights. And if the model can improve its own harness, the ceiling keeps moving upward without a single gradient update.

You can read more about MiniMax M2.7 in the official blog post →

👉 Over to you: Do you think self-evolving harnesses will become standard for agent deployments, or is this still too early for production?

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Mistral open-sourced Voxtral TTS, a 4B-parameter text-to-speech model that clones voices from just 3 seconds of reference audio.

The architecture is a hybrid of autoregressive semantic token generation (built on Ministral 3B) and flow-matching for acoustic tokens. It is just 8 GB in BF16, fits on a single 16GB GPU.

In human evals by native speakers, it scored a 68.4% win rate over ElevenLabs Flash v2.5 for zero-shot voice cloning across 9 languages, with a model latency of 70ms.

The model is open-weights and you can self-host the entire voice pipeline without sending audio to a third party.

Weights on HuggingFace are available here →

And you can read the research paper here →

Claude vs. Claude Code vs. Cowork

Anthropic now offers three distinct ways to interact with Claude, and each one targets a fundamentally different workflow. Think of it as: Chat for thinking, Code for building, and Cowork for doing.

If you’ve been confused about which one to use and when, this newsletter will clear that up in under two minutes.

Here’s a quick breakdown:

1) Claude Chat

This is the conversational AI assistant most people already know. You type a prompt, Claude responds, and you iterate together.

• Turn rough ideas into structured plans through conversation

• Write emails, reports, essays, and long-form content

• Research and summarize complex topics in minutes

• Analyze documents, PDFs, and images

• Build interactive prototypes through Artifacts

The key here is that everything happens through conversation. You’re thinking with Claude, not delegating work to it.

It’s available on every device, has a free tier, and supports persistent memory across sessions.

The tradeoff is that it has no direct access to your local files (upload only), and it can’t generate raster images natively.

2) Claude Code

This is a terminal-native coding agent. You describe what you want in plain English, and Claude reads your codebase, writes code, runs tests, fixes errors, and ships the result.

• Build and debug entire features across the full codebase

• Write, run, and fix tests automatically

• Manage git workflows and create pull requests

• Spawn multiple parallel agents working on different parts of a task simultaneously

It handles the full development cycle end-to-end, from planning to execution to testing. With the CLAUDE(.)md configuration file, you can teach it your project’s conventions, patterns, and constraints so it writes code the way your team expects.

The tradeoff is a steeper learning curve compared to Chat, and token costs can add up during heavy sessions.

3) Claude Cowork

This is the newest addition. Anthropic describes it as Claude Code for the rest of your work.

It’s an agentic desktop assistant that automates file management and repetitive tasks through a GUI. You describe an outcome, and Claude plans, executes, and delivers finished work: formatted documents, organized file systems, spreadsheets with working formulas, and synthesized research.

• Direct local file access and editing (no upload/download cycle)

• Schedule recurring tasks automatically

• Assign tasks remotely via Dispatch from your phone

• Computer Use lets Claude control your screen directly

It runs inside a sandboxed virtual machine on your computer, so Claude can only access folders you explicitly grant. You don’t need to know how to code to use it.

The tradeoff is that your computer must stay awake for tasks to run, and it’s still in research preview.

Here’s how to think about choosing between them:

• If you need to think through a problem or get writing/research help, use Chat

• If you’re building software and want an autonomous coding partner, use Code

• If you have a clearly defined deliverable that involves local files and desktop workflows, use Cowork

All three are included in the same subscription starting at $20/month, which makes it one of the highest-leverage subscriptions in productivity software right now.

We’ve put together a visual below that maps the workflow of each product side by side.

If you want to go deeper into Claude Code specifically, we wrote a detailed article covering the anatomy of the .claude/ folder, a complete guide to CLAUDE.md, custom commands, skills, agents, and permissions, and how to set them all up properly.

You can read it here →

Thanks for reading!

This newsletter regularly breaks down RAG architectures, AI agents, LLM internals, and everything in between.

Now we’re bringing all of that to Instagram too, in a format that’s quick to consume and hard to ignore.

We’re already 240 posts deep with content on RAG vs HyDE, agentic RAG, specialized AI models, prompt techniques, Bayesian optimization, active learning, and a lot more.

You can find the account and follow it here →

DailyDoseofDS Instagram Channel

Concepts of LLM Serving

After covering LLM inference optimization in the full LLMOps course, we now move to the fundamentals of LLM serving.

Read Part 14 of the full LLMOps course here →

LLMOps course Part 14

It covers how to actually make a language model accessible as a service: API-based providers vs. self-hosted inference, deployment topology decisions (on-prem, cloud, hybrid), serving with vLLM, and the practical trade-offs that determine how your LLM runs in production.

Read Part 14 of the full LLMOps course here →

Why care?

Optimizing inference (the previous chapter) is about making a single model run faster. Serving is about making that model reliably available to users.

These are different problems. You can have the most optimized inference stack in the world, but if your serving layer cannot handle concurrent users, if cold starts block requests, if you have no strategy for scaling up and down, none of that optimization matters in practice.

The serving layer is where engineering decisions directly translate into user experience and cost. Choosing between API providers and self-hosting changes your cost structure, latency profile, and data privacy posture. Choosing between on-prem and cloud changes your operational burden and scaling flexibility.

This chapter gives you the conceptual framework to make these decisions thoughtfully, along with hands-on experience serving models with vLLM.

• Read Part 1 on fundamentals of LLMOps here →

• Read Part 2 on understanding the core building blocks of LLMs →

• Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →

• Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →

• Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →

• Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →

• Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →

• Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common context failure modes in agentic applications →

• Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →

• Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →

• Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →

• Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →

• Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →

Over to you: What would you like to learn in the LLMOps course?

Thanks for reading!

Onyx is a self-hostable AI chat platform that works with any LLM, like Claude, GPT, Gemini, Llama, or any open-weight model you want.

Onyx GitHub Repo

Here’s what it ships with:

• Agents that chain multiple tools in sequence

• RAG with full indexing across 40+ connectors (Slack, Drive, Confluence, Jira, GitHub, email, call transcripts)

• Deep research ranked #1 on DeepResearchBench, above every proprietary alternative

• MCP support for connecting to external systems

• Code interpreter for data analysis and file generation

• Self-host on your own infrastructure via Docker in a few mins

Unlike Claude’s MCP-based connectors that query your tools at runtime, Onyx actually indexes all your data. That means faster, more reliable search across everything your team has ever written.

The entire code is open-source under MIT license, so you can see the full implementation on GitHub and try it yourself.

Find the GitHub repo here → (don’t forget to star it ⭐️)

The three layers of context that make coding agents actually useful

GitHub analyzed 2,500+ custom instruction files across public repos to understand what separates effective agent setups from weak ones.

They found that effective setups give agents a specific persona, exact commands to run, defined boundaries, and examples of good output.

Weak ones are vague helpers with no clear job description.

This points to the core friction with coding agents today, which is that they don’t have a capability problem but rather a context problem.

A raw agent can write code, but it doesn’t know the team’s naming conventions, the specific linting setup, or preferred framework patterns.

Without that context, the first PR is often off-target and requires multiple rounds of correction.

Getting this right requires structured context, and GitHub Copilot implements a smart, layered customization system that does exactly this.

Layer 1: Repository-Level Instructions

At the repo level, a .github/copilot-instructions.md file defines project-wide rules like coding conventions, naming standards, security defaults, and prohibited patterns. The agent reads this before generating any code.

Here’s what an effective copilot-instructions.md looks like:

# Project Context

TypeScript monorepo using pnpm workspaces.
Backend: Fastify. Frontend: React + Vite.

## Commands

- Install: `pnpm install`
- Test: `pnpm test`
- Lint: `pnpm lint --fix`

## Coding Standards

- Use TypeScript strict mode
- Prefer named exports over default exports
- Use date-fns instead of moment.js (deprecated)
- Never commit secrets or API keys
- No `any` type, no console.log in production

Layer 2: Path-Specific Instructions

For granular control, instruction files in .github/instructions/ can target specific file paths using applyTo frontmatter. A TypeScript-specific instruction file only activates when the agent works on .ts files:

---
applyTo: "**/tests/*.spec.ts"
---

## Playwright Test Requirements

- Use stable locators: `getByRole()`, `getByText()`, `getByTestId()`
- Avoid CSS selectors or XPath
- Each test should be independent
- Always wait for elements with `await expect(locator).toBeVisible()`

Layer 3: Custom Agents

The most interesting addition is custom agents. These are .agent.md files in .github/agents/ that define specialized personas with their own tool access and MCP server connections.

Here’s a security auditor agent that can only read code and run linters:

---
name: security-auditor
description: Reviews code for vulnerabilities
tools: ['read', 'search']
---

# Security Reviewer

You are a security engineer reviewing code for vulnerabilities.
You can READ code but cannot EDIT files. Flag issues, do not fix them.

Check for: hardcoded secrets, SQL injection, XSS, missing auth checks, unvalidated input.

Output: file path, severity, description, recommended fix.

Frontend conventions, backend patterns, and security policies apply everywhere without duplicating config files in every repo.

Pre-Configured Partner Agents

For teams that don’t want to build from scratch, GitHub Copilot has also shipped pre-configured agents with MCP connections to external dev tools.

Some of the available partner agents include:

• JFrog Security Agent: Scans dependencies and suggests vulnerability fixes

• MongoDB Performance Advisor: Analyzes query performance and recommends optimizations

• PagerDuty Incident Responder: Summarizes incidents and suggests investigation steps

• Terraform Agent: Manages infrastructure-as-code workflows with HCP Terraform

These aren’t generic prompts but rather domain-specific agents with actual tool access to the platforms they integrate with via MCP servers.

Here’s what a partner agent config looks like:

---
name: mongodb-advisor
description: Analyzes MongoDB performance
tools: ['read', 'search']
mcp-servers:
  - url: https://mcp.mongodb.com/sse
    name: mongodb-mcp
---

# MongoDB Performance Advisor

Use the MongoDB MCP server to analyze slow queries, 
suggest index improvements, and identify N+1 patterns.

The custom agents work across VS Code, JetBrains, Eclipse, and Xcode. So the setup meets developers where they already work.

Try it for free here →

Thanks to Microsoft for partnering on this one!