<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:googleplay="http://www.google.com/schemas/play-podcasts/1.0"><channel><title><![CDATA[Daily Dose of Data Science]]></title><description><![CDATA[A free newsletter for continuous learning about data science and ML, lesser-known techniques, and how to apply them in 2 minutes. We keep things no-fluff.

Join 100,000+ data scientists from top companies like Google, NVIDIA, Microsoft, Uber, etc.]]></description><link>https://blog.dailydoseofds.com</link><image><url>https://substackcdn.com/image/fetch/$s_!heKx!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5dc1fee-2d1e-4892-b219-4b96f6998ab5_288x288.png</url><title>Daily Dose of Data Science</title><link>https://blog.dailydoseofds.com</link></image><generator>Substack</generator><lastBuildDate>Fri, 20 Nov 2026 00:31:22 GMT</lastBuildDate><atom:link href="https://blog.dailydoseofds.com/feed" rel="self" type="application/rss+xml"/><copyright><![CDATA[Avi Chawla]]></copyright><language><![CDATA[en]]></language><webMaster><![CDATA[avi@dailydoseofds.com]]></webMaster><itunes:owner><itunes:email><![CDATA[avi@dailydoseofds.com]]></itunes:email><itunes:name><![CDATA[Avi Chawla]]></itunes:name></itunes:owner><itunes:author><![CDATA[Avi Chawla]]></itunes:author><googleplay:owner><![CDATA[avi@dailydoseofds.com]]></googleplay:owner><googleplay:email><![CDATA[avi@dailydoseofds.com]]></googleplay:email><googleplay:author><![CDATA[Avi Chawla]]></googleplay:author><itunes:block><![CDATA[Yes]]></itunes:block><item><title><![CDATA[GPU architecture, clearly explained]]></title><description><![CDATA[...and why memory bandwidth, not compute, sets your token rate.]]></description><link>https://blog.dailydoseofds.com/p/gpu-architecture-clearly-explained</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/gpu-architecture-clearly-explained</guid><dc:creator><![CDATA[Avi Chawla]]></dc:creator><pubDate>Thu, 19 Nov 2026 00:13:29 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/838af6d2-1392-4263-ae54-c6becd6b962a_1838x1574.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><strong>In today&#8217;s newsletter:</strong></p><ul><li><p>Redis is going all in on AI.</p></li><li><p>GPU architecture, clearly explained.</p></li></ul><div><hr></div><h3><a href="https://fandf.co/4kjC0b6">Redis is going all in on AI.</a></h3><p>They just built <strong><a href="https://fandf.co/4kjC0b6">Redis Iris</a></strong>, a real-time engine that delivers relevant and accurate context for production agents.</p><p>In production, support agents receive the same intent in many phrasings:</p><ul><li><p>&#8220;What&#8217;s the status of my refund?&#8221;</p></li><li><p>&#8220;Has my refund gone through yet?&#8221;</p></li></ul><p>A semantic cache matches these paraphrases and returns the stored response, which works well when the answer is static.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!v95r!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc47dff5f-f2ed-4d16-bf52-07cfb8778629_2752x1536.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!v95r!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc47dff5f-f2ed-4d16-bf52-07cfb8778629_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!v95r!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc47dff5f-f2ed-4d16-bf52-07cfb8778629_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!v95r!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc47dff5f-f2ed-4d16-bf52-07cfb8778629_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!v95r!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc47dff5f-f2ed-4d16-bf52-07cfb8778629_2752x1536.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!v95r!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc47dff5f-f2ed-4d16-bf52-07cfb8778629_2752x1536.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c47dff5f-f2ed-4d16-bf52-07cfb8778629_2752x1536.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:2017323,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224671137?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc47dff5f-f2ed-4d16-bf52-07cfb8778629_2752x1536.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!v95r!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc47dff5f-f2ed-4d16-bf52-07cfb8778629_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!v95r!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc47dff5f-f2ed-4d16-bf52-07cfb8778629_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!v95r!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc47dff5f-f2ed-4d16-bf52-07cfb8778629_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!v95r!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc47dff5f-f2ed-4d16-bf52-07cfb8778629_2752x1536.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>But, at times, context is not static over time (like refund status). The stored response might indicate &#8220;processing&#8221; even after the refund has gone through, so the cache returns a stale answer.</p><p>A plain agent avoids this by running the full reasoning loop on every request, which gives the right answer but spends tokens re-deriving the same steps each time.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!pnA0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8c0a92a-0524-4959-ba58-1ce7f57e212a_2752x1536.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!pnA0!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8c0a92a-0524-4959-ba58-1ce7f57e212a_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!pnA0!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8c0a92a-0524-4959-ba58-1ce7f57e212a_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!pnA0!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8c0a92a-0524-4959-ba58-1ce7f57e212a_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!pnA0!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8c0a92a-0524-4959-ba58-1ce7f57e212a_2752x1536.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!pnA0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8c0a92a-0524-4959-ba58-1ce7f57e212a_2752x1536.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d8c0a92a-0524-4959-ba58-1ce7f57e212a_2752x1536.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1329338,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224671137?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8c0a92a-0524-4959-ba58-1ce7f57e212a_2752x1536.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!pnA0!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8c0a92a-0524-4959-ba58-1ce7f57e212a_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!pnA0!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8c0a92a-0524-4959-ba58-1ce7f57e212a_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!pnA0!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8c0a92a-0524-4959-ba58-1ce7f57e212a_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!pnA0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8c0a92a-0524-4959-ba58-1ce7f57e212a_2752x1536.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Here&#8217;s what happens when a request comes in:</p><ul><li><p>The Intent Router recognizes similar requests.</p></li><li><p>It then pulls the Agent Playbook for that request. The Playbook doesn&#8217;t store an answer. It stores the steps to produce one, like &#8220;look up the order, check the refund status, reply using this template.&#8221;</p></li><li><p>The agent follows those steps, so it skips re-planning, and the app fetches the current refund status before the reply goes out.</p></li></ul><p>In short, semantic cache reuses the answer while Iris reuses the steps, so the answer is always computed on live data.</p><p>This lets the workflow avoid any additional reasoning while still working on the live data.</p><p>You can try Redis Iris here: <strong><a href="https://fandf.co/4kjC0b6">https://redis.io/iris</a></strong></p><p>To test this, we built a small app to compare all three setups (plain agent, Semantic cache, and Iris-backed agent). </p><p>We asked about the refund while it was processing, then updated the status to refunded and asked again with a paraphrase.</p><p>The video below shows this in action:</p><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;81607d60-d01c-4523-9fdd-9bef6dd6fbd4&quot;,&quot;duration&quot;:null}"></div><p>Question 1 (status: processing):</p><ul><li><p>Plain agent: 2 LLM calls, 9.4s, correct</p></li><li><p>Semantic cache: cache miss, so 2 LLM calls, 10.2s, correct</p></li><li><p>Iris: 1 LLM call, 5.9s, correct</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!hyKM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e480e4c-b90a-4e15-ac5a-6b7e7265fbf1_2752x1536.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!hyKM!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e480e4c-b90a-4e15-ac5a-6b7e7265fbf1_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!hyKM!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e480e4c-b90a-4e15-ac5a-6b7e7265fbf1_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!hyKM!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e480e4c-b90a-4e15-ac5a-6b7e7265fbf1_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!hyKM!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e480e4c-b90a-4e15-ac5a-6b7e7265fbf1_2752x1536.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!hyKM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e480e4c-b90a-4e15-ac5a-6b7e7265fbf1_2752x1536.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8e480e4c-b90a-4e15-ac5a-6b7e7265fbf1_2752x1536.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1741792,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224671137?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e480e4c-b90a-4e15-ac5a-6b7e7265fbf1_2752x1536.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!hyKM!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e480e4c-b90a-4e15-ac5a-6b7e7265fbf1_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!hyKM!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e480e4c-b90a-4e15-ac5a-6b7e7265fbf1_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!hyKM!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e480e4c-b90a-4e15-ac5a-6b7e7265fbf1_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!hyKM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e480e4c-b90a-4e15-ac5a-6b7e7265fbf1_2752x1536.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Question 2 (status: refunded, paraphrased):</p><ul><li><p>Plain agent: 2 LLM calls, 12.7s, and the reply didn&#8217;t confirm status</p></li><li><p>Semantic cache: cache hit, 0.5s, but it returned the stale &#8220;processing&#8221; answer</p></li><li><p>Iris: 1 LLM call, 5.6s, correct</p></li></ul><p>Iris skips the planning call because the procedure is already stored, so it ran about 2x faster than the plain agent and was the only setup that was right on both turns.</p><p>You can try Redis Iris here: <strong><a href="https://fandf.co/4kjC0b6">https://redis.io/iris</a></strong></p><p><em>Thanks to the Redis team for partnering today!</em></p><div><hr></div><h3><a href="https://www.dailydoseofds.com/p/how-a-gpu-actually-works/">GPU architecture, clearly explained</a></h3><p>A GPU can perform calculations faster than it can load the data those calculations need.</p><p>This often limits token generation. The model reads its weights from memory, uses them for a small amount of arithmetic, then needs more data.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!vEr4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F636ce17c-ea9c-4e2a-ac6c-eb0b7a8ff573_5504x3072.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!vEr4!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F636ce17c-ea9c-4e2a-ac6c-eb0b7a8ff573_5504x3072.png 424w, https://substackcdn.com/image/fetch/$s_!vEr4!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F636ce17c-ea9c-4e2a-ac6c-eb0b7a8ff573_5504x3072.png 848w, https://substackcdn.com/image/fetch/$s_!vEr4!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F636ce17c-ea9c-4e2a-ac6c-eb0b7a8ff573_5504x3072.png 1272w, https://substackcdn.com/image/fetch/$s_!vEr4!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F636ce17c-ea9c-4e2a-ac6c-eb0b7a8ff573_5504x3072.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!vEr4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F636ce17c-ea9c-4e2a-ac6c-eb0b7a8ff573_5504x3072.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/636ce17c-ea9c-4e2a-ac6c-eb0b7a8ff573_5504x3072.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:2816674,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224671137?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F636ce17c-ea9c-4e2a-ac6c-eb0b7a8ff573_5504x3072.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!vEr4!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F636ce17c-ea9c-4e2a-ac6c-eb0b7a8ff573_5504x3072.png 424w, https://substackcdn.com/image/fetch/$s_!vEr4!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F636ce17c-ea9c-4e2a-ac6c-eb0b7a8ff573_5504x3072.png 848w, https://substackcdn.com/image/fetch/$s_!vEr4!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F636ce17c-ea9c-4e2a-ac6c-eb0b7a8ff573_5504x3072.png 1272w, https://substackcdn.com/image/fetch/$s_!vEr4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F636ce17c-ea9c-4e2a-ac6c-eb0b7a8ff573_5504x3072.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Adding more arithmetic capacity does little to help if memory cannot supply those weights fast enough.</p><p>Memory bandwidth measures how much data the GPU can move per second. When generating one response at a time, the rate at which it reads the model&#8217;s weights can limit how quickly it produces tokens.</p><p>To see why, consider what happens when the model uses a weight.</p><p>At 16-bit precision, reading one weight transfers two bytes. Using it involves a multiplication and an addition, giving just two arithmetic operations. The GPU does very little work with each weight before it needs the next one.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!yB3N!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c71b13-dbe4-4183-bd89-613188523adb_5504x3072.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!yB3N!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c71b13-dbe4-4183-bd89-613188523adb_5504x3072.png 424w, https://substackcdn.com/image/fetch/$s_!yB3N!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c71b13-dbe4-4183-bd89-613188523adb_5504x3072.png 848w, https://substackcdn.com/image/fetch/$s_!yB3N!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c71b13-dbe4-4183-bd89-613188523adb_5504x3072.png 1272w, https://substackcdn.com/image/fetch/$s_!yB3N!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c71b13-dbe4-4183-bd89-613188523adb_5504x3072.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!yB3N!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c71b13-dbe4-4183-bd89-613188523adb_5504x3072.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/61c71b13-dbe4-4183-bd89-613188523adb_5504x3072.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:2643584,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224671137?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c71b13-dbe4-4183-bd89-613188523adb_5504x3072.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!yB3N!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c71b13-dbe4-4183-bd89-613188523adb_5504x3072.png 424w, https://substackcdn.com/image/fetch/$s_!yB3N!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c71b13-dbe4-4183-bd89-613188523adb_5504x3072.png 848w, https://substackcdn.com/image/fetch/$s_!yB3N!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c71b13-dbe4-4183-bd89-613188523adb_5504x3072.png 1272w, https://substackcdn.com/image/fetch/$s_!yB3N!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c71b13-dbe4-4183-bd89-613188523adb_5504x3072.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Processing several requests together can improve this. The GPU can load a weight and use it across several requests, doing more calculations with the same data.</p><p>That is one reason a model serving many users at once can make better use of the GPU than a model generating a single response.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!WAPO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9640bce-295b-4e4d-9346-5d26bd6afbb7_5504x3072.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!WAPO!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9640bce-295b-4e4d-9346-5d26bd6afbb7_5504x3072.png 424w, https://substackcdn.com/image/fetch/$s_!WAPO!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9640bce-295b-4e4d-9346-5d26bd6afbb7_5504x3072.png 848w, https://substackcdn.com/image/fetch/$s_!WAPO!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9640bce-295b-4e4d-9346-5d26bd6afbb7_5504x3072.png 1272w, https://substackcdn.com/image/fetch/$s_!WAPO!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9640bce-295b-4e4d-9346-5d26bd6afbb7_5504x3072.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!WAPO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9640bce-295b-4e4d-9346-5d26bd6afbb7_5504x3072.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d9640bce-295b-4e4d-9346-5d26bd6afbb7_5504x3072.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:2920834,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224671137?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9640bce-295b-4e4d-9346-5d26bd6afbb7_5504x3072.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!WAPO!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9640bce-295b-4e4d-9346-5d26bd6afbb7_5504x3072.png 424w, https://substackcdn.com/image/fetch/$s_!WAPO!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9640bce-295b-4e4d-9346-5d26bd6afbb7_5504x3072.png 848w, https://substackcdn.com/image/fetch/$s_!WAPO!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9640bce-295b-4e4d-9346-5d26bd6afbb7_5504x3072.png 1272w, https://substackcdn.com/image/fetch/$s_!WAPO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9640bce-295b-4e4d-9346-5d26bd6afbb7_5504x3072.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The GPU&#8217;s layout also helps it reuse data. It divides work into small groups, each with access to storage near the units doing the calculations.</p><p>There are four names to know:</p><ul><li><p>A thread performs a small part of the computation.</p></li><li><p>A warp groups 32 threads so the GPU can issue instructions to them together.</p></li><li><p>A thread block contains one or more warps that can cooperate and share data.</p></li><li><p>An SM, short for streaming multiprocessor, is a compute unit that runs thread blocks. A block&#8217;s warps run on the same SM.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!zxPs!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fee247eb7-b325-439f-a985-b89e8d6ebab9_5504x3072.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zxPs!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fee247eb7-b325-439f-a985-b89e8d6ebab9_5504x3072.png 424w, https://substackcdn.com/image/fetch/$s_!zxPs!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fee247eb7-b325-439f-a985-b89e8d6ebab9_5504x3072.png 848w, https://substackcdn.com/image/fetch/$s_!zxPs!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fee247eb7-b325-439f-a985-b89e8d6ebab9_5504x3072.png 1272w, https://substackcdn.com/image/fetch/$s_!zxPs!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fee247eb7-b325-439f-a985-b89e8d6ebab9_5504x3072.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!zxPs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fee247eb7-b325-439f-a985-b89e8d6ebab9_5504x3072.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ee247eb7-b325-439f-a985-b89e8d6ebab9_5504x3072.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:2610428,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224671137?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fee247eb7-b325-439f-a985-b89e8d6ebab9_5504x3072.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!zxPs!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fee247eb7-b325-439f-a985-b89e8d6ebab9_5504x3072.png 424w, https://substackcdn.com/image/fetch/$s_!zxPs!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fee247eb7-b325-439f-a985-b89e8d6ebab9_5504x3072.png 848w, https://substackcdn.com/image/fetch/$s_!zxPs!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fee247eb7-b325-439f-a985-b89e8d6ebab9_5504x3072.png 1272w, https://substackcdn.com/image/fetch/$s_!zxPs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fee247eb7-b325-439f-a985-b89e8d6ebab9_5504x3072.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Each SM has storage close to its arithmetic units. Keeping a value there lets the GPU reuse it without fetching it from its larger memory again.</p><p>The main storage levels are:</p><ul><li><p>Registers hold the values an individual thread is working with.</p></li><li><p>Shared memory lets threads in a block exchange and reuse data. L1 is a nearby cache that can also hold recently accessed data.</p></li><li><p>L2 is a larger cache shared across the SMs.</p></li><li><p>HBM, or High Bandwidth Memory, holds the model&#8217;s weights, its stored attention data, and other large arrays. It sits outside the processing chip, so accessing it takes longer than reading local storage.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!4xUS!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fafebc498-906c-4bbe-9f3d-14d6e853c77e_5504x3072.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!4xUS!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fafebc498-906c-4bbe-9f3d-14d6e853c77e_5504x3072.png 424w, https://substackcdn.com/image/fetch/$s_!4xUS!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fafebc498-906c-4bbe-9f3d-14d6e853c77e_5504x3072.png 848w, https://substackcdn.com/image/fetch/$s_!4xUS!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fafebc498-906c-4bbe-9f3d-14d6e853c77e_5504x3072.png 1272w, https://substackcdn.com/image/fetch/$s_!4xUS!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fafebc498-906c-4bbe-9f3d-14d6e853c77e_5504x3072.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!4xUS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fafebc498-906c-4bbe-9f3d-14d6e853c77e_5504x3072.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/afebc498-906c-4bbe-9f3d-14d6e853c77e_5504x3072.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:2515482,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224671137?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fafebc498-906c-4bbe-9f3d-14d6e853c77e_5504x3072.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!4xUS!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fafebc498-906c-4bbe-9f3d-14d6e853c77e_5504x3072.png 424w, https://substackcdn.com/image/fetch/$s_!4xUS!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fafebc498-906c-4bbe-9f3d-14d6e853c77e_5504x3072.png 848w, https://substackcdn.com/image/fetch/$s_!4xUS!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fafebc498-906c-4bbe-9f3d-14d6e853c77e_5504x3072.png 1272w, https://substackcdn.com/image/fetch/$s_!4xUS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fafebc498-906c-4bbe-9f3d-14d6e853c77e_5504x3072.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>When an algorithm keeps useful data in local storage, it can avoid repeated trips to HBM. FlashAttention is an example.</p><p>A conventional attention implementation creates a large table of attention scores, writes it to HBM, and reads intermediate results back for later calculations.</p><p>FlashAttention works on smaller pieces and keeps intermediate results close to the computation. It avoids storing the full table in HBM. It still loads inputs and saves outputs, but moves less data between those steps.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!IAtD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07d333c9-d32a-4a82-8b93-c4706dedcd15_5504x3072.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!IAtD!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07d333c9-d32a-4a82-8b93-c4706dedcd15_5504x3072.png 424w, https://substackcdn.com/image/fetch/$s_!IAtD!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07d333c9-d32a-4a82-8b93-c4706dedcd15_5504x3072.png 848w, https://substackcdn.com/image/fetch/$s_!IAtD!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07d333c9-d32a-4a82-8b93-c4706dedcd15_5504x3072.png 1272w, https://substackcdn.com/image/fetch/$s_!IAtD!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07d333c9-d32a-4a82-8b93-c4706dedcd15_5504x3072.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!IAtD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07d333c9-d32a-4a82-8b93-c4706dedcd15_5504x3072.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/07d333c9-d32a-4a82-8b93-c4706dedcd15_5504x3072.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:2540311,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224671137?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07d333c9-d32a-4a82-8b93-c4706dedcd15_5504x3072.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!IAtD!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07d333c9-d32a-4a82-8b93-c4706dedcd15_5504x3072.png 424w, https://substackcdn.com/image/fetch/$s_!IAtD!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07d333c9-d32a-4a82-8b93-c4706dedcd15_5504x3072.png 848w, https://substackcdn.com/image/fetch/$s_!IAtD!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07d333c9-d32a-4a82-8b93-c4706dedcd15_5504x3072.png 1272w, https://substackcdn.com/image/fetch/$s_!IAtD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07d333c9-d32a-4a82-8b93-c4706dedcd15_5504x3072.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>This is why a GPU&#8217;s compute rating alone cannot predict token speed. Performance also depends on how much data the model moves and how much work it does before loading that data again.</p><p>Our full guide explains how these parts fit together, with diagrams of GPU execution and memory.</p><p><strong><a href="https://www.dailydoseofds.com/p/how-a-gpu-actually-works/">Read the full guide on how a GPU actually works &#8594;</a></strong></p><p>We also put together a 10-week roadmap for engineers who want to run LLM inference in production, with 30 minutes of work per day.</p><p><strong><a href="https://github.com/patchy631/time-to-first-token">Explore the time-to-first-token roadmap &#8594;</a></strong></p><p>If you find it useful, star the repository.</p><p>&#128073; Over to you: Is your current inference workload limited by memory bandwidth or compute?</p><p>Good day!</p><div><hr></div><h3><strong>P.S. For those wanting to develop &#8220;Industry ML&#8221; expertise:</strong></h3><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!cn8y!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!cn8y!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png 424w, https://substackcdn.com/image/fetch/$s_!cn8y!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png 848w, https://substackcdn.com/image/fetch/$s_!cn8y!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png 1272w, https://substackcdn.com/image/fetch/$s_!cn8y!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!cn8y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png" width="1456" height="543" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:543,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!cn8y!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png 424w, https://substackcdn.com/image/fetch/$s_!cn8y!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png 848w, https://substackcdn.com/image/fetch/$s_!cn8y!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png 1272w, https://substackcdn.com/image/fetch/$s_!cn8y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>At the end of the day, all businesses care about <em>impact</em>. That&#8217;s it!</p><ul><li><p>Can you reduce costs?</p></li><li><p>Drive revenue?</p></li><li><p>Can you scale ML models?</p></li><li><p>Predict trends before they happen?</p></li></ul><p>We have discussed several other topics (with implementations) that align with such topics.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.dailydoseofds.com/membership&quot;,&quot;text&quot;:&quot;Develop \&quot;Industry ML\&quot; Skills&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.dailydoseofds.com/membership"><span>Develop "Industry ML" Skills</span></a></p><p>Here are some of them:</p><ul><li><p>Learn everything about MCPs in this <a href="https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/">&#8203;</a><strong><a href="https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/">crash course with 9 parts &#8594;</a></strong><a href="https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/">&#8203;</a></p></li><li><p>Learn how to build Agentic systems in <strong><a href="https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/">a crash course with 14 parts</a></strong>.</p></li><li><p>Learn how to build real-world RAG apps and evaluate and scale them in <strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/">this crash course</a></strong>.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!bv9E!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!bv9E!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png 424w, https://substackcdn.com/image/fetch/$s_!bv9E!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png 848w, https://substackcdn.com/image/fetch/$s_!bv9E!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png 1272w, https://substackcdn.com/image/fetch/$s_!bv9E!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!bv9E!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png" width="1456" height="1128" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/cd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1128,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!bv9E!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png 424w, https://substackcdn.com/image/fetch/$s_!bv9E!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png 848w, https://substackcdn.com/image/fetch/$s_!bv9E!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png 1272w, https://substackcdn.com/image/fetch/$s_!bv9E!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><ul><li><p>Learn sophisticated graph architectures and how to train them on graph data.</p></li><li><p>So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches <strong><a href="https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/">here</a></strong>.</p></li><li><p>Learn how to run large models on small devices using <a href="https://www.dailydoseofds.com/quantization-optimize-ml-models-to-run-them-on-tiny-hardware/">&#8203;</a><strong><a href="https://www.dailydoseofds.com/quantization-optimize-ml-models-to-run-them-on-tiny-hardware/">Quantization techniques</a></strong>.</p></li><li><p>Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using <a href="https://www.dailydoseofds.com/conformal-predictions-build-confidence-in-your-ml-models-predictions/">&#8203;</a><strong><a href="https://www.dailydoseofds.com/conformal-predictions-build-confidence-in-your-ml-models-predictions/">Conformal Predictions</a></strong>.</p></li><li><p>Learn how to identify causal relationships and answer business questions using causal inference in <strong><a href="https://www.dailydoseofds.com/a-crash-course-on-causality-part-1/">this crash course</a></strong>.</p></li><li><p>Learn how to scale and implement ML model training in this <strong><a href="https://www.dailydoseofds.com/how-to-scale-model-training/">practical guide</a></strong>.</p></li><li><p>Learn techniques to reliably <strong><a href="https://www.dailydoseofds.com/5-must-know-ways-to-test-ml-models-in-production-implementation-included/">test new models in production</a></strong>.</p></li><li><p>Learn how to build privacy-first ML systems using <a href="https://www.dailydoseofds.com/federated-learning-a-critical-step-towards-privacy-preserving-machine-learning/">&#8203;</a><strong><a href="https://www.dailydoseofds.com/federated-learning-a-critical-step-towards-privacy-preserving-machine-learning/">Federated Learning</a></strong>.</p></li><li><p>Learn 6 techniques with implementation to <strong><a href="https://www.dailydoseofds.com/model-compression-a-critical-step-towards-efficient-machine-learning/">compress ML models</a></strong>.</p></li></ul><p>All these resources will help you cultivate key skills that businesses and companies care about the most.</p>]]></content:encoded></item><item><title><![CDATA[The next AI inference architecture, clearly explained (4x faster than GPUs)]]></title><description><![CDATA[...plus autoregressive LLMs vs. diffusion LLMs (a popular LLM interview question).]]></description><link>https://blog.dailydoseofds.com/p/the-next-ai-inference-architecture</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/the-next-ai-inference-architecture</guid><dc:creator><![CDATA[Avi Chawla]]></dc:creator><pubDate>Tue, 17 Nov 2026 23:40:37 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/6fb4594a-1250-4b28-8f2e-728074fbfd91_2752x1536.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><strong>In today&#8217;s newsletter:</strong></p><ul><li><p>The next AI inference architecture, clearly explained (4x faster than GPUs).</p></li><li><p>Autoregressive LLMs vs. diffusion LLMs, clearly explained. </p></li></ul><div><hr></div><h3><a href="https://fandf.co/4ezjhEu">The next AI inference architecture, clearly explained (4x faster than GPUs)</a></h3><p><em>Everything you need to understand why LLM decode waits on memory, and how a dataflow chip removes that wait. It covers the memory-bandwidth ceiling, kernel boundaries, the limits of kernel fusion, spatial mapping, memory tiers, disaggregated prefill and decode, and a hands-on speed test of GLM-5.3, a Mixture-of-Experts model on SambaNova Cloud.</em></p><p>A GPU runs an LLM as a long chain of small programs called kernels.</p><p>Even for a small Llama model, that chain has about a hundred kernels for every token the model generates.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Ea9B!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefc28aa9-12d2-4135-a5d9-f79822744cad_2752x1536.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Ea9B!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefc28aa9-12d2-4135-a5d9-f79822744cad_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!Ea9B!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefc28aa9-12d2-4135-a5d9-f79822744cad_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!Ea9B!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefc28aa9-12d2-4135-a5d9-f79822744cad_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!Ea9B!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefc28aa9-12d2-4135-a5d9-f79822744cad_2752x1536.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Ea9B!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefc28aa9-12d2-4135-a5d9-f79822744cad_2752x1536.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/efc28aa9-12d2-4135-a5d9-f79822744cad_2752x1536.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:505815,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225409773?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefc28aa9-12d2-4135-a5d9-f79822744cad_2752x1536.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Ea9B!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefc28aa9-12d2-4135-a5d9-f79822744cad_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!Ea9B!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefc28aa9-12d2-4135-a5d9-f79822744cad_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!Ea9B!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefc28aa9-12d2-4135-a5d9-f79822744cad_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!Ea9B!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fefc28aa9-12d2-4135-a5d9-f79822744cad_2752x1536.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">With one request, the GPU spends much of each step waiting on HBM, not doing math.</figcaption></figure></div><p>Each kernel writes its result to HBM, the GPU&#8217;s main memory, and the next kernel reads it back. With one request at a time, the chip spends much of each step waiting on memory instead of doing math.</p><p>AI agents suffer the most due to this. Anthropic reports that its agents use about 4x the tokens of a chat, and multi-agent systems use about 15x.</p><p>A dataflow chip, like <strong><a href="https://fandf.co/4ezjhEu">SambaNova&#8217;s RDU</a></strong>, is a processor designed to pass each result straight to the next step on the chip, which minimizes trips to main memory. On 16 of SambaNova&#8217;s SN40L chips, this approach runs Llama 3.1 8B at about 1,100 tokens/s.</p><p>The visual below shows how this differs from a GPU:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!g3mT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67b0d73b-febd-4177-b4d4-a1d591bdb261_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!g3mT!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67b0d73b-febd-4177-b4d4-a1d591bdb261_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!g3mT!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67b0d73b-febd-4177-b4d4-a1d591bdb261_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!g3mT!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67b0d73b-febd-4177-b4d4-a1d591bdb261_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!g3mT!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67b0d73b-febd-4177-b4d4-a1d591bdb261_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!g3mT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67b0d73b-febd-4177-b4d4-a1d591bdb261_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/67b0d73b-febd-4177-b4d4-a1d591bdb261_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:143710,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225409773?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67b0d73b-febd-4177-b4d4-a1d591bdb261_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!g3mT!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67b0d73b-febd-4177-b4d4-a1d591bdb261_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!g3mT!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67b0d73b-febd-4177-b4d4-a1d591bdb261_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!g3mT!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67b0d73b-febd-4177-b4d4-a1d591bdb261_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!g3mT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67b0d73b-febd-4177-b4d4-a1d591bdb261_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A GPU runs decode as a chain of kernels that bounce through HBM, while a dataflow chip keeps the same operations in one on-chip pipeline.</figcaption></figure></div><p>So today, let&#8217;s understand why a GPU leaves much of its memory bandwidth unused during decode, why kernel fusion only goes so far, and how SambaNova&#8217;s RDU runs a whole decoder layer as one pipeline that keeps results in on-chip SRAM.</p><p>We&#8217;ll end with a hands-on speed test of GLM-5.3, a Mixture-of-Experts model now running on SambaNova Cloud.</p><div><hr></div><h4>Prefill vs. decode in agent workloads</h4><p>An agent works in a loop. It asks the model for the next action, runs a tool, adds the result to its context and repeats.</p><p>Every model call has two phases.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!-nWy!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06e68ce4-4434-43ac-999c-23534fcd4252_680x380.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!-nWy!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06e68ce4-4434-43ac-999c-23534fcd4252_680x380.png 424w, https://substackcdn.com/image/fetch/$s_!-nWy!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06e68ce4-4434-43ac-999c-23534fcd4252_680x380.png 848w, https://substackcdn.com/image/fetch/$s_!-nWy!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06e68ce4-4434-43ac-999c-23534fcd4252_680x380.png 1272w, https://substackcdn.com/image/fetch/$s_!-nWy!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06e68ce4-4434-43ac-999c-23534fcd4252_680x380.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!-nWy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06e68ce4-4434-43ac-999c-23534fcd4252_680x380.png" width="680" height="380" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/06e68ce4-4434-43ac-999c-23534fcd4252_680x380.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:380,&quot;width&quot;:680,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:69074,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225409773?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06e68ce4-4434-43ac-999c-23534fcd4252_680x380.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!-nWy!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06e68ce4-4434-43ac-999c-23534fcd4252_680x380.png 424w, https://substackcdn.com/image/fetch/$s_!-nWy!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06e68ce4-4434-43ac-999c-23534fcd4252_680x380.png 848w, https://substackcdn.com/image/fetch/$s_!-nWy!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06e68ce4-4434-43ac-999c-23534fcd4252_680x380.png 1272w, https://substackcdn.com/image/fetch/$s_!-nWy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06e68ce4-4434-43ac-999c-23534fcd4252_680x380.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Prefill handles the prompt in parallel, and decode then writes the answer one token at a time.</figcaption></figure></div><ul><li><p>Prefill reads the whole prompt in parallel and builds the KV cache,</p></li><li><p>And decode then writes the answer one token at a time.</p></li></ul><p>A recent study traced this loop on benchmarks like SWE-bench Pro and Terminal-Bench. Model inference took 71% to 98% of the total runtime.</p><p>Each turn reuses almost all of the last turn&#8217;s context, so prefix caching skips most of the prefill work.</p><div><hr></div><h4>Memory-bandwidth ceiling for decode</h4><p>With one request, the model reads every weight once for each new token. That works out to about 1 FLOP of compute for every byte it reads.</p><p>An H100 SXM can do about 989 trillion BF16 FLOPs per second, but its HBM delivers only 3.35 TB/s. So decode sits about 300x below the point where compute power becomes the limit.</p><p>This gives one request a hard speed ceiling, as the formula below shows:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;plaintext&quot;,&quot;nodeId&quot;:null}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-plaintext">tokens/s &lt;= memory bandwidth / bytes of weights

H100, GLM-5.3 in BF16, ~40B active:

3.35e12 / (40e9 * 2) = ~42 tokens/s</code></pre></div><p>The same calculation gives 1,351 tokens/s for Llama-3.2-1B and 209 for an 8B model.</p><p>The chart below shows how far decode sits from the compute limit:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!QJF6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F795d3dba-679d-4bab-962e-3c0bbebe8b79_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!QJF6!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F795d3dba-679d-4bab-962e-3c0bbebe8b79_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!QJF6!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F795d3dba-679d-4bab-962e-3c0bbebe8b79_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!QJF6!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F795d3dba-679d-4bab-962e-3c0bbebe8b79_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!QJF6!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F795d3dba-679d-4bab-962e-3c0bbebe8b79_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!QJF6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F795d3dba-679d-4bab-962e-3c0bbebe8b79_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/795d3dba-679d-4bab-962e-3c0bbebe8b79_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:126108,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225409773?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F795d3dba-679d-4bab-962e-3c0bbebe8b79_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!QJF6!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F795d3dba-679d-4bab-962e-3c0bbebe8b79_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!QJF6!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F795d3dba-679d-4bab-962e-3c0bbebe8b79_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!QJF6!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F795d3dba-679d-4bab-962e-3c0bbebe8b79_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!QJF6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F795d3dba-679d-4bab-962e-3c0bbebe8b79_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">One-request decode sits at about 1 FLOP per byte, far below the H100 compute ridge.</figcaption></figure></div><div><hr></div><h4>Kernel boundaries and issues with fusion</h4><p>Each GPU kernel handles one or a few operations, such as RMSNorm, attention, or the MLP.</p><p>Stanford Hazy Research found that vLLM and SGLang use at most 50% of an H100&#8217;s bandwidth on Llama-3.2-1B.</p><p>At each boundary, the next kernel waits for the last one to finish, then stalls while its weights load from HBM.</p><p>The visual below shows these gaps between kernels:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Gh4t!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40b903d-c354-4bdc-8874-91e59a7052c4_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Gh4t!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40b903d-c354-4bdc-8874-91e59a7052c4_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Gh4t!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40b903d-c354-4bdc-8874-91e59a7052c4_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Gh4t!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40b903d-c354-4bdc-8874-91e59a7052c4_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Gh4t!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40b903d-c354-4bdc-8874-91e59a7052c4_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Gh4t!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40b903d-c354-4bdc-8874-91e59a7052c4_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e40b903d-c354-4bdc-8874-91e59a7052c4_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:151004,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225409773?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40b903d-c354-4bdc-8874-91e59a7052c4_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Gh4t!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40b903d-c354-4bdc-8874-91e59a7052c4_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Gh4t!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40b903d-c354-4bdc-8874-91e59a7052c4_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Gh4t!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40b903d-c354-4bdc-8874-91e59a7052c4_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Gh4t!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40b903d-c354-4bdc-8874-91e59a7052c4_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Each GPU kernel writes to HBM and the next reads it back, with a launch gap between them.</figcaption></figure></div><p>Kernel fusion, which merges several operations into one kernel, is the standard fix. Hazy&#8217;s hand-built megakernel fused the whole 1B forward pass this way and reached 78% of bandwidth.</p><p>GPU fusion still stops at a few operations. Large intermediate results spill out of the small on-chip SRAM, and the programming model cannot run back-to-back operations as a pipeline.</p><div><hr></div><h4>Dataflow execution and memory tiers on an RDU</h4><p>An RDU, or Reconfigurable Dataflow Unit, is SambaNova&#8217;s AI chip.</p><p>Unlike a GPU, it doesn&#8217;t run a model one kernel at a time. Instead, its compiler spreads the steps of a decoder layer across the chip, so every step runs in its own spot.</p><p>Here&#8217;s how that differs from a GPU:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!aTW5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45dd3eed-d7b1-4fad-9051-5f5cf5f5ef13_2752x1536.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!aTW5!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45dd3eed-d7b1-4fad-9051-5f5cf5f5ef13_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!aTW5!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45dd3eed-d7b1-4fad-9051-5f5cf5f5ef13_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!aTW5!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45dd3eed-d7b1-4fad-9051-5f5cf5f5ef13_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!aTW5!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45dd3eed-d7b1-4fad-9051-5f5cf5f5ef13_2752x1536.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!aTW5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45dd3eed-d7b1-4fad-9051-5f5cf5f5ef13_2752x1536.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/45dd3eed-d7b1-4fad-9051-5f5cf5f5ef13_2752x1536.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:513741,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225409773?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45dd3eed-d7b1-4fad-9051-5f5cf5f5ef13_2752x1536.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!aTW5!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45dd3eed-d7b1-4fad-9051-5f5cf5f5ef13_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!aTW5!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45dd3eed-d7b1-4fad-9051-5f5cf5f5ef13_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!aTW5!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45dd3eed-d7b1-4fad-9051-5f5cf5f5ef13_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!aTW5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45dd3eed-d7b1-4fad-9051-5f5cf5f5ef13_2752x1536.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A GPU runs RMSNorm, attention, and the MLP one after another, while an RDU gives each operation its own group of compute units.</figcaption></figure></div><ul><li><p>A GPU runs RMSNorm, then attention, then the MLP, one after another. The RDU gives each of these operations its own group of compute units on the chip.</p></li><li><p>On a GPU, each kernel writes its output to HBM and the next one reads it back. On the RDU, each result passes straight to the next group through on-chip SRAM, so it never leaves the chip.</p></li><li><p>A GPU kernel stalls while its weights load from HBM. The RDU loads the next step&#8217;s weights while the current step is still computing, so no step waits.</p></li></ul><p>Here&#8217;s how this layout looks on the chip:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!zUBC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7eae6098-6b0c-47fe-ab39-b0e25f050520_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zUBC!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7eae6098-6b0c-47fe-ab39-b0e25f050520_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!zUBC!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7eae6098-6b0c-47fe-ab39-b0e25f050520_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!zUBC!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7eae6098-6b0c-47fe-ab39-b0e25f050520_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!zUBC!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7eae6098-6b0c-47fe-ab39-b0e25f050520_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!zUBC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7eae6098-6b0c-47fe-ab39-b0e25f050520_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7eae6098-6b0c-47fe-ab39-b0e25f050520_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:143609,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225409773?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7eae6098-6b0c-47fe-ab39-b0e25f050520_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!zUBC!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7eae6098-6b0c-47fe-ab39-b0e25f050520_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!zUBC!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7eae6098-6b0c-47fe-ab39-b0e25f050520_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!zUBC!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7eae6098-6b0c-47fe-ab39-b0e25f050520_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!zUBC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7eae6098-6b0c-47fe-ab39-b0e25f050520_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">An RDU tile runs a whole decoder layer as one kernel, and activations stay in on-chip SRAM.</figcaption></figure></div><p>The RDU keeps data in three memory tiers. On-chip SRAM holds activations, 64 GB of HBM holds active weights and KV cache, and up to 512 GB of DDR holds other models.</p><p>That DDR sits on the chip itself, so the RDU can switch models without going through the host CPU&#8217;s memory.</p><div><hr></div><h4>Whole-decoder kernels and loop conversion</h4><p>SambaNova applied this to Llama 3.1 8B, which has 32 identical decoder layers. The RDU runs each whole layer as one kernel call, and a compiler pass called loop conversion folds the 32 calls into one loop.</p><p>On 16 SN40L chips, the baseline ran at 100 tokens/s. The single-kernel decoder reached 500, and loop conversion reached 1,100.</p><p>The visual below compares the GPU and RDU layouts:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!mb0s!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe25f6412-b463-44d2-a7ef-cd6458b42ce4_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mb0s!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe25f6412-b463-44d2-a7ef-cd6458b42ce4_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!mb0s!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe25f6412-b463-44d2-a7ef-cd6458b42ce4_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!mb0s!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe25f6412-b463-44d2-a7ef-cd6458b42ce4_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!mb0s!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe25f6412-b463-44d2-a7ef-cd6458b42ce4_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!mb0s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe25f6412-b463-44d2-a7ef-cd6458b42ce4_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e25f6412-b463-44d2-a7ef-cd6458b42ce4_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:160889,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225409773?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe25f6412-b463-44d2-a7ef-cd6458b42ce4_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!mb0s!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe25f6412-b463-44d2-a7ef-cd6458b42ce4_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!mb0s!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe25f6412-b463-44d2-a7ef-cd6458b42ce4_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!mb0s!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe25f6412-b463-44d2-a7ef-cd6458b42ce4_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!mb0s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe25f6412-b463-44d2-a7ef-cd6458b42ce4_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">On 16 SN40L chips, loop conversion took Llama 3.1 8B from a 100 tokens/s baseline to 1,100 tokens/s.</figcaption></figure></div><p>Per the SN40L paper, HBM bandwidth then goes only to weights and KV cache, and loading weights during compute uses over 85% of it.</p><div><hr></div><h4>Multi-chip dataflow through the AGCU</h4><p>A model like GLM-5.3 does not fit on one chip, so its 753B parameters are split across many. Every token then has to move data from chip to chip, and that movement often costs more time than the math.</p><p>On a GPU cluster, each GPU writes its result to memory, sends it over the network to the next GPU, and the next GPU reads it back. Each hop adds a wait.</p><p>SambaNova&#8217;s RDU handles this with the AGCU (Address Generation and Coalescing Unit). The AGCU streams data directly from one RDU to the next, so the dataflow pipeline from the previous section keeps running across chips instead of stopping at every boundary, as shown below.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!kjUY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5ba5d46e-1cc4-4ec1-b53e-b264ceab2dac_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!kjUY!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5ba5d46e-1cc4-4ec1-b53e-b264ceab2dac_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!kjUY!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5ba5d46e-1cc4-4ec1-b53e-b264ceab2dac_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!kjUY!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5ba5d46e-1cc4-4ec1-b53e-b264ceab2dac_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!kjUY!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5ba5d46e-1cc4-4ec1-b53e-b264ceab2dac_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!kjUY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5ba5d46e-1cc4-4ec1-b53e-b264ceab2dac_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5ba5d46e-1cc4-4ec1-b53e-b264ceab2dac_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:123228,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225409773?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5ba5d46e-1cc4-4ec1-b53e-b264ceab2dac_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!kjUY!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5ba5d46e-1cc4-4ec1-b53e-b264ceab2dac_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!kjUY!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5ba5d46e-1cc4-4ec1-b53e-b264ceab2dac_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!kjUY!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5ba5d46e-1cc4-4ec1-b53e-b264ceab2dac_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!kjUY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5ba5d46e-1cc4-4ec1-b53e-b264ceab2dac_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A GPU cluster stops at every chip hop, while the AGCU streams data directly from one RDU to the next.</figcaption></figure></div><p>This matters even more for Mixture-of-Experts models. GLM-5.3 activates only about 40B parameters per token, spread across experts on different chips, so tokens hop between chips all the time.</p><div><hr></div><h4>Measuring decode speed on SambaNova Cloud</h4><p>SambaNova Cloud has an OpenAI-compatible API, so the OpenAI SDK works with a different base URL. The script below streams one request to GLM-5.3 and prints the server&#8217;s own timings:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;python&quot;,&quot;nodeId&quot;:null}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-python">
import os
from openai import OpenAI

client = OpenAI(base_url="https://api.sambanova.ai/v1",
                api_key=os.environ["SAMBANOVA_API_KEY"])

stream = client.chat.completions.create(
    model="GLM-5.3", max_tokens=300, stream=True,
    stream_options={"include_usage": True},
    messages=[{"role": "user", "content": "Explain in about 250 words "
               "why LLM decode is memory-bandwidth bound."}])

u = [c.usage for c in stream if c.usage][0].model_dump()

print(f"prefill: {u['time_to_first_token']:.3f} s")

print(f"decode:  {u['completion_tokens_after_first_per_sec']:.0f} tokens/s")</code></pre></div><p>The screenshot below shows this request, plus a ten-step agent loop built on the same call:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!2s0p!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F022814b8-2dee-4841-87b1-466047cde224_1920x562.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!2s0p!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F022814b8-2dee-4841-87b1-466047cde224_1920x562.png 424w, https://substackcdn.com/image/fetch/$s_!2s0p!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F022814b8-2dee-4841-87b1-466047cde224_1920x562.png 848w, https://substackcdn.com/image/fetch/$s_!2s0p!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F022814b8-2dee-4841-87b1-466047cde224_1920x562.png 1272w, https://substackcdn.com/image/fetch/$s_!2s0p!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F022814b8-2dee-4841-87b1-466047cde224_1920x562.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!2s0p!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F022814b8-2dee-4841-87b1-466047cde224_1920x562.png" width="1456" height="426" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/022814b8-2dee-4841-87b1-466047cde224_1920x562.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:426,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:12575,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225409773?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F022814b8-2dee-4841-87b1-466047cde224_1920x562.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!2s0p!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F022814b8-2dee-4841-87b1-466047cde224_1920x562.png 424w, https://substackcdn.com/image/fetch/$s_!2s0p!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F022814b8-2dee-4841-87b1-466047cde224_1920x562.png 848w, https://substackcdn.com/image/fetch/$s_!2s0p!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F022814b8-2dee-4841-87b1-466047cde224_1920x562.png 1272w, https://substackcdn.com/image/fetch/$s_!2s0p!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F022814b8-2dee-4841-87b1-466047cde224_1920x562.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The GLM-5.3 run decoded at 179 tokens/s, and a ten-step agent loop used 16.85 s of server time.</figcaption></figure></div><p>As the screenshot above shows, the run decoded at 179 tokens/s, and prefill finished in 0.211 s.</p><p>That is about 4x the ~42 tokens/s ceiling of one H100, since the deployment spreads GLM-5.3&#8217;s experts across many chips. In the agent loop, decode took 83% of the 16.85 s of server time.</p><div><hr></div><h4>Disaggregated prefill and decode</h4><p>Prefill and decode stress hardware differently, so disaggregated inference runs them on different chips.</p><p>In a live demo with an NVIDIA B200 for prefill and an SN40 RDU for decode, SambaNova reported 2x the speed of a B200-only setup, verified by Artificial Analysis.</p><p>The visual below shows the split:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!_eiM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6749a286-40fc-4689-add5-54dce447c36e_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_eiM!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6749a286-40fc-4689-add5-54dce447c36e_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!_eiM!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6749a286-40fc-4689-add5-54dce447c36e_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!_eiM!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6749a286-40fc-4689-add5-54dce447c36e_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!_eiM!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6749a286-40fc-4689-add5-54dce447c36e_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!_eiM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6749a286-40fc-4689-add5-54dce447c36e_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6749a286-40fc-4689-add5-54dce447c36e_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:161107,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225409773?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6749a286-40fc-4689-add5-54dce447c36e_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!_eiM!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6749a286-40fc-4689-add5-54dce447c36e_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!_eiM!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6749a286-40fc-4689-add5-54dce447c36e_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!_eiM!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6749a286-40fc-4689-add5-54dce447c36e_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!_eiM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6749a286-40fc-4689-add5-54dce447c36e_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Disaggregated inference runs prefill on a GPU and decode on an RDU, and SambaNova reported 2x the speed of a B200-only setup.</figcaption></figure></div><h4>Takeaways</h4><p>Decode with one request is limited by memory bandwidth, and GPU kernel boundaries waste part of it.</p><p>A dataflow chip lays the decoder out in space and keeps activations in SRAM, so HBM bandwidth goes to weights and KV cache.</p><p>The trade-off is flexibility, since the compiler maps a fixed graph ahead of time.</p><p>If your agent is slow, its prefill and decode split is the first thing to measure.</p><p>&#128073; Over to you: what share of your agent&#8217;s model time goes to decode?</p><p>SambaNova&#8217;s page on its dataflow architecture goes deeper <strong><a href="https://fandf.co/4zxBt9Fhttps://fandf.co/4ezjhEu">here</a></strong>.</p><p>If you will be at NeurIPS, you can find SambaNova at the booth in Sydney (December 6-12) and Atlanta (December 8-13), or <strong><a href="https://sambanova.ai/events">register for SambaHouse AI Night in Sydney on December 9 here</a></strong>.</p><p><em>Thanks to SambaNova for working with us on this article!</em></p><div><hr></div><h3><a href="https://www.dailydoseofds.com/diffusion-models-part-1/">Autoregressive LLMs vs. diffusion LLMs, clearly explained</a></h3><p>Generating 500 tokens doesn&#8217;t always require 500 sequential decoding steps.</p><p>Autoregressive models extend a response one token at a time. Masked diffusion models can predict multiple positions together, then repeat that process to complete the response.</p><p>Both can use transformer components such as embeddings, attention, and feed-forward layers. Their attention patterns and training objectives differ.</p><h4>1) Autoregressive LLMs</h4><p>In standard autoregressive decoding, each forward pass produces a probability distribution for the next token.</p><p>The selected token is appended, and generation continues from that longer sequence.</p><p>Causal attention prevents a position from accessing future positions. Earlier token representations therefore stay unchanged as new tokens arrive.</p><p>That allows their KV to be cached and reused.</p><p>A 500-token response normally requires 500 sequential token-generation steps, although techniques such as speculative decoding can reduce expensive model passes.</p><h4>2) Diffusion LLMs</h4><p>In the masked diffusion approach shown below, generation starts with masked output positions alongside the prompt.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!lWJ1!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b706860-2aad-44d4-8a87-4b058e991c5e_1200x976.gif 424w, https://substackcdn.com/image/fetch/$s_!lWJ1!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b706860-2aad-44d4-8a87-4b058e991c5e_1200x976.gif 848w, https://substackcdn.com/image/fetch/$s_!lWJ1!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b706860-2aad-44d4-8a87-4b058e991c5e_1200x976.gif 1272w, https://substackcdn.com/image/fetch/$s_!lWJ1!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b706860-2aad-44d4-8a87-4b058e991c5e_1200x976.gif 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!lWJ1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b706860-2aad-44d4-8a87-4b058e991c5e_1200x976.gif" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4b706860-2aad-44d4-8a87-4b058e991c5e_1200x976.gif&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:3149620,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/gif&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225409773?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b706860-2aad-44d4-8a87-4b058e991c5e_1200x976.gif&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!lWJ1!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b706860-2aad-44d4-8a87-4b058e991c5e_1200x976.gif 424w, https://substackcdn.com/image/fetch/$s_!lWJ1!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b706860-2aad-44d4-8a87-4b058e991c5e_1200x976.gif 848w, https://substackcdn.com/image/fetch/$s_!lWJ1!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b706860-2aad-44d4-8a87-4b058e991c5e_1200x976.gif 1272w, https://substackcdn.com/image/fetch/$s_!lWJ1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b706860-2aad-44d4-8a87-4b058e991c5e_1200x976.gif 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>A denoising pass predicts candidate tokens for multiple positions using bidirectional context.</p><p>With confidence-based sampling, selected predictions are retained while other positions remain masked for subsequent steps.</p><p>The model progressively fills the sequence. It doesn&#8217;t necessarily produce a complete draft and repeatedly rewrite every token.</p><p>Some implementations generate block by block rather than processing the entire response together.</p><p>Multiple tokens can be finalized per step, allowing fewer steps than output tokens. The step budget affects both speed and quality.</p><p>There are some engineering tradeoffs.</p><p>Bidirectional attention makes standard autoregressive KV reuse harder. A token&#8217;s representation can change when surrounding positions are updated, even if that token itself stays fixed.</p><p>Caching therefore depends on the architecture and decoding strategy. Block diffusion, for example, can support KV caching across completed blocks.</p><p>Fewer steps also don&#8217;t guarantee lower latency. Each step may process many positions, so total computation and implementation matter.</p><p>Bidirectional conditioning also supports filling gaps in text or code.</p><p><strong>You can read the diffusion models series here:</strong></p><ul><li><p><strong><a href="https://www.dailydoseofds.com/diffusion-models-part-1/">Read Part 1 on the theory, math, and why they work &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/diffusion-models-part-2/">Read Part 2 on training, inference, and practical engineering &#8594;</a></strong></p></li></ul><p>&#128073; Over to you: Where would a diffusion LLM be a better fit than an autoregressive one in your stack?</p><p>Good day!</p>]]></content:encoded></item><item><title><![CDATA[Security Breaks in the Gaps Between Your Reviews]]></title><description><![CDATA[Everything you need to understand about why the riskiest access control bugs sit between services, and how to catch them before and after they merge.]]></description><link>https://blog.dailydoseofds.com/p/security-breaks-in-the-gaps-between</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/security-breaks-in-the-gaps-between</guid><pubDate>Mon, 16 Nov 2026 23:02:23 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!AgsQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ac60bbe-7efc-4055-ba4b-3fd9bab152b3_1990x1338.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<h3><a href="https://github.com/alookai/alook/">Always-on agents, clearly explained!</a></h3><p>An agent in a chat window only works while someone is typing to it. Once the tab closes, it stops and can&#8217;t act on anything that happens next.</p><p><strong><a href="https://github.com/alookai/alook">Always-on agents</a></strong> keep state across sessions and act without supervision. Anthropic&#8217;s new guide on building them shows that the agent loop itself doesn&#8217;t change, and everything new is the architecture around it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!MUAY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc14fc234-3ade-4c97-b822-14e64a358c20_910x512.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!MUAY!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc14fc234-3ade-4c97-b822-14e64a358c20_910x512.png 424w, https://substackcdn.com/image/fetch/$s_!MUAY!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc14fc234-3ade-4c97-b822-14e64a358c20_910x512.png 848w, https://substackcdn.com/image/fetch/$s_!MUAY!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc14fc234-3ade-4c97-b822-14e64a358c20_910x512.png 1272w, https://substackcdn.com/image/fetch/$s_!MUAY!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc14fc234-3ade-4c97-b822-14e64a358c20_910x512.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!MUAY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc14fc234-3ade-4c97-b822-14e64a358c20_910x512.png" width="910" height="512" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c14fc234-3ade-4c97-b822-14e64a358c20_910x512.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:512,&quot;width&quot;:910,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:50086,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225265145?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc14fc234-3ade-4c97-b822-14e64a358c20_910x512.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!MUAY!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc14fc234-3ade-4c97-b822-14e64a358c20_910x512.png 424w, https://substackcdn.com/image/fetch/$s_!MUAY!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc14fc234-3ade-4c97-b822-14e64a358c20_910x512.png 848w, https://substackcdn.com/image/fetch/$s_!MUAY!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc14fc234-3ade-4c97-b822-14e64a358c20_910x512.png 1272w, https://substackcdn.com/image/fetch/$s_!MUAY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc14fc234-3ade-4c97-b822-14e64a358c20_910x512.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The agent also isn&#8217;t running all the time. It sleeps and wakes up for short runs, where each run receives an event, does the work, and saves what happened.</p><p>That cycle needs five primitives.</p><p>1) An event inbox</p><p>Webhooks, scheduled tasks, and user messages all land in a durable table instead of calling the agent directly. Direct calls let two events start competing runs, and a crash loses the event.</p><p>A single worker per inbox reads the table and starts runs. Events are deduplicated by source ID, and if a worker dies, a new one retries the pending event.</p><p>2) Timers</p><p>Follow-ups are deadlines the agent sets for itself, and schedules are recurring jobs. A scheduler polls for due timers and adds an event to the inbox, so every run still starts there.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Rw2u!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc124b37a-1ef8-49f5-ad3a-466fb1c78ce1_2752x995.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Rw2u!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc124b37a-1ef8-49f5-ad3a-466fb1c78ce1_2752x995.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Rw2u!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc124b37a-1ef8-49f5-ad3a-466fb1c78ce1_2752x995.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Rw2u!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc124b37a-1ef8-49f5-ad3a-466fb1c78ce1_2752x995.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Rw2u!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc124b37a-1ef8-49f5-ad3a-466fb1c78ce1_2752x995.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Rw2u!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc124b37a-1ef8-49f5-ad3a-466fb1c78ce1_2752x995.jpeg" width="2752" height="995" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c124b37a-1ef8-49f5-ad3a-466fb1c78ce1_2752x995.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:995,&quot;width&quot;:2752,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:227060,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225265145?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86c51f69-7476-4d73-9e45-e87c1c01051f_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Rw2u!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc124b37a-1ef8-49f5-ad3a-466fb1c78ce1_2752x995.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Rw2u!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc124b37a-1ef8-49f5-ad3a-466fb1c78ce1_2752x995.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Rw2u!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc124b37a-1ef8-49f5-ad3a-466fb1c78ce1_2752x995.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Rw2u!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc124b37a-1ef8-49f5-ad3a-466fb1c78ce1_2752x995.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>3) Memory</p><p>Each run starts with a fresh context, so anything worth keeping lives in durable storage. The agent stores only what it can&#8217;t look up again and keeps pointers for the rest.</p><p>Instructions, an index, and a log of what the user was already told load on every run. User notes and per-task progress files are read on demand, and anything that must survive compaction gets written during the run.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!v8UA!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ccdf179-7ba5-4bc9-aeb0-f4434983e915_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!v8UA!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ccdf179-7ba5-4bc9-aeb0-f4434983e915_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!v8UA!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ccdf179-7ba5-4bc9-aeb0-f4434983e915_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!v8UA!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ccdf179-7ba5-4bc9-aeb0-f4434983e915_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!v8UA!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ccdf179-7ba5-4bc9-aeb0-f4434983e915_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!v8UA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ccdf179-7ba5-4bc9-aeb0-f4434983e915_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6ccdf179-7ba5-4bc9-aeb0-f4434983e915_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:157940,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225265145?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ccdf179-7ba5-4bc9-aeb0-f4434983e915_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!v8UA!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ccdf179-7ba5-4bc9-aeb0-f4434983e915_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!v8UA!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ccdf179-7ba5-4bc9-aeb0-f4434983e915_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!v8UA!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ccdf179-7ba5-4bc9-aeb0-f4434983e915_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!v8UA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ccdf179-7ba5-4bc9-aeb0-f4434983e915_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>4) Permissions</p><p>This carries the most risk, since the run holds the user&#8217;s access and nobody is watching. The agent sends requests without credentials, and a proxy adds them before forwarding.</p><p>For decisions that need a human, the agent makes a deferred tool call. The session is saved mid-call, the user sees an approval card, and the answer returns as an inbox event that resumes the session.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Ysf_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68dfc287-3a0b-4e13-bed4-cc3b5dd7a0c5_2752x845.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Ysf_!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68dfc287-3a0b-4e13-bed4-cc3b5dd7a0c5_2752x845.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Ysf_!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68dfc287-3a0b-4e13-bed4-cc3b5dd7a0c5_2752x845.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Ysf_!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68dfc287-3a0b-4e13-bed4-cc3b5dd7a0c5_2752x845.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Ysf_!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68dfc287-3a0b-4e13-bed4-cc3b5dd7a0c5_2752x845.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Ysf_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68dfc287-3a0b-4e13-bed4-cc3b5dd7a0c5_2752x845.jpeg" width="2752" height="845" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/68dfc287-3a0b-4e13-bed4-cc3b5dd7a0c5_2752x845.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:845,&quot;width&quot;:2752,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:160054,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225265145?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52e7c5a0-45a7-4f46-a6b4-14a31cc67a74_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Ysf_!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68dfc287-3a0b-4e13-bed4-cc3b5dd7a0c5_2752x845.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Ysf_!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68dfc287-3a0b-4e13-bed4-cc3b5dd7a0c5_2752x845.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Ysf_!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68dfc287-3a0b-4e13-bed4-cc3b5dd7a0c5_2752x845.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Ysf_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68dfc287-3a0b-4e13-bed4-cc3b5dd7a0c5_2752x845.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>5) An outbox</p><p>The user gets the outcome, not every step. A run ends by sending a message, deciding nothing is worth sending, or asking for permission.</p><p>Too many messages make the agent noisy, and too few miss what matters, so the guide recommends starting quiet and tuning it with evals.</p><p>Timers and user replies keep feeding new events into the inbox, and those short runs repeated indefinitely are what make the agent look always on.</p><p>To see these ideas running in a real system, <strong><a href="https://github.com/alookai/alook">Alook</a></strong> (open-source) turns coding agents like Claude Code into an org chart with roles and real email inboxes for multi-agent collaboration.</p><p>Alook GitHub: <strong><a href="https://github.com/alookai/alook">https://github.com/alookai/alook</a></strong></p><p>(don&#8217;t forget to star it &#11088;&#65039;)</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!mD1P!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff250b92e-a7cf-46ef-a0e2-fda488ae02af_848x677.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mD1P!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff250b92e-a7cf-46ef-a0e2-fda488ae02af_848x677.png 424w, https://substackcdn.com/image/fetch/$s_!mD1P!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff250b92e-a7cf-46ef-a0e2-fda488ae02af_848x677.png 848w, https://substackcdn.com/image/fetch/$s_!mD1P!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff250b92e-a7cf-46ef-a0e2-fda488ae02af_848x677.png 1272w, https://substackcdn.com/image/fetch/$s_!mD1P!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff250b92e-a7cf-46ef-a0e2-fda488ae02af_848x677.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!mD1P!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff250b92e-a7cf-46ef-a0e2-fda488ae02af_848x677.png" width="579" height="462.2441037735849" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f250b92e-a7cf-46ef-a0e2-fda488ae02af_848x677.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:677,&quot;width&quot;:848,&quot;resizeWidth&quot;:579,&quot;bytes&quot;:85985,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225265145?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff250b92e-a7cf-46ef-a0e2-fda488ae02af_848x677.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!mD1P!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff250b92e-a7cf-46ef-a0e2-fda488ae02af_848x677.png 424w, https://substackcdn.com/image/fetch/$s_!mD1P!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff250b92e-a7cf-46ef-a0e2-fda488ae02af_848x677.png 848w, https://substackcdn.com/image/fetch/$s_!mD1P!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff250b92e-a7cf-46ef-a0e2-fda488ae02af_848x677.png 1272w, https://substackcdn.com/image/fetch/$s_!mD1P!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff250b92e-a7cf-46ef-a0e2-fda488ae02af_848x677.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><h3><a href="https://coderabbit.link/dailydoseofds-003">Security breaks in the gaps between your reviews</a></h3><p>On October 21, GitLab patched a flaw dating back to 2014 that let anyone run code on self-managed servers without logging in.</p><p>That same week, researchers using an OpenAI model uncovered critical PHP bugs missed for decades, while the NSA urged organizations to patch or take affected systems offline.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!EOFB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bf57d03-4c09-42bd-9f46-23d7029cf8b3_2752x1536.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!EOFB!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bf57d03-4c09-42bd-9f46-23d7029cf8b3_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!EOFB!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bf57d03-4c09-42bd-9f46-23d7029cf8b3_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!EOFB!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bf57d03-4c09-42bd-9f46-23d7029cf8b3_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!EOFB!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bf57d03-4c09-42bd-9f46-23d7029cf8b3_2752x1536.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!EOFB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bf57d03-4c09-42bd-9f46-23d7029cf8b3_2752x1536.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5bf57d03-4c09-42bd-9f46-23d7029cf8b3_2752x1536.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1495315,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bf57d03-4c09-42bd-9f46-23d7029cf8b3_2752x1536.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!EOFB!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bf57d03-4c09-42bd-9f46-23d7029cf8b3_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!EOFB!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bf57d03-4c09-42bd-9f46-23d7029cf8b3_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!EOFB!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bf57d03-4c09-42bd-9f46-23d7029cf8b3_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!EOFB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bf57d03-4c09-42bd-9f46-23d7029cf8b3_2752x1536.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Code that ran safely for years is now being read by models that can find these bugs, and anyone with a credit card or a laptop can run those models. The time from a bug going public to its first exploit has dropped from almost a year in 2021 to just over a day in 2026.</p><p>In a November 6, 2026 post, <strong><a href="https://x.com/coderabbitai/status/2118824048914227424">CodeRabbit reported finding more than 43,000 high or critical vulnerabilities</a></strong> by running their Deep Scan feature over the last 10 days.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!tBMu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8df1dbb1-499a-4a6f-9edd-b54e9ab00438_2752x1536.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!tBMu!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8df1dbb1-499a-4a6f-9edd-b54e9ab00438_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!tBMu!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8df1dbb1-499a-4a6f-9edd-b54e9ab00438_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!tBMu!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8df1dbb1-499a-4a6f-9edd-b54e9ab00438_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!tBMu!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8df1dbb1-499a-4a6f-9edd-b54e9ab00438_2752x1536.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!tBMu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8df1dbb1-499a-4a6f-9edd-b54e9ab00438_2752x1536.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8df1dbb1-499a-4a6f-9edd-b54e9ab00438_2752x1536.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:753511,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8df1dbb1-499a-4a6f-9edd-b54e9ab00438_2752x1536.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!tBMu!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8df1dbb1-499a-4a6f-9edd-b54e9ab00438_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!tBMu!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8df1dbb1-499a-4a6f-9edd-b54e9ab00438_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!tBMu!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8df1dbb1-499a-4a6f-9edd-b54e9ab00438_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!tBMu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8df1dbb1-499a-4a6f-9edd-b54e9ab00438_2752x1536.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Access control bugs are easy to miss in your own codebase because they rarely live in one file. A gateway checks who you are, but the service behind it never checks whether the order belongs to you.</p><p>We wanted to see whether a security tool could catch that gap, so we built two small services with Claude Code and ran <strong><a href="https://coderabbit.link/dailydoseofds-003">CodeRabbit</a></strong> across both.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!AgsQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ac60bbe-7efc-4055-ba4b-3fd9bab152b3_1990x1338.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!AgsQ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ac60bbe-7efc-4055-ba4b-3fd9bab152b3_1990x1338.png 424w, https://substackcdn.com/image/fetch/$s_!AgsQ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ac60bbe-7efc-4055-ba4b-3fd9bab152b3_1990x1338.png 848w, https://substackcdn.com/image/fetch/$s_!AgsQ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ac60bbe-7efc-4055-ba4b-3fd9bab152b3_1990x1338.png 1272w, https://substackcdn.com/image/fetch/$s_!AgsQ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ac60bbe-7efc-4055-ba4b-3fd9bab152b3_1990x1338.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!AgsQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ac60bbe-7efc-4055-ba4b-3fd9bab152b3_1990x1338.png" width="1456" height="979" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7ac60bbe-7efc-4055-ba4b-3fd9bab152b3_1990x1338.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:979,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1531817,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ac60bbe-7efc-4055-ba4b-3fd9bab152b3_1990x1338.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!AgsQ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ac60bbe-7efc-4055-ba4b-3fd9bab152b3_1990x1338.png 424w, https://substackcdn.com/image/fetch/$s_!AgsQ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ac60bbe-7efc-4055-ba4b-3fd9bab152b3_1990x1338.png 848w, https://substackcdn.com/image/fetch/$s_!AgsQ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ac60bbe-7efc-4055-ba4b-3fd9bab152b3_1990x1338.png 1272w, https://substackcdn.com/image/fetch/$s_!AgsQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ac60bbe-7efc-4055-ba4b-3fd9bab152b3_1990x1338.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><strong><a href="https://coderabbit.link/dailydoseofds-003">CodeRabbit</a></strong> is the trust layer for agentic software development. <strong><a href="https://www.coderabbit.ai/security">CodeRabbit Secure</a></strong> extends that with security checks on pull requests, existing code, and linked repositories, so it can trace a vulnerability from one service into another.</p><p>For the test, we recreated the pattern behind last year&#8217;s Next.js middleware auth bypass.</p><h4>How the Next.js bypass worked</h4><p>Next.js is a React framework with 143K+ GitHub stars. It lets you run middleware before a request reaches a page or route, and many apps check whether the user is logged in there.</p><p>To stop middleware from calling itself in a loop, Next.js marked its own internal requests with a header called <code>x-middleware-subrequest</code>. Any request carrying that header skipped the middleware.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!G4lU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbab290e-5c6a-450d-99b9-dc155d0e5754_2752x1536.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!G4lU!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbab290e-5c6a-450d-99b9-dc155d0e5754_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!G4lU!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbab290e-5c6a-450d-99b9-dc155d0e5754_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!G4lU!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbab290e-5c6a-450d-99b9-dc155d0e5754_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!G4lU!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbab290e-5c6a-450d-99b9-dc155d0e5754_2752x1536.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!G4lU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbab290e-5c6a-450d-99b9-dc155d0e5754_2752x1536.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bbab290e-5c6a-450d-99b9-dc155d0e5754_2752x1536.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1103918,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbab290e-5c6a-450d-99b9-dc155d0e5754_2752x1536.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!G4lU!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbab290e-5c6a-450d-99b9-dc155d0e5754_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!G4lU!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbab290e-5c6a-450d-99b9-dc155d0e5754_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!G4lU!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbab290e-5c6a-450d-99b9-dc155d0e5754_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!G4lU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbab290e-5c6a-450d-99b9-dc155d0e5754_2752x1536.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The problem was that Next.js never checked where that header came from. Anyone could add that header to a request from outside and get past the login check.</p><p>The flaw was tracked as CVE-2025-29927, rated critical, and fixed in 15.2.3 with patches for older versions. Self-hosted apps that checked logins in middleware were exposed.</p><p>The same pattern can sit between two services, where one trusts that the other already checked.</p><h4>The two services we built</h4><p>The test uses two small repositories that depend on each other. <code>shop-gateway</code> handles login, and <code>shop-orders</code> only gets requests after they pass through the gateway.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!4j1a!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38cd1b0a-7534-4d4f-b602-55602f1fd453_1456x813.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!4j1a!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38cd1b0a-7534-4d4f-b602-55602f1fd453_1456x813.png 424w, https://substackcdn.com/image/fetch/$s_!4j1a!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38cd1b0a-7534-4d4f-b602-55602f1fd453_1456x813.png 848w, https://substackcdn.com/image/fetch/$s_!4j1a!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38cd1b0a-7534-4d4f-b602-55602f1fd453_1456x813.png 1272w, https://substackcdn.com/image/fetch/$s_!4j1a!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38cd1b0a-7534-4d4f-b602-55602f1fd453_1456x813.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!4j1a!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38cd1b0a-7534-4d4f-b602-55602f1fd453_1456x813.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/38cd1b0a-7534-4d4f-b602-55602f1fd453_1456x813.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:328999,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38cd1b0a-7534-4d4f-b602-55602f1fd453_1456x813.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!4j1a!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38cd1b0a-7534-4d4f-b602-55602f1fd453_1456x813.png 424w, https://substackcdn.com/image/fetch/$s_!4j1a!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38cd1b0a-7534-4d4f-b602-55602f1fd453_1456x813.png 848w, https://substackcdn.com/image/fetch/$s_!4j1a!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38cd1b0a-7534-4d4f-b602-55602f1fd453_1456x813.png 1272w, https://substackcdn.com/image/fetch/$s_!4j1a!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38cd1b0a-7534-4d4f-b602-55602f1fd453_1456x813.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>We gave Claude Code one plain feature prompt for each, with nothing about security in either. </p><p>The gateway uses FastAPI and JWT login for two users, alice (user 1) and bob (user 2), and forwards <code>/api/orders</code> requests with the user&#8217;s ID in an <code>X-User-Id</code> header.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!_D8e!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdec62861-eaac-443d-9129-9d3f3bd4a7d9_1886x924.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_D8e!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdec62861-eaac-443d-9129-9d3f3bd4a7d9_1886x924.png 424w, https://substackcdn.com/image/fetch/$s_!_D8e!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdec62861-eaac-443d-9129-9d3f3bd4a7d9_1886x924.png 848w, https://substackcdn.com/image/fetch/$s_!_D8e!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdec62861-eaac-443d-9129-9d3f3bd4a7d9_1886x924.png 1272w, https://substackcdn.com/image/fetch/$s_!_D8e!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdec62861-eaac-443d-9129-9d3f3bd4a7d9_1886x924.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!_D8e!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdec62861-eaac-443d-9129-9d3f3bd4a7d9_1886x924.png" width="1456" height="713" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/dec62861-eaac-443d-9129-9d3f3bd4a7d9_1886x924.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:713,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:388148,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdec62861-eaac-443d-9129-9d3f3bd4a7d9_1886x924.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!_D8e!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdec62861-eaac-443d-9129-9d3f3bd4a7d9_1886x924.png 424w, https://substackcdn.com/image/fetch/$s_!_D8e!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdec62861-eaac-443d-9129-9d3f3bd4a7d9_1886x924.png 848w, https://substackcdn.com/image/fetch/$s_!_D8e!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdec62861-eaac-443d-9129-9d3f3bd4a7d9_1886x924.png 1272w, https://substackcdn.com/image/fetch/$s_!_D8e!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdec62861-eaac-443d-9129-9d3f3bd4a7d9_1886x924.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The orders service uses FastAPI and SQLite, reads the user from that header, and starts with orders 1 and 2 for alice and order 3 for bob. Both services ran, and all 27 tests passed.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!DgKu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe50fe563-2b53-43a9-8bb9-cbd595b3de5c_1694x946.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!DgKu!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe50fe563-2b53-43a9-8bb9-cbd595b3de5c_1694x946.png 424w, https://substackcdn.com/image/fetch/$s_!DgKu!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe50fe563-2b53-43a9-8bb9-cbd595b3de5c_1694x946.png 848w, https://substackcdn.com/image/fetch/$s_!DgKu!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe50fe563-2b53-43a9-8bb9-cbd595b3de5c_1694x946.png 1272w, https://substackcdn.com/image/fetch/$s_!DgKu!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe50fe563-2b53-43a9-8bb9-cbd595b3de5c_1694x946.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!DgKu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe50fe563-2b53-43a9-8bb9-cbd595b3de5c_1694x946.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e50fe563-2b53-43a9-8bb9-cbd595b3de5c_1694x946.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:353536,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe50fe563-2b53-43a9-8bb9-cbd595b3de5c_1694x946.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!DgKu!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe50fe563-2b53-43a9-8bb9-cbd595b3de5c_1694x946.png 424w, https://substackcdn.com/image/fetch/$s_!DgKu!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe50fe563-2b53-43a9-8bb9-cbd595b3de5c_1694x946.png 848w, https://substackcdn.com/image/fetch/$s_!DgKu!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe50fe563-2b53-43a9-8bb9-cbd595b3de5c_1694x946.png 1272w, https://substackcdn.com/image/fetch/$s_!DgKu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe50fe563-2b53-43a9-8bb9-cbd595b3de5c_1694x946.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Claude Code also added safeguards we hadn&#8217;t asked for. The gateway strips any <code>X-User-Id</code> a client sends, and the orders service checked that each order belonged to the caller.</p><p>In a 100-line service, that check is easy to get right. To recreate a codebase where the check gets missed, we removed it from the endpoint that fetches a single order (<code>GET /orders/{order_id}</code>), so the service now returns any order by its ID, no matter who asks.</p><p>With both services running, we logged in as bob and asked for his own order, number 3, which came back as expected.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!COd9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5475bdf-073e-417c-8200-a807241abca7_1438x1390.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!COd9!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5475bdf-073e-417c-8200-a807241abca7_1438x1390.png 424w, https://substackcdn.com/image/fetch/$s_!COd9!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5475bdf-073e-417c-8200-a807241abca7_1438x1390.png 848w, https://substackcdn.com/image/fetch/$s_!COd9!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5475bdf-073e-417c-8200-a807241abca7_1438x1390.png 1272w, https://substackcdn.com/image/fetch/$s_!COd9!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5475bdf-073e-417c-8200-a807241abca7_1438x1390.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!COd9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5475bdf-073e-417c-8200-a807241abca7_1438x1390.png" width="1438" height="1390" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e5475bdf-073e-417c-8200-a807241abca7_1438x1390.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1390,&quot;width&quot;:1438,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:430869,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5475bdf-073e-417c-8200-a807241abca7_1438x1390.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!COd9!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5475bdf-073e-417c-8200-a807241abca7_1438x1390.png 424w, https://substackcdn.com/image/fetch/$s_!COd9!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5475bdf-073e-417c-8200-a807241abca7_1438x1390.png 848w, https://substackcdn.com/image/fetch/$s_!COd9!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5475bdf-073e-417c-8200-a807241abca7_1438x1390.png 1272w, https://substackcdn.com/image/fetch/$s_!COd9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5475bdf-073e-417c-8200-a807241abca7_1438x1390.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Then we asked for order 1, which belongs to alice. The gateway returned it with <code>"user_id": 1</code>, so bob could read alice&#8217;s order.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!zpj9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e0cd63b-0ffc-4318-93fa-2ef6cad2be2b_1626x1342.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zpj9!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e0cd63b-0ffc-4318-93fa-2ef6cad2be2b_1626x1342.png 424w, https://substackcdn.com/image/fetch/$s_!zpj9!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e0cd63b-0ffc-4318-93fa-2ef6cad2be2b_1626x1342.png 848w, https://substackcdn.com/image/fetch/$s_!zpj9!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e0cd63b-0ffc-4318-93fa-2ef6cad2be2b_1626x1342.png 1272w, https://substackcdn.com/image/fetch/$s_!zpj9!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e0cd63b-0ffc-4318-93fa-2ef6cad2be2b_1626x1342.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!zpj9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e0cd63b-0ffc-4318-93fa-2ef6cad2be2b_1626x1342.png" width="1456" height="1202" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6e0cd63b-0ffc-4318-93fa-2ef6cad2be2b_1626x1342.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1202,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:476773,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e0cd63b-0ffc-4318-93fa-2ef6cad2be2b_1626x1342.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!zpj9!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e0cd63b-0ffc-4318-93fa-2ef6cad2be2b_1626x1342.png 424w, https://substackcdn.com/image/fetch/$s_!zpj9!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e0cd63b-0ffc-4318-93fa-2ef6cad2be2b_1626x1342.png 848w, https://substackcdn.com/image/fetch/$s_!zpj9!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e0cd63b-0ffc-4318-93fa-2ef6cad2be2b_1626x1342.png 1272w, https://substackcdn.com/image/fetch/$s_!zpj9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e0cd63b-0ffc-4318-93fa-2ef6cad2be2b_1626x1342.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>From there, we ran CodeRabbit on the existing code first, then on a new pull request, a gateway refactor, and a commit that skipped review.</p><h4>Scanning both repositories</h4><p>CodeRabbit Secure scans code in two ways. PR scans review each new change before it merges, and a Deep Scan reviews all the code already in the repository. In my test, the missing check was already in the orders service, so we started with a Deep Scan.</p><p>First, we linked <code>shop-gateway</code> to <code>shop-orders</code> with CodeRabbit&#8217;s Multi-Repo Analysis, so the scan could read and understand context from both repositories.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!8YaP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b11cf9f-9aee-4b27-bcf1-3407f2e4ee7f_2148x1210.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8YaP!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b11cf9f-9aee-4b27-bcf1-3407f2e4ee7f_2148x1210.png 424w, https://substackcdn.com/image/fetch/$s_!8YaP!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b11cf9f-9aee-4b27-bcf1-3407f2e4ee7f_2148x1210.png 848w, https://substackcdn.com/image/fetch/$s_!8YaP!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b11cf9f-9aee-4b27-bcf1-3407f2e4ee7f_2148x1210.png 1272w, https://substackcdn.com/image/fetch/$s_!8YaP!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b11cf9f-9aee-4b27-bcf1-3407f2e4ee7f_2148x1210.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!8YaP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b11cf9f-9aee-4b27-bcf1-3407f2e4ee7f_2148x1210.png" width="1456" height="820" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9b11cf9f-9aee-4b27-bcf1-3407f2e4ee7f_2148x1210.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:820,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:462842,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b11cf9f-9aee-4b27-bcf1-3407f2e4ee7f_2148x1210.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!8YaP!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b11cf9f-9aee-4b27-bcf1-3407f2e4ee7f_2148x1210.png 424w, https://substackcdn.com/image/fetch/$s_!8YaP!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b11cf9f-9aee-4b27-bcf1-3407f2e4ee7f_2148x1210.png 848w, https://substackcdn.com/image/fetch/$s_!8YaP!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b11cf9f-9aee-4b27-bcf1-3407f2e4ee7f_2148x1210.png 1272w, https://substackcdn.com/image/fetch/$s_!8YaP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b11cf9f-9aee-4b27-bcf1-3407f2e4ee7f_2148x1210.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Then we ran the scan with no hints about the bug. It maps where requests enter the code, follows each one, and has an independent verifier re-check every finding before reporting it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!VoR_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9a6509f-7554-418e-834a-d407baf8aff1_1010x1248.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!VoR_!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9a6509f-7554-418e-834a-d407baf8aff1_1010x1248.png 424w, https://substackcdn.com/image/fetch/$s_!VoR_!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9a6509f-7554-418e-834a-d407baf8aff1_1010x1248.png 848w, https://substackcdn.com/image/fetch/$s_!VoR_!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9a6509f-7554-418e-834a-d407baf8aff1_1010x1248.png 1272w, https://substackcdn.com/image/fetch/$s_!VoR_!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9a6509f-7554-418e-834a-d407baf8aff1_1010x1248.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!VoR_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9a6509f-7554-418e-834a-d407baf8aff1_1010x1248.png" width="1010" height="1248" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c9a6509f-7554-418e-834a-d407baf8aff1_1010x1248.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1248,&quot;width&quot;:1010,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:337832,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9a6509f-7554-418e-834a-d407baf8aff1_1010x1248.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!VoR_!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9a6509f-7554-418e-834a-d407baf8aff1_1010x1248.png 424w, https://substackcdn.com/image/fetch/$s_!VoR_!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9a6509f-7554-418e-834a-d407baf8aff1_1010x1248.png 848w, https://substackcdn.com/image/fetch/$s_!VoR_!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9a6509f-7554-418e-834a-d407baf8aff1_1010x1248.png 1272w, https://substackcdn.com/image/fetch/$s_!VoR_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9a6509f-7554-418e-834a-d407baf8aff1_1010x1248.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The scan found the bug we had planted, the missing ownership check on the order lookup. CodeRabbit flagged it as an IDOR (CWE-639) and rated it Medium.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!5YRF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa53c93d7-dfd8-4dcc-85a1-db22f5c995fc_3028x1676.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!5YRF!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa53c93d7-dfd8-4dcc-85a1-db22f5c995fc_3028x1676.png 424w, https://substackcdn.com/image/fetch/$s_!5YRF!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa53c93d7-dfd8-4dcc-85a1-db22f5c995fc_3028x1676.png 848w, https://substackcdn.com/image/fetch/$s_!5YRF!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa53c93d7-dfd8-4dcc-85a1-db22f5c995fc_3028x1676.png 1272w, https://substackcdn.com/image/fetch/$s_!5YRF!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa53c93d7-dfd8-4dcc-85a1-db22f5c995fc_3028x1676.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!5YRF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa53c93d7-dfd8-4dcc-85a1-db22f5c995fc_3028x1676.png" width="1456" height="806" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a53c93d7-dfd8-4dcc-85a1-db22f5c995fc_3028x1676.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:806,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:630597,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa53c93d7-dfd8-4dcc-85a1-db22f5c995fc_3028x1676.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!5YRF!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa53c93d7-dfd8-4dcc-85a1-db22f5c995fc_3028x1676.png 424w, https://substackcdn.com/image/fetch/$s_!5YRF!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa53c93d7-dfd8-4dcc-85a1-db22f5c995fc_3028x1676.png 848w, https://substackcdn.com/image/fetch/$s_!5YRF!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa53c93d7-dfd8-4dcc-85a1-db22f5c995fc_3028x1676.png 1272w, https://substackcdn.com/image/fetch/$s_!5YRF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa53c93d7-dfd8-4dcc-85a1-db22f5c995fc_3028x1676.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The finding also shows how it got there. It marks the bug as reachable from outside, and its trace follows a request in through the gateway, into the orders service, and down to the query that loads the order without checking who owns it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!rj6B!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbecf14f9-d360-4a41-8104-7435803bfe02_1889x1852.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!rj6B!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbecf14f9-d360-4a41-8104-7435803bfe02_1889x1852.png 424w, https://substackcdn.com/image/fetch/$s_!rj6B!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbecf14f9-d360-4a41-8104-7435803bfe02_1889x1852.png 848w, https://substackcdn.com/image/fetch/$s_!rj6B!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbecf14f9-d360-4a41-8104-7435803bfe02_1889x1852.png 1272w, https://substackcdn.com/image/fetch/$s_!rj6B!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbecf14f9-d360-4a41-8104-7435803bfe02_1889x1852.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!rj6B!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbecf14f9-d360-4a41-8104-7435803bfe02_1889x1852.png" width="1456" height="1427" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/becf14f9-d360-4a41-8104-7435803bfe02_1889x1852.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1427,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:548027,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbecf14f9-d360-4a41-8104-7435803bfe02_1889x1852.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!rj6B!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbecf14f9-d360-4a41-8104-7435803bfe02_1889x1852.png 424w, https://substackcdn.com/image/fetch/$s_!rj6B!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbecf14f9-d360-4a41-8104-7435803bfe02_1889x1852.png 848w, https://substackcdn.com/image/fetch/$s_!rj6B!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbecf14f9-d360-4a41-8104-7435803bfe02_1889x1852.png 1272w, https://substackcdn.com/image/fetch/$s_!rj6B!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbecf14f9-d360-4a41-8104-7435803bfe02_1889x1852.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>That trace crosses into the gateway&#8217;s code, so the finding cites four parts of <code>shop-gateway</code>, including how the gateway turns the login token into a user ID and forwards it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!7YLR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F391ef77b-0e55-4ea2-ad27-33f2ad560eb1_1854x960.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!7YLR!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F391ef77b-0e55-4ea2-ad27-33f2ad560eb1_1854x960.png 424w, https://substackcdn.com/image/fetch/$s_!7YLR!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F391ef77b-0e55-4ea2-ad27-33f2ad560eb1_1854x960.png 848w, https://substackcdn.com/image/fetch/$s_!7YLR!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F391ef77b-0e55-4ea2-ad27-33f2ad560eb1_1854x960.png 1272w, https://substackcdn.com/image/fetch/$s_!7YLR!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F391ef77b-0e55-4ea2-ad27-33f2ad560eb1_1854x960.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!7YLR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F391ef77b-0e55-4ea2-ad27-33f2ad560eb1_1854x960.png" width="1456" height="754" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/391ef77b-0e55-4ea2-ad27-33f2ad560eb1_1854x960.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:754,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:331663,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F391ef77b-0e55-4ea2-ad27-33f2ad560eb1_1854x960.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!7YLR!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F391ef77b-0e55-4ea2-ad27-33f2ad560eb1_1854x960.png 424w, https://substackcdn.com/image/fetch/$s_!7YLR!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F391ef77b-0e55-4ea2-ad27-33f2ad560eb1_1854x960.png 848w, https://substackcdn.com/image/fetch/$s_!7YLR!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F391ef77b-0e55-4ea2-ad27-33f2ad560eb1_1854x960.png 1272w, https://substackcdn.com/image/fetch/$s_!7YLR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F391ef77b-0e55-4ea2-ad27-33f2ad560eb1_1854x960.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Nothing in those parts checks who owns an order, so the check had to happen in <code>shop-orders</code>. CodeRabbit could show that because the two repositories were linked and it read both of them.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!dVaM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a993926-2092-41dc-a48f-107672a4c475_3056x1886.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!dVaM!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a993926-2092-41dc-a48f-107672a4c475_3056x1886.png 424w, https://substackcdn.com/image/fetch/$s_!dVaM!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a993926-2092-41dc-a48f-107672a4c475_3056x1886.png 848w, https://substackcdn.com/image/fetch/$s_!dVaM!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a993926-2092-41dc-a48f-107672a4c475_3056x1886.png 1272w, https://substackcdn.com/image/fetch/$s_!dVaM!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a993926-2092-41dc-a48f-107672a4c475_3056x1886.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!dVaM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a993926-2092-41dc-a48f-107672a4c475_3056x1886.png" width="1456" height="899" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0a993926-2092-41dc-a48f-107672a4c475_3056x1886.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:899,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:727061,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a993926-2092-41dc-a48f-107672a4c475_3056x1886.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!dVaM!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a993926-2092-41dc-a48f-107672a4c475_3056x1886.png 424w, https://substackcdn.com/image/fetch/$s_!dVaM!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a993926-2092-41dc-a48f-107672a4c475_3056x1886.png 848w, https://substackcdn.com/image/fetch/$s_!dVaM!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a993926-2092-41dc-a48f-107672a4c475_3056x1886.png 1272w, https://substackcdn.com/image/fetch/$s_!dVaM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a993926-2092-41dc-a48f-107672a4c475_3056x1886.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Along with the finding, the scan built an Attack Surface Map of <code>shop-orders</code>, showing the 37 places in the code that matter for security, like routes, access checks, and database queries. All 37 were verified, and the one with the finding was marked.</p><h4>Fixing what the scan found</h4><p>To fix it, we clicked Fix on the finding, and CodeRabbit&#8217;s <strong><a href="https://docs.coderabbit.ai/security#fix-with-ai">Fix with AI</a></strong> opened a pull request that put the ownership check back, this time inside the database query, so the service never loads another user&#8217;s order.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!YEiF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ba92a62-f094-4100-9ac7-5f9b378a3b8d_2090x2026.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!YEiF!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ba92a62-f094-4100-9ac7-5f9b378a3b8d_2090x2026.png 424w, https://substackcdn.com/image/fetch/$s_!YEiF!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ba92a62-f094-4100-9ac7-5f9b378a3b8d_2090x2026.png 848w, https://substackcdn.com/image/fetch/$s_!YEiF!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ba92a62-f094-4100-9ac7-5f9b378a3b8d_2090x2026.png 1272w, https://substackcdn.com/image/fetch/$s_!YEiF!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ba92a62-f094-4100-9ac7-5f9b378a3b8d_2090x2026.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!YEiF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ba92a62-f094-4100-9ac7-5f9b378a3b8d_2090x2026.png" width="1456" height="1411" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4ba92a62-f094-4100-9ac7-5f9b378a3b8d_2090x2026.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1411,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:703258,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ba92a62-f094-4100-9ac7-5f9b378a3b8d_2090x2026.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!YEiF!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ba92a62-f094-4100-9ac7-5f9b378a3b8d_2090x2026.png 424w, https://substackcdn.com/image/fetch/$s_!YEiF!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ba92a62-f094-4100-9ac7-5f9b378a3b8d_2090x2026.png 848w, https://substackcdn.com/image/fetch/$s_!YEiF!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ba92a62-f094-4100-9ac7-5f9b378a3b8d_2090x2026.png 1272w, https://substackcdn.com/image/fetch/$s_!YEiF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ba92a62-f094-4100-9ac7-5f9b378a3b8d_2090x2026.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The pull request also added a test where another user&#8217;s order returns the same 404 as a missing one, since a different error would tell an attacker which order IDs exist.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!OKyj!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedae278e-6fbf-4cfa-a1e7-106110795a89_2222x1538.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!OKyj!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedae278e-6fbf-4cfa-a1e7-106110795a89_2222x1538.png 424w, https://substackcdn.com/image/fetch/$s_!OKyj!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedae278e-6fbf-4cfa-a1e7-106110795a89_2222x1538.png 848w, https://substackcdn.com/image/fetch/$s_!OKyj!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedae278e-6fbf-4cfa-a1e7-106110795a89_2222x1538.png 1272w, https://substackcdn.com/image/fetch/$s_!OKyj!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedae278e-6fbf-4cfa-a1e7-106110795a89_2222x1538.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!OKyj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedae278e-6fbf-4cfa-a1e7-106110795a89_2222x1538.png" width="1456" height="1008" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/edae278e-6fbf-4cfa-a1e7-106110795a89_2222x1538.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1008,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:607133,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedae278e-6fbf-4cfa-a1e7-106110795a89_2222x1538.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!OKyj!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedae278e-6fbf-4cfa-a1e7-106110795a89_2222x1538.png 424w, https://substackcdn.com/image/fetch/$s_!OKyj!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedae278e-6fbf-4cfa-a1e7-106110795a89_2222x1538.png 848w, https://substackcdn.com/image/fetch/$s_!OKyj!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedae278e-6fbf-4cfa-a1e7-106110795a89_2222x1538.png 1272w, https://substackcdn.com/image/fetch/$s_!OKyj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedae278e-6fbf-4cfa-a1e7-106110795a89_2222x1538.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>We reviewed the diff, ran the tests locally, and merged. Bob&#8217;s request for order 1 now returns <code>404 Not Found</code>.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!LowV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe51adc31-1496-4bb7-a376-d8a3f36bf2f3_1566x834.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!LowV!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe51adc31-1496-4bb7-a376-d8a3f36bf2f3_1566x834.png 424w, https://substackcdn.com/image/fetch/$s_!LowV!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe51adc31-1496-4bb7-a376-d8a3f36bf2f3_1566x834.png 848w, https://substackcdn.com/image/fetch/$s_!LowV!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe51adc31-1496-4bb7-a376-d8a3f36bf2f3_1566x834.png 1272w, https://substackcdn.com/image/fetch/$s_!LowV!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe51adc31-1496-4bb7-a376-d8a3f36bf2f3_1566x834.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!LowV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe51adc31-1496-4bb7-a376-d8a3f36bf2f3_1566x834.png" width="1456" height="775" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e51adc31-1496-4bb7-a376-d8a3f36bf2f3_1566x834.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:775,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:708766,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe51adc31-1496-4bb7-a376-d8a3f36bf2f3_1566x834.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!LowV!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe51adc31-1496-4bb7-a376-d8a3f36bf2f3_1566x834.png 424w, https://substackcdn.com/image/fetch/$s_!LowV!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe51adc31-1496-4bb7-a376-d8a3f36bf2f3_1566x834.png 848w, https://substackcdn.com/image/fetch/$s_!LowV!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe51adc31-1496-4bb7-a376-d8a3f36bf2f3_1566x834.png 1272w, https://substackcdn.com/image/fetch/$s_!LowV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe51adc31-1496-4bb7-a376-d8a3f36bf2f3_1566x834.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>CodeRabbit then marked the finding Fixed and linked it to the merged pull request.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!9bIz!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f30741b-44d4-47b0-b28f-6d14bdf83725_1804x808.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!9bIz!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f30741b-44d4-47b0-b28f-6d14bdf83725_1804x808.png 424w, https://substackcdn.com/image/fetch/$s_!9bIz!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f30741b-44d4-47b0-b28f-6d14bdf83725_1804x808.png 848w, https://substackcdn.com/image/fetch/$s_!9bIz!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f30741b-44d4-47b0-b28f-6d14bdf83725_1804x808.png 1272w, https://substackcdn.com/image/fetch/$s_!9bIz!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f30741b-44d4-47b0-b28f-6d14bdf83725_1804x808.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!9bIz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f30741b-44d4-47b0-b28f-6d14bdf83725_1804x808.png" width="1456" height="652" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4f30741b-44d4-47b0-b28f-6d14bdf83725_1804x808.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:652,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:270761,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f30741b-44d4-47b0-b28f-6d14bdf83725_1804x808.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!9bIz!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f30741b-44d4-47b0-b28f-6d14bdf83725_1804x808.png 424w, https://substackcdn.com/image/fetch/$s_!9bIz!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f30741b-44d4-47b0-b28f-6d14bdf83725_1804x808.png 848w, https://substackcdn.com/image/fetch/$s_!9bIz!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f30741b-44d4-47b0-b28f-6d14bdf83725_1804x808.png 1272w, https://substackcdn.com/image/fetch/$s_!9bIz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f30741b-44d4-47b0-b28f-6d14bdf83725_1804x808.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h4>Catching the bug before merge</h4><p>Next, we asked Claude Code to add a reorder endpoint (<code>POST /orders/{order_id}/reorder</code>) that copies an order&#8217;s items into a new order for the current user. The codebase now had the fixed lookup to copy from, and Claude Code included the ownership check.</p><p>To see whether review would catch a missing check, we removed it and opened the pull request myself.</p><p>This time the PR scan caught it. CodeRabbit&#8217;s Secure flagged the lookup as a Major issue, the same kind of IDOR the Deep Scan had found, reachable from outside. </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!_BqV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e8141dc-1418-4231-8d30-3719560196e9_1640x1627.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_BqV!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e8141dc-1418-4231-8d30-3719560196e9_1640x1627.png 424w, https://substackcdn.com/image/fetch/$s_!_BqV!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e8141dc-1418-4231-8d30-3719560196e9_1640x1627.png 848w, https://substackcdn.com/image/fetch/$s_!_BqV!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e8141dc-1418-4231-8d30-3719560196e9_1640x1627.png 1272w, https://substackcdn.com/image/fetch/$s_!_BqV!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e8141dc-1418-4231-8d30-3719560196e9_1640x1627.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!_BqV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e8141dc-1418-4231-8d30-3719560196e9_1640x1627.png" width="1456" height="1444" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8e8141dc-1418-4231-8d30-3719560196e9_1640x1627.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1444,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:536857,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e8141dc-1418-4231-8d30-3719560196e9_1640x1627.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!_BqV!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e8141dc-1418-4231-8d30-3719560196e9_1640x1627.png 424w, https://substackcdn.com/image/fetch/$s_!_BqV!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e8141dc-1418-4231-8d30-3719560196e9_1640x1627.png 848w, https://substackcdn.com/image/fetch/$s_!_BqV!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e8141dc-1418-4231-8d30-3719560196e9_1640x1627.png 1272w, https://substackcdn.com/image/fetch/$s_!_BqV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e8141dc-1418-4231-8d30-3719560196e9_1640x1627.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Its comment names the gap between the two services. The gateway supplies the logged-in user&#8217;s ID, but nothing checks that the order being copied belongs to that user, so alice could reorder bob&#8217;s order and get its items.</p><p>CodeRabbit can also publish a Security check on each pull request, which we had turned on for <code>shop-orders</code>. With one blocking Security comment, the check failed. The merge button still worked at that point, because GitHub only blocks on a failing check when a rule requires it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!BcNE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe08523b2-631b-43de-876c-081faa9ccf02_1492x731.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!BcNE!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe08523b2-631b-43de-876c-081faa9ccf02_1492x731.png 424w, https://substackcdn.com/image/fetch/$s_!BcNE!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe08523b2-631b-43de-876c-081faa9ccf02_1492x731.png 848w, https://substackcdn.com/image/fetch/$s_!BcNE!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe08523b2-631b-43de-876c-081faa9ccf02_1492x731.png 1272w, https://substackcdn.com/image/fetch/$s_!BcNE!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe08523b2-631b-43de-876c-081faa9ccf02_1492x731.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!BcNE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe08523b2-631b-43de-876c-081faa9ccf02_1492x731.png" width="1456" height="713" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e08523b2-631b-43de-876c-081faa9ccf02_1492x731.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:713,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:286331,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe08523b2-631b-43de-876c-081faa9ccf02_1492x731.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!BcNE!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe08523b2-631b-43de-876c-081faa9ccf02_1492x731.png 424w, https://substackcdn.com/image/fetch/$s_!BcNE!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe08523b2-631b-43de-876c-081faa9ccf02_1492x731.png 848w, https://substackcdn.com/image/fetch/$s_!BcNE!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe08523b2-631b-43de-876c-081faa9ccf02_1492x731.png 1272w, https://substackcdn.com/image/fetch/$s_!BcNE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe08523b2-631b-43de-876c-081faa9ccf02_1492x731.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>We added CodeRabbit Security as a required check in the branch protection rule for <code>main</code>, and GitHub disabled the merge button.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!rbfG!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4924c12-edac-4b60-a411-5f50c5aab9eb_1571x635.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!rbfG!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4924c12-edac-4b60-a411-5f50c5aab9eb_1571x635.png 424w, https://substackcdn.com/image/fetch/$s_!rbfG!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4924c12-edac-4b60-a411-5f50c5aab9eb_1571x635.png 848w, https://substackcdn.com/image/fetch/$s_!rbfG!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4924c12-edac-4b60-a411-5f50c5aab9eb_1571x635.png 1272w, https://substackcdn.com/image/fetch/$s_!rbfG!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4924c12-edac-4b60-a411-5f50c5aab9eb_1571x635.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!rbfG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4924c12-edac-4b60-a411-5f50c5aab9eb_1571x635.png" width="1456" height="589" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c4924c12-edac-4b60-a411-5f50c5aab9eb_1571x635.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:589,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:292763,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4924c12-edac-4b60-a411-5f50c5aab9eb_1571x635.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!rbfG!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4924c12-edac-4b60-a411-5f50c5aab9eb_1571x635.png 424w, https://substackcdn.com/image/fetch/$s_!rbfG!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4924c12-edac-4b60-a411-5f50c5aab9eb_1571x635.png 848w, https://substackcdn.com/image/fetch/$s_!rbfG!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4924c12-edac-4b60-a411-5f50c5aab9eb_1571x635.png 1272w, https://substackcdn.com/image/fetch/$s_!rbfG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4924c12-edac-4b60-a411-5f50c5aab9eb_1571x635.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>We asked Claude Code to restrict reorders to the user&#8217;s own orders. CodeRabbit reviewed the new commit, the check passed with no blocking Security comments, and we merged.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!3XDk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9393c7b-5fcb-4c26-aa04-7c01c39cd8ab_1638x569.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!3XDk!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9393c7b-5fcb-4c26-aa04-7c01c39cd8ab_1638x569.png 424w, https://substackcdn.com/image/fetch/$s_!3XDk!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9393c7b-5fcb-4c26-aa04-7c01c39cd8ab_1638x569.png 848w, https://substackcdn.com/image/fetch/$s_!3XDk!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9393c7b-5fcb-4c26-aa04-7c01c39cd8ab_1638x569.png 1272w, https://substackcdn.com/image/fetch/$s_!3XDk!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9393c7b-5fcb-4c26-aa04-7c01c39cd8ab_1638x569.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!3XDk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9393c7b-5fcb-4c26-aa04-7c01c39cd8ab_1638x569.png" width="1456" height="506" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f9393c7b-5fcb-4c26-aa04-7c01c39cd8ab_1638x569.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:506,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:264302,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9393c7b-5fcb-4c26-aa04-7c01c39cd8ab_1638x569.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!3XDk!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9393c7b-5fcb-4c26-aa04-7c01c39cd8ab_1638x569.png 424w, https://substackcdn.com/image/fetch/$s_!3XDk!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9393c7b-5fcb-4c26-aa04-7c01c39cd8ab_1638x569.png 848w, https://substackcdn.com/image/fetch/$s_!3XDk!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9393c7b-5fcb-4c26-aa04-7c01c39cd8ab_1638x569.png 1272w, https://substackcdn.com/image/fetch/$s_!3XDk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9393c7b-5fcb-4c26-aa04-7c01c39cd8ab_1638x569.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>After the fix, the pull request&#8217;s <strong><a href="https://docs.coderabbit.ai/security/blast-radius">Security Blast Radius</a></strong> in <strong><a href="https://www.coderabbit.ai/change-stack">Change Stack</a></strong> shows what the new endpoint touches, including the <code>Order</code> model it reads, the response schema it returns, and the test fixtures it relies on.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!whV0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1595f7e5-40ee-4264-b6e4-06852981b8d9_1386x906.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!whV0!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1595f7e5-40ee-4264-b6e4-06852981b8d9_1386x906.png 424w, https://substackcdn.com/image/fetch/$s_!whV0!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1595f7e5-40ee-4264-b6e4-06852981b8d9_1386x906.png 848w, https://substackcdn.com/image/fetch/$s_!whV0!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1595f7e5-40ee-4264-b6e4-06852981b8d9_1386x906.png 1272w, https://substackcdn.com/image/fetch/$s_!whV0!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1595f7e5-40ee-4264-b6e4-06852981b8d9_1386x906.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!whV0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1595f7e5-40ee-4264-b6e4-06852981b8d9_1386x906.png" width="1386" height="906" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1595f7e5-40ee-4264-b6e4-06852981b8d9_1386x906.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:906,&quot;width&quot;:1386,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:470880,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1595f7e5-40ee-4264-b6e4-06852981b8d9_1386x906.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!whV0!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1595f7e5-40ee-4264-b6e4-06852981b8d9_1386x906.png 424w, https://substackcdn.com/image/fetch/$s_!whV0!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1595f7e5-40ee-4264-b6e4-06852981b8d9_1386x906.png 848w, https://substackcdn.com/image/fetch/$s_!whV0!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1595f7e5-40ee-4264-b6e4-06852981b8d9_1386x906.png 1272w, https://substackcdn.com/image/fetch/$s_!whV0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1595f7e5-40ee-4264-b6e4-06852981b8d9_1386x906.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h4>Reviewing a gateway refactor</h4><p>The gateway&#8217;s forwarding code decides which user ID the orders service sees, which makes it the most sensitive code in this setup. We wanted to see how CodeRabbit handles a routine change to it.</p><p>We asked Claude Code to pull that code into a shared <code>forward()</code> function and pass tracing and language headers through to the orders service. Claude Code found those headers already passed through, and it kept stripping any <code>X-User-Id</code> a client sends.</p><p>CodeRabbit reviewed the pull request and flagged nothing. In Change Stack, the <strong><a href="https://docs.coderabbit.ai/security/architecture-review">security architecture review</a></strong> shows the before and after, with the forwarding code moved out of the route and into the new shared function.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!yxad!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F860e1b87-ed6f-4c23-8250-87e73c6f21ad_1470x920.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!yxad!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F860e1b87-ed6f-4c23-8250-87e73c6f21ad_1470x920.png 424w, https://substackcdn.com/image/fetch/$s_!yxad!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F860e1b87-ed6f-4c23-8250-87e73c6f21ad_1470x920.png 848w, https://substackcdn.com/image/fetch/$s_!yxad!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F860e1b87-ed6f-4c23-8250-87e73c6f21ad_1470x920.png 1272w, https://substackcdn.com/image/fetch/$s_!yxad!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F860e1b87-ed6f-4c23-8250-87e73c6f21ad_1470x920.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!yxad!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F860e1b87-ed6f-4c23-8250-87e73c6f21ad_1470x920.png" width="1456" height="911" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/860e1b87-ed6f-4c23-8250-87e73c6f21ad_1470x920.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:911,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:586499,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F860e1b87-ed6f-4c23-8250-87e73c6f21ad_1470x920.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!yxad!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F860e1b87-ed6f-4c23-8250-87e73c6f21ad_1470x920.png 424w, https://substackcdn.com/image/fetch/$s_!yxad!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F860e1b87-ed6f-4c23-8250-87e73c6f21ad_1470x920.png 848w, https://substackcdn.com/image/fetch/$s_!yxad!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F860e1b87-ed6f-4c23-8250-87e73c6f21ad_1470x920.png 1272w, https://substackcdn.com/image/fetch/$s_!yxad!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F860e1b87-ed6f-4c23-8250-87e73c6f21ad_1470x920.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>In the pull request, CodeRabbit rated both the merge risk and the security architecture risk as Minimal. Its overview of the change explains why. The same routes still require login and still reach the same orders service, and no new route is open to outside callers.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!GW7i!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe179b315-28d1-4e71-9183-c4cee9ddbc87_1550x938.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!GW7i!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe179b315-28d1-4e71-9183-c4cee9ddbc87_1550x938.png 424w, https://substackcdn.com/image/fetch/$s_!GW7i!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe179b315-28d1-4e71-9183-c4cee9ddbc87_1550x938.png 848w, https://substackcdn.com/image/fetch/$s_!GW7i!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe179b315-28d1-4e71-9183-c4cee9ddbc87_1550x938.png 1272w, https://substackcdn.com/image/fetch/$s_!GW7i!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe179b315-28d1-4e71-9183-c4cee9ddbc87_1550x938.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!GW7i!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe179b315-28d1-4e71-9183-c4cee9ddbc87_1550x938.png" width="1456" height="881" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e179b315-28d1-4e71-9183-c4cee9ddbc87_1550x938.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:881,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:310447,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe179b315-28d1-4e71-9183-c4cee9ddbc87_1550x938.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!GW7i!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe179b315-28d1-4e71-9183-c4cee9ddbc87_1550x938.png 424w, https://substackcdn.com/image/fetch/$s_!GW7i!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe179b315-28d1-4e71-9183-c4cee9ddbc87_1550x938.png 848w, https://substackcdn.com/image/fetch/$s_!GW7i!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe179b315-28d1-4e71-9183-c4cee9ddbc87_1550x938.png 1272w, https://substackcdn.com/image/fetch/$s_!GW7i!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe179b315-28d1-4e71-9183-c4cee9ddbc87_1550x938.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The gateway also still overwrites <code>X-User-Id</code> with the logged-in user&#8217;s ID. That overwrite is what this whole setup depends on, and the overview lists it as something the change keeps.</p><h4>A change that skipped review</h4><p>Not every change goes through a pull request, so we wanted to see what the map does when one doesn&#8217;t. </p><p>We pushed a small change straight to <code>main</code>, adding a one-line description to the function that returns a single order (<code>get_order</code>). </p><p>By the end of the test, the <strong><a href="https://docs.coderabbit.ai/security/attack-surface">Attack Surface Map</a></strong> had gone from 100% verified to 89.2%, with four points marked stale, meaning CodeRabbit no longer counts them as checked.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!EeaS!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d9893a3-a95f-4b08-ac06-7c47cfc885c0_1297x967.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!EeaS!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d9893a3-a95f-4b08-ac06-7c47cfc885c0_1297x967.png 424w, https://substackcdn.com/image/fetch/$s_!EeaS!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d9893a3-a95f-4b08-ac06-7c47cfc885c0_1297x967.png 848w, https://substackcdn.com/image/fetch/$s_!EeaS!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d9893a3-a95f-4b08-ac06-7c47cfc885c0_1297x967.png 1272w, https://substackcdn.com/image/fetch/$s_!EeaS!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d9893a3-a95f-4b08-ac06-7c47cfc885c0_1297x967.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!EeaS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d9893a3-a95f-4b08-ac06-7c47cfc885c0_1297x967.png" width="1297" height="967" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2d9893a3-a95f-4b08-ac06-7c47cfc885c0_1297x967.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:967,&quot;width&quot;:1297,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:531840,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d9893a3-a95f-4b08-ac06-7c47cfc885c0_1297x967.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!EeaS!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d9893a3-a95f-4b08-ac06-7c47cfc885c0_1297x967.png 424w, https://substackcdn.com/image/fetch/$s_!EeaS!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d9893a3-a95f-4b08-ac06-7c47cfc885c0_1297x967.png 848w, https://substackcdn.com/image/fetch/$s_!EeaS!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d9893a3-a95f-4b08-ac06-7c47cfc885c0_1297x967.png 1272w, https://substackcdn.com/image/fetch/$s_!EeaS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d9893a3-a95f-4b08-ac06-7c47cfc885c0_1297x967.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Two of the stale points are in <code>get_order</code>, the function this push touched, because no review covered that change. They stay stale until a reviewed change or a new scan checks them again, which shows the team exactly which code to look at next.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!mmZX!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98b546fd-9d68-459b-9506-68fe6e935bf7_1606x581.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mmZX!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98b546fd-9d68-459b-9506-68fe6e935bf7_1606x581.png 424w, https://substackcdn.com/image/fetch/$s_!mmZX!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98b546fd-9d68-459b-9506-68fe6e935bf7_1606x581.png 848w, https://substackcdn.com/image/fetch/$s_!mmZX!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98b546fd-9d68-459b-9506-68fe6e935bf7_1606x581.png 1272w, https://substackcdn.com/image/fetch/$s_!mmZX!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98b546fd-9d68-459b-9506-68fe6e935bf7_1606x581.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!mmZX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98b546fd-9d68-459b-9506-68fe6e935bf7_1606x581.png" width="1456" height="527" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/98b546fd-9d68-459b-9506-68fe6e935bf7_1606x581.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:527,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:453249,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223433700?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98b546fd-9d68-459b-9506-68fe6e935bf7_1606x581.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!mmZX!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98b546fd-9d68-459b-9506-68fe6e935bf7_1606x581.png 424w, https://substackcdn.com/image/fetch/$s_!mmZX!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98b546fd-9d68-459b-9506-68fe6e935bf7_1606x581.png 848w, https://substackcdn.com/image/fetch/$s_!mmZX!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98b546fd-9d68-459b-9506-68fe6e935bf7_1606x581.png 1272w, https://substackcdn.com/image/fetch/$s_!mmZX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98b546fd-9d68-459b-9506-68fe6e935bf7_1606x581.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Across the test, we tried four points where a change like this can get through, and CodeRabbit handled each one:</p><ul><li><p><strong>Code already in the repository:</strong> a Deep Scan found the missing check, traced it through both repositories, and Fix with AI opened the fix.</p></li><li><p><strong>A new pull request:</strong> the Security Agent flagged the same mistake, and the required Security check blocked the merge until it was fixed.</p></li><li><p><strong>A refactor of the gateway:</strong> CodeRabbit confirmed the gateway still sets the user ID itself and rated the change minimal risk.</p></li><li><p><strong>A change that skipped review:</strong> the Attack Surface Map marked the code it touched as stale.</p></li></ul><p>Start a free 14-day trial. No credit card, 2-click setup.</p><p><strong><a href="https://coderabbit.link/dailydoseofds-003">Ship code you can trust across every service &#8594;</a></strong></p><p>Thanks to CodeRabbit for partnering today!</p>]]></content:encoded></item><item><title><![CDATA[Moving from RAG Apps to RAG Systems Engineering]]></title><description><![CDATA[A hands-on path through retrieval, context, storage, and scale.]]></description><link>https://blog.dailydoseofds.com/p/moving-from-rag-apps-to-rag-systems</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/moving-from-rag-apps-to-rag-systems</guid><dc:creator><![CDATA[Avi Chawla]]></dc:creator><pubDate>Mon, 16 Nov 2026 00:46:08 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!-BPc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F22cdf107-2889-417c-a3b6-31b8fd77a1a0_5504x3072.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>We have added 4 new parts to our hands-on RAG Systems course:</p><ul><li><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-16-with-implementation/">Part 16: When the model decides what to retrieve &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-17-with-implementation/">Part 17: When the context window fills up &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-18-with-implementation/">Part 18: Do you need a vector database? &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-19-with-implementation/">Part 19: When the index no longer fits in memory &#8594;</a></strong></p></li></ul><p>Together, these parts cover the engineering around retrieval: how the model searches, how evidence survives a long task, how the index stays consistent, and how search scales beyond memory.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!-BPc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F22cdf107-2889-417c-a3b6-31b8fd77a1a0_5504x3072.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!-BPc!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F22cdf107-2889-417c-a3b6-31b8fd77a1a0_5504x3072.jpeg 424w, https://substackcdn.com/image/fetch/$s_!-BPc!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F22cdf107-2889-417c-a3b6-31b8fd77a1a0_5504x3072.jpeg 848w, https://substackcdn.com/image/fetch/$s_!-BPc!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F22cdf107-2889-417c-a3b6-31b8fd77a1a0_5504x3072.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!-BPc!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F22cdf107-2889-417c-a3b6-31b8fd77a1a0_5504x3072.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!-BPc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F22cdf107-2889-417c-a3b6-31b8fd77a1a0_5504x3072.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/22cdf107-2889-417c-a3b6-31b8fd77a1a0_5504x3072.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:353682,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225107006?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F22cdf107-2889-417c-a3b6-31b8fd77a1a0_5504x3072.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!-BPc!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F22cdf107-2889-417c-a3b6-31b8fd77a1a0_5504x3072.jpeg 424w, https://substackcdn.com/image/fetch/$s_!-BPc!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F22cdf107-2889-417c-a3b6-31b8fd77a1a0_5504x3072.jpeg 848w, https://substackcdn.com/image/fetch/$s_!-BPc!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F22cdf107-2889-417c-a3b6-31b8fd77a1a0_5504x3072.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!-BPc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F22cdf107-2889-417c-a3b6-31b8fd77a1a0_5504x3072.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Each part includes the complete implementation.</p><div><hr></div><h4>Why care?</h4><p>Knowing how to chunk, embed, and retrieve documents is a useful foundation.</p><p>Production RAG work soon expands into problems usually associated with search engineering, backend systems, ML infrastructure, and agent design.</p><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-16-with-implementation/">Part 16</a></strong> develops the first of those skills: retrieval policy design.</p><p>Instead of treating retrieval as one fixed top-k call, the model can choose between lexical search, semantic search, file inspection, and bounded reading.</p><p>It can reformulate the query after seeing evidence and stop when the required claims are supported.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!fExJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feaff6d85-c7ae-4d60-ae9c-6467681dca53_5504x3072.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!fExJ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feaff6d85-c7ae-4d60-ae9c-6467681dca53_5504x3072.jpeg 424w, https://substackcdn.com/image/fetch/$s_!fExJ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feaff6d85-c7ae-4d60-ae9c-6467681dca53_5504x3072.jpeg 848w, https://substackcdn.com/image/fetch/$s_!fExJ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feaff6d85-c7ae-4d60-ae9c-6467681dca53_5504x3072.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!fExJ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feaff6d85-c7ae-4d60-ae9c-6467681dca53_5504x3072.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!fExJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feaff6d85-c7ae-4d60-ae9c-6467681dca53_5504x3072.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/eaff6d85-c7ae-4d60-ae9c-6467681dca53_5504x3072.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:359857,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225107006?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feaff6d85-c7ae-4d60-ae9c-6467681dca53_5504x3072.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!fExJ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feaff6d85-c7ae-4d60-ae9c-6467681dca53_5504x3072.jpeg 424w, https://substackcdn.com/image/fetch/$s_!fExJ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feaff6d85-c7ae-4d60-ae9c-6467681dca53_5504x3072.jpeg 848w, https://substackcdn.com/image/fetch/$s_!fExJ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feaff6d85-c7ae-4d60-ae9c-6467681dca53_5504x3072.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!fExJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Feaff6d85-c7ae-4d60-ae9c-6467681dca53_5504x3072.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The implementation also covers the failure modes created by that freedom. Retrieval can repeat the same evidence, launch overlapping calls, stay committed to a rejected hypothesis, or consume the context budget without making progress.</p><p>Learning to trace and control those decisions is useful far beyond RAG. The same reasoning applies to agents that browse, investigate code, query databases, or work across several tools.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!w5XO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78120db7-3a4d-438b-a63a-48e97853e88c_5504x2244.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!w5XO!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78120db7-3a4d-438b-a63a-48e97853e88c_5504x2244.jpeg 424w, https://substackcdn.com/image/fetch/$s_!w5XO!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78120db7-3a4d-438b-a63a-48e97853e88c_5504x2244.jpeg 848w, https://substackcdn.com/image/fetch/$s_!w5XO!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78120db7-3a4d-438b-a63a-48e97853e88c_5504x2244.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!w5XO!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78120db7-3a4d-438b-a63a-48e97853e88c_5504x2244.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!w5XO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78120db7-3a4d-438b-a63a-48e97853e88c_5504x2244.jpeg" width="5504" height="2244" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/78120db7-3a4d-438b-a63a-48e97853e88c_5504x2244.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:2244,&quot;width&quot;:5504,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:713950,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225107006?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedfa63ef-9f13-457e-afb0-4bd32cc5bd4f_5504x3072.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!w5XO!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78120db7-3a4d-438b-a63a-48e97853e88c_5504x2244.jpeg 424w, https://substackcdn.com/image/fetch/$s_!w5XO!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78120db7-3a4d-438b-a63a-48e97853e88c_5504x2244.jpeg 848w, https://substackcdn.com/image/fetch/$s_!w5XO!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78120db7-3a4d-438b-a63a-48e97853e88c_5504x2244.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!w5XO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78120db7-3a4d-438b-a63a-48e97853e88c_5504x2244.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-17-with-implementation/">Part 17</a></strong> moves into context engineering.</p><p>A long-running retrieval loop needs rules for what remains in the active context, what gets cleared, what can be compacted, and what should live in an external record.</p><p>The chapter compares clearing tool results, summarising history, external notes, searchable session logs, and sub-agent isolation.</p><p>These are core design choices for agent memory. They determine whether a system can recover an earlier fact, preserve exact identifiers, and continue a long task without carrying every intermediate result in the prompt.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!3jvO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20bef523-6cd1-4450-8923-2e19110d8286_5504x3072.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!3jvO!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20bef523-6cd1-4450-8923-2e19110d8286_5504x3072.jpeg 424w, https://substackcdn.com/image/fetch/$s_!3jvO!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20bef523-6cd1-4450-8923-2e19110d8286_5504x3072.jpeg 848w, https://substackcdn.com/image/fetch/$s_!3jvO!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20bef523-6cd1-4450-8923-2e19110d8286_5504x3072.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!3jvO!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20bef523-6cd1-4450-8923-2e19110d8286_5504x3072.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!3jvO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20bef523-6cd1-4450-8923-2e19110d8286_5504x3072.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/20bef523-6cd1-4450-8923-2e19110d8286_5504x3072.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:337419,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225107006?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20bef523-6cd1-4450-8923-2e19110d8286_5504x3072.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!3jvO!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20bef523-6cd1-4450-8923-2e19110d8286_5504x3072.jpeg 424w, https://substackcdn.com/image/fetch/$s_!3jvO!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20bef523-6cd1-4450-8923-2e19110d8286_5504x3072.jpeg 848w, https://substackcdn.com/image/fetch/$s_!3jvO!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20bef523-6cd1-4450-8923-2e19110d8286_5504x3072.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!3jvO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20bef523-6cd1-4450-8923-2e19110d8286_5504x3072.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-18-with-implementation/">Part 18</a></strong> introduces the backend and distributed-systems side of RAG.</p><p>Choosing between pgvector and a dedicated vector database is only one part of that work. The harder questions concern ownership and consistency.</p><p>The implementation covers what happens when a document changes, gets deleted, or loses a permission. It builds direct dual writes, transactional outboxes, version guards, reconciliation jobs, and filtered retrieval.</p><p>Operating a vector store safely requires both the retrieval API and the update path that prevents stale or unauthorized evidence.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ERlk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff60d3d68-adcb-42fc-b7f3-f47600bcb201_5504x3072.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ERlk!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff60d3d68-adcb-42fc-b7f3-f47600bcb201_5504x3072.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ERlk!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff60d3d68-adcb-42fc-b7f3-f47600bcb201_5504x3072.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ERlk!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff60d3d68-adcb-42fc-b7f3-f47600bcb201_5504x3072.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ERlk!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff60d3d68-adcb-42fc-b7f3-f47600bcb201_5504x3072.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ERlk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff60d3d68-adcb-42fc-b7f3-f47600bcb201_5504x3072.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f60d3d68-adcb-42fc-b7f3-f47600bcb201_5504x3072.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:306030,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225107006?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff60d3d68-adcb-42fc-b7f3-f47600bcb201_5504x3072.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ERlk!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff60d3d68-adcb-42fc-b7f3-f47600bcb201_5504x3072.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ERlk!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff60d3d68-adcb-42fc-b7f3-f47600bcb201_5504x3072.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ERlk!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff60d3d68-adcb-42fc-b7f3-f47600bcb201_5504x3072.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ERlk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff60d3d68-adcb-42fc-b7f3-f47600bcb201_5504x3072.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-19-with-implementation/">Part 19</a></strong> moves into search infrastructure at ten million vectors.</p><p>It covers product quantization, scalar quantization, binary codes, RaBitQ, IVF, HNSW, caching, disk-backed vectors, and exact rescoring.</p><p>The useful skill here is learning to reason about recall, memory, latency, and index freshness together. A faster index is not useful if it misses the evidence the application needs or slows down as soon as its cache becomes too small.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!7J4u!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F038fb11b-774c-4f3c-a4e8-55c50cbfdd07_5504x3072.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!7J4u!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F038fb11b-774c-4f3c-a4e8-55c50cbfdd07_5504x3072.jpeg 424w, https://substackcdn.com/image/fetch/$s_!7J4u!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F038fb11b-774c-4f3c-a4e8-55c50cbfdd07_5504x3072.jpeg 848w, https://substackcdn.com/image/fetch/$s_!7J4u!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F038fb11b-774c-4f3c-a4e8-55c50cbfdd07_5504x3072.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!7J4u!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F038fb11b-774c-4f3c-a4e8-55c50cbfdd07_5504x3072.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!7J4u!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F038fb11b-774c-4f3c-a4e8-55c50cbfdd07_5504x3072.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/038fb11b-774c-4f3c-a4e8-55c50cbfdd07_5504x3072.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:353469,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225107006?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F038fb11b-774c-4f3c-a4e8-55c50cbfdd07_5504x3072.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!7J4u!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F038fb11b-774c-4f3c-a4e8-55c50cbfdd07_5504x3072.jpeg 424w, https://substackcdn.com/image/fetch/$s_!7J4u!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F038fb11b-774c-4f3c-a4e8-55c50cbfdd07_5504x3072.jpeg 848w, https://substackcdn.com/image/fetch/$s_!7J4u!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F038fb11b-774c-4f3c-a4e8-55c50cbfdd07_5504x3072.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!7J4u!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F038fb11b-774c-4f3c-a4e8-55c50cbfdd07_5504x3072.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Together, the four parts provide a practical way to locate failures across the complete retrieval path:</p><ul><li><p>A weak query or repeated search points to the retrieval policy.</p></li><li><p>Relevant candidates that never reach the answer point to context handling.</p></li><li><p>Old documents or incorrect permissions point to synchronization and filtering.</p></li><li><p>Falling recall or rising latency at scale points to the index, cache, or storage layout.</p></li></ul><p>Being able to trace a failure to the right layer gives you something concrete to demonstrate in a portfolio project or explain in a RAG architecture interview.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!H76-!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ff56104-1748-4db9-aa3a-89dbe1085985_5504x1277.jpeg 424w, https://substackcdn.com/image/fetch/$s_!H76-!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ff56104-1748-4db9-aa3a-89dbe1085985_5504x1277.jpeg 848w, https://substackcdn.com/image/fetch/$s_!H76-!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ff56104-1748-4db9-aa3a-89dbe1085985_5504x1277.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!H76-!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ff56104-1748-4db9-aa3a-89dbe1085985_5504x1277.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!H76-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ff56104-1748-4db9-aa3a-89dbe1085985_5504x1277.jpeg" width="5504" height="1277" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1ff56104-1748-4db9-aa3a-89dbe1085985_5504x1277.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1277,&quot;width&quot;:5504,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:451475,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/225107006?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6a7bcf6c-4e04-41e9-af60-b09154e144e5_5504x3072.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!H76-!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ff56104-1748-4db9-aa3a-89dbe1085985_5504x1277.jpeg 424w, https://substackcdn.com/image/fetch/$s_!H76-!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ff56104-1748-4db9-aa3a-89dbe1085985_5504x1277.jpeg 848w, https://substackcdn.com/image/fetch/$s_!H76-!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ff56104-1748-4db9-aa3a-89dbe1085985_5504x1277.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!H76-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ff56104-1748-4db9-aa3a-89dbe1085985_5504x1277.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>A developer who can explain where retrieval failed, which layer owns the fix, and what tradeoff the fix introduces demonstrates more than framework familiarity.</p><p>The progression across these parts supports a clear career path: from building RAG applications to engineering agentic retrieval, search systems, and the infrastructure that keeps them reliable.</p><ul><li><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-16-with-implementation/">Learn to design model-directed retrieval in Part 16 &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-17-with-implementation/">Learn to manage context across long agent runs in Part 17 &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-18-with-implementation/">Learn to operate vector storage safely in Part 18 &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-19-with-implementation/">Learn to scale vector search beyond memory in Part 19 &#8594;</a></strong></p></li></ul><p>&#128073; Over to you: Which layer of production RAG would strengthen your current skill set most?</p><p>Good day!</p>]]></content:encoded></item><item><title><![CDATA[How to Deploy Agent Teams Across Your Company (Complete Guide)]]></title><description><![CDATA[...explained visually, plus a four-agent Alook and Nimble walkthrough.]]></description><link>https://blog.dailydoseofds.com/p/how-to-deploy-agent-teams-across</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/how-to-deploy-agent-teams-across</guid><pubDate>Sat, 14 Nov 2026 00:00:46 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/d427c98d-fc2d-4dc4-8a32-221c768e0229_1983x793.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><strong>In today&#8217;s newsletter:</strong></p><ul><li><p>Skills 2.0 is here!</p></li><li><p>How to Deploy Agent Teams Across Your Company That Work Like Actual Employees (Complete Guide).</p></li></ul><div><hr></div><h3><strong><a href="https://fandf.co/4ipxEwg">Skills 2.0 is here!</a></strong></h3><p>We package repeated workflows as skills all the time, but the agent still re-reads the skill and decides every step on every run, which costs tokens and time.</p><p>Devs just open-sourced <strong><a href="https://fandf.co/4ipxEwg">Swamp</a></strong>, where your coding agent turns a repeated skill into a workflow that it builds once and runs again and again.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!qcIF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7732e370-3890-440f-b413-91b4b586487e_2528x1696.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!qcIF!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7732e370-3890-440f-b413-91b4b586487e_2528x1696.jpeg 424w, https://substackcdn.com/image/fetch/$s_!qcIF!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7732e370-3890-440f-b413-91b4b586487e_2528x1696.jpeg 848w, https://substackcdn.com/image/fetch/$s_!qcIF!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7732e370-3890-440f-b413-91b4b586487e_2528x1696.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!qcIF!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7732e370-3890-440f-b413-91b4b586487e_2528x1696.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!qcIF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7732e370-3890-440f-b413-91b4b586487e_2528x1696.jpeg" width="1456" height="977" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7732e370-3890-440f-b413-91b4b586487e_2528x1696.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:977,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:174498,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7732e370-3890-440f-b413-91b4b586487e_2528x1696.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!qcIF!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7732e370-3890-440f-b413-91b4b586487e_2528x1696.jpeg 424w, https://substackcdn.com/image/fetch/$s_!qcIF!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7732e370-3890-440f-b413-91b4b586487e_2528x1696.jpeg 848w, https://substackcdn.com/image/fetch/$s_!qcIF!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7732e370-3890-440f-b413-91b4b586487e_2528x1696.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!qcIF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7732e370-3890-440f-b413-91b4b586487e_2528x1696.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A skill re-plans every run, while Swamp builds a reusable workflow once and spends model tokens only on judgment.</figcaption></figure></div><p>You describe the task once, and the agent breaks it into reusable behaviors connected in a workflow.</p><p>Every behavior has typed inputs and outputs, and plain code decides which step runs next. Swamp then gives you:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!8pVh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a6272fe-9035-4345-93e5-f57265dc8017_960x960.gif" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8pVh!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a6272fe-9035-4345-93e5-f57265dc8017_960x960.gif 424w, https://substackcdn.com/image/fetch/$s_!8pVh!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a6272fe-9035-4345-93e5-f57265dc8017_960x960.gif 848w, https://substackcdn.com/image/fetch/$s_!8pVh!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a6272fe-9035-4345-93e5-f57265dc8017_960x960.gif 1272w, https://substackcdn.com/image/fetch/$s_!8pVh!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a6272fe-9035-4345-93e5-f57265dc8017_960x960.gif 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!8pVh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a6272fe-9035-4345-93e5-f57265dc8017_960x960.gif" width="960" height="960" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5a6272fe-9035-4345-93e5-f57265dc8017_960x960.gif&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:960,&quot;width&quot;:960,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:823695,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/gif&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a6272fe-9035-4345-93e5-f57265dc8017_960x960.gif&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!8pVh!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a6272fe-9035-4345-93e5-f57265dc8017_960x960.gif 424w, https://substackcdn.com/image/fetch/$s_!8pVh!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a6272fe-9035-4345-93e5-f57265dc8017_960x960.gif 848w, https://substackcdn.com/image/fetch/$s_!8pVh!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a6272fe-9035-4345-93e5-f57265dc8017_960x960.gif 1272w, https://substackcdn.com/image/fetch/$s_!8pVh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a6272fe-9035-4345-93e5-f57265dc8017_960x960.gif 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Skills re-plan the task on every run, while Skills plus Swamp lets code run a workflow the agent built once.</figcaption></figure></div><ul><li><p>model calls only on steps that need judgment</p></li><li><p>secrets pulled from vaults and redacted from output</p></li><li><p>versioned data from every run that you can query later</p></li><li><p>a full record of what happened on each run</p></li></ul><p>Swamp plugs into the coding agent you already use, like:</p><ul><li><p>Claude Code</p></li><li><p>Codex</p></li><li><p>Cursor</p></li><li><p>OpenCode</p></li><li><p>And 5 more agent harnesses</p></li></ul><p>In Swamp, the model is a step in the workflow, not the thing deciding what happens next.</p><p>The model steps still reason like any LLM call.</p><p>However, in this case, code owns the loop, so the agent doesn&#8217;t spend tokens rediscovering the same process every time.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!e-Xi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e07f317-b1b6-4cce-b398-304f68cfb06a_2528x1176.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!e-Xi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e07f317-b1b6-4cce-b398-304f68cfb06a_2528x1176.jpeg 424w, https://substackcdn.com/image/fetch/$s_!e-Xi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e07f317-b1b6-4cce-b398-304f68cfb06a_2528x1176.jpeg 848w, https://substackcdn.com/image/fetch/$s_!e-Xi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e07f317-b1b6-4cce-b398-304f68cfb06a_2528x1176.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!e-Xi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e07f317-b1b6-4cce-b398-304f68cfb06a_2528x1176.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!e-Xi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e07f317-b1b6-4cce-b398-304f68cfb06a_2528x1176.jpeg" width="1456" height="677" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1e07f317-b1b6-4cce-b398-304f68cfb06a_2528x1176.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:677,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:99349,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e07f317-b1b6-4cce-b398-304f68cfb06a_2528x1176.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!e-Xi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e07f317-b1b6-4cce-b398-304f68cfb06a_2528x1176.jpeg 424w, https://substackcdn.com/image/fetch/$s_!e-Xi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e07f317-b1b6-4cce-b398-304f68cfb06a_2528x1176.jpeg 848w, https://substackcdn.com/image/fetch/$s_!e-Xi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e07f317-b1b6-4cce-b398-304f68cfb06a_2528x1176.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!e-Xi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e07f317-b1b6-4cce-b398-304f68cfb06a_2528x1176.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Code owns the loop and calls the model only on the steps that need judgment.</figcaption></figure></div><p>Here&#8217;s an official result, where they turned a code-review skill into a Swamp workflow:</p><ul><li><p>Skill in Codex: about 4.6M tokens, 23 agents, 12.8 minutes</p></li><li><p>Swamp workflow: about 506K tokens, 3 agents, 6.6 minutes</p></li></ul><p>That is ~8x fewer tokens in about half the time, and most of the savings came from replacing the coordinator agent with plain code.</p><p>You can keep using skills while you prototype. Once a task starts repeating, ask your agent to turn it into a Swamp workflow.</p><p>This helps answer practical questions such as:</p><ul><li><p>Which steps actually need a model?</p></li><li><p>What exactly happened on last week&#8217;s run?</p></li><li><p>Can a cheaper model handle the one step that needs judgment?</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!IoyO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40df8c28-ad04-4aae-aafc-74bd787106b3_2528x1147.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!IoyO!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40df8c28-ad04-4aae-aafc-74bd787106b3_2528x1147.jpeg 424w, https://substackcdn.com/image/fetch/$s_!IoyO!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40df8c28-ad04-4aae-aafc-74bd787106b3_2528x1147.jpeg 848w, https://substackcdn.com/image/fetch/$s_!IoyO!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40df8c28-ad04-4aae-aafc-74bd787106b3_2528x1147.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!IoyO!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40df8c28-ad04-4aae-aafc-74bd787106b3_2528x1147.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!IoyO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40df8c28-ad04-4aae-aafc-74bd787106b3_2528x1147.jpeg" width="1456" height="661" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/40df8c28-ad04-4aae-aafc-74bd787106b3_2528x1147.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:661,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:116042,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40df8c28-ad04-4aae-aafc-74bd787106b3_2528x1147.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!IoyO!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40df8c28-ad04-4aae-aafc-74bd787106b3_2528x1147.jpeg 424w, https://substackcdn.com/image/fetch/$s_!IoyO!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40df8c28-ad04-4aae-aafc-74bd787106b3_2528x1147.jpeg 848w, https://substackcdn.com/image/fetch/$s_!IoyO!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40df8c28-ad04-4aae-aafc-74bd787106b3_2528x1147.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!IoyO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40df8c28-ad04-4aae-aafc-74bd787106b3_2528x1147.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Each run loads secrets from a vault and records versioned data, queryable history, and an execution record.</figcaption></figure></div><p>Once it&#8217;s built, you can run the workflow from your agent, the CLI, a schedule, or an API, with every run recorded.</p><p>Instead, you spend tokens only on the steps where the model&#8217;s judgment adds something.</p><p><strong><a href="https://fandf.co/4ipxEwg">You can find Swamp Club here &#8594;</a></strong></p><p><em>Thanks to Swamp Club for partnering today!</em></p><div><hr></div><h3><strong><a href="https://github.com/alookai/alook">How to Deploy Agent Teams Across Your Company (Complete Guide)</a></strong></h3><p>Most coding agents today work for one person. Claude Code or Codex runs in a terminal on your laptop, and only you can talk to it.</p><p>That is fine for solo work, but it does not work once a team depends on the output.</p><p>When a teammate wants the agent&#8217;s latest findings or has a follow-up question, they have to ask you.</p><p>You paste their question into your terminal, then paste the answer back to them, so you end up as the relay between your team and your agents.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!f3nh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9bcd3fa-797b-4382-b58a-55297315a91b_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!f3nh!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9bcd3fa-797b-4382-b58a-55297315a91b_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!f3nh!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9bcd3fa-797b-4382-b58a-55297315a91b_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!f3nh!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9bcd3fa-797b-4382-b58a-55297315a91b_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!f3nh!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9bcd3fa-797b-4382-b58a-55297315a91b_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!f3nh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9bcd3fa-797b-4382-b58a-55297315a91b_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c9bcd3fa-797b-4382-b58a-55297315a91b_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:152423,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9bcd3fa-797b-4382-b58a-55297315a91b_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!f3nh!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9bcd3fa-797b-4382-b58a-55297315a91b_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!f3nh!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9bcd3fa-797b-4382-b58a-55297315a91b_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!f3nh!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9bcd3fa-797b-4382-b58a-55297315a91b_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!f3nh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9bcd3fa-797b-4382-b58a-55297315a91b_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Single-player agent where you relay every teammate question, versus an agent team in a shared room with direct messages and handoffs.</figcaption></figure></div><p>An agent team removes that relay. Each agent gets a role, a handle, and an inbox inside a shared room, much like a team chat channel.</p><p>Anyone in the room can message an agent directly, and agents can hand off work to each other without you in the middle.</p><p>In this article, we will build a four-agent team with Alook and give one agent Nimble&#8217;s web skills. The example job is tracking a competitor&#8217;s pricing page and reporting what changed.</p><p>The first run saved a baseline of Railway&#8217;s pricing page on 9 September 2026. A second run on 23 October compared the live pages with that snapshot and posted what had changed, as shown below.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!x8Eg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37af7e67-c665-4f92-a840-3465496a3a0c_1456x801.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!x8Eg!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37af7e67-c665-4f92-a840-3465496a3a0c_1456x801.png 424w, https://substackcdn.com/image/fetch/$s_!x8Eg!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37af7e67-c665-4f92-a840-3465496a3a0c_1456x801.png 848w, https://substackcdn.com/image/fetch/$s_!x8Eg!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37af7e67-c665-4f92-a840-3465496a3a0c_1456x801.png 1272w, https://substackcdn.com/image/fetch/$s_!x8Eg!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37af7e67-c665-4f92-a840-3465496a3a0c_1456x801.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!x8Eg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37af7e67-c665-4f92-a840-3465496a3a0c_1456x801.png" width="1456" height="801" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/37af7e67-c665-4f92-a840-3465496a3a0c_1456x801.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:801,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:183440,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37af7e67-c665-4f92-a840-3465496a3a0c_1456x801.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!x8Eg!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37af7e67-c665-4f92-a840-3465496a3a0c_1456x801.png 424w, https://substackcdn.com/image/fetch/$s_!x8Eg!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37af7e67-c665-4f92-a840-3465496a3a0c_1456x801.png 848w, https://substackcdn.com/image/fetch/$s_!x8Eg!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37af7e67-c665-4f92-a840-3465496a3a0c_1456x801.png 1272w, https://substackcdn.com/image/fetch/$s_!x8Eg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37af7e67-c665-4f92-a840-3465496a3a0c_1456x801.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The 9 September baseline snapshot cites Railway&#8217;s pricing and plans pages, and records the headline &#8220;Pay only for what your app uses, by the second.&#8221;</figcaption></figure></div><p>The snapshot came from Railway&#8217;s pricing page and its plans documentation. The pricing headline read &#8220;Pay only for what your app uses, by the second&#8221;. The plans were Free ($0, then $1/month), Hobby ($5 minimum), Pro ($20 minimum), and Enterprise (custom).</p><p>This baseline came from Nimble&#8217;s competitor-positioning skill. To run the same skill on a site you care about:</p><p><strong><a href="https://github.com/Nimbleway/agent-skills">Get the Nimble agent skills &#8594;</a></strong></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!zoVi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb323279e-8f54-4c13-9371-c24ada7ab35f_881x594.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zoVi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb323279e-8f54-4c13-9371-c24ada7ab35f_881x594.png 424w, https://substackcdn.com/image/fetch/$s_!zoVi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb323279e-8f54-4c13-9371-c24ada7ab35f_881x594.png 848w, https://substackcdn.com/image/fetch/$s_!zoVi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb323279e-8f54-4c13-9371-c24ada7ab35f_881x594.png 1272w, https://substackcdn.com/image/fetch/$s_!zoVi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb323279e-8f54-4c13-9371-c24ada7ab35f_881x594.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!zoVi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb323279e-8f54-4c13-9371-c24ada7ab35f_881x594.png" width="881" height="594" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b323279e-8f54-4c13-9371-c24ada7ab35f_881x594.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:594,&quot;width&quot;:881,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:37317,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb323279e-8f54-4c13-9371-c24ada7ab35f_881x594.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!zoVi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb323279e-8f54-4c13-9371-c24ada7ab35f_881x594.png 424w, https://substackcdn.com/image/fetch/$s_!zoVi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb323279e-8f54-4c13-9371-c24ada7ab35f_881x594.png 848w, https://substackcdn.com/image/fetch/$s_!zoVi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb323279e-8f54-4c13-9371-c24ada7ab35f_881x594.png 1272w, https://substackcdn.com/image/fetch/$s_!zoVi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb323279e-8f54-4c13-9371-c24ada7ab35f_881x594.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The Nimble agent skills README, licensed under MIT, with a link to the Agent Skills spec.</figcaption></figure></div><div><hr></div><h4><strong>Shared agent rooms and live web data</strong></h4><p>Nimble handles the web research, and Alook gives your teammates a way to work with the agents doing that research, as shown below.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!1RxP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F273b283b-e18f-4cd4-a355-a43008134535_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!1RxP!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F273b283b-e18f-4cd4-a355-a43008134535_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!1RxP!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F273b283b-e18f-4cd4-a355-a43008134535_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!1RxP!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F273b283b-e18f-4cd4-a355-a43008134535_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!1RxP!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F273b283b-e18f-4cd4-a355-a43008134535_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!1RxP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F273b283b-e18f-4cd4-a355-a43008134535_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/273b283b-e18f-4cd4-a355-a43008134535_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:185174,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F273b283b-e18f-4cd4-a355-a43008134535_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!1RxP!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F273b283b-e18f-4cd4-a355-a43008134535_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!1RxP!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F273b283b-e18f-4cd4-a355-a43008134535_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!1RxP!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F273b283b-e18f-4cd4-a355-a43008134535_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!1RxP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F273b283b-e18f-4cd4-a355-a43008134535_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Source pages feed Nimble&#8217;s competitor-positioning skill, which saves snapshots and sends evidence to the Alook room.</figcaption></figure></div><p>Nimble is web data infrastructure for AI agents. Its agent skills are open source under the MIT license. They let coding agents search the web, map sites, scrape pages, and extract structured data.</p><p>We will use the competitor-positioning skill to map a competitor&#8217;s site, collect its pages, and save a dated snapshot with source references.</p><p>When we run it again, the skill can compare the new evidence with the saved snapshot. Snapshot history is a behavior of this skill, not of every Nimble skill.</p><p>You can install the skills with one command and use them with Claude Code, Codex, and other agents that support the Agent Skills specification.</p><p><strong><a href="https://github.com/alookai/alook">Alook</a></strong> is an open-source platform for sharing agents with your team. It is licensed under Apache-2.0.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ERDn!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F906863d7-3968-40a4-a6d5-be4040989c19_874x698.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ERDn!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F906863d7-3968-40a4-a6d5-be4040989c19_874x698.png 424w, https://substackcdn.com/image/fetch/$s_!ERDn!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F906863d7-3968-40a4-a6d5-be4040989c19_874x698.png 848w, https://substackcdn.com/image/fetch/$s_!ERDn!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F906863d7-3968-40a4-a6d5-be4040989c19_874x698.png 1272w, https://substackcdn.com/image/fetch/$s_!ERDn!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F906863d7-3968-40a4-a6d5-be4040989c19_874x698.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ERDn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F906863d7-3968-40a4-a6d5-be4040989c19_874x698.png" width="874" height="698" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/906863d7-3968-40a4-a6d5-be4040989c19_874x698.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:698,&quot;width&quot;:874,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:52786,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F906863d7-3968-40a4-a6d5-be4040989c19_874x698.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ERDn!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F906863d7-3968-40a4-a6d5-be4040989c19_874x698.png 424w, https://substackcdn.com/image/fetch/$s_!ERDn!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F906863d7-3968-40a4-a6d5-be4040989c19_874x698.png 848w, https://substackcdn.com/image/fetch/$s_!ERDn!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F906863d7-3968-40a4-a6d5-be4040989c19_874x698.png 1272w, https://substackcdn.com/image/fetch/$s_!ERDn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F906863d7-3968-40a4-a6d5-be4040989c19_874x698.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The Alook GitHub README describes shared rooms for people and local coding agents, under the Apache-2.0 license.</figcaption></figure></div><p>The agents keep running on your machine with their existing tools. Alook gives each one a handle and an inbox, so teammates can message them in shared rooms, check updates, and work with them without going through you.</p><p>The rooms in this walkthrough are Alook rooms, not a Slack integration.</p><p>We will set up a four-agent Alook server, give one agent Nimble&#8217;s skills, and run the competitor-monitoring workflow to see how the team handles a real job.</p><div><hr></div><h4><strong>Installing Alook and connecting your local agents</strong></h4><p>You will need Node.js 20.9 or later with npm, Claude Code with its default model, and a Nimble account.</p><p>Alook can be installed and connected with one command:</p><pre><code><code>npx @alook/app onboard</code></code></pre><p>The wizard connects your machine, detects installed agent runtimes such as Claude Code and Codex, and starts the local daemon.</p><p>Once it is running, the local dashboard opens at localhost port 15210 as shown below.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!NXt-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc854708e-7327-40fb-be9b-5ee5a3882e45_1769x889.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!NXt-!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc854708e-7327-40fb-be9b-5ee5a3882e45_1769x889.png 424w, https://substackcdn.com/image/fetch/$s_!NXt-!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc854708e-7327-40fb-be9b-5ee5a3882e45_1769x889.png 848w, https://substackcdn.com/image/fetch/$s_!NXt-!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc854708e-7327-40fb-be9b-5ee5a3882e45_1769x889.png 1272w, https://substackcdn.com/image/fetch/$s_!NXt-!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc854708e-7327-40fb-be9b-5ee5a3882e45_1769x889.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!NXt-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc854708e-7327-40fb-be9b-5ee5a3882e45_1769x889.png" width="1456" height="732" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c854708e-7327-40fb-be9b-5ee5a3882e45_1769x889.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:732,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:428120,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc854708e-7327-40fb-be9b-5ee5a3882e45_1769x889.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!NXt-!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc854708e-7327-40fb-be9b-5ee5a3882e45_1769x889.png 424w, https://substackcdn.com/image/fetch/$s_!NXt-!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc854708e-7327-40fb-be9b-5ee5a3882e45_1769x889.png 848w, https://substackcdn.com/image/fetch/$s_!NXt-!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc854708e-7327-40fb-be9b-5ee5a3882e45_1769x889.png 1272w, https://substackcdn.com/image/fetch/$s_!NXt-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc854708e-7327-40fb-be9b-5ee5a3882e45_1769x889.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Alook local dashboard after onboarding, with the daemon connected and agent runtimes detected.</figcaption></figure></div><h4><strong>Agents run locally while the Alook app routes messages</strong></h4><p>Alook splits the work between a local daemon and the app.</p><ol><li><p>The <code>@ alook/daemon</code> runs on your machine, holds the agent workdir, and connects to the app over a WebSocket.</p></li><li><p>The hosted <code>@ alook/app</code> manages queues, the database, and file storage. It runs on the cloud at alook(.)ai or on your own machine in self-hosted mode.</p></li></ol><p>The agents and their working directories stay on your machine. If you use the cloud, the host stores room messages and files shared in rooms. Model calls go to your model provider, while web requests go through Nimble&#8217;s API.</p><p>Each agent runs as a coding-agent session that stays on your machine. It keeps the same identity across rooms and can use its existing tools, as shown below.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!rTMH!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe65b3274-7217-4266-939a-111fd8fdbdc7_1445x290.png 424w, https://substackcdn.com/image/fetch/$s_!rTMH!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe65b3274-7217-4266-939a-111fd8fdbdc7_1445x290.png 848w, https://substackcdn.com/image/fetch/$s_!rTMH!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe65b3274-7217-4266-939a-111fd8fdbdc7_1445x290.png 1272w, https://substackcdn.com/image/fetch/$s_!rTMH!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe65b3274-7217-4266-939a-111fd8fdbdc7_1445x290.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!rTMH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe65b3274-7217-4266-939a-111fd8fdbdc7_1445x290.png" width="1445" height="290" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e65b3274-7217-4266-939a-111fd8fdbdc7_1445x290.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:290,&quot;width&quot;:1445,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:57804,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe65b3274-7217-4266-939a-111fd8fdbdc7_1445x290.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!rTMH!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe65b3274-7217-4266-939a-111fd8fdbdc7_1445x290.png 424w, https://substackcdn.com/image/fetch/$s_!rTMH!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe65b3274-7217-4266-939a-111fd8fdbdc7_1445x290.png 848w, https://substackcdn.com/image/fetch/$s_!rTMH!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe65b3274-7217-4266-939a-111fd8fdbdc7_1445x290.png 1272w, https://substackcdn.com/image/fetch/$s_!rTMH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe65b3274-7217-4266-939a-111fd8fdbdc7_1445x290.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Alook detected Claude Code as the tested runtime on the local machine, with the other listed runtimes not installed.</figcaption></figure></div><p>Alook supports Claude Code, Codex, Cursor, OpenCode, and Pi. This walkthrough uses Claude Code for all four agents.</p><p>The next decision is which agents go in each room, and who should have access.</p><h4><strong>Splitting one job across four agent roles</strong></h4><p>Competitor tracking often requires checking a pricing page, copying the numbers into a spreadsheet, and doing it all again repeatedly.</p><p>We will split the work across four agents: one handles requests, one writes the brief, one researches the competitor, and one posts alerts when something changes. A single agent could do all four jobs, but splitting them up lets us see how work passes between agents.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!0k_F!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe56a8e60-fa53-4e88-8669-2470414fa9da_966x1629.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!0k_F!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe56a8e60-fa53-4e88-8669-2470414fa9da_966x1629.png 424w, https://substackcdn.com/image/fetch/$s_!0k_F!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe56a8e60-fa53-4e88-8669-2470414fa9da_966x1629.png 848w, https://substackcdn.com/image/fetch/$s_!0k_F!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe56a8e60-fa53-4e88-8669-2470414fa9da_966x1629.png 1272w, https://substackcdn.com/image/fetch/$s_!0k_F!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe56a8e60-fa53-4e88-8669-2470414fa9da_966x1629.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!0k_F!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe56a8e60-fa53-4e88-8669-2470414fa9da_966x1629.png" width="966" height="1629" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e56a8e60-fa53-4e88-8669-2470414fa9da_966x1629.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1629,&quot;width&quot;:966,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:373647,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe56a8e60-fa53-4e88-8669-2470414fa9da_966x1629.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!0k_F!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe56a8e60-fa53-4e88-8669-2470414fa9da_966x1629.png 424w, https://substackcdn.com/image/fetch/$s_!0k_F!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe56a8e60-fa53-4e88-8669-2470414fa9da_966x1629.png 848w, https://substackcdn.com/image/fetch/$s_!0k_F!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe56a8e60-fa53-4e88-8669-2470414fa9da_966x1629.png 1272w, https://substackcdn.com/image/fetch/$s_!0k_F!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe56a8e60-fa53-4e88-8669-2470414fa9da_966x1629.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Creating the Atlas bot in Alook, with the CEO role and the Claude Code runtime selected.</figcaption></figure></div><p>The first step is to create four agents and assign each a role and handle, as shown below.</p><ol><li><p>Atlas (CEO) is the agent you talk to. It tracks what is outstanding instead of asking you to hold it.</p></li><li><p>Mara (PM) turns a request into a spec that Theo or Ren can act on.</p></li><li><p>Theo (engineer) runs and maintains the competitive intelligence pipeline.</p></li><li><p>Ren (operations) posts in <code>#alerts</code> when a tracked change appears.</p></li></ol><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!4yLo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d70dbe5-ac28-4ef8-9319-b7ad1dc98779_1993x789.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!4yLo!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d70dbe5-ac28-4ef8-9319-b7ad1dc98779_1993x789.png 424w, https://substackcdn.com/image/fetch/$s_!4yLo!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d70dbe5-ac28-4ef8-9319-b7ad1dc98779_1993x789.png 848w, https://substackcdn.com/image/fetch/$s_!4yLo!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d70dbe5-ac28-4ef8-9319-b7ad1dc98779_1993x789.png 1272w, https://substackcdn.com/image/fetch/$s_!4yLo!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d70dbe5-ac28-4ef8-9319-b7ad1dc98779_1993x789.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!4yLo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d70dbe5-ac28-4ef8-9319-b7ad1dc98779_1993x789.png" width="1456" height="576" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0d70dbe5-ac28-4ef8-9319-b7ad1dc98779_1993x789.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:576,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:303239,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d70dbe5-ac28-4ef8-9319-b7ad1dc98779_1993x789.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!4yLo!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d70dbe5-ac28-4ef8-9319-b7ad1dc98779_1993x789.png 424w, https://substackcdn.com/image/fetch/$s_!4yLo!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d70dbe5-ac28-4ef8-9319-b7ad1dc98779_1993x789.png 848w, https://substackcdn.com/image/fetch/$s_!4yLo!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d70dbe5-ac28-4ef8-9319-b7ad1dc98779_1993x789.png 1272w, https://substackcdn.com/image/fetch/$s_!4yLo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0d70dbe5-ac28-4ef8-9319-b7ad1dc98779_1993x789.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The four agents with their roles: Atlas as CEO, Mara as PM, Theo as engineer, and Ren as operations.</figcaption></figure></div><p>The next step is to create two Alook channels.</p><ul><li><p><code># intel</code> is public. It holds Atlas, Mara, Theo, Ren, and us.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xnaz!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d70eda1-d9c8-42d5-a0f0-b7bb565be255_1886x834.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xnaz!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d70eda1-d9c8-42d5-a0f0-b7bb565be255_1886x834.png 424w, https://substackcdn.com/image/fetch/$s_!xnaz!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d70eda1-d9c8-42d5-a0f0-b7bb565be255_1886x834.png 848w, https://substackcdn.com/image/fetch/$s_!xnaz!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d70eda1-d9c8-42d5-a0f0-b7bb565be255_1886x834.png 1272w, https://substackcdn.com/image/fetch/$s_!xnaz!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d70eda1-d9c8-42d5-a0f0-b7bb565be255_1886x834.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!xnaz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d70eda1-d9c8-42d5-a0f0-b7bb565be255_1886x834.png" width="1456" height="644" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4d70eda1-d9c8-42d5-a0f0-b7bb565be255_1886x834.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:644,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:297331,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d70eda1-d9c8-42d5-a0f0-b7bb565be255_1886x834.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!xnaz!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d70eda1-d9c8-42d5-a0f0-b7bb565be255_1886x834.png 424w, https://substackcdn.com/image/fetch/$s_!xnaz!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d70eda1-d9c8-42d5-a0f0-b7bb565be255_1886x834.png 848w, https://substackcdn.com/image/fetch/$s_!xnaz!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d70eda1-d9c8-42d5-a0f0-b7bb565be255_1886x834.png 1272w, https://substackcdn.com/image/fetch/$s_!xnaz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d70eda1-d9c8-42d5-a0f0-b7bb565be255_1886x834.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The public Alook #intel channel with Atlas, Mara, Theo, Ren, and the human teammates as members.</figcaption></figure></div><ul><li><p><code># alerts</code> is private. It holds Atlas, Ren, and us. Ren posts change summaries there, and Atlas passes the important ones on to us.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ig4y!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbb1da012-c7ca-4b23-b869-1f12ea1df95b_1319x1192.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ig4y!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbb1da012-c7ca-4b23-b869-1f12ea1df95b_1319x1192.png 424w, https://substackcdn.com/image/fetch/$s_!ig4y!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbb1da012-c7ca-4b23-b869-1f12ea1df95b_1319x1192.png 848w, https://substackcdn.com/image/fetch/$s_!ig4y!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbb1da012-c7ca-4b23-b869-1f12ea1df95b_1319x1192.png 1272w, https://substackcdn.com/image/fetch/$s_!ig4y!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbb1da012-c7ca-4b23-b869-1f12ea1df95b_1319x1192.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ig4y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbb1da012-c7ca-4b23-b869-1f12ea1df95b_1319x1192.png" width="1319" height="1192" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bb1da012-c7ca-4b23-b869-1f12ea1df95b_1319x1192.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1192,&quot;width&quot;:1319,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:387771,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbb1da012-c7ca-4b23-b869-1f12ea1df95b_1319x1192.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ig4y!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbb1da012-c7ca-4b23-b869-1f12ea1df95b_1319x1192.png 424w, https://substackcdn.com/image/fetch/$s_!ig4y!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbb1da012-c7ca-4b23-b869-1f12ea1df95b_1319x1192.png 848w, https://substackcdn.com/image/fetch/$s_!ig4y!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbb1da012-c7ca-4b23-b869-1f12ea1df95b_1319x1192.png 1272w, https://substackcdn.com/image/fetch/$s_!ig4y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbb1da012-c7ca-4b23-b869-1f12ea1df95b_1319x1192.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The private Alook #alerts channel with Atlas, Ren, and the human teammates as members.</figcaption></figure></div><p>Channel membership controls who sees each conversation. Mara and Theo are not in <code># alerts</code>, so the research stays in <code>#intel</code> and the alerts stay short. People who only need pricing alerts can join <code>#alerts</code> without following every research discussion.</p><p>Theo still needs a way to compare its findings with previous runs. Otherwise, each check just returns the current page, with no record of what changed.</p><h4>Giving the research agent web access with Nimble skills</h4><p>That comparison is the job of Nimble&#8217;s <code>competitor-positioning</code> skill. Nimble, again, is the web-data layer in this setup. The skill maps a competitor&#8217;s site, finds relevant pages, extracts their content, saves a dated snapshot on your machine, and compares later runs against it.</p><p><strong>1. Install the Nimble CLI</strong></p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;bash&quot;,&quot;nodeId&quot;:&quot;e7ded6cf-3100-42d1-9633-fb56696404da&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-bash">npm i -g @nimble-way/nimble-cli</code></pre></div><p><strong>2. Set your API key</strong></p><p><strong><a href="https://online.nimbleway.com/signup">Sign up for Nimble</a></strong> and get your key from Account Settings, then API Keys.</p><p>The key should be set in the terminal you use to start Alook:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;bash&quot;,&quot;nodeId&quot;:&quot;d673ed8a-7707-4d6d-90c7-0b5bd43a65db&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-bash">export NIMBLE_API_KEY="your-api-key-here"</code></pre></div><p>If Alook is already running from the setup above, restart it from this terminal so the agents pick up the key. Exporting the key in a different terminal does not update a daemon that is already running.</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;bash&quot;,&quot;nodeId&quot;:&quot;fd3dc55e-862b-4b2b-9df0-b66d9131e8a4&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-bash">npx @alook/app stop
npx @alook/app start</code></pre></div><p><strong>3. Add agent skills</strong></p><p>The repository contains 15 MIT-licensed skills built to the Agent Skills specification. This walkthrough uses <code>v1.7.0</code>. The open-source skills repo is separate from Nimble&#8217;s hosted API.</p><p>The skills can be added with this command:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;bash&quot;,&quot;nodeId&quot;:&quot;8be761cd-6158-4f79-86b0-d7865025635e&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-bash">npx skills add Nimbleway/agent-skills -g</code></pre></div><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!jjU2!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc41b8cbd-8c56-468b-9a66-aa191f1083a5_1474x281.png 424w, https://substackcdn.com/image/fetch/$s_!jjU2!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc41b8cbd-8c56-468b-9a66-aa191f1083a5_1474x281.png 848w, https://substackcdn.com/image/fetch/$s_!jjU2!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc41b8cbd-8c56-468b-9a66-aa191f1083a5_1474x281.png 1272w, https://substackcdn.com/image/fetch/$s_!jjU2!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc41b8cbd-8c56-468b-9a66-aa191f1083a5_1474x281.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!jjU2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc41b8cbd-8c56-468b-9a66-aa191f1083a5_1474x281.png" width="1456" height="278" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c41b8cbd-8c56-468b-9a66-aa191f1083a5_1474x281.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:278,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:85413,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc41b8cbd-8c56-468b-9a66-aa191f1083a5_1474x281.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!jjU2!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc41b8cbd-8c56-468b-9a66-aa191f1083a5_1474x281.png 424w, https://substackcdn.com/image/fetch/$s_!jjU2!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc41b8cbd-8c56-468b-9a66-aa191f1083a5_1474x281.png 848w, https://substackcdn.com/image/fetch/$s_!jjU2!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc41b8cbd-8c56-468b-9a66-aa191f1083a5_1474x281.png 1272w, https://substackcdn.com/image/fetch/$s_!jjU2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc41b8cbd-8c56-468b-9a66-aa191f1083a5_1474x281.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Terminal output from installing the Nimbleway/agent-skills package globally.</figcaption></figure></div><p>To confirm Claude Code can see the skills, open it in the same folder and run <code>/skills</code>. <code>competitor-positioning</code> should be in the list, as shown below.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!4lEE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87100f75-6377-454a-861a-2a8fdc9b1e6b_1942x809.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!4lEE!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87100f75-6377-454a-861a-2a8fdc9b1e6b_1942x809.png 424w, https://substackcdn.com/image/fetch/$s_!4lEE!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87100f75-6377-454a-861a-2a8fdc9b1e6b_1942x809.png 848w, https://substackcdn.com/image/fetch/$s_!4lEE!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87100f75-6377-454a-861a-2a8fdc9b1e6b_1942x809.png 1272w, https://substackcdn.com/image/fetch/$s_!4lEE!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87100f75-6377-454a-861a-2a8fdc9b1e6b_1942x809.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!4lEE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87100f75-6377-454a-861a-2a8fdc9b1e6b_1942x809.png" width="1456" height="607" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/87100f75-6377-454a-861a-2a8fdc9b1e6b_1942x809.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:607,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:516986,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87100f75-6377-454a-861a-2a8fdc9b1e6b_1942x809.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!4lEE!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87100f75-6377-454a-861a-2a8fdc9b1e6b_1942x809.png 424w, https://substackcdn.com/image/fetch/$s_!4lEE!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87100f75-6377-454a-861a-2a8fdc9b1e6b_1942x809.png 848w, https://substackcdn.com/image/fetch/$s_!4lEE!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87100f75-6377-454a-861a-2a8fdc9b1e6b_1942x809.png 1272w, https://substackcdn.com/image/fetch/$s_!4lEE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87100f75-6377-454a-861a-2a8fdc9b1e6b_1942x809.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Claude Code /skills list showing competitor-positioning among the installed Nimble skills.</figcaption></figure></div><p>A quick way to check the setup is to run the skill once from Claude Code. If the content comes back, the CLI, the key, and the skills are all wired up.</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;plaintext&quot;,&quot;nodeId&quot;:&quot;a0b03df8-a3cb-4d24-884e-369c67845823&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-plaintext">/competitor-positioning
What's Railway.com's current pricing and messaging?</code></pre></div><p>With the agents and channels set up and Nimble&#8217;s skills installed, the same job can be handed to the team, so anyone in the room can ask for it.</p><h4>How a request passes from one agent to the next</h4><p>The request goes to Atlas in the Alook room. The rest of the team takes it from there.</p><p>We asked Atlas to track pricing and positioning on Railway, as shown below.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!o_P-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec88eb9c-aac6-4386-9f82-8898c2147b66_1200x947.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!o_P-!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec88eb9c-aac6-4386-9f82-8898c2147b66_1200x947.png 424w, https://substackcdn.com/image/fetch/$s_!o_P-!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec88eb9c-aac6-4386-9f82-8898c2147b66_1200x947.png 848w, https://substackcdn.com/image/fetch/$s_!o_P-!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec88eb9c-aac6-4386-9f82-8898c2147b66_1200x947.png 1272w, https://substackcdn.com/image/fetch/$s_!o_P-!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec88eb9c-aac6-4386-9f82-8898c2147b66_1200x947.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!o_P-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec88eb9c-aac6-4386-9f82-8898c2147b66_1200x947.png" width="1200" height="947" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ec88eb9c-aac6-4386-9f82-8898c2147b66_1200x947.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:947,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:235571,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec88eb9c-aac6-4386-9f82-8898c2147b66_1200x947.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!o_P-!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec88eb9c-aac6-4386-9f82-8898c2147b66_1200x947.png 424w, https://substackcdn.com/image/fetch/$s_!o_P-!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec88eb9c-aac6-4386-9f82-8898c2147b66_1200x947.png 848w, https://substackcdn.com/image/fetch/$s_!o_P-!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec88eb9c-aac6-4386-9f82-8898c2147b66_1200x947.png 1272w, https://substackcdn.com/image/fetch/$s_!o_P-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec88eb9c-aac6-4386-9f82-8898c2147b66_1200x947.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A direct message to Atlas asking it to track Railway&#8217;s pricing and alert on changes, and Atlas handing the baseline to Mara.</figcaption></figure></div><p>Atlas opens a task in <code># intel</code> and brings Mara into the conversation. The request and follow-ups stay in the same channel.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!R2NQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c14cabf-d37a-4400-9545-6e7d0b039479_1459x670.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!R2NQ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c14cabf-d37a-4400-9545-6e7d0b039479_1459x670.png 424w, https://substackcdn.com/image/fetch/$s_!R2NQ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c14cabf-d37a-4400-9545-6e7d0b039479_1459x670.png 848w, https://substackcdn.com/image/fetch/$s_!R2NQ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c14cabf-d37a-4400-9545-6e7d0b039479_1459x670.png 1272w, https://substackcdn.com/image/fetch/$s_!R2NQ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c14cabf-d37a-4400-9545-6e7d0b039479_1459x670.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!R2NQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c14cabf-d37a-4400-9545-6e7d0b039479_1459x670.png" width="1456" height="669" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9c14cabf-d37a-4400-9545-6e7d0b039479_1459x670.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:669,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:115622,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c14cabf-d37a-4400-9545-6e7d0b039479_1459x670.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!R2NQ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c14cabf-d37a-4400-9545-6e7d0b039479_1459x670.png 424w, https://substackcdn.com/image/fetch/$s_!R2NQ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c14cabf-d37a-4400-9545-6e7d0b039479_1459x670.png 848w, https://substackcdn.com/image/fetch/$s_!R2NQ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c14cabf-d37a-4400-9545-6e7d0b039479_1459x670.png 1272w, https://substackcdn.com/image/fetch/$s_!R2NQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c14cabf-d37a-4400-9545-6e7d0b039479_1459x670.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Atlas opening a task in the #intel channel and bringing Mara into the conversation.</figcaption></figure></div><p>Mara turns the request into a spec for Theo and Ren. It spells out which pages to track, how to save timestamped snapshots, how to detect changes, and what the final report should include.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!9_0q!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fb89364-84cf-4712-805a-0513eab8e009_2008x783.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!9_0q!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fb89364-84cf-4712-805a-0513eab8e009_2008x783.png 424w, https://substackcdn.com/image/fetch/$s_!9_0q!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fb89364-84cf-4712-805a-0513eab8e009_2008x783.png 848w, https://substackcdn.com/image/fetch/$s_!9_0q!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fb89364-84cf-4712-805a-0513eab8e009_2008x783.png 1272w, https://substackcdn.com/image/fetch/$s_!9_0q!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fb89364-84cf-4712-805a-0513eab8e009_2008x783.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!9_0q!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fb89364-84cf-4712-805a-0513eab8e009_2008x783.png" width="1456" height="568" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6fb89364-84cf-4712-805a-0513eab8e009_2008x783.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:568,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:451408,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fb89364-84cf-4712-805a-0513eab8e009_2008x783.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!9_0q!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fb89364-84cf-4712-805a-0513eab8e009_2008x783.png 424w, https://substackcdn.com/image/fetch/$s_!9_0q!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fb89364-84cf-4712-805a-0513eab8e009_2008x783.png 848w, https://substackcdn.com/image/fetch/$s_!9_0q!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fb89364-84cf-4712-805a-0513eab8e009_2008x783.png 1272w, https://substackcdn.com/image/fetch/$s_!9_0q!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fb89364-84cf-4712-805a-0513eab8e009_2008x783.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Mara&#8217;s spec in #intel describing which pages to track, how to save snapshots, and what the report should include.</figcaption></figure></div><p>Theo runs Nimble&#8217;s <code>competitor-positioning</code> skill and reports back to the room once the snapshot is saved, as shown below.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!QqFh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1490acb1-9b9d-42fa-bbee-e2d97fad1ea7_1993x789.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!QqFh!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1490acb1-9b9d-42fa-bbee-e2d97fad1ea7_1993x789.png 424w, https://substackcdn.com/image/fetch/$s_!QqFh!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1490acb1-9b9d-42fa-bbee-e2d97fad1ea7_1993x789.png 848w, https://substackcdn.com/image/fetch/$s_!QqFh!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1490acb1-9b9d-42fa-bbee-e2d97fad1ea7_1993x789.png 1272w, https://substackcdn.com/image/fetch/$s_!QqFh!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1490acb1-9b9d-42fa-bbee-e2d97fad1ea7_1993x789.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!QqFh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1490acb1-9b9d-42fa-bbee-e2d97fad1ea7_1993x789.png" width="1456" height="576" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1490acb1-9b9d-42fa-bbee-e2d97fad1ea7_1993x789.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:576,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:360427,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1490acb1-9b9d-42fa-bbee-e2d97fad1ea7_1993x789.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!QqFh!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1490acb1-9b9d-42fa-bbee-e2d97fad1ea7_1993x789.png 424w, https://substackcdn.com/image/fetch/$s_!QqFh!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1490acb1-9b9d-42fa-bbee-e2d97fad1ea7_1993x789.png 848w, https://substackcdn.com/image/fetch/$s_!QqFh!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1490acb1-9b9d-42fa-bbee-e2d97fad1ea7_1993x789.png 1272w, https://substackcdn.com/image/fetch/$s_!QqFh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1490acb1-9b9d-42fa-bbee-e2d97fad1ea7_1993x789.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Theo&#8217;s baseline report in #intel, with Mara accepting it against the spec.</figcaption></figure></div><p>The skill does three jobs on this run.</p><ol><li><p>The skill discovers the page structure and finds relevant URLs.</p></li><li><p>It pulls content from the pages we need. For Railway, that meant the pricing page and plan documentation.</p></li><li><p>It checks the new evidence against the previous snapshot. Since this is the first run, there is no snapshot to compare against, so the skill saves a baseline.</p></li></ol><p>Theo posts the baseline in <code># intel</code>. Mara checks it against the spec, and Atlas sends us a summary by DM.</p><p>Here is what the first run saved under <code>~/.nimble/memory/</code>, as shown below.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!u5Yp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F134e6a47-ac8f-459c-952f-20b3ab630817_1472x1069.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!u5Yp!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F134e6a47-ac8f-459c-952f-20b3ab630817_1472x1069.png 424w, https://substackcdn.com/image/fetch/$s_!u5Yp!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F134e6a47-ac8f-459c-952f-20b3ab630817_1472x1069.png 848w, https://substackcdn.com/image/fetch/$s_!u5Yp!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F134e6a47-ac8f-459c-952f-20b3ab630817_1472x1069.png 1272w, https://substackcdn.com/image/fetch/$s_!u5Yp!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F134e6a47-ac8f-459c-952f-20b3ab630817_1472x1069.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!u5Yp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F134e6a47-ac8f-459c-952f-20b3ab630817_1472x1069.png" width="1456" height="1057" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/134e6a47-ac8f-459c-952f-20b3ab630817_1472x1069.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1057,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:490891,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F134e6a47-ac8f-459c-952f-20b3ab630817_1472x1069.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!u5Yp!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F134e6a47-ac8f-459c-952f-20b3ab630817_1472x1069.png 424w, https://substackcdn.com/image/fetch/$s_!u5Yp!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F134e6a47-ac8f-459c-952f-20b3ab630817_1472x1069.png 848w, https://substackcdn.com/image/fetch/$s_!u5Yp!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F134e6a47-ac8f-459c-952f-20b3ab630817_1472x1069.png 1272w, https://substackcdn.com/image/fetch/$s_!u5Yp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F134e6a47-ac8f-459c-952f-20b3ab630817_1472x1069.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Files saved under ~/.nimble/memory/, including positioning/railway.md, the 9 September briefing, and log.md.</figcaption></figure></div><p>The run wrote three main files.</p><ul><li><p><code>positioning/railway .md</code> stores the snapshot and its history</p></li><li><p>An md file in reports contains the full briefing</p></li><li><p>Log md records the run</p></li></ul><p>These are Markdown files on our machine, not records locked in a separate dashboard. You can open them directly, search their contents, or browse them in Obsidian using their <code>[[wiki links]]</code>. Stored files do not stay current on their own. Later runs have to read them and refresh them.</p><p>On the next run, Theo can use the saved history to compare Railway&#8217;s latest pages with the baseline.</p><h4>Diffing a new capture against the saved snapshot</h4><p>We had a baseline from 9 September. On 23 October, we asked Atlas to have Theo run the same skill again and compare the results, as shown below.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!1j7Q!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5220e02d-3d3c-4d9a-806a-dea466e2d308_1200x307.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!1j7Q!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5220e02d-3d3c-4d9a-806a-dea466e2d308_1200x307.png 424w, https://substackcdn.com/image/fetch/$s_!1j7Q!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5220e02d-3d3c-4d9a-806a-dea466e2d308_1200x307.png 848w, https://substackcdn.com/image/fetch/$s_!1j7Q!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5220e02d-3d3c-4d9a-806a-dea466e2d308_1200x307.png 1272w, https://substackcdn.com/image/fetch/$s_!1j7Q!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5220e02d-3d3c-4d9a-806a-dea466e2d308_1200x307.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!1j7Q!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5220e02d-3d3c-4d9a-806a-dea466e2d308_1200x307.png" width="1200" height="307" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5220e02d-3d3c-4d9a-806a-dea466e2d308_1200x307.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:307,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:46015,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5220e02d-3d3c-4d9a-806a-dea466e2d308_1200x307.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!1j7Q!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5220e02d-3d3c-4d9a-806a-dea466e2d308_1200x307.png 424w, https://substackcdn.com/image/fetch/$s_!1j7Q!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5220e02d-3d3c-4d9a-806a-dea466e2d308_1200x307.png 848w, https://substackcdn.com/image/fetch/$s_!1j7Q!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5220e02d-3d3c-4d9a-806a-dea466e2d308_1200x307.png 1272w, https://substackcdn.com/image/fetch/$s_!1j7Q!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5220e02d-3d3c-4d9a-806a-dea466e2d308_1200x307.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A follow-up message asking Atlas to have Theo rerun the skill and compare it with the 9 September snapshot.</figcaption></figure></div><p>The run found several changes on Railway&#8217;s pricing page.</p><ul><li><p>A new section on VM pricing, marked beta in September, now has a full Sandboxes &amp; VMs section.</p></li><li><p>A changed tier, i.e., the $5,000 committed-spend tier changed from &#8220;enterprise support + regions&#8221; (9 Sep) to &#8220;Enterprise Support, Higher Resource Limits&#8221;  (23 Oct), dropping cloud regions.</p></li><li><p>Plan prices remain Free ($0, then $1/month), Hobby ($5 minimum), Pro ($20 minimum), and Enterprise (custom).</p></li></ul><p>Theo posted the diff in <code># intel</code>. Mara passed it to Ren, who posted a summary in <code># alerts</code>, as shown below.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!8kgc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72874b3d-8e71-4318-b9ac-cd57f80f42d1_1456x921.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8kgc!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72874b3d-8e71-4318-b9ac-cd57f80f42d1_1456x921.png 424w, https://substackcdn.com/image/fetch/$s_!8kgc!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72874b3d-8e71-4318-b9ac-cd57f80f42d1_1456x921.png 848w, https://substackcdn.com/image/fetch/$s_!8kgc!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72874b3d-8e71-4318-b9ac-cd57f80f42d1_1456x921.png 1272w, https://substackcdn.com/image/fetch/$s_!8kgc!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72874b3d-8e71-4318-b9ac-cd57f80f42d1_1456x921.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!8kgc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72874b3d-8e71-4318-b9ac-cd57f80f42d1_1456x921.png" width="1456" height="921" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/72874b3d-8e71-4318-b9ac-cd57f80f42d1_1456x921.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:921,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:163844,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224798473?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72874b3d-8e71-4318-b9ac-cd57f80f42d1_1456x921.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!8kgc!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72874b3d-8e71-4318-b9ac-cd57f80f42d1_1456x921.png 424w, https://substackcdn.com/image/fetch/$s_!8kgc!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72874b3d-8e71-4318-b9ac-cd57f80f42d1_1456x921.png 848w, https://substackcdn.com/image/fetch/$s_!8kgc!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72874b3d-8e71-4318-b9ac-cd57f80f42d1_1456x921.png 1272w, https://substackcdn.com/image/fetch/$s_!8kgc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72874b3d-8e71-4318-b9ac-cd57f80f42d1_1456x921.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Ren&#8217;s alert in #alerts with both capture dates, grouped changes, and the source link.</figcaption></figure></div><p>Nobody had to reopen a terminal or check again. If someone wants to know, they can ask Theo in the same thread. The answer stays there for the next person who needs it.</p><p>The workflow flags changes found on a later check, rather than the instant a page updates. In this walkthrough, the second check was a request we sent on 23 October, six weeks after the baseline.</p><div><hr></div><h4>Saved snapshots carry context between runs</h4><p>Every agent here ran on Claude Code, but Alook also supports Codex, Cursor, OpenCode, and Pi in the same room.</p><p>Between runs, Alook carries each agent&#8217;s task context forward and the skill keeps a dated snapshot, so Theo compares new evidence with the last capture instead of starting from scratch. Each run still costs model and API calls. Teammates can also read  the room history to see how a decision was made and ask follow-ups themselves.</p><p>The same pattern also scales to more work. We sent Atlas one message:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;plaintext&quot;,&quot;nodeId&quot;:&quot;26006b6c-c8f7-4039-a344-ee92723b7f80&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-plaintext">Track Railway, Render, Fly.io, Vercel, Netlify, and
Koyeb for competitive positioning.</code></pre></div><p>Atlas handed the five new ones to the team the same way as Railway, and each one came back with its own dated, sourced baseline.</p><p>The agents were already doing the work. The missing piece was letting your team work with them, too.</p><p>A good next step is to pick one competitor and a question you want answered, then try it with the Nimble agent skills.</p><p><strong><a href="https://github.com/Nimbleway/agent-skills">Here&#8217;s the Nimble agent skills repo for web research &#8594;</a></strong></p><p>The shared rooms in this walkthrough are built with Alook, which is open source too.</p><p><strong><a href="https://github.com/alookai/alook">Build your own Multi-Agent collaboration for Team via Alook&#8217;s GitHub repo &#8594;</a></strong></p><p>&#128073; Over to you: Which repeated research job would you hand to an agent team first?</p><p>Good day!</p>]]></content:encoded></item><item><title><![CDATA[How Top AI Teams Scale to 1,000+ Agents]]></title><description><![CDATA[...explained visually, plus a Claude Code and Codex walkthrough.]]></description><link>https://blog.dailydoseofds.com/p/how-top-ai-teams-scale-to-1000-agents</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/how-top-ai-teams-scale-to-1000-agents</guid><dc:creator><![CDATA[Avi Chawla]]></dc:creator><pubDate>Thu, 12 Nov 2026 22:52:10 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/ef84ddc6-9b5f-46b9-b936-e112de4446b6_1983x793.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><strong>In today&#8217;s newsletter:</strong></p><ul><li><p>Turn your agents into a team that works toward shared goals.</p></li><li><p>How top AI teams scale to 1,000+ agents.</p></li></ul><div><hr></div><h3><a href="https://github.com/rowboatlabs/rowboat">Turn your agents into a team that works toward shared goals</a></h3><p>Most of us still glue our agents together by hand. One writes the plan, another looks at the data, a third ships the change, and we sit in the middle passing context around.</p><p><strong><a href="https://github.com/rowboatlabs/rowboat">Rowboat</a></strong> is an open-source, self-hostable layer that turns those agents into a team working toward shared goals.</p><p>It can bring Claude Code, Codex, and Hermes into one team. Any agent that supports MCP can join too, so you stop being the glue between your AI agents.</p><p>You send one message, and a super agent:</p><ul><li><p>Decides which agents the job needs</p></li><li><p>Asks your permission and creates them</p></li><li><p>Hands each one a task</p></li><li><p>Lets them pass work along in threads</p></li><li><p>Builds the result with them on a shared whiteboard</p></li></ul><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;aad44c90-519d-4e29-bafb-4210963c832c&quot;,&quot;duration&quot;:null}"></div><p>What makes it different:</p><ul><li><p>Runs on the subscriptions or API keys you already have</p></li><li><p>See every handoff, and step in whenever you want</p></li><li><p>Or let Rowboat host your agents, if you&#8217;d rather not run servers</p></li><li><p>Free to use</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!25yb!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a0883d9-a145-4baf-8cdc-25514c6f17e0_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!25yb!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a0883d9-a145-4baf-8cdc-25514c6f17e0_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!25yb!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a0883d9-a145-4baf-8cdc-25514c6f17e0_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!25yb!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a0883d9-a145-4baf-8cdc-25514c6f17e0_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!25yb!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a0883d9-a145-4baf-8cdc-25514c6f17e0_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!25yb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a0883d9-a145-4baf-8cdc-25514c6f17e0_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9a0883d9-a145-4baf-8cdc-25514c6f17e0_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:161013,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224646450?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a0883d9-a145-4baf-8cdc-25514c6f17e0_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!25yb!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a0883d9-a145-4baf-8cdc-25514c6f17e0_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!25yb!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a0883d9-a145-4baf-8cdc-25514c6f17e0_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!25yb!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a0883d9-a145-4baf-8cdc-25514c6f17e0_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!25yb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9a0883d9-a145-4baf-8cdc-25514c6f17e0_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">You send one message. A super agent staffs the job, then you only step in when you want to.</figcaption></figure></div><p>The video above shows it in action. We loaded nine weeks of product data for a sample app into PostHog, then asked why new users were dropping off after week one. Hermes found where they dropped off in PostHog, Claude Code found why in the user feedback, and Codex built the dashboard.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!TC-3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad922499-889f-4600-9078-cb86917a3f50_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!TC-3!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad922499-889f-4600-9078-cb86917a3f50_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!TC-3!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad922499-889f-4600-9078-cb86917a3f50_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!TC-3!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad922499-889f-4600-9078-cb86917a3f50_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!TC-3!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad922499-889f-4600-9078-cb86917a3f50_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!TC-3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad922499-889f-4600-9078-cb86917a3f50_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ad922499-889f-4600-9078-cb86917a3f50_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:171254,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224646450?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad922499-889f-4600-9078-cb86917a3f50_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!TC-3!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad922499-889f-4600-9078-cb86917a3f50_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!TC-3!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad922499-889f-4600-9078-cb86917a3f50_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!TC-3!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad922499-889f-4600-9078-cb86917a3f50_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!TC-3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fad922499-889f-4600-9078-cb86917a3f50_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Hermes found the drop-off, Claude Code found the why, and Codex built the dashboard.</figcaption></figure></div><p><strong><a href="https://github.com/rowboatlabs/rowboat">Explore the Rowboat repo &#8594;</a></strong></p><p>(don&#8217;t forget to star it &#11088;)</p><div><hr></div><h3><a href="https://github.com/ModernRelay/omnigraph">How top AI teams scale to 1,000+ agents</a></h3><p>Cognition wrote in April that multi-agent systems work best when &#8220;writes stay single-threaded.&#8221;</p><p>This implies that while many agents can think in parallel, only one of them should save to shared memory.</p><p>Shared writes are risky for a concrete database reason, known as the lost update, in which two writers save the same record and the second save silently erases the first, with no error on either side.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!oonU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1230866-17cd-4ff4-8d77-c5075010398d_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!oonU!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1230866-17cd-4ff4-8d77-c5075010398d_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!oonU!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1230866-17cd-4ff4-8d77-c5075010398d_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!oonU!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1230866-17cd-4ff4-8d77-c5075010398d_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!oonU!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1230866-17cd-4ff4-8d77-c5075010398d_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!oonU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1230866-17cd-4ff4-8d77-c5075010398d_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e1230866-17cd-4ff4-8d77-c5075010398d_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:123801,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224646450?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1230866-17cd-4ff4-8d77-c5075010398d_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!oonU!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1230866-17cd-4ff4-8d77-c5075010398d_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!oonU!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1230866-17cd-4ff4-8d77-c5075010398d_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!oonU!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1230866-17cd-4ff4-8d77-c5075010398d_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!oonU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1230866-17cd-4ff4-8d77-c5075010398d_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Last write wins, and one agent quietly loses its work.</figcaption></figure></div><p>For a developer running Claude Code and Codex in parallel on shared memory, this turns parallelism into a liability, because the more those agents write at once, the more of their saved work can potentially disappear.</p><p>Today, let&#8217;s understand why this happens, the typical solutions and their problems, and how<a href="https://github.com/ModernRelay/omnigraph"> </a><strong><a href="https://github.com/ModernRelay/omnigraph">Omnigraph</a></strong> (open-source), a graph database with the Git model built in, solves it, followed by a hands-on walkthrough with Claude Code and Codex.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!3Hvq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fead495fe-c1cc-45ac-9b0b-848e369d2f72_876x513.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!3Hvq!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fead495fe-c1cc-45ac-9b0b-848e369d2f72_876x513.png 424w, https://substackcdn.com/image/fetch/$s_!3Hvq!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fead495fe-c1cc-45ac-9b0b-848e369d2f72_876x513.png 848w, https://substackcdn.com/image/fetch/$s_!3Hvq!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fead495fe-c1cc-45ac-9b0b-848e369d2f72_876x513.png 1272w, https://substackcdn.com/image/fetch/$s_!3Hvq!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fead495fe-c1cc-45ac-9b0b-848e369d2f72_876x513.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!3Hvq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fead495fe-c1cc-45ac-9b0b-848e369d2f72_876x513.png" width="876" height="513" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ead495fe-c1cc-45ac-9b0b-848e369d2f72_876x513.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:513,&quot;width&quot;:876,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:30616,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224646450?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fead495fe-c1cc-45ac-9b0b-848e369d2f72_876x513.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!3Hvq!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fead495fe-c1cc-45ac-9b0b-848e369d2f72_876x513.png 424w, https://substackcdn.com/image/fetch/$s_!3Hvq!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fead495fe-c1cc-45ac-9b0b-848e369d2f72_876x513.png 848w, https://substackcdn.com/image/fetch/$s_!3Hvq!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fead495fe-c1cc-45ac-9b0b-848e369d2f72_876x513.png 1272w, https://substackcdn.com/image/fetch/$s_!3Hvq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fead495fe-c1cc-45ac-9b0b-848e369d2f72_876x513.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Omnigraph is an open-source graph database with Git-style branching.</figcaption></figure></div><div><hr></div><h4>Why shared agent memory loses writes</h4><p>Agent memory updates are usually read-modify-write cycles, where an agent loads a record into its context window and later writes back whatever version the model produced.</p><p>So consider a shared project graph where Claude Code and Codex read the <code>sessions</code> component at the same moment, when it says Redis and has no owner.</p><p>Claude Code finishes first and saves the store as Postgres, while Codex, still working from its earlier read, sets the owner and saves its own copy of the record.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!CmCY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F805e76fb-a6b2-4755-a794-23dd4c85b961_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!CmCY!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F805e76fb-a6b2-4755-a794-23dd4c85b961_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!CmCY!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F805e76fb-a6b2-4755-a794-23dd4c85b961_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!CmCY!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F805e76fb-a6b2-4755-a794-23dd4c85b961_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!CmCY!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F805e76fb-a6b2-4755-a794-23dd4c85b961_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!CmCY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F805e76fb-a6b2-4755-a794-23dd4c85b961_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/805e76fb-a6b2-4755-a794-23dd4c85b961_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:125551,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224646450?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F805e76fb-a6b2-4755-a794-23dd4c85b961_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!CmCY!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F805e76fb-a6b2-4755-a794-23dd4c85b961_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!CmCY!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F805e76fb-a6b2-4755-a794-23dd4c85b961_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!CmCY!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F805e76fb-a6b2-4755-a794-23dd4c85b961_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!CmCY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F805e76fb-a6b2-4755-a794-23dd4c85b961_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Both agents read sessions at the same version, then Codex&#8217;s stale full-record save puts the store back to Redis.</figcaption></figure></div><p>Codex&#8217;s copy still says Redis, so its save puts <code>sessions</code> back on Redis and erases Claude Code&#8217;s update. Both agents report success, and nothing in the logs shows that one of them was overwritten.</p><p>Agents make this worse than human users do.</p><p>A person editing a record usually changes one field and saves only that field, but an agent typically writes back the entire record it read earlier, so any field another agent updated in the meantime gets reset to the old value.</p><p>There are three common ways to fix this, and each has some trade-offs:</p><ul><li><p>Single writer &#8594; one agent owns every write and the others send it suggestions, which keeps memory correct but caps the whole system at the speed of that one agent.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!DERv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2124acf7-70c0-40fc-a5d5-f4ea404dbc9e_1200x448.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!DERv!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2124acf7-70c0-40fc-a5d5-f4ea404dbc9e_1200x448.png 424w, https://substackcdn.com/image/fetch/$s_!DERv!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2124acf7-70c0-40fc-a5d5-f4ea404dbc9e_1200x448.png 848w, https://substackcdn.com/image/fetch/$s_!DERv!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2124acf7-70c0-40fc-a5d5-f4ea404dbc9e_1200x448.png 1272w, https://substackcdn.com/image/fetch/$s_!DERv!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2124acf7-70c0-40fc-a5d5-f4ea404dbc9e_1200x448.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!DERv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2124acf7-70c0-40fc-a5d5-f4ea404dbc9e_1200x448.png" width="1200" height="448" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2124acf7-70c0-40fc-a5d5-f4ea404dbc9e_1200x448.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:448,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:129191,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224646450?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2124acf7-70c0-40fc-a5d5-f4ea404dbc9e_1200x448.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!DERv!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2124acf7-70c0-40fc-a5d5-f4ea404dbc9e_1200x448.png 424w, https://substackcdn.com/image/fetch/$s_!DERv!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2124acf7-70c0-40fc-a5d5-f4ea404dbc9e_1200x448.png 848w, https://substackcdn.com/image/fetch/$s_!DERv!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2124acf7-70c0-40fc-a5d5-f4ea404dbc9e_1200x448.png 1272w, https://substackcdn.com/image/fetch/$s_!DERv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2124acf7-70c0-40fc-a5d5-f4ea404dbc9e_1200x448.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">One agent owns every write, and the others only send suggestions.</figcaption></figure></div><ul><li><p>Locks and optimistic version checks &#8594; a conflict only fires when two writers touch the same record, so a version check would have rejected Codex&#8217;s stale save in the above case. Both still guard one record at a time, though, while an agent&#8217;s task usually spans several records and several steps, and neither leaves a diff to review or a history to audit.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xB2o!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e1bc4b6-77d8-40fc-8750-bc69d04b43be_3168x1344.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xB2o!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e1bc4b6-77d8-40fc-8750-bc69d04b43be_3168x1344.jpeg 424w, https://substackcdn.com/image/fetch/$s_!xB2o!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e1bc4b6-77d8-40fc-8750-bc69d04b43be_3168x1344.jpeg 848w, https://substackcdn.com/image/fetch/$s_!xB2o!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e1bc4b6-77d8-40fc-8750-bc69d04b43be_3168x1344.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!xB2o!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e1bc4b6-77d8-40fc-8750-bc69d04b43be_3168x1344.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!xB2o!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e1bc4b6-77d8-40fc-8750-bc69d04b43be_3168x1344.jpeg" width="1456" height="618" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7e1bc4b6-77d8-40fc-8750-bc69d04b43be_3168x1344.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:618,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:104051,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224646450?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e1bc4b6-77d8-40fc-8750-bc69d04b43be_3168x1344.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!xB2o!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e1bc4b6-77d8-40fc-8750-bc69d04b43be_3168x1344.jpeg 424w, https://substackcdn.com/image/fetch/$s_!xB2o!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e1bc4b6-77d8-40fc-8750-bc69d04b43be_3168x1344.jpeg 848w, https://substackcdn.com/image/fetch/$s_!xB2o!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e1bc4b6-77d8-40fc-8750-bc69d04b43be_3168x1344.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!xB2o!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7e1bc4b6-77d8-40fc-8750-bc69d04b43be_3168x1344.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Locks and version checks are safe, but they only guard one record at a time.</figcaption></figure></div><ul><li><p>Last write wins &#8594; every agent saves freely, and the latest save overwrites the rest, which keeps everyone fast and is exactly how Claude Code&#8217;s update disappeared in the above example.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!jsrh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5717972-f3da-45eb-a728-ac52d0dcedc4_3168x1344.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!jsrh!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5717972-f3da-45eb-a728-ac52d0dcedc4_3168x1344.jpeg 424w, https://substackcdn.com/image/fetch/$s_!jsrh!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5717972-f3da-45eb-a728-ac52d0dcedc4_3168x1344.jpeg 848w, https://substackcdn.com/image/fetch/$s_!jsrh!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5717972-f3da-45eb-a728-ac52d0dcedc4_3168x1344.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!jsrh!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5717972-f3da-45eb-a728-ac52d0dcedc4_3168x1344.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!jsrh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5717972-f3da-45eb-a728-ac52d0dcedc4_3168x1344.jpeg" width="1456" height="618" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a5717972-f3da-45eb-a728-ac52d0dcedc4_3168x1344.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:618,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:118002,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224646450?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5717972-f3da-45eb-a728-ac52d0dcedc4_3168x1344.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!jsrh!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5717972-f3da-45eb-a728-ac52d0dcedc4_3168x1344.jpeg 424w, https://substackcdn.com/image/fetch/$s_!jsrh!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5717972-f3da-45eb-a728-ac52d0dcedc4_3168x1344.jpeg 848w, https://substackcdn.com/image/fetch/$s_!jsrh!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5717972-f3da-45eb-a728-ac52d0dcedc4_3168x1344.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!jsrh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5717972-f3da-45eb-a728-ac52d0dcedc4_3168x1344.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Last write wins is fast, but writes vanish.</figcaption></figure></div><p>Collectively, these fixes also miss isolation.</p><p>An agent working through a multi-step task needs its own workspace, a diff of what it changed, a review before that change is pushed, and a record of who changed what.</p><p>Code faced the same problem when many engineers edited one codebase, and version control solved it without making anyone wait.</p><p>In Git, each engineer works on an isolated branch, a diff shows exactly what changed, and a merge compares both sides against the commit they started from, called the merge base, so a change built on an outdated version gets caught before it lands, while the history keeps every step.</p><p>Agent memory needs that same versioning model, and that model is implemented by Omnigraph.</p><div><hr></div><h4>How Omnigraph applies the Git model to agent memory</h4><p>Omnigraph is an open-source graph database, written in Rust under the MIT license, with the Git model built in, so agent memory gets branches, diffs, merges, and history.</p><p>Each agent writes to its own branch, so no agent waits on another, and nothing reaches main until a merge compares that branch with its merge base.</p><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;e7eddbdb-a06f-4e90-a281-e53b7d9cdd7d&quot;,&quot;duration&quot;:null}"></div><p>If main has changed the same record since the branch was created, the merge refuses the write, publishes nothing, and tells the agent to refresh and try again.</p><p>The binary also ships an agent skill, so Claude Code and Codex can drive branches and merges from plain-English prompts.</p><p>To test it, we started both agents together on two tasks that touch the same record. The recording below is the real run, with help-page lookups trimmed:</p><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;f429164b-ccbf-4fb8-9f1c-0b1b976b4e0e&quot;,&quot;duration&quot;:null}"></div><p>Claude Code and Codex running in parallel, each on its own Omnigraph branch.</p><div><hr></div><h4>Step 1: The CLI and the agent skill</h4><p>The CLI installs with one script, and <code>skill export</code> writes the skill where Claude Code and Codex look for it:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;bash&quot;,&quot;nodeId&quot;:null}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-bash">curl -fsSL https://raw.githubusercontent.com/ModernRelay/omnigraph/main/scripts/install.sh | bash

omnigraph skill export ~/.claude/skills/omnigraph
omnigraph skill export ~/.agents/skills/omnigraph</code></pre></div><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!mvtA!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f6b0ce-5b19-4820-b2ba-004a2587f964_3200x1253.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mvtA!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f6b0ce-5b19-4820-b2ba-004a2587f964_3200x1253.png 424w, https://substackcdn.com/image/fetch/$s_!mvtA!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f6b0ce-5b19-4820-b2ba-004a2587f964_3200x1253.png 848w, https://substackcdn.com/image/fetch/$s_!mvtA!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f6b0ce-5b19-4820-b2ba-004a2587f964_3200x1253.png 1272w, https://substackcdn.com/image/fetch/$s_!mvtA!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f6b0ce-5b19-4820-b2ba-004a2587f964_3200x1253.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!mvtA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f6b0ce-5b19-4820-b2ba-004a2587f964_3200x1253.png" width="1456" height="570" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/53f6b0ce-5b19-4820-b2ba-004a2587f964_3200x1253.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:570,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:55044,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224646450?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f6b0ce-5b19-4820-b2ba-004a2587f964_3200x1253.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!mvtA!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f6b0ce-5b19-4820-b2ba-004a2587f964_3200x1253.png 424w, https://substackcdn.com/image/fetch/$s_!mvtA!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f6b0ce-5b19-4820-b2ba-004a2587f964_3200x1253.png 848w, https://substackcdn.com/image/fetch/$s_!mvtA!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f6b0ce-5b19-4820-b2ba-004a2587f964_3200x1253.png 1272w, https://substackcdn.com/image/fetch/$s_!mvtA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F53f6b0ce-5b19-4820-b2ba-004a2587f964_3200x1253.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The CLI version check and skill export for Claude Code and Codex.</figcaption></figure></div><div><hr></div><h4>Step 2: An agent-written schema</h4><p>To demonstrate the usage of Omnigraph, we created shared data, which is a small project graph seeded with two components, <code>sessions</code> and <code>billing</code>.</p><p>Since it has Omnigraph skill, Claude Code wrote its schema and seed from one prompt, then created the graph on <code>main</code>, as shown below:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!7rCZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d3c257c-f9a7-4332-8741-171021526979_3200x1820.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!7rCZ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d3c257c-f9a7-4332-8741-171021526979_3200x1820.png 424w, https://substackcdn.com/image/fetch/$s_!7rCZ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d3c257c-f9a7-4332-8741-171021526979_3200x1820.png 848w, https://substackcdn.com/image/fetch/$s_!7rCZ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d3c257c-f9a7-4332-8741-171021526979_3200x1820.png 1272w, https://substackcdn.com/image/fetch/$s_!7rCZ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d3c257c-f9a7-4332-8741-171021526979_3200x1820.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!7rCZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d3c257c-f9a7-4332-8741-171021526979_3200x1820.png" width="1456" height="828" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2d3c257c-f9a7-4332-8741-171021526979_3200x1820.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:828,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:135275,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224646450?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d3c257c-f9a7-4332-8741-171021526979_3200x1820.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!7rCZ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d3c257c-f9a7-4332-8741-171021526979_3200x1820.png 424w, https://substackcdn.com/image/fetch/$s_!7rCZ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d3c257c-f9a7-4332-8741-171021526979_3200x1820.png 848w, https://substackcdn.com/image/fetch/$s_!7rCZ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d3c257c-f9a7-4332-8741-171021526979_3200x1820.png 1272w, https://substackcdn.com/image/fetch/$s_!7rCZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d3c257c-f9a7-4332-8741-171021526979_3200x1820.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Claude Code writes the schema and seed from one prompt, then creates the graph on main.</figcaption></figure></div><p>The only detail worth knowing is <code>@key</code> on each <code>slug</code>, which gives every record a stable ID that merges compare across branches:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!dVvQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb20ea6f4-72bb-4071-ba1a-867084775ef3_3200x2450.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!dVvQ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb20ea6f4-72bb-4071-ba1a-867084775ef3_3200x2450.png 424w, https://substackcdn.com/image/fetch/$s_!dVvQ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb20ea6f4-72bb-4071-ba1a-867084775ef3_3200x2450.png 848w, https://substackcdn.com/image/fetch/$s_!dVvQ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb20ea6f4-72bb-4071-ba1a-867084775ef3_3200x2450.png 1272w, https://substackcdn.com/image/fetch/$s_!dVvQ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb20ea6f4-72bb-4071-ba1a-867084775ef3_3200x2450.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!dVvQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb20ea6f4-72bb-4071-ba1a-867084775ef3_3200x2450.png" width="1456" height="1115" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b20ea6f4-72bb-4071-ba1a-867084775ef3_3200x2450.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1115,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:80336,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224646450?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb20ea6f4-72bb-4071-ba1a-867084775ef3_3200x2450.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!dVvQ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb20ea6f4-72bb-4071-ba1a-867084775ef3_3200x2450.png 424w, https://substackcdn.com/image/fetch/$s_!dVvQ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb20ea6f4-72bb-4071-ba1a-867084775ef3_3200x2450.png 848w, https://substackcdn.com/image/fetch/$s_!dVvQ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb20ea6f4-72bb-4071-ba1a-867084775ef3_3200x2450.png 1272w, https://substackcdn.com/image/fetch/$s_!dVvQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb20ea6f4-72bb-4071-ba1a-867084775ef3_3200x2450.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The schema marks slug as the key on Component, Decision, and Issue nodes.</figcaption></figure></div><div><hr></div><h4>Step 3: Claude Code and Codex on separate branches</h4><p>Here, we gave Claude Code a multi-step task.</p><p>We asked Claude Code to record that <code>sessions</code> moved from Redis to Postgres.</p><p>To do that, it added a new decision, linked it to the component and the open issue, marked the old decision as superseded, changed the store to Postgres, and closed the issue. All six writes landed as one commit on its own branch, <code>claude</code>, as shown below:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!pRwH!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F920c5dd6-8467-4cd7-b780-69b429b49f35_3200x1820.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!pRwH!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F920c5dd6-8467-4cd7-b780-69b429b49f35_3200x1820.png 424w, https://substackcdn.com/image/fetch/$s_!pRwH!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F920c5dd6-8467-4cd7-b780-69b429b49f35_3200x1820.png 848w, https://substackcdn.com/image/fetch/$s_!pRwH!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F920c5dd6-8467-4cd7-b780-69b429b49f35_3200x1820.png 1272w, https://substackcdn.com/image/fetch/$s_!pRwH!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F920c5dd6-8467-4cd7-b780-69b429b49f35_3200x1820.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!pRwH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F920c5dd6-8467-4cd7-b780-69b429b49f35_3200x1820.png" width="1456" height="828" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/920c5dd6-8467-4cd7-b780-69b429b49f35_3200x1820.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:828,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:133691,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224646450?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F920c5dd6-8467-4cd7-b780-69b429b49f35_3200x1820.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!pRwH!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F920c5dd6-8467-4cd7-b780-69b429b49f35_3200x1820.png 424w, https://substackcdn.com/image/fetch/$s_!pRwH!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F920c5dd6-8467-4cd7-b780-69b429b49f35_3200x1820.png 848w, https://substackcdn.com/image/fetch/$s_!pRwH!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F920c5dd6-8467-4cd7-b780-69b429b49f35_3200x1820.png 1272w, https://substackcdn.com/image/fetch/$s_!pRwH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F920c5dd6-8467-4cd7-b780-69b429b49f35_3200x1820.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Claude Code writes six changes as one mutation on the claude branch.</figcaption></figure></div><p>At the same time, we asked Codex to set the owner of <code>sessions</code> to <code>platform-team</code>. This was a one-field change, and Codex made it on its own branch, <code>codex</code>, as shown below:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!2W79!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7933974-40c3-4db4-8e34-73ea9cf15d2c_3200x1379.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!2W79!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7933974-40c3-4db4-8e34-73ea9cf15d2c_3200x1379.png 424w, https://substackcdn.com/image/fetch/$s_!2W79!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7933974-40c3-4db4-8e34-73ea9cf15d2c_3200x1379.png 848w, https://substackcdn.com/image/fetch/$s_!2W79!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7933974-40c3-4db4-8e34-73ea9cf15d2c_3200x1379.png 1272w, https://substackcdn.com/image/fetch/$s_!2W79!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7933974-40c3-4db4-8e34-73ea9cf15d2c_3200x1379.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!2W79!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7933974-40c3-4db4-8e34-73ea9cf15d2c_3200x1379.png" width="1456" height="627" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d7933974-40c3-4db4-8e34-73ea9cf15d2c_3200x1379.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:627,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:91757,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224646450?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7933974-40c3-4db4-8e34-73ea9cf15d2c_3200x1379.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!2W79!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7933974-40c3-4db4-8e34-73ea9cf15d2c_3200x1379.png 424w, https://substackcdn.com/image/fetch/$s_!2W79!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7933974-40c3-4db4-8e34-73ea9cf15d2c_3200x1379.png 848w, https://substackcdn.com/image/fetch/$s_!2W79!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7933974-40c3-4db4-8e34-73ea9cf15d2c_3200x1379.png 1272w, https://substackcdn.com/image/fetch/$s_!2W79!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7933974-40c3-4db4-8e34-73ea9cf15d2c_3200x1379.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Codex sets the sessions owner on its own branch, which still reads store redis.</figcaption></figure></div><p>Both branches started from the same commit of <code>main</code>, so Codex&#8217;s branch still reads <code>store: redis</code>.</p><div><hr></div><h4>Step 4: Diff review and a fast-forward merge</h4><p>Before Claude Code&#8217;s branch goes into <code>main</code>, we can review exactly what it changed.</p><p>Omnigraph compares the branch with the commit both branches started from, so the diff shows only Claude Code&#8217;s six writes and nothing else, as shown below:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!XTjn!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49300b5a-7db8-47c0-bbe0-3ff14387289f_3200x1631.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!XTjn!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49300b5a-7db8-47c0-bbe0-3ff14387289f_3200x1631.png 424w, https://substackcdn.com/image/fetch/$s_!XTjn!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49300b5a-7db8-47c0-bbe0-3ff14387289f_3200x1631.png 848w, https://substackcdn.com/image/fetch/$s_!XTjn!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49300b5a-7db8-47c0-bbe0-3ff14387289f_3200x1631.png 1272w, https://substackcdn.com/image/fetch/$s_!XTjn!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49300b5a-7db8-47c0-bbe0-3ff14387289f_3200x1631.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!XTjn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49300b5a-7db8-47c0-bbe0-3ff14387289f_3200x1631.png" width="1456" height="742" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/49300b5a-7db8-47c0-bbe0-3ff14387289f_3200x1631.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:742,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:74627,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224646450?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49300b5a-7db8-47c0-bbe0-3ff14387289f_3200x1631.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!XTjn!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49300b5a-7db8-47c0-bbe0-3ff14387289f_3200x1631.png 424w, https://substackcdn.com/image/fetch/$s_!XTjn!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49300b5a-7db8-47c0-bbe0-3ff14387289f_3200x1631.png 848w, https://substackcdn.com/image/fetch/$s_!XTjn!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49300b5a-7db8-47c0-bbe0-3ff14387289f_3200x1631.png 1272w, https://substackcdn.com/image/fetch/$s_!XTjn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49300b5a-7db8-47c0-bbe0-3ff14387289f_3200x1631.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The diff of Claude Code&#8217;s branch against the merge base, then a fast-forward merge into main.</figcaption></figure></div><p>Since nobody had changed <code>main</code> after the branch was created, the merge is a fast-forward. That means <code>main</code> simply moves up to Claude Code&#8217;s commit, so all six writes are pushed together.</p><p>This is the part a regular database doesn&#8217;t give you. A row lock protects one record at a time, but it can&#8217;t show you a multi-step change as a single diff or let you approve it before it goes live.</p><div><hr></div><h4>Step 5: The merge check refuses the stale branch</h4><p>Next, we tried to merge Codex&#8217;s branch. Its diff shows only the owner change, but the merge still fails, as shown below:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!yp4e!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ca1296-3220-4ef1-b3e2-50626e052d17_3200x938.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!yp4e!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ca1296-3220-4ef1-b3e2-50626e052d17_3200x938.png 424w, https://substackcdn.com/image/fetch/$s_!yp4e!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ca1296-3220-4ef1-b3e2-50626e052d17_3200x938.png 848w, https://substackcdn.com/image/fetch/$s_!yp4e!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ca1296-3220-4ef1-b3e2-50626e052d17_3200x938.png 1272w, https://substackcdn.com/image/fetch/$s_!yp4e!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ca1296-3220-4ef1-b3e2-50626e052d17_3200x938.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!yp4e!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ca1296-3220-4ef1-b3e2-50626e052d17_3200x938.png" width="1456" height="427" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/96ca1296-3220-4ef1-b3e2-50626e052d17_3200x938.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:427,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:47587,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224646450?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ca1296-3220-4ef1-b3e2-50626e052d17_3200x938.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!yp4e!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ca1296-3220-4ef1-b3e2-50626e052d17_3200x938.png 424w, https://substackcdn.com/image/fetch/$s_!yp4e!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ca1296-3220-4ef1-b3e2-50626e052d17_3200x938.png 848w, https://substackcdn.com/image/fetch/$s_!yp4e!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ca1296-3220-4ef1-b3e2-50626e052d17_3200x938.png 1272w, https://substackcdn.com/image/fetch/$s_!yp4e!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ca1296-3220-4ef1-b3e2-50626e052d17_3200x938.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The merge of the codex branch is refused with divergent_update, and nothing is published.</figcaption></figure></div><p>The merge fails because both <code>main</code> and the <code>codex</code> branch changed <code>sessions</code> after the two branches were created. Omnigraph reports this as a <code>divergent_update</code>, publishes nothing, and tells Codex to refresh its branch, as shown below:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!37gu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c97fea6-100e-4bc5-9223-18922d3c7fa8_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!37gu!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c97fea6-100e-4bc5-9223-18922d3c7fa8_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!37gu!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c97fea6-100e-4bc5-9223-18922d3c7fa8_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!37gu!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c97fea6-100e-4bc5-9223-18922d3c7fa8_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!37gu!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c97fea6-100e-4bc5-9223-18922d3c7fa8_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!37gu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c97fea6-100e-4bc5-9223-18922d3c7fa8_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6c97fea6-100e-4bc5-9223-18922d3c7fa8_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:122426,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224646450?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c97fea6-100e-4bc5-9223-18922d3c7fa8_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!37gu!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c97fea6-100e-4bc5-9223-18922d3c7fa8_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!37gu!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c97fea6-100e-4bc5-9223-18922d3c7fa8_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!37gu!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c97fea6-100e-4bc5-9223-18922d3c7fa8_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!37gu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c97fea6-100e-4bc5-9223-18922d3c7fa8_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The merge base shows that main and the codex branch both changed sessions.</figcaption></figure></div><p>With no version check, a full-record save from Codex would have put <code>sessions</code> back on Redis, even though decision <code>dec-12</code> says it moved to Postgres.</p><p>Omnigraph checks conflicts per record, not per field, so it refused this merge even though the two agents changed different fields. A false alarm costs one retry, while a stale merge that goes through quietly leaves a wrong fact in memory.</p><div><hr></div><h4>Step 6: Refresh, retry and snapshot reads</h4><p>Next, we sent the merge result back to the same Codex session. Codex read the latest <code>main</code>, opened a new branch called <code>codex-2</code>, and made its change again, as shown below:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!_cu0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2913239-5742-4623-9612-c7537a012af5_3200x1442.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_cu0!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2913239-5742-4623-9612-c7537a012af5_3200x1442.png 424w, https://substackcdn.com/image/fetch/$s_!_cu0!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2913239-5742-4623-9612-c7537a012af5_3200x1442.png 848w, https://substackcdn.com/image/fetch/$s_!_cu0!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2913239-5742-4623-9612-c7537a012af5_3200x1442.png 1272w, https://substackcdn.com/image/fetch/$s_!_cu0!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2913239-5742-4623-9612-c7537a012af5_3200x1442.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!_cu0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2913239-5742-4623-9612-c7537a012af5_3200x1442.png" width="1456" height="656" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b2913239-5742-4623-9612-c7537a012af5_3200x1442.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:656,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:101196,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224646450?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2913239-5742-4623-9612-c7537a012af5_3200x1442.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!_cu0!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2913239-5742-4623-9612-c7537a012af5_3200x1442.png 424w, https://substackcdn.com/image/fetch/$s_!_cu0!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2913239-5742-4623-9612-c7537a012af5_3200x1442.png 848w, https://substackcdn.com/image/fetch/$s_!_cu0!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2913239-5742-4623-9612-c7537a012af5_3200x1442.png 1272w, https://substackcdn.com/image/fetch/$s_!_cu0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2913239-5742-4623-9612-c7537a012af5_3200x1442.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Codex refreshes from the latest main and redoes its change on branch codex-2.</figcaption></figure></div><p>This time, the merge is a fast-forward, and <code>main</code> keeps both agents&#8217; work. Old versions also stay readable. A query with <code>--snapshot</code> set to the original commit still returns Redis and no owner, as shown below:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ZxYb!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ca3401d-e4f6-40ed-915a-29229066d5e0_3200x1568.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ZxYb!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ca3401d-e4f6-40ed-915a-29229066d5e0_3200x1568.png 424w, https://substackcdn.com/image/fetch/$s_!ZxYb!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ca3401d-e4f6-40ed-915a-29229066d5e0_3200x1568.png 848w, https://substackcdn.com/image/fetch/$s_!ZxYb!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ca3401d-e4f6-40ed-915a-29229066d5e0_3200x1568.png 1272w, https://substackcdn.com/image/fetch/$s_!ZxYb!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ca3401d-e4f6-40ed-915a-29229066d5e0_3200x1568.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ZxYb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ca3401d-e4f6-40ed-915a-29229066d5e0_3200x1568.png" width="1456" height="713" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2ca3401d-e4f6-40ed-915a-29229066d5e0_3200x1568.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:713,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:105195,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224646450?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ca3401d-e4f6-40ed-915a-29229066d5e0_3200x1568.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ZxYb!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ca3401d-e4f6-40ed-915a-29229066d5e0_3200x1568.png 424w, https://substackcdn.com/image/fetch/$s_!ZxYb!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ca3401d-e4f6-40ed-915a-29229066d5e0_3200x1568.png 848w, https://substackcdn.com/image/fetch/$s_!ZxYb!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ca3401d-e4f6-40ed-915a-29229066d5e0_3200x1568.png 1272w, https://substackcdn.com/image/fetch/$s_!ZxYb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ca3401d-e4f6-40ed-915a-29229066d5e0_3200x1568.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The merge of codex-2, the final state, the commit history, and a snapshot query returning the original Redis state.</figcaption></figure></div><div><hr></div><h4>Step 7: Only a reviewer can merge into main</h4><p>Up to here, an agent could still merge its own branch into <code>main</code> without anyone checking it. Locally, nothing stops that. The <code>--as</code> flag only names who made a change, and any agent can type any name.</p><p>To lock <code>main</code>, we ran Omnigraph as a server, <code>omnigraph-server</code>, and gave each agent and the reviewer their own access token. With tokens in place, the server knows who is making each request.</p><p>Next, we added a policy. Omnigraph writes its policies in Cedar, an open-source language for writing access rules. Our rule says that agents can write on their own branches, and only the reviewer can merge into <code>main</code>.</p><p>Then we tested it. Claude Code&#8217;s token could still write on its branch, but its attempts to write to <code>main</code> or merge into it came back as HTTP 403, which means access denied, as shown below:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!QMW1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e0e9d7d-0d7f-4095-ad5a-6785031d6e1d_3200x2135.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!QMW1!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e0e9d7d-0d7f-4095-ad5a-6785031d6e1d_3200x2135.png 424w, https://substackcdn.com/image/fetch/$s_!QMW1!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e0e9d7d-0d7f-4095-ad5a-6785031d6e1d_3200x2135.png 848w, https://substackcdn.com/image/fetch/$s_!QMW1!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e0e9d7d-0d7f-4095-ad5a-6785031d6e1d_3200x2135.png 1272w, https://substackcdn.com/image/fetch/$s_!QMW1!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e0e9d7d-0d7f-4095-ad5a-6785031d6e1d_3200x2135.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!QMW1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e0e9d7d-0d7f-4095-ad5a-6785031d6e1d_3200x2135.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1e0e9d7d-0d7f-4095-ad5a-6785031d6e1d_3200x2135.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:117548,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224646450?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e0e9d7d-0d7f-4095-ad5a-6785031d6e1d_3200x2135.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!QMW1!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e0e9d7d-0d7f-4095-ad5a-6785031d6e1d_3200x2135.png 424w, https://substackcdn.com/image/fetch/$s_!QMW1!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e0e9d7d-0d7f-4095-ad5a-6785031d6e1d_3200x2135.png 848w, https://substackcdn.com/image/fetch/$s_!QMW1!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e0e9d7d-0d7f-4095-ad5a-6785031d6e1d_3200x2135.png 1272w, https://substackcdn.com/image/fetch/$s_!QMW1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1e0e9d7d-0d7f-4095-ad5a-6785031d6e1d_3200x2135.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">On the server, the policy denies Claude Code&#8217;s write and merge on main, allows its branch, and the reviewer merges.</figcaption></figure></div><p>The reviewer&#8217;s token then merged the branch. So agents still work in parallel, but nothing reaches <code>main</code> until a person approves it.</p><div><hr></div><p>What we ran here was local, with one developer and two agents.</p><p>The same race happens at a much larger scale. Picture a company where many people and their agents read and write one shared database, such as a support bot, a coding agent, and an analyst&#8217;s agent all updating the same customer records.</p><p>There, one stale write can spread wrong facts into every tool that reads that data.</p><p>Omnigraph&#8217;s model holds up at that scale too. Every person or agent works on its own branch, each change shows up as a diff someone can review, and only merges checked against the latest <code>main</code> go live.</p><p><strong><a href="https://github.com/ModernRelay/omnigraph">You can find the Omnigraph GitHub repo here &#8594;</a></strong></p><p>(don&#8217;t forget to star it &#11088;)</p><p>&#128073; Over to you: How do your agents handle two writes to the same record today?</p><p>Good day!</p>]]></content:encoded></item><item><title><![CDATA[Find Production Agent Failures Before Users Do]]></title><description><![CDATA[...explained with a hands-on walkthrough!]]></description><link>https://blog.dailydoseofds.com/p/find-production-agent-failures-before</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/find-production-agent-failures-before</guid><dc:creator><![CDATA[Avi Chawla]]></dc:creator><pubDate>Wed, 11 Nov 2026 23:33:57 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!fd04!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F365e2adb-10d9-4795-9626-a249a4c75965_2752x1536.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><strong>In today&#8217;s newsletter:</strong></p><ul><li><p>Benchmark the harness, not just the model.</p></li><li><p>Find production agent failures before users do.</p></li></ul><div><hr></div><h3><a href="https://github.com/HarnessRouter/harnessrouter">Benchmark the harness, not just the model</a></h3><p>We benchmark models all the time. In production, that model runs inside a harness, and the harness can change the result, the cost, and the runtime.</p><p>Devs just open-sourced <strong><a href="https://github.com/HarnessRouter/harnessrouter">Harness Arena</a></strong>, inside the <strong><a href="https://github.com/HarnessRouter/harnessrouter">HarnessRouter</a></strong> project. It runs one task across several harness and model combinations, side by side.</p><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;42d30fee-7c25-41f3-bab5-671826ca0fa7&quot;,&quot;duration&quot;:null}"></div><p>Every config gets the same prompt, files, tools, skills, and evaluation checks. Arena then returns:</p><ul><li><p>Total cost</p></li><li><p>Execution time</p></li><li><p>Whether the task passed</p></li><li><p>The files and artifacts it produced</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!SncI!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde6d1873-7e1e-463b-871b-682ac6cf069e_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!SncI!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde6d1873-7e1e-463b-871b-682ac6cf069e_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!SncI!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde6d1873-7e1e-463b-871b-682ac6cf069e_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!SncI!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde6d1873-7e1e-463b-871b-682ac6cf069e_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!SncI!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde6d1873-7e1e-463b-871b-682ac6cf069e_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!SncI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde6d1873-7e1e-463b-871b-682ac6cf069e_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/de6d1873-7e1e-463b-871b-682ac6cf069e_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:176809,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224522564?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde6d1873-7e1e-463b-871b-682ac6cf069e_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!SncI!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde6d1873-7e1e-463b-871b-682ac6cf069e_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!SncI!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde6d1873-7e1e-463b-871b-682ac6cf069e_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!SncI!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde6d1873-7e1e-463b-871b-682ac6cf069e_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!SncI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde6d1873-7e1e-463b-871b-682ac6cf069e_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Every config gets the same inputs and checks, so the runtime is the only thing that changes.</figcaption></figure></div><p>HarnessRouter is a plug-and-play layer that lets you run any harness under one interface, including Codex, Hermes, Claude Code, DeepSeek Harness, System One (powered by Jev), and 25 more.</p><p>The cheapest run does not automatically win. A config has to complete the task correctly first. Cost and execution time are compared only among the configurations that pass.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!fd04!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F365e2adb-10d9-4795-9626-a249a4c75965_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!fd04!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F365e2adb-10d9-4795-9626-a249a4c75965_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!fd04!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F365e2adb-10d9-4795-9626-a249a4c75965_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!fd04!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F365e2adb-10d9-4795-9626-a249a4c75965_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!fd04!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F365e2adb-10d9-4795-9626-a249a4c75965_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!fd04!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F365e2adb-10d9-4795-9626-a249a4c75965_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/365e2adb-10d9-4795-9626-a249a4c75965_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:156120,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224522564?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F365e2adb-10d9-4795-9626-a249a4c75965_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!fd04!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F365e2adb-10d9-4795-9626-a249a4c75965_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!fd04!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F365e2adb-10d9-4795-9626-a249a4c75965_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!fd04!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F365e2adb-10d9-4795-9626-a249a4c75965_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!fd04!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F365e2adb-10d9-4795-9626-a249a4c75965_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A cheap run that fails the task is not a bargain, so correctness comes first.</figcaption></figure></div><p>In one Harness Arena test:</p><ul><li><p>Hermes + GPT-5.2: 5/5 successful runs using 0.47 credits</p></li><li><p>Claude Code + Opus 4.8: 3/5 successful runs using 223 credits</p></li></ul><p>That is around a 474x difference in credits, and the more expensive configuration also failed two runs.</p><p>You can keep the model fixed to measure how much the harness changes the result, or vary both to find the best pairing for that task.</p><p>Once you pick a config, later tasks run through the same Unified Harness Protocol (UHP) interface for sessions, streaming, files, artifacts, cancellation, and failures.</p><p>So you are no longer choosing an agent stack from a model leaderboard alone. You can benchmark the complete runtime on the task your product actually needs to solve.</p><p><strong><a href="https://github.com/HarnessRouter/harnessrouter">Explore the HarnessRouter repo &#8594;</a></strong></p><p>(don&#8217;t forget to star it &#11088;)</p><div><hr></div><h3><a href="https://simulation.plurai.ai/connect">Find production agent failures before users do</a></h3><p>There&#8217;s a new system that can test and evaluate your agents against the real world, then write the fixes, all before you deploy them.</p><p>AI agents look great in demos, until real users start breaking them. Your agent can tell a customer &#8220;refund done&#8221; on a $1,080 invoice without ever checking who she is, and the logs will still look clean.</p><p>Normally, you only find a failure like that after a real customer runs into it. </p><p><strong><a href="https://simulation.plurai.ai/connect">PlurAI Simulation</a></strong> lets your agent fail in a simulated world first, so you can fix the problem before any user sees it.</p><p>Here&#8217;s the whole system, step by step, from the first simulated user to a re-tested fix:</p><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;86644840-936a-4f5c-a5c1-e0c9533b3101&quot;,&quot;duration&quot;:null}"></div><ul><li><p>It rebuilds your actual agent, its tools, and its environment from your code</p></li><li><p>Simulated users push the agent into the edge cases that matter for your use case</p></li><li><p>Evals built for your agent check the actions it took, like whether it verified the customer before issuing that refund</p></li><li><p>Every failure comes with the run that exposed it and a fix you can verify</p></li><li><p>Any production failure can become a simulation you replay against every new version</p></li></ul><p>It ships as a single plugin for coding agents like Claude Code and Codex, and it works with whatever agent you&#8217;re building. I&#8217;ve started running it on my own agents too.</p><p><strong><a href="https://simulation.plurai.ai/connect">It&#8217;s free for every AI engineer, and you can find it here &#8594;</a></strong></p><p>&#128073; Over to you: Would you rather find the next agent failure in simulation, or in a customer ticket?</p><p>Thanks to the Plurai team for working with us on this.</p><p>Good day!</p>]]></content:encoded></item><item><title><![CDATA[16 Policy Optimization Methods for LLM Training]]></title><description><![CDATA[...explained visually, plus how hybrid LLMs shrink the KV cache.]]></description><link>https://blog.dailydoseofds.com/p/16-policy-optimization-methods-for</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/16-policy-optimization-methods-for</guid><dc:creator><![CDATA[Avi Chawla]]></dc:creator><pubDate>Wed, 11 Nov 2026 00:07:01 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!W6LW!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc70904f3-7eb3-48e9-8957-93caf292fb80_1024x854.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><strong>In today&#8217;s newsletter:</strong></p><ul><li><p>Build voice agents that hear the details&#8203;.</p></li><li><p>16 policy optimization methods for LLM training.</p></li><li><p>How hybrid LLMs shrink the KV cache, explained visually.</p></li></ul><div><hr></div><h3><a href="https://www.speechmatics.com/voice-agents">Build voice agents that hear the details&#8203;</a></h3><p><a href="https://www.speechmatics.com/voice-agents">&#8203;</a><strong><a href="https://www.speechmatics.com/voice-agents">Speechmatics Linden</a></strong>&nbsp;is a speech-to-text model built specifically for voice agents. It focuses on the inputs that tend to cause expensive mistakes: strong accents, background noise, non-native speech, names, and alphanumeric strings such as booking references.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!nc5P!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc88d54-4dd1-4937-9a5a-c2d172e7f5bf_1753x964.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!nc5P!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc88d54-4dd1-4937-9a5a-c2d172e7f5bf_1753x964.png 424w, https://substackcdn.com/image/fetch/$s_!nc5P!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc88d54-4dd1-4937-9a5a-c2d172e7f5bf_1753x964.png 848w, https://substackcdn.com/image/fetch/$s_!nc5P!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc88d54-4dd1-4937-9a5a-c2d172e7f5bf_1753x964.png 1272w, https://substackcdn.com/image/fetch/$s_!nc5P!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc88d54-4dd1-4937-9a5a-c2d172e7f5bf_1753x964.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!nc5P!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc88d54-4dd1-4937-9a5a-c2d172e7f5bf_1753x964.png" width="1456" height="801" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bbc88d54-4dd1-4937-9a5a-c2d172e7f5bf_1753x964.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:801,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!nc5P!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc88d54-4dd1-4937-9a5a-c2d172e7f5bf_1753x964.png 424w, https://substackcdn.com/image/fetch/$s_!nc5P!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc88d54-4dd1-4937-9a5a-c2d172e7f5bf_1753x964.png 848w, https://substackcdn.com/image/fetch/$s_!nc5P!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc88d54-4dd1-4937-9a5a-c2d172e7f5bf_1753x964.png 1272w, https://substackcdn.com/image/fetch/$s_!nc5P!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbc88d54-4dd1-4937-9a5a-c2d172e7f5bf_1753x964.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><ul><li><p>Real-time transcription across 55+ languages</p></li><li><p>Final transcripts in roughly 250 ms after a <code>ForceEndOfUtterance</code> signal</p></li><li><p>Custom dictionaries for names, product terms, and industry vocabulary</p></li><li><p>Native integrations for LiveKit, Pipecat, Jambonz, and the Speechmatics API</p></li><li><p>Linden 1 costs $0.30 per hour for new users, with volume discounts bringing the price down to $0.16 per hour</p></li></ul><p>New accounts receive $100 in credit with no card required, which covers roughly 333 hours of Linden 1 transcription at the standard $0.30 per-hour rate.</p><p><a href="https://portal.speechmatics.com/signup/">&#8203;</a><strong><a href="https://portal.speechmatics.com/signup/">Try Speechmatics here &#8594;</a></strong><a href="https://portal.speechmatics.com/signup/">&#8203;</a></p><p><em>Thanks to Speechmatics for partnering today!</em></p><div><hr></div><h3><a href="https://www.dailydoseofds.com/rl-course-part-1/">16 policy optimization methods for LLM training</a></h3><p>The fastest way to make LLM RL cheaper is to delete a model.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!gsVv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9ad4476-b4cd-427f-8b4e-ce9bad22c285_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!gsVv!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9ad4476-b4cd-427f-8b4e-ce9bad22c285_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!gsVv!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9ad4476-b4cd-427f-8b4e-ce9bad22c285_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!gsVv!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9ad4476-b4cd-427f-8b4e-ce9bad22c285_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!gsVv!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9ad4476-b4cd-427f-8b4e-ce9bad22c285_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!gsVv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9ad4476-b4cd-427f-8b4e-ce9bad22c285_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b9ad4476-b4cd-427f-8b4e-ce9bad22c285_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:146472,&quot;alt&quot;:&quot;Every new RL method for LLMs edits the baseline, the clip, or both.&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224378592?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9ad4476-b4cd-427f-8b4e-ce9bad22c285_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Every new RL method for LLMs edits the baseline, the clip, or both." title="Every new RL method for LLMs edits the baseline, the clip, or both." srcset="https://substackcdn.com/image/fetch/$s_!gsVv!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9ad4476-b4cd-427f-8b4e-ce9bad22c285_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!gsVv!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9ad4476-b4cd-427f-8b4e-ce9bad22c285_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!gsVv!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9ad4476-b4cd-427f-8b4e-ce9bad22c285_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!gsVv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9ad4476-b4cd-427f-8b4e-ce9bad22c285_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Every new RL method for LLMs edits the baseline, the clip, or both.</figcaption></figure></div><p>ReMax did exactly that in 2023. It dropped PPO&#8217;s value model and cut GPU memory by about 46% when training a 7B model.</p><p>Most of the methods below make a similar trade. They remove a component, or they change how far a single update can move the policy.</p><p>The visual below maps 16 of them.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!W6LW!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc70904f3-7eb3-48e9-8957-93caf292fb80_1024x854.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!W6LW!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc70904f3-7eb3-48e9-8957-93caf292fb80_1024x854.jpeg 424w, https://substackcdn.com/image/fetch/$s_!W6LW!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc70904f3-7eb3-48e9-8957-93caf292fb80_1024x854.jpeg 848w, https://substackcdn.com/image/fetch/$s_!W6LW!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc70904f3-7eb3-48e9-8957-93caf292fb80_1024x854.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!W6LW!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc70904f3-7eb3-48e9-8957-93caf292fb80_1024x854.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!W6LW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc70904f3-7eb3-48e9-8957-93caf292fb80_1024x854.jpeg" width="1024" height="854" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c70904f3-7eb3-48e9-8957-93caf292fb80_1024x854.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:854,&quot;width&quot;:1024,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:106269,&quot;alt&quot;:&quot;Sixteen policy optimization methods, each summarized by the one thing it changes.&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224378592?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc70904f3-7eb3-48e9-8957-93caf292fb80_1024x854.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Sixteen policy optimization methods, each summarized by the one thing it changes." title="Sixteen policy optimization methods, each summarized by the one thing it changes." srcset="https://substackcdn.com/image/fetch/$s_!W6LW!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc70904f3-7eb3-48e9-8957-93caf292fb80_1024x854.jpeg 424w, https://substackcdn.com/image/fetch/$s_!W6LW!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc70904f3-7eb3-48e9-8957-93caf292fb80_1024x854.jpeg 848w, https://substackcdn.com/image/fetch/$s_!W6LW!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc70904f3-7eb3-48e9-8957-93caf292fb80_1024x854.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!W6LW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc70904f3-7eb3-48e9-8957-93caf292fb80_1024x854.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Sixteen policy optimization methods, each summarized by the one thing it changes.</figcaption></figure></div><p>Under the hood, all 16 run the same loop. We sample responses from the current policy and score them with a reward model or a verifier. Then we compute an advantage, which says how much better a response was than expected, and update the policy.</p><p>The update weights each token by an importance ratio, the new policy&#8217;s probability divided by the old one&#8217;s. PPO-style methods clip that ratio so one batch can&#8217;t drag the policy too far.</p><p>So every method answers two questions. What do we compare the reward against, and what do we do with ratios that drift too far?</p><p>Grouping them by those answers gives four families.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!fINW!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F024cfedd-0fec-49c4-a758-119fef04c0fc_2752x1406.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!fINW!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F024cfedd-0fec-49c4-a758-119fef04c0fc_2752x1406.jpeg 424w, https://substackcdn.com/image/fetch/$s_!fINW!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F024cfedd-0fec-49c4-a758-119fef04c0fc_2752x1406.jpeg 848w, https://substackcdn.com/image/fetch/$s_!fINW!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F024cfedd-0fec-49c4-a758-119fef04c0fc_2752x1406.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!fINW!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F024cfedd-0fec-49c4-a758-119fef04c0fc_2752x1406.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!fINW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F024cfedd-0fec-49c4-a758-119fef04c0fc_2752x1406.jpeg" width="1456" height="744" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/024cfedd-0fec-49c4-a758-119fef04c0fc_2752x1406.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:744,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:179349,&quot;alt&quot;:&quot;Pick the family first. The method inside it is a tuning detail.&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224378592?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F024cfedd-0fec-49c4-a758-119fef04c0fc_2752x1406.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Pick the family first. The method inside it is a tuning detail." title="Pick the family first. The method inside it is a tuning detail." srcset="https://substackcdn.com/image/fetch/$s_!fINW!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F024cfedd-0fec-49c4-a758-119fef04c0fc_2752x1406.jpeg 424w, https://substackcdn.com/image/fetch/$s_!fINW!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F024cfedd-0fec-49c4-a758-119fef04c0fc_2752x1406.jpeg 848w, https://substackcdn.com/image/fetch/$s_!fINW!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F024cfedd-0fec-49c4-a758-119fef04c0fc_2752x1406.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!fINW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F024cfedd-0fec-49c4-a758-119fef04c0fc_2752x1406.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Pick the family first. The method inside it is a tuning detail.</figcaption></figure></div><h4>Value-based credit: PPO, VAPO, VinePPO</h4><p><strong>PPO</strong> clips the token ratio, typically to the range 0.8 to 1.2. In RLHF, it also trains a critic that predicts the expected reward at every token, and GAE turns those predictions into per-token advantages.</p><p>The critic is expensive and it isn&#8217;t always accurate. The VinePPO authors found that value networks on reasoning tasks often estimate returns poorly and barely beat a random baseline when ranking alternative steps.</p><p><strong>VAPO</strong> keeps the critic and repairs it for long chains of thought. It pretrains the value model against a fixed policy, decouples GAE between the actor and the critic, and adapts GAE&#8217;s &#955; to response length. Built on a Qwen 32B base model, it reported 60.4 on AIME 2024 within 5,000 steps.</p><p><strong>VinePPO</strong> removes the value network instead. A language environment can restart from any partial answer by re-feeding the prefix, so VinePPO runs extra rollouts from intermediate steps and averages their rewards. That gives an unbiased Monte Carlo value for each step.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!hLrB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b432f5a-6e2c-43a2-80e5-8c621c26e712_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!hLrB!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b432f5a-6e2c-43a2-80e5-8c621c26e712_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!hLrB!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b432f5a-6e2c-43a2-80e5-8c621c26e712_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!hLrB!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b432f5a-6e2c-43a2-80e5-8c621c26e712_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!hLrB!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b432f5a-6e2c-43a2-80e5-8c621c26e712_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!hLrB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b432f5a-6e2c-43a2-80e5-8c621c26e712_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3b432f5a-6e2c-43a2-80e5-8c621c26e712_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:176301,&quot;alt&quot;:&quot;Per-step credit costs either an extra model or extra rollouts.&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224378592?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b432f5a-6e2c-43a2-80e5-8c621c26e712_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Per-step credit costs either an extra model or extra rollouts." title="Per-step credit costs either an extra model or extra rollouts." srcset="https://substackcdn.com/image/fetch/$s_!hLrB!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b432f5a-6e2c-43a2-80e5-8c621c26e712_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!hLrB!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b432f5a-6e2c-43a2-80e5-8c621c26e712_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!hLrB!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b432f5a-6e2c-43a2-80e5-8c621c26e712_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!hLrB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b432f5a-6e2c-43a2-80e5-8c621c26e712_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Per-step credit costs either an extra model or extra rollouts.</figcaption></figure></div><p>On MATH and GSM8K, VinePPO beat PPO in up to 3x less wall-clock time.</p><h4>REINFORCE-style baselines: REINFORCE, RLOO, ReMax, REINFORCE++, OPO</h4><p>These methods skip the critic and treat the whole response as one action with one reward. The design question becomes what to subtract from that reward.</p><p><strong>REINFORCE</strong> is the plain policy gradient from 1992. It raises the log-probability of a sampled response in proportion to its reward, optionally minus a baseline.</p><p><strong>RLOO</strong> samples k responses per prompt and uses the mean reward of the other k-1 responses as each one&#8217;s baseline. Cohere&#8217;s 2024 study showed REINFORCE-style methods like this outperform PPO, DPO, and RAFT in RLHF.</p><p><strong>ReMax</strong> uses the reward of the greedy-decoded response as the baseline. One extra greedy generation replaces an entire value network.</p><p><strong>REINFORCE++</strong> normalizes advantages across the entire global batch instead of within each prompt&#8217;s group. Its authors show prompt-level normalization is biased, while the global version&#8217;s bias vanishes as the batch grows.</p><p><strong>OPO</strong> enforces exact on-policy training and derives the baseline that minimizes gradient variance. Under a simplifying assumption, that baseline becomes a length-weighted mean of rewards, so longer responses count more.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!LtMl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fe89051-77e0-462e-9a55-936b59402d15_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!LtMl!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fe89051-77e0-462e-9a55-936b59402d15_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!LtMl!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fe89051-77e0-462e-9a55-936b59402d15_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!LtMl!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fe89051-77e0-462e-9a55-936b59402d15_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!LtMl!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fe89051-77e0-462e-9a55-936b59402d15_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!LtMl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fe89051-77e0-462e-9a55-936b59402d15_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6fe89051-77e0-462e-9a55-936b59402d15_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:162139,&quot;alt&quot;:&quot;A better baseline cuts gradient variance without training a second model.&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224378592?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fe89051-77e0-462e-9a55-936b59402d15_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A better baseline cuts gradient variance without training a second model." title="A better baseline cuts gradient variance without training a second model." srcset="https://substackcdn.com/image/fetch/$s_!LtMl!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fe89051-77e0-462e-9a55-936b59402d15_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!LtMl!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fe89051-77e0-462e-9a55-936b59402d15_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!LtMl!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fe89051-77e0-462e-9a55-936b59402d15_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!LtMl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fe89051-77e0-462e-9a55-936b59402d15_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A better baseline cuts gradient variance without training a second model.</figcaption></figure></div><p>OPO needs no value model, no reference model, and no KL term. Its implementation is merged into verl.</p><h4>Group-relative: GRPO, Dr. GRPO, DAPO, LitePPO</h4><p><strong>GRPO</strong> came from DeepSeekMath. It samples a group of responses per prompt, normalizes each reward by the group&#8217;s mean and standard deviation, and feeds that advantage into PPO&#8217;s clipped objective with a KL penalty to a reference model.</p><p>There&#8217;s no critic, and DeepSeek later used GRPO to train DeepSeek-R1. But its defaults carry biases.</p><p><strong>Dr. GRPO</strong> shows that GRPO&#8217;s per-response length normalization pushes response length up, especially for wrong answers. It removes both the length and the standard-deviation normalization terms. With it, the authors&#8217; 7B recipe reached 43.3% on AIME 2024.</p><p><strong>DAPO</strong> from ByteDance Seed makes four changes and also drops the KL term:</p><ul><li><p>Clip-Higher decouples the clip range to 0.2 below and 0.28 above, so low-probability exploration tokens can grow.</p></li><li><p>Dynamic Sampling over-samples and filters out prompts where every answer is right or every answer is wrong, since those groups produce zero advantage.</p></li><li><p>A token-level loss weights long responses by their token count.</p></li><li><p>Overlong reward shaping applies a soft, length-aware penalty to responses that run past the length limit.</p></li></ul><p>DAPO reached 50 points on AIME 2024 with a Qwen2.5-32B base model.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ERmB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cca2437-6465-4571-8848-6265b8726499_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ERmB!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cca2437-6465-4571-8848-6265b8726499_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ERmB!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cca2437-6465-4571-8848-6265b8726499_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ERmB!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cca2437-6465-4571-8848-6265b8726499_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ERmB!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cca2437-6465-4571-8848-6265b8726499_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ERmB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cca2437-6465-4571-8848-6265b8726499_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3cca2437-6465-4571-8848-6265b8726499_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:150336,&quot;alt&quot;:&quot;GRPO's successors are small edits to the same loop.&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224378592?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cca2437-6465-4571-8848-6265b8726499_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="GRPO's successors are small edits to the same loop." title="GRPO's successors are small edits to the same loop." srcset="https://substackcdn.com/image/fetch/$s_!ERmB!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cca2437-6465-4571-8848-6265b8726499_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ERmB!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cca2437-6465-4571-8848-6265b8726499_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ERmB!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cca2437-6465-4571-8848-6265b8726499_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ERmB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cca2437-6465-4571-8848-6265b8726499_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">GRPO&#8217;s successors are small edits to the same loop.</figcaption></figure></div><p><strong>LitePPO</strong> tested these tricks one at a time. It kept only two: group-level mean with batch-level standard deviation for the advantage, and token-level loss aggregation, on a vanilla PPO loss without a critic. In the authors&#8217; experiments, that pair beat both GRPO and DAPO.</p><h4>Ratio and clipping redesigns: GSPO, GMPO, CISPO, SAPO</h4><p>Every method above reuses samples from a slightly older policy, so the token ratios drift. The SAPO authors note that this variance gets worse in Mixture-of-Experts models.</p><p><strong>GSPO</strong> from Qwen computes one ratio per sequence, the length-normalized sequence likelihood ratio, which works out to the geometric mean of the token ratios. Clipping happens at the sequence level. The Qwen team reports that it stabilizes MoE RL training and credits it with gains in Qwen3.</p><p><strong>GMPO</strong> maximizes the geometric mean of the token-level objectives instead of the arithmetic mean, so one outlier ratio can&#8217;t dominate. It stays stable with a clip range of e^-0.4 to e^0.4, wider than GRPO&#8217;s 0.8 to 1.2, and improved GRPO&#8217;s average Pass@1 by up to 4.1% at 7B.</p><p><strong>CISPO</strong> comes from the MiniMax-M1 report. It clips the importance-sampling weight and stops the gradient through it, instead of clipping the token update. Every token still contributes a gradient, and in a Qwen2.5-32B study it matched DAPO&#8217;s AIME score in half the training steps.</p><p><strong>SAPO</strong>, also from Qwen, replaces the hard clip with a smooth, temperature-controlled gate. When a few tokens in a sequence go off-policy, GSPO suppresses the gradient for the whole sequence. SAPO down-weights only those tokens. Qwen used it to train the Qwen3-VL series.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!G3jQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F786b0fa4-af9f-414c-bb29-57fd921e75f1_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!G3jQ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F786b0fa4-af9f-414c-bb29-57fd921e75f1_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!G3jQ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F786b0fa4-af9f-414c-bb29-57fd921e75f1_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!G3jQ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F786b0fa4-af9f-414c-bb29-57fd921e75f1_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!G3jQ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F786b0fa4-af9f-414c-bb29-57fd921e75f1_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!G3jQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F786b0fa4-af9f-414c-bb29-57fd921e75f1_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/786b0fa4-af9f-414c-bb29-57fd921e75f1_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:142590,&quot;alt&quot;:&quot;Off-policy tokens are unavoidable. The question is how much signal you throw away.&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224378592?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F786b0fa4-af9f-414c-bb29-57fd921e75f1_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Off-policy tokens are unavoidable. The question is how much signal you throw away." title="Off-policy tokens are unavoidable. The question is how much signal you throw away." srcset="https://substackcdn.com/image/fetch/$s_!G3jQ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F786b0fa4-af9f-414c-bb29-57fd921e75f1_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!G3jQ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F786b0fa4-af9f-414c-bb29-57fd921e75f1_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!G3jQ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F786b0fa4-af9f-414c-bb29-57fd921e75f1_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!G3jQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F786b0fa4-af9f-414c-bb29-57fd921e75f1_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Off-policy tokens are unavoidable. The question is how much signal you throw away.</figcaption></figure></div><p>Laid out this way, the 16 names collapse into two dials: the baseline and the clip.</p><p>When the next RL paper lands, check which of the two it touches first. That usually tells you whether it fixes a problem you actually have.</p><p>If you want to understand where these dials come from, our RL course builds them from scratch, from the Bellman equations to REINFORCE, PPO, RLHF, GRPO, and agentic RL:</p><ul><li><p><strong><a href="https://www.dailydoseofds.com/rl-course-part-1/">Part 1: Foundations of Reinforcement Learning &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/rl-course-part-2/">Part 2: Markov Decision Processes and Value Functions &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/rl-course-part-3/">Part 3: Bellman Equations and Dynamic Programming &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/rl-course-part-4/">Part 4: Model-Free Learning &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/rl-course-part-5/">Part 5: Function Approximation &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/rl-course-part-6/">Part 6: Introduction to Deep RL and DQN &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/rl-course-part-7/">Part 7: Policy Gradients: REINFORCE and Actor-Critic &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/rl-course-part-8/">Part 8: Proximal Policy Optimization &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/rl-course-part-9/">Part 9: RLHF to Align Language Models with Human Feedback &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/rl-course-part-10/">Part 10: Verifiable Rewards and GRPO &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/rl-course-part-11/">Part 11: The Reward Signal Problem for Agents &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/rl-course-part-12/">Part 12: Agentic RL: Environments, Trajectories, and the Training &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/rl-course-part-13/">Part 13: How Do AI Teams Use RL in Production? &#8594;</a></strong></p></li></ul><p>&#128073; Over to you: Which policy optimization method runs in your post-training stack today?</p><div><hr></div><h3><a href="https://www.dailydoseofds.com/p/kv-cache-engineering-for-llm-serving/">How hybrid LLMs shrink the KV cache</a></h3><p>Every attention layer stores a key and a value for every token it has seen.</p><p>Hybrid LLMs keep only a few attention layers. The rest are linear-attention or state-space layers, like Mamba, that compress everything seen so far into a fixed-size state.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ZkX8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46962452-9ebd-43e8-9d72-ea96c6d72ef3_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ZkX8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46962452-9ebd-43e8-9d72-ea96c6d72ef3_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ZkX8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46962452-9ebd-43e8-9d72-ea96c6d72ef3_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ZkX8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46962452-9ebd-43e8-9d72-ea96c6d72ef3_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ZkX8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46962452-9ebd-43e8-9d72-ea96c6d72ef3_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ZkX8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46962452-9ebd-43e8-9d72-ea96c6d72ef3_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/46962452-9ebd-43e8-9d72-ea96c6d72ef3_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:130632,&quot;alt&quot;:&quot;Only the attention layers pay for every token you keep.&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224378592?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46962452-9ebd-43e8-9d72-ea96c6d72ef3_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Only the attention layers pay for every token you keep." title="Only the attention layers pay for every token you keep." srcset="https://substackcdn.com/image/fetch/$s_!ZkX8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46962452-9ebd-43e8-9d72-ea96c6d72ef3_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ZkX8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46962452-9ebd-43e8-9d72-ea96c6d72ef3_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ZkX8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46962452-9ebd-43e8-9d72-ea96c6d72ef3_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ZkX8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46962452-9ebd-43e8-9d72-ea96c6d72ef3_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Only the attention layers pay for every token you keep.</figcaption></figure></div><p>That state does not grow with context. The visual below shows the difference on one 262K-token sequence.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!x3Zs!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd74c31d3-5979-4862-9fde-d58b1da1d6f0_1024x887.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!x3Zs!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd74c31d3-5979-4862-9fde-d58b1da1d6f0_1024x887.jpeg 424w, https://substackcdn.com/image/fetch/$s_!x3Zs!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd74c31d3-5979-4862-9fde-d58b1da1d6f0_1024x887.jpeg 848w, https://substackcdn.com/image/fetch/$s_!x3Zs!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd74c31d3-5979-4862-9fde-d58b1da1d6f0_1024x887.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!x3Zs!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd74c31d3-5979-4862-9fde-d58b1da1d6f0_1024x887.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!x3Zs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd74c31d3-5979-4862-9fde-d58b1da1d6f0_1024x887.jpeg" width="1024" height="887" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d74c31d3-5979-4862-9fde-d58b1da1d6f0_1024x887.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:887,&quot;width&quot;:1024,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:84492,&quot;alt&quot;:&quot;A 3:1 hybrid caches keys and values in 10 layers instead of 40.&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224378592?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd74c31d3-5979-4862-9fde-d58b1da1d6f0_1024x887.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A 3:1 hybrid caches keys and values in 10 layers instead of 40." title="A 3:1 hybrid caches keys and values in 10 layers instead of 40." srcset="https://substackcdn.com/image/fetch/$s_!x3Zs!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd74c31d3-5979-4862-9fde-d58b1da1d6f0_1024x887.jpeg 424w, https://substackcdn.com/image/fetch/$s_!x3Zs!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd74c31d3-5979-4862-9fde-d58b1da1d6f0_1024x887.jpeg 848w, https://substackcdn.com/image/fetch/$s_!x3Zs!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd74c31d3-5979-4862-9fde-d58b1da1d6f0_1024x887.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!x3Zs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd74c31d3-5979-4862-9fde-d58b1da1d6f0_1024x887.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A 3:1 hybrid caches keys and values in 10 layers instead of 40.</figcaption></figure></div><p>Qwen3.5-35B-A3B has 40 layers, laid out as 10 repeats of three Gated DeltaNet layers followed by one gated attention layer. Only those 10 attention layers write to the KV cache.</p><p>Each attention layer caches 2 KV heads of 256 dimensions for keys and values. At 16 bits, that is 2,048 bytes per token per layer, or 512 MiB per layer at 262,144 tokens.</p><ul><li><p>40 layers give 20 GiB.</p></li><li><p>10 layers result in 5 GiB.</p></li></ul><p>Each Gated DeltaNet layer instead carries a recurrent state of 32 heads &#215; 128 &#215; 128 values, whether the context is 1K tokens or 262K.</p><p><strong>So why not do fully linear?</strong></p><p>A fixed-size state is lossy.</p><p>When AI21 built Jamba, a pure Mamba model struggled with in-context learning, for example, following the output format shown in few-shot examples, while the attention-Mamba hybrid behaved like a vanilla Transformer.</p><p>Attention layers are where exact recall of any past token lives. Hybrids keep a few of them and let the rest run in constant memory.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!KTre!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7214fc09-7291-46af-b3d9-85e764f2d64a_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!KTre!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7214fc09-7291-46af-b3d9-85e764f2d64a_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!KTre!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7214fc09-7291-46af-b3d9-85e764f2d64a_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!KTre!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7214fc09-7291-46af-b3d9-85e764f2d64a_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!KTre!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7214fc09-7291-46af-b3d9-85e764f2d64a_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!KTre!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7214fc09-7291-46af-b3d9-85e764f2d64a_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7214fc09-7291-46af-b3d9-85e764f2d64a_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:162475,&quot;alt&quot;:&quot;Keep a few attention layers for recall. Let the rest run in constant memory.&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224378592?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7214fc09-7291-46af-b3d9-85e764f2d64a_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Keep a few attention layers for recall. Let the rest run in constant memory." title="Keep a few attention layers for recall. Let the rest run in constant memory." srcset="https://substackcdn.com/image/fetch/$s_!KTre!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7214fc09-7291-46af-b3d9-85e764f2d64a_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!KTre!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7214fc09-7291-46af-b3d9-85e764f2d64a_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!KTre!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7214fc09-7291-46af-b3d9-85e764f2d64a_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!KTre!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7214fc09-7291-46af-b3d9-85e764f2d64a_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Keep a few attention layers for recall. Let the rest run in constant memory.</figcaption></figure></div><p>Labs have converged on mostly linear stacks but disagree on how much attention to keep:</p><p>&#8627; Qwen3-Next-80B-A3B and Qwen3.5 &#8594; 3 Gated DeltaNet layers per gated attention layer. Qwen reports that Qwen3-Next-80B-A3B gets 10x the inference throughput of Qwen3-32B for contexts over 32K tokens.</p><p>&#8627; Jamba &#8594; 1 attention layer per 7 Mamba layers. Its KV cache at 256K tokens is 4GB, versus 32GB for Mixtral.</p><p>&#8627; MiniMax-M1 &#8594; 1 softmax attention block after every 7 lightning attention blocks. At a 100K-token generation length, it uses about 25% of DeepSeek R1&#8217;s FLOPs.</p><p>&#8627; Granite 4.0-H &#8594; 9 Mamba-2 blocks per transformer block. IBM reports over 70% less RAM for long inputs and concurrent batches.</p><p>&#8627; Nemotron-H &#8594; roughly 8% attention layers, 4 of 52 in the 8B model and 10 of 118 in the 56B model. NVIDIA reports up to 3x faster inference than similarly sized Transformers.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!-Rzt!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb3d9c16-913e-4af7-8030-073772611deb_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!-Rzt!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb3d9c16-913e-4af7-8030-073772611deb_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!-Rzt!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb3d9c16-913e-4af7-8030-073772611deb_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!-Rzt!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb3d9c16-913e-4af7-8030-073772611deb_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!-Rzt!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb3d9c16-913e-4af7-8030-073772611deb_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!-Rzt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb3d9c16-913e-4af7-8030-073772611deb_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/db3d9c16-913e-4af7-8030-073772611deb_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:171123,&quot;alt&quot;:&quot;Everyone went mostly linear. They disagree on how much attention to keep.&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224378592?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb3d9c16-913e-4af7-8030-073772611deb_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Everyone went mostly linear. They disagree on how much attention to keep." title="Everyone went mostly linear. They disagree on how much attention to keep." srcset="https://substackcdn.com/image/fetch/$s_!-Rzt!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb3d9c16-913e-4af7-8030-073772611deb_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!-Rzt!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb3d9c16-913e-4af7-8030-073772611deb_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!-Rzt!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb3d9c16-913e-4af7-8030-073772611deb_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!-Rzt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb3d9c16-913e-4af7-8030-073772611deb_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Everyone went mostly linear. They disagree on how much attention to keep.</figcaption></figure></div><p>This ties back to the main section, too. MiniMax trained M1 with CISPO, and its report credits the hybrid attention design plus CISPO for finishing full RL training on 512 H800 GPUs in three weeks.</p><p>The savings grow with context length and concurrency. At short contexts, the KV cache is small anyway, and the real speedup depends on how well your serving engine implements the linear layers. The Qwen3-Next model card itself notes that the throughput gain &#8220;depends highly on the implementation.&#8221;</p><p>Hybrid layers are one of 12 ways to shrink the KV cache. We covered all of them in our KV cache engineering article, including GQA, multi-head latent attention, sliding windows, quantization, eviction, paging, prefix reuse, and offloading, plus how to combine them.</p><p><strong><a href="https://www.dailydoseofds.com/p/kv-cache-engineering-for-llm-serving/">Read the KV cache engineering article here &#8594;</a></strong></p><p>Good day!</p>]]></content:encoded></item><item><title><![CDATA[Pages in Grok Bot, clearly explained:]]></title><description><![CDATA[...covered with a full hands-on video demo.]]></description><link>https://blog.dailydoseofds.com/p/pages-in-grok-bot-clearly-explained</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/pages-in-grok-bot-clearly-explained</guid><dc:creator><![CDATA[Avi Chawla]]></dc:creator><pubDate>Mon, 09 Nov 2026 23:04:15 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!vqlA!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfac875-3603-4ccd-82e3-4dc6c6b0ffb6_2752x1536.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><strong>In today&#8217;s newsletter:</strong></p><ul><li><p>Your coding agent can now do the one thing it used to need always asked you for.</p></li><li><p>Pages in Grok Bot, clearly explained:</p></li><li><p>Chat with video using EmbeddingGemma 2 (100% local).</p></li></ul><div><hr></div><h3><a href="https://fandf.co/4ze7kLT">Your coding agent can now do the one thing it used to need always asked you for.</a></h3><p>When you ask an agent to build an app, it writes the schema, the routes, and the data layer in just a few minutes.</p><p>But then it needs a database, and everything stops while you  open a browser, create an account, provision a database, set up a user, and paste the connection string back into the session.</p><p>Those steps were  built for a human, so the agent just sits there waiting for you to complete them</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ZoYw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F659480c1-01f3-4d4c-8a3e-21d6c9f439f6_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ZoYw!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F659480c1-01f3-4d4c-8a3e-21d6c9f439f6_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ZoYw!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F659480c1-01f3-4d4c-8a3e-21d6c9f439f6_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ZoYw!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F659480c1-01f3-4d4c-8a3e-21d6c9f439f6_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ZoYw!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F659480c1-01f3-4d4c-8a3e-21d6c9f439f6_2752x1536.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!ZoYw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F659480c1-01f3-4d4c-8a3e-21d6c9f439f6_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/659480c1-01f3-4d4c-8a3e-21d6c9f439f6_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:146141,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224219784?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F659480c1-01f3-4d4c-8a3e-21d6c9f439f6_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ZoYw!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F659480c1-01f3-4d4c-8a3e-21d6c9f439f6_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ZoYw!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F659480c1-01f3-4d4c-8a3e-21d6c9f439f6_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ZoYw!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F659480c1-01f3-4d4c-8a3e-21d6c9f439f6_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ZoYw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F659480c1-01f3-4d4c-8a3e-21d6c9f439f6_2752x1536.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A coding agent can write the app, then stall when the database still needs a human signup.</figcaption></figure></div><p>MongoDB just solved that with <strong><a href="https://fandf.co/4ze7kLT">Atlas Ephemeral Clusters</a></strong>.</p><p>Now, your agent can make one API call and Atlas returns a live database in under a minute.</p><p>Your agent saves the connection string, allowing you to keep building on a real MongoDB Atlas cluster. The video shows this in action:</p><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;3e4a02d5-5110-4691-ba48-299d4e245bc5&quot;,&quot;duration&quot;:null}"></div><p>The part I like most is that the agent can create the database cluster and work with the data, but only a human can make the decision to claim it and take ownership.</p><p>If you don&#8217;t claim it, the cluster pauses after 48 hours and then will be deleted after 7 days.</p><p>If you claim it, it becomes a permanent free Atlas cluster with the same data and the same connection string.</p><p>So the agent handles the setup end-to-end setup, and you only step in once the project is worth keeping.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!uGK8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf91c665-1dee-481b-89c4-e9a7fe5d0ddb_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!uGK8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf91c665-1dee-481b-89c4-e9a7fe5d0ddb_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!uGK8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf91c665-1dee-481b-89c4-e9a7fe5d0ddb_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!uGK8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf91c665-1dee-481b-89c4-e9a7fe5d0ddb_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!uGK8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf91c665-1dee-481b-89c4-e9a7fe5d0ddb_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!uGK8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf91c665-1dee-481b-89c4-e9a7fe5d0ddb_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/af91c665-1dee-481b-89c4-e9a7fe5d0ddb_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:132019,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224219784?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf91c665-1dee-481b-89c4-e9a7fe5d0ddb_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!uGK8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf91c665-1dee-481b-89c4-e9a7fe5d0ddb_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!uGK8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf91c665-1dee-481b-89c4-e9a7fe5d0ddb_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!uGK8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf91c665-1dee-481b-89c4-e9a7fe5d0ddb_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!uGK8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf91c665-1dee-481b-89c4-e9a7fe5d0ddb_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">An agent can create and use the cluster. Only a human can claim it into a lasting Atlas account.</figcaption></figure></div><p>Atlas Ephemeral Clusters are in public preview, and the docs walk you through the full flow.</p><p>You can learn more here: <strong><a href="https://fandf.co/4ze7kLT">https://www.mongodb.com/docs/atlas/tutorial/create-ephemeral-cluster/</a></strong></p><p><em>Thanks to MongoDB for partnering today!</em></p><div><hr></div><h3><a href="https://www.dailydoseofds.com/p/grok-bot-masterclass/">Pages in Grok Bot, clearly explained:</a></h3><p>Agent work that stays buried in chat is hard to revisit, share, or keep current.</p><p>We recorded a short walkthrough of Grok Bot Pages, and three things he built with them.</p><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;fe53362f-53ae-4fb0-b50f-35d93fa49842&quot;,&quot;duration&quot;:null}"></div><p>A Page is a living interactive dashboard that a bot creates for you. It is not a PDF, and it is not a slide deck.</p><p>It can hold text, tables, charts, and controls like filters and switches. The bot can edit the same page later, including from a routine, so one link keeps pointing at the latest version.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!vqlA!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfac875-3603-4ccd-82e3-4dc6c6b0ffb6_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!vqlA!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfac875-3603-4ccd-82e3-4dc6c6b0ffb6_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!vqlA!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfac875-3603-4ccd-82e3-4dc6c6b0ffb6_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!vqlA!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfac875-3603-4ccd-82e3-4dc6c6b0ffb6_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!vqlA!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfac875-3603-4ccd-82e3-4dc6c6b0ffb6_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!vqlA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfac875-3603-4ccd-82e3-4dc6c6b0ffb6_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5cfac875-3603-4ccd-82e3-4dc6c6b0ffb6_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:165124,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224219784?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfac875-3603-4ccd-82e3-4dc6c6b0ffb6_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!vqlA!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfac875-3603-4ccd-82e3-4dc6c6b0ffb6_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!vqlA!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfac875-3603-4ccd-82e3-4dc6c6b0ffb6_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!vqlA!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfac875-3603-4ccd-82e3-4dc6c6b0ffb6_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!vqlA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cfac875-3603-4ccd-82e3-4dc6c6b0ffb6_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>A page is a durable home for agent work, not another message in a thread. A page stays at one address while the bot keeps the content current.</p><p>This is what we cover in the video:</p><ul><li><p>00:00 - HQ office walkthrough</p></li><li><p>01:42 - What a Page is</p></li><li><p>02:04 - Notion board and routine</p></li><li><p>02:57 - Custom X analytics page</p></li><li><p>04:01 - Interactive learning guide</p></li><li><p>04:24 - Share, browser, and mobile</p></li><li><p>04:51 - Projectile demo</p></li><li><p>05:16 - Wrap-up</p></li></ul><p>Pages do not replace the bots. They give the bots a place to publish work that you can reopen without digging through chat.</p><p>If you want the broader org setup behind this demo, our earlier Grok Bot masterclass covers how the 14-bot team is structured.</p><p><strong><a href="https://www.dailydoseofds.com/p/grok-bot-masterclass/">Watch the Grok Bot masterclass here &#8594;</a></strong></p><p>&#128073; Over to you: Would you rather give your bots a live HQ page, or a local video-search page first?</p><div><hr></div><h3><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-3-with-implementation/">Chat with video using EmbeddingGemma 2 (100% local)</a></h3><p>We built a local chat interface that finds matching frames from a text query and jumps to their timestamps.</p><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;dd82576d-a812-4808-ab79-b327dbba1651&quot;,&quot;duration&quot;:null}"></div><p>EmbeddingGemma 2 from Google DeepMind maps text, images, video, and audio into one shared vector space. That is what makes a natural-language query able to retrieve a video moment.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!7Re3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7490a69c-f771-4300-a4b9-70d81d17de66_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!7Re3!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7490a69c-f771-4300-a4b9-70d81d17de66_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!7Re3!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7490a69c-f771-4300-a4b9-70d81d17de66_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!7Re3!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7490a69c-f771-4300-a4b9-70d81d17de66_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!7Re3!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7490a69c-f771-4300-a4b9-70d81d17de66_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!7Re3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7490a69c-f771-4300-a4b9-70d81d17de66_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7490a69c-f771-4300-a4b9-70d81d17de66_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:134174,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224219784?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7490a69c-f771-4300-a4b9-70d81d17de66_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!7Re3!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7490a69c-f771-4300-a4b9-70d81d17de66_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!7Re3!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7490a69c-f771-4300-a4b9-70d81d17de66_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!7Re3!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7490a69c-f771-4300-a4b9-70d81d17de66_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!7Re3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7490a69c-f771-4300-a4b9-70d81d17de66_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The model has about 740M parameters in full multimodal form, with modular text, vision, and audio encoders. Native Rust ports for local inference already exist.</p><p>Unsloth also ships a 176 MB, 4-bit GGUF for text embeddings. That checkpoint does not include the full multimodal pipeline, so video search still needs the vision path.</p><p>We indexed a 60-second video on an Apple M3 Pro over CPU:</p><ul><li><p>Indexing took 29 seconds</p></li><li><p>Search took 25 ms on average across 9 queries</p></li><li><p>The saved frame vectors occupied 180 KiB</p></li></ul><p>The expensive step happens once. Each later search embeds the query and compares it with the saved frame vectors. Nothing leaves the laptop.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!jP7N!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc152d58c-aa47-4938-ab67-a29b5b9fbbfb_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!jP7N!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc152d58c-aa47-4938-ab67-a29b5b9fbbfb_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!jP7N!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc152d58c-aa47-4938-ab67-a29b5b9fbbfb_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!jP7N!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc152d58c-aa47-4938-ab67-a29b5b9fbbfb_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!jP7N!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc152d58c-aa47-4938-ab67-a29b5b9fbbfb_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!jP7N!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc152d58c-aa47-4938-ab67-a29b5b9fbbfb_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c152d58c-aa47-4938-ab67-a29b5b9fbbfb_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:148743,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224219784?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc152d58c-aa47-4938-ab67-a29b5b9fbbfb_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!jP7N!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc152d58c-aa47-4938-ab67-a29b5b9fbbfb_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!jP7N!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc152d58c-aa47-4938-ab67-a29b5b9fbbfb_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!jP7N!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc152d58c-aa47-4938-ab67-a29b5b9fbbfb_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!jP7N!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc152d58c-aa47-4938-ab67-a29b5b9fbbfb_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Pay for indexing once. Every later search is a cheap local vector comparison.</p><p>Model quantization shrinks the weight memory. The retrieval index is a separate cost, and it grows as you add videos.</p><p>To cut that index cost, store vectors with one bit per dimension instead of float32. That is a 32x reduction in the raw vector payload, and we covered a full retrieval implementation in our binary quantization article.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!7a0v!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17db4f76-2bf5-46ed-bbfb-6c45aa3dbd3f_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!7a0v!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17db4f76-2bf5-46ed-bbfb-6c45aa3dbd3f_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!7a0v!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17db4f76-2bf5-46ed-bbfb-6c45aa3dbd3f_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!7a0v!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17db4f76-2bf5-46ed-bbfb-6c45aa3dbd3f_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!7a0v!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17db4f76-2bf5-46ed-bbfb-6c45aa3dbd3f_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!7a0v!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17db4f76-2bf5-46ed-bbfb-6c45aa3dbd3f_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/17db4f76-2bf5-46ed-bbfb-6c45aa3dbd3f_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:152963,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/224219784?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17db4f76-2bf5-46ed-bbfb-6c45aa3dbd3f_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!7a0v!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17db4f76-2bf5-46ed-bbfb-6c45aa3dbd3f_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!7a0v!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17db4f76-2bf5-46ed-bbfb-6c45aa3dbd3f_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!7a0v!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17db4f76-2bf5-46ed-bbfb-6c45aa3dbd3f_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!7a0v!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F17db4f76-2bf5-46ed-bbfb-6c45aa3dbd3f_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Quantize the weights to fit the laptop. Compress the index if the library grows.</figcaption></figure></div><p><strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-3-with-implementation/">Read the binary quantization breakdown here &#8594;</a></strong></p><p>Good day!</p>]]></content:encoded></item><item><title><![CDATA[7 MoE routing strategies in production, clearly explained]]></title><description><![CDATA[+ a practical learning checklist to learn AI engineering.]]></description><link>https://blog.dailydoseofds.com/p/7-moe-routing-strategies-in-production</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/7-moe-routing-strategies-in-production</guid><dc:creator><![CDATA[Avi Chawla]]></dc:creator><pubDate>Sat, 07 Nov 2026 01:58:32 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!2C2x!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F265d349e-5fe7-4eea-8226-9058281c7542_1400x1200.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><strong>In today's newsletter:</strong></p><ul><li><p>An open-source alternative to Claude (33k+ stars)!</p></li><li><p>7 MoE routing strategies in production, clearly explained.</p></li><li><p>As an AI engineer, please learn.</p></li></ul><div><hr></div><h3><strong><a href="https://github.com/onyx-dot-app/onyx">An open-source alternative to Claude (33k+ stars)!</a></strong></h3><p><strong><a href="https://github.com/onyx-dot-app/onyx">Onyx</a></strong> is a self-hostable AI chat platform that works with any LLM, like Claude, GPT, Gemini, Llama, or any open-weight model you want.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://github.com/onyx-dot-app/onyx&quot;,&quot;text&quot;:&quot;Onyx GitHub Repo&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://github.com/onyx-dot-app/onyx"><span>Onyx GitHub Repo</span></a></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!B_Eu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cd6f829-9f6d-4255-a623-894a4d6a7645_970x919.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!B_Eu!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cd6f829-9f6d-4255-a623-894a4d6a7645_970x919.png 424w, https://substackcdn.com/image/fetch/$s_!B_Eu!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cd6f829-9f6d-4255-a623-894a4d6a7645_970x919.png 848w, https://substackcdn.com/image/fetch/$s_!B_Eu!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cd6f829-9f6d-4255-a623-894a4d6a7645_970x919.png 1272w, https://substackcdn.com/image/fetch/$s_!B_Eu!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cd6f829-9f6d-4255-a623-894a4d6a7645_970x919.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!B_Eu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cd6f829-9f6d-4255-a623-894a4d6a7645_970x919.png" width="970" height="919" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3cd6f829-9f6d-4255-a623-894a4d6a7645_970x919.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:919,&quot;width&quot;:970,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:48613,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/198870603?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cd6f829-9f6d-4255-a623-894a4d6a7645_970x919.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!B_Eu!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cd6f829-9f6d-4255-a623-894a4d6a7645_970x919.png 424w, https://substackcdn.com/image/fetch/$s_!B_Eu!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cd6f829-9f6d-4255-a623-894a4d6a7645_970x919.png 848w, https://substackcdn.com/image/fetch/$s_!B_Eu!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cd6f829-9f6d-4255-a623-894a4d6a7645_970x919.png 1272w, https://substackcdn.com/image/fetch/$s_!B_Eu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3cd6f829-9f6d-4255-a623-894a4d6a7645_970x919.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Here&#8217;s what it ships with:</p><ul><li><p>Agents that chain multiple tools in sequence</p></li><li><p>RAG with full indexing across 40+ connectors (Slack, Drive, Confluence, Jira, GitHub, email, call transcripts)</p></li><li><p>Deep research ranked #1 on DeepResearchBench, above every proprietary alternative</p></li><li><p>MCP support for connecting to external systems</p></li><li><p>Code interpreter for data analysis and file generation</p></li><li><p>Self-host on your own infrastructure via Docker in a few mins</p></li></ul><p>Unlike Claude&#8217;s MCP-based connectors that query your tools at runtime, Onyx actually indexes all your data. That means faster, more reliable search across everything your team has ever written.</p><p>The entire code is open-source under the MIT license, so you can see the full implementation on GitHub and try it yourself.</p><p><strong><a href="https://github.com/onyx-dot-app/onyx">Find the GitHub repo here &#8594;</a> (don&#8217;t forget to star it &#11088;&#65039;)</strong></p><div><hr></div><h3><a href="https://www.dailydoseofds.com/p/moe-inference-engineering-clearly-explained/">7 MoE routing strategies in production, clearly explained</a></h3><p>Sending every token to its highest-scoring experts can make an MoE model slower.</p><p>The problem is that many tokens can prefer the same experts. Those experts become overloaded while others remain underused.</p><p>With fixed capacity and a dropping rule, some token assignments may be dropped. Across GPUs, uneven workloads and communication overhead can reduce the compute savings.</p><p>MoE routing has to select useful experts while distributing work efficiently. The seven strategies in the visual handle that tradeoff differently.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!2C2x!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F265d349e-5fe7-4eea-8226-9058281c7542_1400x1200.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!2C2x!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F265d349e-5fe7-4eea-8226-9058281c7542_1400x1200.png 424w, https://substackcdn.com/image/fetch/$s_!2C2x!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F265d349e-5fe7-4eea-8226-9058281c7542_1400x1200.png 848w, https://substackcdn.com/image/fetch/$s_!2C2x!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F265d349e-5fe7-4eea-8226-9058281c7542_1400x1200.png 1272w, https://substackcdn.com/image/fetch/$s_!2C2x!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F265d349e-5fe7-4eea-8226-9058281c7542_1400x1200.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!2C2x!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F265d349e-5fe7-4eea-8226-9058281c7542_1400x1200.png" width="1400" height="1200" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/265d349e-5fe7-4eea-8226-9058281c7542_1400x1200.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1200,&quot;width&quot;:1400,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:155803,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223845806?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F265d349e-5fe7-4eea-8226-9058281c7542_1400x1200.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!2C2x!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F265d349e-5fe7-4eea-8226-9058281c7542_1400x1200.png 424w, https://substackcdn.com/image/fetch/$s_!2C2x!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F265d349e-5fe7-4eea-8226-9058281c7542_1400x1200.png 848w, https://substackcdn.com/image/fetch/$s_!2C2x!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F265d349e-5fe7-4eea-8226-9058281c7542_1400x1200.png 1272w, https://substackcdn.com/image/fetch/$s_!2C2x!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F265d349e-5fe7-4eea-8226-9058281c7542_1400x1200.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>One caveat before we start. These are routing designs, not seven interchangeable settings in a serving engine. Some come from research on training or vision models and need extra care for autoregressive generation.</p><h4>1. Token-choice routing</h4><p>Each token selects its top-k experts using router scores. If we choose two experts per token, four tokens create eight token-expert assignments.</p><p>That fixes the expert count per token. It does not fix how much work each expert receives.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!VGHn!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3f2965d-a6c9-4c6e-8ca9-b1b88541a8bf_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!VGHn!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3f2965d-a6c9-4c6e-8ca9-b1b88541a8bf_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!VGHn!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3f2965d-a6c9-4c6e-8ca9-b1b88541a8bf_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!VGHn!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3f2965d-a6c9-4c6e-8ca9-b1b88541a8bf_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!VGHn!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3f2965d-a6c9-4c6e-8ca9-b1b88541a8bf_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!VGHn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3f2965d-a6c9-4c6e-8ca9-b1b88541a8bf_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f3f2965d-a6c9-4c6e-8ca9-b1b88541a8bf_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:88748,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223845806?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3f2965d-a6c9-4c6e-8ca9-b1b88541a8bf_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!VGHn!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3f2965d-a6c9-4c6e-8ca9-b1b88541a8bf_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!VGHn!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3f2965d-a6c9-4c6e-8ca9-b1b88541a8bf_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!VGHn!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3f2965d-a6c9-4c6e-8ca9-b1b88541a8bf_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!VGHn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3f2965d-a6c9-4c6e-8ca9-b1b88541a8bf_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A fixed expert count per token does not fix expert load.</figcaption></figure></div><p>Say all four tokens select the same two experts in a four-expert layer. Two experts receive four assignments each, while the other two receive none.</p><p>Load-balancing mechanisms help prevent traffic from concentrating on a few experts. A training-time balancing objective still does not guarantee that every serving batch will be balanced.</p><h4>2. Expert-choice routing</h4><p>Here, we reverse the selection. Each expert selects its highest-scoring tokens within a fixed capacity.</p><p>Suppose two experts can each take two tokens. Both experts do the same number of token assignments, but they can select an overlapping set of tokens.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!enWP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc0e671f-8d0f-4a2c-b9ec-e7eaf347b08d_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!enWP!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc0e671f-8d0f-4a2c-b9ec-e7eaf347b08d_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!enWP!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc0e671f-8d0f-4a2c-b9ec-e7eaf347b08d_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!enWP!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc0e671f-8d0f-4a2c-b9ec-e7eaf347b08d_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!enWP!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc0e671f-8d0f-4a2c-b9ec-e7eaf347b08d_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!enWP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc0e671f-8d0f-4a2c-b9ec-e7eaf347b08d_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/cc0e671f-8d0f-4a2c-b9ec-e7eaf347b08d_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:132781,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223845806?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc0e671f-8d0f-4a2c-b9ec-e7eaf347b08d_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!enWP!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc0e671f-8d0f-4a2c-b9ec-e7eaf347b08d_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!enWP!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc0e671f-8d0f-4a2c-b9ec-e7eaf347b08d_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!enWP!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc0e671f-8d0f-4a2c-b9ec-e7eaf347b08d_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!enWP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc0e671f-8d0f-4a2c-b9ec-e7eaf347b08d_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Equal expert capacity can still mean unequal computation per token.</figcaption></figure></div><p>One token may reach both experts. Another may reach neither. Balanced expert workloads therefore do not mean every token receives the same amount of computation.</p><p>The original Expert Choice paper also flags limitations for autoregressive generation and very small serving batches. Selecting across a token group needs care when future tokens are unavailable.</p><h4>3. Dynamic routing</h4><p>The number of selected experts varies across tokens. Confidence-based variants allocate more experts when routing scores are less decisive.</p><p>One approach selects experts in descending score order until their cumulative router probability reaches a threshold.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Lsc6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91e37f5d-3602-45cb-9937-8e2d3ac0d81a_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Lsc6!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91e37f5d-3602-45cb-9937-8e2d3ac0d81a_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Lsc6!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91e37f5d-3602-45cb-9937-8e2d3ac0d81a_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Lsc6!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91e37f5d-3602-45cb-9937-8e2d3ac0d81a_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Lsc6!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91e37f5d-3602-45cb-9937-8e2d3ac0d81a_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Lsc6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91e37f5d-3602-45cb-9937-8e2d3ac0d81a_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/91e37f5d-3602-45cb-9937-8e2d3ac0d81a_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:148140,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223845806?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91e37f5d-3602-45cb-9937-8e2d3ac0d81a_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Lsc6!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91e37f5d-3602-45cb-9937-8e2d3ac0d81a_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Lsc6!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91e37f5d-3602-45cb-9937-8e2d3ac0d81a_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Lsc6!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91e37f5d-3602-45cb-9937-8e2d3ac0d81a_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Lsc6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91e37f5d-3602-45cb-9937-8e2d3ac0d81a_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Dynamic routing lets the number of active experts vary by token.</figcaption></figure></div><p>If one expert already carries enough probability, the router can stop there. If probability is spread across experts, it selects more of them.</p><p>Router confidence does not directly measure token difficulty, though. We should check quality and actual runtime rather than assuming that more experts always means harder input or better answers.</p><h4>4. Balanced assignment</h4><p>A global matching procedure assigns tokens under expert load constraints. Instead of making every decision independently, we consider the token group together.</p><p>The assignment tries to keep high token-expert affinity while respecting how much work each expert can take.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!XpGd!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0ba6e66-2bcd-4515-999b-7e42fa8ca457_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!XpGd!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0ba6e66-2bcd-4515-999b-7e42fa8ca457_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!XpGd!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0ba6e66-2bcd-4515-999b-7e42fa8ca457_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!XpGd!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0ba6e66-2bcd-4515-999b-7e42fa8ca457_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!XpGd!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0ba6e66-2bcd-4515-999b-7e42fa8ca457_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!XpGd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0ba6e66-2bcd-4515-999b-7e42fa8ca457_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a0ba6e66-2bcd-4515-999b-7e42fa8ca457_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:135314,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223845806?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0ba6e66-2bcd-4515-999b-7e42fa8ca457_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!XpGd!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0ba6e66-2bcd-4515-999b-7e42fa8ca457_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!XpGd!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0ba6e66-2bcd-4515-999b-7e42fa8ca457_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!XpGd!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0ba6e66-2bcd-4515-999b-7e42fa8ca457_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!XpGd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0ba6e66-2bcd-4515-999b-7e42fa8ca457_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Balancing a batch can require giving up a token&#8217;s first choice.</figcaption></figure></div><p>Some tokens may miss their highest-scoring expert to achieve a more balanced allocation. Computing that assignment also adds overhead.</p><p>BASE Layers is a research example of this idea. Its balanced assignment should not be read as a promise that every production decoder runs a global matching solver at each step.</p><h4>5. Hash-based routing</h4><p>A deterministic hash maps tokens to experts without a learned gate. In a token-ID-based version, the mapping stays fixed even when the surrounding context changes.</p><p>The expert still processes a contextual token representation. It is the expert selection that does not adapt through a learned router.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!8ar1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74a762d2-b3a6-4a9f-9345-90ac9e6683e1_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8ar1!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74a762d2-b3a6-4a9f-9345-90ac9e6683e1_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!8ar1!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74a762d2-b3a6-4a9f-9345-90ac9e6683e1_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!8ar1!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74a762d2-b3a6-4a9f-9345-90ac9e6683e1_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!8ar1!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74a762d2-b3a6-4a9f-9345-90ac9e6683e1_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!8ar1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74a762d2-b3a6-4a9f-9345-90ac9e6683e1_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/74a762d2-b3a6-4a9f-9345-90ac9e6683e1_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:110984,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223845806?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74a762d2-b3a6-4a9f-9345-90ac9e6683e1_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!8ar1!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74a762d2-b3a6-4a9f-9345-90ac9e6683e1_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!8ar1!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74a762d2-b3a6-4a9f-9345-90ac9e6683e1_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!8ar1!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74a762d2-b3a6-4a9f-9345-90ac9e6683e1_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!8ar1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74a762d2-b3a6-4a9f-9345-90ac9e6683e1_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A fixed hash chooses experts without learning a routing gate.</figcaption></figure></div><p>That removes router learning and makes selection simple. The tradeoff is less flexibility to adapt expert choice to context.</p><p>A fixed mapping also does not automatically balance real traffic. Frequent tokens can concentrate assignments, so the hash design and workload still matter.</p><h4>6. Soft MoE routing</h4><p>Experts process weighted mixtures of token representations through slots. Each slot can combine information from multiple tokens.</p><p>We first form those mixtures, pass each slot through its assigned expert, and then combine the expert outputs back into output token representations.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!G6Jj!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff627c7-8bec-43ef-993c-03e444ad4d14_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!G6Jj!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff627c7-8bec-43ef-993c-03e444ad4d14_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!G6Jj!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff627c7-8bec-43ef-993c-03e444ad4d14_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!G6Jj!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff627c7-8bec-43ef-993c-03e444ad4d14_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!G6Jj!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff627c7-8bec-43ef-993c-03e444ad4d14_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!G6Jj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff627c7-8bec-43ef-993c-03e444ad4d14_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2ff627c7-8bec-43ef-993c-03e444ad4d14_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:144610,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223845806?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff627c7-8bec-43ef-993c-03e444ad4d14_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!G6Jj!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff627c7-8bec-43ef-993c-03e444ad4d14_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!G6Jj!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff627c7-8bec-43ef-993c-03e444ad4d14_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!G6Jj!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff627c7-8bec-43ef-993c-03e444ad4d14_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!G6Jj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ff627c7-8bec-43ef-993c-03e444ad4d14_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Soft MoE changes the inputs to experts into weighted token mixtures.</figcaption></figure></div><p>This changes what an expert processes. It is different from sending an individual token to a discrete top-k set of experts.</p><p>The original Soft MoE work focuses on visual recognition and identifies autoregressive decoding as a limitation. Mixing tokens must preserve causality before we can use the same idea in a decoder.</p><h4>7. Group-limited routing</h4><p>The router restricts eligible expert groups before selecting experts. When groups reflect hardware placement, this can reduce communication across devices or nodes.</p><p>We first narrow the destinations, then select experts within those eligible groups.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!cYfa!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff48978fd-6417-44bf-ac1d-dabc605f1a31_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!cYfa!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff48978fd-6417-44bf-ac1d-dabc605f1a31_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!cYfa!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff48978fd-6417-44bf-ac1d-dabc605f1a31_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!cYfa!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff48978fd-6417-44bf-ac1d-dabc605f1a31_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!cYfa!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff48978fd-6417-44bf-ac1d-dabc605f1a31_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!cYfa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff48978fd-6417-44bf-ac1d-dabc605f1a31_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f48978fd-6417-44bf-ac1d-dabc605f1a31_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:121103,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223845806?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff48978fd-6417-44bf-ac1d-dabc605f1a31_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!cYfa!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff48978fd-6417-44bf-ac1d-dabc605f1a31_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!cYfa!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff48978fd-6417-44bf-ac1d-dabc605f1a31_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!cYfa!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff48978fd-6417-44bf-ac1d-dabc605f1a31_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!cYfa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff48978fd-6417-44bf-ac1d-dabc605f1a31_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Hardware-aligned groups can limit how far a token travels.</figcaption></figure></div><p>For example, DeepSeek-V3 uses node-limited routing that sends each token to at most four nodes. That restriction limits cross-node destinations while allowing expert selection within them.</p><p>The hardware mapping matters. Arbitrary logical groups do not guarantee less network traffic, and adding a routing restriction to an already trained model can change its selected computation.</p><p>These choices affect how serving engines batch expert computation and distribute work across GPUs.</p><p>Evenly distributing tokens does not guarantee low latency. Expert batches may still be small, and moving tokens between devices takes time.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!gHqo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fd54815-f5f7-4a71-9a99-83f0b58ff6f7_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!gHqo!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fd54815-f5f7-4a71-9a99-83f0b58ff6f7_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!gHqo!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fd54815-f5f7-4a71-9a99-83f0b58ff6f7_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!gHqo!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fd54815-f5f7-4a71-9a99-83f0b58ff6f7_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!gHqo!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fd54815-f5f7-4a71-9a99-83f0b58ff6f7_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!gHqo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fd54815-f5f7-4a71-9a99-83f0b58ff6f7_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0fd54815-f5f7-4a71-9a99-83f0b58ff6f7_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:131148,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223845806?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fd54815-f5f7-4a71-9a99-83f0b58ff6f7_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!gHqo!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fd54815-f5f7-4a71-9a99-83f0b58ff6f7_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!gHqo!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fd54815-f5f7-4a71-9a99-83f0b58ff6f7_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!gHqo!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fd54815-f5f7-4a71-9a99-83f0b58ff6f7_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!gHqo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0fd54815-f5f7-4a71-9a99-83f0b58ff6f7_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Measure the full MoE path, not only the routing scores.</figcaption></figure></div><p>When we profile an MoE deployment, we need to separate routing, token movement, expert computation, and combination. The slowest stage tells us what to investigate next.</p><p>We covered the rest of this path in our MoE inference engineering article, including expert parallelism, grouped GEMM, topology-aware placement, memory requirements, and inference optimizations.</p><p><strong><a href="https://www.dailydoseofds.com/p/moe-inference-engineering-clearly-explained/">Read the MoE inference engineering breakdown here &#8594;</a></strong></p><div><hr></div><h3><a href="https://github.com/patchy631/time-to-first-token">As an AI engineer, please learn:</a></h3><p>Knowing a model&#8217;s benchmark score does not tell us how it will behave behind a busy API.</p><p><strong><a href="https://github.com/patchy631/time-to-first-token">We prepared a practical learning checklist</a></strong> for the work that comes after picking a model:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!DF8Q!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fced3b0d4-0826-4c6d-8760-cd30c9b85cd5_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!DF8Q!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fced3b0d4-0826-4c6d-8760-cd30c9b85cd5_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!DF8Q!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fced3b0d4-0826-4c6d-8760-cd30c9b85cd5_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!DF8Q!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fced3b0d4-0826-4c6d-8760-cd30c9b85cd5_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!DF8Q!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fced3b0d4-0826-4c6d-8760-cd30c9b85cd5_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!DF8Q!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fced3b0d4-0826-4c6d-8760-cd30c9b85cd5_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ced3b0d4-0826-4c6d-8760-cd30c9b85cd5_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:126709,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223845806?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fced3b0d4-0826-4c6d-8760-cd30c9b85cd5_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!DF8Q!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fced3b0d4-0826-4c6d-8760-cd30c9b85cd5_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!DF8Q!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fced3b0d4-0826-4c6d-8760-cd30c9b85cd5_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!DF8Q!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fced3b0d4-0826-4c6d-8760-cd30c9b85cd5_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!DF8Q!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fced3b0d4-0826-4c6d-8760-cd30c9b85cd5_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Small-batch decode is often memory-bound, but the bottleneck depends on the workload and hardware.</figcaption></figure></div><ul><li><p>Learn the roofline model and why small-batch decode is often memory-bound.</p></li><li><p>Deploy vLLM and SGLang, then read their schedulers.</p></li><li><p>Understand PagedAttention from the code, not only a blog post.</p></li><li><p>Build observability before optimizing anything.</p></li><li><p>Track time to first token, inter-token latency, throughput, and queue depth.</p></li><li><p>Use Grafana and Prometheus for inference dashboards.</p></li><li><p>Turn on prefix caching and find which workloads benefit. It reuses shared-prefix computation, not the generation of new tokens.</p></li><li><p>Learn continuous batching and chunked prefill.</p></li><li><p>Run load tests that sweep concurrency, including 1,000-plus concurrent requests where the hardware and workload allow it.</p></li><li><p>Report p50, p95, and p99, not only the mean.</p></li><li><p>Study quantization tradeoffs across FP8 and INT4 formats, and methods such as AWQ and GPTQ. Measure quality alongside memory use and speed.</p></li><li><p>Learn speculative decoding and where it stops helping.</p></li><li><p>Test a supported KV-cache eviction strategy for long contexts. Distinguish evicting reusable cached prefixes from discarding attention history that an active request still needs.</p></li><li><p>Try disaggregated prefill and decode serving.</p></li><li><p>Learn Kubernetes for AI workloads and use queue depth as an autoscaling signal.</p></li><li><p>Learn how inference costs affect unit economics.</p></li><li><p>Build a model router that considers cost, latency, and quality.</p></li><li><p>Create a token budget for each request.</p></li><li><p>Build one inference service and benchmark it publicly.</p></li><li><p>Read inference research rather than relying on model-release news.</p></li><li><p>Share your optimization benchmarks.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ZZws!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F413e52b6-a203-4e4e-a99b-b67240b2ae42_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ZZws!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F413e52b6-a203-4e4e-a99b-b67240b2ae42_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ZZws!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F413e52b6-a203-4e4e-a99b-b67240b2ae42_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ZZws!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F413e52b6-a203-4e4e-a99b-b67240b2ae42_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ZZws!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F413e52b6-a203-4e4e-a99b-b67240b2ae42_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ZZws!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F413e52b6-a203-4e4e-a99b-b67240b2ae42_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/413e52b6-a203-4e4e-a99b-b67240b2ae42_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:192774,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223845806?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F413e52b6-a203-4e4e-a99b-b67240b2ae42_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ZZws!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F413e52b6-a203-4e4e-a99b-b67240b2ae42_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ZZws!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F413e52b6-a203-4e4e-a99b-b67240b2ae42_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ZZws!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F413e52b6-a203-4e4e-a99b-b67240b2ae42_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ZZws!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F413e52b6-a203-4e4e-a99b-b67240b2ae42_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Roadmap builds on the same inference service over ten weeks.</figcaption></figure></div><p>We have turned this into a <strong><a href="https://github.com/patchy631/time-to-first-token">10-week roadmap on GitHub</a></strong>. It has 50 sessions of 30 minutes each, with five sessions and two buffer days per week.</p><p>The reading and build sessions feed one project. We deploy the service, instrument it, load test it, tune it, add a cost-aware router, and publish a reproducible benchmark.</p><p>Treat the timing as a study plan, not a guarantee that every deployment or load test fits into half an hour. The 1,000-request target is a load-testing exercise, not a claim that any GPU can serve that workload within a useful latency budget.</p><p>The repo welcomes contributions, including newer sources and benchmark writeups. The useful outcome is a service whose behavior we can measure and explain.</p><p><strong><a href="https://github.com/patchy631/time-to-first-token">Find our inference-serving roadmap here &#8594;</a></strong></p><p>Good day!</p>]]></content:encoded></item><item><title><![CDATA[Finding Every Vulnerability Made Security Harder]]></title><description><![CDATA[+ building an agent engineering team with Grok Bot!]]></description><link>https://blog.dailydoseofds.com/p/finding-every-vulnerability-made</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/finding-every-vulnerability-made</guid><dc:creator><![CDATA[Avi Chawla]]></dc:creator><pubDate>Thu, 05 Nov 2026 22:54:55 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!mgUA!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa45ec282-24a0-46b5-a64d-43262825d0d4_1990x1160.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<h3><a href="https://github.com/InsForge/InsForge">Build an agent engineering team with InsForge</a></h3><p>Giving four agents different job titles doesn&#8217;t tell the frontend agent what the backend actually built.</p><p>We created a four-bot engineering setup on Grok Bot, with <strong><a href="https://github.com/InsForge/InsForge">InsForge</a></strong> handling the backend. The useful part is how work moves between the bots.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!0VFe!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f86322b-8010-484d-a624-0e32f8d88174_1200x1150.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!0VFe!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f86322b-8010-484d-a624-0e32f8d88174_1200x1150.jpeg 424w, https://substackcdn.com/image/fetch/$s_!0VFe!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f86322b-8010-484d-a624-0e32f8d88174_1200x1150.jpeg 848w, https://substackcdn.com/image/fetch/$s_!0VFe!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f86322b-8010-484d-a624-0e32f8d88174_1200x1150.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!0VFe!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f86322b-8010-484d-a624-0e32f8d88174_1200x1150.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!0VFe!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f86322b-8010-484d-a624-0e32f8d88174_1200x1150.jpeg" width="1200" height="1150" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0f86322b-8010-484d-a624-0e32f8d88174_1200x1150.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1150,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:86194,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/221745929?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f86322b-8010-484d-a624-0e32f8d88174_1200x1150.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!0VFe!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f86322b-8010-484d-a624-0e32f8d88174_1200x1150.jpeg 424w, https://substackcdn.com/image/fetch/$s_!0VFe!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f86322b-8010-484d-a624-0e32f8d88174_1200x1150.jpeg 848w, https://substackcdn.com/image/fetch/$s_!0VFe!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f86322b-8010-484d-a624-0e32f8d88174_1200x1150.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!0VFe!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f86322b-8010-484d-a624-0e32f8d88174_1200x1150.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Our setup gives each bot a defined role and keeps project state on one shared computer.</figcaption></figure></div><p>The team has four roles:</p><ul><li><p>Piper, the project manager, breaks features into tasks and checks each handoff.</p></li><li><p>Bay, the backend developer, owns the database, APIs, and server logic.</p></li><li><p>Fin, the frontend developer, builds the interface against Bay&#8217;s documented contract.</p></li><li><p>Tess, the tester, checks requirements and returns defects to the responsible bot.</p></li></ul><p>All four share one cloud computer. Tasks, project context, and handoff notes live in files that the next bot can read.</p><p>Only Piper updates the task board. Each handoff records file paths, the interface contract, decisions, and unfinished work, so Fin doesn&#8217;t have to guess what Bay meant by &#8220;the endpoint is ready.&#8221;</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!dMMe!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73c9f9e8-86b6-47fb-bc7e-e561abc740af_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!dMMe!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73c9f9e8-86b6-47fb-bc7e-e561abc740af_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!dMMe!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73c9f9e8-86b6-47fb-bc7e-e561abc740af_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!dMMe!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73c9f9e8-86b6-47fb-bc7e-e561abc740af_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!dMMe!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73c9f9e8-86b6-47fb-bc7e-e561abc740af_2752x1536.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!dMMe!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73c9f9e8-86b6-47fb-bc7e-e561abc740af_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/73c9f9e8-86b6-47fb-bc7e-e561abc740af_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:139907,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/221745929?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73c9f9e8-86b6-47fb-bc7e-e561abc740af_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!dMMe!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73c9f9e8-86b6-47fb-bc7e-e561abc740af_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!dMMe!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73c9f9e8-86b6-47fb-bc7e-e561abc740af_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!dMMe!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73c9f9e8-86b6-47fb-bc7e-e561abc740af_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!dMMe!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73c9f9e8-86b6-47fb-bc7e-e561abc740af_2752x1536.jpeg 1456w" sizes="100vw"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A handoff file carries the details that the next agent needs to continue.</figcaption></figure></div><p>For Bay&#8217;s backend work, InsForge provides Postgres, authentication, storage, and edge functions. It&#8217;s open source under Apache 2.0.</p><p>Its cloud CLI lets an agent inspect backend state and configure resources. That includes reading schemas and metadata, creating storage buckets, running migrations, and deploying functions.</p><p>In our setup, Bay checks the current backend before starting a task. Otherwise, an incorrect assumption can enter the handoff note and become the frontend&#8217;s implementation plan.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!MHup!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f2a25c-9f7b-417f-bdd0-f6cc2442cfd5_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!MHup!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f2a25c-9f7b-417f-bdd0-f6cc2442cfd5_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!MHup!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f2a25c-9f7b-417f-bdd0-f6cc2442cfd5_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!MHup!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f2a25c-9f7b-417f-bdd0-f6cc2442cfd5_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!MHup!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f2a25c-9f7b-417f-bdd0-f6cc2442cfd5_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!MHup!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f2a25c-9f7b-417f-bdd0-f6cc2442cfd5_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d8f2a25c-9f7b-417f-bdd0-f6cc2442cfd5_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:122691,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/221745929?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f2a25c-9f7b-417f-bdd0-f6cc2442cfd5_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!MHup!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f2a25c-9f7b-417f-bdd0-f6cc2442cfd5_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!MHup!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f2a25c-9f7b-417f-bdd0-f6cc2442cfd5_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!MHup!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f2a25c-9f7b-417f-bdd0-f6cc2442cfd5_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!MHup!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f2a25c-9f7b-417f-bdd0-f6cc2442cfd5_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Bay can inspect and configure the InsForge backend through its cloud CLI.</figcaption></figure></div><p><strong><a href="https://github.com/InsForge/InsForge">Explore InsForge on GitHub &#8594;</a></strong></p><p>For the role prompts, setup steps, and a hands-on example, our article walks through the full workflow.</p><p><strong><a href="https://x.com/akshay_pachaar/status/2092664985197723937">Read &#8220;Build an Engineering Team With Grok Bot&#8221; &#8594;</a></strong></p><div><hr></div><h3><a href="https://coderabbit.link/dailydoseofds-003">Finding every vulnerability made security harder</a></h3><p>Claude Mythos Preview and other Claude models have flagged 29,439 candidate vulnerabilities in open-source projects through Anthropic&#8217;s Project Glasswing.</p><p>Each candidate claims that some input can reach a dangerous call. Someone still has to open the files, follow that input, and check whether a guard somewhere in between stops it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!mtLg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bef7a00-003b-4446-8516-a1c8ab0ac38b_2752x1536.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mtLg!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bef7a00-003b-4446-8516-a1c8ab0ac38b_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!mtLg!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bef7a00-003b-4446-8516-a1c8ab0ac38b_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!mtLg!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bef7a00-003b-4446-8516-a1c8ab0ac38b_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!mtLg!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bef7a00-003b-4446-8516-a1c8ab0ac38b_2752x1536.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!mtLg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bef7a00-003b-4446-8516-a1c8ab0ac38b_2752x1536.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0bef7a00-003b-4446-8516-a1c8ab0ac38b_2752x1536.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1241585,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/221745929?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bef7a00-003b-4446-8516-a1c8ab0ac38b_2752x1536.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!mtLg!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bef7a00-003b-4446-8516-a1c8ab0ac38b_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!mtLg!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bef7a00-003b-4446-8516-a1c8ab0ac38b_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!mtLg!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bef7a00-003b-4446-8516-a1c8ab0ac38b_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!mtLg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bef7a00-003b-4446-8516-a1c8ab0ac38b_2752x1536.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Anthropic names that human triage as the bottleneck. As of October 2, only 516 of the 6,157 findings it reported to maintainers had been patched.</p><p>Your own repository has the same problem at a smaller scale, and a scanner is only as useful as the evidence it gives you for each finding.</p><p>We wanted to see how much of that evidence a scanner can hand you, so we built a small local tool with Claude Code and ran CodeRabbit across the whole repository.</p><p><strong><a href="https://coderabbit.link/dailydoseofds-003">CodeRabbit</a></strong> is the governance and control layer for agentic software development. It helps developers understand and validate what their agents build so they can ship with confidence.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!mgUA!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa45ec282-24a0-46b5-a64d-43262825d0d4_1990x1160.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mgUA!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa45ec282-24a0-46b5-a64d-43262825d0d4_1990x1160.png 424w, https://substackcdn.com/image/fetch/$s_!mgUA!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa45ec282-24a0-46b5-a64d-43262825d0d4_1990x1160.png 848w, https://substackcdn.com/image/fetch/$s_!mgUA!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa45ec282-24a0-46b5-a64d-43262825d0d4_1990x1160.png 1272w, https://substackcdn.com/image/fetch/$s_!mgUA!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa45ec282-24a0-46b5-a64d-43262825d0d4_1990x1160.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!mgUA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa45ec282-24a0-46b5-a64d-43262825d0d4_1990x1160.png" width="1456" height="849" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a45ec282-24a0-46b5-a64d-43262825d0d4_1990x1160.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:849,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:401212,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/221745929?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa45ec282-24a0-46b5-a64d-43262825d0d4_1990x1160.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!mgUA!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa45ec282-24a0-46b5-a64d-43262825d0d4_1990x1160.png 424w, https://substackcdn.com/image/fetch/$s_!mgUA!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa45ec282-24a0-46b5-a64d-43262825d0d4_1990x1160.png 848w, https://substackcdn.com/image/fetch/$s_!mgUA!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa45ec282-24a0-46b5-a64d-43262825d0d4_1990x1160.png 1272w, https://substackcdn.com/image/fetch/$s_!mgUA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa45ec282-24a0-46b5-a64d-43262825d0d4_1990x1160.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><strong><a href="https://coderabbit.link/dailydoseofds-003">CodeRabbit Secure</a></strong> extends that to the full repository. It reasons across the codebase to find vulnerabilities, and a separate agent challenges every finding before it reaches you, so you can eventually craft a vulnerability fix with CodeRabbit&#8217;s Fix with AI feature.</p><p>We modeled the test on a real bug that shipped in OpenCode this year.</p><h4>How the OpenCode bug got in</h4><p><strong><a href="https://github.com/anomalyco/opencode">OpenCode</a></strong> is an open-source coding agent with 224k+ GitHub stars. When you run <code>opencode serve</code>, it starts a local API on <code>127.0.0.1:4096</code>. The server has no authentication by default, so it acts on any request that reaches it.</p><p>Among the requests it accepts is one for updates. You send it a version, and the server installs that version for you. The catch is in how the server installs.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!0IXD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccf23a69-8b39-4224-82b7-0c06d7121f7a_2752x1536.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!0IXD!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccf23a69-8b39-4224-82b7-0c06d7121f7a_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!0IXD!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccf23a69-8b39-4224-82b7-0c06d7121f7a_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!0IXD!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccf23a69-8b39-4224-82b7-0c06d7121f7a_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!0IXD!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccf23a69-8b39-4224-82b7-0c06d7121f7a_2752x1536.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!0IXD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccf23a69-8b39-4224-82b7-0c06d7121f7a_2752x1536.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ccf23a69-8b39-4224-82b7-0c06d7121f7a_2752x1536.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:914836,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/221745929?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccf23a69-8b39-4224-82b7-0c06d7121f7a_2752x1536.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!0IXD!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccf23a69-8b39-4224-82b7-0c06d7121f7a_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!0IXD!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccf23a69-8b39-4224-82b7-0c06d7121f7a_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!0IXD!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccf23a69-8b39-4224-82b7-0c06d7121f7a_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!0IXD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccf23a69-8b39-4224-82b7-0c06d7121f7a_2752x1536.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The installer pulls from any location you give it, and installing a package runs its setup code. So whoever picks the version picks the code that runs on your machine.</p><p>You might think this is safe since the server only listens on your computer. But any web page you open can make your browser send a request to that local address.</p><p>Browsers restrict what a page can read from another origin, but some cross-origin requests can still be sent. A plain, old-style form submission can reach the update endpoint.</p><p>The form only worked because the server never checked what kind of data it received. The form sent plain text, and the server read it as JSON anyway.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!KJCF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90fc606d-a206-4775-8a28-570293a2f9c4_2752x1536.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!KJCF!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90fc606d-a206-4775-8a28-570293a2f9c4_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!KJCF!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90fc606d-a206-4775-8a28-570293a2f9c4_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!KJCF!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90fc606d-a206-4775-8a28-570293a2f9c4_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!KJCF!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90fc606d-a206-4775-8a28-570293a2f9c4_2752x1536.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!KJCF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90fc606d-a206-4775-8a28-570293a2f9c4_2752x1536.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/90fc606d-a206-4775-8a28-570293a2f9c4_2752x1536.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1324561,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/221745929?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90fc606d-a206-4775-8a28-570293a2f9c4_2752x1536.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!KJCF!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90fc606d-a206-4775-8a28-570293a2f9c4_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!KJCF!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90fc606d-a206-4775-8a28-570293a2f9c4_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!KJCF!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90fc606d-a206-4775-8a28-570293a2f9c4_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!KJCF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90fc606d-a206-4775-8a28-570293a2f9c4_2752x1536.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>That check used to be in OpenCode&#8217;s server. An earlier version of its code rejected anything that wasn&#8217;t JSON. But when OpenCode rebuilt that part on a new framework, the check didn&#8217;t come along.</p><p>The flawed code shipped on April 30. Before a fix landed, those versions were downloaded more than 647,000 times.</p><p>Yet no single file looked wrong. The danger only showed up when you followed that version from the request to the install.</p><h4>The tool we built</h4><p>You probably run a few local servers like OpenCode already, such as model runners, MCP servers, and coding agents. We had Claude Code build one of these servers so we could test whether a scanner would catch the same kind of bug.</p><p>We gave Claude Code a single prompt asking for a local dashboard to manage LLM tooling, running on FastAPI with a small HTML frontend.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!07jV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6040818-3d58-4104-8a21-cd00ca3e00f3_2302x1740.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!07jV!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6040818-3d58-4104-8a21-cd00ca3e00f3_2302x1740.png 424w, https://substackcdn.com/image/fetch/$s_!07jV!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6040818-3d58-4104-8a21-cd00ca3e00f3_2302x1740.png 848w, https://substackcdn.com/image/fetch/$s_!07jV!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6040818-3d58-4104-8a21-cd00ca3e00f3_2302x1740.png 1272w, https://substackcdn.com/image/fetch/$s_!07jV!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6040818-3d58-4104-8a21-cd00ca3e00f3_2302x1740.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!07jV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6040818-3d58-4104-8a21-cd00ca3e00f3_2302x1740.png" width="1456" height="1101" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d6040818-3d58-4104-8a21-cd00ca3e00f3_2302x1740.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1101,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:2382633,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/221745929?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6040818-3d58-4104-8a21-cd00ca3e00f3_2302x1740.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!07jV!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6040818-3d58-4104-8a21-cd00ca3e00f3_2302x1740.png 424w, https://substackcdn.com/image/fetch/$s_!07jV!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6040818-3d58-4104-8a21-cd00ca3e00f3_2302x1740.png 848w, https://substackcdn.com/image/fetch/$s_!07jV!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6040818-3d58-4104-8a21-cd00ca3e00f3_2302x1740.png 1272w, https://substackcdn.com/image/fetch/$s_!07jV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6040818-3d58-4104-8a21-cd00ca3e00f3_2302x1740.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The dashboard installs plugins by name or version, pulls models through the local model CLI, and stores provider API keys. It also runs an agent that summarizes a URL or a workspace file, with tools to fetch a page and read files.</p><p>Claude Code produced about 2,150 lines across two dozen modules. The dashboard ran, and all 90 tests passed.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!N5FH!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5b40e10-7ea2-4724-bd35-8d8dd6bddece_3114x1994.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!N5FH!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5b40e10-7ea2-4724-bd35-8d8dd6bddece_3114x1994.png 424w, https://substackcdn.com/image/fetch/$s_!N5FH!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5b40e10-7ea2-4724-bd35-8d8dd6bddece_3114x1994.png 848w, https://substackcdn.com/image/fetch/$s_!N5FH!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5b40e10-7ea2-4724-bd35-8d8dd6bddece_3114x1994.png 1272w, https://substackcdn.com/image/fetch/$s_!N5FH!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5b40e10-7ea2-4724-bd35-8d8dd6bddece_3114x1994.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!N5FH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5b40e10-7ea2-4724-bd35-8d8dd6bddece_3114x1994.png" width="1456" height="932" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d5b40e10-7ea2-4724-bd35-8d8dd6bddece_3114x1994.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:932,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:564562,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/221745929?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5b40e10-7ea2-4724-bd35-8d8dd6bddece_3114x1994.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!N5FH!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5b40e10-7ea2-4724-bd35-8d8dd6bddece_3114x1994.png 424w, https://substackcdn.com/image/fetch/$s_!N5FH!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5b40e10-7ea2-4724-bd35-8d8dd6bddece_3114x1994.png 848w, https://substackcdn.com/image/fetch/$s_!N5FH!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5b40e10-7ea2-4724-bd35-8d8dd6bddece_3114x1994.png 1272w, https://substackcdn.com/image/fetch/$s_!N5FH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5b40e10-7ea2-4724-bd35-8d8dd6bddece_3114x1994.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>We added one module myself. The module recreates the OpenCode bug in Python, with a self-update endpoint that reads incoming data as JSON without checking its type. The endpoint then passes the requested version straight to <code>pip install</code>.</p><p>The handler and the installer live in separate files, as they did in OpenCode, and the repository holds no hints for the scanner. Everything else is the code Claude Code generated.</p><h4>Scanning the whole repository</h4><p>A pull request review focuses on the changes in the diff, with context from the codebase. Code already on main may not show up in a new diff.</p><p>CodeRabbit covers new code and committed code separately. PR security reviews check pull requests CodeRabbit is configured to review, and Deep Scans investigate the committed codebase on demand or, where supported, on a schedule.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!jXkl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd0cbeff-aa34-4ba1-9faa-2e1277caad4b_3108x1990.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!jXkl!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd0cbeff-aa34-4ba1-9faa-2e1277caad4b_3108x1990.png 424w, https://substackcdn.com/image/fetch/$s_!jXkl!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd0cbeff-aa34-4ba1-9faa-2e1277caad4b_3108x1990.png 848w, https://substackcdn.com/image/fetch/$s_!jXkl!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd0cbeff-aa34-4ba1-9faa-2e1277caad4b_3108x1990.png 1272w, https://substackcdn.com/image/fetch/$s_!jXkl!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd0cbeff-aa34-4ba1-9faa-2e1277caad4b_3108x1990.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!jXkl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd0cbeff-aa34-4ba1-9faa-2e1277caad4b_3108x1990.png" width="1456" height="932" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/cd0cbeff-aa34-4ba1-9faa-2e1277caad4b_3108x1990.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:932,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:687411,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/221745929?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd0cbeff-aa34-4ba1-9faa-2e1277caad4b_3108x1990.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!jXkl!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd0cbeff-aa34-4ba1-9faa-2e1277caad4b_3108x1990.png 424w, https://substackcdn.com/image/fetch/$s_!jXkl!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd0cbeff-aa34-4ba1-9faa-2e1277caad4b_3108x1990.png 848w, https://substackcdn.com/image/fetch/$s_!jXkl!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd0cbeff-aa34-4ba1-9faa-2e1277caad4b_3108x1990.png 1272w, https://substackcdn.com/image/fetch/$s_!jXkl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd0cbeff-aa34-4ba1-9faa-2e1277caad4b_3108x1990.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>We ran a Deep Scan on main without giving CodeRabbit any notes about the code. CodeRabbit first maps the application, including the places where requests come in and the controls along their paths.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!IpE9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a90332a-6310-4c47-9cba-f939b87d37d7_2076x1824.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!IpE9!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a90332a-6310-4c47-9cba-f939b87d37d7_2076x1824.png 424w, https://substackcdn.com/image/fetch/$s_!IpE9!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a90332a-6310-4c47-9cba-f939b87d37d7_2076x1824.png 848w, https://substackcdn.com/image/fetch/$s_!IpE9!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a90332a-6310-4c47-9cba-f939b87d37d7_2076x1824.png 1272w, https://substackcdn.com/image/fetch/$s_!IpE9!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a90332a-6310-4c47-9cba-f939b87d37d7_2076x1824.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!IpE9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a90332a-6310-4c47-9cba-f939b87d37d7_2076x1824.png" width="1456" height="1279" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3a90332a-6310-4c47-9cba-f939b87d37d7_2076x1824.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1279,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:695717,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/221745929?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a90332a-6310-4c47-9cba-f939b87d37d7_2076x1824.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!IpE9!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a90332a-6310-4c47-9cba-f939b87d37d7_2076x1824.png 424w, https://substackcdn.com/image/fetch/$s_!IpE9!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a90332a-6310-4c47-9cba-f939b87d37d7_2076x1824.png 848w, https://substackcdn.com/image/fetch/$s_!IpE9!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a90332a-6310-4c47-9cba-f939b87d37d7_2076x1824.png 1272w, https://substackcdn.com/image/fetch/$s_!IpE9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a90332a-6310-4c47-9cba-f939b87d37d7_2076x1824.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The scan then traces potential paths from entry points toward security-sensitive calls, following them across files. CodeRabbit checks the evidence for candidate findings before reporting them.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!qAiu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F501d6cd3-7bfe-45eb-9898-963318b68972_2586x1936.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!qAiu!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F501d6cd3-7bfe-45eb-9898-963318b68972_2586x1936.png 424w, https://substackcdn.com/image/fetch/$s_!qAiu!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F501d6cd3-7bfe-45eb-9898-963318b68972_2586x1936.png 848w, https://substackcdn.com/image/fetch/$s_!qAiu!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F501d6cd3-7bfe-45eb-9898-963318b68972_2586x1936.png 1272w, https://substackcdn.com/image/fetch/$s_!qAiu!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F501d6cd3-7bfe-45eb-9898-963318b68972_2586x1936.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!qAiu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F501d6cd3-7bfe-45eb-9898-963318b68972_2586x1936.png" width="1456" height="1090" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/501d6cd3-7bfe-45eb-9898-963318b68972_2586x1936.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1090,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:740852,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/221745929?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F501d6cd3-7bfe-45eb-9898-963318b68972_2586x1936.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!qAiu!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F501d6cd3-7bfe-45eb-9898-963318b68972_2586x1936.png 424w, https://substackcdn.com/image/fetch/$s_!qAiu!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F501d6cd3-7bfe-45eb-9898-963318b68972_2586x1936.png 848w, https://substackcdn.com/image/fetch/$s_!qAiu!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F501d6cd3-7bfe-45eb-9898-963318b68972_2586x1936.png 1272w, https://substackcdn.com/image/fetch/$s_!qAiu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F501d6cd3-7bfe-45eb-9898-963318b68972_2586x1936.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The scan returned six findings, five rated medium and one low. The Security Dashboard gathered them in one place, and the secrets scan came back clean.</p><h4>Tracing the injection finding</h4><p>We started with the injection in the self-update installer, rated medium, since it maps directly to the OpenCode bug.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!79Qr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2800acbf-706c-4e59-aedc-e5ac278c5463_1898x1972.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!79Qr!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2800acbf-706c-4e59-aedc-e5ac278c5463_1898x1972.png 424w, https://substackcdn.com/image/fetch/$s_!79Qr!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2800acbf-706c-4e59-aedc-e5ac278c5463_1898x1972.png 848w, https://substackcdn.com/image/fetch/$s_!79Qr!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2800acbf-706c-4e59-aedc-e5ac278c5463_1898x1972.png 1272w, https://substackcdn.com/image/fetch/$s_!79Qr!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2800acbf-706c-4e59-aedc-e5ac278c5463_1898x1972.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!79Qr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2800acbf-706c-4e59-aedc-e5ac278c5463_1898x1972.png" width="1456" height="1513" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2800acbf-706c-4e59-aedc-e5ac278c5463_1898x1972.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1513,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:610256,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/221745929?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2800acbf-706c-4e59-aedc-e5ac278c5463_1898x1972.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!79Qr!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2800acbf-706c-4e59-aedc-e5ac278c5463_1898x1972.png 424w, https://substackcdn.com/image/fetch/$s_!79Qr!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2800acbf-706c-4e59-aedc-e5ac278c5463_1898x1972.png 848w, https://substackcdn.com/image/fetch/$s_!79Qr!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2800acbf-706c-4e59-aedc-e5ac278c5463_1898x1972.png 1272w, https://substackcdn.com/image/fetch/$s_!79Qr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2800acbf-706c-4e59-aedc-e5ac278c5463_1898x1972.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The trace began at an unauthenticated POST to <code>/api/update</code>. It carried the <code>target</code> field from the request into a second file, where the value reached <code>pip install</code>.</p><p>Because nothing checks the target, an attacker can make pip ignore the real package name and install their own code instead. pip then runs that code&#8217;s build step under the same user account as the dashboard.</p><p>The bug did not live in one file. The handler parsed the request in one place, and the installer ran the command in another. The finding connected those two steps.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!97_h!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19079a64-3d1f-4a69-a178-5abeafb4a18f_1906x1972.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!97_h!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19079a64-3d1f-4a69-a178-5abeafb4a18f_1906x1972.png 424w, https://substackcdn.com/image/fetch/$s_!97_h!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19079a64-3d1f-4a69-a178-5abeafb4a18f_1906x1972.png 848w, https://substackcdn.com/image/fetch/$s_!97_h!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19079a64-3d1f-4a69-a178-5abeafb4a18f_1906x1972.png 1272w, https://substackcdn.com/image/fetch/$s_!97_h!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19079a64-3d1f-4a69-a178-5abeafb4a18f_1906x1972.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!97_h!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19079a64-3d1f-4a69-a178-5abeafb4a18f_1906x1972.png" width="1456" height="1506" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/19079a64-3d1f-4a69-a178-5abeafb4a18f_1906x1972.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1506,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:630163,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/221745929?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19079a64-3d1f-4a69-a178-5abeafb4a18f_1906x1972.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!97_h!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19079a64-3d1f-4a69-a178-5abeafb4a18f_1906x1972.png 424w, https://substackcdn.com/image/fetch/$s_!97_h!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19079a64-3d1f-4a69-a178-5abeafb4a18f_1906x1972.png 848w, https://substackcdn.com/image/fetch/$s_!97_h!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19079a64-3d1f-4a69-a178-5abeafb4a18f_1906x1972.png 1272w, https://substackcdn.com/image/fetch/$s_!97_h!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19079a64-3d1f-4a69-a178-5abeafb4a18f_1906x1972.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>A second finding, rated low, flagged the same endpoint for cross-site request forgery. The handler never checks which site a request came from, which is what lets a page you visit reach it.</p><p>We reproduced the attack on my own machine. A page served from a different port sent a plain-text form to the local dashboard, and the request reached the installer with my chosen target.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!P0q1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7186773-cd6f-4c7e-a7bc-cbf21b342e54_2480x1928.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!P0q1!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7186773-cd6f-4c7e-a7bc-cbf21b342e54_2480x1928.png 424w, https://substackcdn.com/image/fetch/$s_!P0q1!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7186773-cd6f-4c7e-a7bc-cbf21b342e54_2480x1928.png 848w, https://substackcdn.com/image/fetch/$s_!P0q1!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7186773-cd6f-4c7e-a7bc-cbf21b342e54_2480x1928.png 1272w, https://substackcdn.com/image/fetch/$s_!P0q1!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7186773-cd6f-4c7e-a7bc-cbf21b342e54_2480x1928.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!P0q1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7186773-cd6f-4c7e-a7bc-cbf21b342e54_2480x1928.png" width="1456" height="1132" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a7186773-cd6f-4c7e-a7bc-cbf21b342e54_2480x1928.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1132,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:675958,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/221745929?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7186773-cd6f-4c7e-a7bc-cbf21b342e54_2480x1928.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!P0q1!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7186773-cd6f-4c7e-a7bc-cbf21b342e54_2480x1928.png 424w, https://substackcdn.com/image/fetch/$s_!P0q1!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7186773-cd6f-4c7e-a7bc-cbf21b342e54_2480x1928.png 848w, https://substackcdn.com/image/fetch/$s_!P0q1!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7186773-cd6f-4c7e-a7bc-cbf21b342e54_2480x1928.png 1272w, https://substackcdn.com/image/fetch/$s_!P0q1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7186773-cd6f-4c7e-a7bc-cbf21b342e54_2480x1928.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h4>How each finding proves itself</h4><p>Besides the two findings we traced above, a third flagged the origin check we had removed to match OpenCode. The other three were in code Claude Code wrote.</p><p>One was a server-side request forgery in the agent&#8217;s page fetcher. The fetcher checks a URL against a single DNS lookup, then lets the HTTP client resolve the hostname again when it connects.</p><p>A changing DNS answer can redirect that second request to a private address. CodeRabbit followed the path across four files, from the agent route through the loop into the tool.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!5hB8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ae3e479-bb45-4909-8c65-af3a32d9d4d4_1932x2006.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!5hB8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ae3e479-bb45-4909-8c65-af3a32d9d4d4_1932x2006.png 424w, https://substackcdn.com/image/fetch/$s_!5hB8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ae3e479-bb45-4909-8c65-af3a32d9d4d4_1932x2006.png 848w, https://substackcdn.com/image/fetch/$s_!5hB8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ae3e479-bb45-4909-8c65-af3a32d9d4d4_1932x2006.png 1272w, https://substackcdn.com/image/fetch/$s_!5hB8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ae3e479-bb45-4909-8c65-af3a32d9d4d4_1932x2006.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!5hB8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ae3e479-bb45-4909-8c65-af3a32d9d4d4_1932x2006.png" width="1456" height="1512" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0ae3e479-bb45-4909-8c65-af3a32d9d4d4_1932x2006.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1512,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:639382,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/221745929?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ae3e479-bb45-4909-8c65-af3a32d9d4d4_1932x2006.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!5hB8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ae3e479-bb45-4909-8c65-af3a32d9d4d4_1932x2006.png 424w, https://substackcdn.com/image/fetch/$s_!5hB8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ae3e479-bb45-4909-8c65-af3a32d9d4d4_1932x2006.png 848w, https://substackcdn.com/image/fetch/$s_!5hB8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ae3e479-bb45-4909-8c65-af3a32d9d4d4_1932x2006.png 1272w, https://substackcdn.com/image/fetch/$s_!5hB8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ae3e479-bb45-4909-8c65-af3a32d9d4d4_1932x2006.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The other two were a denial of service in the same fetcher and a settings endpoint that stored provider configuration without an authorization check.</p><p>CodeRabbit returned each finding with the entry point, the path through the code, whether the route was reachable, and the conditions needed to exploit it.</p><p>Those details let me check each finding against the code instead of investigating from scratch.</p><p>The scan also flagged where it could only reason within this repository and said further checks would need external context.</p><h4>Fixing what the scan confirmed</h4><p>For the injection finding, CodeRabbit drafted a fix and opened it as a pull request on a security branch.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!5E1q!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F185c4bfa-64b5-4c6a-b1b4-8eec26363b11_2716x2024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!5E1q!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F185c4bfa-64b5-4c6a-b1b4-8eec26363b11_2716x2024.png 424w, https://substackcdn.com/image/fetch/$s_!5E1q!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F185c4bfa-64b5-4c6a-b1b4-8eec26363b11_2716x2024.png 848w, https://substackcdn.com/image/fetch/$s_!5E1q!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F185c4bfa-64b5-4c6a-b1b4-8eec26363b11_2716x2024.png 1272w, https://substackcdn.com/image/fetch/$s_!5E1q!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F185c4bfa-64b5-4c6a-b1b4-8eec26363b11_2716x2024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!5E1q!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F185c4bfa-64b5-4c6a-b1b4-8eec26363b11_2716x2024.png" width="1456" height="1085" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/185c4bfa-64b5-4c6a-b1b4-8eec26363b11_2716x2024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1085,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:845348,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/221745929?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F185c4bfa-64b5-4c6a-b1b4-8eec26363b11_2716x2024.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!5E1q!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F185c4bfa-64b5-4c6a-b1b4-8eec26363b11_2716x2024.png 424w, https://substackcdn.com/image/fetch/$s_!5E1q!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F185c4bfa-64b5-4c6a-b1b4-8eec26363b11_2716x2024.png 848w, https://substackcdn.com/image/fetch/$s_!5E1q!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F185c4bfa-64b5-4c6a-b1b4-8eec26363b11_2716x2024.png 1272w, https://substackcdn.com/image/fetch/$s_!5E1q!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F185c4bfa-64b5-4c6a-b1b4-8eec26363b11_2716x2024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The fix validates the target before calling pip, rejecting file paths, URLs, and anything else pip could treat as a different package.</p><p>The pull request also adds regression tests for the injection attempts and invalid inputs, and it keeps the evidence with the patch, so we reviewed the fix against the exact attack path.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!QuTo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13ce54ef-4bbd-4f93-a039-ee5d4840dd3d_2052x1470.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!QuTo!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13ce54ef-4bbd-4f93-a039-ee5d4840dd3d_2052x1470.png 424w, https://substackcdn.com/image/fetch/$s_!QuTo!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13ce54ef-4bbd-4f93-a039-ee5d4840dd3d_2052x1470.png 848w, https://substackcdn.com/image/fetch/$s_!QuTo!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13ce54ef-4bbd-4f93-a039-ee5d4840dd3d_2052x1470.png 1272w, https://substackcdn.com/image/fetch/$s_!QuTo!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13ce54ef-4bbd-4f93-a039-ee5d4840dd3d_2052x1470.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!QuTo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13ce54ef-4bbd-4f93-a039-ee5d4840dd3d_2052x1470.png" width="1456" height="1043" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/13ce54ef-4bbd-4f93-a039-ee5d4840dd3d_2052x1470.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1043,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:610109,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/221745929?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13ce54ef-4bbd-4f93-a039-ee5d4840dd3d_2052x1470.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!QuTo!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13ce54ef-4bbd-4f93-a039-ee5d4840dd3d_2052x1470.png 424w, https://substackcdn.com/image/fetch/$s_!QuTo!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13ce54ef-4bbd-4f93-a039-ee5d4840dd3d_2052x1470.png 848w, https://substackcdn.com/image/fetch/$s_!QuTo!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13ce54ef-4bbd-4f93-a039-ee5d4840dd3d_2052x1470.png 1272w, https://substackcdn.com/image/fetch/$s_!QuTo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13ce54ef-4bbd-4f93-a039-ee5d4840dd3d_2052x1470.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>With the target validated, the crafted input is rejected before pip runs. OpenCode&#8217;s fix in 1.18.22 made the same move, requiring the target to be a real version number.</p><p>You can start a free 14-day CodeRabbit trial, with no credit card required. Deep Scan is billed separately, but CodeRabbit covers <strong><a href="https://www.coderabbit.ai/blog/one-million-security-scans-and-fixes">$1 million in security scans and fixes through November 8, 2026</a></strong>.</p><p><strong><a href="https://coderabbit.link/dailydoseofds-003">Ship agent-built code with confidence &#8594;</a></strong></p><p><em>Thanks to CodeRabbit for partnering today!</em></p>]]></content:encoded></item><item><title><![CDATA[Grok Bot Masterclass]]></title><description><![CDATA[...covered with a full hands-on video demo.]]></description><link>https://blog.dailydoseofds.com/p/grok-bot-masterclass-081</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/grok-bot-masterclass-081</guid><dc:creator><![CDATA[Avi Chawla]]></dc:creator><pubDate>Wed, 04 Nov 2026 23:24:08 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/9d1cb505-4873-4ddc-90d1-75286ab7a31e_680x272.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><strong>In today&#8217;s newsletter:</strong></p><ul><li><p>Benchmark agent harnesses side by side.</p></li><li><p>Grok Bot masterclass: from 1 bot to a 14-bot org.</p></li></ul><div><hr></div><h3><a href="https://github.com/HarnessRouter/harnessrouter">Benchmark agent harnesses side by side!</a></h3><p>Most agent benchmarks compare models. But in production, the model runs inside a harness, and the harness changes cost, speed, and whether the task passes at all.</p><p>Harness Arena, inside HarnessRouter, benchmarks the harness and not just the model.</p><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;0894ad01-c878-4975-b337-8e5b1974c55a&quot;,&quot;duration&quot;:null}"></div><p>It runs one task across several harness &#215; model configurations, and every config gets the same prompt, files, tools, skills, and evaluation checks.</p><p>For each config, it returns:</p><p>&#8627; Total cost.</p><p>&#8627; Execution time.</p><p>&#8627; Whether the task passed or failed.</p><p>&#8627; The files and artifacts it produced.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!4deC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9489ad7-9da2-4a10-a4e4-ef72916b4687_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!4deC!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9489ad7-9da2-4a10-a4e4-ef72916b4687_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!4deC!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9489ad7-9da2-4a10-a4e4-ef72916b4687_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!4deC!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9489ad7-9da2-4a10-a4e4-ef72916b4687_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!4deC!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9489ad7-9da2-4a10-a4e4-ef72916b4687_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!4deC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9489ad7-9da2-4a10-a4e4-ef72916b4687_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b9489ad7-9da2-4a10-a4e4-ef72916b4687_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:165788,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223510481?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9489ad7-9da2-4a10-a4e4-ef72916b4687_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!4deC!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9489ad7-9da2-4a10-a4e4-ef72916b4687_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!4deC!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9489ad7-9da2-4a10-a4e4-ef72916b4687_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!4deC!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9489ad7-9da2-4a10-a4e4-ef72916b4687_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!4deC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9489ad7-9da2-4a10-a4e4-ef72916b4687_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Every config gets the same inputs and checks, so the runtime is the only thing that changes.</figcaption></figure></div><p>HarnessRouter supports Codex, Hermes, Claude Code, DeepSeek Harness, System One (powered by Jev), and 25 more.</p><p>GitHub repo: <strong><a href="https://github.com/HarnessRouter/harnessrouter">https://github.com/HarnessRouter/harnessrouter</a></strong></p><p>(don&#8217;t forget to star it &#11088;)</p><p>One thing we like about the design is that the cheapest run does not automatically win.</p><p>A config first has to pass the task. Only then does Arena compare cost and execution time.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!rcoo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a403272-4a79-432e-82e5-c93325487a89_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!rcoo!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a403272-4a79-432e-82e5-c93325487a89_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!rcoo!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a403272-4a79-432e-82e5-c93325487a89_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!rcoo!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a403272-4a79-432e-82e5-c93325487a89_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!rcoo!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a403272-4a79-432e-82e5-c93325487a89_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!rcoo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a403272-4a79-432e-82e5-c93325487a89_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0a403272-4a79-432e-82e5-c93325487a89_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:152774,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223510481?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a403272-4a79-432e-82e5-c93325487a89_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!rcoo!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a403272-4a79-432e-82e5-c93325487a89_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!rcoo!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a403272-4a79-432e-82e5-c93325487a89_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!rcoo!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a403272-4a79-432e-82e5-c93325487a89_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!rcoo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0a403272-4a79-432e-82e5-c93325487a89_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A cheap run that fails the task is not a bargain, so correctness comes first.</figcaption></figure></div><p>In one published result:</p><p>&#8627; Hermes + GPT-5.2 passed 5/5 runs for 0.47 credits.</p><p>&#8627; Claude Code + Opus 4.8 passed 3/5 runs for 223 credits.</p><p>That is about 475x more credits for the config that also failed more often.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!KwCE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07abb77b-b01f-45b1-b6c3-dd60bd9ce167_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!KwCE!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07abb77b-b01f-45b1-b6c3-dd60bd9ce167_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!KwCE!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07abb77b-b01f-45b1-b6c3-dd60bd9ce167_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!KwCE!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07abb77b-b01f-45b1-b6c3-dd60bd9ce167_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!KwCE!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07abb77b-b01f-45b1-b6c3-dd60bd9ce167_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!KwCE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07abb77b-b01f-45b1-b6c3-dd60bd9ce167_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/07abb77b-b01f-45b1-b6c3-dd60bd9ce167_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:131677,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223510481?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07abb77b-b01f-45b1-b6c3-dd60bd9ce167_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!KwCE!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07abb77b-b01f-45b1-b6c3-dd60bd9ce167_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!KwCE!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07abb77b-b01f-45b1-b6c3-dd60bd9ce167_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!KwCE!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07abb77b-b01f-45b1-b6c3-dd60bd9ce167_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!KwCE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07abb77b-b01f-45b1-b6c3-dd60bd9ce167_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The pricier config lost on cost and on reliability for this task.</figcaption></figure></div><p>You can keep the model fixed and compare harnesses, or vary both. Either way, Arena answers practical questions:</p><p>&#8627; Should this task use Codex or Claude Code?</p><p>&#8627; Can a cheaper config still pass?</p><p>&#8627; Which setup works best for this workflow?</p><p>Once you pick a setup, you run it through the same UHP interface, with the same sessions, streaming, files, artifacts, cancellation, and failure handling.</p><p>So the comparison covers the whole runtime, not a model leaderboard.</p><p><a href="https://github.com/HarnessRouter/harnessrouter">&#8203;</a><strong><a href="https://github.com/HarnessRouter/harnessrouter">Here's the repo &#8594;</a></strong></p><div><hr></div><h3><a href="https://www.dailydoseofds.com/p/grok-bot-masterclass/">Grok Bot masterclass: from 1 bot to a 14-bot org</a></h3><p>We spent 100+ hours over two weeks building with Grok Bot. We now run 14 bots and talk to only 4 of them.</p><p>We recorded a 38-minute masterclass that covers all of it, from the first bot to a working org.</p><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;86b00c18-803c-4921-a1af-ec7f06a75219&quot;,&quot;duration&quot;:null}"></div><p>Here&#8217;s what the video covers.</p><ul><li><p>00:00 - Intro</p></li><li><p>01:22 - What a Grok Bot really is (a new hire with a laptop)</p></li><li><p>02:23 - Every bot gets its own computer</p></li><li><p>03:35 - Installation and pricing</p></li><li><p>05:01 - Creating your first bot (no system prompt to write)</p></li><li><p>06:29 - Routines: tasks your bot runs on a schedule</p></li><li><p>07:30 - Computer use and connected apps</p></li><li><p>11:50 - Skills: a recipe card for your AI</p></li><li><p>13:07 - Teach a Task: turning a screen recording into a skill</p></li><li><p>18:38 - One fat bot vs. 14 bots (and why both fail)</p></li><li><p>19:39 - Run it like a company: you, executives, workers</p></li><li><p>21:00 - The Notion task board that tracks every bot</p></li><li><p>21:31 - Designing the org with Claude</p></li><li><p>23:11 - Why my bots are named after TV characters</p></li><li><p>23:50 - Leadership in one group chat</p></li><li><p>26:31 - Chief of staff: inbox sweep across Gmail, Slack and X</p></li><li><p>27:10 - Head of content: post analysis and competitor watch</p></li><li><p>29:44 - Design team: on-brand illustrations and Typefully drafts</p></li><li><p>32:46 - Engineering team: GitHub issue to PR with one label</p></li><li><p>35:17 - Claude Code on your bot&#8217;s computer</p></li><li><p>37:05 - Recap</p></li><li><p>38:12 - Outro</p></li></ul><p>&#128073; Over to you: Which job would you hand to your first bot?</p><p>Good day!</p>]]></content:encoded></item><item><title><![CDATA[Agent Framework vs. Agent Harness, clearly explained!]]></title><description><![CDATA[+ a technique for 23x faster attention!]]></description><link>https://blog.dailydoseofds.com/p/agent-framework-vs-agent-harness</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/agent-framework-vs-agent-harness</guid><dc:creator><![CDATA[Avi Chawla]]></dc:creator><pubDate>Tue, 03 Nov 2026 21:36:04 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!Pwck!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e36d4fa-98f8-40dc-a6d0-f2925369482c_900x900.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><strong>In today&#8217;s newsletter:</strong></p><ul><li><p>LLM routing can cost more than not routing at all.</p></li><li><p>Agent framework vs. agent harness, clearly explained.</p></li><li><p>A new technique just made attention 23x faster.</p></li></ul><div><hr></div><h3><a href="https://do.co/4i8JBrO">LLM routing can cost more than not routing at all</a></h3><p>The basic idea behind LLM routing sounds simple, i.e., send easy requests to cheaper models and save frontier models for hard tasks.</p><p>Production routing is rarely that simple.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!BBBU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c8e2dd2-5bf9-419b-a440-89ddc7c4ce2f_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!BBBU!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c8e2dd2-5bf9-419b-a440-89ddc7c4ce2f_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!BBBU!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c8e2dd2-5bf9-419b-a440-89ddc7c4ce2f_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!BBBU!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c8e2dd2-5bf9-419b-a440-89ddc7c4ce2f_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!BBBU!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c8e2dd2-5bf9-419b-a440-89ddc7c4ce2f_2752x1536.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!BBBU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c8e2dd2-5bf9-419b-a440-89ddc7c4ce2f_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0c8e2dd2-5bf9-419b-a440-89ddc7c4ce2f_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:145077,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223382039?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c8e2dd2-5bf9-419b-a440-89ddc7c4ce2f_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!BBBU!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c8e2dd2-5bf9-419b-a440-89ddc7c4ce2f_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!BBBU!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c8e2dd2-5bf9-419b-a440-89ddc7c4ce2f_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!BBBU!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c8e2dd2-5bf9-419b-a440-89ddc7c4ce2f_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!BBBU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0c8e2dd2-5bf9-419b-a440-89ddc7c4ce2f_2752x1536.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Per-request routing is easy to draw and expensive to get wrong.</figcaption></figure></div><p>Every classification adds latency. General-purpose models can misread intent. Routing rules go stale.</p><p>And switching models during an agent session can wipe out your prefix-cache savings, raise costs, and make the agent behave inconsistently.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!P16Z!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a7e150e-47e6-4ce4-a5f5-179b47ba5407_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!P16Z!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a7e150e-47e6-4ce4-a5f5-179b47ba5407_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!P16Z!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a7e150e-47e6-4ce4-a5f5-179b47ba5407_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!P16Z!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a7e150e-47e6-4ce4-a5f5-179b47ba5407_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!P16Z!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a7e150e-47e6-4ce4-a5f5-179b47ba5407_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!P16Z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a7e150e-47e6-4ce4-a5f5-179b47ba5407_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2a7e150e-47e6-4ce4-a5f5-179b47ba5407_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:158044,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223382039?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a7e150e-47e6-4ce4-a5f5-179b47ba5407_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!P16Z!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a7e150e-47e6-4ce4-a5f5-179b47ba5407_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!P16Z!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a7e150e-47e6-4ce4-a5f5-179b47ba5407_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!P16Z!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a7e150e-47e6-4ce4-a5f5-179b47ba5407_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!P16Z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2a7e150e-47e6-4ce4-a5f5-179b47ba5407_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">In an agent loop, the cheapest model is often the one that already holds your prefix cache.</figcaption></figure></div><p>In this video, we cover:</p><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;74f97c1b-07fd-4a59-a039-d9cfe50bf3d8&quot;,&quot;duration&quot;:null}"></div><ul><li><p>Why DIY routing often breaks in production</p></li><li><p>How prefix caching changes the economics</p></li><li><p>Why agents need session pinning and model affinity</p></li><li><p>How Plano and Arch-Router handle intent resolution</p></li><li><p>How to build, test, and evaluate a router with <strong><a href="https://do.co/4i8JBrO">DigitalOcean</a></strong></p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!fyJg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F960af665-865b-4ca6-b0e9-e1e1126e3b47_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!fyJg!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F960af665-865b-4ca6-b0e9-e1e1126e3b47_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!fyJg!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F960af665-865b-4ca6-b0e9-e1e1126e3b47_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!fyJg!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F960af665-865b-4ca6-b0e9-e1e1126e3b47_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!fyJg!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F960af665-865b-4ca6-b0e9-e1e1126e3b47_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!fyJg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F960af665-865b-4ca6-b0e9-e1e1126e3b47_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/960af665-865b-4ca6-b0e9-e1e1126e3b47_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:137343,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223382039?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F960af665-865b-4ca6-b0e9-e1e1126e3b47_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!fyJg!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F960af665-865b-4ca6-b0e9-e1e1126e3b47_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!fyJg!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F960af665-865b-4ca6-b0e9-e1e1126e3b47_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!fyJg!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F960af665-865b-4ca6-b0e9-e1e1126e3b47_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!fyJg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F960af665-865b-4ca6-b0e9-e1e1126e3b47_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The router first works out what the task is, then picks the model for it.</figcaption></figure></div><p>Routing works, but only when the infrastructure accounts for the full session, not just the latest request.</p><p>So the question worth asking is how much of your AI traffic has been overpaying this whole time.</p><p><strong><a href="https://do.co/4i8JBrO">Get started with DigitalOcean Inference Router &#8594;</a></strong></p><p><em>Thanks to DigitalOcean for partnering today!</em></p><div><hr></div><h3><a href="https://www.dailydoseofds.com/ai-agents-with-langgraph-course-part-1-with-implementation/">Agent Framework vs. Agent Harness, clearly explained!</a></h3><p>Giving an agent tools doesn&#8217;t solve the problem of keeping it productive across a long task. It still needs to track progress, recover from failed actions, and preserve useful context when its conversation grows.</p><p>Those responsibilities help explain the distinction between an agent framework and an agent harness.</p><p>A framework gives you components for building agent behavior. A harness packages an agent loop with an environment and operating conventions. The engineering decision is how much of that surrounding system you want to assemble and maintain.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Pwck!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e36d4fa-98f8-40dc-a6d0-f2925369482c_900x900.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Pwck!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e36d4fa-98f8-40dc-a6d0-f2925369482c_900x900.png 424w, https://substackcdn.com/image/fetch/$s_!Pwck!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e36d4fa-98f8-40dc-a6d0-f2925369482c_900x900.png 848w, https://substackcdn.com/image/fetch/$s_!Pwck!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e36d4fa-98f8-40dc-a6d0-f2925369482c_900x900.png 1272w, https://substackcdn.com/image/fetch/$s_!Pwck!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e36d4fa-98f8-40dc-a6d0-f2925369482c_900x900.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Pwck!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e36d4fa-98f8-40dc-a6d0-f2925369482c_900x900.png" width="900" height="900" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0e36d4fa-98f8-40dc-a6d0-f2925369482c_900x900.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:900,&quot;width&quot;:900,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:95422,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223382039?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e36d4fa-98f8-40dc-a6d0-f2925369482c_900x900.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Pwck!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e36d4fa-98f8-40dc-a6d0-f2925369482c_900x900.png 424w, https://substackcdn.com/image/fetch/$s_!Pwck!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e36d4fa-98f8-40dc-a6d0-f2925369482c_900x900.png 848w, https://substackcdn.com/image/fetch/$s_!Pwck!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e36d4fa-98f8-40dc-a6d0-f2925369482c_900x900.png 1272w, https://substackcdn.com/image/fetch/$s_!Pwck!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e36d4fa-98f8-40dc-a6d0-f2925369482c_900x900.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A framework gives you the parts to assemble, while a harness gives you a working agent to configure.</figcaption></figure></div><h4>A framework gives you control over the design</h4><p>An agent framework provides abstractions for models, tools, state, and orchestration. You use those components to define how your application behaves.</p><p>Consider an invoice-processing system. It might extract fields, validate them against purchase orders, route discrepancies for review, and submit approved records.</p><p>Some steps need model judgment. Others should follow deterministic rules, especially when money moves or records change.</p><p>Explicit orchestration lets you enforce those boundaries. A model can suggest an action, while application code decides whether the required checks have passed.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!y5qx!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81ab6214-f813-4b7e-b9b5-5e86a3a0a578_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!y5qx!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81ab6214-f813-4b7e-b9b5-5e86a3a0a578_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!y5qx!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81ab6214-f813-4b7e-b9b5-5e86a3a0a578_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!y5qx!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81ab6214-f813-4b7e-b9b5-5e86a3a0a578_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!y5qx!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81ab6214-f813-4b7e-b9b5-5e86a3a0a578_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!y5qx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81ab6214-f813-4b7e-b9b5-5e86a3a0a578_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/81ab6214-f813-4b7e-b9b5-5e86a3a0a578_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:156393,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223382039?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81ab6214-f813-4b7e-b9b5-5e86a3a0a578_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!y5qx!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81ab6214-f813-4b7e-b9b5-5e86a3a0a578_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!y5qx!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81ab6214-f813-4b7e-b9b5-5e86a3a0a578_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!y5qx!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81ab6214-f813-4b7e-b9b5-5e86a3a0a578_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!y5qx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81ab6214-f813-4b7e-b9b5-5e86a3a0a578_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Explicit orchestration separates what the model proposes from what the code allows.</figcaption></figure></div><p>However, the left side of the visual needs one qualification: frameworks aren&#8217;t limited to workflows you can draw upfront.</p><p>LangGraph, for example, supports both deterministic steps and model-driven decisions, along with persistence and human intervention. A graph can contain an agent loop whose next tool call depends on what it discovers.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xbwd!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d1227bd-aef0-4294-8781-616372eb3ee2_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xbwd!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d1227bd-aef0-4294-8781-616372eb3ee2_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!xbwd!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d1227bd-aef0-4294-8781-616372eb3ee2_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!xbwd!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d1227bd-aef0-4294-8781-616372eb3ee2_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!xbwd!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d1227bd-aef0-4294-8781-616372eb3ee2_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!xbwd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d1227bd-aef0-4294-8781-616372eb3ee2_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5d1227bd-aef0-4294-8781-616372eb3ee2_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:122004,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223382039?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d1227bd-aef0-4294-8781-616372eb3ee2_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!xbwd!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d1227bd-aef0-4294-8781-616372eb3ee2_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!xbwd!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d1227bd-aef0-4294-8781-616372eb3ee2_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!xbwd!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d1227bd-aef0-4294-8781-616372eb3ee2_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!xbwd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d1227bd-aef0-4294-8781-616372eb3ee2_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A framework graph can hold an open-ended agent loop inside a path you still control.</figcaption></figure></div><p>The useful distinction is how much agent behavior comes assembled, and how much you define yourself.</p><h4>A harness supplies an operating environment</h4><p>An agent harness makes choices about how the model works inside a task.</p><p>Depending on the implementation, that can include file access, command execution, context management, progress tracking, and delegation to subagents.</p><p>Deep Agents is a concrete example. It builds on LangChain and LangGraph, adding filesystem tools, context offloading, and subagent support around the tool-calling loop. The framework and harness occupy different layers of the same system.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!9vOs!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb694876d-9bde-4201-b841-efe73a615d9b_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!9vOs!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb694876d-9bde-4201-b841-efe73a615d9b_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!9vOs!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb694876d-9bde-4201-b841-efe73a615d9b_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!9vOs!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb694876d-9bde-4201-b841-efe73a615d9b_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!9vOs!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb694876d-9bde-4201-b841-efe73a615d9b_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!9vOs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb694876d-9bde-4201-b841-efe73a615d9b_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b694876d-9bde-4201-b841-efe73a615d9b_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:171195,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223382039?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb694876d-9bde-4201-b841-efe73a615d9b_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!9vOs!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb694876d-9bde-4201-b841-efe73a615d9b_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!9vOs!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb694876d-9bde-4201-b841-efe73a615d9b_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!9vOs!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb694876d-9bde-4201-b841-efe73a615d9b_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!9vOs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb694876d-9bde-4201-b841-efe73a615d9b_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A harness like Deep Agents is a layer built on top of a framework and runtime, not a rival to it.</figcaption></figure></div><p>For a coding task, this changes your starting point.</p><p>Instead of implementing file inspection, editing, and execution behavior individually, you configure an existing environment with your repository instructions, tools, and access boundaries.</p><p>The model can inspect a failing test, read the relevant implementation, make a change, and run another check. Its intermediate actions depend on evidence gathered during execution.</p><p>You still own the definition of success. A working tool loop doesn&#8217;t establish that the requested feature works.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Li8v!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1e45df-da5e-46d8-a575-dd86d54a24ae_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Li8v!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1e45df-da5e-46d8-a575-dd86d54a24ae_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Li8v!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1e45df-da5e-46d8-a575-dd86d54a24ae_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Li8v!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1e45df-da5e-46d8-a575-dd86d54a24ae_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Li8v!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1e45df-da5e-46d8-a575-dd86d54a24ae_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Li8v!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1e45df-da5e-46d8-a575-dd86d54a24ae_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3d1e45df-da5e-46d8-a575-dd86d54a24ae_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:147076,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223382039?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1e45df-da5e-46d8-a575-dd86d54a24ae_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Li8v!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1e45df-da5e-46d8-a575-dd86d54a24ae_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Li8v!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1e45df-da5e-46d8-a575-dd86d54a24ae_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Li8v!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1e45df-da5e-46d8-a575-dd86d54a24ae_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Li8v!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1e45df-da5e-46d8-a575-dd86d54a24ae_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The harness runs the loop, but you still decide what counts as done.</figcaption></figure></div><h4>Long tasks</h4><p>Consider a task spans multiple sessions or context windows.</p><p>A conversation summary might preserve the broad objective while losing a detail such as which migration ran or which test still fails. Persisting the conversation alone doesn&#8217;t guarantee that the next session can resume correctly.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Wm3A!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F941b333f-d6b6-4573-bc1f-3c906dc4748f_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Wm3A!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F941b333f-d6b6-4573-bc1f-3c906dc4748f_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Wm3A!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F941b333f-d6b6-4573-bc1f-3c906dc4748f_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Wm3A!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F941b333f-d6b6-4573-bc1f-3c906dc4748f_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Wm3A!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F941b333f-d6b6-4573-bc1f-3c906dc4748f_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Wm3A!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F941b333f-d6b6-4573-bc1f-3c906dc4748f_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/941b333f-d6b6-4573-bc1f-3c906dc4748f_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:141920,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223382039?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F941b333f-d6b6-4573-bc1f-3c906dc4748f_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Wm3A!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F941b333f-d6b6-4573-bc1f-3c906dc4748f_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Wm3A!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F941b333f-d6b6-4573-bc1f-3c906dc4748f_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Wm3A!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F941b333f-d6b6-4573-bc1f-3c906dc4748f_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Wm3A!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F941b333f-d6b6-4573-bc1f-3c906dc4748f_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A summary can carry the goal into the next session and still lose the state of the work.</figcaption></figure></div><p>Anthropic documented this problem in its work on long-running coding agents. Agents sometimes attempted too much at once, left incomplete work, or declared completion prematurely.</p><p>Its approach combined an initial setup phase with incremental coding sessions. Progress files, version history, and explicit feature checks helped subsequent sessions reconstruct the state of work.</p><p>The practical lesson is to make progress inspectable outside the model&#8217;s conversation.</p><p>A repository commit records what changed. A test result records what passed. A task file records what remains.</p><p>These artifacts answer different questions, so compressing everything into one narrative summary can remove information the next session needs.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ybPB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e828f6b-ad4b-4142-b62c-d91a5b049e86_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ybPB!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e828f6b-ad4b-4142-b62c-d91a5b049e86_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ybPB!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e828f6b-ad4b-4142-b62c-d91a5b049e86_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ybPB!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e828f6b-ad4b-4142-b62c-d91a5b049e86_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ybPB!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e828f6b-ad4b-4142-b62c-d91a5b049e86_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ybPB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e828f6b-ad4b-4142-b62c-d91a5b049e86_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9e828f6b-ad4b-4142-b62c-d91a5b049e86_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:186967,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223382039?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e828f6b-ad4b-4142-b62c-d91a5b049e86_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ybPB!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e828f6b-ad4b-4142-b62c-d91a5b049e86_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ybPB!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e828f6b-ad4b-4142-b62c-d91a5b049e86_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ybPB!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e828f6b-ad4b-4142-b62c-d91a5b049e86_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ybPB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e828f6b-ad4b-4142-b62c-d91a5b049e86_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Each artifact answers a different question, so the next session can rebuild state without guessing.</figcaption></figure></div><h4>Choosing what to build yourself</h4><p>Start with the task&#8217;s control requirements.</p><p>For invoice processing, the required sequence and approval rules may be clear before execution. Explicit orchestration makes those rules visible and testable.</p><p>For investigating a failing build, the next useful action may only become clear after reading logs or running a test. A configurable harness can reduce the amount of execution machinery you need to build.</p><p>Many applications need both.</p><p>An outer workflow can enforce authorization and record final outcomes. Inside one step, an agent can investigate an issue using a harness, then return evidence for the workflow to validate.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!R1SD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa061a7a-02b2-4d5a-8fa9-cf51d7859aae_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!R1SD!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa061a7a-02b2-4d5a-8fa9-cf51d7859aae_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!R1SD!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa061a7a-02b2-4d5a-8fa9-cf51d7859aae_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!R1SD!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa061a7a-02b2-4d5a-8fa9-cf51d7859aae_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!R1SD!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa061a7a-02b2-4d5a-8fa9-cf51d7859aae_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!R1SD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa061a7a-02b2-4d5a-8fa9-cf51d7859aae_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/aa061a7a-02b2-4d5a-8fa9-cf51d7859aae_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:157069,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223382039?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa061a7a-02b2-4d5a-8fa9-cf51d7859aae_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!R1SD!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa061a7a-02b2-4d5a-8fa9-cf51d7859aae_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!R1SD!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa061a7a-02b2-4d5a-8fa9-cf51d7859aae_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!R1SD!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa061a7a-02b2-4d5a-8fa9-cf51d7859aae_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!R1SD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa061a7a-02b2-4d5a-8fa9-cf51d7859aae_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The workflow owns authorization and outcomes, and the agent owns the investigation inside one step.</figcaption></figure></div><p>That boundary also improves debugging. You can distinguish a failed business rule from an agent choosing an unhelpful tool or losing context.</p><h4>Evaluate the behavior around the model</h4><p>When comparing options, inspect what happens after a tool fails, what stays a restart, and how the system decides that work is complete.</p><p>Also check which operations require approval and whether those restrictions apply to delegated agents.</p><p>Test those behaviors with a deliberately interrupted task. Resume it and inspect whether the agent repeats completed work, misses an unresolved failure, or claims success without verification.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!V9se!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F697cbaea-5ab0-4715-9b25-3d6d058bb506_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!V9se!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F697cbaea-5ab0-4715-9b25-3d6d058bb506_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!V9se!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F697cbaea-5ab0-4715-9b25-3d6d058bb506_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!V9se!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F697cbaea-5ab0-4715-9b25-3d6d058bb506_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!V9se!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F697cbaea-5ab0-4715-9b25-3d6d058bb506_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!V9se!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F697cbaea-5ab0-4715-9b25-3d6d058bb506_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/697cbaea-5ab0-4715-9b25-3d6d058bb506_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:155760,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223382039?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F697cbaea-5ab0-4715-9b25-3d6d058bb506_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!V9se!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F697cbaea-5ab0-4715-9b25-3d6d058bb506_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!V9se!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F697cbaea-5ab0-4715-9b25-3d6d058bb506_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!V9se!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F697cbaea-5ab0-4715-9b25-3d6d058bb506_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!V9se!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F697cbaea-5ab0-4715-9b25-3d6d058bb506_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">An interrupted run tells you more about an agent system than a clean demo does.</figcaption></figure></div><p>Frameworks give you building blocks. Harnesses package decisions about execution. Your application still needs evidence that those decisions produce the behavior you intended.</p><p>If you want to build both sides of this yourself, our AI Agents course has a full LangChain and LangGraph track. It starts with a single production-ready agent and works up to harness middleware, planning and verification loops, multi-agent orchestration, and durable agents:</p><p><strong><a href="https://www.dailydoseofds.com/ai-agents-with-langgraph-course-part-1-with-implementation/">Start the AI Agents with LangGraph course &#8594;</a></strong></p><ul><li><p><strong><a href="https://www.dailydoseofds.com/ai-agents-with-langgraph-course-part-1-with-implementation/">Part 1: First Steps Towards a Production-Ready Agent with LangChain &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/ai-agents-with-langgraph-course-part-2-with-implementation/">Part 2: Middleware-Driven Context Engineering in LangChain &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/ai-agents-with-langgraph-course-part-3-with-implementation/">Part 3: Agent Harness with LangChain Middleware and Jev &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/ai-agents-with-langgraph-course-part-4-with-implementation/">Part 4: Planning and Verification Loops with LangGraph and Jev &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/ai-agents-with-langgraph-course-part-5-with-implementation/">Part 5: Multi-agent orchestration with LangGraph &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/ai-agents-with-langgraph-course-part-6-with-implementation/">Part 6: Durable Agents with LangGraph and LangSmith &#8594;</a></strong></p></li></ul><div><hr></div><h3><a href="https://arxiv.org/abs/2610.42147">A new technique just made attention 23x faster</a></h3><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!yx31!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb781afef-6493-4567-8905-f1484d8102bf_1000x775.gif" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!yx31!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb781afef-6493-4567-8905-f1484d8102bf_1000x775.gif 424w, https://substackcdn.com/image/fetch/$s_!yx31!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb781afef-6493-4567-8905-f1484d8102bf_1000x775.gif 848w, https://substackcdn.com/image/fetch/$s_!yx31!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb781afef-6493-4567-8905-f1484d8102bf_1000x775.gif 1272w, https://substackcdn.com/image/fetch/$s_!yx31!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb781afef-6493-4567-8905-f1484d8102bf_1000x775.gif 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!yx31!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb781afef-6493-4567-8905-f1484d8102bf_1000x775.gif" width="1000" height="775" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b781afef-6493-4567-8905-f1484d8102bf_1000x775.gif&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:775,&quot;width&quot;:1000,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:964881,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/gif&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223382039?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb781afef-6493-4567-8905-f1484d8102bf_1000x775.gif&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!yx31!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb781afef-6493-4567-8905-f1484d8102bf_1000x775.gif 424w, https://substackcdn.com/image/fetch/$s_!yx31!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb781afef-6493-4567-8905-f1484d8102bf_1000x775.gif 848w, https://substackcdn.com/image/fetch/$s_!yx31!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb781afef-6493-4567-8905-f1484d8102bf_1000x775.gif 1272w, https://substackcdn.com/image/fetch/$s_!yx31!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb781afef-6493-4567-8905-f1484d8102bf_1000x775.gif 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">At 10M tokens, Latent Sparse Attention runs one layer&#8217;s prefill 22.7x faster than GLM-5.3-Flash&#8217;s sparse attention on a single H200.</figcaption></figure></div><p>Full attention compares every query against every earlier key, so prefill costs O(T&#178;). At 10M tokens, that cost dominates everything else.</p><p>Sparse attention reads only k keys per query. But something still has to pick those k keys.</p><p>That picker is now the bottleneck. DeepSeek&#8217;s DSA uses a lightning indexer that scores every key. GLM-5.3-Flash pools 4 keys into 1 before scoring. NSA, MoBA, and Quest score blocks of keys. In each case the indexer stays O(T&#178;), and at 10M tokens it takes 98% of the layer&#8217;s prefill time.</p><p>Blocks also group keys by position, so one relevant key sitting among unrelated ones can get averaged away.</p><p>Latent Sparse Attention (LSA) groups keys by meaning instead:</p><ul><li><p>Keys are projected into a 64-dim index space, and every 4 are pooled into 1.</p></li><li><p>Each 512-key block is clustered into buckets of about 8 keys, each with a centroid. Finished blocks are never rebuilt.</p></li><li><p>A query scores only the centroids, keeps the top 1,024 buckets, and expands them to 4,096 keys, plus 4 sink tokens and the 512 most recent keys.</p></li><li><p>Exact FlashAttention then runs over those keys with an unchanged kernel.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!hDPp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa22ebc29-5abb-4b2e-b2a7-b0ee3b0b3cda_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!hDPp!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa22ebc29-5abb-4b2e-b2a7-b0ee3b0b3cda_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!hDPp!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa22ebc29-5abb-4b2e-b2a7-b0ee3b0b3cda_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!hDPp!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa22ebc29-5abb-4b2e-b2a7-b0ee3b0b3cda_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!hDPp!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa22ebc29-5abb-4b2e-b2a7-b0ee3b0b3cda_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!hDPp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa22ebc29-5abb-4b2e-b2a7-b0ee3b0b3cda_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a22ebc29-5abb-4b2e-b2a7-b0ee3b0b3cda_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:156666,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223382039?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa22ebc29-5abb-4b2e-b2a7-b0ee3b0b3cda_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!hDPp!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa22ebc29-5abb-4b2e-b2a7-b0ee3b0b3cda_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!hDPp!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa22ebc29-5abb-4b2e-b2a7-b0ee3b0b3cda_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!hDPp!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa22ebc29-5abb-4b2e-b2a7-b0ee3b0b3cda_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!hDPp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa22ebc29-5abb-4b2e-b2a7-b0ee3b0b3cda_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">LSA searches bucket centroids instead of scanning every key.</figcaption></figure></div><p>If you&#8217;ve worked with vector search, this will look familiar. It&#8217;s cluster-based search running inside attention, and the paper&#8217;s kernel design draws on FAISS.</p><p>The results at 10M tokens:</p><ul><li><p>One layer&#8217;s prefill runs 72.4x faster than full attention on Qwen-3.8-27B, and 22.7x faster than GLM-5.3-Flash&#8217;s sparse attention.</p></li><li><p>End-to-end time to first token is 9.2x faster on Qwen and 2.2x faster on GLM.</p></li><li><p>Below roughly 512k tokens, the end-to-end gain is about break-even.</p></li></ul><p>So the 23x is a per-layer prefill number. The end-to-end gain is smaller and shows up only at very long contexts.</p><p>One thing LSA doesn&#8217;t change is storage. It makes finding the right keys cheaper, but the KV cache still grows linearly with context.</p><p><strong><a href="https://arxiv.org/abs/2610.42147">Read the LSA paper &#8594;</a></strong></p><p>We covered the storage side in detail in our KV cache engineering deep dive:</p><p><strong><a href="https://www.dailydoseofds.com/p/kv-cache-engineering-for-llm-serving/">Read it here &#8594;</a></strong></p><p>&#128073; Over to you: Which part of your agent system has required more engineering, the workflow itself or keeping the agent productive during execution?</p><p>Good day!</p>]]></content:encoded></item><item><title><![CDATA[12 Optimizers for Training Neural Networks, clearly explained!]]></title><description><![CDATA[...and what each one costs in GPU memory!]]></description><link>https://blog.dailydoseofds.com/p/12-optimizers-for-training-neural</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/12-optimizers-for-training-neural</guid><dc:creator><![CDATA[Avi Chawla]]></dc:creator><pubDate>Mon, 02 Nov 2026 21:30:36 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!gHRY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a6e4e5-3e34-4e37-b073-7ec589d2eb67_1402x1125.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><strong>In today&#8217;s newsletter:</strong></p><ul><li><p>Your AI coding agents should get smarter every time you use them.</p></li><li><p>12 optimizers for training neural networks, clearly explained.</p></li></ul><div><hr></div><h3><a href="https://www.producthunt.com/products/asymptotelabs?launch=beacon-sh">Your AI coding agents should get smarter every time you use them</a></h3><p>AI coding agents write good code now, but they still don&#8217;t learn from experience.</p><p>Say you spend 30 minutes helping Claude Code debug a tricky issue. It explores your codebase, tries several approaches, hits dead ends, and eventually finds the fix.</p><p>Tomorrow you start a new session, and the agent makes the exact same mistakes.</p><p>Switch to Codex or Cursor and it gets worse. They have no idea what your previous agent figured out.</p><p>Every session produces useful knowledge: debugging breakthroughs, failed approaches, architectural decisions, and patterns specific to your codebase.</p><p>But most of it disappears when the session ends.</p><p><strong><a href="https://github.com/Asymptote-Labs/agent-beacon">Beacon</a></strong>, built by Asymptote Labs, is an open-source memory layer that lets your coding agents learn from past sessions.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!zDiJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85527bbd-c50f-41e0-a4ee-54d172053c27_804x568.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zDiJ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85527bbd-c50f-41e0-a4ee-54d172053c27_804x568.png 424w, https://substackcdn.com/image/fetch/$s_!zDiJ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85527bbd-c50f-41e0-a4ee-54d172053c27_804x568.png 848w, https://substackcdn.com/image/fetch/$s_!zDiJ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85527bbd-c50f-41e0-a4ee-54d172053c27_804x568.png 1272w, https://substackcdn.com/image/fetch/$s_!zDiJ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85527bbd-c50f-41e0-a4ee-54d172053c27_804x568.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!zDiJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85527bbd-c50f-41e0-a4ee-54d172053c27_804x568.png" width="625" height="441.5422885572139" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/85527bbd-c50f-41e0-a4ee-54d172053c27_804x568.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:568,&quot;width&quot;:804,&quot;resizeWidth&quot;:625,&quot;bytes&quot;:38160,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223195758?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85527bbd-c50f-41e0-a4ee-54d172053c27_804x568.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!zDiJ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85527bbd-c50f-41e0-a4ee-54d172053c27_804x568.png 424w, https://substackcdn.com/image/fetch/$s_!zDiJ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85527bbd-c50f-41e0-a4ee-54d172053c27_804x568.png 848w, https://substackcdn.com/image/fetch/$s_!zDiJ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85527bbd-c50f-41e0-a4ee-54d172053c27_804x568.png 1272w, https://substackcdn.com/image/fetch/$s_!zDiJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85527bbd-c50f-41e0-a4ee-54d172053c27_804x568.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>It works across Claude Code, Cursor, Codex, OpenCode, Cline, and 20+ other agent harnesses.</p><p>Beacon launched on Product Hunt today.</p><p><strong><a href="https://www.producthunt.com/products/asymptotelabs?launch=beacon-sh">Check out the launch and show your support with an upvote &#8594;</a></strong></p><p>Instead of treating every session as an isolated conversation, Beacon captures what your agents actually do and turns that experience into reusable knowledge.</p><p>Here&#8217;s how it works:</p><p>&#8627; <strong>Capture every session.</strong> Beacon records execution traces across your coding agents, including commands, file changes, errors, and debugging paths.</p><p>&#8627; <strong>Extract reusable knowledge.</strong> It picks out successful fixes, recurring workflows, and lessons worth remembering.</p><p>&#8627; <strong>Improve future sessions.</strong> Those lessons become skills and context your agents can use, even in a different coding tool.</p><p>&#8627; <strong>Close the feedback loop.</strong> Later traces show when saved knowledge stops working, so it can be updated or retired.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!nc8T!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1dc9ca1c-c5a3-40bb-96b8-957aedffe2a9_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!nc8T!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1dc9ca1c-c5a3-40bb-96b8-957aedffe2a9_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!nc8T!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1dc9ca1c-c5a3-40bb-96b8-957aedffe2a9_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!nc8T!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1dc9ca1c-c5a3-40bb-96b8-957aedffe2a9_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!nc8T!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1dc9ca1c-c5a3-40bb-96b8-957aedffe2a9_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!nc8T!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1dc9ca1c-c5a3-40bb-96b8-957aedffe2a9_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1dc9ca1c-c5a3-40bb-96b8-957aedffe2a9_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:164920,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223195758?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1dc9ca1c-c5a3-40bb-96b8-957aedffe2a9_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!nc8T!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1dc9ca1c-c5a3-40bb-96b8-957aedffe2a9_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!nc8T!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1dc9ca1c-c5a3-40bb-96b8-957aedffe2a9_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!nc8T!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1dc9ca1c-c5a3-40bb-96b8-957aedffe2a9_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!nc8T!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1dc9ca1c-c5a3-40bb-96b8-957aedffe2a9_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Beacon turns traces from Claude Code, Cursor, and Codex into skills that the next session can use.</figcaption></figure></div><h4>Beacon Cloud continuously finds ways to improve your agents</h4><p>Capturing sessions is only the first step.</p><p>Every hour, Beacon Cloud reviews your agents&#8217; new activity and turns it into findings in three categories:</p><p>&#8627; <strong>Security: risky things your agents did.</strong> Secrets pasted into prompts, destructive commands, permission bypasses, or connections to unknown MCP servers. Beacon flags the action, shows which session it happened in, and suggests a fix.</p><p>&#8627; <strong>Skill: workflows worth reusing.</strong> When you keep walking agents through the same debugging steps, Beacon notices and drafts a reusable skill. You download it and make it available to all your coding agents.</p><p>&#8627; <strong>Behavior: patterns in how you work with agents.</strong> Retry loops, abandoned sessions, unnecessary approvals, or expensive models used for simple tasks. Beacon points out the waste and recommends faster, cheaper ways to work.</p><p>The findings keep updating as your agents generate new traces.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!sl1j!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06c808dc-2f04-4cee-bde6-3f78689fb49c_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!sl1j!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06c808dc-2f04-4cee-bde6-3f78689fb49c_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!sl1j!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06c808dc-2f04-4cee-bde6-3f78689fb49c_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!sl1j!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06c808dc-2f04-4cee-bde6-3f78689fb49c_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!sl1j!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06c808dc-2f04-4cee-bde6-3f78689fb49c_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!sl1j!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06c808dc-2f04-4cee-bde6-3f78689fb49c_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/06c808dc-2f04-4cee-bde6-3f78689fb49c_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:170690,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223195758?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06c808dc-2f04-4cee-bde6-3f78689fb49c_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!sl1j!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06c808dc-2f04-4cee-bde6-3f78689fb49c_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!sl1j!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06c808dc-2f04-4cee-bde6-3f78689fb49c_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!sl1j!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06c808dc-2f04-4cee-bde6-3f78689fb49c_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!sl1j!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06c808dc-2f04-4cee-bde6-3f78689fb49c_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Beacon Cloud sorts what it finds in your sessions into security, skill, and behavior findings.</figcaption></figure></div><h4>One memory layer for every coding agent</h4><p>Beacon sits across the agent harness layer.</p><p>So your Claude Code debugging lessons can help your next Codex session, and your Cursor discoveries can become skills in OpenCode.</p><p>With <code>beacon handoff</code>, you can also resume work in a different agent when you hit a rate limit or need to switch tools. You don&#8217;t re-explain the repo or retrace failed approaches.</p><p>It matters even more across a team. One engineer spends an hour debugging an infrastructure issue, Beacon captures the approach that worked, and the next engineer can reuse it as a skill in whichever coding agent they use.</p><p>The loop looks like this:</p><p><strong>Run agents &#8594; Capture traces &#8594; Generate findings &#8594; Reuse skills &#8594; Improve &#8594; Repeat</strong></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!vJFU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F615ee745-b4c5-4a21-a31a-22c1ac2f9e7d_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!vJFU!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F615ee745-b4c5-4a21-a31a-22c1ac2f9e7d_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!vJFU!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F615ee745-b4c5-4a21-a31a-22c1ac2f9e7d_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!vJFU!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F615ee745-b4c5-4a21-a31a-22c1ac2f9e7d_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!vJFU!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F615ee745-b4c5-4a21-a31a-22c1ac2f9e7d_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!vJFU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F615ee745-b4c5-4a21-a31a-22c1ac2f9e7d_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/615ee745-b4c5-4a21-a31a-22c1ac2f9e7d_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:150825,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223195758?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F615ee745-b4c5-4a21-a31a-22c1ac2f9e7d_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!vJFU!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F615ee745-b4c5-4a21-a31a-22c1ac2f9e7d_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!vJFU!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F615ee745-b4c5-4a21-a31a-22c1ac2f9e7d_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!vJFU!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F615ee745-b4c5-4a21-a31a-22c1ac2f9e7d_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!vJFU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F615ee745-b4c5-4a21-a31a-22c1ac2f9e7d_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Each run feeds new traces back into Beacon, so the skills keep improving.</figcaption></figure></div><p>Beacon runs locally without a hosted backend. Beacon Cloud adds hosted trace storage, cross-harness search, session analytics, and the findings above.</p><p>A smarter model only gets you so far. The coding agent that keeps getting better is the one that learns from experience.</p><p><strong><a href="https://www.producthunt.com/products/asymptotelabs?launch=beacon-sh">Support Beacon on Product Hunt &#8594;</a></strong></p><p><strong><a href="https://github.com/Asymptote-Labs/agent-beacon">Star Beacon on GitHub &#8594;</a></strong></p><p>Stop teaching your agents the same lessons twice.</p><p><em>Thanks to Asymptote Labs for partnering today!</em></p><div><hr></div><h3><a href="https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/">12 optimizers for training neural networks, clearly explained!</a></h3><p>An optimizer can reduce the number of training steps and still make your run slower. Each update has its own compute and memory cost, which becomes part of the training budget.</p><p>That tradeoff gets easier to reason about once you look at what each optimizer stores and how it transforms the gradient.</p><p>The 12 optimizers in the visual approach this differently. Let&#8217;s break them down through the engineering decisions they change, starting with how much gradient history survives each update.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!gHRY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a6e4e5-3e34-4e37-b073-7ec589d2eb67_1402x1125.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!gHRY!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a6e4e5-3e34-4e37-b073-7ec589d2eb67_1402x1125.png 424w, https://substackcdn.com/image/fetch/$s_!gHRY!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a6e4e5-3e34-4e37-b073-7ec589d2eb67_1402x1125.png 848w, https://substackcdn.com/image/fetch/$s_!gHRY!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a6e4e5-3e34-4e37-b073-7ec589d2eb67_1402x1125.png 1272w, https://substackcdn.com/image/fetch/$s_!gHRY!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a6e4e5-3e34-4e37-b073-7ec589d2eb67_1402x1125.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!gHRY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a6e4e5-3e34-4e37-b073-7ec589d2eb67_1402x1125.png" width="1402" height="1125" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/56a6e4e5-3e34-4e37-b073-7ec589d2eb67_1402x1125.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1125,&quot;width&quot;:1402,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:166536,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223195758?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a6e4e5-3e34-4e37-b073-7ec589d2eb67_1402x1125.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!gHRY!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a6e4e5-3e34-4e37-b073-7ec589d2eb67_1402x1125.png 424w, https://substackcdn.com/image/fetch/$s_!gHRY!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a6e4e5-3e34-4e37-b073-7ec589d2eb67_1402x1125.png 848w, https://substackcdn.com/image/fetch/$s_!gHRY!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a6e4e5-3e34-4e37-b073-7ec589d2eb67_1402x1125.png 1272w, https://substackcdn.com/image/fetch/$s_!gHRY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a6e4e5-3e34-4e37-b073-7ec589d2eb67_1402x1125.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Each of the 12 optimizers changes what it stores and how it turns the gradient into an update.</figcaption></figure></div><h4>Smoothing gradients and adapting step sizes</h4><p>SGD with momentum keeps a running accumulation of gradients. Directions that persist across batches build up, while inconsistent directions tend to cancel.</p><p>This helps reduce oscillation, but momentum alone doesn&#8217;t give each parameter its own adaptive step size.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!DKv8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4815a9f-5317-4047-b3dd-0f5a1b9c0c70_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!DKv8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4815a9f-5317-4047-b3dd-0f5a1b9c0c70_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!DKv8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4815a9f-5317-4047-b3dd-0f5a1b9c0c70_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!DKv8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4815a9f-5317-4047-b3dd-0f5a1b9c0c70_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!DKv8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4815a9f-5317-4047-b3dd-0f5a1b9c0c70_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!DKv8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4815a9f-5317-4047-b3dd-0f5a1b9c0c70_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c4815a9f-5317-4047-b3dd-0f5a1b9c0c70_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:180135,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223195758?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4815a9f-5317-4047-b3dd-0f5a1b9c0c70_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!DKv8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4815a9f-5317-4047-b3dd-0f5a1b9c0c70_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!DKv8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4815a9f-5317-4047-b3dd-0f5a1b9c0c70_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!DKv8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4815a9f-5317-4047-b3dd-0f5a1b9c0c70_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!DKv8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc4815a9f-5317-4047-b3dd-0f5a1b9c0c70_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Momentum averages away the back-and-forth across the valley, so the steady direction builds speed.</figcaption></figure></div><p>AdaGrad adds that adaptation by accumulating squared gradients for each parameter. Frequently updated parameters receive progressively smaller effective steps, which can help with sparse features.</p><p>Its limitation follows directly from that accumulation: the denominator keeps growing, so effective learning rates can become too small.</p><p>RMSprop replaces the full accumulation with an exponential moving average of squared gradients. Older observations fade, allowing the scaling to respond to more recent gradients.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!uY-n!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0535192f-b5f7-472e-a13a-992bb46b3c33_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!uY-n!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0535192f-b5f7-472e-a13a-992bb46b3c33_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!uY-n!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0535192f-b5f7-472e-a13a-992bb46b3c33_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!uY-n!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0535192f-b5f7-472e-a13a-992bb46b3c33_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!uY-n!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0535192f-b5f7-472e-a13a-992bb46b3c33_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!uY-n!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0535192f-b5f7-472e-a13a-992bb46b3c33_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0535192f-b5f7-472e-a13a-992bb46b3c33_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:180718,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223195758?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0535192f-b5f7-472e-a13a-992bb46b3c33_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!uY-n!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0535192f-b5f7-472e-a13a-992bb46b3c33_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!uY-n!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0535192f-b5f7-472e-a13a-992bb46b3c33_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!uY-n!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0535192f-b5f7-472e-a13a-992bb46b3c33_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!uY-n!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0535192f-b5f7-472e-a13a-992bb46b3c33_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">AdaGrad&#8217;s sum only grows, so its steps shrink toward zero, while RMSprop&#8217;s moving average lets old gradients fade.</figcaption></figure></div><p>Adam combines momentum with this second-moment scaling. One moving average tracks gradient direction; another tracks squared gradient magnitude.</p><p>That second quantity isn&#8217;t a direct curvature measurement. Adam adapts to gradient statistics without estimating the Hessian.</p><h4>Changing regularization, memory, and gradient history</h4><p>AdamW separates weight decay from the adaptive gradient update. Adding an L2 penalty to Adam&#8217;s loss lets its adaptive scaling affect that penalty; decoupled weight decay applies parameter shrinkage separately.</p><p>That distinction matters when you&#8217;re tuning regularization. Weight decay becomes easier to reason about independently of the gradient scaling.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!eRLN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcde78a-1dbd-40b9-ac80-afdfbd4f9836_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!eRLN!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcde78a-1dbd-40b9-ac80-afdfbd4f9836_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!eRLN!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcde78a-1dbd-40b9-ac80-afdfbd4f9836_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!eRLN!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcde78a-1dbd-40b9-ac80-afdfbd4f9836_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!eRLN!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcde78a-1dbd-40b9-ac80-afdfbd4f9836_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!eRLN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcde78a-1dbd-40b9-ac80-afdfbd4f9836_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bfcde78a-1dbd-40b9-ac80-afdfbd4f9836_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:170548,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223195758?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcde78a-1dbd-40b9-ac80-afdfbd4f9836_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!eRLN!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcde78a-1dbd-40b9-ac80-afdfbd4f9836_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!eRLN!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcde78a-1dbd-40b9-ac80-afdfbd4f9836_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!eRLN!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcde78a-1dbd-40b9-ac80-afdfbd4f9836_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!eRLN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcde78a-1dbd-40b9-ac80-afdfbd4f9836_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">With L2 inside Adam, the penalty gets divided by &#8730;v too, while AdamW applies the decay to the weights directly.</figcaption></figure></div><p>Memory introduces another constraint. Standard Adam maintains two parameter-sized state arrays, in addition to the model weights and gradients.</p><p>With one billion parameters and FP32 moment buffers, those two arrays alone require about 8 GB before sharding or compression.</p><p>Adafactor reduces the second-moment storage for matrices using row and column statistics. For an n &#215; m matrix, that replaces nm stored second-moment values with roughly n+m.</p><p>You&#8217;re trading a full per-parameter estimate for a factored approximation. That&#8217;s useful when optimizer state limits the model or batch size you can fit.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!CJSW!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf86c3bb-07a6-43f4-86a2-3fccfe03e08a_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!CJSW!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf86c3bb-07a6-43f4-86a2-3fccfe03e08a_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!CJSW!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf86c3bb-07a6-43f4-86a2-3fccfe03e08a_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!CJSW!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf86c3bb-07a6-43f4-86a2-3fccfe03e08a_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!CJSW!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf86c3bb-07a6-43f4-86a2-3fccfe03e08a_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!CJSW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf86c3bb-07a6-43f4-86a2-3fccfe03e08a_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/cf86c3bb-07a6-43f4-86a2-3fccfe03e08a_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:168297,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223195758?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf86c3bb-07a6-43f4-86a2-3fccfe03e08a_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!CJSW!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf86c3bb-07a6-43f4-86a2-3fccfe03e08a_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!CJSW!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf86c3bb-07a6-43f4-86a2-3fccfe03e08a_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!CJSW!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf86c3bb-07a6-43f4-86a2-3fccfe03e08a_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!CJSW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf86c3bb-07a6-43f4-86a2-3fccfe03e08a_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Adam&#8217;s two FP32 moment buffers take 8 GB at 1B parameters, while Adafactor keeps n + m statistics per matrix instead of n &#215; m.</figcaption></figure></div><p>Lion keeps momentum without a second-moment buffer. Its gradient update uses the sign of a momentum-gradient blend, giving coordinates the same update magnitude within a parameter group.</p><p>That changes the update scale, so copying AdamW&#8217;s learning rate into Lion isn&#8217;t a fair comparison. The Lion paper specifically calls for smaller learning rates.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xTfg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa14fe711-8f5f-49f6-a909-f0768da8323d_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xTfg!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa14fe711-8f5f-49f6-a909-f0768da8323d_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!xTfg!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa14fe711-8f5f-49f6-a909-f0768da8323d_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!xTfg!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa14fe711-8f5f-49f6-a909-f0768da8323d_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!xTfg!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa14fe711-8f5f-49f6-a909-f0768da8323d_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!xTfg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa14fe711-8f5f-49f6-a909-f0768da8323d_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a14fe711-8f5f-49f6-a909-f0768da8323d_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:132437,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223195758?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa14fe711-8f5f-49f6-a909-f0768da8323d_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!xTfg!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa14fe711-8f5f-49f6-a909-f0768da8323d_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!xTfg!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa14fe711-8f5f-49f6-a909-f0768da8323d_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!xTfg!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa14fe711-8f5f-49f6-a909-f0768da8323d_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!xTfg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa14fe711-8f5f-49f6-a909-f0768da8323d_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Lion keeps only the sign of the momentum-gradient blend, so every coordinate moves by the same amount.</figcaption></figure></div><p>AdEMAMix takes another approach: it combines fast and slow moving averages of gradients alongside second-moment scaling. Recent gradients remain influential while older directional information survives longer, at the cost of additional state.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!8Ajj!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ba24687-e6f3-41bf-81de-418fc18c1da9_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8Ajj!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ba24687-e6f3-41bf-81de-418fc18c1da9_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!8Ajj!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ba24687-e6f3-41bf-81de-418fc18c1da9_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!8Ajj!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ba24687-e6f3-41bf-81de-418fc18c1da9_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!8Ajj!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ba24687-e6f3-41bf-81de-418fc18c1da9_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!8Ajj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ba24687-e6f3-41bf-81de-418fc18c1da9_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8ba24687-e6f3-41bf-81de-418fc18c1da9_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:171084,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223195758?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ba24687-e6f3-41bf-81de-418fc18c1da9_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!8Ajj!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ba24687-e6f3-41bf-81de-418fc18c1da9_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!8Ajj!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ba24687-e6f3-41bf-81de-418fc18c1da9_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!8Ajj!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ba24687-e6f3-41bf-81de-418fc18c1da9_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!8Ajj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8ba24687-e6f3-41bf-81de-418fc18c1da9_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A fast EMA follows recent gradients while a slow EMA keeps older directions in the update.</figcaption></figure></div><h4>Using matrix structure and curvature</h4><p>Adam scales coordinates individually. Shampoo uses matrix-valued statistics to transform updates across parameter dimensions, capturing relationships that elementwise scaling misses.</p><p>Those richer statistics bring additional storage and matrix computation.</p><p>SOAP connects this approach back to Adam. It rotates gradients into the eigenbasis of Shampoo&#8217;s preconditioners, applies Adam there, and rotates the update back.</p><p>The coordinate system changes how Adam&#8217;s elementwise scaling acts on the update. Computing and refreshing that basis also adds overhead.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ZqGW!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52e1c386-5f30-495d-afb4-2dee58ca6116_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ZqGW!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52e1c386-5f30-495d-afb4-2dee58ca6116_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ZqGW!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52e1c386-5f30-495d-afb4-2dee58ca6116_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ZqGW!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52e1c386-5f30-495d-afb4-2dee58ca6116_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ZqGW!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52e1c386-5f30-495d-afb4-2dee58ca6116_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ZqGW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52e1c386-5f30-495d-afb4-2dee58ca6116_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/52e1c386-5f30-495d-afb4-2dee58ca6116_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:157283,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223195758?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52e1c386-5f30-495d-afb4-2dee58ca6116_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ZqGW!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52e1c386-5f30-495d-afb4-2dee58ca6116_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ZqGW!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52e1c386-5f30-495d-afb4-2dee58ca6116_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ZqGW!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52e1c386-5f30-495d-afb4-2dee58ca6116_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ZqGW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52e1c386-5f30-495d-afb4-2dee58ca6116_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Shampoo preconditions the gradient from both sides, and SOAP runs Adam in the eigenbasis of those preconditioners.</figcaption></figure></div><p>Muon approximately orthogonalizes momentum updates for matrix parameters using Newton&#8211;Schulz iterations. It changes the matrix update&#8217;s singular-value structure rather than scaling every coordinate independently.</p><p>Parameter selection matters here. Muon is typically applied to hidden weight matrices, with another optimizer handling embeddings, output layers, and vector parameters.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!zk2Z!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f107ac6-2681-4e5a-8922-e2c3317249ce_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zk2Z!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f107ac6-2681-4e5a-8922-e2c3317249ce_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!zk2Z!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f107ac6-2681-4e5a-8922-e2c3317249ce_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!zk2Z!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f107ac6-2681-4e5a-8922-e2c3317249ce_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!zk2Z!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f107ac6-2681-4e5a-8922-e2c3317249ce_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!zk2Z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f107ac6-2681-4e5a-8922-e2c3317249ce_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0f107ac6-2681-4e5a-8922-e2c3317249ce_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:172026,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223195758?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f107ac6-2681-4e5a-8922-e2c3317249ce_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!zk2Z!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f107ac6-2681-4e5a-8922-e2c3317249ce_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!zk2Z!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f107ac6-2681-4e5a-8922-e2c3317249ce_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!zk2Z!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f107ac6-2681-4e5a-8922-e2c3317249ce_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!zk2Z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f107ac6-2681-4e5a-8922-e2c3317249ce_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Muon evens out the singular values of hidden-matrix updates, while embeddings, the output layer, and vector parameters stay on AdamW.</figcaption></figure></div><p>Sophia uses a diagonal curvature estimate to scale momentum, then clips updates elementwise. Its curvature estimate is refreshed periodically, limiting the average overhead of obtaining that information.</p><h4>Comparing optimizers on your workload</h4><p>I&#8217;d use AdamW as the reference run for a transformer experiment, then choose alternatives against a specific constraint.</p><p>For memory pressure, investigate Adafactor or Lion. For matrix updates, evaluate Muon, Shampoo, or SOAP while accounting for their parameter coverage and implementation costs.</p><p>Give each optimizer a learning-rate search. Keep the data, model, and evaluation protocol consistent, and report both validation quality at a fixed compute budget and time to a target quality.</p><p>Also measure peak memory and step time. An optimizer that reaches the target in 20% fewer steps but makes each step 30% slower takes about 4% longer overall.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Lf76!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb700a17-0a84-4c3c-957a-d08b9025a81c_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Lf76!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb700a17-0a84-4c3c-957a-d08b9025a81c_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Lf76!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb700a17-0a84-4c3c-957a-d08b9025a81c_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Lf76!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb700a17-0a84-4c3c-957a-d08b9025a81c_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Lf76!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb700a17-0a84-4c3c-957a-d08b9025a81c_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Lf76!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb700a17-0a84-4c3c-957a-d08b9025a81c_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/cb700a17-0a84-4c3c-957a-d08b9025a81c_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:137028,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/223195758?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb700a17-0a84-4c3c-957a-d08b9025a81c_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Lf76!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb700a17-0a84-4c3c-957a-d08b9025a81c_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Lf76!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb700a17-0a84-4c3c-957a-d08b9025a81c_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Lf76!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb700a17-0a84-4c3c-957a-d08b9025a81c_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Lf76!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb700a17-0a84-4c3c-957a-d08b9025a81c_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">80 steps at 1.3 s each take 104 s, so 20% fewer steps still lose to the 100 s baseline on the clock.</figcaption></figure></div><p>The useful result is a reproducible improvement under your training constraints, including the cost of tuning it.</p><p>If you want to build these foundations properly, our Classical ML and Deep Learning roadmap covers the full stack in order, from the assumptions behind linear models to neural network training and the architectures built on top:</p><p><strong><a href="https://www.dailydoseofds.com/where-did-the-assumptions-of-linear-regression-originate-from/">Start the Classical ML and Deep Learning roadmap &#8594;</a></strong></p><p>&#128073; Over to you: Which optimizer has improved your training runs enough to justify changing your default?</p><p>Good day!</p>]]></content:encoded></item><item><title><![CDATA[Why Other Requests Change Your LLM’s Output]]></title><description><![CDATA[+ an update on Harness Engineering 2.0.]]></description><link>https://blog.dailydoseofds.com/p/why-other-requests-change-your-llms</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/why-other-requests-change-your-llms</guid><dc:creator><![CDATA[Avi Chawla]]></dc:creator><pubDate>Sat, 31 Oct 2026 20:21:59 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!fxG4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a8f8bb1-b631-4984-89b2-5f2d80b6df03_900x900.gif" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><strong>In today&#8217;s newsletter:</strong></p><ul><li><p>Harness Engineering 2.0 is here!</p></li><li><p>Why other requests change your LLM&#8217;s output.</p></li><li><p>RAG vs. retriever-free RAG, clearly explained.</p></li></ul><div><hr></div><h3><a href="https://github.com/HarnessRouter/harnessrouter">Harness Engineering 2.0 is here!</a></h3><p>Tools got MCP, agent-to-agent communication got A2A, and agent interfaces got AG-UI. Now harnesses have a protocol too.</p><p>Developers just built UHP (Unified Harness Protocol), a standard interface for running agent tasks on any agent harness. HarnessRouter is its open-source server implementation.</p><p></p><p>Here&#8217;s how it works:</p><p>&#8627; Your backend sends a task with a <code>harness_id</code>.</p><p>&#8627; HarnessRouter starts the selected harness in its own session.</p><p>&#8627; The harness runs its native agent loop and streams back progress.</p><p>&#8627; Files, artifacts, failures, and the final output come back through the same interface.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!g8u1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9824fb-de34-48f8-a4cf-ef730e63fcb3_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!g8u1!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9824fb-de34-48f8-a4cf-ef730e63fcb3_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!g8u1!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9824fb-de34-48f8-a4cf-ef730e63fcb3_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!g8u1!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9824fb-de34-48f8-a4cf-ef730e63fcb3_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!g8u1!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9824fb-de34-48f8-a4cf-ef730e63fcb3_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!g8u1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9824fb-de34-48f8-a4cf-ef730e63fcb3_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4f9824fb-de34-48f8-a4cf-ef730e63fcb3_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:160143,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222929994?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9824fb-de34-48f8-a4cf-ef730e63fcb3_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!g8u1!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9824fb-de34-48f8-a4cf-ef730e63fcb3_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!g8u1!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9824fb-de34-48f8-a4cf-ef730e63fcb3_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!g8u1!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9824fb-de34-48f8-a4cf-ef730e63fcb3_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!g8u1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9824fb-de34-48f8-a4cf-ef730e63fcb3_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Your backend talks to one API, and HarnessRouter runs whichever harness the task names.</figcaption></figure></div><p>Without UHP, every harness needs its own handling for task execution, sessions, streaming, files, recovery, and output formats.</p><p>With UHP, Codex, Claude Code, Gemini CLI, Hermes, Pi, and other harnesses plug into the same product backend. Each one keeps its native tools, skills, models, and agent loop.</p><p>So developers can:</p><p>&#8627; Switch harnesses without rebuilding the product integration.</p><p>&#8627; Run the same task on different harnesses.</p><p>&#8627; Add new harnesses without maintaining another execution stack.</p><p>HarnessRouter supports 30 harness bases today.</p><p>The harness also changes what a task costs. In HarnessRouter&#8217;s published benchmark, keeping the model fixed and switching only the harness cut credits by 32% to 53% on the same healthcare prep task. Across all eight harness and model setups, the cheapest used 99.8% fewer credits than the most expensive.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!kkyo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa57eb893-a631-4279-b6ba-e7bf24f07260_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!kkyo!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa57eb893-a631-4279-b6ba-e7bf24f07260_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!kkyo!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa57eb893-a631-4279-b6ba-e7bf24f07260_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!kkyo!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa57eb893-a631-4279-b6ba-e7bf24f07260_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!kkyo!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa57eb893-a631-4279-b6ba-e7bf24f07260_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!kkyo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa57eb893-a631-4279-b6ba-e7bf24f07260_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a57eb893-a631-4279-b6ba-e7bf24f07260_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:153964,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222929994?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa57eb893-a631-4279-b6ba-e7bf24f07260_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!kkyo!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa57eb893-a631-4279-b6ba-e7bf24f07260_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!kkyo!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa57eb893-a631-4279-b6ba-e7bf24f07260_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!kkyo!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa57eb893-a631-4279-b6ba-e7bf24f07260_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!kkyo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa57eb893-a631-4279-b6ba-e7bf24f07260_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">On the same task with the same model, the harness alone moved cost by up to 2.1x.</figcaption></figure></div><p>That&#8217;s one task, so your numbers will differ. Harness Arena lets you run your own workload across configurations and compare cost and latency. Alibaba&#8217;s Qoder has also adopted UHP and is available as an agent backend through the HarnessRouter API.</p><p>GitHub repo: <strong><a href="https://github.com/HarnessRouter/harnessrouter">https://github.com/HarnessRouter/harnessrouter</a></strong></p><p>(don&#8217;t forget to star it &#11088;)</p><div><hr></div><h3><a href="https://www.dailydoseofds.com/guides/continuous-batching-in-llms/">Why other requests change your LLM&#8217;s output</a></h3><p>Your LLM&#8217;s output depends on requests that aren&#8217;t yours.</p><p>The same prompt can produce different logits depending on how many other requests share its batch on the GPU.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!fxG4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a8f8bb1-b631-4984-89b2-5f2d80b6df03_900x900.gif" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!fxG4!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a8f8bb1-b631-4984-89b2-5f2d80b6df03_900x900.gif 424w, https://substackcdn.com/image/fetch/$s_!fxG4!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a8f8bb1-b631-4984-89b2-5f2d80b6df03_900x900.gif 848w, https://substackcdn.com/image/fetch/$s_!fxG4!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a8f8bb1-b631-4984-89b2-5f2d80b6df03_900x900.gif 1272w, https://substackcdn.com/image/fetch/$s_!fxG4!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a8f8bb1-b631-4984-89b2-5f2d80b6df03_900x900.gif 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!fxG4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a8f8bb1-b631-4984-89b2-5f2d80b6df03_900x900.gif" width="900" height="900" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5a8f8bb1-b631-4984-89b2-5f2d80b6df03_900x900.gif&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:900,&quot;width&quot;:900,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:763612,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/gif&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222929994?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a8f8bb1-b631-4984-89b2-5f2d80b6df03_900x900.gif&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!fxG4!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a8f8bb1-b631-4984-89b2-5f2d80b6df03_900x900.gif 424w, https://substackcdn.com/image/fetch/$s_!fxG4!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a8f8bb1-b631-4984-89b2-5f2d80b6df03_900x900.gif 848w, https://substackcdn.com/image/fetch/$s_!fxG4!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a8f8bb1-b631-4984-89b2-5f2d80b6df03_900x900.gif 1272w, https://substackcdn.com/image/fetch/$s_!fxG4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a8f8bb1-b631-4984-89b2-5f2d80b6df03_900x900.gif 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">On a busy server, the same prompt goes through a different kernel split and can come out with a different token.</figcaption></figure></div><p>Thinking Machines Lab measured this. They sampled 1,000 completions of &#8220;Tell me about Richard Feynman&#8221; from Qwen3-235B at temperature 0 and got 80 unique completions.</p><p>All 1,000 matched for the first 102 tokens. At token 103, 992 continued with &#8220;Queens, New York&#8221; and 8 with &#8220;New York City&#8221;.</p><p>Here&#8217;s what happens inside the GPU.</p><h4>Why the order of additions matters</h4><p>Most of an LLM&#8217;s forward pass is spent accumulating sums. Every matmul, RMSNorm, and attention step reduces thousands of floating-point values into one output, so the order of those additions affects the result.</p><p>The catch is that floating-point addition isn&#8217;t associative. Every addition rounds to the nearest representable value, so <code>(a + b) + c</code> and <code>a + (b + c)</code> can differ in the low-order bits.</p><p>fp16 has an 11-bit significand, so between 1024 and 2048, representable values are 1.0 apart.</p><p>&#8627; Add 1024 + 0.4 + 0.4 left to right, and each step rounds back to 1024.</p><p>&#8627; Add the small values first, and 0.4 + 0.4 gives 0.8, then 1024 + 0.8 rounds to 1025.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!HkUl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91a0718c-33b7-42b1-b126-cccdda8956a9_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!HkUl!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91a0718c-33b7-42b1-b126-cccdda8956a9_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!HkUl!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91a0718c-33b7-42b1-b126-cccdda8956a9_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!HkUl!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91a0718c-33b7-42b1-b126-cccdda8956a9_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!HkUl!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91a0718c-33b7-42b1-b126-cccdda8956a9_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!HkUl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91a0718c-33b7-42b1-b126-cccdda8956a9_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/91a0718c-33b7-42b1-b126-cccdda8956a9_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:133060,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222929994?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91a0718c-33b7-42b1-b126-cccdda8956a9_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!HkUl!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91a0718c-33b7-42b1-b126-cccdda8956a9_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!HkUl!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91a0718c-33b7-42b1-b126-cccdda8956a9_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!HkUl!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91a0718c-33b7-42b1-b126-cccdda8956a9_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!HkUl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91a0718c-33b7-42b1-b126-cccdda8956a9_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The same three fp16 numbers give 1024 or 1025 depending on which pair gets added first.</figcaption></figure></div><h4>Why batch size changes the order</h4><p>GPU kernels pick the parallelization strategy that keeps the GPU busiest, and that choice depends on batch size.</p><p>An H100 SXM has 132 SMs, its compute units. A small batch doesn&#8217;t give each of them enough work.</p><p>So the kernel splits each long sum into chunks, adds the chunks on different SMs at the same time, then adds the partial results together. This is a split reduction, called split-K in matmuls and split-KV in attention.</p><p>A larger batch has enough rows to keep every SM busy, so each row&#8217;s sum stays on one SM and gets added in a single pass.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Jdg8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b81d7d-261b-4b5d-9f24-9b2d5effbd96_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Jdg8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b81d7d-261b-4b5d-9f24-9b2d5effbd96_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Jdg8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b81d7d-261b-4b5d-9f24-9b2d5effbd96_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Jdg8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b81d7d-261b-4b5d-9f24-9b2d5effbd96_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Jdg8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b81d7d-261b-4b5d-9f24-9b2d5effbd96_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Jdg8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b81d7d-261b-4b5d-9f24-9b2d5effbd96_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c0b81d7d-261b-4b5d-9f24-9b2d5effbd96_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:147495,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222929994?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b81d7d-261b-4b5d-9f24-9b2d5effbd96_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Jdg8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b81d7d-261b-4b5d-9f24-9b2d5effbd96_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Jdg8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b81d7d-261b-4b5d-9f24-9b2d5effbd96_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Jdg8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b81d7d-261b-4b5d-9f24-9b2d5effbd96_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Jdg8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0b81d7d-261b-4b5d-9f24-9b2d5effbd96_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A small batch gets its sums split across SMs, while a large batch keeps each sum on one SM.</figcaption></figure></div><p>Different batch size, different split, different order of additions, slightly different logits.</p><p>This isn&#8217;t random noise. Run the exact same batch twice and you get bitwise identical results. The output only changes when the batch around your request changes.</p><h4>How a tiny difference flips a token</h4><p>Usually the difference is negligible. It matters when two tokens are nearly tied.</p><p>Say that in a batch of 8, &#8220;Queens&#8221; scores 12.3417 and &#8220;New&#8221; scores 12.3415. In a batch of 32, rounding shifts them to 12.3414 and 12.3416.</p><p>At temperature 0, argmax picks &#8220;Queens&#8221; in the first case and &#8220;New&#8221; in the second. Every later token is conditioned on a different prefix, so the whole completion diverges.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!FV_n!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8112ea18-a9de-49f1-bfe9-28169b628ee5_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!FV_n!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8112ea18-a9de-49f1-bfe9-28169b628ee5_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!FV_n!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8112ea18-a9de-49f1-bfe9-28169b628ee5_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!FV_n!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8112ea18-a9de-49f1-bfe9-28169b628ee5_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!FV_n!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8112ea18-a9de-49f1-bfe9-28169b628ee5_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!FV_n!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8112ea18-a9de-49f1-bfe9-28169b628ee5_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8112ea18-a9de-49f1-bfe9-28169b628ee5_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:170362,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222929994?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8112ea18-a9de-49f1-bfe9-28169b628ee5_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!FV_n!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8112ea18-a9de-49f1-bfe9-28169b628ee5_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!FV_n!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8112ea18-a9de-49f1-bfe9-28169b628ee5_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!FV_n!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8112ea18-a9de-49f1-bfe9-28169b628ee5_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!FV_n!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8112ea18-a9de-49f1-bfe9-28169b628ee5_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A 0.0002 shift in two nearly tied logits is enough to change the greedy token.</figcaption></figure></div><p>So temperature 0 doesn&#8217;t make inference deterministic. The sampler is deterministic, but it reads logits from a forward pass whose numerics depend on traffic you don&#8217;t control.</p><h4>The fix: batch-invariant kernels</h4><p>The fix is to make every reduction batch invariant. Each row gets reduced in the same fixed order, no matter how many rows share the batch.</p><p>This costs throughput. The kernel can no longer pick the best split for each batch size, so it uses one fixed strategy even when that leaves SMs idle.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!oJ5x!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbafde5b8-35f1-4a6a-80a9-2af9da3f8975_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!oJ5x!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbafde5b8-35f1-4a6a-80a9-2af9da3f8975_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!oJ5x!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbafde5b8-35f1-4a6a-80a9-2af9da3f8975_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!oJ5x!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbafde5b8-35f1-4a6a-80a9-2af9da3f8975_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!oJ5x!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbafde5b8-35f1-4a6a-80a9-2af9da3f8975_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!oJ5x!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbafde5b8-35f1-4a6a-80a9-2af9da3f8975_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bafde5b8-35f1-4a6a-80a9-2af9da3f8975_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:145346,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222929994?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbafde5b8-35f1-4a6a-80a9-2af9da3f8975_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!oJ5x!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbafde5b8-35f1-4a6a-80a9-2af9da3f8975_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!oJ5x!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbafde5b8-35f1-4a6a-80a9-2af9da3f8975_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!oJ5x!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbafde5b8-35f1-4a6a-80a9-2af9da3f8975_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!oJ5x!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbafde5b8-35f1-4a6a-80a9-2af9da3f8975_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A batch-invariant kernel gives your request the same logits at any batch size, at the cost of throughput.</figcaption></figure></div><p>The major inference engines now expose this as a flag:</p><p>&#8627; vLLM has <code>VLLM_BATCH_INVARIANT=1</code>, still in beta.</p><p>&#8627; SGLang has <code>--enable-deterministic-inference</code>, with most slowdowns between 25% and 45% in the LMSYS benchmarks.</p><p>&#8627; DeepSeek&#8217;s DeepInfer has <code>DEEPINFER_BATCH_INVARIANT=1</code>, on NVIDIA GPUs only.</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;bash&quot;,&quot;nodeId&quot;:null}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-bash"># vLLM
VLLM_BATCH_INVARIANT=1 vllm serve Qwen/Qwen3-8B

# SGLang
python3 -m sglang.launch_server --model-path Qwen/Qwen3-8B \
  --attention-backend fa3 --enable-deterministic-inference</code></pre></div><p>With batch-invariant kernels on, Thinking Machines got all 1,000 Feynman completions identical.</p><p>If you run evals, RL rollouts, or anything else that needs reproducible outputs, turn it on and treat batch composition as part of your experimental setup.</p><p>RL gains the most. When the sampler and the trainer compute bitwise identical numbers, training stays truly on-policy. In the Thinking Machines run, the KL divergence between sampler and trainer stayed flat at 0.</p><h4>Why production makes this worse</h4><p>In production, the batch around your request never stays fixed. With continuous batching, the server adds and removes requests at every decoding step.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ksfb!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36cb07a8-6d37-4fe1-9f8c-f127d8cffaf0_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ksfb!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36cb07a8-6d37-4fe1-9f8c-f127d8cffaf0_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ksfb!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36cb07a8-6d37-4fe1-9f8c-f127d8cffaf0_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ksfb!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36cb07a8-6d37-4fe1-9f8c-f127d8cffaf0_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ksfb!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36cb07a8-6d37-4fe1-9f8c-f127d8cffaf0_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ksfb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36cb07a8-6d37-4fe1-9f8c-f127d8cffaf0_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/36cb07a8-6d37-4fe1-9f8c-f127d8cffaf0_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:159982,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222929994?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36cb07a8-6d37-4fe1-9f8c-f127d8cffaf0_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ksfb!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36cb07a8-6d37-4fe1-9f8c-f127d8cffaf0_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!ksfb!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36cb07a8-6d37-4fe1-9f8c-f127d8cffaf0_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!ksfb!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36cb07a8-6d37-4fe1-9f8c-f127d8cffaf0_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!ksfb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36cb07a8-6d37-4fe1-9f8c-f127d8cffaf0_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">With continuous batching, requests join and leave at every step, so the batch size keeps changing.</figcaption></figure></div><p>So your request can run at a different batch size for every token it generates.</p><div><hr></div><h3><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/">RAG vs. retriever-free RAG, clearly explained!</a></h3><p>In standard RAG, similarity search decides what the LLM can ever see.</p><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;9dd45fad-08c1-4113-a185-907aaa1a3fa7&quot;,&quot;duration&quot;:null}"></div><p>Documents are split into chunks, embedded, and stored in a vector index. When a query arrives, the system embeds it and pulls the top-k chunks with the closest vectors. A reranker reorders those candidates, and the best few go to the LLM.</p><p>If the right passage never made the top-k, nothing later in the pipeline can bring it back, not even a strong reranker.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xPFl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F298de5ee-b06c-45cf-a306-209c661f45d9_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xPFl!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F298de5ee-b06c-45cf-a306-209c661f45d9_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!xPFl!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F298de5ee-b06c-45cf-a306-209c661f45d9_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!xPFl!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F298de5ee-b06c-45cf-a306-209c661f45d9_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!xPFl!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F298de5ee-b06c-45cf-a306-209c661f45d9_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!xPFl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F298de5ee-b06c-45cf-a306-209c661f45d9_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/298de5ee-b06c-45cf-a306-209c661f45d9_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:143430,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222929994?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F298de5ee-b06c-45cf-a306-209c661f45d9_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!xPFl!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F298de5ee-b06c-45cf-a306-209c661f45d9_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!xPFl!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F298de5ee-b06c-45cf-a306-209c661f45d9_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!xPFl!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F298de5ee-b06c-45cf-a306-209c661f45d9_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!xPFl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F298de5ee-b06c-45cf-a306-209c661f45d9_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A passage that misses the top-k never reaches the reranker or the LLM.</figcaption></figure></div><p>Yesterday we covered vectorless RAG, where an LLM reads a document tree and picks sections. Retriever-free RAG, from a new Cornell and ASAPP paper, also walks a tree. But it uses a decision model to do it, not a generative LLM.</p><h4>What a decision model does</h4><p>A decision model doesn&#8217;t generate text. It reads one shared context and returns a probability for every yes/no question about it, in one forward pass.</p><p>The paper uses Jev 1.13. Put the query next to every candidate document, ask &#8220;Is this relevant?&#8221; for each, and a single pass scores all of them.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!YUT5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F183aa430-555b-4fd7-a059-3c9c7332ea5b_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!YUT5!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F183aa430-555b-4fd7-a059-3c9c7332ea5b_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!YUT5!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F183aa430-555b-4fd7-a059-3c9c7332ea5b_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!YUT5!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F183aa430-555b-4fd7-a059-3c9c7332ea5b_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!YUT5!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F183aa430-555b-4fd7-a059-3c9c7332ea5b_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!YUT5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F183aa430-555b-4fd7-a059-3c9c7332ea5b_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/183aa430-555b-4fd7-a059-3c9c7332ea5b_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:147981,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222929994?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F183aa430-555b-4fd7-a059-3c9c7332ea5b_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!YUT5!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F183aa430-555b-4fd7-a059-3c9c7332ea5b_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!YUT5!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F183aa430-555b-4fd7-a059-3c9c7332ea5b_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!YUT5!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F183aa430-555b-4fd7-a059-3c9c7332ea5b_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!YUT5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F183aa430-555b-4fd7-a059-3c9c7332ea5b_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A decision model reads one shared context and returns a probability for every question in one pass.</figcaption></figure></div><h4>How a query moves through the tree</h4><p>The documents are organized as a tree, like a table of contents, and the decision model walks down it:</p><p>&#8627; <strong>Top documents:</strong> the model reads the query next to every candidate document, asks &#8220;Is this relevant?&#8221; for each, and keeps the best few.</p><p>&#8627; <strong>Top passages:</strong> it opens only those documents and asks the same question for every passage inside them.</p><p>&#8627; <strong>Most relevant? (optional):</strong> a single multiple-choice question reorders the top 20 passages, or the top 5 on Natural Questions.</p><p>Dropped documents are never opened, so their passages are never scored.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!YYS4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41fa41a8-e30a-4af4-be7f-7d09433dd2d8_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!YYS4!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41fa41a8-e30a-4af4-be7f-7d09433dd2d8_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!YYS4!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41fa41a8-e30a-4af4-be7f-7d09433dd2d8_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!YYS4!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41fa41a8-e30a-4af4-be7f-7d09433dd2d8_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!YYS4!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41fa41a8-e30a-4af4-be7f-7d09433dd2d8_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!YYS4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41fa41a8-e30a-4af4-be7f-7d09433dd2d8_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/41fa41a8-e30a-4af4-be7f-7d09433dd2d8_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:133475,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222929994?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41fa41a8-e30a-4af4-be7f-7d09433dd2d8_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!YYS4!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41fa41a8-e30a-4af4-be7f-7d09433dd2d8_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!YYS4!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41fa41a8-e30a-4af4-be7f-7d09433dd2d8_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!YYS4!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41fa41a8-e30a-4af4-be7f-7d09433dd2d8_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!YYS4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41fa41a8-e30a-4af4-be7f-7d09433dd2d8_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Pruned clusters are never opened, so only passages inside kept documents get scored.</figcaption></figure></div><p>For Wikipedia, the tree was built offline. The 2.68M paragraphs from 108,626 pages were grouped into 1,000 clusters using e5-base-v2 embeddings and k-means, and Qwen3-8B wrote the summaries.</p><h4>The results</h4><p>On Natural Questions, its top pick contained the answer 65.4% of the time. Embedding search plus an 8B reranker got 48.0%, and GPT-5 reranking the dense top 20 got 60.8%.</p><p>The gain comes from ranking, not coverage. Embedding search&#8217;s top 100 holds the answer more often than the navigated shortlist, 98.4% vs. 94.0%. But when both contain it, navigation puts it first 70.1% of the time vs. 61.9%.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!d82E!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5de9d151-9fa2-4153-9b64-17b444513dd3_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!d82E!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5de9d151-9fa2-4153-9b64-17b444513dd3_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!d82E!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5de9d151-9fa2-4153-9b64-17b444513dd3_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!d82E!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5de9d151-9fa2-4153-9b64-17b444513dd3_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!d82E!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5de9d151-9fa2-4153-9b64-17b444513dd3_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!d82E!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5de9d151-9fa2-4153-9b64-17b444513dd3_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5de9d151-9fa2-4153-9b64-17b444513dd3_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:194398,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222929994?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5de9d151-9fa2-4153-9b64-17b444513dd3_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!d82E!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5de9d151-9fa2-4153-9b64-17b444513dd3_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!d82E!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5de9d151-9fa2-4153-9b64-17b444513dd3_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!d82E!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5de9d151-9fa2-4153-9b64-17b444513dd3_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!d82E!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5de9d151-9fa2-4153-9b64-17b444513dd3_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Navigation finds the answer slightly less often but ranks it first much more often.</figcaption></figure></div><p>Median retrieval takes under 2 seconds per query. Each method was timed in a different setup, though, so this isn&#8217;t a controlled speed comparison.</p><p>There are limits:</p><p>&#8627; It needs a sensible hierarchy.</p><p>&#8627; A document pruned early is gone for good.</p><p>&#8627; It skips embedding search only at query time, since the Wikipedia tree was built offline with embeddings.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!4RQi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0841e56e-0a96-431f-92ab-30644c50e415_1080x1080.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!4RQi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0841e56e-0a96-431f-92ab-30644c50e415_1080x1080.png 424w, https://substackcdn.com/image/fetch/$s_!4RQi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0841e56e-0a96-431f-92ab-30644c50e415_1080x1080.png 848w, https://substackcdn.com/image/fetch/$s_!4RQi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0841e56e-0a96-431f-92ab-30644c50e415_1080x1080.png 1272w, https://substackcdn.com/image/fetch/$s_!4RQi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0841e56e-0a96-431f-92ab-30644c50e415_1080x1080.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!4RQi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0841e56e-0a96-431f-92ab-30644c50e415_1080x1080.png" width="1080" height="1080" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0841e56e-0a96-431f-92ab-30644c50e415_1080x1080.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1080,&quot;width&quot;:1080,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:134039,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222929994?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0841e56e-0a96-431f-92ab-30644c50e415_1080x1080.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!4RQi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0841e56e-0a96-431f-92ab-30644c50e415_1080x1080.png 424w, https://substackcdn.com/image/fetch/$s_!4RQi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0841e56e-0a96-431f-92ab-30644c50e415_1080x1080.png 848w, https://substackcdn.com/image/fetch/$s_!4RQi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0841e56e-0a96-431f-92ab-30644c50e415_1080x1080.png 1272w, https://substackcdn.com/image/fetch/$s_!4RQi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0841e56e-0a96-431f-92ab-30644c50e415_1080x1080.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">RAG retrieves by similarity and reranks, while retriever-free RAG navigates a document tree with a decision model.</figcaption></figure></div><p>If you&#8217;re building a RAG system, this is worth benchmarking against your current setup.</p><p>Here&#8217;s the paper: <strong><a href="https://arxiv.org/abs/2610.35868">Retriever-Free Retrieval: Navigating Documents with Decision Models &#8594;</a></strong></p><p>Whichever retrieval you pick, a RAG system only works when every stage holds up: chunking, embeddings, retrieval, reranking, evaluation, and latency. Our full RAG crash course covers RAG from basics to beyond:</p><ul><li><p><strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/">Part 1: Build the complete RAG workflow, including chunking, embeddings, retrieval, and reranking &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-2-with-implementations/">Part 2: Evaluate retrieval and generation quality with RAG-specific metrics &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-3-with-implementation/">Part 3: Reduce RAG latency with faster retrieval and generation paths &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-4-with-implementation/">Part 4: Extend RAG beyond text to multiple data types &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-5-with-implementation/">Part 5: Understand CLIP embeddings, multimodal prompting, and tool calling &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-6-with-implementation/">Part 6: Build a multimodal RAG system over real-world data &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-7-with-implementation/">Part 7: Use Graph RAG for relationships that ordinary chunk retrieval misses &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-8-with-implementation/">Part 8: Improve retrieval with ColBERT and ColBERTv2 &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-9-with-implementation/">Part 9: Build vision-driven retrieval with ColPali &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/16-techniques-to-supercharge-and-build-real-world-rag-systems-part-1/">Part 10: Apply the first set of techniques for production RAG systems &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/16-techniques-to-supercharge-and-build-real-world-rag-systems-part-2/">Part 11: Complete the production RAG techniques and implementation patterns &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-12-with-implementation/">Part 12: Reduce prefill latency by reusing KV caches without a shared prefix &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-13-with-implementation/">Part 13: Preload a corpus before queries arrive and examine the production constraints &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-14-with-implementation/">Part 14: Compress preloaded caches and measure where the approaches break &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-15-with-implementation/">Part 15: Implement Block-Attention, Cartridges, and production preloading &#8594;</a></strong></p></li></ul><p>&#128073; Over to you: Do you turn on batch-invariant mode for your evals, or accept a little drift?</p><p>Good day!</p>]]></content:encoded></item><item><title><![CDATA[RAG vs. Vectorless RAG, clearly explained!]]></title><description><![CDATA[...explained visually (with tradeoffs in production).]]></description><link>https://blog.dailydoseofds.com/p/rag-vs-vectorless-rag-clearly-explained</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/rag-vs-vectorless-rag-clearly-explained</guid><dc:creator><![CDATA[Avi Chawla]]></dc:creator><pubDate>Fri, 30 Oct 2026 20:22:09 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!I9LO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa86e8c9a-f522-4db7-8663-e5a94aec7689_1080x1080.gif" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><strong>In today&#8217;s newsletter:</strong></p><ul><li><p>Speculative decoding, now 2.6x faster on a MacBook.</p></li><li><p>RAG vs. Vectorless RAG, clearly explained.</p></li></ul><div><hr></div><h3><a href="https://github.com/trymirai/uzu">Speculative decoding, now 2.6x faster on a MacBook:</a></h3><p>We ran Qwen3.5 9B on an M5 MacBook Pro with the same prompt, three ways:</p><ul><li><p>No speculative decoding: 22.0 tokens/s, done in 42.2s</p></li><li><p>Regular speculative decoding: 34.7 tokens/s, done in 30.4s</p></li><li><p>DFlash + Weaver in <strong><a href="https://github.com/trymirai/uzu">Uzu</a></strong>: 92.1 tokens/s, done in 11.0s</p></li></ul><p>That is about 4x faster than plain decoding and 2.6x faster than regular speculative decoding. The video shows all three runs side by side.</p><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;9333b42d-a898-419a-a397-81c8bc0751c4&quot;,&quot;duration&quot;:null}"></div><p>Speculative decoding lets a cheap drafter guess the next few tokens, and the main model checks them all in one pass.</p><p>The fastest drafters, like DFlash, guess every position in parallel. But each guess is made on its own, so position 5 never sees what was picked at position 4.</p><p>One early mismatch, and everything after it gets thrown away.</p><p>Weaver fixes this. It is a tiny adapter, a single Transformer layer with 56.7M parameters, that sits on top of DFlash and:</p><p>&#8627; picks among DFlash&#8217;s top candidates left to right, so each pick depends on the last one.</p><p>&#8627; grows a tree of likely continuations instead of a single chain.</p><p>&#8627; lets the main model verify the whole tree in one pass.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ASJB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9cb501-42fc-4ee1-957d-530b1a5fa2cd_1000x911.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ASJB!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9cb501-42fc-4ee1-957d-530b1a5fa2cd_1000x911.png 424w, https://substackcdn.com/image/fetch/$s_!ASJB!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9cb501-42fc-4ee1-957d-530b1a5fa2cd_1000x911.png 848w, https://substackcdn.com/image/fetch/$s_!ASJB!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9cb501-42fc-4ee1-957d-530b1a5fa2cd_1000x911.png 1272w, https://substackcdn.com/image/fetch/$s_!ASJB!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9cb501-42fc-4ee1-957d-530b1a5fa2cd_1000x911.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ASJB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9cb501-42fc-4ee1-957d-530b1a5fa2cd_1000x911.png" width="1000" height="911" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4f9cb501-42fc-4ee1-957d-530b1a5fa2cd_1000x911.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:911,&quot;width&quot;:1000,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:106827,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222815220?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9cb501-42fc-4ee1-957d-530b1a5fa2cd_1000x911.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ASJB!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9cb501-42fc-4ee1-957d-530b1a5fa2cd_1000x911.png 424w, https://substackcdn.com/image/fetch/$s_!ASJB!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9cb501-42fc-4ee1-957d-530b1a5fa2cd_1000x911.png 848w, https://substackcdn.com/image/fetch/$s_!ASJB!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9cb501-42fc-4ee1-957d-530b1a5fa2cd_1000x911.png 1272w, https://substackcdn.com/image/fetch/$s_!ASJB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9cb501-42fc-4ee1-957d-530b1a5fa2cd_1000x911.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>On the MacBook, every pass of the 9B model produced 7.5 tokens instead of 1.</p><p>On a B200 GPU, the <strong><a href="https://arxiv.org/abs/2607.06763">Weaver paper</a></strong> reports a 4.08x speedup over standard decoding, measured across chat, math, and code benchmarks.</p><p>Mirai open-sourced the whole stack in Uzu, its inference engine built for Apple Silicon. Trying it takes two commands:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;bash&quot;,&quot;nodeId&quot;:null}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-bash">brew install mirai
mirai --model trymirai/Qwen3.5-9B-M -m \
 "Write a python function to merge two sorted lists"</code></pre></div><p>GitHub repo: <strong><a href="https://github.com/trymirai/uzu">https://github.com/trymirai/uzu</a></strong></p><p>(don&#8217;t forget to star &#127775;)</p><div><hr></div><h3><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/">RAG vs. Vectorless RAG, clearly explained!</a></h3><p>The chunk closest to a question is often not the chunk that answers it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!I9LO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa86e8c9a-f522-4db7-8663-e5a94aec7689_1080x1080.gif" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!I9LO!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa86e8c9a-f522-4db7-8663-e5a94aec7689_1080x1080.gif 424w, https://substackcdn.com/image/fetch/$s_!I9LO!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa86e8c9a-f522-4db7-8663-e5a94aec7689_1080x1080.gif 848w, https://substackcdn.com/image/fetch/$s_!I9LO!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa86e8c9a-f522-4db7-8663-e5a94aec7689_1080x1080.gif 1272w, https://substackcdn.com/image/fetch/$s_!I9LO!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa86e8c9a-f522-4db7-8663-e5a94aec7689_1080x1080.gif 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!I9LO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa86e8c9a-f522-4db7-8663-e5a94aec7689_1080x1080.gif" width="533" height="533" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a86e8c9a-f522-4db7-8663-e5a94aec7689_1080x1080.gif&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1080,&quot;width&quot;:1080,&quot;resizeWidth&quot;:533,&quot;bytes&quot;:708693,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/gif&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222815220?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa86e8c9a-f522-4db7-8663-e5a94aec7689_1080x1080.gif&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!I9LO!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa86e8c9a-f522-4db7-8663-e5a94aec7689_1080x1080.gif 424w, https://substackcdn.com/image/fetch/$s_!I9LO!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa86e8c9a-f522-4db7-8663-e5a94aec7689_1080x1080.gif 848w, https://substackcdn.com/image/fetch/$s_!I9LO!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa86e8c9a-f522-4db7-8663-e5a94aec7689_1080x1080.gif 1272w, https://substackcdn.com/image/fetch/$s_!I9LO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa86e8c9a-f522-4db7-8663-e5a94aec7689_1080x1080.gif 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>In a standard RAG setup, documents are split into chunks, embedded, and stored in a vector DB. The query gets embedded too, and the top-k closest chunks go to the LLM.</p><p>This works when the closest text also holds the answer. On long, structured documents, that assumption breaks in two ways.</p><h4>Problems with similarity search</h4><p>Five annual reports from one company usually follow the same template every year, and only the numbers change.</p><p>So a question about FY2022 can land just as close to an FY2025 chunk as to the FY2022 one. The embedding captures what a chunk is about, and both chunks are about the same thing.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!GDNN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68f89e2a-62e0-45ec-bfa3-e32d70b45ee8_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!GDNN!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68f89e2a-62e0-45ec-bfa3-e32d70b45ee8_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!GDNN!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68f89e2a-62e0-45ec-bfa3-e32d70b45ee8_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!GDNN!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68f89e2a-62e0-45ec-bfa3-e32d70b45ee8_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!GDNN!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68f89e2a-62e0-45ec-bfa3-e32d70b45ee8_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!GDNN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68f89e2a-62e0-45ec-bfa3-e32d70b45ee8_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/68f89e2a-62e0-45ec-bfa3-e32d70b45ee8_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:129850,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222815220?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68f89e2a-62e0-45ec-bfa3-e32d70b45ee8_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!GDNN!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68f89e2a-62e0-45ec-bfa3-e32d70b45ee8_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!GDNN!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68f89e2a-62e0-45ec-bfa3-e32d70b45ee8_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!GDNN!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68f89e2a-62e0-45ec-bfa3-e32d70b45ee8_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!GDNN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F68f89e2a-62e0-45ec-bfa3-e32d70b45ee8_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Two debt sections from different years look almost identical to an embedding model.</figcaption></figure></div><p>The LLM then answers from the wrong year. Nothing in the pipeline flags it, because the retrieved chunk really is on topic.</p><p>Cross-references cause the second failure. A chunk might say &#8220;see Appendix G&#8221;, while the table in Appendix G holds the actual numbers.</p><p>That table looks nothing like the query, so it never makes it to the top-k. The pointer gets retrieved, but the answer it points to does not.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!1PrK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d6b025e-e3ee-44c9-8863-101e6a0a3b80_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!1PrK!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d6b025e-e3ee-44c9-8863-101e6a0a3b80_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!1PrK!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d6b025e-e3ee-44c9-8863-101e6a0a3b80_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!1PrK!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d6b025e-e3ee-44c9-8863-101e6a0a3b80_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!1PrK!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d6b025e-e3ee-44c9-8863-101e6a0a3b80_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!1PrK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d6b025e-e3ee-44c9-8863-101e6a0a3b80_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2d6b025e-e3ee-44c9-8863-101e6a0a3b80_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:141113,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222815220?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d6b025e-e3ee-44c9-8863-101e6a0a3b80_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!1PrK!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d6b025e-e3ee-44c9-8863-101e6a0a3b80_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!1PrK!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d6b025e-e3ee-44c9-8863-101e6a0a3b80_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!1PrK!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d6b025e-e3ee-44c9-8863-101e6a0a3b80_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!1PrK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d6b025e-e3ee-44c9-8863-101e6a0a3b80_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The chunk that mentions Appendix G gets retrieved, but the appendix table itself never does.</figcaption></figure></div><p>Fixes on the vector side do help. Anthropic&#8217;s contextual retrieval adds document context to each chunk before embedding, and with a reranker it cut top-20 retrieval failures from 5.7% to 1.9%.</p><p>But these fixes still rank text by how similar it looks to the query.</p><h4>How vectorless RAG works</h4><p>Vectorless RAG drops the embedding model and the vector DB, and replaces similarity search with LLM reasoning.</p><p>There are three parts to the process.</p><p>First, a preprocessing step parses each document&#8217;s layout into a ToC tree index. Every node holds a section title, its page range, and a short summary.</p><p>Nothing gets chunked, so a table never gets split across two chunks. And the whole tree is small enough to fit in the LLM&#8217;s context. In PageIndex, the most popular open-source implementation, the structure comes from the layout itself, and a basic model only writes the summaries.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!skgC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1d96027-735d-413d-87cc-e7ab7ad71d88_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!skgC!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1d96027-735d-413d-87cc-e7ab7ad71d88_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!skgC!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1d96027-735d-413d-87cc-e7ab7ad71d88_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!skgC!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1d96027-735d-413d-87cc-e7ab7ad71d88_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!skgC!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1d96027-735d-413d-87cc-e7ab7ad71d88_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!skgC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1d96027-735d-413d-87cc-e7ab7ad71d88_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e1d96027-735d-413d-87cc-e7ab7ad71d88_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:171037,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222815220?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1d96027-735d-413d-87cc-e7ab7ad71d88_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!skgC!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1d96027-735d-413d-87cc-e7ab7ad71d88_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!skgC!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1d96027-735d-413d-87cc-e7ab7ad71d88_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!skgC!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1d96027-735d-413d-87cc-e7ab7ad71d88_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!skgC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1d96027-735d-413d-87cc-e7ab7ad71d88_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The index is a table of contents where every node stores a title, a page range and a short summary.</figcaption></figure></div><p>Next, at query time, the LLM gets the query along with the tree. It reads the section titles and summaries and picks the few sections most likely to hold the answer.</p><p>For example, for a question about debt, it might pick the &#8220;Financial Summary&#8221; section and also &#8220;Appendix G&#8221;, because its summary says the debt details are there.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!y7hc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36747d1e-8b6e-430e-9631-2e04374a67a0_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!y7hc!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36747d1e-8b6e-430e-9631-2e04374a67a0_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!y7hc!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36747d1e-8b6e-430e-9631-2e04374a67a0_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!y7hc!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36747d1e-8b6e-430e-9631-2e04374a67a0_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!y7hc!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36747d1e-8b6e-430e-9631-2e04374a67a0_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!y7hc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36747d1e-8b6e-430e-9631-2e04374a67a0_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/36747d1e-8b6e-430e-9631-2e04374a67a0_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:164444,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222815220?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36747d1e-8b6e-430e-9631-2e04374a67a0_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!y7hc!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36747d1e-8b6e-430e-9631-2e04374a67a0_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!y7hc!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36747d1e-8b6e-430e-9631-2e04374a67a0_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!y7hc!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36747d1e-8b6e-430e-9631-2e04374a67a0_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!y7hc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36747d1e-8b6e-430e-9631-2e04374a67a0_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The LLM picks sections by reading their summaries, so it can follow a reference to an appendix.</figcaption></figure></div><p>That is how an analyst reads a 10-K. They open the table of contents, go to the financial summary, and follow the note to the appendix.</p><p>Finally, the system pulls those full sections and checks whether the evidence is enough.</p><p>&#8627; If it is, the LLM answers from the full sections, with page citations.</p><p>&#8627; If not, it goes back to the tree and opens more nodes.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!DmBh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b1ad9d1-83ac-462f-9d0c-2dad81c802b0_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!DmBh!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b1ad9d1-83ac-462f-9d0c-2dad81c802b0_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!DmBh!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b1ad9d1-83ac-462f-9d0c-2dad81c802b0_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!DmBh!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b1ad9d1-83ac-462f-9d0c-2dad81c802b0_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!DmBh!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b1ad9d1-83ac-462f-9d0c-2dad81c802b0_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!DmBh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b1ad9d1-83ac-462f-9d0c-2dad81c802b0_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3b1ad9d1-83ac-462f-9d0c-2dad81c802b0_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:116650,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222815220?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b1ad9d1-83ac-462f-9d0c-2dad81c802b0_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!DmBh!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b1ad9d1-83ac-462f-9d0c-2dad81c802b0_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!DmBh!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b1ad9d1-83ac-462f-9d0c-2dad81c802b0_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!DmBh!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b1ad9d1-83ac-462f-9d0c-2dad81c802b0_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!DmBh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b1ad9d1-83ac-462f-9d0c-2dad81c802b0_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">When the evidence falls short, the system goes back to the tree and opens more nodes.</figcaption></figure></div><p>The LLM also sees the chat history. So if you ask about assets and then follow up with &#8220;what about liabilities?&#8221;, it goes back to the same report.</p><h4>What it costs</h4><p>The trade-off is that every query now runs LLM calls over the tree.</p><p>PageIndex tested this on 63 questions over NVIDIA&#8217;s FY2022 to FY2026 annual reports. Note that the benchmark comes from the PageIndex team itself.</p><ul><li><p>Tree retrieval answered all 63 correctly, at about $110 per 1,000 queries.</p></li><li><p>The best vector setup reached 92.1%, and only by sending the top 100 chunks, at about $134.</p></li><li><p>At top-10, vector search cost about $15 per 1,000 queries, but no embedding model got past 76.2%.</p></li></ul><p>So vector search is cheaper per query at a normal k, but it can be less accurate on documents like these.</p><p>Latency follows the same pattern. Every hop through the tree is another LLM call, and PageIndex has not published latency numbers.</p><p>The index costs more too. One independent benchmark across about 1,000 documents skipped the tree entirely, because building one took 2 to 5 minutes of LLM calls per document.</p><p>And the tree only helps when documents have real structure. A pile of support tickets has no table of contents to reason over.</p><p>So the two suit different data:</p><p>&#8627; Use a tree for long, structured documents like filings, contracts, and manuals.</p><p>&#8627; Stick with a vector DB for lots of short, unrelated text.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Ak_o!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70cd3966-4aef-4495-a768-7926c413fccf_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Ak_o!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70cd3966-4aef-4495-a768-7926c413fccf_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Ak_o!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70cd3966-4aef-4495-a768-7926c413fccf_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Ak_o!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70cd3966-4aef-4495-a768-7926c413fccf_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Ak_o!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70cd3966-4aef-4495-a768-7926c413fccf_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Ak_o!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70cd3966-4aef-4495-a768-7926c413fccf_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/70cd3966-4aef-4495-a768-7926c413fccf_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:131784,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222815220?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70cd3966-4aef-4495-a768-7926c413fccf_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Ak_o!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70cd3966-4aef-4495-a768-7926c413fccf_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Ak_o!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70cd3966-4aef-4495-a768-7926c413fccf_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Ak_o!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70cd3966-4aef-4495-a768-7926c413fccf_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Ak_o!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F70cd3966-4aef-4495-a768-7926c413fccf_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Long structured documents suit a tree index, while large piles of short text suit a vector DB.</figcaption></figure></div><p>Vector search asks which text looks like the question. Vectorless RAG asks where in the document the answer should be.</p><p>If you want to dive deeper, our full RAG course covers RAG from basics to beyond:</p><ul><li><p><strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/">Part 1: Build the complete RAG workflow, including chunking, embeddings, retrieval, and reranking &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-2-with-implementations/">Part 2: Evaluate retrieval and generation quality with RAG-specific metrics &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-3-with-implementation/">Part 3: Reduce RAG latency with faster retrieval and generation paths &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-4-with-implementation/">Part 4: Extend RAG beyond text to multiple data types &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-5-with-implementation/">Part 5: Understand CLIP embeddings, multimodal prompting, and tool calling &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-6-with-implementation/">Part 6: Build a multimodal RAG system over real-world data &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-7-with-implementation/">Part 7: Use Graph RAG for relationships that ordinary chunk retrieval misses &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-8-with-implementation/">Part 8: Improve retrieval with ColBERT and ColBERTv2 &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-9-with-implementation/">Part 9: Build vision-driven retrieval with ColPali &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/16-techniques-to-supercharge-and-build-real-world-rag-systems-part-1/">Part 10: Apply the first set of techniques for production RAG systems &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/16-techniques-to-supercharge-and-build-real-world-rag-systems-part-2/">Part 11: Complete the production RAG techniques and implementation patterns &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-12-with-implementation/">Part 12: Reduce prefill latency by reusing KV caches without a shared prefix &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-13-with-implementation/">Part 13: Preload a corpus before queries arrive and examine the production constraints &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-14-with-implementation/">Part 14: Compress preloaded caches and measure where the approaches break &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/building-rag-systems-course-part-15-with-implementation/">Part 15: Implement Block-Attention, Cartridges, and production preloading &#8594;</a></strong></p></li></ul><p>&#128073; Over to you: Do your documents have enough structure for a tree index, or does a vector DB still fit better?</p><p>Good day!</p>]]></content:encoded></item><item><title><![CDATA[Prefix-Aware LLM Routing, clearly explained]]></title><description><![CDATA[...explained visually (the trick to get the first LLM token 170x sooner).]]></description><link>https://blog.dailydoseofds.com/p/prefix-aware-llm-routing-clearly</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/prefix-aware-llm-routing-clearly</guid><dc:creator><![CDATA[Avi Chawla]]></dc:creator><pubDate>Thu, 29 Oct 2026 20:42:27 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!9iec!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d894230-732b-485d-9d5e-3702526b42eb_1100x928.gif" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><strong>In today&#8217;s newsletter:</strong></p><ul><li><p>Training LLMs without one correct answer.</p></li><li><p>Prefix-aware LLM routing, clearly explained.</p></li><li><p>Regular ML Inference vs. LLM Inference</p></li></ul><div><hr></div><h3><a href="https://fandf.co/46QX2pe">Training LLMs without one correct answer</a></h3><p>Suppose a model must translate questions from finance analysts into SQL.</p><p>A basic verifier might run each generated query and return a reward of 1 when it executes without an error.</p><p>Due to this, the model will learn to produce executable SQL, but it will not necessarily learn to produce correct SQL.</p><p>A query can select the wrong columns, omit a filter, duplicate rows after a join, or calculate revenue over the wrong period. The database will execute all of them without complaint.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!DJfF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F596500a0-7f57-4345-8713-028e2f27bddc_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!DJfF!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F596500a0-7f57-4345-8713-028e2f27bddc_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!DJfF!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F596500a0-7f57-4345-8713-028e2f27bddc_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!DJfF!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F596500a0-7f57-4345-8713-028e2f27bddc_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!DJfF!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F596500a0-7f57-4345-8713-028e2f27bddc_2752x1536.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!DJfF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F596500a0-7f57-4345-8713-028e2f27bddc_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/596500a0-7f57-4345-8713-028e2f27bddc_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:145085,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222635654?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F596500a0-7f57-4345-8713-028e2f27bddc_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!DJfF!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F596500a0-7f57-4345-8713-028e2f27bddc_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!DJfF!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F596500a0-7f57-4345-8713-028e2f27bddc_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!DJfF!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F596500a0-7f57-4345-8713-028e2f27bddc_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!DJfF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F596500a0-7f57-4345-8713-028e2f27bddc_2752x1536.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The database executes all three queries, but only one of them returns the right rows.</figcaption></figure></div><p>This is the part of RLVR that deserves more attention. A verifier inherently does not discover what correctness means, which is why the engineer must encode it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!oVT1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a0ed61-dd53-462e-8d7e-d659a53ea920_1080x1080.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!oVT1!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a0ed61-dd53-462e-8d7e-d659a53ea920_1080x1080.png 424w, https://substackcdn.com/image/fetch/$s_!oVT1!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a0ed61-dd53-462e-8d7e-d659a53ea920_1080x1080.png 848w, https://substackcdn.com/image/fetch/$s_!oVT1!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a0ed61-dd53-462e-8d7e-d659a53ea920_1080x1080.png 1272w, https://substackcdn.com/image/fetch/$s_!oVT1!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a0ed61-dd53-462e-8d7e-d659a53ea920_1080x1080.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!oVT1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a0ed61-dd53-462e-8d7e-d659a53ea920_1080x1080.png" width="1080" height="1080" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/59a0ed61-dd53-462e-8d7e-d659a53ea920_1080x1080.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1080,&quot;width&quot;:1080,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:47969,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222635654?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a0ed61-dd53-462e-8d7e-d659a53ea920_1080x1080.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!oVT1!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a0ed61-dd53-462e-8d7e-d659a53ea920_1080x1080.png 424w, https://substackcdn.com/image/fetch/$s_!oVT1!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a0ed61-dd53-462e-8d7e-d659a53ea920_1080x1080.png 848w, https://substackcdn.com/image/fetch/$s_!oVT1!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a0ed61-dd53-462e-8d7e-d659a53ea920_1080x1080.png 1272w, https://substackcdn.com/image/fetch/$s_!oVT1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59a0ed61-dd53-462e-8d7e-d659a53ea920_1080x1080.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Verifier engineering turns a vague goal into checks that a reward function can <strong>run.</strong></figcaption></figure></div><p><strong><a href="https://fandf.co/46QX2pe">Amazon SageMaker AI</a></strong> makes it simple to test that definition through its serverless model customization workflow.</p><p>In SageMaker Studio, the engineer can select a base model, prepare the examples, attach a custom reward function, and submit an RLVR job without managing the training infrastructure.</p><p>SageMaker AI provisions the compute, runs and tracks the job, stores the resulting model artifacts, and tears the infrastructure down afterward.</p><p>That makes it easier to iterate on the model and reward logic as separate parts of the experiment.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!5MZk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc65ac6ad-95b3-482b-9671-55a6ebdb4e71_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!5MZk!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc65ac6ad-95b3-482b-9671-55a6ebdb4e71_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!5MZk!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc65ac6ad-95b3-482b-9671-55a6ebdb4e71_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!5MZk!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc65ac6ad-95b3-482b-9671-55a6ebdb4e71_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!5MZk!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc65ac6ad-95b3-482b-9671-55a6ebdb4e71_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!5MZk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc65ac6ad-95b3-482b-9671-55a6ebdb4e71_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c65ac6ad-95b3-482b-9671-55a6ebdb4e71_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:145602,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222635654?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc65ac6ad-95b3-482b-9671-55a6ebdb4e71_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!5MZk!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc65ac6ad-95b3-482b-9671-55a6ebdb4e71_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!5MZk!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc65ac6ad-95b3-482b-9671-55a6ebdb4e71_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!5MZk!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc65ac6ad-95b3-482b-9671-55a6ebdb4e71_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!5MZk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc65ac6ad-95b3-482b-9671-55a6ebdb4e71_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">SageMaker AI runs the training job while the engineer defines what success means.</figcaption></figure></div><p>For text-to-SQL, you can split the verifier into layers.</p><ul><li><p>The first layer checks whether the query parses, executes, and remains read-only.</p></li><li><p>The second compares the returned rows with the reference result. This catches queries that are valid SQL but answer a different question.</p></li><li><p>The final layer can score operational properties such as latency or scanned data, but only after the query has passed the correctness checks.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!avzQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa480df81-ed84-4c7b-ae7c-1367e2472ba3_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!avzQ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa480df81-ed84-4c7b-ae7c-1367e2472ba3_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!avzQ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa480df81-ed84-4c7b-ae7c-1367e2472ba3_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!avzQ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa480df81-ed84-4c7b-ae7c-1367e2472ba3_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!avzQ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa480df81-ed84-4c7b-ae7c-1367e2472ba3_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!avzQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa480df81-ed84-4c7b-ae7c-1367e2472ba3_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a480df81-ed84-4c7b-ae7c-1367e2472ba3_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:119236,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222635654?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa480df81-ed84-4c7b-ae7c-1367e2472ba3_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!avzQ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa480df81-ed84-4c7b-ae7c-1367e2472ba3_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!avzQ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa480df81-ed84-4c7b-ae7c-1367e2472ba3_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!avzQ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa480df81-ed84-4c7b-ae7c-1367e2472ba3_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!avzQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa480df81-ed84-4c7b-ae7c-1367e2472ba3_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Correctness works as a gate, and efficiency only ranks the queries that pass it.</figcaption></figure></div><p>The order is important because a fast query that returns the wrong rows should never outrank a slower query that returns the right ones.</p><p>During customization, the model generates several candidates for each prompt. The verifier scores them, and GRPO updates the model toward the candidates that satisfy the full specification.</p><p>This also gives the evaluation stage a clear purpose. A higher training reward is not enough. The customized model must improve on held-out execution accuracy without increasing unsafe queries or weakening result correctness.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!oPae!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc955f56e-4104-4866-b7d5-82871fbb880f_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!oPae!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc955f56e-4104-4866-b7d5-82871fbb880f_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!oPae!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc955f56e-4104-4866-b7d5-82871fbb880f_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!oPae!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc955f56e-4104-4866-b7d5-82871fbb880f_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!oPae!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc955f56e-4104-4866-b7d5-82871fbb880f_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!oPae!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc955f56e-4104-4866-b7d5-82871fbb880f_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c955f56e-4104-4866-b7d5-82871fbb880f_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:150350,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222635654?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc955f56e-4104-4866-b7d5-82871fbb880f_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!oPae!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc955f56e-4104-4866-b7d5-82871fbb880f_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!oPae!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc955f56e-4104-4866-b7d5-82871fbb880f_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!oPae!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc955f56e-4104-4866-b7d5-82871fbb880f_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!oPae!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc955f56e-4104-4866-b7d5-82871fbb880f_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A higher training reward only matters when behavior on held-out prompts improves too.</figcaption></figure></div><p><strong><a href="https://fandf.co/46QX2pe">SageMaker AI</a></strong> agent skills can help with model selection, data preparation, job configuration, evaluation, and deployment through Claude Code, Codex, Kiro, or another compatible coding agent.</p><p>AWS has published an interactive workshop for anyone who wants to get hands-on with serverless model customization in SageMaker AI:</p><p><strong><a href="https://fandf.co/46QX2pe">Try the SageMaker AI model customization workshop &#8594;</a></strong></p><p>Thanks to AWS for working with us on today&#8217;s issue!</p><div><hr></div><h3><a href="https://www.dailydoseofds.com/p/kv-vs-prefix-vs-prompt-vs-semantic-caching/">Prefix-Aware LLM Routing, clearly explained</a></h3><p>The same request on the same GPUs can wait 92 seconds or half a second for its first token, depending only on where the router sends it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!9iec!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d894230-732b-485d-9d5e-3702526b42eb_1100x928.gif" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!9iec!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d894230-732b-485d-9d5e-3702526b42eb_1100x928.gif 424w, https://substackcdn.com/image/fetch/$s_!9iec!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d894230-732b-485d-9d5e-3702526b42eb_1100x928.gif 848w, https://substackcdn.com/image/fetch/$s_!9iec!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d894230-732b-485d-9d5e-3702526b42eb_1100x928.gif 1272w, https://substackcdn.com/image/fetch/$s_!9iec!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d894230-732b-485d-9d5e-3702526b42eb_1100x928.gif 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!9iec!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d894230-732b-485d-9d5e-3702526b42eb_1100x928.gif" width="1100" height="928" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4d894230-732b-485d-9d5e-3702526b42eb_1100x928.gif&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:928,&quot;width&quot;:1100,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:711967,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/gif&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222635654?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d894230-732b-485d-9d5e-3702526b42eb_1100x928.gif&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!9iec!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d894230-732b-485d-9d5e-3702526b42eb_1100x928.gif 424w, https://substackcdn.com/image/fetch/$s_!9iec!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d894230-732b-485d-9d5e-3702526b42eb_1100x928.gif 848w, https://substackcdn.com/image/fetch/$s_!9iec!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d894230-732b-485d-9d5e-3702526b42eb_1100x928.gif 1272w, https://substackcdn.com/image/fetch/$s_!9iec!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d894230-732b-485d-9d5e-3702526b42eb_1100x928.gif 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A prefix-aware router sends each request to the worker that already holds the longest matching part of its prompt.</figcaption></figure></div><p>Production prompts start with the same system prompt, tool definitions and earlier turns. Only the last part of each request is new.</p><p>So the GPU that served the previous turn still holds the KV cache for that shared prefix.</p><p>But a cache-blind router may send the next request to a different GPU, which then recomputes the whole prefix before it can write a single token.</p><p>A prefix-aware router avoids this by tracking which GPU holds which part of each prompt.</p><p>Let&#8217;s walk through how it works.</p><h4>1. Why routing decides the cache hit</h4><p>Inside one engine, prefix caching already works well. In llm-d&#8217;s tests, sending a roughly 10,000-token prompt to a Qwen3-32B vLLM instance a second time cut time to first token from 4.3s to 0.6s.</p><p>The trouble starts when we scale out. Each replica keeps its own KV cache, and no replica can see another&#8217;s.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!tv5H!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27d0d71a-d9cc-464d-b8e7-1ac13353c264_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!tv5H!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27d0d71a-d9cc-464d-b8e7-1ac13353c264_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!tv5H!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27d0d71a-d9cc-464d-b8e7-1ac13353c264_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!tv5H!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27d0d71a-d9cc-464d-b8e7-1ac13353c264_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!tv5H!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27d0d71a-d9cc-464d-b8e7-1ac13353c264_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!tv5H!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27d0d71a-d9cc-464d-b8e7-1ac13353c264_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/27d0d71a-d9cc-464d-b8e7-1ac13353c264_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:173464,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222635654?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27d0d71a-d9cc-464d-b8e7-1ac13353c264_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!tv5H!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27d0d71a-d9cc-464d-b8e7-1ac13353c264_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!tv5H!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27d0d71a-d9cc-464d-b8e7-1ac13353c264_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!tv5H!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27d0d71a-d9cc-464d-b8e7-1ac13353c264_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!tv5H!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27d0d71a-d9cc-464d-b8e7-1ac13353c264_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A round-robin router sends the second turn to a GPU that has to recompute the whole prefix.</figcaption></figure></div><p>With 8 replicas and random routing, the next turn lands on the warm GPU about 1 time in 8.</p><p>The other 7 times, a GPU recomputes a prefix that already sits in memory somewhere else in the cluster.</p><p>So in a cluster, the router decides whether the engine&#8217;s prefix cache gets used at all.</p><h4>2. Hash the prompt block by block</h4><p>The engine stores the KV cache in fixed-size blocks. vLLM uses 16 tokens per block by default and only caches full blocks.</p><p>Each block gets a hash built from its parent block&#8217;s hash plus its own token IDs. So the hash of block D covers A, B, C and D together.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!I4AF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76f70169-d599-4687-8922-b14beac0ede5_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!I4AF!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76f70169-d599-4687-8922-b14beac0ede5_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!I4AF!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76f70169-d599-4687-8922-b14beac0ede5_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!I4AF!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76f70169-d599-4687-8922-b14beac0ede5_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!I4AF!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76f70169-d599-4687-8922-b14beac0ede5_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!I4AF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76f70169-d599-4687-8922-b14beac0ede5_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/76f70169-d599-4687-8922-b14beac0ede5_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:156793,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222635654?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76f70169-d599-4687-8922-b14beac0ede5_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!I4AF!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76f70169-d599-4687-8922-b14beac0ede5_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!I4AF!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76f70169-d599-4687-8922-b14beac0ede5_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!I4AF!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76f70169-d599-4687-8922-b14beac0ede5_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!I4AF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76f70169-d599-4687-8922-b14beac0ede5_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Each block hash includes its parent hash, so one changed token changes every hash after it.</figcaption></figure></div><p>Here is a simplified version of what vLLM does:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;python&quot;,&quot;nodeId&quot;:null}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-python">def block_hashes(token_ids, block_size=16):
    hashes, parent = [], None
    for i in range(0, len(token_ids) - block_size + 1, block_size):
        block = tuple(token_ids[i:i + block_size])
        parent = hash((parent, block))
        hashes.append(parent)
    return hashes</code></pre></div><p>This chaining turns a block lookup into a prefix lookup. If block D&#8217;s hash matches, everything before it matched too.</p><p>The router computes the same hashes for each incoming request, so it can compare prompts without touching any KV tensors.</p><h4>3. Look up which worker holds those blocks</h4><p>The router keeps a block-to-worker index. It maps each block hash to the workers that hold that block, and it never stores the KV tensors themselves.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!h_wN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4171ec24-e962-4b24-8da8-72dc676337ef_1250x427.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!h_wN!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4171ec24-e962-4b24-8da8-72dc676337ef_1250x427.jpeg 424w, https://substackcdn.com/image/fetch/$s_!h_wN!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4171ec24-e962-4b24-8da8-72dc676337ef_1250x427.jpeg 848w, https://substackcdn.com/image/fetch/$s_!h_wN!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4171ec24-e962-4b24-8da8-72dc676337ef_1250x427.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!h_wN!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4171ec24-e962-4b24-8da8-72dc676337ef_1250x427.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!h_wN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4171ec24-e962-4b24-8da8-72dc676337ef_1250x427.jpeg" width="1250" height="427" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4171ec24-e962-4b24-8da8-72dc676337ef_1250x427.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:427,&quot;width&quot;:1250,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:51599,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222635654?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4171ec24-e962-4b24-8da8-72dc676337ef_1250x427.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!h_wN!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4171ec24-e962-4b24-8da8-72dc676337ef_1250x427.jpeg 424w, https://substackcdn.com/image/fetch/$s_!h_wN!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4171ec24-e962-4b24-8da8-72dc676337ef_1250x427.jpeg 848w, https://substackcdn.com/image/fetch/$s_!h_wN!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4171ec24-e962-4b24-8da8-72dc676337ef_1250x427.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!h_wN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4171ec24-e962-4b24-8da8-72dc676337ef_1250x427.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The router hashes the prompt and checks how many leading blocks each worker already holds.</figcaption></figure></div><p>That keeps the index tiny. llm-d ran the numbers for DeepSeek-R1 in FP8 on 8 H200s, where each 128-token block takes about 8.6 MB of VRAM but only an 8-byte hash in the index.</p><p>So a 365 GB KV cache pool needs only about 339 KB of index.</p><p>For a request with prefix blocks A to D and a new suffix E, the router walks the hashes in order:</p><p>&#8627; Worker A holds 2 matching blocks.</p><p>&#8627; Worker C holds 3.</p><p>&#8627; Worker B holds all 4, so Worker B gets the request.</p><p>If no worker has a match, the request gets a full prefill on whichever worker the router picks.</p><h4>4. Reuse the cache and report back</h4><p>Worker B reuses the cached KV for A to D, computes KV only for E, and starts decoding.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!_njm!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe250ac89-f5b3-455c-9a4f-05702de5367f_1250x443.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_njm!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe250ac89-f5b3-455c-9a4f-05702de5367f_1250x443.jpeg 424w, https://substackcdn.com/image/fetch/$s_!_njm!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe250ac89-f5b3-455c-9a4f-05702de5367f_1250x443.jpeg 848w, https://substackcdn.com/image/fetch/$s_!_njm!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe250ac89-f5b3-455c-9a4f-05702de5367f_1250x443.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!_njm!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe250ac89-f5b3-455c-9a4f-05702de5367f_1250x443.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!_njm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe250ac89-f5b3-455c-9a4f-05702de5367f_1250x443.jpeg" width="1250" height="443" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e250ac89-f5b3-455c-9a4f-05702de5367f_1250x443.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:443,&quot;width&quot;:1250,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:42517,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222635654?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe250ac89-f5b3-455c-9a4f-05702de5367f_1250x443.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!_njm!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe250ac89-f5b3-455c-9a4f-05702de5367f_1250x443.jpeg 424w, https://substackcdn.com/image/fetch/$s_!_njm!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe250ac89-f5b3-455c-9a4f-05702de5367f_1250x443.jpeg 848w, https://substackcdn.com/image/fetch/$s_!_njm!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe250ac89-f5b3-455c-9a4f-05702de5367f_1250x443.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!_njm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe250ac89-f5b3-455c-9a4f-05702de5367f_1250x443.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Worker B skips prefill for the cached blocks and reports its new block back to the router.</figcaption></figure></div><p>Once E is cached, Worker B emits a cache event and the router records that E now lives on Worker B. So the next turn of this conversation lands on Worker B too.</p><p>Evictions emit events as well. In llm-d, each vLLM pod publishes these KV events over ZMQ every time a block is created or evicted, so the index tracks what each GPU actually holds.</p><p>That is the difference between precise and approximate routing. An approximate router guesses cache locations from its own routing history instead.</p><h4>5. Weigh load against cache hits</h4><p>If the router only chased cache matches, the warmest GPU would end up with the longest queue, because every matching request wants it.</p><p>So real routers score load too. NVIDIA Dynamo&#8217;s KV router adds the prefill a worker would still owe after cache credit to that worker&#8217;s decode load, then picks the lowest total.</p><p>Here is the example from Dynamo&#8217;s docs, for a request with 10 prefill blocks:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;plaintext&quot;,&quot;nodeId&quot;:null}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-plaintext">cost = (prefill blocks - cached blocks) + decode blocks

Worker 1: (10 - 2) + 10 = 18
Worker 2: (10 - 5) + 5  = 10   &lt;- picked
Worker 3: (10 - 8) + 9  = 11</code></pre></div><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!DTX9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda64df2f-f0fe-430c-a952-1bdf52419c02_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!DTX9!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda64df2f-f0fe-430c-a952-1bdf52419c02_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!DTX9!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda64df2f-f0fe-430c-a952-1bdf52419c02_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!DTX9!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda64df2f-f0fe-430c-a952-1bdf52419c02_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!DTX9!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda64df2f-f0fe-430c-a952-1bdf52419c02_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!DTX9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda64df2f-f0fe-430c-a952-1bdf52419c02_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/da64df2f-f0fe-430c-a952-1bdf52419c02_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:189184,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222635654?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda64df2f-f0fe-430c-a952-1bdf52419c02_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!DTX9!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda64df2f-f0fe-430c-a952-1bdf52419c02_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!DTX9!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda64df2f-f0fe-430c-a952-1bdf52419c02_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!DTX9!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda64df2f-f0fe-430c-a952-1bdf52419c02_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!DTX9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda64df2f-f0fe-430c-a952-1bdf52419c02_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Worker 3 has the best cache match, but Worker 2 wins because it carries less load.</figcaption></figure></div><p>Worker 3 has the best match, yet Worker 2 wins. The router accepts recomputing 3 extra blocks to avoid the busier GPU.</p><p>Dynamo exposes this tradeoff as a knob. More cache credit lowers time to first token, and less credit spreads load more evenly and lowers inter-token latency.</p><div><hr></div><p>llm-d compared four routing strategies on 8 vLLM pods across 16 H100s.</p><p>The workload simulated 150 customers, each with a 6,000-token shared context and 5 users sending 1,200-token questions, at 3 to 60 QPS. Those shared prefixes needed about 73% of the cluster&#8217;s KV cache, roughly 6x what one pod could hold.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!wAmH!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febafd696-0ae4-4c26-a171-6d4abdfa32d2_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!wAmH!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febafd696-0ae4-4c26-a171-6d4abdfa32d2_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!wAmH!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febafd696-0ae4-4c26-a171-6d4abdfa32d2_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!wAmH!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febafd696-0ae4-4c26-a171-6d4abdfa32d2_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!wAmH!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febafd696-0ae4-4c26-a171-6d4abdfa32d2_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!wAmH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febafd696-0ae4-4c26-a171-6d4abdfa32d2_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ebafd696-0ae4-4c26-a171-6d4abdfa32d2_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:158381,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222635654?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febafd696-0ae4-4c26-a171-6d4abdfa32d2_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!wAmH!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febafd696-0ae4-4c26-a171-6d4abdfa32d2_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!wAmH!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febafd696-0ae4-4c26-a171-6d4abdfa32d2_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!wAmH!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febafd696-0ae4-4c26-a171-6d4abdfa32d2_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!wAmH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Febafd696-0ae4-4c26-a171-6d4abdfa32d2_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Precise prefix-aware routing cut p90 time to first token from 92.6 seconds to 0.54 seconds on the same cluster.</figcaption></figure></div><p>Here are the p90 times to first token:</p><ul><li><p>Random: 92.6s</p></li><li><p>Load-aware only: 94.9s</p></li><li><p>Approximate prefix-aware: 31.1s</p></li><li><p>Precise prefix-aware: 0.54s</p></li></ul><p>That is about 170x sooner than random routing. Output throughput also roughly doubled, from 4,429 to 8,730 tokens/s.</p><p>Load-aware routing did no better than random. Spreading requests evenly doesn&#8217;t help when each one lands on a GPU that has to recompute its prefix.</p><p>A later llm-d rerun on a newer vLLM, against a plain Kubernetes round-robin Service, showed a smaller gap, but p90 time to first token still dropped by more than 99%.</p><p>All of this depends on the prefix cache itself working.</p><p>A small change early in the prompt, like a timestamp at the top of the system prompt, changes every block hash after it, and no router can match those blocks.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!c_4L!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b7f32cc-d8c9-48a5-9041-3c3e9644a88d_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!c_4L!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b7f32cc-d8c9-48a5-9041-3c3e9644a88d_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!c_4L!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b7f32cc-d8c9-48a5-9041-3c3e9644a88d_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!c_4L!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b7f32cc-d8c9-48a5-9041-3c3e9644a88d_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!c_4L!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b7f32cc-d8c9-48a5-9041-3c3e9644a88d_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!c_4L!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b7f32cc-d8c9-48a5-9041-3c3e9644a88d_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2b7f32cc-d8c9-48a5-9041-3c3e9644a88d_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:135461,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222635654?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b7f32cc-d8c9-48a5-9041-3c3e9644a88d_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!c_4L!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b7f32cc-d8c9-48a5-9041-3c3e9644a88d_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!c_4L!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b7f32cc-d8c9-48a5-9041-3c3e9644a88d_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!c_4L!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b7f32cc-d8c9-48a5-9041-3c3e9644a88d_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!c_4L!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b7f32cc-d8c9-48a5-9041-3c3e9644a88d_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A timestamp at the top of a prompt breaks every block after it, while the same timestamp at the end costs only the tail.</figcaption></figure></div><p>So keep the stable parts first, like the system prompt, tool definitions, and documents. Put whatever changes per request at the end.</p><p>A router can only find a cached prefix if the prompt still produces the same hashes.</p><p>We covered how prefix caching works, how it differs from KV, prompt, and semantic caching, and which prompt changes cause cache misses, in <strong><a href="https://www.dailydoseofds.com/p/kv-vs-prefix-vs-prompt-vs-semantic-caching/">a detailed article</a></strong>.</p><p><strong><a href="https://www.dailydoseofds.com/p/kv-vs-prefix-vs-prompt-vs-semantic-caching/">Read it here &#8594;</a></strong></p><p>&#128073; Over to you: Does your serving stack route requests by cache location, or does it still round-robin across replicas?</p><div><hr></div><h3><a href="https://www.dailydoseofds.com/llmops-crash-course-part-1/">&#8203;</a><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-1/">Regular ML inference vs. LLM inference</a></strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-1/">&#8203;</a></h3><p>LLM inference presents unique challenges over regular ML inference, due to which we have specialized, high-performance LLM inference engines, like vLLM, LMCache, SGLang, and TensorRT-LLM.</p><p>Let&#8217;s understand these challenges today and how we solve them!</p><h4><strong>Continuous batching</strong></h4><p>Traditional models, like CNNs, usually deal with a fixed-size image input and a fixed-length output (like a label). This makes batching easy.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!6M3E!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe71fbda8-e4bf-4686-92d3-c6e4cef4130a_627x167.png 424w, https://substackcdn.com/image/fetch/$s_!6M3E!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe71fbda8-e4bf-4686-92d3-c6e4cef4130a_627x167.png 848w, https://substackcdn.com/image/fetch/$s_!6M3E!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe71fbda8-e4bf-4686-92d3-c6e4cef4130a_627x167.png 1272w, https://substackcdn.com/image/fetch/$s_!6M3E!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe71fbda8-e4bf-4686-92d3-c6e4cef4130a_627x167.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!6M3E!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe71fbda8-e4bf-4686-92d3-c6e4cef4130a_627x167.png" width="627" height="167" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e71fbda8-e4bf-4686-92d3-c6e4cef4130a_627x167.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:167,&quot;width&quot;:627,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!6M3E!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe71fbda8-e4bf-4686-92d3-c6e4cef4130a_627x167.png 424w, https://substackcdn.com/image/fetch/$s_!6M3E!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe71fbda8-e4bf-4686-92d3-c6e4cef4130a_627x167.png 848w, https://substackcdn.com/image/fetch/$s_!6M3E!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe71fbda8-e4bf-4686-92d3-c6e4cef4130a_627x167.png 1272w, https://substackcdn.com/image/fetch/$s_!6M3E!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe71fbda8-e4bf-4686-92d3-c6e4cef4130a_627x167.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>LLMs, however, deal with variable-length inputs (the prompt) and generate variable-length outputs.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!-1w1!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34adfce4-ac6f-4447-adbe-1bcba3021497_627x167.png 424w, https://substackcdn.com/image/fetch/$s_!-1w1!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34adfce4-ac6f-4447-adbe-1bcba3021497_627x167.png 848w, https://substackcdn.com/image/fetch/$s_!-1w1!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34adfce4-ac6f-4447-adbe-1bcba3021497_627x167.png 1272w, https://substackcdn.com/image/fetch/$s_!-1w1!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34adfce4-ac6f-4447-adbe-1bcba3021497_627x167.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!-1w1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34adfce4-ac6f-4447-adbe-1bcba3021497_627x167.png" width="627" height="167" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/34adfce4-ac6f-4447-adbe-1bcba3021497_627x167.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:167,&quot;width&quot;:627,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!-1w1!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34adfce4-ac6f-4447-adbe-1bcba3021497_627x167.png 424w, https://substackcdn.com/image/fetch/$s_!-1w1!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34adfce4-ac6f-4447-adbe-1bcba3021497_627x167.png 848w, https://substackcdn.com/image/fetch/$s_!-1w1!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34adfce4-ac6f-4447-adbe-1bcba3021497_627x167.png 1272w, https://substackcdn.com/image/fetch/$s_!-1w1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34adfce4-ac6f-4447-adbe-1bcba3021497_627x167.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>So if you batch some requests, all will finish at different times, and the GPU would have to wait for the longest request to finish before it can process new requests. This leads to idle time on the GPU:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!6u-i!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f7947bf-3b07-4076-9e5c-382717d41b09_820x647.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!6u-i!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f7947bf-3b07-4076-9e5c-382717d41b09_820x647.png 424w, https://substackcdn.com/image/fetch/$s_!6u-i!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f7947bf-3b07-4076-9e5c-382717d41b09_820x647.png 848w, https://substackcdn.com/image/fetch/$s_!6u-i!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f7947bf-3b07-4076-9e5c-382717d41b09_820x647.png 1272w, https://substackcdn.com/image/fetch/$s_!6u-i!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f7947bf-3b07-4076-9e5c-382717d41b09_820x647.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!6u-i!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f7947bf-3b07-4076-9e5c-382717d41b09_820x647.png" width="820" height="647" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6f7947bf-3b07-4076-9e5c-382717d41b09_820x647.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:647,&quot;width&quot;:820,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!6u-i!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f7947bf-3b07-4076-9e5c-382717d41b09_820x647.png 424w, https://substackcdn.com/image/fetch/$s_!6u-i!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f7947bf-3b07-4076-9e5c-382717d41b09_820x647.png 848w, https://substackcdn.com/image/fetch/$s_!6u-i!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f7947bf-3b07-4076-9e5c-382717d41b09_820x647.png 1272w, https://substackcdn.com/image/fetch/$s_!6u-i!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6f7947bf-3b07-4076-9e5c-382717d41b09_820x647.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Continuous Batching solves this.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!2L6U!,w_424,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76879841-a90f-4278-97d0-a2e6cb14255c_550x428.gif 424w, https://substackcdn.com/image/fetch/$s_!2L6U!,w_848,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76879841-a90f-4278-97d0-a2e6cb14255c_550x428.gif 848w, https://substackcdn.com/image/fetch/$s_!2L6U!,w_1272,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76879841-a90f-4278-97d0-a2e6cb14255c_550x428.gif 1272w, https://substackcdn.com/image/fetch/$s_!2L6U!,w_1456,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76879841-a90f-4278-97d0-a2e6cb14255c_550x428.gif 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!2L6U!,w_1456,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76879841-a90f-4278-97d0-a2e6cb14255c_550x428.gif" width="550" height="auto" data-attrs="{&quot;src&quot;:&quot;https://substackcdn.com/image/fetch/$s_!2L6U!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76879841-a90f-4278-97d0-a2e6cb14255c_550x428.gif&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:&quot;auto&quot;,&quot;width&quot;:550,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!2L6U!,w_424,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76879841-a90f-4278-97d0-a2e6cb14255c_550x428.gif 424w, https://substackcdn.com/image/fetch/$s_!2L6U!,w_848,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76879841-a90f-4278-97d0-a2e6cb14255c_550x428.gif 848w, https://substackcdn.com/image/fetch/$s_!2L6U!,w_1272,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76879841-a90f-4278-97d0-a2e6cb14255c_550x428.gif 1272w, https://substackcdn.com/image/fetch/$s_!2L6U!,w_1456,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76879841-a90f-4278-97d0-a2e6cb14255c_550x428.gif 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>Instead of waiting for the entire batch to finish, the system monitors all sequences and swaps completed ones (<code>&lt;EOS&gt;</code> token) with new queries:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!O_gJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F260c0368-3012-4022-974e-1632a306649a_820x647.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!O_gJ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F260c0368-3012-4022-974e-1632a306649a_820x647.png 424w, https://substackcdn.com/image/fetch/$s_!O_gJ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F260c0368-3012-4022-974e-1632a306649a_820x647.png 848w, https://substackcdn.com/image/fetch/$s_!O_gJ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F260c0368-3012-4022-974e-1632a306649a_820x647.png 1272w, https://substackcdn.com/image/fetch/$s_!O_gJ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F260c0368-3012-4022-974e-1632a306649a_820x647.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!O_gJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F260c0368-3012-4022-974e-1632a306649a_820x647.png" width="820" height="647" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/260c0368-3012-4022-974e-1632a306649a_820x647.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:647,&quot;width&quot;:820,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!O_gJ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F260c0368-3012-4022-974e-1632a306649a_820x647.png 424w, https://substackcdn.com/image/fetch/$s_!O_gJ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F260c0368-3012-4022-974e-1632a306649a_820x647.png 848w, https://substackcdn.com/image/fetch/$s_!O_gJ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F260c0368-3012-4022-974e-1632a306649a_820x647.png 1272w, https://substackcdn.com/image/fetch/$s_!O_gJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F260c0368-3012-4022-974e-1632a306649a_820x647.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>This keeps the GPU pipeline full and maximizes utilization.</p><h4><strong>Prefill-decode disaggregation</strong></h4><p>LLM inference is a two-stage process with fundamentally different resource requirements.</p><ul><li><p>The prefill stage processes all the input prompt tokens at once, so this is compute-heavy.</p></li><li><p>The decode stage autoregressively generates the output, and this demands low latency.</p></li></ul><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!4H-B!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a4aaf46-75c2-4959-aff9-16d6bf071894_906x161.png 424w, https://substackcdn.com/image/fetch/$s_!4H-B!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a4aaf46-75c2-4959-aff9-16d6bf071894_906x161.png 848w, https://substackcdn.com/image/fetch/$s_!4H-B!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a4aaf46-75c2-4959-aff9-16d6bf071894_906x161.png 1272w, https://substackcdn.com/image/fetch/$s_!4H-B!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a4aaf46-75c2-4959-aff9-16d6bf071894_906x161.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!4H-B!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a4aaf46-75c2-4959-aff9-16d6bf071894_906x161.png" width="906" height="161" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5a4aaf46-75c2-4959-aff9-16d6bf071894_906x161.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:161,&quot;width&quot;:906,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!4H-B!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a4aaf46-75c2-4959-aff9-16d6bf071894_906x161.png 424w, https://substackcdn.com/image/fetch/$s_!4H-B!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a4aaf46-75c2-4959-aff9-16d6bf071894_906x161.png 848w, https://substackcdn.com/image/fetch/$s_!4H-B!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a4aaf46-75c2-4959-aff9-16d6bf071894_906x161.png 1272w, https://substackcdn.com/image/fetch/$s_!4H-B!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a4aaf46-75c2-4959-aff9-16d6bf071894_906x161.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>Running both stages on the GPU means the compute-heavy prefill requests will interfere with the latency-sensitive decode requests.</p><p>Prefill-decode disaggregation solves this by allocating a dedicated pool of GPUs for the prefill stage and another pool for the decode stage.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!DynL!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ef916a4-59b1-44c2-aa56-f4f86ff5cbde_886x514.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!DynL!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ef916a4-59b1-44c2-aa56-f4f86ff5cbde_886x514.png 424w, https://substackcdn.com/image/fetch/$s_!DynL!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ef916a4-59b1-44c2-aa56-f4f86ff5cbde_886x514.png 848w, https://substackcdn.com/image/fetch/$s_!DynL!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ef916a4-59b1-44c2-aa56-f4f86ff5cbde_886x514.png 1272w, https://substackcdn.com/image/fetch/$s_!DynL!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ef916a4-59b1-44c2-aa56-f4f86ff5cbde_886x514.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!DynL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ef916a4-59b1-44c2-aa56-f4f86ff5cbde_886x514.png" width="886" height="514" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6ef916a4-59b1-44c2-aa56-f4f86ff5cbde_886x514.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:514,&quot;width&quot;:886,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!DynL!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ef916a4-59b1-44c2-aa56-f4f86ff5cbde_886x514.png 424w, https://substackcdn.com/image/fetch/$s_!DynL!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ef916a4-59b1-44c2-aa56-f4f86ff5cbde_886x514.png 848w, https://substackcdn.com/image/fetch/$s_!DynL!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ef916a4-59b1-44c2-aa56-f4f86ff5cbde_886x514.png 1272w, https://substackcdn.com/image/fetch/$s_!DynL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ef916a4-59b1-44c2-aa56-f4f86ff5cbde_886x514.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>In contrast, a standard ML model typically has a single, unified computation phase.</p><h4><strong>GPU memory management + KV caching</strong></h4><p>Generating a new token uses the key and value vectors of all previous tokens. To avoid recomputing these vectors for all tokens over and over, we cache them (<a href="https://www.dailydoseofds.com/p/kv-caching-in-llms-explained-visually/">&#8203;</a><strong><a href="https://www.dailydoseofds.com/p/kv-caching-in-llms-explained-visually/">we covered KV caching in detail here</a></strong><a href="https://www.dailydoseofds.com/p/kv-caching-in-llms-explained-visually/">&#8203;</a>):</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!_QIc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86b13d74-204f-4165-81ed-13ef6655dd34_996x1016.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_QIc!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86b13d74-204f-4165-81ed-13ef6655dd34_996x1016.jpeg 424w, https://substackcdn.com/image/fetch/$s_!_QIc!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86b13d74-204f-4165-81ed-13ef6655dd34_996x1016.jpeg 848w, https://substackcdn.com/image/fetch/$s_!_QIc!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86b13d74-204f-4165-81ed-13ef6655dd34_996x1016.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!_QIc!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86b13d74-204f-4165-81ed-13ef6655dd34_996x1016.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!_QIc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86b13d74-204f-4165-81ed-13ef6655dd34_996x1016.jpeg" width="996" height="1016" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/86b13d74-204f-4165-81ed-13ef6655dd34_996x1016.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1016,&quot;width&quot;:996,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!_QIc!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86b13d74-204f-4165-81ed-13ef6655dd34_996x1016.jpeg 424w, https://substackcdn.com/image/fetch/$s_!_QIc!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86b13d74-204f-4165-81ed-13ef6655dd34_996x1016.jpeg 848w, https://substackcdn.com/image/fetch/$s_!_QIc!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86b13d74-204f-4165-81ed-13ef6655dd34_996x1016.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!_QIc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86b13d74-204f-4165-81ed-13ef6655dd34_996x1016.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>This KV Cache grows linearly with the total length of the conversation history.</p><p>But in many workflows, inputs like the system prompts are shared across many requests. So we can avoid recomputing them by using these KV vectors across all chats:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!emXD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F870313d3-6afb-4573-adec-ed767fd38f6d_1227x807.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!emXD!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F870313d3-6afb-4573-adec-ed767fd38f6d_1227x807.png 424w, https://substackcdn.com/image/fetch/$s_!emXD!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F870313d3-6afb-4573-adec-ed767fd38f6d_1227x807.png 848w, https://substackcdn.com/image/fetch/$s_!emXD!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F870313d3-6afb-4573-adec-ed767fd38f6d_1227x807.png 1272w, https://substackcdn.com/image/fetch/$s_!emXD!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F870313d3-6afb-4573-adec-ed767fd38f6d_1227x807.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!emXD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F870313d3-6afb-4573-adec-ed767fd38f6d_1227x807.png" width="1227" height="807" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/870313d3-6afb-4573-adec-ed767fd38f6d_1227x807.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:807,&quot;width&quot;:1227,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!emXD!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F870313d3-6afb-4573-adec-ed767fd38f6d_1227x807.png 424w, https://substackcdn.com/image/fetch/$s_!emXD!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F870313d3-6afb-4573-adec-ed767fd38f6d_1227x807.png 848w, https://substackcdn.com/image/fetch/$s_!emXD!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F870313d3-6afb-4573-adec-ed767fd38f6d_1227x807.png 1272w, https://substackcdn.com/image/fetch/$s_!emXD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F870313d3-6afb-4573-adec-ed767fd38f6d_1227x807.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>That said, KV cache takes up a significant memory since it&#8217;s stored in contiguous blocks. This wastes GPU memory and leads to memory fragmentation:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!vvO7!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d247ef-5945-4107-bb9b-03fc7ac12531_2452x676.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!vvO7!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d247ef-5945-4107-bb9b-03fc7ac12531_2452x676.png 424w, https://substackcdn.com/image/fetch/$s_!vvO7!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d247ef-5945-4107-bb9b-03fc7ac12531_2452x676.png 848w, https://substackcdn.com/image/fetch/$s_!vvO7!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d247ef-5945-4107-bb9b-03fc7ac12531_2452x676.png 1272w, https://substackcdn.com/image/fetch/$s_!vvO7!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d247ef-5945-4107-bb9b-03fc7ac12531_2452x676.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!vvO7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d247ef-5945-4107-bb9b-03fc7ac12531_2452x676.png" width="1456" height="401" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b3d247ef-5945-4107-bb9b-03fc7ac12531_2452x676.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:401,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!vvO7!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d247ef-5945-4107-bb9b-03fc7ac12531_2452x676.png 424w, https://substackcdn.com/image/fetch/$s_!vvO7!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d247ef-5945-4107-bb9b-03fc7ac12531_2452x676.png 848w, https://substackcdn.com/image/fetch/$s_!vvO7!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d247ef-5945-4107-bb9b-03fc7ac12531_2452x676.png 1272w, https://substackcdn.com/image/fetch/$s_!vvO7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d247ef-5945-4107-bb9b-03fc7ac12531_2452x676.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Paged Attention solves this problem by storing KV caching in non-contiguous blocks and then using a lookup table to track these blocks. The LLM only loads the blocks it needs, instead of loading everything at once.</p><p>We will cover Paged Attention in another issue.</p><h4><strong>Prefix-aware routing</strong></h4><p>To scale standard ML models, you can simply replicate the model across multiple servers/GPUs and use straightforward load-balancing schemes like Round Robin or routing to the least-busy server.</p><p>Since each request is independent, this works fine.</p><p>But LLMs heavily rely on caching (like the shared KV prefix discussed above), so requests are no longer independent.</p><p>If a new query comes in with a shared prefix that has already been cached on <code>Replica A</code>, but the router sends it to <code>Replica B</code> (which is less busy), <code>Replica B</code> has to recompute the entire prefix&#8217;s KV cache.</p><p>Prefix-aware routing solves this.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!UurJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41885dc8-c398-473c-b39e-a3c186099e44_1440x1490.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!UurJ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41885dc8-c398-473c-b39e-a3c186099e44_1440x1490.png 424w, https://substackcdn.com/image/fetch/$s_!UurJ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41885dc8-c398-473c-b39e-a3c186099e44_1440x1490.png 848w, https://substackcdn.com/image/fetch/$s_!UurJ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41885dc8-c398-473c-b39e-a3c186099e44_1440x1490.png 1272w, https://substackcdn.com/image/fetch/$s_!UurJ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41885dc8-c398-473c-b39e-a3c186099e44_1440x1490.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!UurJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41885dc8-c398-473c-b39e-a3c186099e44_1440x1490.png" width="1440" height="1490" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/41885dc8-c398-473c-b39e-a3c186099e44_1440x1490.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1490,&quot;width&quot;:1440,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!UurJ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41885dc8-c398-473c-b39e-a3c186099e44_1440x1490.png 424w, https://substackcdn.com/image/fetch/$s_!UurJ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41885dc8-c398-473c-b39e-a3c186099e44_1440x1490.png 848w, https://substackcdn.com/image/fetch/$s_!UurJ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41885dc8-c398-473c-b39e-a3c186099e44_1440x1490.png 1272w, https://substackcdn.com/image/fetch/$s_!UurJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41885dc8-c398-473c-b39e-a3c186099e44_1440x1490.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Different open-source frameworks each have their own implementations for prefix-aware routing.</p><p>Generally, prefix-aware routing requires the router to maintain a map or table (or use a predictive algorithm) that tracks which KV prefixes are currently cached on which GPU replicas.</p><p>When a new query arrives, the router sends the query to the replica that has the relevant prefix already cached.</p><h4><strong>Model sharding strategies</strong></h4><p>There are several strategies to scale a dense ML model (<a href="https://www.dailydoseofds.com/a-beginner-friendly-guide-to-multi-gpu-model-training/">&#8203;</a><strong><a href="https://www.dailydoseofds.com/a-beginner-friendly-guide-to-multi-gpu-model-training/">discussed here</a></strong><a href="https://www.dailydoseofds.com/a-beginner-friendly-guide-to-multi-gpu-model-training/">&#8203;</a>):</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!LCSj!,w_424,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19c29872-5d12-4426-9d6f-50db8bf6d0dd_1022x1138.gif 424w, https://substackcdn.com/image/fetch/$s_!LCSj!,w_848,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19c29872-5d12-4426-9d6f-50db8bf6d0dd_1022x1138.gif 848w, https://substackcdn.com/image/fetch/$s_!LCSj!,w_1272,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19c29872-5d12-4426-9d6f-50db8bf6d0dd_1022x1138.gif 1272w, https://substackcdn.com/image/fetch/$s_!LCSj!,w_1456,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19c29872-5d12-4426-9d6f-50db8bf6d0dd_1022x1138.gif 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!LCSj!,w_1456,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19c29872-5d12-4426-9d6f-50db8bf6d0dd_1022x1138.gif" width="556" height="auto" data-attrs="{&quot;src&quot;:&quot;https://substackcdn.com/image/fetch/$s_!LCSj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19c29872-5d12-4426-9d6f-50db8bf6d0dd_1022x1138.gif&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:&quot;auto&quot;,&quot;width&quot;:556,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!LCSj!,w_424,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19c29872-5d12-4426-9d6f-50db8bf6d0dd_1022x1138.gif 424w, https://substackcdn.com/image/fetch/$s_!LCSj!,w_848,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19c29872-5d12-4426-9d6f-50db8bf6d0dd_1022x1138.gif 848w, https://substackcdn.com/image/fetch/$s_!LCSj!,w_1272,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19c29872-5d12-4426-9d6f-50db8bf6d0dd_1022x1138.gif 1272w, https://substackcdn.com/image/fetch/$s_!LCSj!,w_1456,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F19c29872-5d12-4426-9d6f-50db8bf6d0dd_1022x1138.gif 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>LLMs, like Mixture of Experts (MoE), are complicated.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!6dGF!,w_424,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab3bce91-359f-4410-8019-3518b4535022_1116x1126.gif 424w, https://substackcdn.com/image/fetch/$s_!6dGF!,w_848,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab3bce91-359f-4410-8019-3518b4535022_1116x1126.gif 848w, https://substackcdn.com/image/fetch/$s_!6dGF!,w_1272,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab3bce91-359f-4410-8019-3518b4535022_1116x1126.gif 1272w, https://substackcdn.com/image/fetch/$s_!6dGF!,w_1456,c_limit,f_webp,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab3bce91-359f-4410-8019-3518b4535022_1116x1126.gif 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!6dGF!,w_1456,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab3bce91-359f-4410-8019-3518b4535022_1116x1126.gif" width="800" height="auto" data-attrs="{&quot;src&quot;:&quot;https://substackcdn.com/image/fetch/$s_!6dGF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab3bce91-359f-4410-8019-3518b4535022_1116x1126.gif&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:&quot;auto&quot;,&quot;width&quot;:800,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!6dGF!,w_424,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab3bce91-359f-4410-8019-3518b4535022_1116x1126.gif 424w, https://substackcdn.com/image/fetch/$s_!6dGF!,w_848,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab3bce91-359f-4410-8019-3518b4535022_1116x1126.gif 848w, https://substackcdn.com/image/fetch/$s_!6dGF!,w_1272,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab3bce91-359f-4410-8019-3518b4535022_1116x1126.gif 1272w, https://substackcdn.com/image/fetch/$s_!6dGF!,w_1456,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab3bce91-359f-4410-8019-3518b4535022_1116x1126.gif 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>MoE models use a specialized parallelism strategy called expert parallelism, which splits the experts themselves across different devices, and the attention layers are replicated across all GPUs:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!sgBg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e34f5f8-d504-42bc-b298-1853f4dc0065_1745x1596.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!sgBg!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e34f5f8-d504-42bc-b298-1853f4dc0065_1745x1596.png 424w, https://substackcdn.com/image/fetch/$s_!sgBg!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e34f5f8-d504-42bc-b298-1853f4dc0065_1745x1596.png 848w, https://substackcdn.com/image/fetch/$s_!sgBg!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e34f5f8-d504-42bc-b298-1853f4dc0065_1745x1596.png 1272w, https://substackcdn.com/image/fetch/$s_!sgBg!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e34f5f8-d504-42bc-b298-1853f4dc0065_1745x1596.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!sgBg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e34f5f8-d504-42bc-b298-1853f4dc0065_1745x1596.png" width="1456" height="1332" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4e34f5f8-d504-42bc-b298-1853f4dc0065_1745x1596.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1332,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!sgBg!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e34f5f8-d504-42bc-b298-1853f4dc0065_1745x1596.png 424w, https://substackcdn.com/image/fetch/$s_!sgBg!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e34f5f8-d504-42bc-b298-1853f4dc0065_1745x1596.png 848w, https://substackcdn.com/image/fetch/$s_!sgBg!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e34f5f8-d504-42bc-b298-1853f4dc0065_1745x1596.png 1272w, https://substackcdn.com/image/fetch/$s_!sgBg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e34f5f8-d504-42bc-b298-1853f4dc0065_1745x1596.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>So each GPU holds the full weights of only some experts, not all. This means that each GPU processes only the tokens assigned to the experts stored on that GPU.</p><p>Now, when a query arrives, the gating network in the MoE layer dynamically decides which GPU it should go to, depending on which experts are activated.</p><p>This is a complex internal routing problem that cannot be treated like a simple replicated model. You need a sophisticated inference engine to manage the dynamic flow of computation across the sharded expert pool.</p><p>If you want to learn the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:</p><ul><li><p><a href="https://www.dailydoseofds.com/llmops-crash-course-part-1/">&#8203;</a><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-1/">Read Part 1 on fundamentals of LLMOps here &#8594;</a></strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-1/">&#8203;</a></p></li><li><p><a href="https://www.dailydoseofds.com/llmops-crash-course-part-2">&#8203;</a><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-2">Read Part 2 on understanding the core building blocks of LLMs &#8594;</a></strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-2">&#8203;</a></p></li><li><p><a href="https://www.dailydoseofds.com/llmops-crash-course-part-3">&#8203;</a><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-3">Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning &#8594;</a></strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-3">&#8203;</a></p></li><li><p><a href="https://www.dailydoseofds.com/llmops-crash-course-part-4">&#8203;</a><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-4">Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications &#8594;</a></strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-4">&#8203;</a></p></li><li><p><a href="https://www.dailydoseofds.com/llmops-crash-course-part-5">&#8203;</a><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-5">Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques &#8594;</a></strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-5">&#8203;</a></p></li><li><p><a href="https://www.dailydoseofds.com/llmops-crash-course-part-6">&#8203;</a><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-6">Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more &#8594;</a></strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-6">&#8203;</a></p></li><li><p><a href="https://www.dailydoseofds.com/llmops-crash-course-part-7">&#8203;</a><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-7">Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs &#8594;</a></strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-7">&#8203;</a></p></li><li><p><a href="https://www.dailydoseofds.com/llmops-crash-course-part-8">&#8203;</a><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-8">Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications &#8594;</a></strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-8">&#8203;</a></p></li><li><p><a href="https://www.dailydoseofds.com/llmops-crash-course-part-9">&#8203;</a><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-9">Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts &#8594;</a></strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-9">&#8203;</a></p></li><li><p><a href="https://www.dailydoseofds.com/llmops-crash-course-part-10">&#8203;</a><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-10">Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps &#8594;</a></strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-10">&#8203;</a></p></li><li><p><a href="https://www.dailydoseofds.com/llmops-crash-course-part-11">&#8203;</a><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-11">Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming &#8594;</a></strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-11">&#8203;</a></p></li><li><p><a href="https://www.dailydoseofds.com/llmops-crash-course-part-12/">&#8203;</a><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-12/">Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO &#8594;</a></strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-12/">&#8203;</a></p></li><li><p><a href="https://www.dailydoseofds.com/llmops-crash-course-part-13/">&#8203;</a><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-13/">Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism &#8594;</a></strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-13/">&#8203;</a></p></li><li><p><a href="https://www.dailydoseofds.com/llmops-crash-course-part-14">&#8203;</a><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-14">Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions &#8594;</a></strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-14">&#8203;</a></p></li></ul><p>&#128073; Over to you: What are some other differences between LLM inference and regular inference?</p><p>Good day!</p>]]></content:encoded></item><item><title><![CDATA[Why DeepSeek stores its KV cache on disk]]></title><description><![CDATA[DeepSeek's engineering behind 26x faster time to first token.]]></description><link>https://blog.dailydoseofds.com/p/why-deepseek-stores-its-kv-cache</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/why-deepseek-stores-its-kv-cache</guid><dc:creator><![CDATA[Avi Chawla]]></dc:creator><pubDate>Wed, 28 Oct 2026 22:56:03 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!Ppcc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7767340f-c861-4882-a49f-2f4f3fafcf58_1200x806.gif" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><strong>In today&#8217;s newsletter:</strong></p><ul><li><p>An open-source engine for faster local LLMs on Mac</p></li><li><p>Why DeepSeek stores its KV cache on disk</p></li><li><p>What is Early Exaggeration in tSNE?</p></li></ul><div><hr></div><h3><a href="https://github.com/magnitudedev/magnitude">An open-source engine for faster local LLMs on Mac</a></h3><p>This open-source engine generates tokens 2.7x faster than Ollama on a MacBook:</p><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;b6077648-24b3-42a1-9213-441679c0c584&quot;,&quot;duration&quot;:null}"></div><p>I tested it on a base M5 MacBook Pro with Qwen3.5 4B at 4-bit and the same 9.2k-token prompt. Ollama ran its MTP build, so both had speculative decoding.</p><ul><li><p>Prefill: Ollama 464 tok/s, Magnitude 755 tok/s (1.6x)</p></li><li><p>Decode: Ollama 22.6 tok/s, Magnitude 60.0 tok/s (2.7x)</p></li><li><p>End-to-end, Magnitude finished in half the time</p></li></ul><p>The video above shows both runs side by side.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!_u5H!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10a3dbc1-05f5-4951-8327-54943ff6bdc2_2752x1536.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_u5H!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10a3dbc1-05f5-4951-8327-54943ff6bdc2_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!_u5H!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10a3dbc1-05f5-4951-8327-54943ff6bdc2_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!_u5H!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10a3dbc1-05f5-4951-8327-54943ff6bdc2_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!_u5H!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10a3dbc1-05f5-4951-8327-54943ff6bdc2_2752x1536.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!_u5H!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10a3dbc1-05f5-4951-8327-54943ff6bdc2_2752x1536.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/10a3dbc1-05f5-4951-8327-54943ff6bdc2_2752x1536.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1106964,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222493290?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10a3dbc1-05f5-4951-8327-54943ff6bdc2_2752x1536.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!_u5H!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10a3dbc1-05f5-4951-8327-54943ff6bdc2_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!_u5H!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10a3dbc1-05f5-4951-8327-54943ff6bdc2_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!_u5H!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10a3dbc1-05f5-4951-8327-54943ff6bdc2_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!_u5H!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10a3dbc1-05f5-4951-8327-54943ff6bdc2_2752x1536.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The Ollama numbers don&#8217;t mean it&#8217;s badly built. Its kernels were simply tuned in advance.</p><p>A kernel is the small GPU program behind every matrix multiply and attention step. Its speed depends on choices like how work is split across GPU cores and how data moves through memory.</p><p>The best choices change with every chip. Apple Silicon alone spans 8 to 80 GPU cores and under 70 to over 800 GB/s of bandwidth, and every model has its own layer sizes.</p><p>A prebuilt engine has to pick one set of settings for all of them, so most local setups run on a compromise.</p><p><strong><a href="https://github.com/magnitudedev/magnitude">Magnitude</a></strong> replaces that compromise with tuning on your machine:</p><p>&#8627; When you download a model, it compiles that model&#8217;s kernels locally.</p><p>&#8627; It tries different kernel configurations and keeps whichever runs fastest on your chip.</p><p>&#8627; It pairs each model with the fastest speculative decoding drafter it has found, so each pass of the model produces more than one token.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!gXTi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7c05558-3cda-4428-ba5d-48e87052c68e_2752x1536.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!gXTi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7c05558-3cda-4428-ba5d-48e87052c68e_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!gXTi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7c05558-3cda-4428-ba5d-48e87052c68e_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!gXTi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7c05558-3cda-4428-ba5d-48e87052c68e_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!gXTi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7c05558-3cda-4428-ba5d-48e87052c68e_2752x1536.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!gXTi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7c05558-3cda-4428-ba5d-48e87052c68e_2752x1536.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c7c05558-3cda-4428-ba5d-48e87052c68e_2752x1536.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1326060,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222493290?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7c05558-3cda-4428-ba5d-48e87052c68e_2752x1536.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!gXTi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7c05558-3cda-4428-ba5d-48e87052c68e_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!gXTi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7c05558-3cda-4428-ba5d-48e87052c68e_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!gXTi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7c05558-3cda-4428-ba5d-48e87052c68e_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!gXTi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7c05558-3cda-4428-ba5d-48e87052c68e_2752x1536.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>This matters most for agents. Every file and tool output has to be processed before the model writes anything, and that prefill step is where kernel efficiency shows up. Concurrent sessions also share prefix caches, so parallel agents don&#8217;t slow each other down.</p><p>Magnitude is an open-source (Apache 2.0) inference engine written in Rust. It plugs into Claude Code, Codex, OpenCode, Pi and other agents with one click.</p><p>GitHub repo: <strong><a href="https://github.com/magnitudedev/magnitude">github.com/magnitudedev/magnitude</a></strong></p><p>(don&#8217;t forget to star it &#127775;)</p><div><hr></div><h3><a href="https://www.dailydoseofds.com/p/kv-cache-engineering-for-llm-serving/">Why DeepSeek stores its KV cache on disk</a></h3><p>They just open-sourced DeepInfer, the inference stack behind cache hits that return the first token up to 26x faster.</p><p>When a request repeats a long prefix, like a system prompt or earlier chat turns, the server can reuse the KV cache it already computed instead of running prefill again.</p><p>Ideally, you&#8217;d expect that cache to sit in GPU memory, since disk is supposed to be far too slow for anything on the request path.</p><p>In practice, DeepSeek keeps it on SSDs.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Ppcc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7767340f-c861-4882-a49f-2f4f3fafcf58_1200x806.gif" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Ppcc!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7767340f-c861-4882-a49f-2f4f3fafcf58_1200x806.gif 424w, https://substackcdn.com/image/fetch/$s_!Ppcc!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7767340f-c861-4882-a49f-2f4f3fafcf58_1200x806.gif 848w, https://substackcdn.com/image/fetch/$s_!Ppcc!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7767340f-c861-4882-a49f-2f4f3fafcf58_1200x806.gif 1272w, https://substackcdn.com/image/fetch/$s_!Ppcc!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7767340f-c861-4882-a49f-2f4f3fafcf58_1200x806.gif 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Ppcc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7767340f-c861-4882-a49f-2f4f3fafcf58_1200x806.gif" width="1200" height="806" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7767340f-c861-4882-a49f-2f4f3fafcf58_1200x806.gif&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:806,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:641745,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/gif&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222493290?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7767340f-c861-4882-a49f-2f4f3fafcf58_1200x806.gif&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Ppcc!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7767340f-c861-4882-a49f-2f4f3fafcf58_1200x806.gif 424w, https://substackcdn.com/image/fetch/$s_!Ppcc!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7767340f-c861-4882-a49f-2f4f3fafcf58_1200x806.gif 848w, https://substackcdn.com/image/fetch/$s_!Ppcc!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7767340f-c861-4882-a49f-2f4f3fafcf58_1200x806.gif 1272w, https://substackcdn.com/image/fetch/$s_!Ppcc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7767340f-c861-4882-a49f-2f4f3fafcf58_1200x806.gif 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>When DeepSeek launched this in 2024, a 128K-token prompt that hit the cache got its first token in 500ms instead of 13s.</p><p>And in one day of production traffic last year, 56% of all input tokens were served straight from disk.</p><p>So disk isn&#8217;t the bottleneck here, as long as what you store is small enough and the storage is wide enough.</p><p>Let&#8217;s understand how this works.</p><h4>1. Why reuse the KV cache at all</h4><p>Before generating anything, an LLM reads the whole prompt and computes a key and a value for every token in every layer.</p><p>This is prefill, and on long prompts it&#8217;s the slow part.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!V0dc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0555954-6500-4943-af2c-59a79f268c7c_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!V0dc!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0555954-6500-4943-af2c-59a79f268c7c_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!V0dc!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0555954-6500-4943-af2c-59a79f268c7c_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!V0dc!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0555954-6500-4943-af2c-59a79f268c7c_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!V0dc!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0555954-6500-4943-af2c-59a79f268c7c_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!V0dc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0555954-6500-4943-af2c-59a79f268c7c_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f0555954-6500-4943-af2c-59a79f268c7c_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:138915,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222493290?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0555954-6500-4943-af2c-59a79f268c7c_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!V0dc!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0555954-6500-4943-af2c-59a79f268c7c_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!V0dc!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0555954-6500-4943-af2c-59a79f268c7c_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!V0dc!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0555954-6500-4943-af2c-59a79f268c7c_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!V0dc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0555954-6500-4943-af2c-59a79f268c7c_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A request with a cached prefix only runs prefill on its new tokens.</figcaption></figure></div><p>Many requests start the same way.</p><p>For instance, a chat resends every earlier turn, and an agent resends the same system prompt and tool definitions on every call.</p><p>If those KV are saved, the next request with the same prefix loads them and only computes the new tokens.</p><h4>2. Why GPU memory is the default</h4><p>Every token stores KV in every layer, so on a standard model a long prompt&#8217;s KV cache runs into many GBs.</p><p>Reading that much from disk can take longer than recomputing it on the GPU.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Z4Hx!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff89f7a62-9f6a-4e5f-aabb-51530956c422_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Z4Hx!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff89f7a62-9f6a-4e5f-aabb-51530956c422_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Z4Hx!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff89f7a62-9f6a-4e5f-aabb-51530956c422_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Z4Hx!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff89f7a62-9f6a-4e5f-aabb-51530956c422_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Z4Hx!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff89f7a62-9f6a-4e5f-aabb-51530956c422_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Z4Hx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff89f7a62-9f6a-4e5f-aabb-51530956c422_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f89f7a62-9f6a-4e5f-aabb-51530956c422_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:197966,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222493290?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff89f7a62-9f6a-4e5f-aabb-51530956c422_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Z4Hx!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff89f7a62-9f6a-4e5f-aabb-51530956c422_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Z4Hx!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff89f7a62-9f6a-4e5f-aabb-51530956c422_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Z4Hx!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff89f7a62-9f6a-4e5f-aabb-51530956c422_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Z4Hx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff89f7a62-9f6a-4e5f-aabb-51530956c422_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">GPU memory is fast but small, so old prefixes get evicted.</figcaption></figure></div><p>So engines keep it in GPU memory.</p><p>But GPU memory is small, so old prefixes get evicted fast and many repeats miss.</p><h4>3. DeepSeek made the KV cache small</h4><p>DeepSeek-V2 introduced Multi-head Latent Attention (MLA).</p><p>Instead of storing separate KV for every attention head, MLA stores one small compressed vector per token per layer. The keys and values are rebuilt from it during attention.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!p6EP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcaa918-f77a-4d4c-99d1-e4fe182899b6_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!p6EP!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcaa918-f77a-4d4c-99d1-e4fe182899b6_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!p6EP!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcaa918-f77a-4d4c-99d1-e4fe182899b6_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!p6EP!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcaa918-f77a-4d4c-99d1-e4fe182899b6_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!p6EP!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcaa918-f77a-4d4c-99d1-e4fe182899b6_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!p6EP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcaa918-f77a-4d4c-99d1-e4fe182899b6_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bfcaa918-f77a-4d4c-99d1-e4fe182899b6_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:171364,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222493290?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcaa918-f77a-4d4c-99d1-e4fe182899b6_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!p6EP!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcaa918-f77a-4d4c-99d1-e4fe182899b6_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!p6EP!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcaa918-f77a-4d4c-99d1-e4fe182899b6_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!p6EP!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcaa918-f77a-4d4c-99d1-e4fe182899b6_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!p6EP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbfcaa918-f77a-4d4c-99d1-e4fe182899b6_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">MLA stores one compressed vector per token per layer instead of per-head keys and values.</figcaption></figure></div><p>In the V2 paper, this cut the KV cache by 93.3% compared to DeepSeek&#8217;s earlier 67B model.</p><p>That made the KV cache small enough to store on low-cost disks.</p><p>Fewer bytes per cached token means fewer bytes to read on every hit, and that made loading from SSD faster than recomputing.</p><h4>4. 3FS reads it from many SSDs at once</h4><p>One SSD would still be too slow, so the cache lives on 3FS, DeepSeek&#8217;s own distributed file system.</p><p>3FS splits files into chunks and spreads them across thousands of SSDs on many storage nodes. A read pulls chunks from many drives in parallel, so their bandwidth adds up.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!SReE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50dac1af-2575-4969-bd11-ed6b1e367fb3_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!SReE!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50dac1af-2575-4969-bd11-ed6b1e367fb3_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!SReE!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50dac1af-2575-4969-bd11-ed6b1e367fb3_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!SReE!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50dac1af-2575-4969-bd11-ed6b1e367fb3_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!SReE!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50dac1af-2575-4969-bd11-ed6b1e367fb3_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!SReE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50dac1af-2575-4969-bd11-ed6b1e367fb3_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/50dac1af-2575-4969-bd11-ed6b1e367fb3_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:225966,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222493290?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50dac1af-2575-4969-bd11-ed6b1e367fb3_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!SReE!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50dac1af-2575-4969-bd11-ed6b1e367fb3_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!SReE!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50dac1af-2575-4969-bd11-ed6b1e367fb3_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!SReE!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50dac1af-2575-4969-bd11-ed6b1e367fb3_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!SReE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50dac1af-2575-4969-bd11-ed6b1e367fb3_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">3FS reads a cached prefix from many SSDs in parallel and streams it to the GPU layer by layer.</figcaption></figure></div><p>When a request arrives, the system matches its prefix against cached blocks and fetches those blocks from 3FS into host memory.</p><p>Then it copies them to the GPU layer by layer while the model is already running.</p><p>And since every prefill machine can reach every cached prefix, a request doesn&#8217;t have to go back to the machine that served it last.</p><p>Disk is also cheap, so the cache can stay for days. As per the DeepInfer docs, cached prefixes are retained for 3 days for agent traffic and 7 days for chat.</p><p>Disk caching works when the KV cache is small and the storage is wide. DeepSeek got the first from its attention design and the second from 3FS.</p><p>This also shows up in the price. On deepseek-flash today, a cache hit costs 50x less than a miss.</p><p>If you use the API, keep the stable parts of your prompt, like the system prompt, tools, and documents, at the front. Only an exact match from the very first token counts as a hit.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!HnXT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8aa3ddd0-3fb9-47b5-932b-e9d83f8780d4_2752x1536.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!HnXT!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8aa3ddd0-3fb9-47b5-932b-e9d83f8780d4_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!HnXT!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8aa3ddd0-3fb9-47b5-932b-e9d83f8780d4_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!HnXT!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8aa3ddd0-3fb9-47b5-932b-e9d83f8780d4_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!HnXT!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8aa3ddd0-3fb9-47b5-932b-e9d83f8780d4_2752x1536.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!HnXT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8aa3ddd0-3fb9-47b5-932b-e9d83f8780d4_2752x1536.png" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8aa3ddd0-3fb9-47b5-932b-e9d83f8780d4_2752x1536.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1800709,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222493290?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8aa3ddd0-3fb9-47b5-932b-e9d83f8780d4_2752x1536.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!HnXT!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8aa3ddd0-3fb9-47b5-932b-e9d83f8780d4_2752x1536.png 424w, https://substackcdn.com/image/fetch/$s_!HnXT!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8aa3ddd0-3fb9-47b5-932b-e9d83f8780d4_2752x1536.png 848w, https://substackcdn.com/image/fetch/$s_!HnXT!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8aa3ddd0-3fb9-47b5-932b-e9d83f8780d4_2752x1536.png 1272w, https://substackcdn.com/image/fetch/$s_!HnXT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8aa3ddd0-3fb9-47b5-932b-e9d83f8780d4_2752x1536.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>That said, prefix caching is only one of the ways to manage the KV cache in LLM serving.</p><p>We wrote <strong><a href="https://www.dailydoseofds.com/p/kv-cache-engineering-for-llm-serving/">an open-access KV Cache Engineering article</a></strong> covering quantization and other ways to reduce cache memory, with code examples and their trade-offs.</p><p><strong><a href="https://www.dailydoseofds.com/p/kv-cache-engineering-for-llm-serving/">Read it here &#8594;</a></strong></p><p>&#128073; Over to you: Do you order your prompts for cache hits, or does every request start from scratch?</p><div><hr></div><h3><a href="https://www.dailydoseofds.com/formulating-and-implementing-the-t-sne-algorithm-from-scratch/">&#8203;</a><strong><a href="https://www.dailydoseofds.com/formulating-and-implementing-the-t-sne-algorithm-from-scratch/">What is Early Exaggeration in tSNE?</a></strong><a href="https://www.dailydoseofds.com/formulating-and-implementing-the-t-sne-algorithm-from-scratch/">&#8203;</a></h3><p>tSNE algorithm has an <code>early_exaggeration</code> parameter:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Szg_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28ab3c83-ed1b-477e-aa31-e82c748cd1ab_943x411.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Szg_!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28ab3c83-ed1b-477e-aa31-e82c748cd1ab_943x411.png 424w, https://substackcdn.com/image/fetch/$s_!Szg_!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28ab3c83-ed1b-477e-aa31-e82c748cd1ab_943x411.png 848w, https://substackcdn.com/image/fetch/$s_!Szg_!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28ab3c83-ed1b-477e-aa31-e82c748cd1ab_943x411.png 1272w, https://substackcdn.com/image/fetch/$s_!Szg_!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28ab3c83-ed1b-477e-aa31-e82c748cd1ab_943x411.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Szg_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28ab3c83-ed1b-477e-aa31-e82c748cd1ab_943x411.png" width="943" height="411" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/28ab3c83-ed1b-477e-aa31-e82c748cd1ab_943x411.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:411,&quot;width&quot;:943,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Szg_!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28ab3c83-ed1b-477e-aa31-e82c748cd1ab_943x411.png 424w, https://substackcdn.com/image/fetch/$s_!Szg_!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28ab3c83-ed1b-477e-aa31-e82c748cd1ab_943x411.png 848w, https://substackcdn.com/image/fetch/$s_!Szg_!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28ab3c83-ed1b-477e-aa31-e82c748cd1ab_943x411.png 1272w, https://substackcdn.com/image/fetch/$s_!Szg_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28ab3c83-ed1b-477e-aa31-e82c748cd1ab_943x411.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>What is it, and how does it help?</p><p>Let&#8217;s understand this today, along with a strategy to improve it.</p><p>Let&#8217;s begin!</p><div><hr></div><h4>A brief about tSNE</h4><p>To understand early exaggeration, we need to briefly overview how tSNE works.</p><p><em>The steps discussed below have been discussed in a mathematical manner here: </em><a href="https://www.dailydoseofds.com/formulating-and-implementing-the-t-sne-algorithm-from-scratch/">&#8203;</a><em><strong><a href="https://www.dailydoseofds.com/formulating-and-implementing-the-t-sne-algorithm-from-scratch/">Formulating and Implementing the t-SNE Algorithm From Scratch</a></strong></em><a href="https://www.dailydoseofds.com/formulating-and-implementing-the-t-sne-algorithm-from-scratch/">&#8203;</a><em>.</em></p><p>The core objective of tSNE is to project a given high-dimensional data into low-dimensions (mostly 2D) for visualisation.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!BCSZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc17850c-4bf2-4557-b67c-9dfa011b6c2f_2400x848.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!BCSZ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc17850c-4bf2-4557-b67c-9dfa011b6c2f_2400x848.png 424w, https://substackcdn.com/image/fetch/$s_!BCSZ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc17850c-4bf2-4557-b67c-9dfa011b6c2f_2400x848.png 848w, https://substackcdn.com/image/fetch/$s_!BCSZ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc17850c-4bf2-4557-b67c-9dfa011b6c2f_2400x848.png 1272w, https://substackcdn.com/image/fetch/$s_!BCSZ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc17850c-4bf2-4557-b67c-9dfa011b6c2f_2400x848.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!BCSZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc17850c-4bf2-4557-b67c-9dfa011b6c2f_2400x848.png" width="1456" height="514" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bc17850c-4bf2-4557-b67c-9dfa011b6c2f_2400x848.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:514,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!BCSZ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc17850c-4bf2-4557-b67c-9dfa011b6c2f_2400x848.png 424w, https://substackcdn.com/image/fetch/$s_!BCSZ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc17850c-4bf2-4557-b67c-9dfa011b6c2f_2400x848.png 848w, https://substackcdn.com/image/fetch/$s_!BCSZ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc17850c-4bf2-4557-b67c-9dfa011b6c2f_2400x848.png 1272w, https://substackcdn.com/image/fetch/$s_!BCSZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc17850c-4bf2-4557-b67c-9dfa011b6c2f_2400x848.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>This is how we do it:</p><p>Step 1) For every point (<code>i</code>) in the given high-dimensional data, plot Euclidean distances to all other points (<code>j</code>), which gives you a Gaussian-like probability distribution. For instance, consider the dark red point marked on the left below. Other red points that are closer to it have a higher probability of being <code>i</code>&#8216;s neighbor than those that are far:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!gQVi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd493615-502f-4770-bffa-49b1ade9b011_2400x784.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!gQVi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd493615-502f-4770-bffa-49b1ade9b011_2400x784.png 424w, https://substackcdn.com/image/fetch/$s_!gQVi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd493615-502f-4770-bffa-49b1ade9b011_2400x784.png 848w, https://substackcdn.com/image/fetch/$s_!gQVi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd493615-502f-4770-bffa-49b1ade9b011_2400x784.png 1272w, https://substackcdn.com/image/fetch/$s_!gQVi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd493615-502f-4770-bffa-49b1ade9b011_2400x784.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!gQVi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd493615-502f-4770-bffa-49b1ade9b011_2400x784.png" width="1456" height="476" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/cd493615-502f-4770-bffa-49b1ade9b011_2400x784.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:476,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!gQVi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd493615-502f-4770-bffa-49b1ade9b011_2400x784.png 424w, https://substackcdn.com/image/fetch/$s_!gQVi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd493615-502f-4770-bffa-49b1ade9b011_2400x784.png 848w, https://substackcdn.com/image/fetch/$s_!gQVi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd493615-502f-4770-bffa-49b1ade9b011_2400x784.png 1272w, https://substackcdn.com/image/fetch/$s_!gQVi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd493615-502f-4770-bffa-49b1ade9b011_2400x784.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The above step assigns a neighbourhood distribution to every data point.</p><p>Step 2) Next, for every data point x&#7522; in the given data (which is in high-dimensions), randomly initialize its counterpart y&#7522; in 2D space. These will be our projections.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!TwDW!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9bbbd967-7469-44a5-a86c-f8eff7f60db4_2316x1476.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!TwDW!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9bbbd967-7469-44a5-a86c-f8eff7f60db4_2316x1476.png 424w, https://substackcdn.com/image/fetch/$s_!TwDW!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9bbbd967-7469-44a5-a86c-f8eff7f60db4_2316x1476.png 848w, https://substackcdn.com/image/fetch/$s_!TwDW!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9bbbd967-7469-44a5-a86c-f8eff7f60db4_2316x1476.png 1272w, https://substackcdn.com/image/fetch/$s_!TwDW!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9bbbd967-7469-44a5-a86c-f8eff7f60db4_2316x1476.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!TwDW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9bbbd967-7469-44a5-a86c-f8eff7f60db4_2316x1476.png" width="1456" height="928" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9bbbd967-7469-44a5-a86c-f8eff7f60db4_2316x1476.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:928,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!TwDW!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9bbbd967-7469-44a5-a86c-f8eff7f60db4_2316x1476.png 424w, https://substackcdn.com/image/fetch/$s_!TwDW!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9bbbd967-7469-44a5-a86c-f8eff7f60db4_2316x1476.png 848w, https://substackcdn.com/image/fetch/$s_!TwDW!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9bbbd967-7469-44a5-a86c-f8eff7f60db4_2316x1476.png 1272w, https://substackcdn.com/image/fetch/$s_!TwDW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9bbbd967-7469-44a5-a86c-f8eff7f60db4_2316x1476.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Step 3) Just like we created a distribution of Euclidean distances using high-dimensional data in Step 1, we create a distribution of the random data as well.</p><p>Step 4) At this point, every data point (i) has a high-dimensional distribution and a random low-dimensional distribution. The objective is to match these two distributions so that the low-dimensional representation preserves the neighborhood structure of the high-dimensional data.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!9Q7w!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2889187b-4e9d-4e70-90dc-5ab7b8c3e73d_2400x497.png 424w, https://substackcdn.com/image/fetch/$s_!9Q7w!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2889187b-4e9d-4e70-90dc-5ab7b8c3e73d_2400x497.png 848w, https://substackcdn.com/image/fetch/$s_!9Q7w!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2889187b-4e9d-4e70-90dc-5ab7b8c3e73d_2400x497.png 1272w, https://substackcdn.com/image/fetch/$s_!9Q7w!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2889187b-4e9d-4e70-90dc-5ab7b8c3e73d_2400x497.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!9Q7w!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2889187b-4e9d-4e70-90dc-5ab7b8c3e73d_2400x497.png" width="1456" height="302" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2889187b-4e9d-4e70-90dc-5ab7b8c3e73d_2400x497.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:302,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!9Q7w!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2889187b-4e9d-4e70-90dc-5ab7b8c3e73d_2400x497.png 424w, https://substackcdn.com/image/fetch/$s_!9Q7w!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2889187b-4e9d-4e70-90dc-5ab7b8c3e73d_2400x497.png 848w, https://substackcdn.com/image/fetch/$s_!9Q7w!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2889187b-4e9d-4e70-90dc-5ab7b8c3e73d_2400x497.png 1272w, https://substackcdn.com/image/fetch/$s_!9Q7w!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2889187b-4e9d-4e70-90dc-5ab7b8c3e73d_2400x497.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><ul><li><p>Using KL divergence as a loss function helps us achieve this. It measures how much information is lost when we use a distribution <code>Q</code> to approximate distribution <code>P</code>.</p></li></ul><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!iRuW!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74706d79-308a-49eb-b071-8bfb6dbffad5_1191x246.png 424w, https://substackcdn.com/image/fetch/$s_!iRuW!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74706d79-308a-49eb-b071-8bfb6dbffad5_1191x246.png 848w, https://substackcdn.com/image/fetch/$s_!iRuW!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74706d79-308a-49eb-b071-8bfb6dbffad5_1191x246.png 1272w, https://substackcdn.com/image/fetch/$s_!iRuW!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74706d79-308a-49eb-b071-8bfb6dbffad5_1191x246.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!iRuW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74706d79-308a-49eb-b071-8bfb6dbffad5_1191x246.png" width="1191" height="246" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/74706d79-308a-49eb-b071-8bfb6dbffad5_1191x246.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:246,&quot;width&quot;:1191,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!iRuW!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74706d79-308a-49eb-b071-8bfb6dbffad5_1191x246.png 424w, https://substackcdn.com/image/fetch/$s_!iRuW!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74706d79-308a-49eb-b071-8bfb6dbffad5_1191x246.png 848w, https://substackcdn.com/image/fetch/$s_!iRuW!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74706d79-308a-49eb-b071-8bfb6dbffad5_1191x246.png 1272w, https://substackcdn.com/image/fetch/$s_!iRuW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74706d79-308a-49eb-b071-8bfb6dbffad5_1191x246.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><ul><li><p>Ideally, we want to have the minimum loss value (which is zero), and this will be achieved when <code>P=Q</code>.</p></li></ul><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!uvoJ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F745754db-473a-4253-bcca-e9590849c2e6_1615x393.png 424w, https://substackcdn.com/image/fetch/$s_!uvoJ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F745754db-473a-4253-bcca-e9590849c2e6_1615x393.png 848w, https://substackcdn.com/image/fetch/$s_!uvoJ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F745754db-473a-4253-bcca-e9590849c2e6_1615x393.png 1272w, https://substackcdn.com/image/fetch/$s_!uvoJ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F745754db-473a-4253-bcca-e9590849c2e6_1615x393.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!uvoJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F745754db-473a-4253-bcca-e9590849c2e6_1615x393.png" width="1456" height="354" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/745754db-473a-4253-bcca-e9590849c2e6_1615x393.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:354,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!uvoJ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F745754db-473a-4253-bcca-e9590849c2e6_1615x393.png 424w, https://substackcdn.com/image/fetch/$s_!uvoJ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F745754db-473a-4253-bcca-e9590849c2e6_1615x393.png 848w, https://substackcdn.com/image/fetch/$s_!uvoJ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F745754db-473a-4253-bcca-e9590849c2e6_1615x393.png 1272w, https://substackcdn.com/image/fetch/$s_!uvoJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F745754db-473a-4253-bcca-e9590849c2e6_1615x393.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>The model can be trained using gradient descent, and eventually, it generates projections that preserve the local neighborhood structure of the original high-dimensional data.</p><p>But having early exaggeration makes it even better.</p><div><hr></div><h4><strong>Early exaggeration</strong></h4><p>The step exaggerates (inflates) the values of all high-dimensional probabilities (<code>P</code>) during the initial stages of the optimization by multiplying a constant factor greater than 1:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!CcgK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!CcgK!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png 424w, https://substackcdn.com/image/fetch/$s_!CcgK!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png 848w, https://substackcdn.com/image/fetch/$s_!CcgK!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png 1272w, https://substackcdn.com/image/fetch/$s_!CcgK!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!CcgK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png" width="1456" height="440" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:440,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!CcgK!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png 424w, https://substackcdn.com/image/fetch/$s_!CcgK!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png 848w, https://substackcdn.com/image/fetch/$s_!CcgK!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png 1272w, https://substackcdn.com/image/fetch/$s_!CcgK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Here&#8217;s why this is useful:</p><p>Recall that the KL divergence is minimized when the distributions P and Q are identical.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!uvoJ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F745754db-473a-4253-bcca-e9590849c2e6_1615x393.png 424w, https://substackcdn.com/image/fetch/$s_!uvoJ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F745754db-473a-4253-bcca-e9590849c2e6_1615x393.png 848w, https://substackcdn.com/image/fetch/$s_!uvoJ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F745754db-473a-4253-bcca-e9590849c2e6_1615x393.png 1272w, https://substackcdn.com/image/fetch/$s_!uvoJ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F745754db-473a-4253-bcca-e9590849c2e6_1615x393.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!uvoJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F745754db-473a-4253-bcca-e9590849c2e6_1615x393.png" width="1456" height="354" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/745754db-473a-4253-bcca-e9590849c2e6_1615x393.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:354,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!uvoJ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F745754db-473a-4253-bcca-e9590849c2e6_1615x393.png 424w, https://substackcdn.com/image/fetch/$s_!uvoJ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F745754db-473a-4253-bcca-e9590849c2e6_1615x393.png 848w, https://substackcdn.com/image/fetch/$s_!uvoJ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F745754db-473a-4253-bcca-e9590849c2e6_1615x393.png 1272w, https://substackcdn.com/image/fetch/$s_!uvoJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F745754db-473a-4253-bcca-e9590849c2e6_1615x393.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>By increasing <code>P</code>, the optimization process learns larger values of <code>Q</code> since it must match them to get minimum loss.</p><p>As a result, the low-dimensional clusters created will be very tight, and all individual clusters will be widely separated from each other.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xZVi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4edc59c-cabb-44e9-b58a-dc8bc4c7d27d_1920x900.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xZVi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4edc59c-cabb-44e9-b58a-dc8bc4c7d27d_1920x900.png 424w, https://substackcdn.com/image/fetch/$s_!xZVi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4edc59c-cabb-44e9-b58a-dc8bc4c7d27d_1920x900.png 848w, https://substackcdn.com/image/fetch/$s_!xZVi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4edc59c-cabb-44e9-b58a-dc8bc4c7d27d_1920x900.png 1272w, https://substackcdn.com/image/fetch/$s_!xZVi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4edc59c-cabb-44e9-b58a-dc8bc4c7d27d_1920x900.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!xZVi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4edc59c-cabb-44e9-b58a-dc8bc4c7d27d_1920x900.png" width="1456" height="683" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f4edc59c-cabb-44e9-b58a-dc8bc4c7d27d_1920x900.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:683,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!xZVi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4edc59c-cabb-44e9-b58a-dc8bc4c7d27d_1920x900.png 424w, https://substackcdn.com/image/fetch/$s_!xZVi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4edc59c-cabb-44e9-b58a-dc8bc4c7d27d_1920x900.png 848w, https://substackcdn.com/image/fetch/$s_!xZVi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4edc59c-cabb-44e9-b58a-dc8bc4c7d27d_1920x900.png 1272w, https://substackcdn.com/image/fetch/$s_!xZVi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4edc59c-cabb-44e9-b58a-dc8bc4c7d27d_1920x900.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>This is why we use early exaggeration.</p><div><hr></div><h4>Improving early exaggeration</h4><p>Early exaggeration multiplies all high-dimensional probabilities (<code>P</code>) by a constant factor greater than 1.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!CcgK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!CcgK!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png 424w, https://substackcdn.com/image/fetch/$s_!CcgK!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png 848w, https://substackcdn.com/image/fetch/$s_!CcgK!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png 1272w, https://substackcdn.com/image/fetch/$s_!CcgK!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!CcgK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png" width="1456" height="440" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:440,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!CcgK!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png 424w, https://substackcdn.com/image/fetch/$s_!CcgK!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png 848w, https://substackcdn.com/image/fetch/$s_!CcgK!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png 1272w, https://substackcdn.com/image/fetch/$s_!CcgK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a42e2de-49e4-48eb-a57c-4a32d05349bc_2400x726.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>While this inflates the probability of data points that are close, the downside is that it also inflates the probability of data points that are far (likely in different clusters).</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!2xb3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91309f0e-a0c2-4b89-9a76-2d9df05efff1_2097x1109.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!2xb3!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91309f0e-a0c2-4b89-9a76-2d9df05efff1_2097x1109.png 424w, https://substackcdn.com/image/fetch/$s_!2xb3!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91309f0e-a0c2-4b89-9a76-2d9df05efff1_2097x1109.png 848w, https://substackcdn.com/image/fetch/$s_!2xb3!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91309f0e-a0c2-4b89-9a76-2d9df05efff1_2097x1109.png 1272w, https://substackcdn.com/image/fetch/$s_!2xb3!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91309f0e-a0c2-4b89-9a76-2d9df05efff1_2097x1109.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!2xb3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91309f0e-a0c2-4b89-9a76-2d9df05efff1_2097x1109.png" width="1456" height="770" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/91309f0e-a0c2-4b89-9a76-2d9df05efff1_2097x1109.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:770,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!2xb3!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91309f0e-a0c2-4b89-9a76-2d9df05efff1_2097x1109.png 424w, https://substackcdn.com/image/fetch/$s_!2xb3!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91309f0e-a0c2-4b89-9a76-2d9df05efff1_2097x1109.png 848w, https://substackcdn.com/image/fetch/$s_!2xb3!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91309f0e-a0c2-4b89-9a76-2d9df05efff1_2097x1109.png 1272w, https://substackcdn.com/image/fetch/$s_!2xb3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91309f0e-a0c2-4b89-9a76-2d9df05efff1_2097x1109.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>A better strategy is to set a threshold (&#955;) for inflating values:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!aT5A!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87f94e99-b81e-42ca-926c-52ab7eabad73_1670x841.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!aT5A!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87f94e99-b81e-42ca-926c-52ab7eabad73_1670x841.png 424w, https://substackcdn.com/image/fetch/$s_!aT5A!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87f94e99-b81e-42ca-926c-52ab7eabad73_1670x841.png 848w, https://substackcdn.com/image/fetch/$s_!aT5A!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87f94e99-b81e-42ca-926c-52ab7eabad73_1670x841.png 1272w, https://substackcdn.com/image/fetch/$s_!aT5A!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87f94e99-b81e-42ca-926c-52ab7eabad73_1670x841.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!aT5A!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87f94e99-b81e-42ca-926c-52ab7eabad73_1670x841.png" width="1456" height="733" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/87f94e99-b81e-42ca-926c-52ab7eabad73_1670x841.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:733,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!aT5A!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87f94e99-b81e-42ca-926c-52ab7eabad73_1670x841.png 424w, https://substackcdn.com/image/fetch/$s_!aT5A!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87f94e99-b81e-42ca-926c-52ab7eabad73_1670x841.png 848w, https://substackcdn.com/image/fetch/$s_!aT5A!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87f94e99-b81e-42ca-926c-52ab7eabad73_1670x841.png 1272w, https://substackcdn.com/image/fetch/$s_!aT5A!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87f94e99-b81e-42ca-926c-52ab7eabad73_1670x841.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><ul><li><p>Only exaggerate the joint probabilities if they exceed a certain threshold.</p></li><li><p>Otherwise, understate (or reduce) them.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!2QTi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F324edf28-4d81-45cd-8f7a-a005522f63dc_2400x622.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!2QTi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F324edf28-4d81-45cd-8f7a-a005522f63dc_2400x622.png 424w, https://substackcdn.com/image/fetch/$s_!2QTi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F324edf28-4d81-45cd-8f7a-a005522f63dc_2400x622.png 848w, https://substackcdn.com/image/fetch/$s_!2QTi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F324edf28-4d81-45cd-8f7a-a005522f63dc_2400x622.png 1272w, https://substackcdn.com/image/fetch/$s_!2QTi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F324edf28-4d81-45cd-8f7a-a005522f63dc_2400x622.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!2QTi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F324edf28-4d81-45cd-8f7a-a005522f63dc_2400x622.png" width="1456" height="377" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/324edf28-4d81-45cd-8f7a-a005522f63dc_2400x622.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:377,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!2QTi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F324edf28-4d81-45cd-8f7a-a005522f63dc_2400x622.png 424w, https://substackcdn.com/image/fetch/$s_!2QTi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F324edf28-4d81-45cd-8f7a-a005522f63dc_2400x622.png 848w, https://substackcdn.com/image/fetch/$s_!2QTi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F324edf28-4d81-45cd-8f7a-a005522f63dc_2400x622.png 1272w, https://substackcdn.com/image/fetch/$s_!2QTi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F324edf28-4d81-45cd-8f7a-a005522f63dc_2400x622.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>This makes intuitive sense as well.</p><p>This is how early exaggeration works and why it is so useful in tSNE.</p><p>We went into much more detail about tSNE in a beginner-friendly manner in the full article: <a href="https://www.dailydoseofds.com/formulating-and-implementing-the-t-sne-algorithm-from-scratch/">&#8203;</a><strong><a href="https://www.dailydoseofds.com/formulating-and-implementing-the-t-sne-algorithm-from-scratch/">Formulating and Implementing t-SNE From Scratch &#8594;</a></strong><a href="https://www.dailydoseofds.com/formulating-and-implementing-the-t-sne-algorithm-from-scratch/">&#8203;</a></p><p>This article covers:</p><ul><li><p>The limitations of PCA.</p></li><li><p>The entire end-to-end working of the SNE algorithm (an earlier version of t-SNE).</p></li><li><p>What does SNE try to optimize?</p></li><li><p>What are local and global structures?</p></li><li><p>Understanding the idea of <code>perplexity</code> visually.</p></li><li><p>The limitations of the SNE algorithm.</p></li><li><p>How is t-SNE different from SNE?</p></li><li><p>Advanced optimization methods used in t-SNE.</p></li><li><p>Implementing the entire t-SNE algorithm by only using NumPy.</p></li><li><p>A practice Jupyter notebook for you to implement t-SNE yourself.</p></li><li><p>Understanding some potential gaps in t-SNE and how to possibly improve it.</p></li><li><p>PCA vs. t-SNE comparison.</p></li><li><p>Limitations of t-SNE.</p></li><li><p>Why is <code>perplexity</code> the most important hyperparameter of t-SNE?</p></li><li><p>Important takeaways for using t-SNE.</p></li></ul><p>Similar to this, we also formulated the PCA algorithm from scratch here: <a href="https://www.dailydoseofds.com/formulating-the-principal-component-analysis-algorithm-from-scratch/">&#8203;</a><strong><a href="https://www.dailydoseofds.com/formulating-the-principal-component-analysis-algorithm-from-scratch/">Formulating the PCA Algorithm From Scratch &#8594;</a></strong><a href="https://www.dailydoseofds.com/formulating-the-principal-component-analysis-algorithm-from-scratch/">&#8203;</a></p><p>&#128073; Over to you: Can you identify a bottleneck in the t-SNE algorithm?</p><p>Good day!</p>]]></content:encoded></item><item><title><![CDATA[Which OCR Solution Works Best for Your Documents?]]></title><description><![CDATA[...DeepSeek-OCR vs. PaddleOCR-VL vs. Datalab Surya, tested on the same 150 pages.]]></description><link>https://blog.dailydoseofds.com/p/which-ocr-solution-works-best-for</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/which-ocr-solution-works-best-for</guid><dc:creator><![CDATA[Avi Chawla]]></dc:creator><pubDate>Tue, 27 Oct 2026 23:58:18 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!EVcr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0393bbae-fa58-42fc-b96d-2f145945f1d0_1080x1080.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<h3><a href="https://github.com/trymirai/uzu">This inference engine runs LLMs 4x faster than llama.cpp:</a></h3><p>We ran Qwen3.5 9B on a base M5 MacBook Pro with three local inference engines, one at a time.</p><ul><li><p>llama.cpp reached 22.0 tokens/s</p></li><li><p>MLX reached 25.1 tokens/s</p></li><li><p><strong><a href="https://github.com/trymirai/uzu">uzu</a></strong> reached 92.1 tokens/s</p></li></ul><p>The video shows all three runs side by side.</p><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;d8015117-9d3c-4a7e-9f02-c537ecf72238&quot;,&quot;duration&quot;:null}"></div><p>The first two numbers don&#8217;t mean llama.cpp or MLX are badly built. Both sit right under a hard limit of local decoding.</p><p>Every new token needs one full pass through the model, and that pass reads all of the model&#8217;s weights from memory.</p><p>A base M5 reads memory at about 153 GB/s, and a 4-bit 9B model weighs about 5.2 GB.</p><p>So at one token per pass, the ceiling is roughly 153 &#247; 5.2, or about 29 tokens/s, however fast the GPU computes.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!UZyT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5ee20dc-670f-4db4-8de2-1e01542e917d_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!UZyT!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5ee20dc-670f-4db4-8de2-1e01542e917d_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!UZyT!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5ee20dc-670f-4db4-8de2-1e01542e917d_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!UZyT!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5ee20dc-670f-4db4-8de2-1e01542e917d_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!UZyT!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5ee20dc-670f-4db4-8de2-1e01542e917d_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!UZyT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5ee20dc-670f-4db4-8de2-1e01542e917d_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e5ee20dc-670f-4db4-8de2-1e01542e917d_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:167227,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222396733?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5ee20dc-670f-4db4-8de2-1e01542e917d_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!UZyT!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5ee20dc-670f-4db4-8de2-1e01542e917d_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!UZyT!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5ee20dc-670f-4db4-8de2-1e01542e917d_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!UZyT!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5ee20dc-670f-4db4-8de2-1e01542e917d_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!UZyT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5ee20dc-670f-4db4-8de2-1e01542e917d_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><em>Ordinary decoding stops near 29 tokens/s, while uzu gets about 7.5 tokens out of every pass.</em></figcaption></figure></div><p><strong><a href="https://github.com/trymirai/uzu">uzu</a></strong> gets past that ceiling by producing several tokens per pass instead of one.</p><p>&#8627; A small drafter (DFlash) proposes candidates for the next 16 positions at once.</p><p>&#8627; A tiny adapter (Weaver, 56.7M params) picks among them left to right so the guesses form coherent text, and keeps a few alternative branches as a tree.</p><p>&#8627; The 9B model checks the whole tree in a single pass and keeps the longest path it agrees with.</p><p>The main model still decides every token that ends up in the output.</p><p>In our run, each pass of the 9B model produced 7.5 tokens on average. Ordinary decoding produces exactly 1.</p><p>The weights and kernels are built around this path too. uzu ships its own 4-bit checkpoints, and on M5 chips it runs the verification step on the GPU&#8217;s int8 matrix path.</p><p>uzu is an open-source inference engine by Mirai Labs, written in Rust with Metal kernels. It has Swift, Python, TypeScript, and Rust bindings, so the same engine runs inside a Mac or iOS app.</p><p>We covered why local inference is memory-bound and how uzu&#8217;s speculative decoding gets past it in <strong><a href="https://blog.dailydoseofds.com/p/karpathys-trick-for-faster-local">a full engineering deep dive</a></strong>, with a benchmark client you can point at your own prompts.</p><p><strong><a href="https://blog.dailydoseofds.com/p/karpathys-trick-for-faster-local">Read it here &#8594;</a></strong></p><p><strong><a href="https://github.com/trymirai/uzu">Find the GitHub repo here &#8594;</a></strong></p><p>(don&#8217;t forget to star it &#127775;)</p><div><hr></div><h3><a href="https://github.com/datalab-to/omni_parse_bench">Which OCR solution works best for your documents?</a></h3><p>The OCR model with the best average score can still be the wrong pick for your documents.</p><p>We ran DeepSeek-OCR, PaddleOCR-VL and Datalab Surya on the same 150 pages, from handwritten notes and rotated scans to dense tables and a filled tax form.</p><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;502cd25a-5ce4-4bdb-ab89-3bc553a0e8a6&quot;,&quot;duration&quot;:null}"></div><p><em>The video shows every page, what each parser returned, and why each test passed or failed.</em></p><p>Most OCR comparisons end with one number per parser.</p><p>That number tells you who&#8217;s ahead on average. It doesn&#8217;t tell you where a parser breaks on documents like yours.</p><p>And nobody parses an average document. An insurance team parses scanned claim forms, a research team parses two-column papers full of equations, and a finance team parses dense tables.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!uidj!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9a6c6f4-03eb-4720-a531-2ef245e0b7cc_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!uidj!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9a6c6f4-03eb-4720-a531-2ef245e0b7cc_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!uidj!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9a6c6f4-03eb-4720-a531-2ef245e0b7cc_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!uidj!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9a6c6f4-03eb-4720-a531-2ef245e0b7cc_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!uidj!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9a6c6f4-03eb-4720-a531-2ef245e0b7cc_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!uidj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9a6c6f4-03eb-4720-a531-2ef245e0b7cc_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e9a6c6f4-03eb-4720-a531-2ef245e0b7cc_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:155201,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222396733?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9a6c6f4-03eb-4720-a531-2ef245e0b7cc_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!uidj!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9a6c6f4-03eb-4720-a531-2ef245e0b7cc_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!uidj!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9a6c6f4-03eb-4720-a531-2ef245e0b7cc_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!uidj!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9a6c6f4-03eb-4720-a531-2ef245e0b7cc_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!uidj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9a6c6f4-03eb-4720-a531-2ef245e0b7cc_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><em>The parser that leads the average can still fail on the one page type you care about.</em></figcaption></figure></div><p>So instead of one score, we checked every page with simple yes/no tests:</p><ul><li><p>Is this line present in the output?</p></li><li><p>Is this cell under the right headings?</p></li><li><p>Does A come before B?</p></li></ul><p>Each test passes or fails, and every failure comes with a reason, like &#8220;text missing&#8221; or &#8220;marked as table, not heading&#8221;.</p><p>The scoring runs as open code. No LLM judges the output.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!pjpA!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F579a5cdc-cd09-4032-83cd-d609da8b2ea9_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!pjpA!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F579a5cdc-cd09-4032-83cd-d609da8b2ea9_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!pjpA!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F579a5cdc-cd09-4032-83cd-d609da8b2ea9_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!pjpA!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F579a5cdc-cd09-4032-83cd-d609da8b2ea9_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!pjpA!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F579a5cdc-cd09-4032-83cd-d609da8b2ea9_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!pjpA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F579a5cdc-cd09-4032-83cd-d609da8b2ea9_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/579a5cdc-cd09-4032-83cd-d609da8b2ea9_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:132705,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222396733?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F579a5cdc-cd09-4032-83cd-d609da8b2ea9_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!pjpA!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F579a5cdc-cd09-4032-83cd-d609da8b2ea9_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!pjpA!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F579a5cdc-cd09-4032-83cd-d609da8b2ea9_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!pjpA!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F579a5cdc-cd09-4032-83cd-d609da8b2ea9_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!pjpA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F579a5cdc-cd09-4032-83cd-d609da8b2ea9_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><em>Each check is a small yes/no question, and the score is the share of checks a parser passes.</em></figcaption></figure></div><h4>What we found</h4><p>Here&#8217;s the overall scoreboard from our 150-page run:</p><p>&#8594; Datalab: 88.2</p><p>&#8594; DeepSeek-OCR: 77.5</p><p>&#8594; PaddleOCR-VL: 73.4</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!EVcr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0393bbae-fa58-42fc-b96d-2f145945f1d0_1080x1080.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!EVcr!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0393bbae-fa58-42fc-b96d-2f145945f1d0_1080x1080.png 424w, https://substackcdn.com/image/fetch/$s_!EVcr!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0393bbae-fa58-42fc-b96d-2f145945f1d0_1080x1080.png 848w, https://substackcdn.com/image/fetch/$s_!EVcr!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0393bbae-fa58-42fc-b96d-2f145945f1d0_1080x1080.png 1272w, https://substackcdn.com/image/fetch/$s_!EVcr!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0393bbae-fa58-42fc-b96d-2f145945f1d0_1080x1080.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!EVcr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0393bbae-fa58-42fc-b96d-2f145945f1d0_1080x1080.png" width="1080" height="1080" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0393bbae-fa58-42fc-b96d-2f145945f1d0_1080x1080.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1080,&quot;width&quot;:1080,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:136516,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222396733?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0393bbae-fa58-42fc-b96d-2f145945f1d0_1080x1080.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!EVcr!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0393bbae-fa58-42fc-b96d-2f145945f1d0_1080x1080.png 424w, https://substackcdn.com/image/fetch/$s_!EVcr!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0393bbae-fa58-42fc-b96d-2f145945f1d0_1080x1080.png 848w, https://substackcdn.com/image/fetch/$s_!EVcr!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0393bbae-fa58-42fc-b96d-2f145945f1d0_1080x1080.png 1272w, https://substackcdn.com/image/fetch/$s_!EVcr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0393bbae-fa58-42fc-b96d-2f145945f1d0_1080x1080.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><em>Datalab led the overall score in our run, but DeepSeek-OCR led on tables.</em></figcaption></figure></div><p>If we had stopped at the overall column, the answer would look obvious. It isn&#8217;t.</p><p>DeepSeek-OCR was the best on tables at 87.2, ahead of Datalab at 78.9 and PaddleOCR-VL at 78.0.</p><p>So a team whose documents are mostly dense tables would pick a different parser than the average suggests.</p><p>On handwritten notes, the gap went the other way. One parser (Datalab) passed 5 of 6 tests, while the other two passed 0.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!c_wY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda10076d-57db-4c2c-aea7-8895a756b04a_1080x1080.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!c_wY!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda10076d-57db-4c2c-aea7-8895a756b04a_1080x1080.png 424w, https://substackcdn.com/image/fetch/$s_!c_wY!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda10076d-57db-4c2c-aea7-8895a756b04a_1080x1080.png 848w, https://substackcdn.com/image/fetch/$s_!c_wY!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda10076d-57db-4c2c-aea7-8895a756b04a_1080x1080.png 1272w, https://substackcdn.com/image/fetch/$s_!c_wY!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda10076d-57db-4c2c-aea7-8895a756b04a_1080x1080.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!c_wY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda10076d-57db-4c2c-aea7-8895a756b04a_1080x1080.png" width="1080" height="1080" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/da10076d-57db-4c2c-aea7-8895a756b04a_1080x1080.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1080,&quot;width&quot;:1080,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:213259,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222396733?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda10076d-57db-4c2c-aea7-8895a756b04a_1080x1080.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!c_wY!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda10076d-57db-4c2c-aea7-8895a756b04a_1080x1080.png 424w, https://substackcdn.com/image/fetch/$s_!c_wY!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda10076d-57db-4c2c-aea7-8895a756b04a_1080x1080.png 848w, https://substackcdn.com/image/fetch/$s_!c_wY!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda10076d-57db-4c2c-aea7-8895a756b04a_1080x1080.png 1272w, https://substackcdn.com/image/fetch/$s_!c_wY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda10076d-57db-4c2c-aea7-8895a756b04a_1080x1080.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><em>On a handwritten page, Datalab passed 5 of 6 tests while PaddleOCR-VL misread most of the lines.</em></figcaption></figure></div><p>The filled 1040 tax form showed the starkest failure.</p><p>PaddleOCR-VL returned only the form&#8217;s title, so every test looking for a filled-in value failed with &#8220;text missing&#8221;.</p><p>On the same page, Datalab passed 8 of 8 tests and DeepSeek-OCR passed 6 of 8.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!-ns2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d5df3d4-31d3-4077-828b-751c4e179c85_1080x1080.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!-ns2!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d5df3d4-31d3-4077-828b-751c4e179c85_1080x1080.png 424w, https://substackcdn.com/image/fetch/$s_!-ns2!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d5df3d4-31d3-4077-828b-751c4e179c85_1080x1080.png 848w, https://substackcdn.com/image/fetch/$s_!-ns2!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d5df3d4-31d3-4077-828b-751c4e179c85_1080x1080.png 1272w, https://substackcdn.com/image/fetch/$s_!-ns2!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d5df3d4-31d3-4077-828b-751c4e179c85_1080x1080.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!-ns2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d5df3d4-31d3-4077-828b-751c4e179c85_1080x1080.png" width="1080" height="1080" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1d5df3d4-31d3-4077-828b-751c4e179c85_1080x1080.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1080,&quot;width&quot;:1080,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:127781,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222396733?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d5df3d4-31d3-4077-828b-751c4e179c85_1080x1080.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!-ns2!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d5df3d4-31d3-4077-828b-751c4e179c85_1080x1080.png 424w, https://substackcdn.com/image/fetch/$s_!-ns2!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d5df3d4-31d3-4077-828b-751c4e179c85_1080x1080.png 848w, https://substackcdn.com/image/fetch/$s_!-ns2!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d5df3d4-31d3-4077-828b-751c4e179c85_1080x1080.png 1272w, https://substackcdn.com/image/fetch/$s_!-ns2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d5df3d4-31d3-4077-828b-751c4e179c85_1080x1080.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><em>On the filled 1040 form, PaddleOCR-VL returned only the title and passed 0 of 8 tests.</em></figcaption></figure></div><p>A parse that returns only a title is a different risk from one with a few wrong cells. A single average folds both into the same number.</p><p>No single parser won on every type of page. The right pick depends on your documents, not on who has the best average.</p><h4>The benchmark behind it</h4><p>The tool we used is <strong><a href="https://github.com/datalab-to/omni_parse_bench">OmniParse Bench</a></strong>, and it&#8217;s 100% open-source.</p><p>It has 16,288 tests across 2,937 pages in 92 languages, drawn from 2,343 source documents.</p><p>It builds on what olmOCR-bench showed, that pass/fail unit tests are a simple proxy for parsing quality that anyone can check. Datalab&#8217;s view is that olmOCR-bench has saturated, and what now separates the top systems there is mostly output conventions and overfitting rather than reading.</p><p>Every test belongs to one of three families: text, tables, and layout. A family&#8217;s score is the share of its tests that pass, and the headline is the mean of the three.</p><p>A page where the parser call fails counts as 0, so a parser can&#8217;t look better by quietly dropping the hard pages.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ozwX!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2663f599-da1e-4eed-9d0c-d6d936c7c290_2000x1305.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ozwX!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2663f599-da1e-4eed-9d0c-d6d936c7c290_2000x1305.png 424w, https://substackcdn.com/image/fetch/$s_!ozwX!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2663f599-da1e-4eed-9d0c-d6d936c7c290_2000x1305.png 848w, https://substackcdn.com/image/fetch/$s_!ozwX!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2663f599-da1e-4eed-9d0c-d6d936c7c290_2000x1305.png 1272w, https://substackcdn.com/image/fetch/$s_!ozwX!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2663f599-da1e-4eed-9d0c-d6d936c7c290_2000x1305.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ozwX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2663f599-da1e-4eed-9d0c-d6d936c7c290_2000x1305.png" width="1456" height="950" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2663f599-da1e-4eed-9d0c-d6d936c7c290_2000x1305.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:950,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:34811,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222396733?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2663f599-da1e-4eed-9d0c-d6d936c7c290_2000x1305.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ozwX!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2663f599-da1e-4eed-9d0c-d6d936c7c290_2000x1305.png 424w, https://substackcdn.com/image/fetch/$s_!ozwX!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2663f599-da1e-4eed-9d0c-d6d936c7c290_2000x1305.png 848w, https://substackcdn.com/image/fetch/$s_!ozwX!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2663f599-da1e-4eed-9d0c-d6d936c7c290_2000x1305.png 1272w, https://substackcdn.com/image/fetch/$s_!ozwX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2663f599-da1e-4eed-9d0c-d6d936c7c290_2000x1305.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><em>The headline averages three family scores, and tags only decide which tests you look at.</em></figcaption></figure></div><p>The tags are where it gets useful.</p><p>Each test is tagged by what makes its content hard, like handwriting, rotation, tiny text, degraded scans, forms, tables, math and multi-column layouts, along with its script and language.</p><p>Tags never change whether a test passes. They only select tests, so you can score a parser on handwriting alone, or on French tables alone.</p><p>Every score also comes with the number of tests and documents behind it, so a thin slice tells you it&#8217;s thin.</p><p>Each test also records how its answer was established, such as the publisher&#8217;s own markup, a form the authors filled in themselves, or agreement across a panel of independent models.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!EAWu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9fbe6b8b-1da7-4dba-94fe-ae313fd218fc_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!EAWu!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9fbe6b8b-1da7-4dba-94fe-ae313fd218fc_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!EAWu!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9fbe6b8b-1da7-4dba-94fe-ae313fd218fc_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!EAWu!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9fbe6b8b-1da7-4dba-94fe-ae313fd218fc_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!EAWu!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9fbe6b8b-1da7-4dba-94fe-ae313fd218fc_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!EAWu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9fbe6b8b-1da7-4dba-94fe-ae313fd218fc_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9fbe6b8b-1da7-4dba-94fe-ae313fd218fc_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:122495,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222396733?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9fbe6b8b-1da7-4dba-94fe-ae313fd218fc_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!EAWu!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9fbe6b8b-1da7-4dba-94fe-ae313fd218fc_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!EAWu!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9fbe6b8b-1da7-4dba-94fe-ae313fd218fc_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!EAWu!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9fbe6b8b-1da7-4dba-94fe-ae313fd218fc_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!EAWu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9fbe6b8b-1da7-4dba-94fe-ae313fd218fc_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><em>Tags turn one benchmark into many small ones, so you can score parsers only on pages that look like yours.</em></figcaption></figure></div><p>One thing to note is that Datalab built this benchmark, and Surya is Datalab&#8217;s own model.</p><p>That&#8217;s exactly why the open scorer matters. You don&#8217;t have to trust anyone&#8217;s leaderboard, since you can rerun every test yourself and read why each one passed or failed.</p><p>So you can plug in any OCR solution and see exactly where it holds up and where it breaks.</p><p>Before you commit to a parser, run it on the slices that match your documents and read the failures yourself.</p><p>GitHub repo: <strong><a href="https://github.com/datalab-to/omni_parse_bench">https://github.com/datalab-to/omni_parse_bench</a></strong></p><p>(don&#8217;t forget to star &#127775;)</p><div><hr></div><h3><a href="https://github.com/onyx-dot-app/onyx">The No. 1 Deep Researcher Beats Claude and ChatGPT Using a Counterintuitive Trick</a></h3><p>The No. 1 deep researcher beats Claude and ChatGPT with a trick neither uses.</p><p>We studied the open-source architecture behind it.</p><p>A counterintuitive thing we found is that the orchestrator agent that runs the entire research strategy has no search access.</p><p>It can&#8217;t query the web or open URLs.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!9Gva!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F674a1a5b-f7d5-4266-8575-2cc2851e8e33_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!9Gva!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F674a1a5b-f7d5-4266-8575-2cc2851e8e33_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!9Gva!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F674a1a5b-f7d5-4266-8575-2cc2851e8e33_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!9Gva!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F674a1a5b-f7d5-4266-8575-2cc2851e8e33_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!9Gva!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F674a1a5b-f7d5-4266-8575-2cc2851e8e33_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!9Gva!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F674a1a5b-f7d5-4266-8575-2cc2851e8e33_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/674a1a5b-f7d5-4266-8575-2cc2851e8e33_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:161726,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222396733?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F674a1a5b-f7d5-4266-8575-2cc2851e8e33_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!9Gva!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F674a1a5b-f7d5-4266-8575-2cc2851e8e33_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!9Gva!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F674a1a5b-f7d5-4266-8575-2cc2851e8e33_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!9Gva!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F674a1a5b-f7d5-4266-8575-2cc2851e8e33_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!9Gva!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F674a1a5b-f7d5-4266-8575-2cc2851e8e33_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>This looks wrong at first glance. Every other deep research system gives its coordinator far more capability.</p><p>For instance:</p><ul><li><p>OpenAI&#8217;s approach trains a single model for many consecutive tool calls. It searches, reads, reasons, and writes the report in a long sequential chain.</p></li></ul><p>&#8627; The researchers behind the No. 1 system (Onyx<strong>; <a href="https://github.com/onyx-dot-app/onyx">GitHub Repo</a></strong>) observed that this causes the model to spend cycles on low-value searches instead of maintaining a high-level research strategy.</p><ul><li><p>Anthropic and Google use an orchestrator-researcher pattern similar to Onyx&#8217;s system. The key difference is how aggressively Onyx constrains the orchestrator.</p></li></ul><p>Most orchestrators have access to search and retrieval tools alongside dispatch capabilities. And the moment an orchestrator can search, it will.</p><p>So instead of decomposing a query into focused research threads, it starts answering the question itself.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!veEJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76c58abd-8d83-434d-baf8-2dc65fc0ad9a_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!veEJ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76c58abd-8d83-434d-baf8-2dc65fc0ad9a_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!veEJ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76c58abd-8d83-434d-baf8-2dc65fc0ad9a_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!veEJ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76c58abd-8d83-434d-baf8-2dc65fc0ad9a_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!veEJ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76c58abd-8d83-434d-baf8-2dc65fc0ad9a_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!veEJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76c58abd-8d83-434d-baf8-2dc65fc0ad9a_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/76c58abd-8d83-434d-baf8-2dc65fc0ad9a_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:140948,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222396733?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76c58abd-8d83-434d-baf8-2dc65fc0ad9a_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!veEJ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76c58abd-8d83-434d-baf8-2dc65fc0ad9a_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!veEJ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76c58abd-8d83-434d-baf8-2dc65fc0ad9a_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!veEJ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76c58abd-8d83-434d-baf8-2dc65fc0ad9a_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!veEJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76c58abd-8d83-434d-baf8-2dc65fc0ad9a_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>It pulls a few results, skips proper task decomposition, and produces a surface-level report from whatever it found first.</p><p>Stripping search from the orchestrator forces it to write self-contained and coherent task briefs for each research agent.</p><p>The researchers also kept the architecture only two levels deep. When info passes through multiple agents, each one subtly distorts it through summarization/reinterpretation. Keeping it to two levels prevents this.</p><p>These two constraints sit inside a larger three-phase pipeline (the visual below maps this):</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!t-Kb!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b05eeba-5445-4fea-8b6d-de8afc010a22_1210x1160.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!t-Kb!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b05eeba-5445-4fea-8b6d-de8afc010a22_1210x1160.jpeg 424w, https://substackcdn.com/image/fetch/$s_!t-Kb!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b05eeba-5445-4fea-8b6d-de8afc010a22_1210x1160.jpeg 848w, https://substackcdn.com/image/fetch/$s_!t-Kb!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b05eeba-5445-4fea-8b6d-de8afc010a22_1210x1160.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!t-Kb!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b05eeba-5445-4fea-8b6d-de8afc010a22_1210x1160.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!t-Kb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b05eeba-5445-4fea-8b6d-de8afc010a22_1210x1160.jpeg" width="1210" height="1160" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0b05eeba-5445-4fea-8b6d-de8afc010a22_1210x1160.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1160,&quot;width&quot;:1210,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:150696,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222396733?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b05eeba-5445-4fea-8b6d-de8afc010a22_1210x1160.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!t-Kb!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b05eeba-5445-4fea-8b6d-de8afc010a22_1210x1160.jpeg 424w, https://substackcdn.com/image/fetch/$s_!t-Kb!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b05eeba-5445-4fea-8b6d-de8afc010a22_1210x1160.jpeg 848w, https://substackcdn.com/image/fetch/$s_!t-Kb!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b05eeba-5445-4fea-8b6d-de8afc010a22_1210x1160.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!t-Kb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b05eeba-5445-4fea-8b6d-de8afc010a22_1210x1160.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><ul><li><p>Phase 1 decomposes the query into up to 6 research directions. No tool access prevents the model from prematurely answering.</p></li><li><p>Phase 2 dispatches 3 isolated research agents. Each runs up to 8 sub-cycles of search, read, and think, to produce an intermediate report with citations. The agents can also search internal enterprise docs (Confluence, Slack, 100+ connectors) with document-level permissions enforced, unlike proprietary solutions.</p></li><li><p>Phase 3 runs a deterministic step that renumbers and deduplicates to produce a report with a unified citation map.</p></li></ul><p>This pattern has been ranked No. 1 on DeepResearch Bench. The whole implementation is available on GitHub, and you can try it yourself.</p><p><strong><a href="https://github.com/onyx-dot-app/onyx">Here&#8217;s the Onyx Repo &#8594;</a></strong> (don&#8217;t forget to star &#11088; )</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://github.com/onyx-dot-app/onyx&quot;,&quot;text&quot;:&quot;Onyx GitHub Repo&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://github.com/onyx-dot-app/onyx"><span>Onyx GitHub Repo</span></a></p><p>&#128073; Over to you: Which page type breaks your current OCR pipeline?</p><p>Good day!</p><div><hr></div><h3><strong>P.S. For those wanting to develop &#8220;Industry ML&#8221; expertise:</strong></h3><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!cn8y!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!cn8y!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png 424w, https://substackcdn.com/image/fetch/$s_!cn8y!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png 848w, https://substackcdn.com/image/fetch/$s_!cn8y!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png 1272w, https://substackcdn.com/image/fetch/$s_!cn8y!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!cn8y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png" width="1456" height="543" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:543,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!cn8y!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png 424w, https://substackcdn.com/image/fetch/$s_!cn8y!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png 848w, https://substackcdn.com/image/fetch/$s_!cn8y!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png 1272w, https://substackcdn.com/image/fetch/$s_!cn8y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F939bede7-b0de-4770-a3e9-34d39488e776_2733x1020.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>At the end of the day, all businesses care about <em>impact</em>. That&#8217;s it!</p><ul><li><p>Can you reduce costs?</p></li><li><p>Drive revenue?</p></li><li><p>Can you scale ML models?</p></li><li><p>Predict trends before they happen?</p></li></ul><p>We have discussed several other topics (with implementations) that align with such topics.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.dailydoseofds.com/membership&quot;,&quot;text&quot;:&quot;Develop \&quot;Industry ML\&quot; Skills&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.dailydoseofds.com/membership"><span>Develop "Industry ML" Skills</span></a></p><p>Here are some of them:</p><ul><li><p>Learn everything about MCPs in this <a href="https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/">&#8203;</a><strong><a href="https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/">crash course with 9 parts &#8594;</a></strong><a href="https://www.dailydoseofds.com/model-context-protocol-crash-course-part-1/">&#8203;</a></p></li><li><p>Learn how to build Agentic systems in <strong><a href="https://www.dailydoseofds.com/ai-agents-crash-course-part-1-with-implementation/">a crash course with 14 parts</a></strong>.</p></li><li><p>Learn how to build real-world RAG apps and evaluate and scale them in <strong><a href="https://www.dailydoseofds.com/a-crash-course-on-building-rag-systems-part-1-with-implementations/">this crash course</a></strong>.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!bv9E!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!bv9E!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png 424w, https://substackcdn.com/image/fetch/$s_!bv9E!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png 848w, https://substackcdn.com/image/fetch/$s_!bv9E!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png 1272w, https://substackcdn.com/image/fetch/$s_!bv9E!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!bv9E!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png" width="1456" height="1128" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/cd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1128,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!bv9E!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png 424w, https://substackcdn.com/image/fetch/$s_!bv9E!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png 848w, https://substackcdn.com/image/fetch/$s_!bv9E!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png 1272w, https://substackcdn.com/image/fetch/$s_!bv9E!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd9f458a-9e46-487b-b611-b512a7dc74d8_1782x1380.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><ul><li><p>Learn sophisticated graph architectures and how to train them on graph data.</p></li><li><p>So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches <strong><a href="https://www.dailydoseofds.com/bi-encoders-and-cross-encoders-for-sentence-pair-similarity-scoring-part-1/">here</a></strong>.</p></li><li><p>Learn how to run large models on small devices using <a href="https://www.dailydoseofds.com/quantization-optimize-ml-models-to-run-them-on-tiny-hardware/">&#8203;</a><strong><a href="https://www.dailydoseofds.com/quantization-optimize-ml-models-to-run-them-on-tiny-hardware/">Quantization techniques</a></strong>.</p></li><li><p>Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using <a href="https://www.dailydoseofds.com/conformal-predictions-build-confidence-in-your-ml-models-predictions/">&#8203;</a><strong><a href="https://www.dailydoseofds.com/conformal-predictions-build-confidence-in-your-ml-models-predictions/">Conformal Predictions</a></strong>.</p></li><li><p>Learn how to identify causal relationships and answer business questions using causal inference in <strong><a href="https://www.dailydoseofds.com/a-crash-course-on-causality-part-1/">this crash course</a></strong>.</p></li><li><p>Learn how to scale and implement ML model training in this <strong><a href="https://www.dailydoseofds.com/how-to-scale-model-training/">practical guide</a></strong>.</p></li><li><p>Learn techniques to reliably <strong><a href="https://www.dailydoseofds.com/5-must-know-ways-to-test-ml-models-in-production-implementation-included/">test new models in production</a></strong>.</p></li><li><p>Learn how to build privacy-first ML systems using <a href="https://www.dailydoseofds.com/federated-learning-a-critical-step-towards-privacy-preserving-machine-learning/">&#8203;</a><strong><a href="https://www.dailydoseofds.com/federated-learning-a-critical-step-towards-privacy-preserving-machine-learning/">Federated Learning</a></strong>.</p></li><li><p>Learn 6 techniques with implementation to <strong><a href="https://www.dailydoseofds.com/model-compression-a-critical-step-towards-efficient-machine-learning/">compress ML models</a></strong>.</p></li></ul><p>All these resources will help you cultivate key skills that businesses and companies care about the most.</p>]]></content:encoded></item><item><title><![CDATA[Why Temperature 0 Isn't Deterministic in LLM Inference]]></title><description><![CDATA[...explained visually (a popular LLM interview question).]]></description><link>https://blog.dailydoseofds.com/p/why-temperature-0-isnt-deterministic</link><guid isPermaLink="false">https://blog.dailydoseofds.com/p/why-temperature-0-isnt-deterministic</guid><dc:creator><![CDATA[Avi Chawla]]></dc:creator><pubDate>Mon, 26 Oct 2026 23:12:43 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/df8439c9-d228-4f03-849d-6ec88d67a471_1200x1200.gif" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><strong>In today&#8217;s newsletter:</strong></p><ul><li><p>A voice agent that gets the details right.</p></li><li><p>Why temperature 0 isn&#8217;t deterministic in LLM inference.</p></li></ul><div><hr></div><h3><a href="https://fandf.co/46ixQI7">A voice agent that gets the details right</a></h3><p>Speech accuracy matters on both sides of an LLM. Speech-to-text must preserve what the caller says, while text-to-speech must preserve the response the LLM generates.</p><p>That becomes especially important with medication names, dosages, appointment times, and alphanumeric reference codes.</p><p><strong><a href="https://fandf.co/46ixQI7">Deepgram's Flux TTS</a></strong> handles the outbound speech layer. It recorded a 3.4% word error rate in an evaluation containing drug names, tracking numbers, technical strings, and alphanumerics.</p><p>We tested it through a pet-hospital conversation about a dog&#8217;s prescription refill. The agent retained the medication, dosage, prescription number, and a collection time that changed during the call.</p><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;ccd58f41-3e09-4d0f-9476-f550f3830be6&quot;,&quot;duration&quot;:null}"></div><p>The details remained correct, the voice sounded natural, and the interaction felt like a call I would be comfortable having with an actual pet hospital.</p><p><strong><a href="https://fandf.co/46ixQI7">Try Flux TTS here &#8594;</a></strong></p><p><em>Thanks to Deepgram for partnering today.</em></p><div><hr></div><h3><a href="https://www.dailydoseofds.com/llmops-crash-course-part-1/">Why temperature 0 isn&#8217;t deterministic in LLM inference</a></h3><p>At temperature 0, the model does not sample. At every step, it picks the token with the highest logit, so the same prompt should give the same output every time.</p><p>In practice, it doesn&#8217;t.</p><p>Thinking Machines sent &#8220;Tell me about Richard Feynman&#8221; to Qwen3-235B 1,000 times at temperature 0 and got 80 different completions.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!yMSt!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e9b6f98-162e-4336-90f2-b7de4fd12ec2_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!yMSt!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e9b6f98-162e-4336-90f2-b7de4fd12ec2_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!yMSt!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e9b6f98-162e-4336-90f2-b7de4fd12ec2_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!yMSt!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e9b6f98-162e-4336-90f2-b7de4fd12ec2_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!yMSt!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e9b6f98-162e-4336-90f2-b7de4fd12ec2_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!yMSt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e9b6f98-162e-4336-90f2-b7de4fd12ec2_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0e9b6f98-162e-4336-90f2-b7de4fd12ec2_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:116734,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222233470?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e9b6f98-162e-4336-90f2-b7de4fd12ec2_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!yMSt!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e9b6f98-162e-4336-90f2-b7de4fd12ec2_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!yMSt!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e9b6f98-162e-4336-90f2-b7de4fd12ec2_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!yMSt!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e9b6f98-162e-4336-90f2-b7de4fd12ec2_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!yMSt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0e9b6f98-162e-4336-90f2-b7de4fd12ec2_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The same prompt sent 1,000 times at temperature 0 produced 80 unique completions.</figcaption></figure></div><p>Picking the highest logit is perfectly deterministic. The variation comes from computing the logits themselves, which happens from scratch on every request through billions of additions.</p><p>If those numbers come out slightly different, argmax can pick a different token.</p><p>Let&#8217;s understand why the logits change.</p><h4>1) The order of addition changes the result</h4><p>GPUs store numbers with limited precision. fp16 keeps only 11 significant bits (10 stored mantissa bits plus 1 implicit bit).</p><p>Between 1024 and 2048, that leaves no room for anything after the decimal point. Representable values are spaced exactly 1 apart, so 1024.4 gets rounded back to 1024.</p><p>Now consider 1024 + 0.4 + 0.4 in two orders. The diagram below shows both.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!mfKE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F77d6778a-d405-4864-b1af-305fb20e8201_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mfKE!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F77d6778a-d405-4864-b1af-305fb20e8201_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!mfKE!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F77d6778a-d405-4864-b1af-305fb20e8201_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!mfKE!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F77d6778a-d405-4864-b1af-305fb20e8201_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!mfKE!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F77d6778a-d405-4864-b1af-305fb20e8201_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!mfKE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F77d6778a-d405-4864-b1af-305fb20e8201_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/77d6778a-d405-4864-b1af-305fb20e8201_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:117965,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222233470?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F77d6778a-d405-4864-b1af-305fb20e8201_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!mfKE!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F77d6778a-d405-4864-b1af-305fb20e8201_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!mfKE!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F77d6778a-d405-4864-b1af-305fb20e8201_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!mfKE!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F77d6778a-d405-4864-b1af-305fb20e8201_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!mfKE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F77d6778a-d405-4864-b1af-305fb20e8201_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Adding the same three numbers in a different order gives a different answer in fp16.</figcaption></figure></div><p>Left to right, 1024 + 0.4 rounds back to 1024, and adding 0.4 again rounds back to 1024 once more. The answer is 1024.</p><p>If the small numbers go first, 0.4 + 0.4 gives 0.8, and 1024 + 0.8 rounds up to 1025. The answer is 1025.</p><p>You can verify this yourself:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;python&quot;,&quot;nodeId&quot;:null}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-python">import numpy as np

a, b = np.float16(1024), np.float16(0.4)
print((a + b) + b)   # 1024.0
print(a + (b + b))   # 1025.0</code></pre></div><p>In other words, floating-point addition is not associative. An LLM performs billions of such additions in every forward pass, so if their order changes, the logits change slightly too.</p><h4>2) What changes the order?</h4><p>The popular answer is that parallel GPU threads finish in an unpredictable order.</p><p>That isn&#8217;t the main reason. The kernels in a typical forward pass compute each sum in a fixed order, and the same matrix multiplication on the same inputs gives bit-for-bit identical results every time you run it.</p><p>So the forward pass itself is run-to-run deterministic. Something else must be changing between requests.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!2PYe!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14d9787f-15fc-4d99-b969-e419ce536ddc_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!2PYe!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14d9787f-15fc-4d99-b969-e419ce536ddc_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!2PYe!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14d9787f-15fc-4d99-b969-e419ce536ddc_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!2PYe!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14d9787f-15fc-4d99-b969-e419ce536ddc_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!2PYe!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14d9787f-15fc-4d99-b969-e419ce536ddc_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!2PYe!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14d9787f-15fc-4d99-b969-e419ce536ddc_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/14d9787f-15fc-4d99-b969-e419ce536ddc_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:170141,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222233470?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14d9787f-15fc-4d99-b969-e419ce536ddc_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!2PYe!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14d9787f-15fc-4d99-b969-e419ce536ddc_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!2PYe!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14d9787f-15fc-4d99-b969-e419ce536ddc_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!2PYe!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14d9787f-15fc-4d99-b969-e419ce536ddc_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!2PYe!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14d9787f-15fc-4d99-b969-e419ce536ddc_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Thread timing is not the cause because the same kernel on the same inputs always returns the same bits.</figcaption></figure></div><h4>3) The real reason is the other requests on the server</h4><p>An inference server groups requests from many users into a batch and runs them together.</p><p>The kernels decide how to split their work based on the batch size. With a large batch, every GPU core is busy, so each core computes its sum in one go.</p><p>With a small batch, many cores would sit idle. So the kernel cuts each long sum into chunks, spreads them across cores, and combines the partial results at the end. This shows up in matmuls (split-K), in RMSNorm, and in attention during decoding (split-KV, as in FlashDecoding).</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!kgu7!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d78bad-d6ae-4739-9c68-87d15fa1ce91_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!kgu7!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d78bad-d6ae-4739-9c68-87d15fa1ce91_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!kgu7!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d78bad-d6ae-4739-9c68-87d15fa1ce91_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!kgu7!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d78bad-d6ae-4739-9c68-87d15fa1ce91_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!kgu7!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d78bad-d6ae-4739-9c68-87d15fa1ce91_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!kgu7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d78bad-d6ae-4739-9c68-87d15fa1ce91_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b3d78bad-d6ae-4739-9c68-87d15fa1ce91_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:176696,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222233470?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d78bad-d6ae-4739-9c68-87d15fa1ce91_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!kgu7!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d78bad-d6ae-4739-9c68-87d15fa1ce91_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!kgu7!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d78bad-d6ae-4739-9c68-87d15fa1ce91_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!kgu7!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d78bad-d6ae-4739-9c68-87d15fa1ce91_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!kgu7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb3d78bad-d6ae-4739-9c68-87d15fa1ce91_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The batch size decides how a reduction is split and therefore the order of addition.</figcaption></figure></div><p>A different split means a different order of addition. The batch size, in turn, depends on how many other people are using the server at that moment.</p><p>Since you have no control over that, the output looks random from your side. Thinking Machines call the missing property <strong>batch invariance</strong>: a request&#8217;s numbers should not depend on what else is in the batch.</p><h4>4) How a tiny difference changes the entire answer</h4><p>A difference in the fourth decimal place sounds harmless, and most of the time it is.</p><p>Consider two runs of the same prompt. The numbers below are illustrative, but the mechanism is real.</p><p>&#8594; In a batch of 8, the logits come out as &#8220;Queens&#8221; = 12.3417 and &#8220;New&#8221; = 12.3415, so argmax picks &#8220;Queens&#8221;.</p><p>&#8594; In a batch of 32, the rounding works out slightly differently, giving &#8220;Queens&#8221; = 12.3414 and &#8220;New&#8221; = 12.3416, so argmax picks &#8220;New&#8221;.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!OYsa!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e3b7233-46d0-4401-9c99-36b895303e42_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!OYsa!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e3b7233-46d0-4401-9c99-36b895303e42_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!OYsa!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e3b7233-46d0-4401-9c99-36b895303e42_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!OYsa!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e3b7233-46d0-4401-9c99-36b895303e42_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!OYsa!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e3b7233-46d0-4401-9c99-36b895303e42_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!OYsa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e3b7233-46d0-4401-9c99-36b895303e42_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9e3b7233-46d0-4401-9c99-36b895303e42_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:156830,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222233470?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e3b7233-46d0-4401-9c99-36b895303e42_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!OYsa!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e3b7233-46d0-4401-9c99-36b895303e42_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!OYsa!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e3b7233-46d0-4401-9c99-36b895303e42_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!OYsa!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e3b7233-46d0-4401-9c99-36b895303e42_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!OYsa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e3b7233-46d0-4401-9c99-36b895303e42_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">These illustrative logits show how a near tie lets tiny rounding noise flip the chosen token.</figcaption></figure></div><p>The output only flips at such near ties. Most of the time, the top token wins by a comfortable margin and rounding noise can&#8217;t touch it.</p><p>The problem is what happens after a flip. Generation is autoregressive, so every token after that point is computed from a different prefix.</p><p>That&#8217;s exactly what Thinking Machines saw. In the Feynman test, all 1,000 runs matched for the first 102 tokens. At token 103, 992 runs wrote &#8220;Queens, New York&#8221; while 8 wrote &#8220;New York City&#8221;. From there, the completions diverged into 80 unique outputs.</p><h4>5) The fix and its cost</h4><p>The fix follows directly from the cause. Every kernel should split its work the same way regardless of batch size, even if that leaves some cores idle on small batches.</p><p>For attention, Thinking Machines fix the split size rather than the number of splits, so a given token always lands in the same chunk. They also update the KV cache before the attention kernel runs, so cached and freshly computed tokens are reduced in the same order.</p><p>With these batch-invariant kernels, all 1,000 Feynman runs were identical.</p><p>The cost is speed. On Qwen3-8B with 1,000 requests, default vLLM took 26 seconds. The first deterministic version took 55 seconds, and an improved attention kernel brought it down to 42 seconds.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!r9F1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb389fdcf-3c82-4653-bedc-f4ee99d5822c_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!r9F1!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb389fdcf-3c82-4653-bedc-f4ee99d5822c_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!r9F1!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb389fdcf-3c82-4653-bedc-f4ee99d5822c_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!r9F1!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb389fdcf-3c82-4653-bedc-f4ee99d5822c_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!r9F1!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb389fdcf-3c82-4653-bedc-f4ee99d5822c_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!r9F1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb389fdcf-3c82-4653-bedc-f4ee99d5822c_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b389fdcf-3c82-4653-bedc-f4ee99d5822c_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:200405,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222233470?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb389fdcf-3c82-4653-bedc-f4ee99d5822c_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!r9F1!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb389fdcf-3c82-4653-bedc-f4ee99d5822c_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!r9F1!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb389fdcf-3c82-4653-bedc-f4ee99d5822c_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!r9F1!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb389fdcf-3c82-4653-bedc-f4ee99d5822c_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!r9F1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb389fdcf-3c82-4653-bedc-f4ee99d5822c_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Batch-invariant kernels make all 1,000 runs identical at some cost in speed.</figcaption></figure></div><p>Both major open-source engines now ship this:</p><ul><li><p>vLLM: the <code>VLLM_BATCH_INVARIANT=1</code> environment variable (beta).</p></li><li><p>SGLang: the <code>--enable-deterministic-inference</code> flag, with a reported average slowdown of 34.35%.</p></li></ul><p>If you just chat with a model, none of this matters much. It matters a lot for evals that need to reproduce, for replaying a bad production output while debugging, and for RL, where the sampler and the trainer must compute the same probabilities for training to stay truly on-policy.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!sNDC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fffaf5fb7-10a1-473b-8a34-0319386b3868_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!sNDC!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fffaf5fb7-10a1-473b-8a34-0319386b3868_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!sNDC!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fffaf5fb7-10a1-473b-8a34-0319386b3868_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!sNDC!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fffaf5fb7-10a1-473b-8a34-0319386b3868_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!sNDC!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fffaf5fb7-10a1-473b-8a34-0319386b3868_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!sNDC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fffaf5fb7-10a1-473b-8a34-0319386b3868_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ffaf5fb7-10a1-473b-8a34-0319386b3868_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:196423,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222233470?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fffaf5fb7-10a1-473b-8a34-0319386b3868_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!sNDC!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fffaf5fb7-10a1-473b-8a34-0319386b3868_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!sNDC!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fffaf5fb7-10a1-473b-8a34-0319386b3868_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!sNDC!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fffaf5fb7-10a1-473b-8a34-0319386b3868_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!sNDC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fffaf5fb7-10a1-473b-8a34-0319386b3868_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Both vLLM and SGLang let you turn this on when reproducibility is worth the slowdown.</figcaption></figure></div><p>So temperature 0 removes the sampling randomness, but it doesn&#8217;t remove the batch.</p><p>The visual below depicts the entire process.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!JVC3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0223c341-c74d-4649-aa3b-d451d41dbac1_2752x1536.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!JVC3!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0223c341-c74d-4649-aa3b-d451d41dbac1_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!JVC3!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0223c341-c74d-4649-aa3b-d451d41dbac1_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!JVC3!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0223c341-c74d-4649-aa3b-d451d41dbac1_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!JVC3!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0223c341-c74d-4649-aa3b-d451d41dbac1_2752x1536.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!JVC3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0223c341-c74d-4649-aa3b-d451d41dbac1_2752x1536.jpeg" width="1456" height="813" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0223c341-c74d-4649-aa3b-d451d41dbac1_2752x1536.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:813,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:98918,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://blog.dailydoseofds.com/i/222233470?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0223c341-c74d-4649-aa3b-d451d41dbac1_2752x1536.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!JVC3!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0223c341-c74d-4649-aa3b-d451d41dbac1_2752x1536.jpeg 424w, https://substackcdn.com/image/fetch/$s_!JVC3!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0223c341-c74d-4649-aa3b-d451d41dbac1_2752x1536.jpeg 848w, https://substackcdn.com/image/fetch/$s_!JVC3!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0223c341-c74d-4649-aa3b-d451d41dbac1_2752x1536.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!JVC3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0223c341-c74d-4649-aa3b-d451d41dbac1_2752x1536.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Server load sets the batch size that ultimately decides which token gets picked.</figcaption></figure></div><p>If you&#8217;re shipping LLMs to production, issues like this one rarely show up in a demo. They show up later as evals that won&#8217;t reproduce, latency that spikes under load, or a serving bill that grows faster than traffic.</p><p>That&#8217;s the gap the LLMOps course fills. It walks you through the full lifecycle, from how LLMs work under the hood to prompting, context, evaluation, fine-tuning, inference optimization, and serving, so you can reason about these problems before they hit your users.</p><p>If you want to learn the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:</p><ul><li><p><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-1/">Read Part 1 on fundamentals of LLMOps here &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-2">Read Part 2 on understanding the core building blocks of LLMs &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-3">Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-4">Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-5">Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-6">Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-7">Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-8">Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-9">Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-10">Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-11">Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-12/">Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-13/">Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism &#8594;</a></strong></p></li><li><p><strong><a href="https://www.dailydoseofds.com/llmops-crash-course-part-14">Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions &#8594;</a></strong></p></li></ul><p>&#128073; Over to you: Have you ever had an eval score move between two identical runs?</p><p>Good day!</p>]]></content:encoded></item></channel></rss>