<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/"><channel><title>Furkan Köykıran Blog</title><link>https://blog.furkankoykiran.com.tr/en/</link><description>Furkan Köykıran's engineering blog: MCP servers and AI agents, open-source contributions, Python trading bots, Python backend and DevOps, written from hands-on work.</description><generator>Hugo 0.167.0</generator><language>en-US</language><copyright>© 2026 Furkan Köykıran. Posts and images all rights reserved; code samples MIT licensed.</copyright><managingEditor>furkankoykiran@gmail.com (Furkan Köykıran)</managingEditor><lastBuildDate>Wed, 06 May 2026 12:00:00 +0300</lastBuildDate><atom:link href="https://blog.furkankoykiran.com.tr/en/feed.xml" rel="self" type="application/rss+xml"/><atom:link href="https://pubsubhubbub.appspot.com/" rel="hub"/><image><url>https://blog.furkankoykiran.com.tr/android-chrome-192x192.png</url><title>Furkan Köykıran Blog</title><link>https://blog.furkankoykiran.com.tr/en/</link></image><item><title>GitHub Actions Self-Hosted Runner on Linux: Setup Guide</title><link>https://blog.furkankoykiran.com.tr/en/posts/github-actions-self-hosted-runner/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/github-actions-self-hosted-runner/</guid><pubDate>Wed, 06 May 2026 12:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>DevOps</category><category>github-actions</category><category>ci-cd</category><category>devops</category><category>linux</category><category>security</category><description>My free GitHub Actions minutes (2,000 a month) ran out, so I made a Linux server a self-hosted runner: dedicated user, SHA256 check, systemd, labels, security.</description><content:encoded><![CDATA[<p>GitHub Actions gives private repositories a monthly quota of free minutes: 2,000 on the Free plan, 3,000 on Pro and Team. Run a multi-step CI pipeline on every push and every PR and the quota goes faster than you'd expect. Mine ran out in May 2026. Instead of paying for more minutes, I turned a Linux server I already had running into a self-hosted runner. This guide walks through the setup step by step: a dedicated user, checksum verification, a systemd service, labels, and the security cost of a runner that keeps its disk.</p>

<h2 class="relative group">Why a self-hosted runner
    <div id="why-a-self-hosted-runner" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-a-self-hosted-runner" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>According to GitHub's <a href="https://docs.github.com/en/billing/concepts/product-billing/github-actions"  target="_blank" rel="noreferrer">billing docs</a>, jobs on self-hosted runners don't consume Actions minutes. Past the quota, a standard Linux runner costs $0.006 a minute. My server was on anyway, so the extra cost was CPU and disk.</p>
<p>There's a second benefit: the runner's disk persists between jobs. In my project the first run was slow because Poetry installed every dependency from scratch. After that, runs were noticeably faster because the virtualenv was still there. That persistence is also the biggest downside, which I'll come back to in the security section.</p>
<p>It helps to know how the runner talks to GitHub. It opens an outbound HTTPS connection and waits for work, so you don't need to open any inbound ports on the server:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/github-actions-self-hosted-runner/img/self-hosted-runner-flow_hu_f090857d7ce2c8a5.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/github-actions-self-hosted-runner/img/self-hosted-runner-flow_hu_2881d0b8cfbe0638.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/github-actions-self-hosted-runner/img/self-hosted-runner-flow_hu_d20d25c0136dac88.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/github-actions-self-hosted-runner/img/self-hosted-runner-flow_hu_d920671300665a09.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/github-actions-self-hosted-runner/img/self-hosted-runner-flow_hu_4bcca2d419ea038.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/github-actions-self-hosted-runner/img/self-hosted-runner-flow_hu_63c4d5c5492e4e66.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/github-actions-self-hosted-runner/img/self-hosted-runner-flow_hu_4bcca2d419ea038.webp" alt="The runner long-polls GitHub over HTTPS, takes a job whose labels match, runs it as github-runner and sends the results back" width="1600" height="986"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>The runner always opens the connection, so no inbound ports are needed</figcaption></figure>
<h2 class="relative group">Create a dedicated user
    <div id="create-a-dedicated-user" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#create-a-dedicated-user" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Don't run the runner as root; <code>config.sh</code> refuses to anyway. Create a system user for it:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo useradd -r -m -d /opt/github-runner -s /bin/bash <span class="se">\
</span></span></span><span class="line"><span class="cl">  -c <span class="s2">&#34;GitHub Actions Runner&#34;</span> github-runner</span></span></code></pre></div></div>
<p><code>-r</code> makes it a system account and <code>-m -d</code> creates its home at <code>/opt/github-runner</code>. The account has no password, so nobody can log in with one, and it has no sudo rights. I give it <code>/bin/bash</code> because the next steps run as this user.</p>

<h2 class="relative group">Download the runner and verify its hash
    <div id="download-the-runner-and-verify-its-hash" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#download-the-runner-and-verify-its-hash" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Do everything from here on as the runner user, so files get the right owner from the start:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo -iu github-runner
</span></span><span class="line"><span class="cl"><span class="nb">cd</span> /opt/github-runner
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nv">VERSION</span><span class="o">=</span>2.338.1
</span></span><span class="line"><span class="cl">curl -fsSLO <span class="s2">&#34;https://github.com/actions/runner/releases/download/v</span><span class="si">${</span><span class="nv">VERSION</span><span class="si">}</span><span class="s2">/actions-runner-linux-x64-</span><span class="si">${</span><span class="nv">VERSION</span><span class="si">}</span><span class="s2">.tar.gz&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;08e9ccb3604d2462c24b5e33604dc30810296ca15cc65f23cffd4317eb144c37  actions-runner-linux-x64-</span><span class="si">${</span><span class="nv">VERSION</span><span class="si">}</span><span class="s2">.tar.gz&#34;</span> <span class="p">|</span> sha256sum -c
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">tar xzf <span class="s2">&#34;actions-runner-linux-x64-</span><span class="si">${</span><span class="nv">VERSION</span><span class="si">}</span><span class="s2">.tar.gz&#34;</span></span></span></code></pre></div></div>
<p>The version and hash in the example belong to runner 2.338.1. Every release lists the hash for each package in its notes, right below the Linux x64 install commands:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/posts/github-actions-self-hosted-runner-tr/img/runner-release-linux-x64-sha256_hu_cfeafead80cbc9cb.avif 480w, https://blog.furkankoykiran.com.tr/posts/github-actions-self-hosted-runner-tr/img/runner-release-linux-x64-sha256_hu_d4f4831e9edf9d92.avif 800w, https://blog.furkankoykiran.com.tr/posts/github-actions-self-hosted-runner-tr/img/runner-release-linux-x64-sha256_hu_fbb6be42c6e05c0c.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/posts/github-actions-self-hosted-runner-tr/img/runner-release-linux-x64-sha256_hu_3df76b76752ef757.webp 480w, https://blog.furkankoykiran.com.tr/posts/github-actions-self-hosted-runner-tr/img/runner-release-linux-x64-sha256_hu_8c15ed3042b8ca6a.webp 800w, https://blog.furkankoykiran.com.tr/posts/github-actions-self-hosted-runner-tr/img/runner-release-linux-x64-sha256_hu_e9eec2eb6fbf86ab.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/posts/github-actions-self-hosted-runner-tr/img/runner-release-linux-x64-sha256_hu_8c15ed3042b8ca6a.webp" alt="actions/runner 2.338.1 release notes: the Linux x64 install commands and the package&#39;s SHA-256 checksum" width="1600" height="614"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Source: GitHub, actions/runner v2.338.1 release notes (quotation)</figcaption></figure><p>Use the current version from the <a href="https://github.com/actions/runner/releases"  target="_blank" rel="noreferrer">releases page</a> and the hash from that release. You're downloading a binary that will run code on your server; don't skip the checksum.</p>

<h2 class="relative group">Register the runner with your repository
    <div id="register-the-runner-with-your-repository" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#register-the-runner-with-your-repository" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Repository → Settings → Actions → Runners → New self-hosted runner shows a registration token. It's valid for one hour, so have the command ready before you generate it:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">./config.sh <span class="se">\
</span></span></span><span class="line"><span class="cl">  --url https://github.com/YOUR_USER/YOUR_REPO <span class="se">\
</span></span></span><span class="line"><span class="cl">  --token YOUR_REGISTRATION_TOKEN <span class="se">\
</span></span></span><span class="line"><span class="cl">  --name prod-server-01 <span class="se">\
</span></span></span><span class="line"><span class="cl">  --labels prod <span class="se">\
</span></span></span><span class="line"><span class="cl">  --unattended</span></span></code></pre></div></div>
<p>The runner adds the <code>self-hosted</code>, <code>linux</code> and <code>x64</code> labels on its own, so <code>--labels</code> only needs your custom ones. <code>--unattended</code> skips the interactive prompts. On success you'll see something like:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">√ Connected to GitHub
</span></span><span class="line"><span class="cl">√ Runner successfully added
</span></span><span class="line"><span class="cl">√ Settings Saved</span></span></code></pre></div></div>
<p>Then <code>exit</code> back to your own user.</p>

<h2 class="relative group">Install it as a systemd service
    <div id="install-it-as-a-systemd-service" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#install-it-as-a-systemd-service" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p><code>svc.sh</code> in the runner folder generates a systemd unit. Run it as root and tell it which user the service runs as:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="nb">cd</span> /opt/github-runner
</span></span><span class="line"><span class="cl">sudo ./svc.sh install github-runner
</span></span><span class="line"><span class="cl">sudo ./svc.sh start
</span></span><span class="line"><span class="cl">sudo ./svc.sh status</span></span></code></pre></div></div>
<p>Logs go to journald:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sudo journalctl -u <span class="s1">&#39;actions.runner.*&#39;</span> -n <span class="m">50</span> --no-pager</span></span></code></pre></div></div>
<p>Look for <code>Listening for Jobs</code>. Once that line shows up, the runner is waiting for work, and it appears as &quot;Idle&quot; in the repository's Runners list.</p>

<h2 class="relative group">Point your workflows at it
    <div id="point-your-workflows-at-it" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#point-your-workflows-at-it" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Replace <code>runs-on</code> with the labels:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">jobs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">lint</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">self-hosted, linux, prod]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/checkout@v7</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">poetry install --no-interaction</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">poetry run ruff check .</span></span></span></code></pre></div></div>
<p>A job goes to a runner that has every label in the list. With several servers, labels like <code>prod</code> and <code>staging</code> let you choose where each job runs.</p>
<p>When I opened my first PR after the switch, the Actions page showed <code>Running job: Backend Lint (ruff)</code>. The job was running on the server in front of me, not on a VM spun up somewhere in Azure. The whole setup, from creating the user to updating the workflow files, took about 20 minutes.</p>

<h2 class="relative group">Security: the cost of a persistent runner
    <div id="security-the-cost-of-a-persistent-runner" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#security-the-cost-of-a-persistent-runner" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>GitHub-hosted runners start every job on a fresh VM. A self-hosted runner doesn't. Anything one job leaves on disk is there for the next one. These are the rules I follow:</p>
<ul>
<li>Don't use self-hosted runners on public repositories. A pull request from a fork can change your workflow and run code on your server, and GitHub's <a href="https://docs.github.com/en/actions/reference/security/secure-use"  target="_blank" rel="noreferrer">secure use guide</a> says as much. My runner only serves private repos.</li>
<li>Lock the runner directory to the runner user: <code>sudo chown -R github-runner:github-runner /opt/github-runner</code> and <code>sudo chmod 700 /opt/github-runner</code>.</li>
<li>Keep the runner user out of sudoers. If a job really needs a root command, write a narrow sudoers rule for that one command.</li>
<li>Adding the runner user to the <code>docker</code> group is effectively giving it root. If jobs need Docker, make that choice knowingly or use rootless Docker.</li>
<li>Avoid steps that write secrets to disk, and delete them at the end of the job if you must. The next job sees the same disk.</li>
<li>If every job needs a clean start, register the runner with <code>--ephemeral</code>. It takes one job and then deregisters. The trade-off is that something has to register a new runner for each job, usually a script or Actions Runner Controller.</li>
</ul>
<p>By default the runner downloads and installs new versions itself. Leave that on: GitHub stops sending jobs to runners that fall too far behind. OS patches are still your job.</p>
<p>To remove the runner, stop the service first, then delete the registration:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="nb">cd</span> /opt/github-runner
</span></span><span class="line"><span class="cl">sudo ./svc.sh stop
</span></span><span class="line"><span class="cl">sudo ./svc.sh uninstall
</span></span><span class="line"><span class="cl">sudo -u github-runner ./config.sh remove --token YOUR_REMOVAL_TOKEN</span></span></code></pre></div></div>
<p>You get the removal token from the runner's menu on the Runners page.</p>

<h2 class="relative group">Which one when
    <div id="which-one-when" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#which-one-when" aria-label="Anchor">#</a>
    </span>
    
</h2>
<div class="fk-table" role="region" tabindex="0" aria-label="Scrollable table">
<table>
  <thead>
    <tr>
      <th></th>
      <th>GitHub-hosted</th>
      <th>Self-hosted</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Cost</td>
      <td>Free within quota, then per minute</td>
      <td>No Actions minutes, you pay for the server</td>
    </tr>
    <tr>
      <td>Maintenance</td>
      <td>None</td>
      <td>Runner and OS updates</td>
    </tr>
    <tr>
      <td>Speed</td>
      <td>Setup and cache download every job</td>
      <td>Fast after the first job thanks to local cache</td>
    </tr>
    <tr>
      <td>Isolation</td>
      <td>Fresh VM every job</td>
      <td>Jobs share the same disk</td>
    </tr>
    <tr>
      <td>Network</td>
      <td>GitHub's network</td>
      <td>Your network, easy access to internal services</td>
    </tr>
  </tbody>
</table>
</div><p>If you're under the quota, GitHub-hosted runners are the right default. If you regularly blow through it and have a server sitting idle, a self-hosted runner makes sense as long as you follow the security rules above.</p>

<h2 class="relative group">Where to go next
    <div id="where-to-go-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-to-go-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>My <a href="https://blog.furkankoykiran.com.tr/en/posts/awesome-trending-repos-web/" >awesome-trending-repos post</a> shows a cron-driven Actions workflow that deploys to GitHub Pages, and the <a href="https://blog.furkankoykiran.com.tr/en/posts/gitlab-github-jekyll-migration/" >GitLab to GitHub migration story</a> compares GitHub Actions with GitLab CI from the time I had to use both. For the official steps, <a href="https://docs.github.com/en/actions/how-tos/manage-runners/self-hosted-runners/add-runners"  target="_blank" rel="noreferrer">GitHub's guide to adding self-hosted runners</a> is the most current source.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/github-actions-self-hosted-runner/feature_hu_3fbf37ffe0bda6c1.jpg" medium="image" type="image/jpeg"/></item><item><title>freqtrade-mcp: An MCP Server for Your Freqtrade Bot</title><link>https://blog.furkankoykiran.com.tr/en/posts/freqtrade-mcp-server/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/freqtrade-mcp-server/</guid><pubDate>Sat, 04 Apr 2026 13:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>MCP &amp; AI</category><category>mcp</category><category>trading</category><category>typescript</category><category>claude</category><category>ai-agents</category><description>My TypeScript MCP server that exposes the Freqtrade REST API to Claude and other MCP clients: 19 tools, setup, code layout and live-trading caveats.</description><content:encoded><![CDATA[<p>I run a Freqtrade bot. Like most people with a trading bot, I tell myself I don't check it all the time, and then I check it all the time: open FreqUI, click through tabs, look at one number, close the tab, open it again fifteen minutes later.</p>
<p>Claude was already open, and the bot already had a perfectly good REST API. So I wrote <a href="https://github.com/furkankoykiran/freqtrade-mcp"  target="_blank" rel="noreferrer">freqtrade-mcp</a>, a TypeScript MCP server that exposes Freqtrade's REST API as MCP tools. It works with Claude Desktop, Claude Code, Cursor, Cline and anything else that speaks the <a href="https://modelcontextprotocol.io"  target="_blank" rel="noreferrer">Model Context Protocol</a>. Version 1.0.0 went to npm on April 4, 2026.</p>

<h2 class="relative group">What it does
    <div id="what-it-does" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-it-does" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The questions I actually ask look like this:</p>
<blockquote><p>&quot;Am I up this month?&quot;</p>
<p>&quot;Are any open trades in the red?&quot;</p>
<p>&quot;What do the last 50 BTC/USDT candles on the 4h look like?&quot;</p>
</blockquote><p>Claude maps each question to a tool, reads the JSON that comes back from Freqtrade and summarizes it, so I don't have to parse numbers at 11pm. The tools fall into six groups:</p>
<div class="fk-table" role="region" tabindex="0" aria-label="Scrollable table">
<table>
  <thead>
    <tr>
      <th>Group</th>
      <th>Tools</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Account</td>
      <td><code>get_profit_stats</code>, <code>get_balance</code>, <code>get_performance</code></td>
    </tr>
    <tr>
      <td>Trades</td>
      <td><code>get_open_trades</code>, <code>get_trade</code>, <code>get_trade_history</code></td>
    </tr>
    <tr>
      <td>Market data</td>
      <td><code>get_market_data</code></td>
    </tr>
    <tr>
      <td>Pair lists</td>
      <td><code>get_whitelist</code>, <code>get_blacklist</code>, <code>add_to_blacklist</code>, <code>remove_from_blacklist</code></td>
    </tr>
    <tr>
      <td>Execution</td>
      <td><code>execute_trade</code>, <code>force_exit_trade</code></td>
    </tr>
    <tr>
      <td>Lifecycle</td>
      <td><code>start_bot</code>, <code>stop_bot</code>, <code>reload_config</code>, <code>get_bot_info</code>, <code>get_locks</code>, <code>delete_lock</code></td>
    </tr>
  </tbody>
</table>
</div><p>That's 19 tools in the code. The README still says 15, which is on my list to fix.</p>
<p>A few practical notes:</p>
<ul>
<li><code>get_market_data</code> uses Freqtrade's <code>/pair_candles</code> endpoint, which returns the candle data the bot has analyzed. In practice that means pairs the bot is tracking, not any pair on the exchange.</li>
<li><code>get_trade_history</code> and <code>get_market_data</code> return at most 500 rows.</li>
<li><code>get_bot_info</code> returns the bot version together with the time of the last strategy cycle. That answers &quot;is the bot running, or has it been silently stuck?&quot;</li>
</ul>

<h2 class="relative group">Architecture
    <div id="architecture" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#architecture" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The server is a single Node.js process that talks stdio to the client and HTTP Basic Auth to Freqtrade under <code>/api/v1</code>:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/freqtrade-mcp-server/img/freqtrade-mcp-request-flow_hu_436eb35942594d88.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/freqtrade-mcp-server/img/freqtrade-mcp-request-flow_hu_43d26f2730bab15d.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/freqtrade-mcp-server/img/freqtrade-mcp-request-flow_hu_c77c007fc3790aae.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/freqtrade-mcp-server/img/freqtrade-mcp-request-flow_hu_23be5b2e31a357de.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/freqtrade-mcp-server/img/freqtrade-mcp-request-flow_hu_d59e87f10f9e459f.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/freqtrade-mcp-server/img/freqtrade-mcp-request-flow_hu_cb1d72189feb8965.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/freqtrade-mcp-server/img/freqtrade-mcp-request-flow_hu_d59e87f10f9e459f.webp" alt="Three steps: a chat question, Claude calling the get_open_trades tool over stdio, and the Freqtrade REST API answering under /api/v1" width="1600" height="1088"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>A chat question becomes one MCP tool call and one REST request</figcaption></figure><p>The code has three layers:</p>
<ul>
<li><code>src/client.ts</code> is a typed HTTP client for the Freqtrade API. It uses the <code>fetch</code> built into Node 18+, aborts each request after 10 seconds with an <code>AbortController</code>, and puts Freqtrade's <code>detail</code> field into the error message on non-2xx responses.</li>
<li><code>src/tools/*.ts</code> has one file per group, and each tool declares its parameters with a zod schema.</li>
<li><code>src/index.ts</code> builds the client from environment variables, registers the tools and connects a <code>StdioServerTransport</code>.</li>
</ul>
<p>That keeps the repo small: the code under <code>src/</code>, a <code>.env.example</code>, and a <code>.github/</code> folder for CI and the npm release.</p>
<p>A complete tool is this short:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="nx">server</span><span class="p">.</span><span class="nx">tool</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">  <span class="s1">&#39;get_trade&#39;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="s1">&#39;Retrieve detailed information about a specific trade by its ID, including full history, fees, and profit/loss.&#39;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nx">trade_id</span>: <span class="kt">z.number</span><span class="p">().</span><span class="nx">int</span><span class="p">().</span><span class="nx">positive</span><span class="p">().</span><span class="nx">describe</span><span class="p">(</span><span class="s1">&#39;The numeric trade ID to look up.&#39;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">  <span class="p">},</span>
</span></span><span class="line"><span class="cl">  <span class="kr">async</span> <span class="p">({</span> <span class="nx">trade_id</span> <span class="p">})</span> <span class="o">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="kr">const</span> <span class="nx">data</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">client</span><span class="p">.</span><span class="nx">getTrade</span><span class="p">(</span><span class="nx">trade_id</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">      <span class="k">return</span> <span class="p">{</span> <span class="nx">content</span><span class="o">:</span> <span class="p">[{</span> <span class="kr">type</span><span class="o">:</span> <span class="s1">&#39;text&#39;</span><span class="p">,</span> <span class="nx">text</span>: <span class="kt">JSON.stringify</span><span class="p">(</span><span class="nx">data</span><span class="p">,</span> <span class="kc">null</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> <span class="p">}]</span> <span class="p">};</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="k">return</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="nx">content</span><span class="o">:</span> <span class="p">[{</span> <span class="kr">type</span><span class="o">:</span> <span class="s1">&#39;text&#39;</span><span class="p">,</span> <span class="nx">text</span><span class="o">:</span> <span class="sb">`Error fetching trade </span><span class="si">${</span><span class="nx">trade_id</span><span class="si">}</span><span class="sb">: </span><span class="si">${</span><span class="p">(</span><span class="nx">err</span> <span class="kr">as</span> <span class="nb">Error</span><span class="p">).</span><span class="nx">message</span><span class="si">}</span><span class="sb">`</span> <span class="p">}],</span>
</span></span><span class="line"><span class="cl">        <span class="nx">isError</span>: <span class="kt">true</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="p">};</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">},</span>
</span></span><span class="line"><span class="cl"><span class="p">);</span></span></span></code></pre></div></div>
<p>I return errors with <code>isError: true</code> instead of throwing them. The model sees &quot;trade 42 not found&quot;, can tell the user something sensible, and the session stays up. The descriptions matter too, because they decide when the model picks a tool. For <code>execute_trade</code> I put natural-language examples like &quot;buy 100 USDT of BTC&quot; in the description, along with the parameters each one maps to.</p>
<p>If I started it today I'd change one thing: newer releases of the MCP TypeScript SDK recommend <code>registerTool()</code> over <code>server.tool()</code>. The server works fine on the 1.x SDK, but a new project should start with <code>registerTool()</code>.</p>

<h2 class="relative group">Setup
    <div id="setup" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#setup" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>First, turn on Freqtrade's REST API, which is off by default. In <code>config.json</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="s2">&#34;api_server&#34;</span><span class="err">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;enabled&#34;</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;listen_ip_address&#34;</span><span class="p">:</span> <span class="s2">&#34;127.0.0.1&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;listen_port&#34;</span><span class="p">:</span> <span class="mi">8080</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;verbosity&#34;</span><span class="p">:</span> <span class="s2">&#34;error&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;enable_openapi&#34;</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;jwt_secret_key&#34;</span><span class="p">:</span> <span class="s2">&#34;some-long-random-value&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;username&#34;</span><span class="p">:</span> <span class="s2">&#34;Freqtrader&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;password&#34;</span><span class="p">:</span> <span class="s2">&#34;YourStrongPassword&#34;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>Keep <code>listen_ip_address</code> at <code>127.0.0.1</code>. Exposing the API to the internet puts your bot and your exchange account behind a single password. The Freqtrade docs say the same:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/posts/freqtrade-mcp-server-tr/img/freqtrade-api-security-warning_hu_e3d5aaf45bda498.avif 480w, https://blog.furkankoykiran.com.tr/posts/freqtrade-mcp-server-tr/img/freqtrade-api-security-warning_hu_76b6fb1fec82dcd8.avif 800w, https://blog.furkankoykiran.com.tr/posts/freqtrade-mcp-server-tr/img/freqtrade-api-security-warning_hu_30d88752e30e9cda.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/posts/freqtrade-mcp-server-tr/img/freqtrade-api-security-warning_hu_c04f9184d2231b5e.webp 480w, https://blog.furkankoykiran.com.tr/posts/freqtrade-mcp-server-tr/img/freqtrade-api-security-warning_hu_ecab3187c24485f6.webp 800w, https://blog.furkankoykiran.com.tr/posts/freqtrade-mcp-server-tr/img/freqtrade-api-security-warning_hu_651f23b9b9275804.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/posts/freqtrade-mcp-server-tr/img/freqtrade-api-security-warning_hu_ecab3187c24485f6.webp" alt="Security warning from the Freqtrade docs: don&#39;t expose the API to the internet and pick a strong, unique password" width="1544" height="256"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Source: Freqtrade documentation, REST API page (quotation)</figcaption></figure><p>Then add the server to Claude Desktop's <code>claude_desktop_config.json</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;mcpServers&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;freqtrade&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;command&#34;</span><span class="p">:</span> <span class="s2">&#34;npx&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;args&#34;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&#34;-y&#34;</span><span class="p">,</span> <span class="s2">&#34;@furkankoykiran/freqtrade-mcp&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;env&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="nt">&#34;FREQTRADE_API_URL&#34;</span><span class="p">:</span> <span class="s2">&#34;http://127.0.0.1:8080&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="nt">&#34;FREQTRADE_USERNAME&#34;</span><span class="p">:</span> <span class="s2">&#34;Freqtrader&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="nt">&#34;FREQTRADE_PASSWORD&#34;</span><span class="p">:</span> <span class="s2">&#34;YourStrongPassword&#34;</span>
</span></span><span class="line"><span class="cl">      <span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>Restart Claude Desktop and the tools show up. With Claude Code it's one command:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">claude mcp add freqtrade <span class="se">\
</span></span></span><span class="line"><span class="cl">  -e <span class="nv">FREQTRADE_API_URL</span><span class="o">=</span>http://127.0.0.1:8080 <span class="se">\
</span></span></span><span class="line"><span class="cl">  -e <span class="nv">FREQTRADE_USERNAME</span><span class="o">=</span>Freqtrader <span class="se">\
</span></span></span><span class="line"><span class="cl">  -e <span class="nv">FREQTRADE_PASSWORD</span><span class="o">=</span>YourStrongPassword <span class="se">\
</span></span></span><span class="line"><span class="cl">  -- npx -y @furkankoykiran/freqtrade-mcp</span></span></code></pre></div></div>
<p>To poke at the tools before wiring them into a client, use the MCP Inspector: <code>npx @modelcontextprotocol/inspector npx -y @furkankoykiran/freqtrade-mcp</code>.</p>

<h2 class="relative group">Live trading caveats
    <div id="live-trading-caveats" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#live-trading-caveats" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p><code>execute_trade</code> and <code>force_exit_trade</code> skip the strategy and call Freqtrade's <code>/forceenter</code> and <code>/forceexit</code> endpoints directly. The server has no confirmation step.</p>
<div class="admonition relative overflow-hidden rounded-lg border-l-4 my-3 px-4 py-3 shadow-sm" data-type="warning">
      <div class="flex items-center gap-2 font-semibold text-inherit">
        <div class="flex shrink-0 h-5 w-5 items-center justify-center text-lg"><span class="relative block icon"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M506.3 417l-213.3-364c-16.33-28-57.54-28-73.98 0l-213.2 364C-10.59 444.9 9.849 480 42.74 480h426.6C502.1 480 522.6 445 506.3 417zM232 168c0-13.25 10.75-24 24-24S280 154.8 280 168v128c0 13.25-10.75 24-23.1 24S232 309.3 232 296V168zM256 416c-17.36 0-31.44-14.08-31.44-31.44c0-17.36 14.07-31.44 31.44-31.44s31.44 14.08 31.44 31.44C287.4 401.9 273.4 416 256 416z"/></svg>
</span></div>
        <div class="grow">
          Warning
        </div>
      </div><div class="admonition-content mt-3 text-base leading-relaxed text-inherit"><p>If the bot is live, these tools place a real order on a real exchange, and you can't take it back.</p></div></div><p>Things to know before you rely on it:</p>
<ul>
<li>Freqtrade only accepts <code>/forceenter</code> when the config has <code>force_entry_enable: true</code>. Without it, <code>execute_trade</code> returns an error. Turning that on should be a deliberate choice.</li>
<li><code>side: &quot;short&quot;</code> only makes sense for bots running in futures or margin mode. You can't short in spot mode.</li>
<li>Try everything with <code>dry_run: true</code> first. Freqtrade simulates the orders, so you can see how the tools behave without risk.</li>
<li>Claude Desktop and Claude Code ask for permission before tool calls. I never pick &quot;always allow&quot; for the tools that place orders.</li>
</ul>

<h2 class="relative group">Where the code is
    <div id="where-the-code-is" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-the-code-is" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>What I wanted was simple: check the bot without opening FreqUI. The REST API was already there; the MCP server moved it into the chat. I still wonder whether something changed while I wasn't looking, but now finding out takes one sentence.</p>
<p>The code is MIT licensed and <a href="https://github.com/furkankoykiran/freqtrade-mcp"  target="_blank" rel="noreferrer">on GitHub</a>, with a summary on my <a href="https://furkankoykiran.com.tr/en/projects/freqtrade-mcp/"  target="_blank" rel="noreferrer">project page</a>. Before you point it at your own bot, run it for a week with <code>dry_run: true</code> and compare what <code>get_profit_stats</code> and <code>get_open_trades</code> report with FreqUI. To see a different MCP integration, where I wrapped someone else's MCP server as a CLI, read <a href="https://blog.furkankoykiran.com.tr/en/posts/cli-anything-browser-automation-contribution/" >my CLI-Anything contribution</a>. If your Python services lean on async I/O the way trading bots do, the <a href="https://blog.furkankoykiran.com.tr/en/posts/python-asyncio-concurrent-processing/" >asyncio guide</a> is a good next read.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/freqtrade-mcp-server/feature_hu_d232e793ec37ecc.jpg" medium="image" type="image/jpeg"/></item><item><title>awesome-trending-repos: Tracking GitHub Trending Daily</title><link>https://blog.furkankoykiran.com.tr/en/posts/awesome-trending-repos-web/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/awesome-trending-repos-web/</guid><pubDate>Fri, 27 Mar 2026 18:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Projects</category><category>github-actions</category><category>automation</category><category>ci-cd</category><category>web</category><category>open-source</category><description>How my open source tracker scrapes GitHub Trending every day, compares it with past snapshots and publishes a React dashboard to GitHub Pages with Actions.</description><content:encoded><![CDATA[<p><a href="https://github.com/furkankoykiran/awesome-trending-repos"  target="_blank" rel="noreferrer">awesome-trending-repos</a> is a small project of mine that tracks GitHub Trending on its own. Every night at 00:00 UTC a GitHub Actions job scrapes the trending list, compares it with the previous day, commits the result as JSON and redeploys a React dashboard to GitHub Pages. It has run without me touching it since March 2026. This post walks through both versions of the project, the data pipeline, and the mistakes I made along the way.</p>

<h2 class="relative group">Why I built it
    <div id="why-i-built-it" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-i-built-it" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>I looked at the trending page most days but couldn't remember what was on it yesterday. Seeing which repos were new, which were climbing and which languages were hot meant keeping history somewhere. GitHub doesn't keep it for you, so I started keeping it myself.</p>

<h2 class="relative group">Version 1: a script that wrote the README
    <div id="version-1-a-script-that-wrote-the-readme" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#version-1-a-script-that-wrote-the-readme" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The March 17, 2026 release was plain Node.js. <code>scripts/update.js</code> scraped the trending page and wrote everything straight into <code>README.md</code>: language breakdowns, rank changes, ASCII charts, the fastest risers. The last 7 days of data lived in <code>.data/</code>.</p>
<p>It worked, but a README is a poor data UI. No filters, no search, and the charts were text. Ten days later I split the interface out.</p>

<h2 class="relative group">Version 2: JSON data, React front end
    <div id="version-2-json-data-react-front-end" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#version-2-json-data-react-front-end" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p><a href="https://github.com/furkankoykiran/awesome-trending-repos/pull/5"  target="_blank" rel="noreferrer">PR #5</a>, merged on March 27, split the project in two. The script no longer touches the README; it only writes <code>public/data/trending.json</code> and <code>public/data/feed.json</code>, and the README became a static landing page. The front end is a single-page app that reads that JSON:</p>
<ul>
<li>React 19 and Vite 6</li>
<li>Tailwind CSS v4 through the <code>@tailwindcss/vite</code> plugin</li>
<li>Framer Motion for transitions, Recharts for the language distribution chart</li>
<li>Language filter, instant search across name, owner and description, and a feed that lists each update</li>
</ul>
<p>Here is the top of the live site. The Trending, Stats and Feed tabs, the language chips and the search box all read the same JSON:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/posts/awesome-trending-repos-web/img/trending-dashboard_hu_2769a0020d38637.avif 480w, https://blog.furkankoykiran.com.tr/posts/awesome-trending-repos-web/img/trending-dashboard_hu_f7d53a046ffcb49b.avif 800w, https://blog.furkankoykiran.com.tr/posts/awesome-trending-repos-web/img/trending-dashboard_hu_da8a9e66f5dee22f.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/posts/awesome-trending-repos-web/img/trending-dashboard_hu_ba076c60816545e5.webp 480w, https://blog.furkankoykiran.com.tr/posts/awesome-trending-repos-web/img/trending-dashboard_hu_f834555f00aa20a9.webp 800w, https://blog.furkankoykiran.com.tr/posts/awesome-trending-repos-web/img/trending-dashboard_hu_556fdf35347f5807.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/posts/awesome-trending-repos-web/img/trending-dashboard_hu_f834555f00aa20a9.webp" alt="The live awesome-trending-repos site: Trending, Stats and Feed tabs, language filters, a search box and repo cards" width="1600" height="1125"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>The React front end at furkankoykiran.com.tr/awesome-trending-repos</figcaption></figure><p>With 25 repos, filtering doesn't need anything clever:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-javascript" data-lang="javascript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">q</span> <span class="o">=</span> <span class="nx">searchQuery</span><span class="p">.</span><span class="nx">toLowerCase</span><span class="p">();</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">filteredRepos</span> <span class="o">=</span> <span class="nx">repos</span><span class="p">.</span><span class="nx">filter</span><span class="p">((</span><span class="nx">repo</span><span class="p">)</span> <span class="p">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">matchesLang</span> <span class="o">=</span> <span class="nx">filter</span> <span class="o">===</span> <span class="s1">&#39;All&#39;</span> <span class="o">||</span> <span class="nx">repo</span><span class="p">.</span><span class="nx">language</span> <span class="o">===</span> <span class="nx">filter</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">matchesSearch</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">    <span class="nx">repo</span><span class="p">.</span><span class="nx">name</span><span class="p">.</span><span class="nx">toLowerCase</span><span class="p">().</span><span class="nx">includes</span><span class="p">(</span><span class="nx">q</span><span class="p">)</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">    <span class="nx">repo</span><span class="p">.</span><span class="nx">owner</span><span class="p">.</span><span class="nx">toLowerCase</span><span class="p">().</span><span class="nx">includes</span><span class="p">(</span><span class="nx">q</span><span class="p">)</span> <span class="o">||</span>
</span></span><span class="line"><span class="cl">    <span class="p">(</span><span class="nx">repo</span><span class="p">.</span><span class="nx">description</span> <span class="o">??</span> <span class="s1">&#39;&#39;</span><span class="p">).</span><span class="nx">toLowerCase</span><span class="p">().</span><span class="nx">includes</span><span class="p">(</span><span class="nx">q</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="k">return</span> <span class="nx">matchesLang</span> <span class="o">&amp;&amp;</span> <span class="nx">matchesSearch</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span></span></span></code></pre></div></div>

<h2 class="relative group">The data pipeline
    <div id="the-data-pipeline" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-data-pipeline" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The real work happens in <code>scripts/update.js</code>:</p>
<ol>
<li>It fetches <code>github.com/trending/&lt;language&gt;?since=daily</code> for 15 languages with axios and parses the HTML with cheerio. The all-languages page changed structure at some point, and going language by language turned out to be more reliable.</li>
<li>If that yields fewer than 25 repos, the GitHub Search API fills the gap.</li>
<li>Invalid entries and anything outside a list of popular languages are dropped.</li>
<li>If the list is still short, repos from the last 7 days of snapshots are added.</li>
<li>The result is compared with yesterday's snapshot: new entries, risers, fallers and a &quot;rising star&quot;.</li>
<li><code>.data/history.json</code> keeps at most 7 snapshots, and older snapshot files are deleted.</li>
</ol>
<p>Two numbers in there are guesses. The Search API doesn't tell you how many stars a repo got today, so for repos that come from search I estimate daily stars as 1% of total stars. Repos pulled back in from history get half of their old daily count. Both are rough fillers, not measurements. Flagging them in the UI is on my list.</p>

<h2 class="relative group">The GitHub Actions workflow
    <div id="the-github-actions-workflow" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-github-actions-workflow" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The workflow runs once a day on a cron, on manual dispatch, and on pushes that touch <code>src/</code> or <code>scripts/</code>. It updates the data, builds, commits any changes and deploys to Pages. Here is a trimmed version of the one in the repo:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Update and Deploy Trends</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">on</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">schedule</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">cron</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;0 0 * * *&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">workflow_dispatch</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">push</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">branches</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">main]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">paths</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s1">&#39;src/**&#39;</span><span class="p">,</span><span class="w"> </span><span class="s1">&#39;scripts/**&#39;</span><span class="p">,</span><span class="w"> </span><span class="s1">&#39;package.json&#39;</span><span class="p">,</span><span class="w"> </span><span class="s1">&#39;package-lock.json&#39;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">permissions</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">contents</span><span class="p">:</span><span class="w"> </span><span class="l">write</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">pages</span><span class="p">:</span><span class="w"> </span><span class="l">write</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">id-token</span><span class="p">:</span><span class="w"> </span><span class="l">write</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">concurrency</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span><span class="l">pages</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">cancel-in-progress</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">jobs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">update-and-build</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/checkout@v7</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/setup-node@v7</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">node-version</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;22&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">cache</span><span class="p">:</span><span class="w"> </span><span class="l">npm</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">npm ci</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">npm run update</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">env</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">GITHUB_TOKEN</span><span class="p">:</span><span class="w"> </span><span class="l">${{ secrets.GITHUB_TOKEN }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">npm run build</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Commit data changes</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          if [ -z &#34;$(git status --porcelain .data/ public/data/)&#34; ]; then exit 0; fi
</span></span></span><span class="line"><span class="cl"><span class="sd">          git config user.name &#39;github-actions[bot]&#39;
</span></span></span><span class="line"><span class="cl"><span class="sd">          git config user.email &#39;github-actions[bot]@users.noreply.github.com&#39;
</span></span></span><span class="line"><span class="cl"><span class="sd">          git add .data/ public/data/
</span></span></span><span class="line"><span class="cl"><span class="sd">          git commit -m &#34;chore: update trending repositories [skip ci]&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">          git push</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/upload-pages-artifact@v5</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">./dist</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">deploy</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">needs</span><span class="p">:</span><span class="w"> </span><span class="l">update-and-build</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">environment</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">github-pages</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">url</span><span class="p">:</span><span class="w"> </span><span class="l">${{ steps.deployment.outputs.page_url }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l">deployment</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/deploy-pages@v5</span></span></span></code></pre></div></div>
<p>The file in the repo still pins <code>checkout@v4</code>, <code>setup-node@v4</code>, <code>upload-pages-artifact@v3</code> and Node 20, which reached end of life in April 2026. If you're starting fresh, use the versions above.</p>
<p>A few details that pull their weight:</p>
<ul>
<li><code>[skip ci]</code> keeps the bot's commit from kicking off another run. Pushes made with <code>GITHUB_TOKEN</code> don't start new workflow runs anyway, but the marker costs nothing.</li>
<li><code>concurrency</code> stops two deploys from racing each other.</li>
<li>The build runs after the update. Vite copies <code>public/</code> into <code>dist/</code>, so the site always ships with that day's JSON.</li>
</ul>
<p>My commit step above adds the whole <code>public/data/</code> folder. The workflow in the repo only commits <code>public/data/trending.json</code> and leaves <code>feed.json</code> out. So each run reads the stale feed from the repo and appends one entry, and the live feed always shows the March 27 entry plus today's. The fix is the one line shown above.</p>

<h2 class="relative group">Vite on GitHub Pages: the base path
    <div id="vite-on-github-pages-the-base-path" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#vite-on-github-pages-the-base-path" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The site is a project page served under <code>/awesome-trending-repos/</code>. If Vite doesn't know that, the built asset URLs point at the domain root and everything 404s in production:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-javascript" data-lang="javascript"><span class="line"><span class="cl"><span class="c1">// vite.config.js
</span></span></span><span class="line"><span class="cl"><span class="kr">import</span> <span class="p">{</span> <span class="nx">defineConfig</span> <span class="p">}</span> <span class="nx">from</span> <span class="s1">&#39;vite&#39;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">import</span> <span class="nx">react</span> <span class="nx">from</span> <span class="s1">&#39;@vitejs/plugin-react&#39;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">import</span> <span class="nx">tailwindcss</span> <span class="nx">from</span> <span class="s1">&#39;@tailwindcss/vite&#39;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">export</span> <span class="k">default</span> <span class="nx">defineConfig</span><span class="p">({</span>
</span></span><span class="line"><span class="cl">  <span class="nx">plugins</span><span class="o">:</span> <span class="p">[</span><span class="nx">react</span><span class="p">(),</span> <span class="nx">tailwindcss</span><span class="p">()],</span>
</span></span><span class="line"><span class="cl">  <span class="nx">base</span><span class="o">:</span> <span class="s1">&#39;/awesome-trending-repos/&#39;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span></span></span></code></pre></div></div>
<p>Vite already adds content hashes to file names in production builds, so you don't need extra <code>rollupOptions</code> for cache busting. The app loads its data with <code>fetch('./data/trending.json')</code>, a relative path that works with any <code>base</code>.</p>

<h2 class="relative group">The custom domain mistake
    <div id="the-custom-domain-mistake" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-custom-domain-mistake" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Putting the root domain in a project's <code>public/CNAME</code> is a common mistake, and this repo had exactly that file. <code>furkankoykiran.com.tr</code> belongs to my user site (<code>furkankoykiran.github.io</code>). Project pages inherit that domain automatically and are served at <code>furkankoykiran.com.tr/awesome-trending-repos/</code>. A project repo with the root domain in its own CNAME is trying to claim the same domain as the user site.</p>
<p><a href="https://github.com/furkankoykiran/awesome-trending-repos/pull/7"  target="_blank" rel="noreferrer">PR #7</a> removed <code>public/CNAME</code> and <code>dist/CNAME</code>. With <code>actions/deploy-pages</code>, a CNAME file inside the artifact is ignored anyway; the domain comes from Settings → Pages → Custom domain, and for a project page that field should stay empty.</p>

<h2 class="relative group">What's next
    <div id="whats-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#whats-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The project still runs every day. Next on the list: fix the feed bug, mark estimated star counts in the UI, and bump the workflow's action versions.</p>
<p>The live site is at <a href="https://furkankoykiran.com.tr/awesome-trending-repos/"  target="_blank" rel="noreferrer">furkankoykiran.com.tr/awesome-trending-repos</a>, and there's a summary on the <a href="https://furkankoykiran.com.tr/en/projects/awesome-trending-repos/"  target="_blank" rel="noreferrer">project page</a>. If your Actions minutes are running low, my <a href="https://blog.furkankoykiran.com.tr/en/posts/github-actions-self-hosted-runner/" >self-hosted runner post</a> shows how I moved CI onto my own server. The story of how this blog ended up on GitHub Pages is in <a href="https://blog.furkankoykiran.com.tr/en/posts/gitlab-github-jekyll-migration/" >my GitLab and GitHub migration post</a>.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/awesome-trending-repos-web/feature_hu_2595d34818f8825c.jpg" medium="image" type="image/jpeg"/></item><item><title>CLI-Anything Browser Harness: My First MCP Backend PR</title><link>https://blog.furkankoykiran.com.tr/en/posts/cli-anything-browser-automation-contribution/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/cli-anything-browser-automation-contribution/</guid><pubDate>Sun, 22 Mar 2026 21:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Open Source</category><category>mcp</category><category>ai-agents</category><category>python</category><category>open-source</category><category>automation</category><description>How PR #118 added browser automation to CLI-Anything through the DOMShell MCP server: architecture, review, bugs found after merge, and a security fix.</description><content:encoded><![CDATA[<p>In March 2026 I sent a pull request that added browser automation to <a href="https://github.com/HKUDS/CLI-Anything"  target="_blank" rel="noreferrer">CLI-Anything</a>. <a href="https://github.com/HKUDS/CLI-Anything/pull/118"  target="_blank" rel="noreferrer">PR #118</a> became the project's first harness that talks to an MCP server instead of calling an application directly. This post covers how the harness is built, what came up in review, and the bugs that only showed up after it was merged.</p>
<p>CLI-Anything's premise is that if you wrap software in a clean command-line interface, AI agents can drive it. The repo has &quot;harnesses&quot; for GIMP, Blender, LibreOffice and dozens of other tools. Browsers were missing.</p>

<h2 class="relative group">Issue #90: browsing a page like a filesystem
    <div id="issue-90-browsing-a-page-like-a-filesystem" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#issue-90-browsing-a-page-like-a-filesystem" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>On March 16, <a href="https://github.com/apireno"  target="_blank" rel="noreferrer">@apireno</a>, the author of <a href="https://github.com/apireno/DOMShell"  target="_blank" rel="noreferrer">DOMShell</a>, opened <a href="https://github.com/HKUDS/CLI-Anything/issues/90"  target="_blank" rel="noreferrer">issue #90</a>. DOMShell is a Chrome extension plus MCP server that exposes Chrome's Accessibility Tree as a virtual filesystem, so an agent moves through a page with <code>ls</code>, <code>cd</code>, <code>grep</code> and <code>click</code>. In apireno's benchmarks with Claude, this cut API calls by half compared with screenshot-based browsing. That number is his; I didn't measure it myself.</p>
<p>The Accessibility Tree is the simplified tree the browser derives from the DOM for screen readers. Each node has a role such as button, link or textbox, and the CSS classes and wrapper <code>div</code>s are gone. That makes it less brittle than CSS selectors when a site's markup changes, and the page → section → element hierarchy maps cleanly onto directories.</p>
<p>In the issue thread, <a href="https://github.com/omerarslan0"  target="_blank" rel="noreferrer">@omerarslan0</a> sketched an integration: DOMShell's MCP server as the backend, command groups like <code>page</code> on the CLI side, and state (URL, working directory) kept by the CLI. apireno asked to keep the filesystem-style commands rather than switch to selectors, and recommended the official <code>mcp</code> Python SDK over stdio. I built on that discussion and opened the PR on March 21.</p>

<h2 class="relative group">How the harness is built
    <div id="how-the-harness-is-built" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#how-the-harness-is-built" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Most harnesses shell out to an application or call its API. This one puts an MCP client in the middle:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/cli-anything-browser-automation-contribution/img/harness-steps_hu_89384f06c32ef360.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/cli-anything-browser-automation-contribution/img/harness-steps_hu_7c6bd5e976d0132c.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/cli-anything-browser-automation-contribution/img/harness-steps_hu_d5e752ffb79f546d.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/cli-anything-browser-automation-contribution/img/harness-steps_hu_33e1ee5fe889b759.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/cli-anything-browser-automation-contribution/img/harness-steps_hu_59a742136c0ec627.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/cli-anything-browser-automation-contribution/img/harness-steps_hu_f950ee9e6b50a965.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/cli-anything-browser-automation-contribution/img/harness-steps_hu_59a742136c0ec627.webp" alt="Click commands call domshell_backend.py, an MCP client that talks over stdio to DOMShell, which exposes Chrome&#39;s Accessibility Tree as a filesystem" width="1600" height="1112"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>From command to Chrome, with an MCP client in between</figcaption></figure><p>There are four command groups:</p>
<ul>
<li><code>page</code>: open, reload, back, forward, info</li>
<li><code>fs</code>: ls, cd, cat, grep, pwd</li>
<li><code>act</code>: click, type</li>
<li><code>session</code>: status, daemon-start, daemon-stop</li>
</ul>
<p>In use:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">cli-anything-browser page open https://example.com
</span></span><span class="line"><span class="cl">cli-anything-browser fs ls /
</span></span><span class="line"><span class="cl">cli-anything-browser fs grep <span class="s2">&#34;Login&#34;</span>
</span></span><span class="line"><span class="cl">cli-anything-browser act click /main/button<span class="o">[</span>0<span class="o">]</span>
</span></span><span class="line"><span class="cl">cli-anything-browser --json fs ls /</span></span></code></pre></div></div>
<p>Every one-shot command starts a fresh session. Stateful flows, like <code>cd</code> followed by a bare <code>ls</code>, belong in the REPL, which opens when you run <code>cli-anything-browser</code> with no arguments.</p>

<h3 class="relative group">The MCP call in Python
    <div id="the-mcp-call-in-python" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-mcp-call-in-python" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>When I wrote the PR, DOMShell exposed one MCP tool per command (<code>domshell_ls</code>, <code>domshell_click</code> and so on). DOMShell 2.0 collapsed those into a single <code>domshell_execute</code> tool, and the harness was updated for it. A minimal client for DOMShell 2.x looks like this:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">asyncio</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">os</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">mcp</span> <span class="kn">import</span> <span class="n">ClientSession</span><span class="p">,</span> <span class="n">StdioServerParameters</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">mcp.client.stdio</span> <span class="kn">import</span> <span class="n">stdio_client</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">params</span> <span class="o">=</span> <span class="n">StdioServerParameters</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">command</span><span class="o">=</span><span class="s2">&#34;npx&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">args</span><span class="o">=</span><span class="p">[</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;-p&#34;</span><span class="p">,</span> <span class="s2">&#34;@apireno/domshell&#34;</span><span class="p">,</span> <span class="s2">&#34;domshell-proxy&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;--port&#34;</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;DOMSHELL_PORT&#34;</span><span class="p">,</span> <span class="s2">&#34;3001&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;--token&#34;</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">&#34;DOMSHELL_TOKEN&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">    <span class="p">],</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">main</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">with</span> <span class="n">stdio_client</span><span class="p">(</span><span class="n">params</span><span class="p">)</span> <span class="k">as</span> <span class="p">(</span><span class="n">read</span><span class="p">,</span> <span class="n">write</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">async</span> <span class="k">with</span> <span class="n">ClientSession</span><span class="p">(</span><span class="n">read</span><span class="p">,</span> <span class="n">write</span><span class="p">)</span> <span class="k">as</span> <span class="n">session</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">await</span> <span class="n">session</span><span class="o">.</span><span class="n">initialize</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">            <span class="n">result</span> <span class="o">=</span> <span class="k">await</span> <span class="n">session</span><span class="o">.</span><span class="n">call_tool</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;domshell_execute&#34;</span><span class="p">,</span> <span class="p">{</span><span class="s2">&#34;command&#34;</span><span class="p">:</span> <span class="s2">&#34;ls&#34;</span><span class="p">,</span> <span class="s2">&#34;group_id&#34;</span><span class="p">:</span> <span class="s2">&#34;new&#34;</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">            <span class="p">)</span>
</span></span><span class="line"><span class="cl">            <span class="nb">print</span><span class="p">(</span><span class="n">result</span><span class="o">.</span><span class="n">content</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">asyncio</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">main</span><span class="p">())</span></span></span></code></pre></div></div>
<p><code>domshell-proxy</code> is a stdio bridge to a DOMShell server that is already running, and <code>DOMSHELL_TOKEN</code> is the token that server prints at startup. <code>group_id: &quot;new&quot;</code> asks DOMShell 2.x for an isolated tab group for the agent. None of this was in my original PR, and the section after the review explains why.</p>

<h3 class="relative group">A sync CLI on top of an async SDK
    <div id="a-sync-cli-on-top-of-an-async-sdk" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#a-sync-cli-on-top-of-an-async-sdk" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>The MCP Python SDK is fully async and Click commands are sync, so each command opens a short session inside <code>asyncio.run()</code> and closes it. The cost is spawning an <code>npx</code> subprocess on every command. To avoid that during interactive use I added a daemon mode: while the REPL runs, one MCP connection stays open and every command reuses it. It only works inside a single process. Separate CLI invocations don't share it, and the README says so.</p>

<h2 class="relative group">Tests
    <div id="tests" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#tests" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Two layers. The 31 unit tests in <code>test_core.py</code> mock MCP responses and cover path resolution and session state, with no Chrome needed. The end-to-end tests in <code>test_full_e2e.py</code> drive a real Chrome with the DOMShell extension and only run when you ask for them:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">pytest cli_anything/browser/tests/test_core.py -v
</span></span><span class="line"><span class="cl"><span class="nv">DOMSHELL_E2E</span><span class="o">=</span><span class="m">1</span> pytest cli_anything/browser/tests/test_full_e2e.py -v</span></span></code></pre></div></div>
<p>The <code>DOMSHELL_E2E</code> gate came out of a Copilot comment. Before it, pytest could try to run <code>npx</code> during test collection.</p>

<h2 class="relative group">Review
    <div id="review" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#review" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Copilot reviewed the PR twice, with 10 comments and then 9. The ones that mattered most:</p>
<ul>
<li>A context manager leak in daemon mode. <code>_start_daemon()</code> called <code>__aenter__</code> on <code>stdio_client(...)</code> without keeping a reference, and <code>_stop_daemon()</code> called <code>__aexit__</code> on a brand-new client, so the original subprocess never shut down. I now keep the reference in a module-level <code>_daemon_client_context</code>.</li>
<li><code>page back</code> and <code>page forward</code> replayed navigation from a local history list, which is always empty in one-shot mode. They now call DOMShell's own back and forward tools.</li>
<li>The REPL split quoted arguments. <code>act type /path &quot;Hello, World!&quot;</code> now goes through <code>shlex.split()</code>.</li>
<li><code>act type</code> echoed the typed text, which leaves passwords in terminal scrollback. It no longer does.</li>
<li><code>fs cd</code> only updates the working directory when the backend reports success.</li>
</ul>
<p>omerarslan0 listed 8 issues in his first review, all of which earlier commits had already fixed. I replied with the commit for each one, and he checked and approved. <a href="https://github.com/yuh-yang"  target="_blank" rel="noreferrer">@yuh-yang</a> asked me to fix the contributor field in <code>registry.json</code>, then approved and merged on March 22. Final size: 15 commits, 21 files, 3,095 lines added:</p>
<p><figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/posts/cli-anything-browser-automation-katkim/img/cli-anything-pr-118-header_hu_4eab2511cd4c9292.avif 480w, https://blog.furkankoykiran.com.tr/posts/cli-anything-browser-automation-katkim/img/cli-anything-pr-118-header_hu_621c8c758db7b92a.avif 800w, https://blog.furkankoykiran.com.tr/posts/cli-anything-browser-automation-katkim/img/cli-anything-pr-118-header_hu_98b6ea5ff08f7dc4.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/posts/cli-anything-browser-automation-katkim/img/cli-anything-pr-118-header_hu_5c58a899edecbb8.webp 480w, https://blog.furkankoykiran.com.tr/posts/cli-anything-browser-automation-katkim/img/cli-anything-pr-118-header_hu_7496048ed2b970d2.webp 800w, https://blog.furkankoykiran.com.tr/posts/cli-anything-browser-automation-katkim/img/cli-anything-pr-118-header_hu_54c467404ed0def8.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/posts/cli-anything-browser-automation-katkim/img/cli-anything-pr-118-header_hu_7496048ed2b970d2.webp" alt="Header of CLI-Anything PR #118: merged by yuh-yang, 15 commits, &#43;3,095 −1" width="1600" height="200"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>CLI-Anything PR #118. Source: GitHub</figcaption></figure> I also added an &quot;MCP Backend Pattern&quot; section to <code>cli-anything-plugin/HARNESS.md</code> so the next MCP integration has something to follow.</p>

<h2 class="relative group">What broke after the merge
    <div id="what-broke-after-the-merge" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-broke-after-the-merge" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>I learned the most from this PR after it was merged. Right after the merge, apireno tested the harness against a real DOMShell server. All 31 unit tests passed and 9 of 10 end-to-end tests passed, but several commands failed when he used them by hand:</p>
<ol>
<li>In one-shot mode every command started a new <code>npx @apireno/domshell</code> process, which tried to bind ports 9876 and 3001. If another client such as Claude Desktop was already running DOMShell, every command failed.</li>
<li>I was sending the wrong parameter names to some tools (<code>ls</code>, <code>cat</code>, <code>click</code>).</li>
<li>DOMShell didn't support <code>--version</code>, which my availability check relied on. apireno fixed that on the DOMShell side.</li>
</ol>
<p>apireno fixed the port and parameter problems in <a href="https://github.com/HKUDS/CLI-Anything/pull/135"  target="_blank" rel="noreferrer">PR #135</a>, merged on March 24, which moved the connection to the <code>domshell-proxy</code> bridge with token auth. My unit tests passed because they mocked a server interface that didn't match the real one. Pulling the tool schemas from the server with <code>list_tools</code> and building the mocks from them would have caught this before review.</p>

<h2 class="relative group">Security hardening in PR #156
    <div id="security-hardening-in-pr-156" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#security-hardening-in-pr-156" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>After I shared the contribution on LinkedIn, Ilhan Goktas pointed out several attack surfaces: SSRF through arbitrary navigation, prompt injection hidden in ARIA labels, and escaping the virtual filesystem. I followed up with <a href="https://github.com/HKUDS/CLI-Anything/pull/156"  target="_blank" rel="noreferrer">PR #156</a>, which omerarslan0 merged on April 7.</p>
<p>It added:</p>
<ul>
<li>URL validation that blocks <code>file://</code>, <code>javascript:</code>, <code>data:</code> and <code>vbscript:</code> schemes</li>
<li>Optional blocking of private network addresses, switched on with an environment variable</li>
<li>A basic sanitizer for known prompt injection patterns in page content</li>
<li>42 security tests, plus security notes for agents in <code>SKILL.md</code></li>
</ul>
<p>The sanitizer catches obvious patterns and nothing more. If you give an agent a browser, treat every piece of page text as untrusted input.</p>

<h2 class="relative group">Where the code went next
    <div id="where-the-code-went-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-the-code-went-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>I wrote the harness, and most of the code that runs now has been through other hands since. apireno did the DOMShell 2.0 migration and the fixes after it, and other contributors followed. That's normal for open source, and it's a good reason to write code that's easy for the next person to change.</p>
<p>What I took away from this PR:</p>
<ul>
<li>When you wrap an MCP server, run at least one end-to-end test against the real server before asking for a merge.</li>
<li>When you bridge an async SDK into a sync CLI, keep the connection lifecycle in one place. That's where the leaks come from. I go deeper into the asyncio side in my <a href="https://blog.furkankoykiran.com.tr/en/posts/python-asyncio-concurrent-processing/" >asyncio guide</a>.</li>
<li>Answer review comments with the commit hash for each fix. It makes the reviewer's job much easier.</li>
</ul>

<h2 class="relative group">Where to go next
    <div id="where-to-go-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-to-go-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The code is in <a href="https://github.com/HKUDS/CLI-Anything/tree/main/browser/agent-harness"  target="_blank" rel="noreferrer">browser/agent-harness</a>. For an MCP server I built from scratch, read the <a href="https://blog.furkankoykiran.com.tr/en/posts/freqtrade-mcp-server/" >freqtrade-mcp post</a>. For another open source contribution, with a very different codebase, see <a href="https://blog.furkankoykiran.com.tr/en/posts/my-open-source-contribution-keploy/" >my Keploy contribution</a>.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/cli-anything-browser-automation-contribution/feature_hu_5f137ff1dbb71eb1.jpg" medium="image" type="image/jpeg"/></item><item><title>Keploy Contribution: Go defer and HTTP Body Leaks</title><link>https://blog.furkankoykiran.com.tr/en/posts/my-open-source-contribution-keploy/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/my-open-source-contribution-keploy/</guid><pubDate>Fri, 20 Mar 2026 18:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Open Source</category><category>go</category><category>open-source</category><category>testing</category><category>api</category><description>Two merged PRs to Keploy: a function that read from a closed file and HTTP response bodies that were never closed. What they taught me about defer in Go.</description><content:encoded><![CDATA[<p>While following GitHub trending with <a href="https://github.com/furkankoykiran/awesome-trending-repos"  target="_blank" rel="noreferrer">awesome-trending-repos</a>, I came across <a href="https://github.com/keploy/keploy"  target="_blank" rel="noreferrer">Keploy</a>, an API testing tool written in Go. Its issue tracker had two resource management bugs, and I sent a PR for each. This post covers that Keploy contribution: using <code>defer</code> correctly in Go, and why an HTTP response body needs to be both drained and closed. Both PRs were merged on March 20, 2026. They were my first contributions to Keploy.</p>

<h2 class="relative group">What Keploy does
    <div id="what-keploy-does" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-keploy-does" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Keploy captures your application's real traffic with eBPF and turns it into API tests and mocks for external dependencies. When the tests run again, database and third-party calls are answered from those mocks.</p>
<p>The two bugs I fixed sit at the edges of this flow: one where Keploy detects the application's language, the other in the HTTP client Keploy uses to talk to its own agent.</p>

<h2 class="relative group">PR #3927: reading from a closed file
    <div id="pr-3927-reading-from-a-closed-file" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#pr-3927-reading-from-a-closed-file" aria-label="Anchor">#</a>
    </span>
    
</h2>

<h3 class="relative group">The bug
    <div id="the-bug" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-bug" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Issue <a href="https://github.com/keploy/keploy/issues/3821"  target="_blank" rel="noreferrer">#3821</a> pointed at <code>isGoBinary</code> in <code>utils/utils.go</code>. The function opened an ELF file, closed it right away, and then tried to read sections from the closed file:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">f</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">elf</span><span class="p">.</span><span class="nf">Open</span><span class="p">(</span><span class="nx">filePath</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="nx">logger</span><span class="p">.</span><span class="nf">Debug</span><span class="p">(</span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Sprintf</span><span class="p">(</span><span class="s">&#34;failed to open file %s&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">filePath</span><span class="p">),</span><span class="w"> </span><span class="nx">zap</span><span class="p">.</span><span class="nf">Error</span><span class="p">(</span><span class="nx">err</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="k">return</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">f</span><span class="p">.</span><span class="nf">Close</span><span class="p">();</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="c1">// the file is closed here</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="nf">LogError</span><span class="p">(</span><span class="nx">logger</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;failed to close file&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">zap</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="s">&#34;file&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">filePath</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">sections</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">&#34;.go.buildinfo&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;.gopclntab&#34;</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">section</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">sections</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="k">if</span><span class="w"> </span><span class="nx">sect</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">f</span><span class="p">.</span><span class="nf">Section</span><span class="p">(</span><span class="nx">section</span><span class="p">);</span><span class="w"> </span><span class="nx">sect</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="c1">// ...and used here</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="nx">section</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="k">return</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p><code>DetectLanguage</code> relies on <code>isGoBinary</code>, so the bug sat right where Keploy decides whether an application is written in Go.</p>

<h3 class="relative group">The fix
    <div id="the-fix" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-fix" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>I moved <code>Close()</code> into a <code>defer</code>. A deferred call runs when the function returns, whichever way it exits: an early <code>return</code>, the normal path, or a panic. I also removed a leftover debug <code>fmt.Println(section)</code>.</p>
<p>The PR's first commit used a plain <code>defer f.Close()</code>. Copilot's automated review pointed out that this swallows the error from <code>Close()</code>, while the rest of the file logs close errors, and that it would probably trip <code>golangci-lint</code>'s <code>errcheck</code>. The final version matches the other call sites:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">f</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">elf</span><span class="p">.</span><span class="nf">Open</span><span class="p">(</span><span class="nx">filePath</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="nx">logger</span><span class="p">.</span><span class="nf">Debug</span><span class="p">(</span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Sprintf</span><span class="p">(</span><span class="s">&#34;failed to open file %s&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">filePath</span><span class="p">),</span><span class="w"> </span><span class="nx">zap</span><span class="p">.</span><span class="nf">Error</span><span class="p">(</span><span class="nx">err</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="k">return</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">defer</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">f</span><span class="p">.</span><span class="nf">Close</span><span class="p">();</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="nf">LogError</span><span class="p">(</span><span class="nx">logger</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;failed to close file&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">zap</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="s">&#34;file&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">filePath</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">sections</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">&#34;.go.buildinfo&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;.gopclntab&#34;</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">section</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">sections</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="k">if</span><span class="w"> </span><span class="nx">sect</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">f</span><span class="p">.</span><span class="nf">Section</span><span class="p">(</span><span class="nx">section</span><span class="p">);</span><span class="w"> </span><span class="nx">sect</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="k">return</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">return</span><span class="w"> </span><span class="kc">false</span></span></span></code></pre></div></div>
<p><a href="https://github.com/keploy/keploy/pull/3927"  target="_blank" rel="noreferrer">PR #3927</a> was a +5/−4 change. AkashKumar7902 merged it on March 20.</p>

<h2 class="relative group">PR #3932: response bodies that were never closed
    <div id="pr-3932-response-bodies-that-were-never-closed" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#pr-3932-response-bodies-that-were-never-closed" aria-label="Anchor">#</a>
    </span>
    
</h2>

<h3 class="relative group">The bug
    <div id="the-bug-1" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-bug-1" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Issue <a href="https://github.com/keploy/keploy/issues/3854"  target="_blank" rel="noreferrer">#3854</a> pointed at <code>MockOutgoing</code> and <code>UpdateMockParams</code> in <code>pkg/platform/http/agent.go</code>. Both sent a request to the agent and never closed <code>res.Body</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">res</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">a</span><span class="p">.</span><span class="nx">client</span><span class="p">.</span><span class="nf">Do</span><span class="p">(</span><span class="nx">req</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="k">return</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;failed to send request for mockOutgoing: %s&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">.</span><span class="nf">Error</span><span class="p">())</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">// res.Body is never closed</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">var</span><span class="w"> </span><span class="nx">mockResp</span><span class="w"> </span><span class="nx">models</span><span class="p">.</span><span class="nx">AgentResp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">json</span><span class="p">.</span><span class="nf">NewDecoder</span><span class="p">(</span><span class="nx">res</span><span class="p">.</span><span class="nx">Body</span><span class="p">).</span><span class="nf">Decode</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">mockResp</span><span class="p">)</span></span></span></code></pre></div></div>
<p>The <code>net/http</code> docs are explicit: the caller must close the body. An unclosed body holds on to its TCP connection, so the connection can't go back to the pool and every request opens a new one. In a long-running process that can end in exhausted file descriptors or ports.</p>

<h3 class="relative group">The fix: drain, then close
    <div id="the-fix-drain-then-close" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-fix-drain-then-close" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Side by side, the whole change is one deferred function:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/my-open-source-contribution-keploy/img/keploy-body-close-fix_hu_2e3cb57d7cfeb0c3.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/my-open-source-contribution-keploy/img/keploy-body-close-fix_hu_c56e2c53d854f6a1.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/my-open-source-contribution-keploy/img/keploy-body-close-fix_hu_66ee3d61d9dbae49.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/my-open-source-contribution-keploy/img/keploy-body-close-fix_hu_c5a43c2e438b4a4.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/my-open-source-contribution-keploy/img/keploy-body-close-fix_hu_26ca08f65737db14.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/my-open-source-contribution-keploy/img/keploy-body-close-fix_hu_1c75148fc89f890.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/my-open-source-contribution-keploy/img/keploy-body-close-fix_hu_26ca08f65737db14.webp" alt="Before: res.Body is never closed. After: a deferred function drains the body with io.Copy(io.Discard, res.Body), then closes it" width="1600" height="1048"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>The fix from PR #3932, shortened</figcaption></figure><div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">res</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">a</span><span class="p">.</span><span class="nx">client</span><span class="p">.</span><span class="nf">Do</span><span class="p">(</span><span class="nx">req</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="k">return</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;failed to send request for mockOutgoing: %s&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">.</span><span class="nf">Error</span><span class="p">())</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">defer</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="nx">io</span><span class="p">.</span><span class="nf">Copy</span><span class="p">(</span><span class="nx">io</span><span class="p">.</span><span class="nx">Discard</span><span class="p">,</span><span class="w"> </span><span class="nx">res</span><span class="p">.</span><span class="nx">Body</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">res</span><span class="p">.</span><span class="nx">Body</span><span class="p">.</span><span class="nf">Close</span><span class="p">();</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="nx">utils</span><span class="p">.</span><span class="nf">LogError</span><span class="p">(</span><span class="nx">a</span><span class="p">.</span><span class="nx">logger</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;failed to close response body for mockOutgoing&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">var</span><span class="w"> </span><span class="nx">mockResp</span><span class="w"> </span><span class="nx">models</span><span class="p">.</span><span class="nx">AgentResp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">json</span><span class="p">.</span><span class="nf">NewDecoder</span><span class="p">(</span><span class="nx">res</span><span class="p">.</span><span class="nx">Body</span><span class="p">).</span><span class="nf">Decode</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">mockResp</span><span class="p">)</span></span></span></code></pre></div></div>
<p>The <code>io.Copy(io.Discard, res.Body)</code> line was the most useful thing I learned from this PR. The maintainer, AkashKumar7902, asked me to address the Copilot comments, and one of them explained it: <code>json.Decoder.Decode</code> doesn't promise to read the body to the end. If the body is closed before it reaches EOF, the <code>Transport</code> can't reuse the connection. Draining it first is what makes keep-alive actually work.</p>
<p>One caveat: if the body can be large, an unbounded <code>io.Copy</code> may pull megabytes you don't need. In that case, cap the drain:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">io</span><span class="p">.</span><span class="nf">Copy</span><span class="p">(</span><span class="nx">io</span><span class="p">.</span><span class="nx">Discard</span><span class="p">,</span><span class="w"> </span><span class="nx">io</span><span class="p">.</span><span class="nf">LimitReader</span><span class="p">(</span><span class="nx">res</span><span class="p">.</span><span class="nx">Body</span><span class="p">,</span><span class="w"> </span><span class="mi">64</span><span class="o">&lt;&lt;</span><span class="mi">10</span><span class="p">))</span><span class="w"> </span><span class="c1">// at most 64 KiB</span></span></span></code></pre></div></div>
<p>Keploy's agent responses are small, so the unbounded version was fine there. <a href="https://github.com/keploy/keploy/pull/3932"  target="_blank" rel="noreferrer">PR #3932</a> was merged with two commits and 12 added lines.</p>

<h2 class="relative group">Rules I now follow with defer
    <div id="rules-i-now-follow-with-defer" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#rules-i-now-follow-with-defer" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Since these two PRs I stick to a few rules with <code>defer</code>.</p>
<p>Put the <code>defer</code> right after you acquire the resource and after the error check. If <code>os.Open</code> fails, <code>f</code> is nil, and a <code>defer f.Close()</code> placed before the check will panic.</p>
<p>Don't <code>defer</code> inside a loop. Deferred calls run when the function returns, not at the end of each iteration, so thousands of files can stay open. Move the loop body into its own function:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">countLines</span><span class="p">(</span><span class="nx">paths</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="nx">total</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="mi">0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">p</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">paths</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="nx">n</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">countFile</span><span class="p">(</span><span class="nx">p</span><span class="p">)</span><span class="w"> </span><span class="c1">// the defer runs when countFile returns</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">			</span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="nx">total</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="nx">n</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="k">return</span><span class="w"> </span><span class="nx">total</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">countFile</span><span class="p">(</span><span class="nx">path</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="nx">f</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nf">Open</span><span class="p">(</span><span class="nx">path</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="k">defer</span><span class="w"> </span><span class="nx">f</span><span class="p">.</span><span class="nf">Close</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="nx">n</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="mi">0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="nx">scanner</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">bufio</span><span class="p">.</span><span class="nf">NewScanner</span><span class="p">(</span><span class="nx">f</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="k">for</span><span class="w"> </span><span class="nx">scanner</span><span class="p">.</span><span class="nf">Scan</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="nx">n</span><span class="o">++</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="k">return</span><span class="w"> </span><span class="nx">n</span><span class="p">,</span><span class="w"> </span><span class="nx">scanner</span><span class="p">.</span><span class="nf">Err</span><span class="p">()</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>Multiple defers run in reverse order (LIFO). Their arguments are evaluated when the <code>defer</code> statement runs, not when the function returns, so use a closure if you need a variable's final value.</p>
<p>For a file you only read, the <code>Close()</code> error rarely matters. For a file you write, it does: that's where you learn buffered data didn't make it to disk. In that case, return the error from the function instead of only logging it.</p>

<h2 class="relative group">Where to go next
    <div id="where-to-go-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-to-go-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The two PRs added 17 lines in total, and I only learned the body-draining detail from the review comments. In your own Go code, grep for <code>client.Do</code> and check that every response body is drained and closed in a <code>defer</code>. For more, see the Go docs for <a href="https://pkg.go.dev/net/http#Client.Do"  target="_blank" rel="noreferrer">net/http Client.Do</a> and the <a href="https://go.dev/wiki/CommonMistakes"  target="_blank" rel="noreferrer">CommonMistakes page on the Go wiki</a>.</p>
<p>The tool I found Keploy with, <a href="https://furkankoykiran.com.tr/en/projects/awesome-trending-repos/"  target="_blank" rel="noreferrer">awesome-trending-repos</a>, has <a href="https://blog.furkankoykiran.com.tr/en/posts/awesome-trending-repos-web/" >its own post</a>. My other open source work is in <a href="https://blog.furkankoykiran.com.tr/en/posts/mcp-ai-open-source-contributions/" >MCP open source contributions</a> and <a href="https://blog.furkankoykiran.com.tr/en/posts/security-and-api-fixes/" >security and API fixes</a>.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/my-open-source-contribution-keploy/feature_hu_44b12ba836832d1e.jpg" medium="image" type="image/jpeg"/></item><item><title>SDK and CLI PRs: Jules, NotebookLM, Mission Control</title><link>https://blog.furkankoykiran.com.tr/en/posts/python-sdk-and-cli-contributions/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/python-sdk-and-cli-contributions/</guid><pubDate>Wed, 11 Mar 2026 18:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Open Source</category><category>python</category><category>typescript</category><category>open-source</category><category>tooling</category><category>ai-agents</category><category>asyncio</category><description>Four PRs to Jules SDK, notebooklm-py, cli-agent and Mission Control: two merged, two closed. The code, the design choices, and why each one ended as it did.</description><content:encoded><![CDATA[<p>In the first two weeks of March 2026 I opened four PRs against the SDK and CLI layers of AI tools, two in Python projects and two in TypeScript. Two were merged. The other two were closed: one was a feature the maintainer didn't want, and the idea in the other made it in through the maintainer's own PR. This post walks through the code, the design choices, and why each PR ended the way it did.</p>

<h2 class="relative group">Four PRs and how they ended
    <div id="four-prs-and-how-they-ended" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#four-prs-and-how-they-ended" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Statuses and dates come from the PR records on GitHub.</p>
<div class="fk-table" role="region" tabindex="0" aria-label="Scrollable table">
<table>
  <thead>
    <tr>
      <th style="text-align:left">Project</th>
      <th style="text-align:left">PR</th>
      <th style="text-align:left">Language</th>
      <th style="text-align:left">Topic</th>
      <th style="text-align:left">Status</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align:left">google-labs-code/jules-sdk</td>
      <td style="text-align:left"><a href="https://github.com/google-labs-code/jules-sdk/pull/184"  target="_blank" rel="noreferrer">#184</a></td>
      <td style="text-align:left">TypeScript</td>
      <td style="text-align:left"><code>session.delete()</code></td>
      <td style="text-align:left">Merged, March 16, 2026</td>
    </tr>
    <tr>
      <td style="text-align:left">teng-lin/notebooklm-py</td>
      <td style="text-align:left"><a href="https://github.com/teng-lin/notebooklm-py/pull/174"  target="_blank" rel="noreferrer">#174</a></td>
      <td style="text-align:left">Python</td>
      <td style="text-align:left">Notebook metadata export</td>
      <td style="text-align:left">Merged, March 11, 2026</td>
    </tr>
    <tr>
      <td style="text-align:left">amranu/cli-agent</td>
      <td style="text-align:left"><a href="https://github.com/amranu/cli-agent/pull/7"  target="_blank" rel="noreferrer">#7</a></td>
      <td style="text-align:left">Python</td>
      <td style="text-align:left"><code>/health</code> command</td>
      <td style="text-align:left">Closed</td>
    </tr>
    <tr>
      <td style="text-align:left">builderz-labs/mission-control</td>
      <td style="text-align:left"><a href="https://github.com/builderz-labs/mission-control/pull/271"  target="_blank" rel="noreferrer">#271</a></td>
      <td style="text-align:left">TypeScript</td>
      <td style="text-align:left">Gateway-optional mode</td>
      <td style="text-align:left">Closed, folded into #291</td>
    </tr>
  </tbody>
</table>
</div>
<h2 class="relative group">Jules SDK: deleting sessions
    <div id="jules-sdk-deleting-sessions" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#jules-sdk-deleting-sessions" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Jules is Google Labs' asynchronous coding agent, and jules-sdk is its TypeScript client. The API had an endpoint for deleting a session, but the SDK didn't expose it; issue <a href="https://github.com/google-labs-code/jules-sdk/issues/105"  target="_blank" rel="noreferrer">#105</a> tracked the gap. My PR added a <code>delete()</code> method to <code>SessionClient</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ts" data-lang="ts"><span class="line"><span class="cl"><span class="kr">async</span> <span class="k">delete</span><span class="p">()</span><span class="o">:</span> <span class="nx">Promise</span><span class="p">&lt;</span><span class="nt">void</span><span class="p">&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="k">try</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">await</span> <span class="k">this</span><span class="p">.</span><span class="nx">request</span><span class="p">(</span><span class="sb">`sessions/</span><span class="si">${</span><span class="k">this</span><span class="p">.</span><span class="nx">id</span><span class="si">}</span><span class="sb">`</span><span class="p">,</span> <span class="p">{</span> <span class="nx">method</span><span class="o">:</span> <span class="s1">&#39;DELETE&#39;</span> <span class="p">});</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">error</span>: <span class="kt">any</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="c1">// 404: the session is already gone, still clear the local cache
</span></span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="p">(</span><span class="nx">error</span><span class="p">.</span><span class="nx">status</span> <span class="o">!==</span> <span class="mi">404</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="k">throw</span> <span class="nx">error</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="k">await</span> <span class="k">this</span><span class="p">.</span><span class="nx">sessionStorage</span><span class="p">.</span><span class="k">delete</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">id</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>Two decisions are baked in. A 404 means the session no longer exists, so the local cache is cleared and the call succeeds. Any other error is rethrown without touching the cache, so the API and the cache never drift apart. The request helper only accepted <code>'GET' | 'POST'</code>, so I added <code>'DELETE'</code> to that union. Three tests cover success, 404 and 403. Usage is one line: <code>await session.delete();</code>. davideast merged the PR on March 16:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/posts/python-sdk-cli/img/jules-sdk-pr-184-header_hu_66ce58dd30f64649.avif 480w, https://blog.furkankoykiran.com.tr/posts/python-sdk-cli/img/jules-sdk-pr-184-header_hu_c370909b67baa56b.avif 800w, https://blog.furkankoykiran.com.tr/posts/python-sdk-cli/img/jules-sdk-pr-184-header_hu_bbe70fd5900bff1d.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/posts/python-sdk-cli/img/jules-sdk-pr-184-header_hu_57eecc4b85ed1599.webp 480w, https://blog.furkankoykiran.com.tr/posts/python-sdk-cli/img/jules-sdk-pr-184-header_hu_e5d6b7593e8afebc.webp 800w, https://blog.furkankoykiran.com.tr/posts/python-sdk-cli/img/jules-sdk-pr-184-header_hu_55a87b45f5af22de.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/posts/python-sdk-cli/img/jules-sdk-pr-184-header_hu_e5d6b7593e8afebc.webp" alt="Header of jules-sdk PR #184: merged by davideast, four commits, &#43;134 −1" width="1600" height="200"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Jules SDK PR #184. Source: GitHub</figcaption></figure>
<h2 class="relative group">notebooklm-py: exporting notebook metadata
    <div id="notebooklm-py-exporting-notebook-metadata" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#notebooklm-py-exporting-notebook-metadata" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>notebooklm-py is an unofficial Python API and CLI for Google NotebookLM. Google later renamed the product to Gemini Notebook; the library kept its name. In <a href="https://github.com/teng-lin/notebooklm-py/issues/171"  target="_blank" rel="noreferrer">#171</a>, a user wanted a notebook's details and its source list in one call. My PR added two things:</p>
<ul>
<li><code>client.notebooks.get_metadata(notebook_id)</code> fetches the notebook and its sources concurrently with <code>asyncio.gather</code> and returns a <code>NotebookMetadata</code>. Sources are reduced to <code>SourceSummary(kind, title, url)</code>, and <code>to_dict()</code> always emits the same keys, with <code>null</code> for missing values.</li>
<li>A <code>notebooklm metadata</code> CLI command with a <code>--json</code> flag.</li>
</ul>
<p>There's also a small safety net: if the notebook reports a non-zero source count but the source listing comes back empty, the code logs a warning, so possible data loss doesn't go unnoticed.</p>
<p>Run <code>notebooklm login</code> once, then:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">asyncio</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">notebooklm</span> <span class="kn">import</span> <span class="n">NotebookLMClient</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">main</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">with</span> <span class="n">NotebookLMClient</span><span class="o">.</span><span class="n">from_storage</span><span class="p">()</span> <span class="k">as</span> <span class="n">client</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">notebooks</span> <span class="o">=</span> <span class="k">await</span> <span class="n">client</span><span class="o">.</span><span class="n">notebooks</span><span class="o">.</span><span class="n">list</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="n">meta</span> <span class="o">=</span> <span class="k">await</span> <span class="n">client</span><span class="o">.</span><span class="n">notebooks</span><span class="o">.</span><span class="n">get_metadata</span><span class="p">(</span><span class="n">notebooks</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">meta</span><span class="o">.</span><span class="n">title</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">meta</span><span class="o">.</span><span class="n">sources</span><span class="p">)</span><span class="si">}</span><span class="s2"> sources&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">meta</span><span class="o">.</span><span class="n">to_dict</span><span class="p">(),</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="nb">str</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">asyncio</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">main</span><span class="p">())</span></span></span></code></pre></div></div>
<p>From the terminal, <code>notebooklm metadata --json</code> gives you the same data. The PR was merged on March 11. The library has grown a lot since then and the method now delegates to a separate service layer, but the public API is unchanged.</p>

<h2 class="relative group">cli-agent: a /health command (closed)
    <div id="cli-agent-a-health-command-closed" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#cli-agent-a-health-command-closed" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>cli-agent is a terminal agent that talks to several LLM providers. I proposed a <code>/health</code> command to show at a glance which API keys work. It checked six providers (Anthropic, OpenAI, DeepSeek, Google, OpenRouter, Ollama) concurrently with <code>asyncio.gather</code> and summarized the results with ✓, ✗ and ○. With tests and README changes, the diff went past 850 lines.</p>
<p>The maintainer's answer was short: not necessary, fork it if it's useful to you, the license is permissive.</p>
<p>Part of that was on me. I opened issue <a href="https://github.com/amranu/cli-agent/issues/6"  target="_blank" rel="noreferrer">#6</a> sixteen seconds before the PR, which is announcing a feature rather than asking about it. For a new feature, open the issue and wait for the maintainer to agree, especially when the diff is that large.</p>
<p>The idea is still handy. Here's a minimal version with three providers:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">asyncio</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">os</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">httpx</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">targets</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]]</span> <span class="o">|</span> <span class="kc">None</span><span class="p">]:</span>
</span></span><span class="line"><span class="cl">    <span class="n">openai_key</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&#34;OPENAI_API_KEY&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">anthropic_key</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&#34;ANTHROPIC_API_KEY&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;openai&#34;</span><span class="p">:</span> <span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;https://api.openai.com/v1/models&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="p">{</span><span class="s2">&#34;Authorization&#34;</span><span class="p">:</span> <span class="sa">f</span><span class="s2">&#34;Bearer </span><span class="si">{</span><span class="n">openai_key</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span> <span class="k">if</span> <span class="n">openai_key</span> <span class="k">else</span> <span class="kc">None</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;anthropic&#34;</span><span class="p">:</span> <span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;https://api.anthropic.com/v1/models&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="p">{</span><span class="s2">&#34;x-api-key&#34;</span><span class="p">:</span> <span class="n">anthropic_key</span><span class="p">,</span> <span class="s2">&#34;anthropic-version&#34;</span><span class="p">:</span> <span class="s2">&#34;2023-06-01&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span> <span class="k">if</span> <span class="n">anthropic_key</span> <span class="k">else</span> <span class="kc">None</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;ollama&#34;</span><span class="p">:</span> <span class="p">(</span><span class="s2">&#34;http://localhost:11434/api/tags&#34;</span><span class="p">,</span> <span class="p">{}),</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">check</span><span class="p">(</span><span class="n">client</span><span class="p">:</span> <span class="n">httpx</span><span class="o">.</span><span class="n">AsyncClient</span><span class="p">,</span> <span class="n">name</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">target</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">target</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;○ </span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">: not configured&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">url</span><span class="p">,</span> <span class="n">headers</span> <span class="o">=</span> <span class="n">target</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">response</span> <span class="o">=</span> <span class="k">await</span> <span class="n">client</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">response</span><span class="o">.</span><span class="n">raise_for_status</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;✓ </span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="n">response</span><span class="o">.</span><span class="n">elapsed</span><span class="o">.</span><span class="n">total_seconds</span><span class="p">()</span> <span class="o">*</span> <span class="mi">1000</span><span class="si">:</span><span class="s2">.0f</span><span class="si">}</span><span class="s2"> ms&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="n">httpx</span><span class="o">.</span><span class="n">HTTPError</span> <span class="k">as</span> <span class="n">exc</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;✗ </span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="n">exc</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">main</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">with</span> <span class="n">httpx</span><span class="o">.</span><span class="n">AsyncClient</span><span class="p">(</span><span class="n">timeout</span><span class="o">=</span><span class="mf">5.0</span><span class="p">)</span> <span class="k">as</span> <span class="n">client</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">results</span> <span class="o">=</span> <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="o">*</span><span class="p">(</span><span class="n">check</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">target</span><span class="p">)</span> <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">target</span> <span class="ow">in</span> <span class="n">targets</span><span class="p">()</span><span class="o">.</span><span class="n">items</span><span class="p">())</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="nb">print</span><span class="p">(</span><span class="s2">&#34;</span><span class="se">\n</span><span class="s2">&#34;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">results</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">asyncio</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">main</span><span class="p">())</span></span></span></code></pre></div></div>
<p>With <code>asyncio.gather</code> you wait as long as the slowest provider, not the sum of all of them. <code>httpx.HTTPError</code> covers both connection failures and 4xx/5xx responses.</p>

<h2 class="relative group">Mission Control: running without the gateway
    <div id="mission-control-running-without-the-gateway" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#mission-control-running-without-the-gateway" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Mission Control is a self-hosted control panel for AI agents, built with Next.js. In <a href="https://github.com/builderz-labs/mission-control/issues/270"  target="_blank" rel="noreferrer">#270</a>, a user reported that the gateway always showed as offline on a VPS. The browser connects to the gateway over WebSocket on non-standard ports (18789 and 18790), and most VPS firewalls block them.</p>
<p>My PR did two things. First, a standalone mode enabled with <code>NEXT_PUBLIC_GATEWAY_OPTIONAL=true</code>: instead of a red &quot;Disconnected&quot; warning, the HUD shows a blue &quot;Gateway Optional (Standalone)&quot; badge. The task board, projects, agent registry, cron jobs and webhooks keep working without a gateway; live session updates and agent-to-agent messaging don't.</p>
<p>Second, documentation for the proper fix: proxy the WebSocket traffic through port 443. The general nginx pattern looks like this (adjust the path and port to your setup):</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-nginx" data-lang="nginx"><span class="line"><span class="cl"><span class="k">location</span> <span class="s">/gateway/</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="kn">proxy_pass</span> <span class="s">http://127.0.0.1:18789/</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="kn">proxy_http_version</span> <span class="mi">1</span><span class="s">.1</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="kn">proxy_set_header</span> <span class="s">Upgrade</span> <span class="nv">$http_upgrade</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="kn">proxy_set_header</span> <span class="s">Connection</span> <span class="s">&#34;upgrade&#34;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="kn">proxy_set_header</span> <span class="s">Host</span> <span class="nv">$host</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="kn">proxy_read_timeout</span> <span class="s">1h</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>Without the <code>Upgrade</code> and <code>Connection</code> headers, nginx won't forward the WebSocket handshake. <code>proxy_read_timeout</code> keeps idle connections from being dropped after the default 60 seconds.</p>
<p>The same day, the maintainer 0xNyk collected the open deployment issues into one PR. <a href="https://github.com/builderz-labs/mission-control/pull/291"  target="_blank" rel="noreferrer">#291</a> bundled two Docker fixes with the gateway-optional behavior and its docs, was merged on March 11, and closed my issue <a href="https://github.com/builderz-labs/mission-control/issues/289"  target="_blank" rel="noreferrer">#289</a> along the way. #271 was closed with a note pointing to #291:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/posts/python-sdk-cli/img/mission-control-pr-271-header_hu_cf8b20c8b736634a.avif 480w, https://blog.furkankoykiran.com.tr/posts/python-sdk-cli/img/mission-control-pr-271-header_hu_8df6212a04e2ca8.avif 800w, https://blog.furkankoykiran.com.tr/posts/python-sdk-cli/img/mission-control-pr-271-header_hu_cedf19e3c60b9c0e.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/posts/python-sdk-cli/img/mission-control-pr-271-header_hu_2fafb1bb7cfb048.webp 480w, https://blog.furkankoykiran.com.tr/posts/python-sdk-cli/img/mission-control-pr-271-header_hu_380c4ed260b7ab99.webp 800w, https://blog.furkankoykiran.com.tr/posts/python-sdk-cli/img/mission-control-pr-271-header_hu_e638be927ae8d31.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/posts/python-sdk-cli/img/mission-control-pr-271-header_hu_380c4ed260b7ab99.webp" alt="Header of mission-control PR #271: closed, four commits from the fix/270-gateway-optional-mode branch, &#43;173" width="1600" height="200"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Mission Control PR #271. Source: GitHub</figcaption></figure><p>The <code>NEXT_PUBLIC_GATEWAY_OPTIONAL</code> line is in the project's <code>.env.example</code>.</p>

<h2 class="relative group">What four PRs taught me
    <div id="what-four-prs-taught-me" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-four-prs-taught-me" aria-label="Anchor">#</a>
    </span>
    
</h2>
<ul>
<li>Both merged PRs solved a well-scoped issue that someone else had opened.</li>
<li>In Jules SDK the new method reused the existing <code>request</code> helper and test layout, so there was little to discuss in review.</li>
<li>In Mission Control my PR was closed, the idea shipped through the maintainer's PR, and the user's problem got fixed.</li>
</ul>

<h2 class="relative group">What's next
    <div id="whats-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#whats-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>My other contributions from those weeks are in <a href="https://blog.furkankoykiran.com.tr/en/posts/mcp-ai-open-source-contributions/" >MCP open source contributions</a> and <a href="https://blog.furkankoykiran.com.tr/en/posts/security-and-api-fixes/" >security and API fixes</a>. For more on <code>asyncio.gather</code> and async clients, see <a href="https://blog.furkankoykiran.com.tr/en/posts/python-asyncio-concurrent-processing/" >Python asyncio for concurrent processing</a>. If you're looking for a first SDK contribution, start with issues of the form &quot;the API supports this but the SDK doesn't&quot;. The scope is clear and the change is easy to test. For a Go example, see <a href="https://blog.furkankoykiran.com.tr/en/posts/my-open-source-contribution-keploy/" >my Keploy contribution</a>.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/python-sdk-and-cli-contributions/feature_hu_918a19a8cddb241f.jpg" medium="image" type="image/jpeg"/></item><item><title>Security and API Fixes in Open Source: 3 Pull Requests</title><link>https://blog.furkankoykiran.com.tr/en/posts/security-and-api-fixes/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/security-and-api-fixes/</guid><pubDate>Wed, 11 Mar 2026 17:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Open Source</category><category>security</category><category>open-source</category><category>python</category><category>api</category><category>databases</category><category>ethereum</category><description>A Cypher injection in LightRAG, a wrong enum value in the OpenAI Python SDK and hardcoded decimals in Coinbase AgentKit: three PRs, how each ended, and why.</description><content:encoded><![CDATA[<p>In the last week of February 2026 I opened three PRs for security and API contract bugs: a Cypher injection in LightRAG, a misspelled enum value in the OpenAI Python SDK, and hardcoded token decimals in Coinbase AgentKit. This post walks through what I changed in each and what happened next. They all ended differently: one landed through the maintainer's PR, one turned out to be unnecessary, and one is waiting for review.</p>

<h2 class="relative group">Three PRs and where they stand
    <div id="three-prs-and-where-they-stand" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#three-prs-and-where-they-stand" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Statuses and dates come from the PR records on GitHub.</p>
<div class="fk-table" role="region" tabindex="0" aria-label="Scrollable table">
<table>
  <thead>
    <tr>
      <th style="text-align:left">Project</th>
      <th style="text-align:left">PR</th>
      <th style="text-align:left">Topic</th>
      <th style="text-align:left">Status</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align:left">HKUDS/LightRAG</td>
      <td style="text-align:left"><a href="https://github.com/HKUDS/LightRAG/pull/2713"  target="_blank" rel="noreferrer">#2713</a></td>
      <td style="text-align:left">Cypher injection via workspace label</td>
      <td style="text-align:left">Closed. My commits were merged through <a href="https://github.com/HKUDS/LightRAG/pull/2792"  target="_blank" rel="noreferrer">#2792</a> on March 18, 2026</td>
    </tr>
    <tr>
      <td style="text-align:left">openai/openai-python</td>
      <td style="text-align:left"><a href="https://github.com/openai/openai-python/pull/2893"  target="_blank" rel="noreferrer">#2893</a></td>
      <td style="text-align:left"><code>prompt_cache_retention</code> value</td>
      <td style="text-align:left">Closed, already fixed upstream</td>
    </tr>
    <tr>
      <td style="text-align:left">coinbase/agentkit</td>
      <td style="text-align:left"><a href="https://github.com/coinbase/agentkit/pull/965"  target="_blank" rel="noreferrer">#965</a></td>
      <td style="text-align:left">Hardcoded 18 decimals in Morpho withdraw</td>
      <td style="text-align:left">Open</td>
    </tr>
  </tbody>
</table>
</div>
<h2 class="relative group">LightRAG: Cypher injection through the workspace name
    <div id="lightrag-cypher-injection-through-the-workspace-name" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#lightrag-cypher-injection-through-the-workspace-name" aria-label="Anchor">#</a>
    </span>
    
</h2>

<h3 class="relative group">The hole
    <div id="the-hole" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-hole" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>LightRAG can store its knowledge graph in Neo4j or Memgraph and separates workspaces with node labels. The API server can read the workspace name from the <code>LIGHTRAG-WORKSPACE</code> HTTP header. In <code>Neo4JStorage</code> and <code>MemgraphStorage</code>, <code>_get_workspace_label()</code> returned that value untouched, and it went into queries through f-strings. The pattern looks like this:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">query</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;MATCH (n:`</span><span class="si">{</span><span class="n">workspace_label</span><span class="si">}</span><span class="s2">` </span><span class="se">{{</span><span class="s2">entity_id: $entity_id</span><span class="se">}}</span><span class="s2">) RETURN n&#34;</span></span></span></code></pre></div></div>
<p>Values like <code>$entity_id</code> are passed as parameters, which is correct. The label, though, is pasted into the query text. A header value containing a backtick could close the label and append arbitrary Cypher. lighthousekeeper1212 reported it in <a href="https://github.com/HKUDS/LightRAG/issues/2698"  target="_blank" rel="noreferrer">#2698</a>, and I picked up the fix.</p>

<h3 class="relative group">First commit and the collision problem
    <div id="first-commit-and-the-collision-problem" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#first-commit-and-the-collision-problem" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>The PostgreSQL backend already sanitized workspace names with <code>re.sub(r&quot;[^a-zA-Z0-9_]&quot;, &quot;_&quot;, ...)</code>, so my first commit applied the same rule to Neo4j and Memgraph and added 20 tests. Then a side effect came up: <code>team-a</code> and <code>team_a</code> map to the same label, so two different workspaces could share data.</p>
<p>In the second commit I stopped stripping characters and escaped backticks by doubling them. Every workspace name now maps to exactly one label:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">_get_workspace_label</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">workspace</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">workspace</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">workspace</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="s2">&#34;base&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">workspace</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">&#34;`&#34;</span><span class="p">,</span> <span class="s2">&#34;``&#34;</span><span class="p">)</span></span></span></code></pre></div></div>

<h3 class="relative group">Review: escaping depends on context
    <div id="review-escaping-depends-on-context" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#review-escaping-depends-on-context" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>The maintainer, danielaskdd, found three problems in review. All three came from using the escaped value everywhere.</p>
<ol>
<li>Labels returned by the driver are raw (<code>foo</code>bar<code>), and the code compared them with the escaped value (</code>foo``bar`), so the comparison never matched.</li>
<li>The Memgraph index creation query used the label without backticks, where the escaped value produces invalid Cypher.</li>
<li>One query used the label as a string literal, <code>WHERE '{label}' IN labels(n)</code>. Doubled backticks mean nothing inside a string, so that query never matched either.</li>
</ol>
<p>Backtick escaping only applies to an identifier inside backticks. In a string context, bind the raw value as a parameter. Here are both, side by side, with the official <code>neo4j</code> Python driver:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">neo4j</span> <span class="kn">import</span> <span class="n">GraphDatabase</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">identifier_label</span><span class="p">(</span><span class="n">workspace</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Label for use inside backticks.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="p">(</span><span class="n">workspace</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span> <span class="ow">or</span> <span class="s2">&#34;base&#34;</span><span class="p">)</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">&#34;`&#34;</span><span class="p">,</span> <span class="s2">&#34;``&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">raw_label</span><span class="p">(</span><span class="n">workspace</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Label for binding as a query parameter.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">workspace</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span> <span class="ow">or</span> <span class="s2">&#34;base&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">find_entity</span><span class="p">(</span><span class="n">driver</span><span class="p">,</span> <span class="n">workspace</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">entity_id</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">label</span> <span class="o">=</span> <span class="n">identifier_label</span><span class="p">(</span><span class="n">workspace</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># Identifier context: escaped value inside backticks</span>
</span></span><span class="line"><span class="cl">    <span class="n">by_label</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;MATCH (n:`</span><span class="si">{</span><span class="n">label</span><span class="si">}</span><span class="s2">` </span><span class="se">{{</span><span class="s2">entity_id: $entity_id</span><span class="se">}}</span><span class="s2">) RETURN n&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># String context: raw value as a parameter, never interpolated</span>
</span></span><span class="line"><span class="cl">    <span class="n">by_filter</span> <span class="o">=</span> <span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;MATCH (n) WHERE $label IN labels(n) &#34;</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;AND n.entity_id = $entity_id RETURN n&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">driver</span><span class="o">.</span><span class="n">session</span><span class="p">()</span> <span class="k">as</span> <span class="n">session</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">first</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">by_label</span><span class="p">,</span> <span class="n">entity_id</span><span class="o">=</span><span class="n">entity_id</span><span class="p">)</span><span class="o">.</span><span class="n">single</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="n">second</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">run</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="n">by_filter</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="n">raw_label</span><span class="p">(</span><span class="n">workspace</span><span class="p">),</span> <span class="n">entity_id</span><span class="o">=</span><span class="n">entity_id</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span><span class="o">.</span><span class="n">single</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">first</span><span class="p">,</span> <span class="n">second</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&#34;__main__&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">GraphDatabase</span><span class="o">.</span><span class="n">driver</span><span class="p">(</span><span class="s2">&#34;bolt://localhost:7687&#34;</span><span class="p">,</span> <span class="n">auth</span><span class="o">=</span><span class="p">(</span><span class="s2">&#34;neo4j&#34;</span><span class="p">,</span> <span class="s2">&#34;password&#34;</span><span class="p">))</span> <span class="k">as</span> <span class="n">driver</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="n">find_entity</span><span class="p">(</span><span class="n">driver</span><span class="p">,</span> <span class="s2">&#34;team-a&#34;</span><span class="p">,</span> <span class="s2">&#34;Alan Turing&#34;</span><span class="p">))</span></span></span></code></pre></div></div>

<h3 class="relative group">How it landed: validation at the entry point too
    <div id="how-it-landed-validation-at-the-entry-point-too" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#how-it-landed-validation-at-the-entry-point-too" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>danielaskdd took my branch and added a commit on top. The workspace value is now sanitized where it enters the system, both from <code>--workspace</code> / <code>WORKSPACE</code> and from the <code>LIGHTRAG-WORKSPACE</code> header: anything outside <code>[a-zA-Z0-9_]</code> becomes <code>_</code>, with a warning logged when the value changes. The combined change was merged as <a href="https://github.com/HKUDS/LightRAG/pull/2792"  target="_blank" rel="noreferrer">#2792</a> on March 18, 2026:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/posts/security-api-fixes/img/lightrag-pr-2792-header_hu_b5f5b3052f4f16d1.avif 480w, https://blog.furkankoykiran.com.tr/posts/security-api-fixes/img/lightrag-pr-2792-header_hu_9af19a1e731fb745.avif 800w, https://blog.furkankoykiran.com.tr/posts/security-api-fixes/img/lightrag-pr-2792-header_hu_60c33f8a25dce2c3.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/posts/security-api-fixes/img/lightrag-pr-2792-header_hu_f250eb6d44339464.webp 480w, https://blog.furkankoykiran.com.tr/posts/security-api-fixes/img/lightrag-pr-2792-header_hu_a6083fcb679546e2.webp 800w, https://blog.furkankoykiran.com.tr/posts/security-api-fixes/img/lightrag-pr-2792-header_hu_49fb4700020909f0.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/posts/security-api-fixes/img/lightrag-pr-2792-header_hu_a6083fcb679546e2.webp" alt="Header of LightRAG PR #2792: merged by danielaskdd, seven commits, &#43;305 −72" width="1600" height="200"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>LightRAG PR #2792. Source: GitHub</figcaption></figure><p>That's why my #2713 shows as closed while my commits are in main. <code>_get_workspace_label()</code> uses my escaping, and a separate helper binds the raw label as a parameter for string contexts. The two layers work together.</p>
<p>Don't rely on a single layer. Validate input against an allowlist at the edge, and escape or parameterize correctly for each context in the query.</p>

<h2 class="relative group">OpenAI Python SDK: <code>in-memory</code> or <code>in_memory</code>?
    <div id="openai-python-sdk-in-memory-or-in_memory" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#openai-python-sdk-in-memory-or-in_memory" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Issue <a href="https://github.com/openai/openai-python/issues/2883"  target="_blank" rel="noreferrer">#2883</a> pointed out that the SDK typed <code>prompt_cache_retention</code> as <code>Literal[&quot;in-memory&quot;, &quot;24h&quot;]</code>, while the API rejected <code>&quot;in-memory&quot;</code> with a 400 and only accepted <code>&quot;in_memory&quot;</code>. The type checker suggested the wrong value and flagged the right one. The bug was in the value's spelling, not the parameter name.</p>
<p>My PR fixed the literal in nine files (+41/−41, across the Chat Completions and Responses types and tests). Six months later a maintainer closed it: the same fix had already landed in main through a commit from the SDK's code generation pipeline. openai-python isn't written by hand. It's generated with Stainless from OpenAI's OpenAPI spec, and the spec is where a lasting fix goes.</p>
<p>Before opening a PR against a generated SDK, read the README and CONTRIBUTING file. For projects like this, a good issue is often more useful than a PR. Correct usage:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">openai</span> <span class="kn">import</span> <span class="n">OpenAI</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">client</span> <span class="o">=</span> <span class="n">OpenAI</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">response</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">responses</span><span class="o">.</span><span class="n">create</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">model</span><span class="o">=</span><span class="s2">&#34;gpt-5.1&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nb">input</span><span class="o">=</span><span class="s2">&#34;Summarize this text in three bullet points: ...&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">prompt_cache_retention</span><span class="o">=</span><span class="s2">&#34;24h&#34;</span><span class="p">,</span>  <span class="c1"># or &#34;in_memory&#34;</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">output_text</span><span class="p">)</span></span></span></code></pre></div></div>

<h2 class="relative group">Coinbase AgentKit: decimals hardcoded to 18
    <div id="coinbase-agentkit-decimals-hardcoded-to-18" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#coinbase-agentkit-decimals-hardcoded-to-18" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>AgentKit is a toolkit that lets AI agents act onchain. Its Morpho <code>withdraw</code> action converted the amount with <code>Web3.to_wei(assets, &quot;ether&quot;)</code>, and the <code>&quot;ether&quot;</code> unit always means 18 decimals. USDC uses 6. An agent withdrawing 100 USDC would request 100 × 10¹⁸ base units, 10¹² times too many. The bug was reported in <a href="https://github.com/coinbase/agentkit/issues/946"  target="_blank" rel="noreferrer">#946</a>.</p>
<p>The <code>deposit</code> action in the same provider already did this right by reading <code>decimals()</code> from the token contract. My PR made <code>withdraw</code> follow the same path: it adds a <code>token_address</code> field to the schema, reads decimals through <code>wallet_provider.read_contract(...)</code>, converts the amount accordingly, and adds a test for a 6-decimal token.</p>
<p>Here's the conversion on its own, independent of AgentKit and runnable as is:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">decimal</span> <span class="kn">import</span> <span class="n">Decimal</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">to_atomic</span><span class="p">(</span><span class="n">amount</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">decimals</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">value</span> <span class="o">=</span> <span class="n">Decimal</span><span class="p">(</span><span class="n">amount</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">value</span> <span class="o">&lt;=</span> <span class="mi">0</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">&#34;amount must be positive&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">scaled</span> <span class="o">=</span> <span class="n">value</span><span class="o">.</span><span class="n">scaleb</span><span class="p">(</span><span class="n">decimals</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">scaled</span> <span class="o">!=</span> <span class="n">scaled</span><span class="o">.</span><span class="n">to_integral_value</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">        <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">amount</span><span class="si">}</span><span class="s2"> has more than </span><span class="si">{</span><span class="n">decimals</span><span class="si">}</span><span class="s2"> decimal places&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="nb">int</span><span class="p">(</span><span class="n">scaled</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">assert</span> <span class="n">to_atomic</span><span class="p">(</span><span class="s2">&#34;100&#34;</span><span class="p">,</span> <span class="mi">6</span><span class="p">)</span> <span class="o">==</span> <span class="mi">100_000_000</span>                 <span class="c1"># USDC</span>
</span></span><span class="line"><span class="cl"><span class="k">assert</span> <span class="n">to_atomic</span><span class="p">(</span><span class="s2">&#34;0.5&#34;</span><span class="p">,</span> <span class="mi">18</span><span class="p">)</span> <span class="o">==</span> <span class="mi">500_000_000_000_000_000</span>    <span class="c1"># WETH</span></span></span></code></pre></div></div>
<p>I keep the amount as a <code>Decimal</code> built from a string, never a <code>float</code>, so values like <code>0.1</code> don't pick up rounding errors. Reading decimals with web3.py is short:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">web3</span> <span class="kn">import</span> <span class="n">Web3</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">ERC20_DECIMALS_ABI</span> <span class="o">=</span> <span class="p">[{</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;name&#34;</span><span class="p">:</span> <span class="s2">&#34;decimals&#34;</span><span class="p">,</span> <span class="s2">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;function&#34;</span><span class="p">,</span> <span class="s2">&#34;stateMutability&#34;</span><span class="p">:</span> <span class="s2">&#34;view&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;inputs&#34;</span><span class="p">:</span> <span class="p">[],</span> <span class="s2">&#34;outputs&#34;</span><span class="p">:</span> <span class="p">[{</span><span class="s2">&#34;name&#34;</span><span class="p">:</span> <span class="s2">&#34;&#34;</span><span class="p">,</span> <span class="s2">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;uint8&#34;</span><span class="p">}],</span>
</span></span><span class="line"><span class="cl"><span class="p">}]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">w3</span> <span class="o">=</span> <span class="n">Web3</span><span class="p">(</span><span class="n">Web3</span><span class="o">.</span><span class="n">HTTPProvider</span><span class="p">(</span><span class="s2">&#34;https://mainnet.base.org&#34;</span><span class="p">))</span>
</span></span><span class="line"><span class="cl"><span class="n">usdc</span> <span class="o">=</span> <span class="n">w3</span><span class="o">.</span><span class="n">eth</span><span class="o">.</span><span class="n">contract</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">address</span><span class="o">=</span><span class="s2">&#34;0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913&#34;</span><span class="p">,</span>  <span class="c1"># USDC on Base</span>
</span></span><span class="line"><span class="cl">    <span class="n">abi</span><span class="o">=</span><span class="n">ERC20_DECIMALS_ABI</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="n">usdc</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">decimals</span><span class="p">()</span><span class="o">.</span><span class="n">call</span><span class="p">())</span>  <span class="c1"># 6</span></span></span></code></pre></div></div>
<p>The PR is open and waiting for review:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/posts/security-api-fixes/img/agentkit-pr-965-header_hu_372c135835b95aed.avif 480w, https://blog.furkankoykiran.com.tr/posts/security-api-fixes/img/agentkit-pr-965-header_hu_8811bfb2390d77d4.avif 800w, https://blog.furkankoykiran.com.tr/posts/security-api-fixes/img/agentkit-pr-965-header_hu_7818dbf64314dece.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/posts/security-api-fixes/img/agentkit-pr-965-header_hu_adbe087c79376d83.webp 480w, https://blog.furkankoykiran.com.tr/posts/security-api-fixes/img/agentkit-pr-965-header_hu_40804f389b8aa9d7.webp 800w, https://blog.furkankoykiran.com.tr/posts/security-api-fixes/img/agentkit-pr-965-header_hu_cb8791b3f478cc6e.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/posts/security-api-fixes/img/agentkit-pr-965-header_hu_40804f389b8aa9d7.webp" alt="Header of coinbase/agentkit PR #965: open, one commit from the fix/morpho-withdraw-decimals branch, &#43;92 −29" width="1600" height="253"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>AgentKit PR #965. Source: GitHub</figcaption></figure><p>Until it's merged, <code>withdraw</code> in main keeps using <code>to_wei(..., &quot;ether&quot;)</code>. If you call it with a token that doesn't use 18 decimals, check the amount yourself.</p>

<h2 class="relative group">Three PRs, three endings
    <div id="three-prs-three-endings" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#three-prs-three-endings" aria-label="Anchor">#</a>
    </span>
    
</h2>
<ul>
<li>In LightRAG my code landed, and the fix was complete once the maintainer added validation at the entry point.</li>
<li>In the OpenAI SDK I fixed a real bug in the wrong place. In a generated codebase, the source spec is where the patch belongs.</li>
<li>In AgentKit the fix and its test are ready, and the PR is waiting for review.</li>
</ul>

<h2 class="relative group">What's next
    <div id="whats-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#whats-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>My other contributions from those weeks are in <a href="https://blog.furkankoykiran.com.tr/en/posts/mcp-ai-open-source-contributions/" >MCP open source contributions</a> and <a href="https://blog.furkankoykiran.com.tr/en/posts/python-sdk-and-cli-contributions/" >SDK and CLI contributions</a>, and the Go side is in <a href="https://blog.furkankoykiran.com.tr/en/posts/my-open-source-contribution-keploy/" >my Keploy contribution</a>. If your own code talks to a graph database, start by checking where the labels and relationship types in your f-string queries come from.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/security-and-api-fixes/feature_hu_85fa9ca1d983a706.jpg" medium="image" type="image/jpeg"/></item><item><title>MCP Open Source Contributions: Playwright, GitHub, LobeHub</title><link>https://blog.furkankoykiran.com.tr/en/posts/mcp-ai-open-source-contributions/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/mcp-ai-open-source-contributions/</guid><pubDate>Wed, 11 Mar 2026 16:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Open Source</category><category>mcp</category><category>open-source</category><category>typescript</category><category>go</category><category>ai-agents</category><description>My PRs to Playwright MCP, the GitHub MCP Server and LobeHub: which were merged, which were closed, and what the reviews taught me. With code and links.</description><content:encoded><![CDATA[<p>Once I started building my own MCP servers, I began reading the code of the bigger projects in the same ecosystem. This post covers five PRs I opened around the Model Context Protocol (MCP) and AI tooling. Both PRs to Playwright were merged. One PR to GitHub's MCP server is open, and the other was closed in favor of a better fix the maintainer wrote. A one-line fix in LobeHub is waiting for review. For each one: the bug, the fix, and what the review taught me.</p>

<h2 class="relative group">Five PRs and where they stand
    <div id="five-prs-and-where-they-stand" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#five-prs-and-where-they-stand" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Statuses and dates come from the PR records on GitHub.</p>
<div class="fk-table" role="region" tabindex="0" aria-label="Scrollable table">
<table>
  <thead>
    <tr>
      <th style="text-align:left">Project</th>
      <th style="text-align:left">PR</th>
      <th style="text-align:left">Topic</th>
      <th style="text-align:left">Status</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align:left">microsoft/playwright</td>
      <td style="text-align:left"><a href="https://github.com/microsoft/playwright/pull/39401"  target="_blank" rel="noreferrer">#39401</a></td>
      <td style="text-align:left">Truncated <code>--cdp-header</code> values</td>
      <td style="text-align:left">Merged, March 10, 2026</td>
    </tr>
    <tr>
      <td style="text-align:left">microsoft/playwright</td>
      <td style="text-align:left"><a href="https://github.com/microsoft/playwright/pull/39625"  target="_blank" rel="noreferrer">#39625</a></td>
      <td style="text-align:left">Malformed Unicode in MCP responses</td>
      <td style="text-align:left">Merged, March 14, 2026</td>
    </tr>
    <tr>
      <td style="text-align:left">github/github-mcp-server</td>
      <td style="text-align:left"><a href="https://github.com/github/github-mcp-server/pull/1519"  target="_blank" rel="noreferrer">#1519</a></td>
      <td style="text-align:left"><code>create_discussion</code> tool</td>
      <td style="text-align:left">Open</td>
    </tr>
    <tr>
      <td style="text-align:left">github/github-mcp-server</td>
      <td style="text-align:left"><a href="https://github.com/github/github-mcp-server/pull/2200"  target="_blank" rel="noreferrer">#2200</a></td>
      <td style="text-align:left"><code>--read-only</code> in HTTP mode</td>
      <td style="text-align:left">Closed, superseded by #2208</td>
    </tr>
    <tr>
      <td style="text-align:left">lobehub/lobehub</td>
      <td style="text-align:left"><a href="https://github.com/lobehub/lobehub/pull/12818"  target="_blank" rel="noreferrer">#12818</a></td>
      <td style="text-align:left">Model picker crash during onboarding</td>
      <td style="text-align:left">Open</td>
    </tr>
  </tbody>
</table>
</div>
<h2 class="relative group">Playwright MCP: header values with colons
    <div id="playwright-mcp-header-values-with-colons" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#playwright-mcp-header-values-with-colons" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>When you connect Playwright MCP to a remote browser over CDP, you can pass headers with <code>--cdp-header</code> or the <code>PLAYWRIGHT_MCP_CDP_HEADERS</code> environment variable. The <code>headerParser</code> function split each value with <code>arg.split(':')</code> and destructured the first two parts, so <code>X-Custom: http://example.com</code> silently became <code>http</code>. Anything with a colon in the value got cut: URLs, <code>host:port</code> pairs, some tokens.</p>
<p>The fix is to split on the first colon only:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ts" data-lang="ts"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">colonIndex</span> <span class="o">=</span> <span class="nx">arg</span><span class="p">.</span><span class="nx">indexOf</span><span class="p">(</span><span class="s1">&#39;:&#39;</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">name</span> <span class="o">=</span> <span class="nx">colonIndex</span> <span class="o">===</span> <span class="o">-</span><span class="mi">1</span> <span class="o">?</span> <span class="nx">arg</span><span class="p">.</span><span class="nx">trim</span><span class="p">()</span> <span class="o">:</span> <span class="nx">arg</span><span class="p">.</span><span class="nx">substring</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="nx">colonIndex</span><span class="p">).</span><span class="nx">trim</span><span class="p">();</span>
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">value</span> <span class="o">=</span> <span class="nx">colonIndex</span> <span class="o">===</span> <span class="o">-</span><span class="mi">1</span> <span class="o">?</span> <span class="s1">&#39;&#39;</span> <span class="o">:</span> <span class="nx">arg</span><span class="p">.</span><span class="nx">substring</span><span class="p">(</span><span class="nx">colonIndex</span> <span class="o">+</span> <span class="mi">1</span><span class="p">).</span><span class="nx">trim</span><span class="p">();</span></span></span></code></pre></div></div>
<p>The test sends <code>X-Forwarded-Proto: value:with:colons</code> plus an empty <code>X-Empty</code> header and checks what actually reaches the test server. It was merged on March 10.</p>

<h2 class="relative group">Playwright MCP: malformed Unicode in responses
    <div id="playwright-mcp-malformed-unicode-in-responses" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#playwright-mcp-malformed-unicode-in-responses" aria-label="Anchor">#</a>
    </span>
    
</h2>

<h3 class="relative group">The problem
    <div id="the-problem" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-problem" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>In <a href="https://github.com/microsoft/playwright-mcp/issues/1447"  target="_blank" rel="noreferrer">playwright-mcp#1447</a>, frankhommers reported that MCP responses from some pages failed with JSON errors. The cause was lone surrogates: half of a UTF-16 surrogate pair with no partner, for example an emoji cut in the middle.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-js" data-lang="js"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">s</span> <span class="o">=</span> <span class="sb">`Hello </span><span class="si">${</span><span class="nb">String</span><span class="p">.</span><span class="nx">fromCharCode</span><span class="p">(</span><span class="mh">0xD800</span><span class="p">)</span><span class="si">}</span><span class="sb">`</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nx">s</span><span class="p">.</span><span class="nx">isWellFormed</span><span class="p">();</span>   <span class="c1">// false
</span></span></span><span class="line"><span class="cl"><span class="nx">JSON</span><span class="p">.</span><span class="nx">stringify</span><span class="p">(</span><span class="nx">s</span><span class="p">);</span>  <span class="c1">// &#39;&#34;Hello \\ud800&#34;&#39; (no error, but the lone surrogate is still there)
</span></span></span><span class="line"><span class="cl"><span class="nx">s</span><span class="p">.</span><span class="nx">toWellFormed</span><span class="p">();</span>   <span class="c1">// &#39;Hello �&#39;, displayed as &#34;Hello �&#34;
</span></span></span></code></pre></div></div>
<p>Modern <code>JSON.stringify</code> doesn't throw here. It escapes the character as <code>\ud800</code>. The failure shows up further down the chain, when a stricter JSON parser on the client side rejects the payload with errors along the lines of &quot;invalid high surrogate&quot;. The client sees the error, but the server is where it should be fixed.</p>

<h3 class="relative group">Where the fix belongs
    <div id="where-the-fix-belongs" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-the-fix-belongs" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>frankhommers' first attempt (<a href="https://github.com/microsoft/playwright-mcp/pull/1448"  target="_blank" rel="noreferrer">#1448</a>) sanitized the output in <code>cli.js</code>, at the transport layer, and was closed. I looked at where the text is produced instead. In <code>packages/playwright-core/src/tools/backend/response.ts</code>, every response already went through <code>redactText()</code>, so adding the cleanup there covered every tool in one place. The diagram shows where each attempt sits in the chain:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/mcp-ai-open-source-contributions/img/unicode-fix-location_hu_1e38690780f419b9.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/mcp-ai-open-source-contributions/img/unicode-fix-location_hu_14ce362653d1dd11.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/mcp-ai-open-source-contributions/img/unicode-fix-location_hu_a57998bc403cadc9.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/mcp-ai-open-source-contributions/img/unicode-fix-location_hu_ec10188895484a61.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/mcp-ai-open-source-contributions/img/unicode-fix-location_hu_16735491d2e4cb1.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/mcp-ai-open-source-contributions/img/unicode-fix-location_hu_ad8998ea210509f4.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/mcp-ai-open-source-contributions/img/unicode-fix-location_hu_16735491d2e4cb1.webp" alt="Page text passes redactText and sanitizeUnicode in response.ts before the JSON-RPC response; the closed #1448 sat at the transport layer" width="1600" height="1235"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>The cleanup happens where the text is produced</figcaption></figure><p>The change itself is short:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ts" data-lang="ts"><span class="line"><span class="cl"><span class="nx">text</span>: <span class="kt">sanitizeUnicode</span><span class="p">(</span><span class="nx">redactText</span><span class="p">(</span><span class="nx">text</span><span class="p">.</span><span class="nx">join</span><span class="p">(</span><span class="s1">&#39;\n&#39;</span><span class="p">))),</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// ...
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kd">function</span> <span class="nx">sanitizeUnicode</span><span class="p">(</span><span class="nx">text</span>: <span class="kt">string</span><span class="p">)</span><span class="o">:</span> <span class="kt">string</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="k">return</span> <span class="nx">text</span><span class="p">.</span><span class="nx">toWellFormed</span><span class="p">();</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p><code>toWellFormed()</code> replaces lone surrogates with U+FFFD and leaves valid emoji and CJK text alone.</p>

<h3 class="relative group">What the review taught me
    <div id="what-the-review-taught-me" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-the-review-taught-me" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>The PR's initial commit had a regex fallback for Node 18 and six tests. Pavel Feldman's review was short: it looked like an agent-generated blob that failed lint; the location was right, but I should assume Node 20 and hand-write the tests. He was right. I dropped the fallback, since <code>toWellFormed()</code> has been in Node since version 20, and cut the tests down to three that match the style of the existing MCP tests: a page with a lone surrogate, a page with valid emoji, and a <code>console.log</code> that prints a lone surrogate. Pavel Feldman merged it on March 14:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/posts/mcp-ai-contributions/img/playwright-pr-39625-header_hu_7af375b48318a430.avif 480w, https://blog.furkankoykiran.com.tr/posts/mcp-ai-contributions/img/playwright-pr-39625-header_hu_3e018f80dd009213.avif 800w, https://blog.furkankoykiran.com.tr/posts/mcp-ai-contributions/img/playwright-pr-39625-header_hu_75e2aa4e3a897c15.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/posts/mcp-ai-contributions/img/playwright-pr-39625-header_hu_7a86f372d35a34ba.webp 480w, https://blog.furkankoykiran.com.tr/posts/mcp-ai-contributions/img/playwright-pr-39625-header_hu_c81a8640a4356e5c.webp 800w, https://blog.furkankoykiran.com.tr/posts/mcp-ai-contributions/img/playwright-pr-39625-header_hu_147b7b5079d48b9.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/posts/mcp-ai-contributions/img/playwright-pr-39625-header_hu_c81a8640a4356e5c.webp" alt="Header of Playwright PR #39625: merged by Pavel Feldman, three commits, &#43;65 −1" width="1600" height="200"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Playwright PR #39625. Source: GitHub</figcaption></figure><p>The maintainer spotted the generated code at first glance. The short version, written in the style of the project's tests, landed within days.</p>

<h2 class="relative group">GitHub MCP Server
    <div id="github-mcp-server" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#github-mcp-server" aria-label="Anchor">#</a>
    </span>
    
</h2>

<h3 class="relative group">A create_discussion tool (#1519, open)
    <div id="a-create_discussion-tool-1519-open" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#a-create_discussion-tool-1519-open" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>GitHub's official MCP server could read discussions but not create them. In December 2025 I opened a PR adding a <code>create_discussion</code> tool. It fetches the repository ID over GraphQL and then calls the <code>createDiscussion</code> mutation. If <code>repo</code> is omitted, the discussion is created at the organization level. The codebase moved to a new <code>NewTool()</code> pattern in the meantime, so I rewrote the tool during a rebase. The PR is open. I wrote up the full story in <a href="https://blog.furkankoykiran.com.tr/en/posts/github-mcp-server-contribution/" >a separate post</a>.</p>

<h3 class="relative group">--read-only in HTTP mode (#2200, closed)
    <div id="--read-only-in-http-mode-2200-closed" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#--read-only-in-http-mode-2200-closed" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>In <a href="https://github.com/github/github-mcp-server/issues/2156"  target="_blank" rel="noreferrer">#2156</a>, a user ran the server in Docker with <code>http --read-only</code> and still saw write tools like <code>create_branch</code> and <code>create_pull_request</code>. The flag worked in stdio mode but was never passed to the HTTP server's config. My PR added it to <code>ServerConfig</code> and registered a middleware that marks every request context as read-only:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">withGlobalReadonly</span><span class="p">(</span><span class="nx">cfg</span><span class="w"> </span><span class="o">*</span><span class="nx">ServerConfig</span><span class="p">)</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="p">)</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="k">return</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">next</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="p">)</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="k">return</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nf">HandlerFunc</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">			</span><span class="k">if</span><span class="w"> </span><span class="nx">cfg</span><span class="p">.</span><span class="nx">ReadOnly</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">				</span><span class="nx">ctx</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">ghcontext</span><span class="p">.</span><span class="nf">WithReadonly</span><span class="p">(</span><span class="nx">r</span><span class="p">.</span><span class="nf">Context</span><span class="p">(),</span><span class="w"> </span><span class="kc">true</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">				</span><span class="nx">next</span><span class="p">.</span><span class="nf">ServeHTTP</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nf">WithContext</span><span class="p">(</span><span class="nx">ctx</span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">			</span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">				</span><span class="nx">next</span><span class="p">.</span><span class="nf">ServeHTTP</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">			</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="p">})</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>The maintainer, Sam Morrow, replied that this wasn't a bug but an open design decision. In HTTP mode, each user can enable read-only mode with an <code>x-mcp-readonly: true</code> header, the same way GitHub's hosted remote server works. I made the case for a server-level setting: consistency with stdio, clients that can't send custom headers, and the Docker setup from the issue.</p>
<p>Sam then opened his own PR. <a href="https://github.com/github/github-mcp-server/pull/2208"  target="_blank" rel="noreferrer">#2208</a> treats the CLI flags as an upper bound, and per-request headers can only narrow it. That's a cleaner model than my middleware. I confirmed it fixed the Docker case. #2208 was merged on April 13, and my PR was closed.</p>
<p>My middleware never landed, and the behavior I asked for shipped with #2208. The PR taught me that a bug report is sometimes a design question, and arguing it through is part of the contribution.</p>

<h2 class="relative group">LobeHub: a one-line regression fix (#12818, open)
    <div id="lobehub-a-one-line-regression-fix-12818-open" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#lobehub-a-one-line-regression-fix-12818-open" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>On LobeHub's onboarding page, opening the model picker crashed the app with React Error #185. I reported it in <a href="https://github.com/lobehub/lobehub/issues/12817"  target="_blank" rel="noreferrer">#12817</a>. Digging through history, I found the same crash had been fixed in January by <a href="https://github.com/lobehub/lobehub/pull/11905"  target="_blank" rel="noreferrer">#11905</a>, and the fix was lost during the base-ui migration: <code>showInfoTag</code> on <code>ModelItemRender</code> was back to <code>true</code>. My PR restores it:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-tsx" data-lang="tsx"><span class="line"><span class="cl"><span class="p">&lt;</span><span class="nt">ModelItemRender</span>
</span></span><span class="line"><span class="cl">  <span class="p">{</span><span class="na">...</span><span class="err">(</span><span class="na">option</span> <span class="na">as</span> <span class="na">ModelOption</span><span class="err">)</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">{</span><span class="na">...</span><span class="err">(</span><span class="na">option</span> <span class="na">as</span> <span class="na">ModelOption</span><span class="err">)</span><span class="na">.abilities</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="na">showInfoTag</span><span class="o">=</span><span class="p">{</span><span class="kc">false</span><span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">/&gt;</span></span></span></code></pre></div></div>
<p>One maintainer commented that it &quot;seems still buggy&quot;. I asked which scenario still failed and noted that the crash was gone in the Vercel preview and locally. There was no reply. Another user had mentioned in the issue that switching models from the top menu crashed too, so my one line may not cover every path. The PR is still open; the issue was closed later.</p>
<p>Around the same time I opened <a href="https://github.com/lobehub/lobehub/issues/12805"  target="_blank" rel="noreferrer">#12805</a>, asking for some of my MCP servers, DevTo-MCP and OmniWire-MCP among them, to be listed in the LobeHub marketplace. It was closed as not planned.</p>

<h2 class="relative group">What I took away
    <div id="what-i-took-away" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-i-took-away" aria-label="Anchor">#</a>
    </span>
    
</h2>
<ul>
<li>Fix the problem where the data is produced, not where the error shows up. That's what got the Unicode PR merged.</li>
<li>A short diff with hand-written tests gets reviewed faster.</li>
<li>Ask before calling something a bug. The read-only flag was a design decision.</li>
<li>Even a one-line fix needs evidence: reproduction steps, before and after screenshots, or a test.</li>
</ul>

<h2 class="relative group">Where to go from here
    <div id="where-to-go-from-here" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-to-go-from-here" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>My own MCP servers are <a href="https://furkankoykiran.com.tr/en/projects/devto-mcp/"  target="_blank" rel="noreferrer">DevTo-MCP</a> and <a href="https://furkankoykiran.com.tr/en/projects/omniwire-mcp/"  target="_blank" rel="noreferrer">OmniWire-MCP</a>. The other PRs from the same weeks are in <a href="https://blog.furkankoykiran.com.tr/en/posts/security-and-api-fixes/" >security and API fixes</a> and <a href="https://blog.furkankoykiran.com.tr/en/posts/python-sdk-and-cli-contributions/" >SDK and CLI contributions</a>, and there's a Go-focused one in <a href="https://blog.furkankoykiran.com.tr/en/posts/my-open-source-contribution-keploy/" >my Keploy contribution</a>. If you want to contribute to the MCP ecosystem yourself, Playwright's <code>tests/mcp</code> folder is a good place to start: the tests are short and adding a new one is easy.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/mcp-ai-open-source-contributions/feature_hu_df20567488fcc09.jpg" medium="image" type="image/jpeg"/></item><item><title>AI Face Detector: Spotting AI-Made Faces with MobileNetV2</title><link>https://blog.furkankoykiran.com.tr/en/posts/ai-face-detector/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/ai-face-detector/</guid><pubDate>Sat, 07 Mar 2026 18:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Projects</category><category>python</category><category>fastapi</category><category>api</category><category>open-source</category><description>AI Face Detector classifies a face photo as real or StyleGAN-generated: MobileNetV2 transfer learning, a FastAPI service, and an honest look at its limits.</description><content:encoded><![CDATA[<p><a href="https://github.com/furkankoykiran/ai-face-detector"  target="_blank" rel="noreferrer">AI Face Detector</a> is an open source project that predicts whether a face photo is real or AI-generated. It's a MobileNetV2 classifier trained with PyTorch, served through a FastAPI REST API, with a simple drag-and-drop page styled with Tailwind CSS. I built it in March 2026 and released it under the MIT license.</p>
<p>This post covers how I trained the model, how the API works, and, most importantly, what the model does and doesn't know.</p>

<h2 class="relative group">What it does and what it doesn't
    <div id="what-it-does-and-what-it-doesnt" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-it-does-and-what-it-doesnt" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>For a single face image, the model returns one of two classes: <code>REAL</code> or <code>AI_GENERATED</code>. The real faces in the training data come from FFHQ and the fake ones from StyleGAN. That has three consequences worth stating up front:</p>
<ul>
<li>The model was trained to spot fully synthetic, StyleGAN-style faces. Images from diffusion models such as Stable Diffusion or Midjourney aren't in the training set. The repo description mentions them, but there's no measured accuracy on those generators.</li>
<li>Face swaps in video, the classic deepfake, are out of scope. The model looks at one still image.</li>
<li>A classifier's output is a probability. Don't use it as sole evidence for identity verification or fact-checking.</li>
</ul>
<p>The banner at the top of the repo README names diffusion models next to GANs. The limit above still applies: the only measured results are on StyleGAN faces.</p>

<h2 class="relative group">Architecture
    <div id="architecture" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#architecture" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The card below follows one request: the uploaded file is validated, resized to 224×224 and normalized, passed through the model, and returned as JSON.</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/ai-face-detector/img/detect-flow_hu_7dab1e3570633457.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/ai-face-detector/img/detect-flow_hu_97ee8235aa90e867.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/ai-face-detector/img/detect-flow_hu_eae06c1792f7fe5c.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/ai-face-detector/img/detect-flow_hu_5e54bd2ff2a38d75.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/ai-face-detector/img/detect-flow_hu_db83511327e10e7f.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/ai-face-detector/img/detect-flow_hu_d6f860ac0152021d.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/ai-face-detector/img/detect-flow_hu_db83511327e10e7f.webp" alt="Steps of a /detect request: POST up to 5 MB, resize to 224×224 and normalize, a MobileNetV2 logit, sigmoid below 0.5 means AI_GENERATED" width="1600" height="1234"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>The path of a /detect request</figcaption></figure><p>The model is loaded once at startup in FastAPI's <code>lifespan</code> handler. If <code>model.pth</code> is missing, the API still starts but <code>/detect</code> returns 503, and <code>/health</code> reports whether the model is loaded.</p>

<h2 class="relative group">Dataset
    <div id="dataset" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#dataset" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>I trained on the <a href="https://www.kaggle.com/datasets/xhlulu/140k-real-and-fake-faces"  target="_blank" rel="noreferrer">140k Real and Fake Faces</a> dataset from Kaggle: 70,000 real faces from FFHQ and 70,000 StyleGAN faces, all 256×256. The provided split is 100,000 training, 20,000 validation and 20,000 test images. With balanced classes, the model can't score well just by leaning towards one class.</p>

<h2 class="relative group">The model: transfer learning with MobileNetV2
    <div id="the-model-transfer-learning-with-mobilenetv2" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-model-transfer-learning-with-mobilenetv2" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Instead of training a CNN from scratch, I started from MobileNetV2 pretrained on ImageNet, froze the feature extractor, and replaced and trained only the final classifier. At about 3.5 million parameters it runs quickly on a CPU, and the weights file is around 14 MB.</p>
<p>The core of the model setup in <code>training/train.py</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">torch.nn</span> <span class="k">as</span> <span class="nn">nn</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">torchvision</span> <span class="kn">import</span> <span class="n">models</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">torchvision.models</span> <span class="kn">import</span> <span class="n">MobileNet_V2_Weights</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">create_model</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">model</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">mobilenet_v2</span><span class="p">(</span><span class="n">weights</span><span class="o">=</span><span class="n">MobileNet_V2_Weights</span><span class="o">.</span><span class="n">IMAGENET1K_V1</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">param</span> <span class="ow">in</span> <span class="n">model</span><span class="o">.</span><span class="n">features</span><span class="o">.</span><span class="n">parameters</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">        <span class="n">param</span><span class="o">.</span><span class="n">requires_grad</span> <span class="o">=</span> <span class="kc">False</span>  <span class="c1"># frozen backbone, only the head trains</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">in_features</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">classifier</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">in_features</span>  <span class="c1"># 1280</span>
</span></span><span class="line"><span class="cl">    <span class="n">model</span><span class="o">.</span><span class="n">classifier</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Sequential</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">nn</span><span class="o">.</span><span class="n">Dropout</span><span class="p">(</span><span class="n">p</span><span class="o">=</span><span class="mf">0.3</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">        <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="n">in_features</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span>  <span class="c1"># one logit: 0 = AI, 1 = REAL</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">model</span></span></span></code></pre></div></div>
<p>The rest of the training setup:</p>
<ul>
<li><code>BCEWithLogitsLoss</code> with <code>Adam</code> (learning rate 0.001, weight decay 1e-4).</li>
<li><code>ReduceLROnPlateau</code> divides the learning rate by 10 after 3 epochs without validation-loss improvement. Early stopping kicks in after 5. At most 15 epochs.</li>
<li>Inputs are resized to 224×224 and normalized with ImageNet mean and standard deviation. The only augmentation is a random horizontal flip; I removed heavier transforms from the script to keep training fast.</li>
<li>On Kaggle I used two T4 GPUs with <code>DataParallel</code>, mixed precision (<code>torch.amp</code>) and a batch size of 256. The script detects whether it runs on Kaggle, Colab or locally.</li>
</ul>

<h2 class="relative group">Results
    <div id="results" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#results" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The README reports these numbers on the dataset's test split (20,000 images):</p>
<div class="fk-table" role="region" tabindex="0" aria-label="Scrollable table">
<table>
  <thead>
    <tr>
      <th style="text-align:left">Metric</th>
      <th style="text-align:left">Value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align:left">Accuracy</td>
      <td style="text-align:left">94.5%</td>
    </tr>
    <tr>
      <td style="text-align:left">Precision</td>
      <td style="text-align:left">94.2%</td>
    </tr>
    <tr>
      <td style="text-align:left">Recall</td>
      <td style="text-align:left">94.8%</td>
    </tr>
    <tr>
      <td style="text-align:left">F1</td>
      <td style="text-align:left">0.945</td>
    </tr>
    <tr>
      <td style="text-align:left">AUC-ROC</td>
      <td style="text-align:left">0.978</td>
    </tr>
    <tr>
      <td style="text-align:left">Inference (CPU / GPU)</td>
      <td style="text-align:left">~45 ms / ~8 ms</td>
    </tr>
  </tbody>
</table>
</div><p>Keep in mind that the test set comes from the same distribution as the training set: the same StyleGAN, the same resolution, the same crops. I'd expect accuracy to drop on photos from another generator, or on images that were compressed or resized along the way. Building a separate test set to measure that is on my list.</p>

<h2 class="relative group">The API
    <div id="the-api" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-api" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The endpoint that matters is <code>POST /detect</code>. It accepts a JPG, PNG or WEBP file up to 5 MB:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">curl -X POST <span class="s2">&#34;http://localhost:8000/detect&#34;</span> -F <span class="s2">&#34;file=@face.jpg&#34;</span></span></span></code></pre></div></div>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;status&#34;</span><span class="p">:</span> <span class="s2">&#34;success&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;result&#34;</span><span class="p">:</span> <span class="s2">&#34;AI_GENERATED&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;confidence&#34;</span><span class="p">:</span> <span class="mf">0.98</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;probabilities&#34;</span><span class="p">:</span> <span class="p">{</span> <span class="nt">&#34;AI_GENERATED&#34;</span><span class="p">:</span> <span class="mf">0.98</span><span class="p">,</span> <span class="nt">&#34;REAL&#34;</span><span class="p">:</span> <span class="mf">0.02</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;inference_time_ms&#34;</span><span class="p">:</span> <span class="mf">45.2</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;image_size&#34;</span><span class="p">:</span> <span class="s2">&#34;512x512&#34;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>The model outputs a single logit. After the sigmoid, the value is the probability of <code>REAL</code>. Anything below 0.5 is returned as <code>AI_GENERATED</code>, and <code>confidence</code> is the probability of whichever class won.</p>

<h2 class="relative group">Running it locally
    <div id="running-it-locally" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#running-it-locally" aria-label="Anchor">#</a>
    </span>
    
</h2>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">git clone https://github.com/furkankoykiran/ai-face-detector.git
</span></span><span class="line"><span class="cl"><span class="nb">cd</span> ai-face-detector
</span></span><span class="line"><span class="cl">python -m venv .venv <span class="o">&amp;&amp;</span> <span class="nb">source</span> .venv/bin/activate
</span></span><span class="line"><span class="cl">pip install -r requirements.txt</span></span></code></pre></div></div>
<p>The trained weights aren't in the repo. You can download them from the <a href="https://www.kaggle.com/models/furkankoykiran/ai-face-detector-mobilenetv2"  target="_blank" rel="noreferrer">Kaggle model page</a>, or open a GPU notebook on Kaggle, attach the dataset and run <code>training/train.py</code>. According to the README, training on two T4s takes about 43 minutes. Put the resulting <code>model.pth</code> in the project root and start the API:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">uvicorn app.main:app --reload --port <span class="m">8000</span></span></span></code></pre></div></div>
<p>Swagger UI is at <code>http://localhost:8000/docs</code>. Serve the web page with a small server rather than opening the file directly, because the API's CORS list includes <code>http://localhost:3000</code> but not <code>file://</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">python -m http.server <span class="m">3000</span> --directory static</span></span></code></pre></div></div>
<p>The quickest way to try the model without installing anything is the <a href="https://www.kaggle.com/code/furkankoykiran/ai-face-detector-model-demo"  target="_blank" rel="noreferrer">demo notebook on Kaggle</a>. It attaches the trained model and the 140k dataset as inputs and runs on two T4 GPUs.</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/posts/ai-face-detector-yapay-zeka-ile-yuz-dogrulama/img/kaggle-demo-notebook_hu_8f3e2d86260ef0a2.avif 480w, https://blog.furkankoykiran.com.tr/posts/ai-face-detector-yapay-zeka-ile-yuz-dogrulama/img/kaggle-demo-notebook_hu_74d24e82d0f102cc.avif 800w, https://blog.furkankoykiran.com.tr/posts/ai-face-detector-yapay-zeka-ile-yuz-dogrulama/img/kaggle-demo-notebook_hu_59fedb8eaaab37da.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/posts/ai-face-detector-yapay-zeka-ile-yuz-dogrulama/img/kaggle-demo-notebook_hu_4d7863a0f78e7ac0.webp 480w, https://blog.furkankoykiran.com.tr/posts/ai-face-detector-yapay-zeka-ile-yuz-dogrulama/img/kaggle-demo-notebook_hu_9749538659e44061.webp 800w, https://blog.furkankoykiran.com.tr/posts/ai-face-detector-yapay-zeka-ile-yuz-dogrulama/img/kaggle-demo-notebook_hu_3d5f7c0345d67a51.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/posts/ai-face-detector-yapay-zeka-ile-yuz-dogrulama/img/kaggle-demo-notebook_hu_9749538659e44061.webp" alt="The AI Face Detector Model Demo notebook on Kaggle, with a GPU T4 x2 runtime and the 140k dataset and MobileNetV2 model attached as inputs" width="1250" height="800"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>The demo notebook on Kaggle. Source: Kaggle</figcaption></figure>
<h2 class="relative group">Known rough edges
    <div id="known-rough-edges" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#known-rough-edges" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Rereading the repo, I found three things to fix:</p>
<ul>
<li><code>app/model_loader.py</code> still uses the old <code>pretrained=True</code> argument, which torchvision deprecated in 0.13. Since we load our own weights anyway, there's no reason to download the ImageNet weights on every startup.</li>
<li>Training on two GPUs with <code>DataParallel</code> saves <code>state_dict</code> keys with a <code>module.</code> prefix. The API doesn't strip it, so it can't load such a file.</li>
<li><code>requirements.txt</code> pins early-2024 versions (torch 2.1, FastAPI 0.109).</li>
</ul>
<p>For the first two, the loader should look like this:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">torch</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">torch.nn</span> <span class="k">as</span> <span class="nn">nn</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">torchvision</span> <span class="kn">import</span> <span class="n">models</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">create_model</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">model</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">mobilenet_v2</span><span class="p">(</span><span class="n">weights</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>  <span class="c1"># weights come from our file</span>
</span></span><span class="line"><span class="cl">    <span class="n">in_features</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">classifier</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">in_features</span>
</span></span><span class="line"><span class="cl">    <span class="n">model</span><span class="o">.</span><span class="n">classifier</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Sequential</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">Dropout</span><span class="p">(</span><span class="n">p</span><span class="o">=</span><span class="mf">0.3</span><span class="p">),</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="n">in_features</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">model</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">load_model</span><span class="p">(</span><span class="n">path</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">device</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">device</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">state</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">map_location</span><span class="o">=</span><span class="n">device</span><span class="p">,</span> <span class="n">weights_only</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">state</span> <span class="o">=</span> <span class="p">{</span><span class="n">k</span><span class="o">.</span><span class="n">removeprefix</span><span class="p">(</span><span class="s2">&#34;module.&#34;</span><span class="p">):</span> <span class="n">v</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">state</span><span class="o">.</span><span class="n">items</span><span class="p">()}</span>
</span></span><span class="line"><span class="cl">    <span class="n">model</span> <span class="o">=</span> <span class="n">create_model</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">model</span><span class="o">.</span><span class="n">load_state_dict</span><span class="p">(</span><span class="n">state</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">model</span><span class="o">.</span><span class="n">to</span><span class="p">(</span><span class="n">device</span><span class="p">)</span><span class="o">.</span><span class="n">eval</span><span class="p">()</span></span></span></code></pre></div></div>
<p><code>weights_only=True</code> has been the default since PyTorch 2.6. It stops a <code>.pth</code> file from an unknown source from executing code on load, so I prefer to spell it out.</p>

<h2 class="relative group">What's next
    <div id="whats-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#whats-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The code is <a href="https://github.com/furkankoykiran/ai-face-detector"  target="_blank" rel="noreferrer">on GitHub</a>, and there's a summary on <a href="https://furkankoykiran.com.tr/en/projects/ai-face-detector/"  target="_blank" rel="noreferrer">my portfolio</a>. Next up: fix the rough edges above and build a separate test set of faces from diffusion models. If the model holds up there, it will be fair to widen the claim in the README.</p>
<p>If you're interested in the API side, <a href="https://blog.furkankoykiran.com.tr/en/posts/python-fastapi-rest-api/" >REST API development with FastAPI</a> goes deeper into Pydantic v2 and async. For preprocessing, see <a href="https://blog.furkankoykiran.com.tr/en/posts/python-image-processing-pillow-opencv/" >image processing with Pillow and OpenCV</a>, and for shipping the service, <a href="https://blog.furkankoykiran.com.tr/en/posts/docker-python-app-containerization/" >containerizing a Python app with Docker</a>.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/ai-face-detector/feature_hu_d45d696a86f9e44c.jpg" medium="image" type="image/jpeg"/></item><item><title>Letting an AI Agent Publish My Gemini Challenge Entry</title><link>https://blog.furkankoykiran.com.tr/en/posts/mcp-gemini-challenge-experience/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/mcp-gemini-challenge-experience/</guid><pubDate>Fri, 27 Feb 2026 15:20:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Stories</category><category>mcp</category><category>ai-agents</category><category>llm</category><category>automation</category><description>For the Built with Google Gemini challenge, an AI agent wrote my entry and published it to DEV through DevTo-MCP. What worked, and the errors that slipped in.</description><content:encoded><![CDATA[<p>In February 2026 DEV Community ran the &quot;Built with Google Gemini: Writing Challenge,&quot; presented by Major League Hacking (MLH). To enter, you published a post about something you'd built with Gemini. A few days earlier I had written an MCP server for the DEV API, so I decided to have an AI agent write my entry and publish it to DEV through that server.</p>
<p>This post covers that experiment: what the agent did, what went well, and the errors I only noticed later.</p>

<h2 class="relative group">The tool: DevTo-MCP
    <div id="the-tool-devto-mcp" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-tool-devto-mcp" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p><a href="https://blog.furkankoykiran.com.tr/en/posts/devto-mcp-server/" >DevTo-MCP</a> is my server that turns the Forem API into MCP tools. With it an assistant can search DEV, list my posts and create new ones with <code>create_article</code>. While testing it on February 18 I had created and updated a throwaway post titled &quot;DevTo MCP Test Article [Automated]&quot; on my account. The challenge was the first time I used it for real work.</p>
<p>Project page: <a href="https://furkankoykiran.com.tr/en/projects/devto-mcp/"  target="_blank" rel="noreferrer">DevTo-MCP</a></p>

<h2 class="relative group">The task I gave the agent
    <div id="the-task-i-gave-the-agent" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-task-i-gave-the-agent" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Without leaving my editor, I gave a Gemini-based agent roughly this instruction:</p>
<blockquote><p>Research the Built with Google Gemini challenge. Read the DevTo-MCP post on my blog. Write an English article about the process that follows the challenge template. When you're done, publish it on my DEV account.</p>
</blockquote><p>The agent then:</p>
<ol>
<li>Read the challenge announcement and submission template, which asked for sections like &quot;What I Built with Google Gemini,&quot; &quot;Demo&quot; and what you learned.</li>
<li>Read my blog post about DevTo-MCP and mapped its technical content onto the template.</li>
<li>Wrote the article and added a closing note saying the post had been drafted and published autonomously through DevTo-MCP.</li>
<li>Called <code>create_article</code> with <code>published: true</code>.</li>
</ol>
<p>The article <a href="https://dev.to/furkankoykiran/building-devto-mcp-bridging-forem-api-and-ai-assistants-with-google-gemini-4a58"  target="_blank" rel="noreferrer">went live on DEV</a> on February 27, 2026, titled &quot;Building DevTo-MCP: Bridging Forem API and AI Assistants with Google Gemini&quot; and tagged <code>devchallenge</code>, <code>geminireflections</code>, <code>gemini</code> and <code>mcp</code>. The whole run took a few minutes.</p>
<p>Here is the whole chain, from one instruction to a published post:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/mcp-gemini-challenge-experience/img/agent-publish-steps_hu_9b1acdba7604f766.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/mcp-gemini-challenge-experience/img/agent-publish-steps_hu_3da7822f36410b21.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/mcp-gemini-challenge-experience/img/agent-publish-steps_hu_1a069f6fde40e4cf.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/mcp-gemini-challenge-experience/img/agent-publish-steps_hu_4adff731dc85221a.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/mcp-gemini-challenge-experience/img/agent-publish-steps_hu_b2fadaa5d992c11.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/mcp-gemini-challenge-experience/img/agent-publish-steps_hu_64ba81f72bf2fca8.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/mcp-gemini-challenge-experience/img/agent-publish-steps_hu_b2fadaa5d992c11.webp" alt="The agent read the challenge template, read my DevTo-MCP post, wrote the entry and called create_article with published: true" width="1600" height="1134"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Four steps from one instruction; the last one published the post without a review.</figcaption></figure>
<h2 class="relative group">What went well
    <div id="what-went-well" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-went-well" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Mechanically, everything worked. The agent's <code>create_article</code> call went through on the first try. It read the template correctly, put the submission line where it belonged and used the challenge's tags.</p>
<p>This kind of chain is where an MCP server earns its keep: research, reading, writing and publishing tied together by one instruction. I stated the goal and the agent planned the steps in between.</p>

<h2 class="relative group">What slipped through
    <div id="what-slipped-through" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-slipped-through" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The DevTo-MCP write-up the agent used as its source made several claims the code doesn't back up. Since the agent built the challenge entry from that write-up, the same errors went into the entry:</p>
<ul>
<li>It says the API client reads <code>x-ratelimit-remaining</code> and slows down before hitting the limit. The client doesn't; it only uses the header in error messages.</li>
<li>It says the npm release used OIDC trusted publishing, that a missing <code>environment: production</code> line caused 404s, and that Gemini spotted the missing line. The workflow in the repo has had that line since its first commit, and publishing uses an npm token.</li>
<li>The install section uses <code>npx -y devto-mcp</code>. In March I moved the package to <code>@furkankoykiran/devto-mcp</code>, and the unscoped <code>devto-mcp</code> name now belongs to a different project on npm. Anyone running that command installs someone else's package and hands it their DEV API key.</li>
</ul>
<p>The last one is the serious one. The command was right on the day the agent wrote it; the package name changed later. Code gets updated, but a published post stays as it was.</p>
<p>The agent also framed the article as my development story, with Gemini as a pair programmer at every step. The template invited that framing, and the agent didn't hesitate to draw connections the source didn't contain. Anything published under my name has to be something I can stand behind sentence by sentence, and that entry has sentences I can't.</p>

<h2 class="relative group">What I'll do differently
    <div id="what-ill-do-differently" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-ill-do-differently" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>I won't let the agent publish. <code>create_article</code> has a <code>published: false</code> option: the agent creates a draft, and I read it in the DEV editor and publish it myself. The minutes saved by automation easily cover a few minutes of reading.</p>
<p>I'll verify the source first. The agent worked from a blog post, so every error in that post carried over. A model that rewrites a flawed source quickly and fluently makes the flaws more convincing.</p>
<p>The server can help too: default <code>create_article</code> to drafts and require an explicit confirmation parameter to publish. Not making a write tool's least reversible side effect its default is a rule I want to follow in every MCP server I design.</p>

<h2 class="relative group">Where this leaves me
    <div id="where-this-leaves-me" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-this-leaves-me" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The experiment showed that an agent can do a job end to end over MCP. It also showed that an agent with write access publishes whatever its source says. My next steps are switching DevTo-MCP's default to drafts and adding a correction note with the current install command to the DEV article.</p>
<p>The technical details of the server are in <a href="https://blog.furkankoykiran.com.tr/en/posts/devto-mcp-server/" >the DevTo-MCP post</a>, and another server from the same series is <a href="https://blog.furkankoykiran.com.tr/en/posts/omniwire-mcp-news-server/" >OmniWire-MCP</a>. The code is <a href="https://github.com/furkankoykiran/DevTo-MCP"  target="_blank" rel="noreferrer">on GitHub</a>.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/mcp-gemini-challenge-experience/feature_hu_ba7e747b34a1f416.jpg" medium="image" type="image/jpeg"/></item><item><title>Telegram Wallet P2P SDKs for Python, Node.js and MCP</title><link>https://blog.furkankoykiran.com.tr/en/posts/telegram-wallet-p2p-mcp-sdk/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/telegram-wallet-p2p-mcp-sdk/</guid><pubDate>Tue, 24 Feb 2026 19:30:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Projects</category><category>telegram</category><category>python</category><category>typescript</category><category>mcp</category><category>trading</category><category>asyncio</category><description>Unofficial Python and Node.js SDKs plus an MCP server for Telegram Wallet's read-only P2P market API, with notes on decimals, pagination and rate limits.</description><content:encoded><![CDATA[<p>Wallet, the crypto wallet built into Telegram, opened its P2P market to an API. You can pull active ads, buy and sell prices, fiat and crypto currencies, maker details and payment methods without opening the app. I wrote two unofficial SDKs for that API, one for Python and one for Node.js, plus an MCP server so assistants like Claude can query it. All three live in one MIT-licensed monorepo.</p>
<p>Project page: <a href="https://furkankoykiran.com.tr/en/projects/telegram-wallet-p2p-sdk/"  target="_blank" rel="noreferrer">telegram-wallet-p2p-sdk</a> · Source: <a href="https://github.com/furkankoykiran/telegram-wallet-p2p-sdk"  target="_blank" rel="noreferrer">GitHub</a></p>
<p>The project isn't affiliated with Telegram or Wallet. And the API is strictly read-only: you can list ads, but you can't post ads or trade. This is a market data and analytics toolkit, not a trading bot.</p>

<h2 class="relative group">What the API looks like
    <div id="what-the-api-looks-like" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-the-api-looks-like" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>There's a single endpoint. You POST to <code>https://p2p.walletbot.me/p2p/integration-api/v1/item/online</code> with an <code>X-API-Key</code> header:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;cryptoCurrency&#34;</span><span class="p">:</span> <span class="s2">&#34;USDT&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;fiatCurrency&#34;</span><span class="p">:</span> <span class="s2">&#34;EUR&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;side&#34;</span><span class="p">:</span> <span class="s2">&#34;SELL&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;page&#34;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;pageSize&#34;</span><span class="p">:</span> <span class="mi">50</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>The response is <code>{&quot;status&quot;: &quot;SUCCESS&quot;, &quot;data&quot;: [...]}</code>. Each ad carries the price, remaining quantity, min and max order amounts, payment methods, the maker's completed order count and completion rate, merchant level (regular user, merchant, trusted merchant) and whether auto-accept is on. Pages hold at most 50 ads.</p>
<p>You create an API key inside Telegram under Wallet → P2P Market → your profile → API Keys. The key doesn't come from browser DevTools; you create it in that menu.</p>

<h2 class="relative group">Why a monorepo
    <div id="why-a-monorepo" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-a-monorepo" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>All three packages wrap the same API, and their models are coupled. When the API adds a field, the Python model, the TypeScript types and the MCP server should change in the same PR. Across separate repos that sync drifts quickly. The card shows the three packages and where each one is used.</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/telegram-wallet-p2p-mcp-sdk/img/monorepo-packages_hu_ec82b4841930d42e.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/telegram-wallet-p2p-mcp-sdk/img/monorepo-packages_hu_39b30cc42e5bbf84.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/telegram-wallet-p2p-mcp-sdk/img/monorepo-packages_hu_f2b52a22dd0774d3.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/telegram-wallet-p2p-mcp-sdk/img/monorepo-packages_hu_ec4458614c164136.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/telegram-wallet-p2p-mcp-sdk/img/monorepo-packages_hu_a700d514371b03ee.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/telegram-wallet-p2p-mcp-sdk/img/monorepo-packages_hu_1d04ee932894efd5.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/telegram-wallet-p2p-mcp-sdk/img/monorepo-packages_hu_a700d514371b03ee.webp" alt="Three packages around one API: Python with aiohttp and Pydantic v2, dependency-free Node.js, and an MCP server with the MCP SDK and Zod" width="1600" height="1162"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Three packages wrapping one API</figcaption></figure><p>The published packages:</p>
<ul>
<li>PyPI: <code>telegram-wallet-p2p</code></li>
<li>npm: <code>@furkankoykiran/telegram-wallet-p2p</code></li>
<li>npm: <code>@furkankoykiran/telegram-wallet-p2p-mcp</code></li>
</ul>
<p>All three live in the repo's <code>packages</code> folder.</p>
<p>As the card shows, the MCP server doesn't depend on the Node SDK; it keeps its own types and HTTP call. That was deliberate: <code>npx</code> downloads a single package, and there's no workspace dependency to complicate publish order. The cost is the same types living in two places.</p>

<h2 class="relative group">Why prices are strings
    <div id="why-prices-are-strings" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-prices-are-strings" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The API returns prices, quantities and rates as strings, and both SDKs keep them that way:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ts" data-lang="ts"><span class="line"><span class="cl"><span class="kr">export</span> <span class="kr">interface</span> <span class="nx">OnlineItem</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nx">id</span>: <span class="kt">string</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">price</span>: <span class="kt">string</span><span class="p">;</span>          <span class="c1">// price per unit
</span></span></span><span class="line"><span class="cl">  <span class="nx">lastQuantity</span>: <span class="kt">string</span><span class="p">;</span>   <span class="c1">// remaining quantity
</span></span></span><span class="line"><span class="cl">  <span class="nx">minAmount</span>: <span class="kt">string</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">maxAmount</span>: <span class="kt">string</span> <span class="o">|</span> <span class="kc">null</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">executeRate</span>: <span class="kt">string</span><span class="p">;</span>    <span class="c1">// completion rate, 0-1
</span></span></span><span class="line"><span class="cl">  <span class="nx">payments</span>: <span class="kt">string</span><span class="p">[];</span>
</span></span><span class="line"><span class="cl">  <span class="nx">merchantLevel</span>: <span class="kt">MerchantLevel</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">isAutoAccept</span>: <span class="kt">boolean</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>Everyone knows <code>0.1 + 0.2</code> is <code>0.30000000000000004</code> in JavaScript. Convert a price to <code>number</code> once and that error leaks into sums and comparisons. Store strings and do the math with a decimal type.</p>
<p>The Python analytics helpers actually do this; every calculation uses <code>decimal.Decimal</code>. The Node side has an inconsistency: helpers like <code>priceSpread</code> and <code>bestPrice</code> convert prices with <code>parseFloat</code>. That's acceptable for statistics like mean and median, which get rounded to four decimals anyway, but don't use their output for accounting or order sizing. Moving the Node helpers to a decimal library is on my list.</p>
<p>The Node SDK doesn't validate API responses with Zod. It doesn't use Zod at all; the types are plain TypeScript interfaces with no runtime validation. Zod is only in the MCP server, for tool arguments. On the Python side, Pydantic v2 models do validate responses and map camelCase fields to snake_case.</p>

<h2 class="relative group">The Node.js SDK
    <div id="the-nodejs-sdk" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-nodejs-sdk" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The Node SDK has no runtime dependencies. It uses the <code>fetch</code> built into Node since 18:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ts" data-lang="ts"><span class="line"><span class="cl"><span class="kr">import</span> <span class="p">{</span> <span class="nx">WalletP2PClient</span><span class="p">,</span> <span class="nx">TradeSide</span><span class="p">,</span> <span class="nx">bestPrice</span><span class="p">,</span> <span class="nx">priceSpread</span> <span class="p">}</span> <span class="kr">from</span> <span class="s2">&#34;@furkankoykiran/telegram-wallet-p2p&#34;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">client</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">WalletP2PClient</span><span class="p">({</span> <span class="nx">apiKey</span>: <span class="kt">process.env.WALLET_P2P_API_KEY</span><span class="o">!</span> <span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="p">{</span> <span class="nx">data</span> <span class="p">}</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">client</span><span class="p">.</span><span class="nx">getOnlineItems</span><span class="p">({</span>
</span></span><span class="line"><span class="cl">  <span class="nx">cryptoCurrency</span><span class="o">:</span> <span class="s2">&#34;USDT&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">fiatCurrency</span><span class="o">:</span> <span class="s2">&#34;EUR&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">side</span>: <span class="kt">TradeSide.SELL</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">pageSize</span>: <span class="kt">50</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">priceSpread</span><span class="p">(</span><span class="nx">data</span><span class="p">));</span>
</span></span><span class="line"><span class="cl"><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">bestPrice</span><span class="p">(</span><span class="nx">data</span><span class="p">,</span> <span class="nx">TradeSide</span><span class="p">.</span><span class="nx">SELL</span><span class="p">)</span><span class="o">?</span><span class="p">.</span><span class="nx">price</span><span class="p">);</span></span></span></code></pre></div></div>
<p>Top-level <code>await</code> needs an ES module (<code>.mjs</code>, <code>.mts</code>, or <code>&quot;type&quot;: &quot;module&quot;</code> in <code>package.json</code>).</p>
<p>The client retries 429 and 5xx responses on its own: 3 attempts by default with exponential backoff of 1, 2 and 4 seconds. Other failures arrive as typed errors: <code>AuthenticationError</code> for a bad key, <code>BadRequestError</code> for an unsupported currency, <code>RateLimitError</code> once retries run out. Requests time out after 30 seconds via <code>AbortController</code>.</p>

<h2 class="relative group">The Python SDK
    <div id="the-python-sdk" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-python-sdk" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The Python SDK is built on <code>aiohttp</code> and used as an async context manager. Scanning several currencies at once is a job for <code>asyncio.gather</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">asyncio</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">os</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">telegram_wallet_p2p</span> <span class="kn">import</span> <span class="n">TradeSide</span><span class="p">,</span> <span class="n">WalletP2PClient</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">telegram_wallet_p2p.analytics</span> <span class="kn">import</span> <span class="n">price_spread</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">scan</span><span class="p">(</span><span class="n">client</span><span class="p">:</span> <span class="n">WalletP2PClient</span><span class="p">,</span> <span class="n">fiat</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">resp</span> <span class="o">=</span> <span class="k">await</span> <span class="n">client</span><span class="o">.</span><span class="n">get_online_items</span><span class="p">(</span><span class="s2">&#34;USDT&#34;</span><span class="p">,</span> <span class="n">fiat</span><span class="p">,</span> <span class="n">TradeSide</span><span class="o">.</span><span class="n">SELL</span><span class="p">,</span> <span class="n">page_size</span><span class="o">=</span><span class="mi">50</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">fiat</span><span class="p">,</span> <span class="n">price_spread</span><span class="p">(</span><span class="n">resp</span><span class="o">.</span><span class="n">data</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">main</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">with</span> <span class="n">WalletP2PClient</span><span class="p">(</span><span class="n">api_key</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">&#34;WALLET_P2P_API_KEY&#34;</span><span class="p">])</span> <span class="k">as</span> <span class="n">client</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">results</span> <span class="o">=</span> <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span><span class="o">*</span><span class="p">(</span><span class="n">scan</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="n">f</span><span class="p">)</span> <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="p">[</span><span class="s2">&#34;EUR&#34;</span><span class="p">,</span> <span class="s2">&#34;USD&#34;</span><span class="p">,</span> <span class="s2">&#34;TRY&#34;</span><span class="p">]))</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">fiat</span><span class="p">,</span> <span class="n">stats</span> <span class="ow">in</span> <span class="n">results</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="n">fiat</span><span class="p">,</span> <span class="n">stats</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&#34;__main__&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">asyncio</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">main</span><span class="p">())</span></span></span></code></pre></div></div>
<p>The client shares one <code>aiohttp.ClientSession</code> and closes it when the <code>async with</code> block exits. Opening a session per request wastes the connection pool, so create the client once and reuse it.</p>
<p>This isn't &quot;high-frequency data collection.&quot; You're sending a handful of parallel requests and you have to respect the API's rate limit. Polling every minute, or every few minutes, and writing the results to a time-series database is plenty to see how P2P spreads move during the day.</p>

<h2 class="relative group">Pagination and rate limits
    <div id="pagination-and-rate-limits" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#pagination-and-rate-limits" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>To walk the whole order book at 50 ads per page you request pages in sequence. You don't need your own 429 handling because the client already retries; a short pause between pages keeps you well clear of the limit. Here's the loop:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ts" data-lang="ts"><span class="line"><span class="cl"><span class="kr">import</span> <span class="p">{</span> <span class="nx">WalletP2PClient</span><span class="p">,</span> <span class="nx">TradeSide</span><span class="p">,</span> <span class="kr">type</span> <span class="nx">OnlineItem</span> <span class="p">}</span> <span class="kr">from</span> <span class="s2">&#34;@furkankoykiran/telegram-wallet-p2p&#34;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">client</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">WalletP2PClient</span><span class="p">({</span> <span class="nx">apiKey</span>: <span class="kt">process.env.WALLET_P2P_API_KEY</span><span class="o">!</span> <span class="p">});</span>
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">sleep</span> <span class="o">=</span> <span class="p">(</span><span class="nx">ms</span>: <span class="kt">number</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="k">new</span> <span class="nx">Promise</span><span class="p">((</span><span class="nx">resolve</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">setTimeout</span><span class="p">(</span><span class="nx">resolve</span><span class="p">,</span> <span class="nx">ms</span><span class="p">));</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">async</span> <span class="kd">function</span> <span class="nx">fetchAll</span><span class="p">(</span><span class="nx">crypto</span>: <span class="kt">string</span><span class="p">,</span> <span class="nx">fiat</span>: <span class="kt">string</span><span class="p">,</span> <span class="nx">side</span>: <span class="kt">TradeSide</span><span class="p">)</span><span class="o">:</span> <span class="nx">Promise</span><span class="p">&lt;</span><span class="nt">OnlineItem</span><span class="err">[]</span><span class="p">&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">all</span>: <span class="kt">OnlineItem</span><span class="p">[]</span> <span class="o">=</span> <span class="p">[];</span>
</span></span><span class="line"><span class="cl">  <span class="k">for</span> <span class="p">(</span><span class="kd">let</span> <span class="nx">page</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> <span class="p">;</span> <span class="nx">page</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="kr">const</span> <span class="p">{</span> <span class="nx">data</span> <span class="p">}</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">client</span><span class="p">.</span><span class="nx">getOnlineItems</span><span class="p">({</span>
</span></span><span class="line"><span class="cl">      <span class="nx">cryptoCurrency</span>: <span class="kt">crypto</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nx">fiatCurrency</span>: <span class="kt">fiat</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nx">side</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nx">page</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nx">pageSize</span>: <span class="kt">50</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">});</span>
</span></span><span class="line"><span class="cl">    <span class="nx">all</span><span class="p">.</span><span class="nx">push</span><span class="p">(...</span><span class="nx">data</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="p">(</span><span class="nx">data</span><span class="p">.</span><span class="nx">length</span> <span class="o">&lt;</span> <span class="mi">50</span><span class="p">)</span> <span class="k">break</span><span class="p">;</span> <span class="c1">// last page
</span></span></span><span class="line"><span class="cl">    <span class="k">await</span> <span class="nx">sleep</span><span class="p">(</span><span class="mi">500</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="k">return</span> <span class="nx">all</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>Don't try to catch 429s with <code>error.message.includes('429')</code>. This SDK's error messages don't include the status code, so that branch never fires; use <code>error instanceof RateLimitError</code> if you need it. Using <code>push</code> instead of rebuilding the array with <code>[...all, ...page]</code> on every page also avoids needless copying on deep books.</p>

<h2 class="relative group">The MCP server
    <div id="the-mcp-server" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-mcp-server" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The MCP server exposes three tools:</p>
<ul>
<li><code>get_p2p_ads</code> lists ads by crypto, fiat and side.</li>
<li><code>get_market_summary</code> returns price statistics, payment method and merchant level breakdowns, and the average completion rate.</li>
<li><code>get_best_price</code> returns the best-priced ad for a side.</li>
</ul>
<p>Claude Desktop config:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;mcpServers&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;telegram-wallet-p2p&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;command&#34;</span><span class="p">:</span> <span class="s2">&#34;npx&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;args&#34;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&#34;-y&#34;</span><span class="p">,</span> <span class="s2">&#34;@furkankoykiran/telegram-wallet-p2p-mcp&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;env&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="nt">&#34;WALLET_P2P_API_KEY&#34;</span><span class="p">:</span> <span class="s2">&#34;your-key&#34;</span>
</span></span><span class="line"><span class="cl">      <span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>Then you can ask things like &quot;What's the cheapest USDT I can buy with EUR right now?&quot; or &quot;Summarize the TON/USD market. How many sellers are merchants?&quot; The assistant fetches the data and does the filtering itself.</p>
<p>The meaning of <code>side</code> isn't documented consistently across the packages. The MCP tool describes it from the ad maker's perspective, while the analytics docs describe it from the taker's. <code>bestPrice</code> picks the lowest price for BUY and the highest for SELL. Before trusting &quot;best&quot; from these helpers, compare a few ads against what the app shows to confirm which side means what. Pinning down one definition and applying it to all three packages is my next task.</p>

<h2 class="relative group">What's next
    <div id="whats-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#whats-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Short term, the two fixes above: one definition of <code>side</code>, and decimal math in the Node helpers. The Python package's <code>__version__</code> also disagrees with <code>pyproject.toml</code>, which needs fixing.</p>
<p>If Wallet ever adds write access to the API, order creation would be the obvious addition, but no such endpoint exists today and I'm not writing code against an API that isn't there. A short-lived cache for frequently queried pairs is an easy way to stay further under the rate limit.</p>
<p>I cover parallel requests, timeouts and concurrency limits in more depth in my <a href="https://blog.furkankoykiran.com.tr/en/posts/python-asyncio-concurrent-processing/" >Python asyncio guide</a>. Other MCP servers in this series: <a href="https://blog.furkankoykiran.com.tr/en/posts/freqtrade-mcp-server/" >freqtrade-mcp</a> and <a href="https://blog.furkankoykiran.com.tr/en/posts/devto-mcp-server/" >DevTo-MCP</a>.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/telegram-wallet-p2p-mcp-sdk/feature_hu_c3d64e954737727f.jpg" medium="image" type="image/jpeg"/></item><item><title>DevTo-MCP: Building an MCP Server for the DEV Community API</title><link>https://blog.furkankoykiran.com.tr/en/posts/devto-mcp-server/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/devto-mcp-server/</guid><pubDate>Wed, 18 Feb 2026 16:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>MCP &amp; AI</category><category>mcp</category><category>typescript</category><category>api</category><category>ci-cd</category><category>open-source</category><description>DevTo-MCP connects the DEV Community (Forem) API to assistants like Claude: Zod input checks, a retrying API client, a draft fallback and an npm naming lesson.</description><content:encoded><![CDATA[<p>DevTo-MCP is an MCP server that turns the Forem API, the platform behind DEV Community (dev.to), into tools. Connect it to Claude Desktop, Cursor or any other MCP client and the assistant can search DEV, list your own posts, and create or update drafts. It's written in TypeScript and published on npm as <code>@furkankoykiran/devto-mcp</code>.</p>
<p>Project page: <a href="https://furkankoykiran.com.tr/en/projects/devto-mcp/"  target="_blank" rel="noreferrer">DevTo-MCP</a> · Source: <a href="https://github.com/furkankoykiran/DevTo-MCP"  target="_blank" rel="noreferrer">GitHub</a></p>
<p>This post covers how the server is put together, the choices in its API client, a feature I switched off, and a lesson about npm package names.</p>

<h2 class="relative group">Why an MCP server for DEV
    <div id="why-an-mcp-server-for-dev" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-an-mcp-server-for-dev" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>I cross-post some of my writing to DEV. Writing a draft in my editor, pasting it into the browser, typing tags one by one, then going back to the site to check comments got tedious. I wanted to tell my assistant &quot;open this markdown as a draft on DEV&quot; or &quot;list this week's popular TypeScript posts.&quot;</p>
<p>MCP has three roles. The host is the app running the assistant, such as Claude Desktop. The MCP client inside the host speaks JSON-RPC 2.0 to a server. DevTo-MCP is the server: it advertises its tools, and when the client calls one it hits the Forem API and returns the result.</p>

<h2 class="relative group">Layout and tools
    <div id="layout-and-tools" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#layout-and-tools" aria-label="Anchor">#</a>
    </span>
    
</h2>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">src/
</span></span><span class="line"><span class="cl">├── index.ts            # reads DEVTO_API_KEY, starts over stdio
</span></span><span class="line"><span class="cl">├── server.ts           # registers the tool groups
</span></span><span class="line"><span class="cl">├── services/
</span></span><span class="line"><span class="cl">│   └── api-client.ts   # Forem API client
</span></span><span class="line"><span class="cl">├── tools/              # articles, comments, users, tags, organizations,
</span></span><span class="line"><span class="cl">│                       # reading-list, followers
</span></span><span class="line"><span class="cl">└── types/</span></span></code></pre></div></div>
<p>The first release shipped 17 tools; 16 are enabled today. For articles there are <code>get_articles</code>, <code>get_article_by_id</code>, <code>create_article</code>, <code>update_article</code> and <code>get_my_articles</code>. The rest are read tools for comments, users, organizations, tags, reading lists and followers. The Forem API has no endpoint for writing comments, so the comment tools are read-only.</p>

<h2 class="relative group">Validating inputs with Zod
    <div id="validating-inputs-with-zod" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#validating-inputs-with-zod" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The model generates tool arguments itself, and sometimes gets them wrong. DEV allows at most four tags per article; a model can easily send five. So every tool's parameters are a Zod schema:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ts" data-lang="ts"><span class="line"><span class="cl"><span class="nx">server</span><span class="p">.</span><span class="nx">tool</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">  <span class="s2">&#34;create_article&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="s2">&#34;Create a new article on DEV Community. Set published to false to save as draft.&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nx">title</span>: <span class="kt">z.string</span><span class="p">().</span><span class="nx">describe</span><span class="p">(</span><span class="s2">&#34;Title of the article&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="nx">body_markdown</span>: <span class="kt">z.string</span><span class="p">().</span><span class="nx">optional</span><span class="p">().</span><span class="nx">describe</span><span class="p">(</span><span class="s2">&#34;Article content in markdown format&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="nx">published</span>: <span class="kt">z.boolean</span><span class="p">().</span><span class="nx">optional</span><span class="p">().</span><span class="nx">describe</span><span class="p">(</span><span class="s2">&#34;Whether to publish immediately (false = draft)&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="nx">tags</span>: <span class="kt">z.array</span><span class="p">(</span><span class="nx">z</span><span class="p">.</span><span class="kt">string</span><span class="p">()).</span><span class="nx">max</span><span class="p">(</span><span class="mi">4</span><span class="p">).</span><span class="nx">optional</span><span class="p">().</span><span class="nx">describe</span><span class="p">(</span><span class="s2">&#34;List of tags (max 4)&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="nx">series</span>: <span class="kt">z.string</span><span class="p">().</span><span class="nx">optional</span><span class="p">().</span><span class="nx">describe</span><span class="p">(</span><span class="s2">&#34;Article series name&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="nx">canonical_url</span>: <span class="kt">z.string</span><span class="p">().</span><span class="nx">url</span><span class="p">().</span><span class="nx">optional</span><span class="p">().</span><span class="nx">describe</span><span class="p">(</span><span class="s2">&#34;Original URL if cross-posting&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="nx">description</span>: <span class="kt">z.string</span><span class="p">().</span><span class="nx">optional</span><span class="p">().</span><span class="nx">describe</span><span class="p">(</span><span class="s2">&#34;Short description for the article&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="nx">main_image</span>: <span class="kt">z.string</span><span class="p">().</span><span class="nx">url</span><span class="p">().</span><span class="nx">optional</span><span class="p">().</span><span class="nx">describe</span><span class="p">(</span><span class="s2">&#34;Main cover image URL&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="nx">organization_id</span>: <span class="kt">z.number</span><span class="p">().</span><span class="nx">int</span><span class="p">().</span><span class="nx">optional</span><span class="p">().</span><span class="nx">describe</span><span class="p">(</span><span class="s2">&#34;Organization ID to publish under&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">  <span class="p">},</span>
</span></span><span class="line"><span class="cl">  <span class="kr">async</span> <span class="p">(</span><span class="nx">params</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span> <span class="cm">/* ... */</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl"><span class="p">);</span></span></span></code></pre></div></div>
<p>The SDK turns this into JSON Schema for the model and validates incoming arguments against it. A fifth tag or a malformed <code>canonical_url</code> never reaches the API; the model gets a readable error and usually fixes it on the next try.</p>
<p>API responses don't go through Zod. The client reads <code>response.json()</code> and casts it with <code>as T</code>; there's no runtime check. If Forem returns a field in an unexpected shape, TypeScript won't notice. Since what goes back to the assistant is JSON text anyway, this doesn't crash anything in practice, but the response types are compile-time only.</p>

<h2 class="relative group">The API client: timeouts, retries and Retry-After
    <div id="the-api-client-timeouts-retries-and-retry-after" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-api-client-timeouts-retries-and-retry-after" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p><code>api-client.ts</code> is a bit more than a <code>fetch</code> wrapper:</p>
<ul>
<li>Every request is cut off after 30 seconds with <code>AbortSignal.timeout(30_000)</code>.</li>
<li>On 429, 5xx and network errors it retries up to 3 times, waiting 1, 2 and 4 seconds.</li>
<li>If the server sends <code>Retry-After</code>, it waits that long instead.</li>
<li>On failure it puts the <code>x-ratelimit-*</code> headers and <code>x-request-id</code> into the error message, so both the assistant and I can see what went wrong.</li>
</ul>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ts" data-lang="ts"><span class="line"><span class="cl"><span class="kr">private</span> <span class="nx">getRetryDelay</span><span class="p">(</span><span class="nx">attempt</span>: <span class="kt">number</span><span class="p">,</span> <span class="nx">response?</span>: <span class="kt">Response</span><span class="p">)</span><span class="o">:</span> <span class="kt">number</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">retryAfter</span> <span class="o">=</span> <span class="nx">response</span><span class="o">?</span><span class="p">.</span><span class="nx">headers</span><span class="p">.</span><span class="kr">get</span><span class="p">(</span><span class="s2">&#34;retry-after&#34;</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="p">(</span><span class="nx">retryAfter</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="kr">const</span> <span class="nx">seconds</span> <span class="o">=</span> <span class="nb">Number</span><span class="p">(</span><span class="nx">retryAfter</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nb">isNaN</span><span class="p">(</span><span class="nx">seconds</span><span class="p">)</span> <span class="o">&amp;&amp;</span> <span class="nx">seconds</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">)</span> <span class="k">return</span> <span class="nx">seconds</span> <span class="o">*</span> <span class="mi">1</span><span class="nx">_000</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="k">return</span> <span class="nx">INITIAL_BACKOFF_MS</span> <span class="o">*</span> <span class="nb">Math</span><span class="p">.</span><span class="nx">pow</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="nx">attempt</span><span class="p">);</span> <span class="c1">// 1s, 2s, 4s
</span></span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">private</span> <span class="nx">isRetryable</span><span class="p">(</span><span class="nx">status</span>: <span class="kt">number</span><span class="p">)</span><span class="o">:</span> <span class="kr">boolean</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="k">return</span> <span class="nx">status</span> <span class="o">===</span> <span class="mi">429</span> <span class="o">||</span> <span class="p">(</span><span class="nx">status</span> <span class="o">&gt;=</span> <span class="mi">500</span> <span class="o">&amp;&amp;</span> <span class="nx">status</span> <span class="o">&lt;</span> <span class="mi">600</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>The client doesn't watch <code>x-ratelimit-remaining</code> to slow down before the limit. It only reads that header for error messages and reacts to rate limits after a 429.</p>
<p>There's also a risk I should have thought through: the retry loop runs for POST and PUT as well. If <code>create_article</code> succeeds on the server but the response comes back as a 502, the client sends it again and you can end up with two copies of the same draft on DEV. Non-idempotent requests should only be retried when they provably never reached the server, or not at all.</p>

<h2 class="relative group">The feature I switched off: reactions
    <div id="the-feature-i-switched-off-reactions" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-feature-i-switched-off-reactions" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The initial release had tools for leaving reactions (likes, unicorns and so on) on articles. During testing they started returning 401 even with a valid API key, and some calls reported success while nothing showed up on the site.</p>
<p>An assistant that says &quot;done&quot; when nothing happened is worse than a missing tool. On February 18 I took the reaction tools out of registration:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ts" data-lang="ts"><span class="line"><span class="cl"><span class="c1">// registerReactionTools(server, client); // Disabled due to API 401 issues
</span></span></span></code></pre></div></div>
<p>The code is still in the repo for anyone who wants to track down the cause.</p>

<h2 class="relative group">Drafts and the 404
    <div id="drafts-and-the-404" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#drafts-and-the-404" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p><code>get_article_by_id</code> uses <code>/articles/{id}</code>, which only returns published articles and gives 404 for drafts. The assistant read that as &quot;no such article,&quot; even though it existed.</p>
<p>On a 404, the tool now fetches all of the user's own articles, drafts included, from <code>/articles/me/all</code> and looks for the ID there. The flow is on the card, the code below it:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/devto-mcp-server/img/draft-404-fallback_hu_cc793024f1c2bb2c.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/devto-mcp-server/img/draft-404-fallback_hu_3995a2fc974dee82.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/devto-mcp-server/img/draft-404-fallback_hu_e8bc368eca5943ad.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/devto-mcp-server/img/draft-404-fallback_hu_f680150fddb15603.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/devto-mcp-server/img/draft-404-fallback_hu_601f3c7bf65db755.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/devto-mcp-server/img/draft-404-fallback_hu_716ea874bd36ae28.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/devto-mcp-server/img/draft-404-fallback_hu_601f3c7bf65db755.webp" alt="get_article_by_id tries GET /articles/{id}; on a 404 it fetches /articles/me/all and returns the draft or a not-found error" width="1600" height="955"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>The 404 fallback for drafts</figcaption></figure><div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ts" data-lang="ts"><span class="line"><span class="cl"><span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">error</span>: <span class="kt">unknown</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">is404</span> <span class="o">=</span>
</span></span><span class="line"><span class="cl">    <span class="k">typeof</span> <span class="nx">error</span> <span class="o">===</span> <span class="s2">&#34;object&#34;</span> <span class="o">&amp;&amp;</span> <span class="nx">error</span> <span class="o">!==</span> <span class="kc">null</span> <span class="o">&amp;&amp;</span> <span class="s2">&#34;status&#34;</span> <span class="k">in</span> <span class="nx">error</span> <span class="o">&amp;&amp;</span>
</span></span><span class="line"><span class="cl">    <span class="p">(</span><span class="nx">error</span> <span class="kr">as</span> <span class="p">{</span> <span class="nx">status?</span>: <span class="kt">number</span> <span class="p">}).</span><span class="nx">status</span> <span class="o">===</span> <span class="mi">404</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="p">(</span><span class="nx">is404</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="kr">const</span> <span class="nx">allArticles</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">client</span><span class="p">.</span><span class="kr">get</span><span class="p">&lt;</span><span class="nt">Article</span><span class="err">[]</span><span class="p">&gt;(</span><span class="s2">&#34;/articles/me/all&#34;</span><span class="p">,</span> <span class="p">{</span> <span class="nx">per_page</span>: <span class="kt">1000</span> <span class="p">});</span>
</span></span><span class="line"><span class="cl">      <span class="kr">const</span> <span class="nx">article</span> <span class="o">=</span> <span class="nx">allArticles</span><span class="p">.</span><span class="nx">find</span><span class="p">((</span><span class="nx">a</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">a</span><span class="p">.</span><span class="nx">id</span> <span class="o">===</span> <span class="nx">params</span><span class="p">.</span><span class="nx">id</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">      <span class="k">if</span> <span class="p">(</span><span class="nx">article</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="p">{</span> <span class="nx">content</span><span class="o">:</span> <span class="p">[{</span> <span class="kr">type</span><span class="o">:</span> <span class="s2">&#34;text&#34;</span><span class="p">,</span> <span class="nx">text</span>: <span class="kt">JSON.stringify</span><span class="p">(</span><span class="nx">article</span><span class="p">,</span> <span class="kc">null</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> <span class="p">}]</span> <span class="p">};</span>
</span></span><span class="line"><span class="cl">      <span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">fallbackError</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="s2">&#34;Fallback lookup failed:&#34;</span><span class="p">,</span> <span class="nx">fallbackError</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="c1">// ...
</span></span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>Note the <code>console.error</code>. In a stdio server, stdout carries the JSON-RPC messages, and any log line written there corrupts the protocol. Logs belong on stderr.</p>
<p>The fallback misses articles beyond the first 1,000, which is fine for someone with a few dozen posts like me.</p>

<h2 class="relative group">Publishing, and a lesson about package names
    <div id="publishing-and-a-lesson-about-package-names" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#publishing-and-a-lesson-about-package-names" aria-label="Anchor">#</a>
    </span>
    
</h2>

<h3 class="relative group">How the workflow evolved
    <div id="how-the-workflow-evolved" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#how-the-workflow-evolved" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>The first workflow was manual (<code>workflow_dispatch</code>) with a dry-run option. With v1.0.2 I switched the trigger to published releases:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">on</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">release</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">types</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">published]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">permissions</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">contents</span><span class="p">:</span><span class="w"> </span><span class="l">read</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">id-token</span><span class="p">:</span><span class="w"> </span><span class="l">write</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">jobs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">publish</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">environment</span><span class="p">:</span><span class="w"> </span><span class="l">production</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="c"># checkout, setup-node (Node 22), npm ci, lint, typecheck, test, build</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">npm publish --provenance --access public</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">env</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">NODE_AUTH_TOKEN</span><span class="p">:</span><span class="w"> </span><span class="l">${{ secrets.NPM_TOKEN }}</span></span></span></code></pre></div></div>
<p>This workflow doesn't use OIDC trusted publishing. <code>environment: production</code> has been in it since its first committed version, and publishing authenticates with an <code>NPM_TOKEN</code> secret; <code>id-token: write</code> is there for the provenance signature. The npm 10 that ships with Node 22 can't do trusted publishing anyway; that needs npm 11.5.1 or later.</p>
<p>The <code>environment</code> line can still matter. If you define a secret on an environment instead of the repository, a job that doesn't name that environment sees an empty secret. And npm often answers unauthorized publishes with 404 rather than 403, so the error points you nowhere near the cause.</p>
<p>When I merged PR #1 with the reactions and draft changes, I expected a new release to go out on its own. It didn't: the trigger was a release, not a merge, and I hadn't bumped the version in <code>package.json</code>. Those changes shipped in v1.0.3 in March.</p>

<h3 class="relative group">Moving to a scoped name
    <div id="moving-to-a-scoped-name" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#moving-to-a-scoped-name" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>In February the package was published unscoped, as <code>devto-mcp</code>. In March I renamed it to <code>@furkankoykiran/devto-mcp</code> so it could also go to GitHub Packages, which requires scoped names.</p>
<p>That had a consequence I didn't expect. The unscoped <code>devto-mcp</code> name now belongs to a different project on npm, with a different maintainer, a different repository, and 0.x releases. The <code>npx -y devto-mcp</code> command in my DEV challenge article now runs that package instead of mine. An MCP server receives your API key through its environment, so a wrong package name here is a security problem.</p>
<p>Two rules came out of it: publish personal packages under a scope from day one and don't rename them later, and update every <code>npx</code> command in docs and posts the moment a package name changes. The latest README commit in the repo does exactly that.</p>

<h2 class="relative group">Installing it
    <div id="installing-it" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#installing-it" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>You need Node.js 18 or later and a DEV API key, which you can create under Settings → Extensions → &quot;DEV Community API Keys&quot;. Add this to Claude Desktop's <code>claude_desktop_config.json</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;mcpServers&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;devto&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;command&#34;</span><span class="p">:</span> <span class="s2">&#34;npx&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;args&#34;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&#34;-y&#34;</span><span class="p">,</span> <span class="s2">&#34;@furkankoykiran/devto-mcp&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;env&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="nt">&#34;DEVTO_API_KEY&#34;</span><span class="p">:</span> <span class="s2">&#34;your-key&#34;</span>
</span></span><span class="line"><span class="cl">      <span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>The file lives in <code>~/Library/Application Support/Claude/</code> on macOS and <code>%APPDATA%\Claude\</code> on Windows. After restarting Claude, try prompts like:</p>
<ul>
<li>&quot;Summarize this week's most popular TypeScript articles.&quot;</li>
<li>&quot;Create this markdown file as a draft on my DEV account, tagged mcp and typescript.&quot;</li>
<li>&quot;List the comments on my latest article and pull out the questions that still need an answer.&quot;</li>
</ul>

<h2 class="relative group">Next steps
    <div id="next-steps" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#next-steps" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>On my list: closing the POST retry risk above, moving the tools to the SDK's newer <code>registerTool()</code> API, and finding the cause of the reactions 401. The first real job I gave DevTo-MCP, having an assistant write and publish a challenge entry on DEV, is in <a href="https://blog.furkankoykiran.com.tr/en/posts/mcp-gemini-challenge-experience/" >the Gemini challenge post</a>. Other servers in this series: <a href="https://blog.furkankoykiran.com.tr/en/posts/omniwire-mcp-news-server/" >OmniWire-MCP</a> and <a href="https://blog.furkankoykiran.com.tr/en/posts/freqtrade-mcp-server/" >freqtrade-mcp</a>.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/devto-mcp-server/feature_hu_dac1af1159a7a183.jpg" medium="image" type="image/jpeg"/></item><item><title>OmniWire-MCP: A Fault-Tolerant News Server for AI Agents</title><link>https://blog.furkankoykiran.com.tr/en/posts/omniwire-mcp-news-server/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/omniwire-mcp-news-server/</guid><pubDate>Fri, 06 Feb 2026 20:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>MCP &amp; AI</category><category>mcp</category><category>typescript</category><category>ai-agents</category><category>github-actions</category><category>ci-cd</category><description>OmniWire-MCP serves RSS, Atom, JSON and HTML sources to AI assistants, with a circuit breaker per source, content sniffing, Zod schemas and a bumpy npm release.</description><content:encoded><![CDATA[<p>OmniWire-MCP is an MCP server that gives AI assistants news and content feeds as tools. It fetches RSS, Atom, JSON and HTML sources, normalizes them into one format, and lets an assistant like Claude answer &quot;what happened in tech today?&quot; with current data. It's written in TypeScript and published on npm as <code>@furkankoykiran/omniwire-mcp</code>.</p>
<p>This post covers three parts: the per-source circuit breaker that keeps one broken feed from slowing down the rest, the parser that tells formats apart, and the string of failures I hit publishing to npm. I also point out three gaps I found when I reread the code, each in its section.</p>
<p>Project page: <a href="https://furkankoykiran.com.tr/en/projects/omniwire-mcp/"  target="_blank" rel="noreferrer">OmniWire-MCP</a> · Source: <a href="https://github.com/furkankoykiran/OmniWire-MCP"  target="_blank" rel="noreferrer">GitHub</a></p>

<h2 class="relative group">What the server exposes
    <div id="what-the-server-exposes" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-the-server-exposes" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>It runs over stdio and registers four tools:</p>
<ul>
<li><code>fetch-news</code> pulls items from enabled sources, optionally filtered by keyword or a single source, with a result limit.</li>
<li><code>check-health</code> reports each source's circuit state and failure counts.</li>
<li><code>refresh-config</code> reloads the source list from remote config.</li>
<li><code>reset-source</code> manually resets one source's circuit breaker.</li>
</ul>
<p><code>fetch-news</code> fetches all healthy sources in parallel with <code>Promise.allSettled</code>, merges the results, sorts them by date and returns the requested number. One failing source can't spoil the response; with <code>Promise.all</code>, a single rejection would throw away everything.</p>
<p>Sources come from the <code>RSS_FEEDS</code> environment variable. It can be a URL to a JSON file, in which case the server re-reads it every 60 seconds by default and you can add feeds without a restart. It can also be the JSON itself. If it's unset, the server starts with Hacker News, Wired and The Verge.</p>

<h2 class="relative group">A circuit breaker per source
    <div id="a-circuit-breaker-per-source" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#a-circuit-breaker-per-source" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>News sources break all the time. One site drops its RSS feed, another returns 503 for a few minutes, a third hangs without answering. Retrying them on every request is slow and pointless.</p>
<p>So each source gets its own circuit breaker. In the codebase this layer is called Sentinel, and it's the standard three-state pattern shown in the diagram:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/omniwire-mcp-news-server/img/circuit-breaker-states_hu_dabd4ac7d04bf78b.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/omniwire-mcp-news-server/img/circuit-breaker-states_hu_4aba7aa1722e1ebf.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/omniwire-mcp-news-server/img/circuit-breaker-states_hu_a55ed6ba4641615b.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/omniwire-mcp-news-server/img/circuit-breaker-states_hu_ba600e99f1051ca0.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/omniwire-mcp-news-server/img/circuit-breaker-states_hu_1bcc3b73581e7fca.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/omniwire-mcp-news-server/img/circuit-breaker-states_hu_2b307532691c57d5.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/omniwire-mcp-news-server/img/circuit-breaker-states_hu_1bcc3b73581e7fca.webp" alt="Circuit breaker states: CLOSED opens after 3 failures, moves to HALF_OPEN after 60 seconds, closes after 2 successes and reopens on any failure" width="1600" height="1080"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>The three states of each source's circuit breaker</figcaption></figure><p>In CLOSED, requests go through and failures are counted. When the count hits the threshold (3 by default), the circuit opens and no requests are sent to that source. After the recovery timeout (60 seconds by default) it moves to HALF_OPEN and the source gets another chance: two successes close the circuit, a single failure opens it again.</p>
<p>The method that decides:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ts" data-lang="ts"><span class="line"><span class="cl"><span class="kr">public</span> <span class="nx">canExecute</span><span class="p">()</span><span class="o">:</span> <span class="kr">boolean</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">currentState</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">getState</span><span class="p">();</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="p">(</span><span class="nx">currentState</span> <span class="o">===</span> <span class="nx">CircuitState</span><span class="p">.</span><span class="nx">CLOSED</span><span class="p">)</span> <span class="k">return</span> <span class="kc">true</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="p">(</span><span class="nx">currentState</span> <span class="o">===</span> <span class="nx">CircuitState</span><span class="p">.</span><span class="nx">HALF_OPEN</span><span class="p">)</span> <span class="k">return</span> <span class="kc">true</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">shouldAttemptRecovery</span><span class="p">())</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">this</span><span class="p">.</span><span class="nx">transitionTo</span><span class="p">(</span><span class="nx">CircuitState</span><span class="p">.</span><span class="nx">HALF_OPEN</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="kc">true</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="k">return</span> <span class="kc">false</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p><code>fetch-news</code> filters out open circuits with <code>canRequest()</code> before sending anything. If 5 of 20 sources are down, the assistant gets the other 15 without waiting on timeouts. Each request also has a 10-second timeout, so a source that never answers counts as a failure instead of hanging forever.</p>
<p>There's a gap here. The comment says &quot;allow one test request&quot; in HALF_OPEN, but the code returns <code>true</code> for every request in that state. If several <code>fetch-news</code> calls arrive at once, they all hit the half-open source. On a single-user stdio server that rarely matters, but the correct behavior is to track the in-flight probe with a flag and reject the others until it resolves.</p>

<h2 class="relative group">Detecting the content type
    <div id="detecting-the-content-type" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#detecting-the-content-type" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Sources come in every shape: RSS 2.0, Atom, JSON APIs, and HTML for sites without a feed. Each source config has a <code>type</code> field that defaults to <code>auto</code>, and in that case the parser guesses:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ts" data-lang="ts"><span class="line"><span class="cl"><span class="kr">public</span> <span class="nx">detectContentType</span><span class="p">(</span><span class="nx">content</span>: <span class="kt">string</span><span class="p">,</span> <span class="nx">contentTypeHeader?</span>: <span class="kt">string</span><span class="p">)</span><span class="o">:</span> <span class="nx">ContentType</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="p">(</span><span class="nx">contentTypeHeader</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="kr">const</span> <span class="nx">lower</span> <span class="o">=</span> <span class="nx">contentTypeHeader</span><span class="p">.</span><span class="nx">toLowerCase</span><span class="p">();</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="p">(</span><span class="nx">lower</span><span class="p">.</span><span class="nx">includes</span><span class="p">(</span><span class="s1">&#39;rss&#39;</span><span class="p">)</span> <span class="o">||</span> <span class="nx">lower</span><span class="p">.</span><span class="nx">includes</span><span class="p">(</span><span class="s1">&#39;atom&#39;</span><span class="p">))</span> <span class="k">return</span> <span class="nx">ContentType</span><span class="p">.</span><span class="nx">RSS</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="p">(</span><span class="nx">lower</span><span class="p">.</span><span class="nx">includes</span><span class="p">(</span><span class="s1">&#39;json&#39;</span><span class="p">))</span> <span class="k">return</span> <span class="nx">ContentType</span><span class="p">.</span><span class="nx">JSON</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="p">(</span><span class="nx">lower</span><span class="p">.</span><span class="nx">includes</span><span class="p">(</span><span class="s1">&#39;html&#39;</span><span class="p">))</span> <span class="k">return</span> <span class="nx">ContentType</span><span class="p">.</span><span class="nx">HTML</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="p">(</span><span class="nx">lower</span><span class="p">.</span><span class="nx">includes</span><span class="p">(</span><span class="s1">&#39;xml&#39;</span><span class="p">))</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="k">return</span> <span class="nx">content</span><span class="p">.</span><span class="nx">includes</span><span class="p">(</span><span class="s1">&#39;&lt;feed&#39;</span><span class="p">)</span> <span class="o">?</span> <span class="nx">ContentType.ATOM</span> : <span class="kt">ContentType.RSS</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">trimmed</span> <span class="o">=</span> <span class="nx">content</span><span class="p">.</span><span class="nx">trim</span><span class="p">();</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="p">(</span><span class="nx">trimmed</span><span class="p">.</span><span class="nx">startsWith</span><span class="p">(</span><span class="s1">&#39;&lt;?xml&#39;</span><span class="p">)</span> <span class="o">||</span> <span class="nx">trimmed</span><span class="p">.</span><span class="nx">startsWith</span><span class="p">(</span><span class="s1">&#39;&lt;rss&#39;</span><span class="p">)</span> <span class="o">||</span> <span class="nx">trimmed</span><span class="p">.</span><span class="nx">startsWith</span><span class="p">(</span><span class="s1">&#39;&lt;feed&#39;</span><span class="p">))</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="nx">trimmed</span><span class="p">.</span><span class="nx">includes</span><span class="p">(</span><span class="s1">&#39;&lt;feed&#39;</span><span class="p">)</span> <span class="o">?</span> <span class="nx">ContentType.ATOM</span> : <span class="kt">ContentType.RSS</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="p">(</span><span class="nx">trimmed</span><span class="p">.</span><span class="nx">startsWith</span><span class="p">(</span><span class="s1">&#39;[&#39;</span><span class="p">)</span> <span class="o">||</span> <span class="nx">trimmed</span><span class="p">.</span><span class="nx">startsWith</span><span class="p">(</span><span class="s1">&#39;{&#39;</span><span class="p">))</span> <span class="k">return</span> <span class="nx">ContentType</span><span class="p">.</span><span class="nx">JSON</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="p">(</span><span class="nx">trimmed</span><span class="p">.</span><span class="nx">toLowerCase</span><span class="p">().</span><span class="nx">startsWith</span><span class="p">(</span><span class="s1">&#39;&lt;!doctype&#39;</span><span class="p">)</span> <span class="o">||</span> <span class="nx">trimmed</span><span class="p">.</span><span class="nx">toLowerCase</span><span class="p">().</span><span class="nx">startsWith</span><span class="p">(</span><span class="s1">&#39;&lt;html&#39;</span><span class="p">))</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="nx">ContentType</span><span class="p">.</span><span class="nx">HTML</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">return</span> <span class="nx">ContentType</span><span class="p">.</span><span class="nx">UNKNOWN</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>This code misclassifies servers that send XML as <code>text/html</code>. The header is checked first, and any header containing <code>html</code> is classified as HTML without looking at the body. Sniffing only kicks in when the header is missing or unrecognized, like <code>application/octet-stream</code> or <code>text/plain</code>.</p>
<p>The fix is to reverse the order: look at the first bytes of the body first and fall back to the header only when the body is ambiguous. The first bytes of a body reliably show its format, while headers are sometimes wrong. Until then, setting <code>type: &quot;rss&quot;</code> in the source config works around it.</p>

<h2 class="relative group">One Zod schema for every item
    <div id="one-zod-schema-for-every-item" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#one-zod-schema-for-every-item" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>All adapters produce the same shape:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ts" data-lang="ts"><span class="line"><span class="cl"><span class="kr">export</span> <span class="kr">const</span> <span class="nx">NewsItemSchema</span> <span class="o">=</span> <span class="nx">z</span><span class="p">.</span><span class="kt">object</span><span class="p">({</span>
</span></span><span class="line"><span class="cl">  <span class="nx">id</span>: <span class="kt">z.string</span><span class="p">(),</span>
</span></span><span class="line"><span class="cl">  <span class="nx">title</span>: <span class="kt">z.string</span><span class="p">(),</span>
</span></span><span class="line"><span class="cl">  <span class="nx">link</span>: <span class="kt">z.string</span><span class="p">().</span><span class="nx">url</span><span class="p">(),</span>
</span></span><span class="line"><span class="cl">  <span class="nx">description</span>: <span class="kt">z.string</span><span class="p">().</span><span class="nx">optional</span><span class="p">(),</span>
</span></span><span class="line"><span class="cl">  <span class="nx">content</span>: <span class="kt">z.string</span><span class="p">().</span><span class="nx">optional</span><span class="p">(),</span>
</span></span><span class="line"><span class="cl">  <span class="nx">author</span>: <span class="kt">z.string</span><span class="p">().</span><span class="nx">optional</span><span class="p">(),</span>
</span></span><span class="line"><span class="cl">  <span class="nx">publishedAt</span>: <span class="kt">z.date</span><span class="p">().</span><span class="nx">optional</span><span class="p">(),</span>
</span></span><span class="line"><span class="cl">  <span class="nx">categories</span>: <span class="kt">z.array</span><span class="p">(</span><span class="nx">z</span><span class="p">.</span><span class="kt">string</span><span class="p">()).</span><span class="k">default</span><span class="p">([]),</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">export</span> <span class="kr">type</span> <span class="nx">NewsItem</span> <span class="o">=</span> <span class="nx">z</span><span class="p">.</span><span class="k">infer</span><span class="p">&lt;</span><span class="nt">typeof</span> <span class="na">NewsItemSchema</span><span class="p">&gt;;</span></span></span></code></pre></div></div>
<p>With <code>z.infer</code> I don't write the TypeScript type separately, so the schema and the type can't drift apart.</p>
<p>The schema doesn't catch malformed items at runtime, though. The adapters only use it for the type and never call <code>parse</code> on what they produce, so an item with an invalid link still reaches the assistant. The fix is to run adapter output through the schema and drop what fails:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ts" data-lang="ts"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">items</span> <span class="o">=</span> <span class="nx">candidates</span><span class="p">.</span><span class="nx">flatMap</span><span class="p">((</span><span class="nx">candidate</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">parsed</span> <span class="o">=</span> <span class="nx">NewsItemSchema</span><span class="p">.</span><span class="nx">safeParse</span><span class="p">(</span><span class="nx">candidate</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">parsed</span><span class="p">.</span><span class="nx">success</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nx">logger</span><span class="p">.</span><span class="nx">warn</span><span class="p">(</span><span class="sb">`Dropped invalid item from </span><span class="si">${</span><span class="nx">source</span><span class="p">.</span><span class="nx">id</span><span class="si">}</span><span class="sb">: </span><span class="si">${</span><span class="nx">parsed</span><span class="p">.</span><span class="nx">error</span><span class="p">.</span><span class="nx">message</span><span class="si">}</span><span class="sb">`</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="p">[];</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="k">return</span> <span class="p">[</span><span class="nx">parsed</span><span class="p">.</span><span class="nx">data</span><span class="p">];</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span></span></span></code></pre></div></div>
<p>Source configuration, on the other hand, really is validated with Zod. Malformed JSON in <code>RSS_FEEDS</code> doesn't crash the server; it logs the error and falls back to the default sources.</p>

<h2 class="relative group">Repository hygiene
    <div id="repository-hygiene" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#repository-hygiene" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>It's a small project, but I set the repo up so other people could contribute. <code>main</code> is protected by an active ruleset, and changes go in through PRs. There's a <code>CONTRIBUTING.md</code>, <code>CODE_OF_CONDUCT.md</code>, <code>SECURITY.md</code>, and issue and PR templates. CI runs tests and lint, CodeQL scans for security issues, and <code>stale.yml</code> labels issues and PRs after 30 days of inactivity and closes them 5 days later.</p>
<p>Adding all of that on day one can feel like overkill. It's also the stuff nobody gets around to adding later, and the first thing a newcomer looks for.</p>

<h2 class="relative group">Publishing to npm: four failed runs
    <div id="publishing-to-npm-four-failed-runs" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#publishing-to-npm-four-failed-runs" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The goal was simple: create a release on GitHub, and a workflow runs the tests, builds, and publishes to npm. In practice I had one failed run before the first successful publish, and three more when I added GitHub Packages. The workflow history shows all of it:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/posts/omniwire-mcp-ai-news-server/img/npm-publish-workflow-runs_hu_c5e5a13bb8c6a324.avif 480w, https://blog.furkankoykiran.com.tr/posts/omniwire-mcp-ai-news-server/img/npm-publish-workflow-runs_hu_ac696066924d4b09.avif 800w, https://blog.furkankoykiran.com.tr/posts/omniwire-mcp-ai-news-server/img/npm-publish-workflow-runs_hu_2b570556c5c41256.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/posts/omniwire-mcp-ai-news-server/img/npm-publish-workflow-runs_hu_783c443c796bfbf4.webp 480w, https://blog.furkankoykiran.com.tr/posts/omniwire-mcp-ai-news-server/img/npm-publish-workflow-runs_hu_a3dfa6fd49071f78.webp 800w, https://blog.furkankoykiran.com.tr/posts/omniwire-mcp-ai-news-server/img/npm-publish-workflow-runs_hu_3f60c7748d97d11c.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/posts/omniwire-mcp-ai-news-server/img/npm-publish-workflow-runs_hu_a3dfa6fd49071f78.webp" alt="OmniWire-MCP npm-publish workflow history: v1.0.2 failed, v1.0.3 passed, v1.0.4, v1.0.5 and the first v1.0.6 attempt failed, the retry passed" width="1484" height="1118"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Six runs of the Node.js Package workflow</figcaption></figure><p>A missing <code>environment: production</code> line wasn't the problem; this workflow never had one. Here's what actually happened.</p>

<h3 class="relative group">v1.0.2 failed, v1.0.3 worked
    <div id="v102-failed-v103-worked" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#v102-failed-v103-worked" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p><code>package.json</code> had <code>&quot;provenance&quot;: true</code>. npm provenance records which workflow built the package, signed, and to do that it needs an OIDC token from GitHub Actions. The workflow didn't grant that permission. On top of that, <code>files</code> only shipped <code>dist</code>, but nothing ran the build before publishing, so the package would have gone out empty. The fix:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">permissions</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">contents</span><span class="p">:</span><span class="w"> </span><span class="l">read</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">id-token</span><span class="p">:</span><span class="w"> </span><span class="l">write</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c"># in the publish-npm job</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">npm ci</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">npm run build</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">npm publish  </span><span class="w"> </span><span class="c"># provenance still came from package.json here</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">env</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">NODE_AUTH_TOKEN</span><span class="p">:</span><span class="w"> </span><span class="l">${{ secrets.npm_token }}</span></span></span></code></pre></div></div>

<h3 class="relative group">v1.0.4 and v1.0.5: npm passed, GitHub Packages failed
    <div id="v104-and-v105-npm-passed-github-packages-failed" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#v104-and-v105-npm-passed-github-packages-failed" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>In March I added a second job to also publish to GitHub Packages. Because <code>&quot;provenance&quot;: true</code> lived in <code>publishConfig</code>, it applied to both registries, and provenance is only supported on npmjs.com. In v1.0.6 I removed it from <code>package.json</code> and passed <code>--provenance</code> only in the npm job, and both publishes went through.</p>
<p>Everything in <code>publishConfig</code> applies to every registry the package is published to. Keep registry-specific behavior in command-line flags.</p>

<h3 class="relative group">What I'd do today
    <div id="what-id-do-today" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-id-do-today" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Starting fresh today, I'd drop the token entirely. npm <a href="https://docs.npmjs.com/trusted-publishers"  target="_blank" rel="noreferrer">trusted publishing</a> uses the GitHub Actions OIDC token directly and adds provenance automatically. You link the package to the repo and workflow file on npmjs.com, grant <code>id-token: write</code>, and use npm CLI 11.5.1 or later. The npm 10 that ships with Node 20 doesn't support it, so either move to Node 24 or run <code>npm install -g npm@latest</code> in the job. Then there's no <code>NPM_TOKEN</code> in the repo to expire or leak.</p>

<h2 class="relative group">Installing it
    <div id="installing-it" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#installing-it" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Node.js 18 or later is enough. For Claude Desktop, add this to <code>claude_desktop_config.json</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;mcpServers&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nt">&#34;omniwire&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;command&#34;</span><span class="p">:</span> <span class="s2">&#34;npx&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;args&#34;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&#34;-y&#34;</span><span class="p">,</span> <span class="s2">&#34;@furkankoykiran/omniwire-mcp&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">      <span class="nt">&#34;env&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="nt">&#34;RSS_FEEDS&#34;</span><span class="p">:</span> <span class="s2">&#34;https://example.com/my-feeds.json&#34;</span>
</span></span><span class="line"><span class="cl">      <span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>Note the scoped package name. An unscoped <code>npx omniwire-mcp</code> doesn't work, because no package by that name exists. Leave out <code>RSS_FEEDS</code> to start with the three default sources.</p>
<p>There's no published Docker image; build it from the repo's Dockerfile:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">git clone https://github.com/furkankoykiran/OmniWire-MCP.git
</span></span><span class="line"><span class="cl"><span class="nb">cd</span> OmniWire-MCP
</span></span><span class="line"><span class="cl">docker build -t omniwire-mcp .
</span></span><span class="line"><span class="cl">docker run -i --rm -e <span class="nv">RSS_FEEDS</span><span class="o">=</span><span class="s2">&#34;https://example.com/my-feeds.json&#34;</span> omniwire-mcp</span></span></code></pre></div></div>
<p>Keep <code>-i</code>: the server talks over stdio and exits when stdin closes.</p>

<h2 class="relative group">Next steps
    <div id="next-steps" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#next-steps" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>First the three fixes above: a single probe in HALF_OPEN, sniffing the body before trusting the header, and validating items against the Zod schema. After that I want to move tool definitions from <code>server.tool()</code> to <code>registerTool()</code>, since the MCP TypeScript SDK is deprecating the old API, and change the Dockerfile's <code>npm ci --only=production</code> to <code>npm ci --omit=dev</code>. Longer term, writing items into a vector store would allow semantic questions like &quot;what came out about X last month?&quot;</p>
<p>Other servers in this series: <a href="https://blog.furkankoykiran.com.tr/en/posts/devto-mcp-server/" >DevTo-MCP</a> for the DEV Community API, and <a href="https://blog.furkankoykiran.com.tr/en/posts/freqtrade-mcp-server/" >freqtrade-mcp</a> for asking a trading bot questions through Claude. I also wrote about running CI on my own machine in <a href="https://blog.furkankoykiran.com.tr/en/posts/github-actions-self-hosted-runner/" >the self-hosted runner post</a>.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/omniwire-mcp-news-server/feature_hu_7773b6aabcf11f34.jpg" medium="image" type="image/jpeg"/></item><item><title>Memoryze: A Spaced Repetition App with React and Supabase</title><link>https://blog.furkankoykiran.com.tr/en/posts/memoryze-spaced-repetition-flashcards/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/memoryze-spaced-repetition-flashcards/</guid><pubDate>Thu, 05 Feb 2026 14:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Projects</category><category>typescript</category><category>web</category><category>databases</category><category>open-source</category><description>How I built Memoryze in one night: the SM-2 algorithm, Supabase auth and row level security, a 3D card flip in Framer Motion, and two bugs I still need to fix.</description><content:encoded><![CDATA[<p>Memoryze is an open-source spaced repetition app I wrote for memorizing words and concepts. You build decks of flashcards, and the SM-2 algorithm decides when each card comes back. It's React, TypeScript, Supabase and Framer Motion, hosted on GitHub Pages.</p>
<p><a href="https://furkankoykiran.com.tr/Memoryze/"  target="_blank" rel="noreferrer">Try the app</a> · <a href="https://github.com/furkankoykiran/Memoryze"  target="_blank" rel="noreferrer">Source code</a> · <a href="https://furkankoykiran.com.tr/en/projects/memoryze/"  target="_blank" rel="noreferrer">Project page</a></p>
<p>This post walks through how it works, the 3D card flip that ate most of my time, and two bugs I found when I reread the code.</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/posts/memoryze-flashcard-spaced-repetition/img/memoryze-landing-page_hu_554b8d5059f2ff64.avif 480w, https://blog.furkankoykiran.com.tr/posts/memoryze-flashcard-spaced-repetition/img/memoryze-landing-page_hu_b3a77c80736727f0.avif 800w, https://blog.furkankoykiran.com.tr/posts/memoryze-flashcard-spaced-repetition/img/memoryze-landing-page_hu_b1cca9d5f540a15e.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/posts/memoryze-flashcard-spaced-repetition/img/memoryze-landing-page_hu_e06b4725702294c2.webp 480w, https://blog.furkankoykiran.com.tr/posts/memoryze-flashcard-spaced-repetition/img/memoryze-landing-page_hu_468d3f1771840f1e.webp 800w, https://blog.furkankoykiran.com.tr/posts/memoryze-flashcard-spaced-repetition/img/memoryze-landing-page_hu_5623c7cc889be27.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/posts/memoryze-flashcard-spaced-repetition/img/memoryze-landing-page_hu_468d3f1771840f1e.webp" alt="Memoryze landing page with its dark UI, a Get Started button and feature cards, shown here in the Turkish locale" width="1600" height="1000"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Memoryze's landing page on GitHub Pages (Turkish locale)</figcaption></figure>
<h2 class="relative group">Why build my own
    <div id="why-build-my-own" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-build-my-own" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Duolingo feels too gamified to me and doesn't focus on the words I pick. Most of Quizlet's useful features are behind a paywall. Anki's scheduling is excellent, but the interface is dated and desktop-first. I wanted my own decks, a clean UI that opens in a browser, and a proper review algorithm underneath.</p>
<p>I wrote it on the night of February 5, 2026, in a session of roughly two hours. All 25 commits in the repo are from that night, and some of the bugs below come straight from that pace.</p>

<h2 class="relative group">Stack and architecture
    <div id="stack-and-architecture" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#stack-and-architecture" aria-label="Anchor">#</a>
    </span>
    
</h2>
<ul>
<li>React 19 and TypeScript, built with Vite 7.</li>
<li>Tailwind CSS 3 for a dark, glassy UI.</li>
<li>Framer Motion 12 for animation.</li>
<li>Supabase for the PostgreSQL database and email/password auth, password reset included.</li>
<li>react-i18next for English and Turkish, picked from the browser language.</li>
<li>React Router 7, deployed to GitHub Pages by a GitHub Actions workflow.</li>
</ul>
<p>There's no backend of my own; the browser talks to Supabase directly. The data model is two tables: <code>decks</code>, and <code>cards</code>, which holds each card's front, back and review state.</p>
<p>GitHub Pages doesn't support SPA routing, so the workflow generates a <code>404.html</code> and the router uses <code>/Memoryze</code> as its <code>basename</code>. Opening a deep link like <code>/Memoryze/dashboard</code> then loads the app instead of a 404.</p>

<h2 class="relative group">SM-2: when cards come back
    <div id="sm-2-when-cards-come-back" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#sm-2-when-cards-come-back" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>SM-2 is SuperMemo's algorithm from the late 1980s, and Anki's default scheduler is based on it. It tracks three numbers per card: how many times in a row you got it right (<code>repetition</code>), an ease factor (<code>ef</code>) and the interval in days (<code>interval</code>). The card below shows what a grade does, and the Memoryze code follows it:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/memoryze-spaced-repetition-flashcards/img/sm2-grades_hu_5ce27491fa76e519.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/memoryze-spaced-repetition-flashcards/img/sm2-grades_hu_5e4812786e712b60.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/memoryze-spaced-repetition-flashcards/img/sm2-grades_hu_62cb806bf0f33fd5.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/memoryze-spaced-repetition-flashcards/img/sm2-grades_hu_17d83a037585c9e.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/memoryze-spaced-repetition-flashcards/img/sm2-grades_hu_c3267faf61e8f7e4.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/memoryze-spaced-repetition-flashcards/img/sm2-grades_hu_c217ef64045a1f32.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/memoryze-spaced-repetition-flashcards/img/sm2-grades_hu_c3267faf61e8f7e4.webp" alt="SM-2: a grade of 3 or more adds one to repetition and sets the next review to 1, 6, then interval times EF days; below 3 it resets to 1 day" width="1600" height="1068"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>One SM-2 review step</figcaption></figure><div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ts" data-lang="ts"><span class="line"><span class="cl"><span class="kr">export</span> <span class="kr">const</span> <span class="nx">calculateSM2</span> <span class="o">=</span> <span class="p">({</span> <span class="nx">q</span><span class="p">,</span> <span class="nx">repetition</span><span class="p">,</span> <span class="nx">ef</span><span class="p">,</span> <span class="nx">interval</span> <span class="p">}</span><span class="o">:</span> <span class="nx">SM2Input</span><span class="p">)</span><span class="o">:</span> <span class="nx">SM2Output</span> <span class="o">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kd">let</span> <span class="nx">newRepetition</span> <span class="o">=</span> <span class="nx">repetition</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="kd">let</span> <span class="nx">newInterval</span> <span class="o">=</span> <span class="nx">interval</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="p">(</span><span class="nx">q</span> <span class="o">&gt;=</span> <span class="mi">3</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="p">(</span><span class="nx">repetition</span> <span class="o">===</span> <span class="mi">0</span><span class="p">)</span> <span class="nx">newInterval</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nx">repetition</span> <span class="o">===</span> <span class="mi">1</span><span class="p">)</span> <span class="nx">newInterval</span> <span class="o">=</span> <span class="mi">6</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="k">else</span> <span class="nx">newInterval</span> <span class="o">=</span> <span class="nb">Math</span><span class="p">.</span><span class="nx">round</span><span class="p">(</span><span class="nx">interval</span> <span class="o">*</span> <span class="nx">ef</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="nx">newRepetition</span> <span class="o">+=</span> <span class="mi">1</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nx">newRepetition</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="nx">newInterval</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// EF&#39; = EF + (0.1 - (5 - q) * (0.08 + (5 - q) * 0.02)), floor of 1.3
</span></span></span><span class="line"><span class="cl">  <span class="kd">let</span> <span class="nx">newEf</span> <span class="o">=</span> <span class="nx">ef</span> <span class="o">+</span> <span class="p">(</span><span class="mf">0.1</span> <span class="o">-</span> <span class="p">(</span><span class="mi">5</span> <span class="o">-</span> <span class="nx">q</span><span class="p">)</span> <span class="o">*</span> <span class="p">(</span><span class="mf">0.08</span> <span class="o">+</span> <span class="p">(</span><span class="mi">5</span> <span class="o">-</span> <span class="nx">q</span><span class="p">)</span> <span class="o">*</span> <span class="mf">0.02</span><span class="p">));</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="p">(</span><span class="nx">newEf</span> <span class="o">&lt;</span> <span class="mf">1.3</span><span class="p">)</span> <span class="nx">newEf</span> <span class="o">=</span> <span class="mf">1.3</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">nextReviewDate</span> <span class="o">=</span> <span class="k">new</span> <span class="nb">Date</span><span class="p">();</span>
</span></span><span class="line"><span class="cl">  <span class="nx">nextReviewDate</span><span class="p">.</span><span class="nx">setDate</span><span class="p">(</span><span class="nx">nextReviewDate</span><span class="p">.</span><span class="nx">getDate</span><span class="p">()</span> <span class="o">+</span> <span class="nx">newInterval</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">return</span> <span class="p">{</span> <span class="nx">repetition</span>: <span class="kt">newRepetition</span><span class="p">,</span> <span class="nx">ef</span>: <span class="kt">newEf</span><span class="p">,</span> <span class="nx">interval</span>: <span class="kt">newInterval</span><span class="p">,</span> <span class="nx">nextReviewDate</span> <span class="p">};</span>
</span></span><span class="line"><span class="cl"><span class="p">};</span></span></span></code></pre></div></div>
<p><code>q</code> is a grade from 0 to 5. The UI has four buttons: Again (0), Hard (3), Good (4) and Easy (5). A card you know comes back after 1 day, then 6, then each time after the previous interval multiplied by EF. Easy raises EF by 0.1 and Hard lowers it by 0.14, so intervals on hard cards grow more slowly.</p>
<p>The review screen loads up to 20 due cards ordered by <code>next_review</code>. Anything graded below 4 goes back to the end of the session queue, so a card you missed shows up again before the session ends.</p>

<h2 class="relative group">The 3D card flip
    <div id="the-3d-card-flip" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-3d-card-flip" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>This took the most time. The last six commits of that night are all about this animation.</p>
<p>I started with the classic CSS approach: rotate the card with <code>rotateY(180deg)</code> and give both faces <code>backface-visibility: hidden</code>. It worked in desktop Chrome, but sometimes the back face rendered mirrored, sometimes it didn't render at all, and sometimes the front face stayed on top and swallowed clicks meant for the buttons on the back.</p>
<p>Next I tried swapping opacity at 90 degrees with Framer Motion's <code>useMotionValue</code> and <code>useTransform</code>. The opacity approach didn't hold up either, and two commits later I went back to a proper 3D structure. This is what the repo runs:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-tsx" data-lang="tsx"><span class="line"><span class="cl"><span class="p">&lt;</span><span class="nt">div</span> <span class="na">style</span><span class="o">=</span><span class="p">{{</span> <span class="nx">perspective</span><span class="o">:</span> <span class="s2">&#34;1000px&#34;</span> <span class="p">}}&gt;</span>
</span></span><span class="line"><span class="cl">  <span class="p">&lt;</span><span class="nt">motion.div</span>
</span></span><span class="line"><span class="cl">    <span class="na">onClick</span><span class="o">=</span><span class="p">{()</span> <span class="o">=&gt;</span> <span class="o">!</span><span class="nx">isFlipped</span> <span class="o">&amp;&amp;</span> <span class="nx">setIsFlipped</span><span class="p">(</span><span class="kc">true</span><span class="p">)}</span>
</span></span><span class="line"><span class="cl">    <span class="na">initial</span><span class="o">=</span><span class="p">{</span><span class="kc">false</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="na">animate</span><span class="o">=</span><span class="p">{{</span> <span class="nx">rotateY</span>: <span class="kt">isFlipped</span> <span class="o">?</span> <span class="nx">180</span> : <span class="kt">0</span> <span class="p">}}</span>
</span></span><span class="line"><span class="cl">    <span class="na">transition</span><span class="o">=</span><span class="p">{{</span> <span class="kr">type</span><span class="o">:</span> <span class="s2">&#34;spring&#34;</span><span class="p">,</span> <span class="nx">stiffness</span>: <span class="kt">260</span><span class="p">,</span> <span class="nx">damping</span>: <span class="kt">20</span> <span class="p">}}</span>
</span></span><span class="line"><span class="cl">    <span class="na">style</span><span class="o">=</span><span class="p">{{</span> <span class="nx">transformStyle</span><span class="o">:</span> <span class="s2">&#34;preserve-3d&#34;</span> <span class="p">}}</span>
</span></span><span class="line"><span class="cl">    <span class="na">className</span><span class="o">=</span><span class="s">&#34;relative w-full min-h-[350px]&#34;</span>
</span></span><span class="line"><span class="cl">  <span class="p">&gt;</span>
</span></span><span class="line"><span class="cl">    <span class="p">&lt;</span><span class="nt">div</span>
</span></span><span class="line"><span class="cl">      <span class="na">className</span><span class="o">=</span><span class="s">&#34;absolute inset-0&#34;</span>
</span></span><span class="line"><span class="cl">      <span class="na">style</span><span class="o">=</span><span class="p">{{</span>
</span></span><span class="line"><span class="cl">        <span class="nx">backfaceVisibility</span><span class="o">:</span> <span class="s2">&#34;hidden&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="nx">WebkitBackfaceVisibility</span><span class="o">:</span> <span class="s2">&#34;hidden&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="nx">zIndex</span>: <span class="kt">isFlipped</span> <span class="o">?</span> <span class="nx">0</span> : <span class="kt">1</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="p">}}</span>
</span></span><span class="line"><span class="cl">    <span class="p">&gt;</span>
</span></span><span class="line"><span class="cl">      <span class="p">{</span><span class="nx">card</span><span class="p">.</span><span class="nx">front</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span>
</span></span><span class="line"><span class="cl">    <span class="p">&lt;</span><span class="nt">div</span>
</span></span><span class="line"><span class="cl">      <span class="na">className</span><span class="o">=</span><span class="s">&#34;absolute inset-0&#34;</span>
</span></span><span class="line"><span class="cl">      <span class="na">style</span><span class="o">=</span><span class="p">{{</span>
</span></span><span class="line"><span class="cl">        <span class="nx">backfaceVisibility</span><span class="o">:</span> <span class="s2">&#34;hidden&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="nx">WebkitBackfaceVisibility</span><span class="o">:</span> <span class="s2">&#34;hidden&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="nx">transform</span><span class="o">:</span> <span class="s2">&#34;rotateY(180deg)&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="nx">zIndex</span>: <span class="kt">isFlipped</span> <span class="o">?</span> <span class="nx">1</span> : <span class="kt">0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">      <span class="p">}}</span>
</span></span><span class="line"><span class="cl">    <span class="p">&gt;</span>
</span></span><span class="line"><span class="cl">      <span class="p">{</span><span class="nx">card</span><span class="p">.</span><span class="nx">back</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span>
</span></span><span class="line"><span class="cl">  <span class="p">&lt;/</span><span class="nt">motion.div</span><span class="p">&gt;</span>
</span></span><span class="line"><span class="cl"><span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span></span></span></code></pre></div></div>
<p>Four details make it work:</p>
<ol>
<li>The rotation lives on the shared parent, and the parent has <code>transform-style: preserve-3d</code>. Without it the faces stay flat in their own plane and the back face can't be hidden.</li>
<li>The back face starts rotated by 180 degrees. When the parent turns 180 degrees, the two rotations cancel and the text reads normally. Missing this is what produced the mirrored text.</li>
<li>Safari needs the <code>WebkitBackfaceVisibility</code> prefix as well.</li>
<li><code>zIndex</code> follows the flip state. The hidden face is still in the DOM and can still catch clicks, so whichever face is showing has to be on top.</li>
</ol>

<h2 class="relative group">The Supabase key and RLS
    <div id="the-supabase-key-and-rls" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-supabase-key-and-rls" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The repo has a committed <code>.env</code> with the Supabase URL and the anon key. That looks like a leak, but the anon key ships inside the JavaScript bundle anyway; anyone can read it in DevTools. In a SPA, the real protection is row level security: a user should only see their own decks and the cards in them.</p>
<p><code>cards</code> has no <code>user_id</code>; ownership comes through the deck. For this schema the policy has this shape:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">alter</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">cards</span><span class="w"> </span><span class="n">enable</span><span class="w"> </span><span class="k">row</span><span class="w"> </span><span class="k">level</span><span class="w"> </span><span class="k">security</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">create</span><span class="w"> </span><span class="n">policy</span><span class="w"> </span><span class="s2">&#34;own cards&#34;</span><span class="w"> </span><span class="k">on</span><span class="w"> </span><span class="n">cards</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">for</span><span class="w"> </span><span class="k">all</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">using</span><span class="w"> </span><span class="p">(</span><span class="k">exists</span><span class="w"> </span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">select</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">decks</span><span class="w"> </span><span class="n">d</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">where</span><span class="w"> </span><span class="n">d</span><span class="p">.</span><span class="n">id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cards</span><span class="p">.</span><span class="n">deck_id</span><span class="w"> </span><span class="k">and</span><span class="w"> </span><span class="n">d</span><span class="p">.</span><span class="n">user_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="k">select</span><span class="w"> </span><span class="n">auth</span><span class="p">.</span><span class="n">uid</span><span class="p">())</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="p">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">with</span><span class="w"> </span><span class="k">check</span><span class="w"> </span><span class="p">(</span><span class="k">exists</span><span class="w"> </span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">select</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">decks</span><span class="w"> </span><span class="n">d</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">where</span><span class="w"> </span><span class="n">d</span><span class="p">.</span><span class="n">id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cards</span><span class="p">.</span><span class="n">deck_id</span><span class="w"> </span><span class="k">and</span><span class="w"> </span><span class="n">d</span><span class="p">.</span><span class="n">user_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="k">select</span><span class="w"> </span><span class="n">auth</span><span class="p">.</span><span class="n">uid</span><span class="p">())</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="p">));</span></span></span></code></pre></div></div>
<p>Wrapping <code>auth.uid()</code> in a <code>select</code> is an optimization Supabase recommends, because Postgres then evaluates it once per query instead of once per row.</p>
<p>Even so, it's cleaner to take <code>.env</code> out of the repo and pass the values to the build as GitHub Actions secrets. The anon key isn't secret, but keeping a <code>.env</code> in git makes it easy for someone to drop the service role key in there one day.</p>

<h2 class="relative group">Two bugs I found rereading the code
    <div id="two-bugs-i-found-rereading-the-code" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#two-bugs-i-found-rereading-the-code" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Reading the code again later, I found two problems.</p>
<p>The first is the Hard button. It sends grade 3, which SM-2 treats as a pass: <code>repetition</code> goes up and the interval grows. The UI, though, treats anything below 4 as a miss and requeues the card in the same session. So a card you mark Hard is pushed to tomorrow and also shown again tonight. Mark it Good the second time and <code>repetition</code> increments again, so the interval grows faster than it should. The fix is to use one pass threshold everywhere, or to skip the second database update for requeued cards.</p>
<p>The second is the end of a session. Requeuing updates state with <code>setCards</code>, but the check for &quot;session complete&quot; right after it still uses the old <code>cards.length</code>. Miss the last card and the session ends on the &quot;complete&quot; screen, even though the card was requeued. Computing the new length once and using it for both the state update and the check fixes it.</p>
<p>Both are typical of a two-hour sprint, and neither got caught that night because I wrote no tests. <code>calculateSM2</code> is a pure function, so a handful of Vitest cases would have caught the first one.</p>

<h2 class="relative group">What's next
    <div id="whats-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#whats-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Ideas on the list: shareable decks, pronunciation through the browser's Web Speech API, and a study heatmap like GitHub's contribution graph. First, though, I want to fix the two bugs above and add tests for SM-2. After that I may try replacing SM-2 with <a href="https://github.com/open-spaced-repetition/fsrs4anki"  target="_blank" rel="noreferrer">FSRS</a>, the newer scheduler that Anki also supports.</p>
<p>The code is MIT licensed and <a href="https://github.com/furkankoykiran/Memoryze"  target="_blank" rel="noreferrer">on GitHub</a>. Create a Supabase project, put your own URL and anon key in <code>.env</code>, and run <code>npm install &amp;&amp; npm run dev</code>. Supabase's free plan has a low email sending limit, so turning off &quot;Confirm email&quot; during development makes sign-ups much less painful.</p>
<p>Another small React and Vite project of mine is <a href="https://blog.furkankoykiran.com.tr/en/posts/awesome-trending-repos-web/" >awesome-trending-repos</a>, and I wrote about building interfaces without a framework in <a href="https://blog.furkankoykiran.com.tr/en/posts/modern-web-ui-css-javascript/" >modern web UI without a framework</a>.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/memoryze-spaced-repetition-flashcards/feature_hu_e71985fc3b00bbab.jpg" medium="image" type="image/jpeg"/></item><item><title>Adding Lookaround to Google RE2: A Proof of Concept</title><link>https://blog.furkankoykiran.com.tr/en/posts/re2-lookaround-proof-of-concept/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/re2-lookaround-proof-of-concept/</guid><pubDate>Wed, 04 Feb 2026 23:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Open Source</category><category>open-source</category><category>performance</category><category>security</category><description>My fork that adds bounded lookaround to Google RE2 without backtracking, the #585 issue I opened, and what EPFL's unbounded lookbehind work showed me.</description><content:encoded><![CDATA[<p>Google RE2 is a regex library that guarantees matching in time linear in the input length. To keep that guarantee it leaves some features out, and lookaround (<code>(?=...)</code>, <code>(?!...)</code>, <code>(?&lt;=...)</code>, <code>(?&lt;!...)</code>) is one of them. In November 2025 I added bounded lookaround to RE2 in my own fork and shared it as a proof of concept in issue <a href="https://github.com/google/re2/issues/585"  target="_blank" rel="noreferrer">#585</a>. This post covers the design, a flaw I later found in my own complexity claim, and what a comment from EPFL taught me. I never opened a PR against RE2 and nothing landed upstream: what exists is a fork, one commit and an open discussion.</p>

<h2 class="relative group">Why RE2 has no lookaround
    <div id="why-re2-has-no-lookaround" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-re2-has-no-lookaround" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Backtracking engines such as PCRE, Python's <code>re</code> and the JavaScript engines can take exponential time on some patterns. The textbook case is <code>(a+)+$</code> against <code>aaaaaaaaaaaaaaaaaaaaaaaaaaaaX</code>: before the match fails, the engine tries every way of splitting the <code>a</code>s between the two groups.</p>
<p>This has taken down real sites. Stack Overflow was unreachable for about half an hour in 2016 because of a regex that trimmed trailing whitespace (<a href="https://stackstatus.tumblr.com/post/147710624694/outage-postmortem-july-20-2016"  target="_blank" rel="noreferrer">postmortem</a>), and Cloudflare's July 2019 outage came from a WAF rule regex that pegged CPUs (<a href="https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/"  target="_blank" rel="noreferrer">Cloudflare's write-up</a>). Any system that feeds user-controlled patterns or input into a backtracking engine carries this risk, known as ReDoS.</p>
<p>RE2 avoids it with automata theory. It compiles the pattern into an automaton and reads the input once, tracking all possible states at the same time. With no backtracking, run time stays linear. Google Sheets' <code>REGEXMATCH</code> uses RE2, and Go's <code>regexp</code> package follows the same design, which is why regexes in Prometheus queries are safe to run.</p>
<p>Backreferences (<code>\1</code>) and lookaround don't fit that model directly. A 2017 request for negative lookahead, <a href="https://github.com/google/re2/issues/156"  target="_blank" rel="noreferrer">#156</a>, was closed the same day with a short answer: RE2 doesn't support constructs that require backtracking. It still collected 61 thumbs-up reactions after being closed.</p>

<h2 class="relative group">What lookaround buys you
    <div id="what-lookaround-buys-you" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-lookaround-buys-you" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Lookaround checks whether something is present at a position without consuming it. The cases I see most:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">(?&lt;!old_)password    matches &#34;password&#34; in &#34;new_password&#34; but not in &#34;old_password&#34;
</span></span><span class="line"><span class="cl">test(?!\d)           matches &#34;test&#34; when no digit follows
</span></span><span class="line"><span class="cl">(?=.*\d)(?=.*[a-z]).{8,}   8+ characters with at least one digit and one lowercase letter</span></span></code></pre></div></div>
<p>The last one is the classic password rule. On RE2 you can't express it as a single regex and need two or three separate checks. That's usually fine, but porting patterns from another engine to RE2 means rewriting all of them.</p>

<h2 class="relative group">My approach: bounded subprograms
    <div id="my-approach-bounded-subprograms" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#my-approach-bounded-subprograms" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>I had read a 2024 paper from EPFL showing that JavaScript regexes, lookarounds included, can be matched in linear time (<a href="https://doi.org/10.1145/3656431"  target="_blank" rel="noreferrer">Barrière and Pit-Claudel, PLDI 2024</a>). The idea I took from it was that lookaround doesn't inherently need backtracking. I then tried a simpler, cruder approach than theirs.</p>
<p>Each lookaround compiles to its own subprogram. While the main program runs on the NFA, a lookaround instruction runs the matching subprogram at the current position, and the thread continues or dies depending on the result. The card shows this for the pattern <code>(?&lt;!pre)main(?=post)</code>.</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/re2-lookaround-proof-of-concept/img/bounded-subprograms_hu_5bc9a071e39dec60.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/re2-lookaround-proof-of-concept/img/bounded-subprograms_hu_367bbc87348b4309.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/re2-lookaround-proof-of-concept/img/bounded-subprograms_hu_2ea860209ead48ea.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/re2-lookaround-proof-of-concept/img/bounded-subprograms_hu_b765062aed425f82.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/re2-lookaround-proof-of-concept/img/bounded-subprograms_hu_4f99b7ed3be3742b.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/re2-lookaround-proof-of-concept/img/bounded-subprograms_hu_60619e34e79fba15.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/re2-lookaround-proof-of-concept/img/bounded-subprograms_hu_4f99b7ed3be3742b.webp" alt="Three steps: the pattern compiles to a main program and one subprogram per lookaround, the NFA runs the subprogram, and the thread continues or dies" width="1600" height="1204"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Each lookaround compiles to its own subprogram</figcaption></figure><p>Lookbehind is capped at 255 characters. The subprogram looks back at most 255 characters at each position, so the extra work per position has a constant upper bound. A small cap is also cheap to store, and real-world lookbehinds are almost always a few characters long.</p>
<p>The change is one commit, 569 added lines across 14 files:</p>
<ul>
<li><code>re2/parse.cc</code>: parsing the four lookaround forms</li>
<li><code>re2/regexp.h</code>: four new <code>RegexpOp</code> values</li>
<li><code>re2/prog.h</code> and <code>re2/prog.cc</code>: the <code>kInstLookBehind</code> and <code>kInstLookAhead</code> instructions and subprogram storage</li>
<li><code>re2/compile.cc</code>: compiling the subprograms</li>
<li><code>re2/nfa.cc</code>: running a subprogram when the NFA hits one of those instructions</li>
<li><code>re2/testing/lookbehind_test.cc</code>: tests</li>
</ul>
<p>This compiles and passes against the fork:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-cpp" data-lang="cpp"><span class="line"><span class="cl"><span class="cp">#include</span> <span class="cpf">&lt;re2/re2.h&gt;</span><span class="cp">
</span></span></span><span class="line"><span class="cl"><span class="cp">#include</span> <span class="cpf">&lt;cassert&gt;</span><span class="cp">
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kt">int</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="n">RE2</span> <span class="n">not_after_digit</span><span class="p">(</span><span class="s">&#34;(?&lt;!</span><span class="se">\\</span><span class="s">d)test&#34;</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="n">assert</span><span class="p">(</span><span class="n">RE2</span><span class="o">::</span><span class="n">PartialMatch</span><span class="p">(</span><span class="s">&#34;hello test&#34;</span><span class="p">,</span> <span class="n">not_after_digit</span><span class="p">));</span>
</span></span><span class="line"><span class="cl">  <span class="n">assert</span><span class="p">(</span><span class="o">!</span><span class="n">RE2</span><span class="o">::</span><span class="n">PartialMatch</span><span class="p">(</span><span class="s">&#34;123test&#34;</span><span class="p">,</span> <span class="n">not_after_digit</span><span class="p">));</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="n">RE2</span> <span class="n">after_hello</span><span class="p">(</span><span class="s">&#34;(?&lt;=hello )world&#34;</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="n">assert</span><span class="p">(</span><span class="n">RE2</span><span class="o">::</span><span class="n">PartialMatch</span><span class="p">(</span><span class="s">&#34;hello world&#34;</span><span class="p">,</span> <span class="n">after_hello</span><span class="p">));</span>
</span></span><span class="line"><span class="cl">  <span class="n">assert</span><span class="p">(</span><span class="o">!</span><span class="n">RE2</span><span class="o">::</span><span class="n">PartialMatch</span><span class="p">(</span><span class="s">&#34;goodbye world&#34;</span><span class="p">,</span> <span class="n">after_hello</span><span class="p">));</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="n">RE2</span> <span class="n">not_before_digit</span><span class="p">(</span><span class="s">&#34;test(?!</span><span class="se">\\</span><span class="s">d)&#34;</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="n">assert</span><span class="p">(</span><span class="n">RE2</span><span class="o">::</span><span class="n">PartialMatch</span><span class="p">(</span><span class="s">&#34;test hello&#34;</span><span class="p">,</span> <span class="n">not_before_digit</span><span class="p">));</span>
</span></span><span class="line"><span class="cl">  <span class="n">assert</span><span class="p">(</span><span class="o">!</span><span class="n">RE2</span><span class="o">::</span><span class="n">PartialMatch</span><span class="p">(</span><span class="s">&#34;test123&#34;</span><span class="p">,</span> <span class="n">not_before_digit</span><span class="p">));</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>Against upstream RE2 the same file compiles, but the patterns are rejected at runtime and the first <code>assert</code> fails.</p>
<p>I listed the limits in the issue. Any pattern with lookaround always falls back to the NFA, with no DFA or OnePass, which means much larger constant factors.</p>

<h2 class="relative group">The flaw I found later
    <div id="the-flaw-i-found-later" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-flaw-i-found-later" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>In the issue I gave the complexity as O(n × m × k), with n the input length, m the number of lookarounds and k the subpattern cost bounded by 255. That holds for lookbehind. It doesn't hold for lookahead.</p>
<p>I left lookahead unbounded and called it &quot;naturally bounded by the remaining text.&quot; The remaining text can be n characters long. A lookahead like <code>(?=.*\d)</code> can scan to the end of the input from every position, which is O(n²) in the worst case. So with lookahead, my prototype breaks the one promise RE2 exists to keep. Fixing it means either bounding lookahead too, or sharing results across positions instead of rerunning the same subprogram everywhere.</p>

<h2 class="relative group">The comment from EPFL
    <div id="the-comment-from-epfl" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-comment-from-epfl" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>About two weeks after I opened the issue, on December 9, 2025, Aurèle Barrière replied. Barrière is one of the authors of the paper I had read, and the comment pointed out that unbounded lookbehinds can also be supported in linear time. A year earlier, Erik Giorgis from their group had implemented their algorithm in a fork of RE2. The code is at <a href="https://github.com/epfl-systemf/re2-lookbehinds"  target="_blank" rel="noreferrer">epfl-systemf/re2-lookbehinds</a>, it's explained in <a href="https://systemf.epfl.ch/blog/re2-lookbehinds/"  target="_blank" rel="noreferrer">a post on the EPFL blog</a>, and they had shared it on the RE2 mailing list. Their approach avoids redundant work when the same lookbehind is tried at different positions and supports all unbounded lookbehinds that have no capture groups.</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/posts/google-re2-lookaround-proof-of-concept/img/issue-585-epfl-comment_hu_b71365c06e7e465e.avif 480w, https://blog.furkankoykiran.com.tr/posts/google-re2-lookaround-proof-of-concept/img/issue-585-epfl-comment_hu_eb3fb99cf494d52e.avif 800w, https://blog.furkankoykiran.com.tr/posts/google-re2-lookaround-proof-of-concept/img/issue-585-epfl-comment_hu_2158c1bf18d536a4.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/posts/google-re2-lookaround-proof-of-concept/img/issue-585-epfl-comment_hu_1009dd7762cc5a94.webp 480w, https://blog.furkankoykiran.com.tr/posts/google-re2-lookaround-proof-of-concept/img/issue-585-epfl-comment_hu_56a522eb47de6838.webp 800w, https://blog.furkankoykiran.com.tr/posts/google-re2-lookaround-proof-of-concept/img/issue-585-epfl-comment_hu_b7202c4e385d0782.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/posts/google-re2-lookaround-proof-of-concept/img/issue-585-epfl-comment_hu_56a522eb47de6838.webp" alt="Aurèle Barrière&#39;s comment on RE2 issue #585: unbounded lookbehinds work in linear time, with links to EPFL&#39;s RE2 fork and blog post" width="1600" height="416"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>The comment from EPFL on issue #585</figcaption></figure><p>So this wasn't two teams racing on the same problem. EPFL had done it a year before me, on stronger theory, and I hadn't found it before opening my issue. Side by side:</p>
<div class="fk-table" role="region" tabindex="0" aria-label="Scrollable table">
<table>
  <thead>
    <tr>
      <th></th>
      <th>My fork</th>
      <th>EPFL fork</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Lookbehind</td>
      <td>Capped at 255 characters</td>
      <td>Unbounded (no captures)</td>
    </tr>
    <tr>
      <td>Lookahead</td>
      <td>Supported, but not linear in the worst case</td>
      <td>Focused on lookbehind</td>
    </tr>
    <tr>
      <td>Repeated work</td>
      <td>Subprogram reruns at every position</td>
      <td>Shared across positions</td>
    </tr>
    <tr>
      <td>Basis</td>
      <td>Engineering shortcut</td>
      <td>Published algorithm</td>
    </tr>
  </tbody>
</table>
</div><p>The one thing on my side is that I attempted lookahead, and the flaw above makes that half a point.</p>

<h2 class="relative group">What I'd do differently
    <div id="what-id-do-differently" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-id-do-differently" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Search the literature and the mailing list before writing code. I had read the paper but never looked for the same group's RE2 fork. One search would have saved me weeks of duplicated work and a false sense of being first.</p>
<p>Check a complexity claim for every construct separately. Bounding lookbehind and waving lookahead through as &quot;naturally bounded&quot; looked reasonable on paper and left exactly the kind of hole RE2 is designed to avoid.</p>
<p>It also makes sense that none of this went upstream. RE2 runs underneath Google's own services, and staying small and predictable is a deliberate choice. A change like this has to be mature enough that the maintainers would want to own it for years, and my prototype isn't.</p>
<p>Still, reading a production C++ regex engine from the parser down to the NFA taught me a lot.</p>

<h2 class="relative group">Where the code is
    <div id="where-the-code-is" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-the-code-is" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>My next step is to study the EPFL fork and try their idea for lookahead. The code and notes on the experimental features are in <a href="https://github.com/furkankoykiran/re2"  target="_blank" rel="noreferrer">furkankoykiran/re2</a>, and the discussion is in issue <a href="https://github.com/google/re2/issues/585"  target="_blank" rel="noreferrer">#585</a>. More from this series: <a href="https://blog.furkankoykiran.com.tr/en/posts/github-mcp-server-contribution/" >my two PRs to github-mcp-server</a> and <a href="https://blog.furkankoykiran.com.tr/en/posts/my-open-source-contribution-keploy/" >my Go contributions to Keploy</a>.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/re2-lookaround-proof-of-concept/feature_hu_d402e108355f6eea.jpg" medium="image" type="image/jpeg"/></item><item><title>Contributing to github-mcp-server: Two PRs, Zero Merges</title><link>https://blog.furkankoykiran.com.tr/en/posts/github-mcp-server-contribution/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/github-mcp-server-contribution/</guid><pubDate>Wed, 04 Feb 2026 14:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Open Source</category><category>mcp</category><category>open-source</category><category>go</category><category>tooling</category><description>My two pull requests to GitHub's official MCP server: a create_discussion tool that is still open, and a read-only fix closed in favor of the maintainer's.</description><content:encoded><![CDATA[<p><code>github/github-mcp-server</code> is GitHub's official MCP server, the Go project that lets assistants like Copilot and Claude call the GitHub API as tools. I opened two pull requests against it. Neither was merged. This post covers what I built, how the codebase changed under me while I was locked out of my account, and why the second PR was closed. If you're about to send your first PR to a large project, the unmerged ones teach you at least as much as the merged ones.</p>

<h2 class="relative group">The missing tool: create_discussion
    <div id="the-missing-tool-create_discussion" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-missing-tool-create_discussion" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>In November 2025 I was reading the Discussions toolset and noticed a gap. It had <code>list_discussions</code>, <code>get_discussion</code>, <code>get_discussion_comments</code> and <code>list_discussion_categories</code>, but nothing that creates a discussion. An assistant could read threads and had no way to start one.</p>
<p>On December 1, 2025 I opened issue <a href="https://github.com/github/github-mcp-server/issues/1517"  target="_blank" rel="noreferrer">#1517</a> and a few hours later PR <a href="https://github.com/github/github-mcp-server/pull/1519"  target="_blank" rel="noreferrer">#1519</a>. No maintainer replied on the issue, and the only review the PR got was Copilot's automated one. In other words, I started the work without a maintainer's go-ahead.</p>
<p>On the GraphQL side the tool takes two steps. The <code>createDiscussion</code> mutation needs the repository's node ID, so the tool resolves the ID from owner and name first and sends the mutation second.</p>
<p>The lookup helper:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">getDiscussionRepositoryID</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">client</span><span class="w"> </span><span class="o">*</span><span class="nx">githubv4</span><span class="p">.</span><span class="nx">Client</span><span class="p">,</span><span class="w"> </span><span class="nx">owner</span><span class="p">,</span><span class="w"> </span><span class="nx">repo</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nx">githubv4</span><span class="p">.</span><span class="nx">ID</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="kd">var</span><span class="w"> </span><span class="nx">q</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="nx">Repository</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">			</span><span class="nx">ID</span><span class="w"> </span><span class="nx">githubv4</span><span class="p">.</span><span class="nx">ID</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="p">}</span><span class="w"> </span><span class="s">`graphql:&#34;repository(owner: $owner, name: $repo)&#34;`</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="nx">vars</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kt">any</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="s">&#34;owner&#34;</span><span class="p">:</span><span class="w"> </span><span class="nx">githubv4</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="nx">owner</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="s">&#34;repo&#34;</span><span class="p">:</span><span class="w">  </span><span class="nx">githubv4</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="nx">repo</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">client</span><span class="p">.</span><span class="nf">Query</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">q</span><span class="p">,</span><span class="w"> </span><span class="nx">vars</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="k">return</span><span class="w"> </span><span class="s">&#34;&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="k">return</span><span class="w"> </span><span class="nx">q</span><span class="p">.</span><span class="nx">Repository</span><span class="p">.</span><span class="nx">ID</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>Then the mutation runs and the tool returns the new discussion's ID, number and URL. If <code>repo</code> is omitted, it falls back to the org's <code>.github</code> repository, which is where GitHub hosts organization-level discussions.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">input</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">githubv4</span><span class="p">.</span><span class="nx">CreateDiscussionInput</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="nx">RepositoryID</span><span class="p">:</span><span class="w"> </span><span class="nx">repoID</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="nx">CategoryID</span><span class="p">:</span><span class="w">   </span><span class="nx">githubv4</span><span class="p">.</span><span class="nf">ID</span><span class="p">(</span><span class="nx">categoryID</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="nx">Title</span><span class="p">:</span><span class="w">        </span><span class="nx">githubv4</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="nx">title</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="nx">Body</span><span class="p">:</span><span class="w">         </span><span class="nx">githubv4</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="nx">body</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">client</span><span class="p">.</span><span class="nf">Mutate</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="nx">mutation</span><span class="p">,</span><span class="w"> </span><span class="nx">input</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="k">return</span><span class="w"> </span><span class="nx">utils</span><span class="p">.</span><span class="nf">NewToolResultError</span><span class="p">(</span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Sprintf</span><span class="p">(</span><span class="s">&#34;failed to create discussion: %v&#34;</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)),</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>The tests mock GraphQL responses with <code>githubv4mock</code> and add the tool's JSON schema to the project's <code>toolsnaps</code> snapshot system. They cover a successful create, an org-level create, missing parameters, and unknown repositories or categories.</p>

<h2 class="relative group">Two months away
    <div id="two-months-away" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#two-months-away" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A few days after I opened the PR, GitHub suspended my account. As far as I could tell it was flagged because of suspicious traffic on a shared network I was using. I filed an appeal and waited about two months. During that time my PR, my repos and my profile were invisible to everyone else.</p>
<p>When the account came back, the PR said &quot;This branch has conflicts that must be resolved.&quot; The conflicts weren't the real problem. Between the day I opened the PR and the day I returned, 204 commits had landed on <code>main</code>, touching roughly 300 files. The way tools are defined had been rewritten.</p>

<h2 class="relative group">The architecture that changed without me
    <div id="the-architecture-that-changed-without-me" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-architecture-that-changed-without-me" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>In December every tool returned an <code>mcp.Tool</code> plus a separate handler, and the GraphQL client was passed in as a parameter:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">CreateDiscussion</span><span class="p">(</span><span class="nx">getGQLClient</span><span class="w"> </span><span class="nx">GetGQLClientFn</span><span class="p">,</span><span class="w"> </span><span class="nx">t</span><span class="w"> </span><span class="nx">translations</span><span class="p">.</span><span class="nx">TranslationHelperFunc</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nx">mcp</span><span class="p">.</span><span class="nx">Tool</span><span class="p">,</span><span class="w"> </span><span class="nx">mcp</span><span class="p">.</span><span class="nx">ToolHandlerFor</span><span class="p">[</span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kt">any</span><span class="p">,</span><span class="w"> </span><span class="kt">any</span><span class="p">])</span></span></span></code></pre></div></div>
<p>By February, every tool was built with a <code>NewTool()</code> factory and returned an <code>inventory.ServerTool</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">CreateDiscussion</span><span class="p">(</span><span class="nx">t</span><span class="w"> </span><span class="nx">translations</span><span class="p">.</span><span class="nx">TranslationHelperFunc</span><span class="p">)</span><span class="w"> </span><span class="nx">inventory</span><span class="p">.</span><span class="nx">ServerTool</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="k">return</span><span class="w"> </span><span class="nf">NewTool</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="nx">ToolsetMetadataDiscussions</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="nx">mcp</span><span class="p">.</span><span class="nx">Tool</span><span class="p">{</span><span class="w"> </span><span class="cm">/* name, description, schema */</span><span class="w"> </span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="p">[]</span><span class="nx">scopes</span><span class="p">.</span><span class="nx">Scope</span><span class="p">{</span><span class="nx">scopes</span><span class="p">.</span><span class="nx">Repo</span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="kd">func</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">deps</span><span class="w"> </span><span class="nx">ToolDependencies</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="o">*</span><span class="nx">mcp</span><span class="p">.</span><span class="nx">CallToolRequest</span><span class="p">,</span><span class="w"> </span><span class="nx">args</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kt">any</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">mcp</span><span class="p">.</span><span class="nx">CallToolResult</span><span class="p">,</span><span class="w"> </span><span class="kt">any</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">			</span><span class="nx">client</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">deps</span><span class="p">.</span><span class="nf">GetGQLClient</span><span class="p">(</span><span class="nx">ctx</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">			</span><span class="c1">// ...</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">		</span><span class="p">},</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">	</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>A tool now declares its own toolset through metadata, receives clients through <code>ToolDependencies</code>, and lists the OAuth scopes it needs (<code>scopes.Repo</code> here), so the server can filter the tool list by what the token is allowed to do. That's a new contract more than a refactor, and every tool in the repo had been migrated to it.</p>

<h2 class="relative group">Rewriting instead of rebasing
    <div id="rewriting-instead-of-rebasing" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#rewriting-instead-of-rebasing" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>I could rebase my old commits onto the new <code>main</code> and resolve conflicts one by one. That meant forcing a function with the old signature into the new contract by hand, and I'd end up rewriting most lines anyway.</p>
<p>Or I could reset the branch to upstream and write the tool again with the new pattern. The PR wasn't merged and nobody depended on its history, so I went with the reset:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">git fetch upstream
</span></span><span class="line"><span class="cl">git checkout feature/add-create-discussion-tool
</span></span><span class="line"><span class="cl">git reset --hard upstream/main
</span></span><span class="line"><span class="cl"><span class="c1"># rewrite create_discussion with NewTool(), update the tests</span>
</span></span><span class="line"><span class="cl">go build ./...
</span></span><span class="line"><span class="cl"><span class="nv">UPDATE_TOOLSNAPS</span><span class="o">=</span><span class="nb">true</span> go <span class="nb">test</span> ./pkg/github/ -run Test_CreateDiscussion
</span></span><span class="line"><span class="cl">git push origin feature/add-create-discussion-tool --force-with-lease</span></span></code></pre></div></div>
<p>Use <code>--force-with-lease</code> for this. A plain <code>--force</code> silently overwrites any commit on the remote branch you haven't seen; <code>--force-with-lease</code> refuses the push in that case.</p>
<p>The rewrite went faster than I expected because the repo now had dozens of tools written in the new style. I opened <code>ListDiscussionCategories</code> and followed the same skeleton. The tests used the newer helpers: a type assertion on <code>tool.InputSchema.(*jsonschema.Schema)</code>, <code>githubv4mock.DataResponse</code> for fake responses, and <code>toolsnaps.Test</code> for the schema snapshot.</p>
<p>The final version is one commit, three files (<code>discussions.go</code>, <code>discussions_test.go</code> and <code>create_discussion.snap</code>) and 336 added lines. I force-pushed on February 4, 2026 and left a comment telling the maintainer the PR had been rebuilt on current <code>main</code> and was ready for review.</p>

<h2 class="relative group">Where #1519 stands
    <div id="where-1519-stands" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-1519-stands" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>#1519 is open and hasn't had a human review yet. #1517 is open too, and <code>main</code> has no <code>create_discussion</code> tool. The PR header shows the single commit and the 336 added lines:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/posts/github-mcp-server-acik-kaynak-katki/img/pr-1519-create-discussion-header_hu_6aaca9249a65bc69.avif 480w, https://blog.furkankoykiran.com.tr/posts/github-mcp-server-acik-kaynak-katki/img/pr-1519-create-discussion-header_hu_29ed9d240d459b1a.avif 800w, https://blog.furkankoykiran.com.tr/posts/github-mcp-server-acik-kaynak-katki/img/pr-1519-create-discussion-header_hu_3cf76dbec0c66863.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/posts/github-mcp-server-acik-kaynak-katki/img/pr-1519-create-discussion-header_hu_d70c71da562a63ad.webp 480w, https://blog.furkankoykiran.com.tr/posts/github-mcp-server-acik-kaynak-katki/img/pr-1519-create-discussion-header_hu_2df90004034d11b7.webp 800w, https://blog.furkankoykiran.com.tr/posts/github-mcp-server-acik-kaynak-katki/img/pr-1519-create-discussion-header_hu_b6cd31aca47892c4.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/posts/github-mcp-server-acik-kaynak-katki/img/pr-1519-create-discussion-header_hu_2df90004034d11b7.webp" alt="Header of github-mcp-server PR #1519: Open, 1 commit, 3 files changed, &#43;336 lines" width="1600" height="200"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>PR #1519, waiting for review</figcaption></figure><p>Meanwhile the team added <code>discussion_comment_write</code> for comments, so they are working on the Discussions write side on their own schedule.</p>
<p>I don't take that personally. Plenty of outside PRs are waiting in that repo, and the team treats write tools, scopes and security as one design problem. Adding a new write tool is a product decision for them, and whether my code is correct is only part of it.</p>

<h2 class="relative group">The second PR: read-only mode, closed
    <div id="the-second-pr-read-only-mode-closed" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-second-pr-read-only-mode-closed" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>In March 2026 I tried something smaller. Issue <a href="https://github.com/github/github-mcp-server/issues/2156"  target="_blank" rel="noreferrer">#2156</a> reported that <code>--read-only</code> did nothing in HTTP mode, leaving write tools like <code>create_pull_request</code> available. PR <a href="https://github.com/github/github-mcp-server/pull/2200"  target="_blank" rel="noreferrer">#2200</a> passed the flag into the HTTP server config and added a middleware that applied it to every request.</p>
<p>The maintainer replied within minutes that this was intended. In HTTP mode users enable read-only per request with the <code>X-MCP-Readonly</code> header, the same way the hosted remote server works, and the team hadn't decided whether to allow a global setting. I argued for consistency with stdio mode and for multi-tenant deployments.</p>
<p>The maintainer then fixed it in a PR of their own, <a href="https://github.com/github/github-mcp-server/pull/2208"  target="_blank" rel="noreferrer">#2208</a>: CLI flags became a static upper bound, and headers can only narrow it. It was merged on April 13, 2026. The diagram below shows the two layers.</p>
<p><figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/github-mcp-server-contribution/img/read-only-layers_hu_1d0980b9a06451b8.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/github-mcp-server-contribution/img/read-only-layers_hu_e45ef279b8779c57.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/github-mcp-server-contribution/img/read-only-layers_hu_13798435b25e20b5.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/github-mcp-server-contribution/img/read-only-layers_hu_3048e2a2e0bfd26d.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/github-mcp-server-contribution/img/read-only-layers_hu_2abb48d7c48b728.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/github-mcp-server-contribution/img/read-only-layers_hu_71829b04a245932.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/github-mcp-server-contribution/img/read-only-layers_hu_2abb48d7c48b728.webp" alt="The --read-only CLI flag caps the allowed tools, the X-MCP-Readonly header can only narrow that set, and the client sees what remains" width="1600" height="1030"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Read-only layers after #2208</figcaption></figure>
They asked whether that solved my case. I confirmed it did and that it was cleaner than my middleware, and #2200 was closed unmerged on April 15.</p>
<p>I count that as a good outcome. The bug was real and the Docker scenario from the issue works now; someone else wrote the fix.</p>

<h2 class="relative group">What I took from both
    <div id="what-i-took-from-both" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-i-took-from-both" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>On a busy project a PR can outlive the code it was written against. When I came back after two months, every line I had written targeted an old contract. Diffing a waiting PR against upstream every week or two beats discovering the drift all at once.</p>
<p>Write tools are much harder to land than read tools. Letting an assistant open discussions on someone's behalf raises questions about permissions, spam and abuse. Next time I'll wait for an explicit yes from a maintainer on the issue before writing code.</p>
<p>A &quot;this is a design decision&quot; reply is also a result. Reading the maintainer's reasoning on #2200 showed me my middleware sat at the wrong layer, and the fix landed where it belongs, in the server configuration.</p>

<h2 class="relative group">Where the PRs are
    <div id="where-the-prs-are" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-the-prs-are" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Other contributions in this series: <a href="https://blog.furkankoykiran.com.tr/en/posts/re2-lookaround-proof-of-concept/" >a lookaround prototype for Google's RE2</a>, <a href="https://blog.furkankoykiran.com.tr/en/posts/my-open-source-contribution-keploy/" >my Go work on Keploy</a> and <a href="https://blog.furkankoykiran.com.tr/en/posts/cli-anything-browser-automation-contribution/" >browser automation for CLI-Anything</a>. The current state of #1519 is on <a href="https://github.com/github/github-mcp-server/pull/1519"  target="_blank" rel="noreferrer">the PR page</a>. If you need a write tool for Discussions, a +1 reaction on <a href="https://github.com/github/github-mcp-server/issues/1517"  target="_blank" rel="noreferrer">#1517</a> will help the team prioritize it more than anything I can add.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/github-mcp-server-contribution/feature_hu_8ab3ccab78d93940.jpg" medium="image" type="image/jpeg"/></item><item><title>GitLab and Back to GitHub: Migrating My Jekyll Blog</title><link>https://blog.furkankoykiran.com.tr/en/posts/gitlab-github-jekyll-migration/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/gitlab-github-jekyll-migration/</guid><pubDate>Mon, 02 Feb 2026 10:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Stories</category><category>github-actions</category><category>ci-cd</category><category>devops</category><category>web</category><description>When my GitHub account was suspended I moved my repos and Jekyll blog to GitLab, then came back. How GitLab CI and GitHub Actions differed, and what I kept.</description><content:encoded><![CDATA[<p>One morning I found my GitHub account suspended. My open source projects, my portfolio site and my Jekyll blog were suddenly out of reach. It turned out to be a simple mistake and the account came back, but while it was down I moved every repository to GitLab and started publishing the sites from there. Here is that migration, the differences I noticed between GitLab CI and GitHub Actions, and how I set up my infrastructure afterwards.</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/gitlab-github-jekyll-migration/img/migration-stops_hu_4132b60483687dc.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/gitlab-github-jekyll-migration/img/migration-stops_hu_7c15dfd553beb85d.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/gitlab-github-jekyll-migration/img/migration-stops_hu_be97ada12daf6ea3.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/gitlab-github-jekyll-migration/img/migration-stops_hu_be04d409547aeda6.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/gitlab-github-jekyll-migration/img/migration-stops_hu_58365e803b572a71.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/gitlab-github-jekyll-migration/img/migration-stops_hu_df1a041dd275c6f1.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/gitlab-github-jekyll-migration/img/migration-stops_hu_58365e803b572a71.webp" alt="Four stops: Jekyll on GitHub Pages, GitLab CI and Pages after the suspension, back to GitHub Actions, then Hugo on GitHub" width="1600" height="1225"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Four stops: suspension, GitLab, the return and Hugo</figcaption></figure>
<h2 class="relative group">The original setup
    <div id="the-original-setup" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-original-setup" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>At the time I had two sites, both Jekyll on GitHub Pages:</p>
<ul>
<li>The portfolio (<code>furkankoykiran.com.tr</code>) lived in the <code>&lt;username&gt;.github.io</code> repository with a minimal Jekyll theme. Settings were in <code>_config.yml</code>, content was Markdown, and the domain sat on Cloudflare.</li>
<li>The blog (<code>blog.furkankoykiran.com.tr</code>) lived in its own repository with the Chirpy theme. The theme's <code>tools/init.sh</code> script cleaned the repo up for personal use.</li>
</ul>
<p>One thing that tripped me up when testing locally with <code>bundle exec jekyll s</code>: if <code>url</code> and <code>baseurl</code> in <code>_config.yml</code> don't match the local address, CSS and JS return 404. The page loads, just without any styling.</p>

<h2 class="relative group">The GitLab stop
    <div id="the-gitlab-stop" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-gitlab-stop" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>With GitHub closed, I moved the repositories to GitLab. Git itself was easy because I had local clones. The real work was rewriting CI/CD, since GitHub Actions workflows don't run on GitLab.</p>
<p>On GitLab everything lives in one <code>.gitlab-ci.yml</code> at the repository root. Here's the config that publishes a Jekyll site to GitLab Pages, written with today's syntax:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">ruby:3.3</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">cache</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">key</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">files</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">Gemfile.lock</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">paths</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="l">vendor/bundle</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">pages</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">stage</span><span class="p">:</span><span class="w"> </span><span class="l">deploy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">script</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="l">bundle config set --local path vendor/bundle</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="l">bundle install</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="l">bundle exec jekyll build -d public</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">artifacts</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">paths</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="l">public</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">rules</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l">$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH</span></span></span></code></pre></div></div>
<p>Some things to watch:</p>
<ul>
<li>GitLab Pages expects the output in an artifact folder named <code>public</code>. Jekyll writes to <code>_site</code> by default, hence <code>-d public</code>.</li>
<li>Keying the cache on <code>Gemfile.lock</code> means gems are only downloaded again when the lock file changes.</li>
<li>My file back then used <code>only: [master]</code>. GitLab now treats <code>only</code> as legacy and recommends <code>rules</code>.</li>
</ul>
<p>The GitHub Actions equivalent is shorter thanks to <code>ruby/setup-ruby</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Deploy Jekyll site</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">on</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">push</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">branches</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">main]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">permissions</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">contents</span><span class="p">:</span><span class="w"> </span><span class="l">read</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">pages</span><span class="p">:</span><span class="w"> </span><span class="l">write</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">id-token</span><span class="p">:</span><span class="w"> </span><span class="l">write</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">jobs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">build</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/checkout@v7</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">ruby/setup-ruby@v1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">with</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">ruby-version</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;3.3&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">bundler-cache</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="l">bundle exec jekyll build</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">env</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">JEKYLL_ENV</span><span class="p">:</span><span class="w"> </span><span class="l">production</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/upload-pages-artifact@v5</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">deploy</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">needs</span><span class="p">:</span><span class="w"> </span><span class="l">build</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">runs-on</span><span class="p">:</span><span class="w"> </span><span class="l">ubuntu-latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">environment</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">github-pages</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">url</span><span class="p">:</span><span class="w"> </span><span class="l">${{ steps.deployment.outputs.page_url }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">steps</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l">deployment</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">uses</span><span class="p">:</span><span class="w"> </span><span class="l">actions/deploy-pages@v5</span></span></span></code></pre></div></div>
<p><code>bundler-cache: true</code> replaces the <code>bundle install</code> and cache steps with one line, and <code>upload-pages-artifact</code> uploads <code>_site</code> by default, so Jekyll's output path can stay as it is. Older Jekyll guides still show <code>actions/setup-ruby@v1</code> with <code>cache: 'bundler'</code>. That action was archived long ago, and the <code>ruby/setup-ruby</code> README documents <code>bundler-cache</code> instead:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/posts/gitlab-github-macerasi-jekyll-altyapisi/img/setup-ruby-bundler-cache-readme_hu_36949f5773e37097.avif 480w, https://blog.furkankoykiran.com.tr/posts/gitlab-github-macerasi-jekyll-altyapisi/img/setup-ruby-bundler-cache-readme_hu_954ac51f94896b5d.avif 800w, https://blog.furkankoykiran.com.tr/posts/gitlab-github-macerasi-jekyll-altyapisi/img/setup-ruby-bundler-cache-readme_hu_e254b6846375bd1c.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/posts/gitlab-github-macerasi-jekyll-altyapisi/img/setup-ruby-bundler-cache-readme_hu_fdb590d633c5aaa3.webp 480w, https://blog.furkankoykiran.com.tr/posts/gitlab-github-macerasi-jekyll-altyapisi/img/setup-ruby-bundler-cache-readme_hu_6f5e6949270a3301.webp 800w, https://blog.furkankoykiran.com.tr/posts/gitlab-github-macerasi-jekyll-altyapisi/img/setup-ruby-bundler-cache-readme_hu_ef5a2b2b2c6fe172.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/posts/gitlab-github-macerasi-jekyll-altyapisi/img/setup-ruby-bundler-cache-readme_hu_6f5e6949270a3301.webp" alt="The ruby/setup-ruby README section showing bundler-cache: true, which runs bundle install and caches the result" width="1364" height="520"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Source: GitHub, ruby/setup-ruby README (quotation)</figcaption></figure>
<h2 class="relative group">Differences I noticed
    <div id="differences-i-noticed" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#differences-i-noticed" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>I used both platforms on real projects, if only for a short while. These are the differences that showed up in daily use rather than on feature pages:</p>
<div class="fk-table" role="region" tabindex="0" aria-label="Scrollable table">
<table>
  <thead>
    <tr>
      <th></th>
      <th>GitLab CI/CD</th>
      <th>GitHub Actions</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Configuration</td>
      <td>One <code>.gitlab-ci.yml</code> with stages and jobs</td>
      <td>Several workflow files, jobs triggered by events</td>
    </tr>
    <tr>
      <td>Reusable pieces</td>
      <td>A CI/CD Catalog exists but is small; you mostly write scripts</td>
      <td>Huge Marketplace with an action for most tasks</td>
    </tr>
    <tr>
      <td>Caching</td>
      <td>You define the paths and key yourself with <code>cache:</code></td>
      <td>One line in the <code>setup-*</code> actions</td>
    </tr>
    <tr>
      <td>Your own runners</td>
      <td>Docker and Kubernetes executors out of the box</td>
      <td>Possible, but the security is on you</td>
    </tr>
    <tr>
      <td>Artifacts</td>
      <td>Very visible which job produced what</td>
      <td>Less prominent, but enough for Pages</td>
    </tr>
  </tbody>
</table>
</div><p>I liked how GitLab keeps everything in one place, and how easy it makes running your own runners. For a team that wants to self-host the whole toolchain, I'd recommend it without hesitation. For personal projects, GitHub's ready-made actions get me done faster. I later set up my own runner on the GitHub side as well, which I wrote about in the <a href="https://blog.furkankoykiran.com.tr/en/posts/github-actions-self-hosted-runner/" >self-hosted runner guide</a>.</p>

<h2 class="relative group">Why I went back to GitHub
    <div id="why-i-went-back-to-github" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-i-went-back-to-github" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Once the account was restored I moved everything back. The reasons were practical:</p>
<ul>
<li>Nearly every open source project I contribute to is on GitHub. Issues, pull requests and reviews happen there.</li>
<li>My editor and tools integrate with GitHub directly.</li>
<li>Dependabot opening PRs for dependency updates on its own cuts down my maintenance work. GitLab has dependency scanning too, but GitHub's flow suited me better.</li>
</ul>

<h2 class="relative group">Habits I kept
    <div id="habits-i-kept" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#habits-i-kept" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The real gain was being ready for the next outage, wherever my code lives:</p>
<ul>
<li>
<p>Important repositories have a copy on another platform. To do it by hand:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">git clone --mirror git@github.com:YOUR_USER/YOUR_REPO.git
</span></span><span class="line"><span class="cl"><span class="nb">cd</span> YOUR_REPO.git
</span></span><span class="line"><span class="cl">git push --mirror git@gitlab.com:YOUR_USER/YOUR_REPO.git</span></span></code></pre></div></div>
<p><code>--mirror</code> carries every branch and tag. GitLab's push mirroring can automate it.</p>
</li>
<li>
<p>The build is a single command or script, and the CI file only calls it. When the platform changes, you only rewrite that thin layer.</p>
</li>
<li>
<p>I don't rely on the cloud alone. A local copy from a regular <code>git clone --mirror</code> is what helps when nothing else works.</p>
</li>
<li>
<p>The domain lives with a DNS provider (Cloudflare), separate from the code host. Pointing the sites at GitLab Pages meant changing a few DNS records.</p>
</li>
</ul>

<h2 class="relative group">Last stop: Hugo
    <div id="last-stop-hugo" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#last-stop-hugo" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>In the end I moved both the blog and the portfolio from Jekyll to <a href="https://gohugo.io/"  target="_blank" rel="noreferrer">Hugo</a>. Both use the Blowfish theme and deploy to GitHub Pages through GitHub Actions. Turkish posts kept their <code>/posts/&lt;slug&gt;/</code> URLs, English posts moved under <code>/en/</code>, and old addresses redirect to the new ones. The whole build is now one <code>hugo --gc --minify</code> command, which is the cleanest version of the &quot;single build command&quot; habit above.</p>

<h2 class="relative group">What to try next
    <div id="what-to-try-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-to-try-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>If you don't have a backup and mirror plan, mirror one repository somewhere else today and try restoring from it. To simplify the branches you'd be mirroring, see <a href="https://blog.furkankoykiran.com.tr/en/posts/git-workflow-branching-strategies/" >Git workflow and branching strategies</a>. For a project that publishes to GitHub Pages on a schedule, see the <a href="https://blog.furkankoykiran.com.tr/en/posts/awesome-trending-repos-web/" >awesome-trending-repos post</a>.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/gitlab-github-jekyll-migration/feature_hu_45d50048cb65cd45.jpg" medium="image" type="image/jpeg"/></item><item><title>Apache Kafka Event Streaming with KRaft and Python</title><link>https://blog.furkankoykiran.com.tr/en/posts/apache-kafka-event-streaming/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/apache-kafka-event-streaming/</guid><pubDate>Thu, 04 Dec 2025 14:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Backend</category><category>messaging</category><category>python</category><category>devops</category><category>monitoring</category><description>Event streaming on Kafka 4 without ZooKeeper (KRaft): partition sizing, a no-loss producer, offset commits and rebalance tuning, with confluent-kafka in Python.</description><content:encoded><![CDATA[<p>Apache Kafka event streaming is a different model from a classic message queue. Kafka is an ordered, distributed log that doesn't delete messages once they're read. Used well, that gives you replay, many independent consumers and very high throughput. Used badly, it gives you consumers stuck in rebalance loops and lost messages.</p>
<p>This post covers what a Kafka 4 cluster looks like in KRaft mode without ZooKeeper, how to pick a partition count, and how to write a producer and consumer in Python with <code>confluent-kafka</code> that don't lose data.</p>

<h2 class="relative group">How Kafka differs from RabbitMQ
    <div id="how-kafka-differs-from-rabbitmq" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#how-kafka-differs-from-rabbitmq" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>RabbitMQ is a smart broker: it routes messages into queues and deletes them when the consumer acks. A Kafka broker does something simpler. It appends messages to a topic's partitions in order and keeps them until retention expires. Each consumer group tracks how far it has read (its offset) on its own.</p>
<p>What that means in practice:</p>
<ul>
<li>Many consumer groups can read the same topic without knowing about each other. Billing, analytics and the search indexer can all consume order events separately.</li>
<li>After fixing a bug, you can rewind a group's offset and reprocess history.</li>
<li>Ordering is only guaranteed within a partition. If a user's events must stay in order, send them with the same key so they land on the same partition.</li>
</ul>
<p>For &quot;someone do this job and be done&quot; work queues, RabbitMQ is less hassle; I wrote about it in my <a href="https://blog.furkankoykiran.com.tr/en/posts/rabbitmq-message-queue/" >RabbitMQ post</a>. I pick Kafka for event streams, multiple consumers and replay.</p>

<h2 class="relative group">KRaft: ZooKeeper is gone
    <div id="kraft-zookeeper-is-gone" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#kraft-zookeeper-is-gone" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>For years Kafka needed a separate ZooKeeper cluster to store cluster metadata. In KRaft mode, Kafka's own controllers handle that with Raft consensus. Kafka 4.0 removed ZooKeeper support entirely. If you still run a ZooKeeper-based cluster, migrate to KRaft on the 3.x line first (3.9 is the bridge release) and upgrade to 4.x after; a ZooKeeper cluster can't be upgraded to 4.0 directly.</p>
<p>In production, the recommendation is three (or five) dedicated controller nodes plus separate brokers. For development, a single node can be both controller and broker, and that's exactly what the official Docker image does by default:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># compose.yaml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">services</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">kafka</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">apache/kafka:4.1.0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">container_name</span><span class="p">:</span><span class="w"> </span><span class="l">kafka</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">ports</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="s2">&#34;9092:9092&#34;</span></span></span></code></pre></div></div>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">docker compose up -d
</span></span><span class="line"><span class="cl">docker <span class="nb">exec</span> -it kafka /opt/kafka/bin/kafka-topics.sh <span class="se">\
</span></span></span><span class="line"><span class="cl">  --bootstrap-server localhost:9092 <span class="se">\
</span></span></span><span class="line"><span class="cl">  --create --topic orders --partitions <span class="m">6</span> --replication-factor <span class="m">1</span></span></span></code></pre></div></div>
<p>On your own installation, each node sets <code>process.roles</code> (<code>controller</code>, <code>broker</code> or both), and the storage directory has to be formatted with a cluster ID using <code>kafka-storage.sh format</code> before the first start. ZooKeeper setups had no such step, which is why it's the one people skip the first time.</p>

<h2 class="relative group">Choosing a partition count
    <div id="choosing-a-partition-count" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#choosing-a-partition-count" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Partitions are Kafka's unit of parallelism. Within a consumer group, only one consumer reads a given partition at a time. Attach 8 consumers to a 6-partition topic and 2 of them sit idle.</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/apache-kafka-event-streaming/img/partition-count_hu_6cb7c9b5a3dacccd.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/apache-kafka-event-streaming/img/partition-count_hu_b963e876fd17ba1b.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/apache-kafka-event-streaming/img/partition-count_hu_14a39b48e40e60fb.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/apache-kafka-event-streaming/img/partition-count_hu_c685d9458547955e.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/apache-kafka-event-streaming/img/partition-count_hu_e1484403b9104d64.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/apache-kafka-event-streaming/img/partition-count_hu_38dc5a0b27ca8467.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/apache-kafka-event-streaming/img/partition-count_hu_e1484403b9104d64.webp" alt="A producer writes to three partitions of the orders topic by key hash; in the billing group consumer 1 reads two partitions and consumer 2 reads one" width="1600" height="979"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Within a group, each partition has exactly one consumer</figcaption></figure><p>A rough calculation:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">partitions ≥ target throughput / throughput of one consumer</span></span></code></pre></div></div>
<p>If 1,000 messages arrive per second and one consumer can handle 100 per second (because it writes to a database), you need at least 10 partitions. Leave some headroom for growth, but don't create thousands: every partition costs files, memory and leader elections.</p>
<p>You can add partitions later but you can't remove them. Adding them also changes which partition a key maps to, so old and new messages for the same key can end up on different partitions. If you rely on per-key ordering, be generous up front.</p>

<h2 class="relative group">A producer that doesn't lose data
    <div id="a-producer-that-doesnt-lose-data" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#a-producer-that-doesnt-lose-data" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>For Python I use <code>confluent-kafka</code>, built on librdkafka (written in C) and fast. The settings that matter for reliability:</p>
<ul>
<li><code>acks=all</code>: a write only counts as successful once all in-sync replicas have it, not just the leader.</li>
<li><code>enable.idempotence=True</code>: retries after a network error don't write duplicates, and order within a partition is preserved. The Java client enables it by default; librdkafka doesn't, so set it explicitly.</li>
<li>On the broker side, replication factor 3 with <code>min.insync.replicas=2</code>. Losing one broker then neither stops writes nor leaves data with a single copy.</li>
</ul>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">confluent_kafka</span> <span class="kn">import</span> <span class="n">Producer</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">producer</span> <span class="o">=</span> <span class="n">Producer</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;bootstrap.servers&#34;</span><span class="p">:</span> <span class="s2">&#34;localhost:9092&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;acks&#34;</span><span class="p">:</span> <span class="s2">&#34;all&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;enable.idempotence&#34;</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;linger.ms&#34;</span><span class="p">:</span> <span class="mi">10</span><span class="p">,</span>  <span class="c1"># batch small messages for up to 10 ms</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;compression.type&#34;</span><span class="p">:</span> <span class="s2">&#34;zstd&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">on_delivery</span><span class="p">(</span><span class="n">err</span><span class="p">,</span> <span class="n">msg</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">err</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;delivery failed: </span><span class="si">{</span><span class="n">err</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">msg</span><span class="o">.</span><span class="n">topic</span><span class="p">()</span><span class="si">}</span><span class="s2">[</span><span class="si">{</span><span class="n">msg</span><span class="o">.</span><span class="n">partition</span><span class="p">()</span><span class="si">}</span><span class="s2">] offset=</span><span class="si">{</span><span class="n">msg</span><span class="o">.</span><span class="n">offset</span><span class="p">()</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">order</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;id&#34;</span><span class="p">:</span> <span class="mi">42</span><span class="p">,</span> <span class="s2">&#34;user_id&#34;</span><span class="p">:</span> <span class="mi">7</span><span class="p">,</span> <span class="s2">&#34;total&#34;</span><span class="p">:</span> <span class="mf">199.9</span><span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="n">producer</span><span class="o">.</span><span class="n">produce</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;orders&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">key</span><span class="o">=</span><span class="nb">str</span><span class="p">(</span><span class="n">order</span><span class="p">[</span><span class="s2">&#34;user_id&#34;</span><span class="p">]),</span>  <span class="c1"># same user, same partition</span>
</span></span><span class="line"><span class="cl">    <span class="n">value</span><span class="o">=</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">order</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(),</span>
</span></span><span class="line"><span class="cl">    <span class="n">on_delivery</span><span class="o">=</span><span class="n">on_delivery</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">producer</span><span class="o">.</span><span class="n">poll</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>    <span class="c1"># serve delivery callbacks</span>
</span></span><span class="line"><span class="cl"><span class="n">producer</span><span class="o">.</span><span class="n">flush</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>  <span class="c1"># send what&#39;s queued before shutting down</span></span></span></code></pre></div></div>
<p><code>produce</code> is asynchronous: it puts the message in a local queue and returns immediately. You learn the outcome in the callback, so logging or retrying failures there is your job. The producer is heavy and thread-safe; create one per application, not one per request.</p>
<p>Keep messages small. The broker's default limit is about 1 MB. Send a pointer to object storage instead of an image or a large file.</p>

<h2 class="relative group">Consumers: commit after the work is done
    <div id="consumers-commit-after-the-work-is-done" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#consumers-commit-after-the-work-is-done" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>With the default <code>enable.auto.commit=true</code>, offsets are committed periodically in the background. You receive a message, its offset gets committed, the process crashes mid-task: that message never comes back. I turn auto-commit off and commit after processing. That gives at-least-once delivery. After a crash a message may be processed twice, so make processing idempotent, for example with a unique constraint on the order ID.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">confluent_kafka</span> <span class="kn">import</span> <span class="n">Consumer</span><span class="p">,</span> <span class="n">KafkaException</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">consumer</span> <span class="o">=</span> <span class="n">Consumer</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;bootstrap.servers&#34;</span><span class="p">:</span> <span class="s2">&#34;localhost:9092&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;group.id&#34;</span><span class="p">:</span> <span class="s2">&#34;billing&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;enable.auto.commit&#34;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;auto.offset.reset&#34;</span><span class="p">:</span> <span class="s2">&#34;earliest&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;partition.assignment.strategy&#34;</span><span class="p">:</span> <span class="s2">&#34;cooperative-sticky&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;group.instance.id&#34;</span><span class="p">:</span> <span class="s2">&#34;billing-1&#34;</span><span class="p">,</span>  <span class="c1"># static membership; unique per instance</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">consumer</span><span class="o">.</span><span class="n">subscribe</span><span class="p">([</span><span class="s2">&#34;orders&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="n">order</span><span class="p">:</span> <span class="nb">dict</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="nb">print</span><span class="p">(</span><span class="s2">&#34;billed:&#34;</span><span class="p">,</span> <span class="n">order</span><span class="p">[</span><span class="s2">&#34;id&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">msg</span> <span class="o">=</span> <span class="n">consumer</span><span class="o">.</span><span class="n">poll</span><span class="p">(</span><span class="mf">1.0</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">msg</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">continue</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">msg</span><span class="o">.</span><span class="n">error</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">            <span class="k">raise</span> <span class="n">KafkaException</span><span class="p">(</span><span class="n">msg</span><span class="o">.</span><span class="n">error</span><span class="p">())</span>
</span></span><span class="line"><span class="cl">        <span class="n">process</span><span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">msg</span><span class="o">.</span><span class="n">value</span><span class="p">()))</span>
</span></span><span class="line"><span class="cl">        <span class="n">consumer</span><span class="o">.</span><span class="n">commit</span><span class="p">(</span><span class="n">message</span><span class="o">=</span><span class="n">msg</span><span class="p">,</span> <span class="n">asynchronous</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="k">finally</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">consumer</span><span class="o">.</span><span class="n">close</span><span class="p">()</span></span></span></code></pre></div></div>
<p>A synchronous commit after every message is simple but slow. When you need throughput, committing every few hundred messages or every few seconds is a common middle ground, at the cost of more reprocessing after a crash. For an asyncio service, <code>aiokafka</code> offers the same model with <code>async/await</code>.</p>

<h2 class="relative group">Rebalance storms
    <div id="rebalance-storms" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#rebalance-storms" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>When a consumer joins or leaves a group, partitions are reassigned. Under the old &quot;eager&quot; protocol, the whole group stops reading while that happens. The most common trigger: a consumer that doesn't call <code>poll</code> again within <code>max.poll.interval.ms</code> (5 minutes by default) is considered dead and kicked out. One slow database write can push the whole group into a rebalance loop.</p>
<p>What I set:</p>
<ol>
<li><code>partition.assignment.strategy=cooperative-sticky</code>: during a rebalance only the partitions that need to move are released, and the rest keep being consumed.</li>
<li>Static membership through <code>group.instance.id</code>: a consumer that restarts during a deploy and returns within <code>session.timeout.ms</code> triggers no rebalance at all.</li>
<li>Keep the work done per <code>poll</code> loop small enough to stay under <code>max.poll.interval.ms</code>. The Java client has <code>max.poll.records</code> for this. <code>confluent-kafka</code>'s <code>poll</code> returns one message anyway; if you batch with <code>consume(num_messages=...)</code>, keep the number low.</li>
</ol>
<p>Kafka 4.0 also made the new consumer group protocol (KIP-848) generally available. The broker does the assignment, and rebalances are incremental without stopping the group. Clients opt in with <code>group.protocol=consumer</code>, and recent librdkafka releases support it. It's worth trying on a new project.</p>

<h2 class="relative group">Log compaction
    <div id="log-compaction" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#log-compaction" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Normally, old segments are deleted when retention expires. On a topic with <code>cleanup.policy=compact</code>, Kafka keeps at least the latest value for every key. If <code>user:123</code> gets &quot;Alice&quot; and later &quot;Alicia&quot;, only &quot;Alicia&quot; survives compaction. The topic then behaves like the changelog of a key-value table: a new service can rebuild the current state by reading the topic from the start. To delete a key, send a message with a null value (a tombstone) for it.</p>

<h2 class="relative group">Keep unclean leader election off
    <div id="keep-unclean-leader-election-off" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#keep-unclean-leader-election-off" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>When a partition's leader dies, Kafka elects a new one from the in-sync replicas. With <code>unclean.leader.election.enable=true</code>, if no in-sync replica is left, Kafka promotes a replica that had fallen behind: the partition stays available, but every message that replica missed is gone. The default is <code>false</code>, and for orders, payments and similar data it should stay that way. You can turn it on per topic for logs or metrics where availability matters more than completeness.</p>

<h2 class="relative group">Monitoring
    <div id="monitoring" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#monitoring" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Running Kafka without monitoring means hearing about problems from users. I collect JMX metrics with the Prometheus JMX exporter and consumer lag with a tool like kafka-exporter or Burrow, then graph them in Grafana; the general setup is in my <a href="https://blog.furkankoykiran.com.tr/en/posts/prometheus-grafana-monitoring/" >Prometheus and Grafana post</a>. The metrics I alert on:</p>
<ul>
<li>Consumer lag: how far behind a group is. If it keeps growing, consumers can't keep up. Scale out or speed up processing.</li>
<li><code>UnderReplicatedPartitions</code>: above zero means a broker is lagging or down, and data has fewer copies than it should.</li>
<li><code>OfflinePartitionsCount</code>: partitions with no leader. Should be zero.</li>
<li><code>ActiveControllerCount</code>: summed across controllers, it should be exactly 1.</li>
</ul>

<h2 class="relative group">Where to go next
    <div id="where-to-go-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-to-go-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Start the single-node Kafka from the compose file above, run the producer, then start the consumer in two terminals with different <code>group.instance.id</code> values and watch how partitions are shared. If your events come from external systems, my <a href="https://blog.furkankoykiran.com.tr/en/posts/webhook-handler-event-driven-architecture/" >webhook handler post</a> covers that side, and if you feed a search index from Kafka, see my <a href="https://blog.furkankoykiran.com.tr/en/posts/elasticsearch-full-text-search/" >Elasticsearch post</a>. For the full list of settings, the <a href="https://kafka.apache.org/documentation/"  target="_blank" rel="noreferrer">Kafka documentation</a> and the <a href="https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html"  target="_blank" rel="noreferrer">confluent-kafka Python docs</a> are the references to trust.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/apache-kafka-event-streaming/feature_hu_daefe2752ba4b418.jpg" medium="image" type="image/jpeg"/></item><item><title>Git Branching Strategies: GitFlow vs Trunk-Based</title><link>https://blog.furkankoykiran.com.tr/en/posts/git-workflow-branching-strategies/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/git-workflow-branching-strategies/</guid><pubDate>Sat, 29 Nov 2025 10:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>DevOps</category><category>devops</category><category>ci-cd</category><category>github-actions</category><category>open-source</category><category>tooling</category><description>Git branching strategies compared: GitFlow, trunk-based development, merge queues, squash merges, Conventional Commits, pre-commit hooks and git bisect.</description><content:encoded><![CDATA[<p>Choosing a Git branching strategy eats more time than you'd expect once a team grows. Everyone has seen the Friday evening where three feature branches get merged into <code>develop</code> and the result is hundreds of conflicts. In this post I compare GitFlow with trunk-based development, then move on to the tools that help whichever one you pick: merge strategy, commit message conventions, pre-commit hooks and <code>git bisect</code>.</p>
<p>The setup I use in my own projects and open source contributions is simple: short-lived branch, pull request, squash merge once CI is green. Here's why.</p>

<h2 class="relative group">GitFlow: when it still makes sense
    <div id="gitflow-when-it-still-makes-sense" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#gitflow-when-it-still-makes-sense" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>GitFlow has two long-lived branches: <code>main</code> (or <code>master</code>) holds what's in production, <code>develop</code> holds the next release. Features are built on <code>feature/*</code> branches, a <code>release/*</code> branch is cut for each release, and urgent fixes go through <code>hotfix/*</code> into both <code>main</code> and <code>develop</code>.</p>
<p>The trouble is how long branches live. While you spend two weeks on <code>feature-x</code>, a teammate restructures the same module on <code>feature-y</code>. On merge day you both deal with two weeks of drift. Vincent Driessen, who described the model, added a note to <a href="https://nvie.com/posts/a-successful-git-branching-model/"  target="_blank" rel="noreferrer">the original article</a> in 2020 recommending a simpler workflow for continuously delivered web apps.</p>
<p>GitFlow still fits some cases: libraries that support several release lines at once, mobile apps waiting on store review, on-premise software shipped to customers. For a web service deployed several times a day, I find it too much ceremony.</p>

<h2 class="relative group">Trunk-based development
    <div id="trunk-based-development" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#trunk-based-development" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>One rule: everyone merges into the trunk (<code>main</code>) often, in small pieces. Branches exist, but they live for hours, a day or two at most.</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/git-workflow-branching-strategies/img/trunk-based-development_hu_f3dc519a7135a39b.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/git-workflow-branching-strategies/img/trunk-based-development_hu_6d076dda1380157c.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/git-workflow-branching-strategies/img/trunk-based-development_hu_3b76cee45039e215.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/git-workflow-branching-strategies/img/trunk-based-development_hu_b0e0fc44212ecda4.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/git-workflow-branching-strategies/img/trunk-based-development_hu_c82d8df6f8421d93.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/git-workflow-branching-strategies/img/trunk-based-development_hu_3f678864d4f48221.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/git-workflow-branching-strategies/img/trunk-based-development_hu_c82d8df6f8421d93.webp" alt="Trunk-based history: fix-login, feat-search and feat-search-ui each merge back into main after a single commit" width="1600" height="1204"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Branches live for hours, a day or two at most, so main stays current</figcaption></figure><p>Unfinished features go behind a feature flag. The code lands on <code>main</code> and even ships to production, but users don't see it while the flag is off. The simplest version is an environment variable:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">os</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">new_checkout_enabled</span><span class="p">(</span><span class="n">user_id</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&#34;FEATURE_NEW_CHECKOUT&#34;</span><span class="p">)</span> <span class="o">!=</span> <span class="s2">&#34;1&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="kc">False</span>
</span></span><span class="line"><span class="cl">    <span class="n">rollout</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&#34;NEW_CHECKOUT_ROLLOUT_PERCENT&#34;</span><span class="p">,</span> <span class="s2">&#34;0&#34;</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">user_id</span> <span class="o">%</span> <span class="mi">100</span> <span class="o">&lt;</span> <span class="n">rollout</span></span></span></code></pre></div></div>
<p>That's enough for a small team. Once you have many flags, move to a flag service or a standard like OpenFeature, otherwise stale flags pile up in the code and become their own kind of debt. Opening an issue to delete each flag when the feature is done is a good habit.</p>
<p>Trunk-based work depends on good CI. If everyone merges into <code>main</code> often, tests have to run on every PR and a red build has to block the merge.</p>
<p>The card below puts the two models side by side.</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/git-workflow-branching-strategies/img/gitflow-vs-trunk_hu_10eb48dd0a9a2896.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/git-workflow-branching-strategies/img/gitflow-vs-trunk_hu_a4c68d0452467910.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/git-workflow-branching-strategies/img/gitflow-vs-trunk_hu_829f684ac5372682.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/git-workflow-branching-strategies/img/gitflow-vs-trunk_hu_5f08306036b3e35.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/git-workflow-branching-strategies/img/gitflow-vs-trunk_hu_175c4e305fc9d7a3.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/git-workflow-branching-strategies/img/gitflow-vs-trunk_hu_2dfe3d83dd8f6730.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/git-workflow-branching-strategies/img/gitflow-vs-trunk_hu_175c4e305fc9d7a3.webp" alt="GitFlow keeps main and develop, with branches that live for weeks; trunk-based uses only main, short branches and feature flags" width="1600" height="1284"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>GitFlow and trunk-based side by side</figcaption></figure>
<h2 class="relative group">Merge queues
    <div id="merge-queues" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#merge-queues" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Bigger teams hit a new problem: two PRs are green on their own but break <code>main</code> together, because each was tested without the other. GitHub's merge queue fixes that. When you hit merge, the PR joins a queue, GitHub builds a temporary branch on top of the current <code>main</code> plus the PRs ahead of it, runs the checks there and merges only if they pass.</p>
<p>That removes the &quot;two green PRs broke main&quot; class of failures. Flaky tests, or tests that don't run in CI, can still break <code>main</code>. To use the queue, your workflow has to listen for the <code>merge_group</code> event:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">on</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">pull_request</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="l">merge_group:</span></span></span></code></pre></div></div>
<p>Forget this and the queue sits waiting for checks that never start.</p>

<h2 class="relative group">Merge, rebase or squash
    <div id="merge-rebase-or-squash" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#merge-rebase-or-squash" aria-label="Anchor">#</a>
    </span>
    
</h2>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Before:            A---B---C            main
</span></span><span class="line"><span class="cl">                        \
</span></span><span class="line"><span class="cl">                         D---E          feature
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">Merge commit:      A---B---C-------M    main
</span></span><span class="line"><span class="cl">                        \         /
</span></span><span class="line"><span class="cl">                         D-------E
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">Rebase and merge:  A---B---C---D&#39;--E&#39;   main   (D&#39; and E&#39; are new commits)
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">Squash and merge:  A---B---C---S        main   (S holds D + E as one commit)</span></span></code></pre></div></div>
<ul>
<li>Merge commit: the branch history stays as is, plus a merge commit on top. You can see who merged what and when, but the log gets busy.</li>
<li>Rebase and merge: commits are replayed one by one on top of <code>main</code>, so history stays linear. Good when every commit on the branch is meaningful.</li>
<li>Squash and merge: all commits on the branch become one.</li>
</ul>
<p>I prefer squash. The &quot;wip&quot;, &quot;typo&quot; and &quot;fix test&quot; commits never reach <code>main</code>, and every PR becomes one commit you can revert cleanly. The PR title becomes the commit message, so write it with care.</p>
<p>To keep your own branch current, rebase it, and when you need to force push, always use a lease:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">git fetch origin
</span></span><span class="line"><span class="cl">git rebase origin/main
</span></span><span class="line"><span class="cl">git push --force-with-lease</span></span></code></pre></div></div>
<p><code>--force-with-lease</code> refuses to push if someone else pushed to the remote branch since you last fetched. Plain <code>--force</code> silently deletes their commits. If you work with stacked branches, <code>git rebase --update-refs</code> moves the intermediate branches along with yours.</p>

<h2 class="relative group">Conventional Commits and automated releases
    <div id="conventional-commits-and-automated-releases" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#conventional-commits-and-automated-releases" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>When commit messages follow a format, tools can produce version numbers and changelogs for you. The <a href="https://www.conventionalcommits.org/"  target="_blank" rel="noreferrer">Conventional Commits</a> format:</p>
<ul>
<li><code>fix(payment): handle timeout</code> bumps the patch version (1.1.0 → 1.1.1).</li>
<li><code>feat(auth): add Google sign-in</code> bumps the minor version (1.1.0 → 1.2.0).</li>
<li><code>feat!: drop the v1 API</code>, or <code>BREAKING CHANGE:</code> in the body, bumps the major version (1.1.0 → 2.0.0).</li>
</ul>
<p>semantic-release and release-please read these messages and generate the version, tag and changelog. With squash merges, the PR title is what has to follow the format.</p>

<h2 class="relative group">Let tools enforce the rules
    <div id="let-tools-enforce-the-rules" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#let-tools-enforce-the-rules" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Leaving &quot;your commit message doesn't follow the format&quot; comments in code review wastes everyone's time. Check it at commit time with <a href="https://pre-commit.com/"  target="_blank" rel="noreferrer">pre-commit</a>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># .pre-commit-config.yaml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">default_install_hook_types</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">pre-commit, commit-msg]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">repos</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">repo</span><span class="p">:</span><span class="w"> </span><span class="l">https://github.com/pre-commit/pre-commit-hooks</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">rev</span><span class="p">:</span><span class="w"> </span><span class="l">v6.0.0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">hooks</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l">trailing-whitespace</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l">end-of-file-fixer</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l">check-yaml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l">check-added-large-files</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">repo</span><span class="p">:</span><span class="w"> </span><span class="l">https://github.com/astral-sh/ruff-pre-commit</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">rev</span><span class="p">:</span><span class="w"> </span><span class="l">v0.17.2</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">hooks</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l">ruff-check</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">args</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>--<span class="l">fix]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l">ruff-format</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">repo</span><span class="p">:</span><span class="w"> </span><span class="l">https://github.com/gitleaks/gitleaks</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">rev</span><span class="p">:</span><span class="w"> </span><span class="l">v8.30.1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">hooks</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l">gitleaks</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">repo</span><span class="p">:</span><span class="w"> </span><span class="l">https://github.com/compilerla/conventional-pre-commit</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">rev</span><span class="p">:</span><span class="w"> </span><span class="l">v4.4.0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">hooks</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l">conventional-pre-commit</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">stages</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">commit-msg]</span></span></span></code></pre></div></div>
<p>Running <code>pre-commit install</code> once sets up both the regular hooks and the commit message check. Bump versions now and then with <code>pre-commit autoupdate</code>. Run the same checks in CI too, because <code>git commit --no-verify</code> skips hooks. In JavaScript projects, Husky and commitlint do the same job. The file above is in pre-commit's own format, not a Husky config.</p>
<p>If you manage Python dependencies with lock files, the same &quot;let tools enforce it&quot; idea applies there; my <a href="https://blog.furkankoykiran.com.tr/en/posts/poetry-python-project-management/" >Poetry vs uv post</a> covers checking lock freshness in CI.</p>

<h2 class="relative group">git bisect: finding the commit that broke it
    <div id="git-bisect-finding-the-commit-that-broke-it" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#git-bisect-finding-the-commit-that-broke-it" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>&quot;It worked yesterday&quot; and there are a hundred commits in between. Instead of checking them one by one, <code>git bisect</code> runs a binary search; even with 1,000 commits you're done in about 10 steps. Give it a test command and it drives the steps itself:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">git bisect start
</span></span><span class="line"><span class="cl">git bisect bad HEAD
</span></span><span class="line"><span class="cl">git bisect good v1.4.0
</span></span><span class="line"><span class="cl">git bisect run pytest tests/test_checkout.py -x -q
</span></span><span class="line"><span class="cl">git bisect reset</span></span></code></pre></div></div>
<p><code>git bisect run</code> treats exit code 0 as good and 1 to 127 (except 125) as bad. Exit code 125 means &quot;can't test this commit, skip it&quot;.</p>

<h2 class="relative group">Common mistakes and how to recover
    <div id="common-mistakes-and-how-to-recover" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#common-mistakes-and-how-to-recover" aria-label="Anchor">#</a>
    </span>
    
</h2>
<ul>
<li>Commits you think you lost with <code>git reset --hard</code> usually aren't gone. <code>git reflog</code> keeps every position HEAD has been at for a while; get one back with <code>git branch rescued &lt;hash&gt;</code>.</li>
<li>If <code>.git</code> has grown to gigabytes, someone probably committed a database dump or a video. Remove it from history with <a href="https://github.com/newren/git-filter-repo"  target="_blank" rel="noreferrer">git-filter-repo</a>, after which everyone needs a fresh clone.</li>
<li>If you pushed an API key, revoke it and issue a new one first; cleaning history comes second. Treat any pushed secret as copied. Turn on GitHub secret scanning and push protection, and let the gitleaks hook above be your first line of defense locally.</li>
<li>Block direct pushes and force pushes to <code>main</code> with branch protection or rulesets. Code gets in through a PR with green CI, nothing else.</li>
<li>When a lock file (<code>poetry.lock</code>, <code>uv.lock</code>) conflicts, don't merge it by hand. Take the version from <code>main</code> and regenerate the lock.</li>
</ul>

<h2 class="relative group">Monorepo or polyrepo?
    <div id="monorepo-or-polyrepo" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#monorepo-or-polyrepo" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>In short: a monorepo lets you change the backend, frontend and a shared library in one commit, but at scale it needs tools like Nx, Turborepo or Bazel and a CI that only tests what changed. In a polyrepo setup every service deploys independently, but rolling a fix in a shared library out to fifty repos becomes a project of its own. For a five-person team either works; let team size and how often services change together decide.</p>

<h2 class="relative group">Tools that make Git nicer
    <div id="tools-that-make-git-nicer" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#tools-that-make-git-nicer" aria-label="Anchor">#</a>
    </span>
    
</h2>
<ul>
<li><a href="https://github.com/dandavison/delta"  target="_blank" rel="noreferrer">delta</a> adds syntax highlighting and side-by-side views to <code>git diff</code>.</li>
<li><a href="https://github.com/jesseduffield/lazygit"  target="_blank" rel="noreferrer">lazygit</a> is a terminal UI for Git. Staging hunks and interactive rebases are much faster with it.</li>
<li><code>git switch</code> and <code>git restore</code> split the two jobs of <code>git checkout</code> into commands that are harder to misuse.</li>
</ul>
<p>If your CI runs on GitHub Actions, my <a href="https://blog.furkankoykiran.com.tr/en/posts/github-actions-ci-cd-pipeline/" >CI/CD pipeline post</a> goes deeper into that side.</p>

<h2 class="relative group">What's next
    <div id="whats-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#whats-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>If your team still works with long-lived branches, don't change everything in a day. Put a limit on branch age (two days, say), switch to squash merges and commit a pre-commit config. Feature flags and merge queues come much easier once those habits stick.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/git-workflow-branching-strategies/feature_hu_23072c0129155aa5.jpg" medium="image" type="image/jpeg"/></item><item><title>Python Context Managers: Resource Management with with</title><link>https://blog.furkankoykiran.com.tr/en/posts/python-context-managers/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/python-context-managers/</guid><pubDate>Sat, 22 Nov 2025 15:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Python</category><category>python</category><category>asyncio</category><category>databases</category><description>A practical guide to Python context managers: __enter__ and __exit__, contextlib.contextmanager, ExitStack, async with, FastAPI lifespan, locks and common bugs.</description><content:encoded><![CDATA[<p>Everyone who writes Python knows <code>with open(&quot;file.txt&quot;) as f:</code>. The mechanism behind it, the Python context manager, isn't limited to files. Database transactions, locks, HTTP clients, temporary directories, timing: anything with a clear start and end can use it.</p>
<p>The point is that cleanup runs even when the block raises. Forgetting a <code>close()</code> or a <code>rollback()</code> is easy, and <code>with</code> guarantees it for you. This post shows how to write your own context managers, what <code>contextlib</code> already gives you, and how the async side works.</p>

<h2 class="relative group">What a with block actually does
    <div id="what-a-with-block-actually-does" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-a-with-block-actually-does" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>This code:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">with</span> <span class="n">manager</span> <span class="k">as</span> <span class="n">value</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">do_something</span><span class="p">(</span><span class="n">value</span><span class="p">)</span></span></span></code></pre></div></div>
<p>is roughly equivalent to:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">value</span> <span class="o">=</span> <span class="n">manager</span><span class="o">.</span><span class="fm">__enter__</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"><span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">do_something</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="k">except</span> <span class="ne">BaseException</span> <span class="k">as</span> <span class="n">exc</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">manager</span><span class="o">.</span><span class="fm">__exit__</span><span class="p">(</span><span class="nb">type</span><span class="p">(</span><span class="n">exc</span><span class="p">),</span> <span class="n">exc</span><span class="p">,</span> <span class="n">exc</span><span class="o">.</span><span class="n">__traceback__</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">raise</span>
</span></span><span class="line"><span class="cl"><span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">manager</span><span class="o">.</span><span class="fm">__exit__</span><span class="p">(</span><span class="kc">None</span><span class="p">,</span> <span class="kc">None</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span></span></span></code></pre></div></div>
<p><code>__enter__</code> sets up the resource and returns whatever gets bound by <code>as</code>. <code>__exit__</code> always runs. If something was raised, it receives the exception type, the exception and the traceback; otherwise all three are <code>None</code>. If <code>__exit__</code> returns <code>True</code>, the exception is swallowed; any other return value lets it propagate.</p>

<h2 class="relative group">A class-based context manager
    <div id="a-class-based-context-manager" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#a-class-based-context-manager" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Here's one for an SQLite transaction: commit on success, roll back on error, close the connection either way.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">sqlite3</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">Transaction</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">path</span> <span class="o">=</span> <span class="n">path</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="p">:</span> <span class="n">sqlite3</span><span class="o">.</span><span class="n">Connection</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__enter__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">sqlite3</span><span class="o">.</span><span class="n">Cursor</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">conn</span> <span class="o">=</span> <span class="n">sqlite3</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">path</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__exit__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">exc_type</span><span class="p">,</span> <span class="n">exc</span><span class="p">,</span> <span class="n">tb</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">if</span> <span class="n">exc_type</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">            <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="o">.</span><span class="n">rollback</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="k">finally</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="kc">False</span>  <span class="c1"># don&#39;t swallow the exception</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">Transaction</span><span class="p">(</span><span class="s2">&#34;app.db&#34;</span><span class="p">)</span> <span class="k">as</span> <span class="n">cur</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s2">&#34;CREATE TABLE IF NOT EXISTS users (name TEXT)&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s2">&#34;INSERT INTO users VALUES (&#39;furkan&#39;)&#34;</span><span class="p">)</span></span></span></code></pre></div></div>
<p><code>close()</code> goes in <code>finally</code> because the rollback itself can fail, and the connection still shouldn't be left open.</p>

<h3 class="relative group">An sqlite3 gotcha
    <div id="an-sqlite3-gotcha" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#an-sqlite3-gotcha" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p><code>sqlite3.Connection</code> is a context manager too, but it doesn't do what most people expect:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">conn</span> <span class="o">=</span> <span class="n">sqlite3</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="s2">&#34;app.db&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">conn</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">conn</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s2">&#34;INSERT INTO users VALUES (&#39;x&#39;)&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="c1"># The transaction is committed here, but the connection is STILL OPEN.</span></span></span></code></pre></div></div>
<p><code>with conn:</code> manages the transaction only; it doesn't close the connection. If you want that too, wrap it in <code>contextlib.closing</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">contextlib</span> <span class="kn">import</span> <span class="n">closing</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">closing</span><span class="p">(</span><span class="n">sqlite3</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="s2">&#34;app.db&#34;</span><span class="p">))</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">conn</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">conn</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s2">&#34;INSERT INTO users VALUES (&#39;y&#39;)&#34;</span><span class="p">)</span></span></span></code></pre></div></div>
<p>Running both side by side makes the difference obvious. After <code>with conn:</code> the connection still answers queries; after <code>closing()</code> the same query raises:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/python-context-managers/img/sqlite-with-vs-closing_hu_4e4910c5b468b1f3.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/python-context-managers/img/sqlite-with-vs-closing_hu_3d7ddfdb363e272c.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/python-context-managers/img/sqlite-with-vs-closing_hu_fcbc2d8bbfa9153e.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/python-context-managers/img/sqlite-with-vs-closing_hu_ba5e175859eee160.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/python-context-managers/img/sqlite-with-vs-closing_hu_364d0e4af3eefa9f.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/python-context-managers/img/sqlite-with-vs-closing_hu_e83932d7015da75f.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/python-context-managers/img/sqlite-with-vs-closing_hu_364d0e4af3eefa9f.webp" alt="Output of sqlite_trap.py: the query after with conn: returns (1,), the query after closing() raises ProgrammingError" width="1494" height="284"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>A real run on Python 3.12</figcaption></figure><p>It pays to check in the docs exactly what a library's context manager closes.</p>

<h2 class="relative group">The shortcut: contextlib.contextmanager
    <div id="the-shortcut-contextlibcontextmanager" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-shortcut-contextlibcontextmanager" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Writing a class every time is overkill. The <code>@contextmanager</code> decorator turns a generator with a single <code>yield</code> into a context manager. Code before the <code>yield</code> acts as <code>__enter__</code>, code after it as <code>__exit__</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">sqlite3</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">contextlib</span> <span class="kn">import</span> <span class="n">contextmanager</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@contextmanager</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">transaction</span><span class="p">(</span><span class="n">path</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">conn</span> <span class="o">=</span> <span class="n">sqlite3</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">yield</span> <span class="n">conn</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="ne">BaseException</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">conn</span><span class="o">.</span><span class="n">rollback</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="k">raise</span>
</span></span><span class="line"><span class="cl">    <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">conn</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">finally</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span></span></span></code></pre></div></div>
<p>An exception inside the <code>with</code> block is re-raised at the <code>yield</code> line. If the <code>yield</code> isn't inside a <code>try</code>, your cleanup never runs on error. That's the bug I see most often.</p>
<p>Context managers built with <code>@contextmanager</code> also work as decorators:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">time</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">contextlib</span> <span class="kn">import</span> <span class="n">contextmanager</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@contextmanager</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">timer</span><span class="p">(</span><span class="n">label</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">start</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">perf_counter</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">yield</span>
</span></span><span class="line"><span class="cl">    <span class="k">finally</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;[</span><span class="si">{</span><span class="n">label</span><span class="si">}</span><span class="s2">] </span><span class="si">{</span><span class="n">time</span><span class="o">.</span><span class="n">perf_counter</span><span class="p">()</span> <span class="o">-</span> <span class="n">start</span><span class="si">:</span><span class="s2">.3f</span><span class="si">}</span><span class="s2">s&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">timer</span><span class="p">(</span><span class="s2">&#34;report&#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">build_report</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@timer</span><span class="p">(</span><span class="s2">&#34;nightly job&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">nightly_job</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">    <span class="o">...</span></span></span></code></pre></div></div>
<p>Used as a decorator, a fresh context manager is created on every call, so the function can be called as many times as you like.</p>

<h2 class="relative group">An unknown number of resources: ExitStack
    <div id="an-unknown-number-of-resources-exitstack" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#an-unknown-number-of-resources-exitstack" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>When you only learn at runtime how many resources you'll open, nested <code>with</code> statements don't work. <code>ExitStack</code> handles it:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">contextlib</span> <span class="kn">import</span> <span class="n">ExitStack</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">paths</span> <span class="o">=</span> <span class="p">[</span><span class="s2">&#34;a.log&#34;</span><span class="p">,</span> <span class="s2">&#34;b.log&#34;</span><span class="p">,</span> <span class="s2">&#34;c.log&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">ExitStack</span><span class="p">()</span> <span class="k">as</span> <span class="n">stack</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">files</span> <span class="o">=</span> <span class="p">[</span><span class="n">stack</span><span class="o">.</span><span class="n">enter_context</span><span class="p">(</span><span class="nb">open</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="s2">&#34;a&#34;</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">&#34;utf-8&#34;</span><span class="p">))</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">paths</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">files</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s2">&#34;deploy finished</span><span class="se">\n</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="c1"># On exit, every opened file is closed in reverse order.</span></span></span></code></pre></div></div>
<p>If opening the third file fails, the first two are still closed. <code>stack.callback(fn, *args)</code> registers cleanup that isn't a context manager.</p>
<p>Another useful pattern: undo everything if setup fails halfway, and take ownership of the resources if it succeeds.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">with</span> <span class="n">ExitStack</span><span class="p">()</span> <span class="k">as</span> <span class="n">stack</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">db</span> <span class="o">=</span> <span class="n">stack</span><span class="o">.</span><span class="n">enter_context</span><span class="p">(</span><span class="n">open_db</span><span class="p">())</span>
</span></span><span class="line"><span class="cl">    <span class="n">cache</span> <span class="o">=</span> <span class="n">stack</span><span class="o">.</span><span class="n">enter_context</span><span class="p">(</span><span class="n">open_cache</span><span class="p">())</span>
</span></span><span class="line"><span class="cl">    <span class="n">resources</span> <span class="o">=</span> <span class="n">stack</span><span class="o">.</span><span class="n">pop_all</span><span class="p">()</span>  <span class="c1"># no error so far: take over the cleanup</span>
</span></span><span class="line"><span class="cl"><span class="c1"># calling resources.close() is now your job</span></span></span></code></pre></div></div>

<h2 class="relative group">async with
    <div id="async-with" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#async-with" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>In async code, opening and closing connections needs <code>await</code> too. That's what <code>__aenter__</code>, <code>__aexit__</code> and <code>async with</code> are for, and <code>asynccontextmanager</code> is the shortcut:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">contextlib</span> <span class="kn">import</span> <span class="n">asynccontextmanager</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">httpx</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@asynccontextmanager</span>
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">api_client</span><span class="p">(</span><span class="n">base_url</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">client</span> <span class="o">=</span> <span class="n">httpx</span><span class="o">.</span><span class="n">AsyncClient</span><span class="p">(</span><span class="n">base_url</span><span class="o">=</span><span class="n">base_url</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">10</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">yield</span> <span class="n">client</span>
</span></span><span class="line"><span class="cl">    <span class="k">finally</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">await</span> <span class="n">client</span><span class="o">.</span><span class="n">aclose</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">with</span> <span class="n">api_client</span><span class="p">(</span><span class="s2">&#34;https://api.example.com&#34;</span><span class="p">)</span> <span class="k">as</span> <span class="n">client</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">resp</span> <span class="o">=</span> <span class="k">await</span> <span class="n">client</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;/health&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="n">resp</span><span class="o">.</span><span class="n">status_code</span><span class="p">)</span></span></span></code></pre></div></div>
<p>Where I use this most is FastAPI's <code>lifespan</code> parameter, to open a connection pool at startup and close it at shutdown:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">contextlib</span> <span class="kn">import</span> <span class="n">asynccontextmanager</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">httpx</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">fastapi</span> <span class="kn">import</span> <span class="n">FastAPI</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@asynccontextmanager</span>
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">lifespan</span><span class="p">(</span><span class="n">app</span><span class="p">:</span> <span class="n">FastAPI</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">app</span><span class="o">.</span><span class="n">state</span><span class="o">.</span><span class="n">http</span> <span class="o">=</span> <span class="n">httpx</span><span class="o">.</span><span class="n">AsyncClient</span><span class="p">(</span><span class="n">timeout</span><span class="o">=</span><span class="mi">10</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">yield</span>
</span></span><span class="line"><span class="cl">    <span class="k">await</span> <span class="n">app</span><span class="o">.</span><span class="n">state</span><span class="o">.</span><span class="n">http</span><span class="o">.</span><span class="n">aclose</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">app</span> <span class="o">=</span> <span class="n">FastAPI</span><span class="p">(</span><span class="n">lifespan</span><span class="o">=</span><span class="n">lifespan</span><span class="p">)</span></span></span></code></pre></div></div>
<p>Connections that never get closed are one of the most common problems I run into in async services under load. Once the pool is exhausted, new requests queue up and everything slows down. Pool tuning is covered in the <a href="https://blog.furkankoykiran.com.tr/en/posts/database-connection-pool-optimization/" >connection pooling post</a>, and the FastAPI skeleton in the <a href="https://blog.furkankoykiran.com.tr/en/posts/python-fastapi-rest-api/" >FastAPI post</a>.</p>
<p>For opening a dynamic number of async resources there's <code>AsyncExitStack</code>, which works like <code>ExitStack</code>. asyncio's own <code>asyncio.timeout()</code> and <code>asyncio.TaskGroup</code> are async context managers as well; I cover them in the <a href="https://blog.furkankoykiran.com.tr/en/posts/python-asyncio-concurrent-processing/" >asyncio post</a>.</p>

<h2 class="relative group">What contextlib already gives you
    <div id="what-contextlib-already-gives-you" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-contextlib-already-gives-you" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Check the standard library before writing your own:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">os</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">threading</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">contextlib</span> <span class="kn">import</span> <span class="n">chdir</span><span class="p">,</span> <span class="n">nullcontext</span><span class="p">,</span> <span class="n">redirect_stdout</span><span class="p">,</span> <span class="n">suppress</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">io</span> <span class="kn">import</span> <span class="n">StringIO</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Ignore one specific exception on purpose</span>
</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">suppress</span><span class="p">(</span><span class="ne">FileNotFoundError</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">os</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="s2">&#34;temp.tmp&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Change the working directory temporarily (Python 3.11+)</span>
</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">chdir</span><span class="p">(</span><span class="s2">&#34;/tmp&#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">run_build</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Capture print output</span>
</span></span><span class="line"><span class="cl"><span class="n">buffer</span> <span class="o">=</span> <span class="n">StringIO</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">redirect_stdout</span><span class="p">(</span><span class="n">buffer</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">legacy_function_that_prints</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># A &#34;do nothing&#34; context manager for the conditional case</span>
</span></span><span class="line"><span class="cl"><span class="n">use_lock</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;THREADED&#34;</span><span class="p">)</span> <span class="o">==</span> <span class="s2">&#34;1&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">lock</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Lock</span><span class="p">()</span> <span class="k">if</span> <span class="n">use_lock</span> <span class="k">else</span> <span class="n">nullcontext</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">lock</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">update_counter</span><span class="p">()</span></span></span></code></pre></div></div>
<p><code>suppress</code> swallows only the exception types you name and states the intent more clearly than <code>try/except: pass</code>. Prefer it to a broad <code>except Exception: pass</code>. <code>chdir</code> is not thread-safe, because the working directory belongs to the whole process.</p>
<p>Since Python 3.10 you can group several context managers in parentheses, one per line:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">with</span> <span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="nb">open</span><span class="p">(</span><span class="s2">&#34;input.csv&#34;</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">&#34;utf-8&#34;</span><span class="p">)</span> <span class="k">as</span> <span class="n">src</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nb">open</span><span class="p">(</span><span class="s2">&#34;output.csv&#34;</span><span class="p">,</span> <span class="s2">&#34;w&#34;</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">&#34;utf-8&#34;</span><span class="p">)</span> <span class="k">as</span> <span class="n">dst</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">dst</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">src</span><span class="o">.</span><span class="n">read</span><span class="p">())</span></span></span></code></pre></div></div>

<h2 class="relative group">Locks
    <div id="locks" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#locks" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Locks are context managers too. Use <code>with</code> instead of <code>lock.acquire()</code> and <code>lock.release()</code>; otherwise an exception leaves the lock held and the program deadlocks:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">threading</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">lock</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Lock</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"><span class="n">shared_counter</span> <span class="o">=</span> <span class="mi">0</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">lock</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">shared_counter</span> <span class="o">+=</span> <span class="mi">1</span></span></span></code></pre></div></div>
<p>In async code, <code>asyncio.Lock</code> works the same way with <code>async with</code>. When separate processes write to the same file, the <code>filelock</code> package does the job through a lock file:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">filelock</span> <span class="kn">import</span> <span class="n">FileLock</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">FileLock</span><span class="p">(</span><span class="s2">&#34;data.txt.lock&#34;</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">10</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s2">&#34;data.txt&#34;</span><span class="p">,</span> <span class="s2">&#34;a&#34;</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">&#34;utf-8&#34;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s2">&#34;safe line</span><span class="se">\n</span><span class="s2">&#34;</span><span class="p">)</span></span></span></code></pre></div></div>
<p>Without a <code>timeout</code>, a lock that's never released makes the process wait forever.</p>

<h2 class="relative group">Common mistakes
    <div id="common-mistakes" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#common-mistakes" aria-label="Anchor">#</a>
    </span>
    
</h2>
<ul>
<li>An unconditional <code>return True</code> in <code>__exit__</code>. It silently discards every exception from the block. Return <code>True</code> only for exception types you mean to swallow.</li>
<li>Leaving the <code>yield</code> outside <code>try/finally</code> in a <code>@contextmanager</code> function. Cleanup won't run on error.</li>
<li>More than one <code>yield</code> in the generator. <code>@contextmanager</code> expects exactly one and raises <code>RuntimeError</code> on the second.</li>
<li>Using the resource after the <code>with</code> block. By then the file is closed and the connection is back in the pool.</li>
<li>Acquiring the resource in <code>__init__</code>. If the object is created but never entered, the resource leaks. Acquire it in <code>__enter__</code>.</li>
</ul>
<p>You already use a version of the same idea in tests: pytest fixtures with <code>yield</code> split setup and teardown the same way. More in the <a href="https://blog.furkankoykiran.com.tr/en/posts/pytest-advanced-testing-techniques/" >pytest post</a>.</p>

<h2 class="relative group">Where to go from here
    <div id="where-to-go-from-here" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-to-go-from-here" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Search your code for repeated <code>try ... finally</code> blocks. If the same setup and teardown shows up in two or three places, move it into a <code>@contextmanager</code> function. If you're on FastAPI, replace <code>on_event(&quot;startup&quot;)</code> and <code>on_event(&quot;shutdown&quot;)</code> with <code>lifespan</code>.</p>
<p>Docs: <a href="https://docs.python.org/3/library/contextlib.html"  target="_blank" rel="noreferrer">contextlib</a>, <a href="https://docs.python.org/3/reference/compound_stmts.html#the-with-statement"  target="_blank" rel="noreferrer">the with statement</a>, <a href="https://peps.python.org/pep-0343/"  target="_blank" rel="noreferrer">PEP 343</a>.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/python-context-managers/feature_hu_cfb57c51f7cb812f.jpg" medium="image" type="image/jpeg"/></item><item><title>Modern Web UI Without a Framework: CSS and JavaScript</title><link>https://blog.furkankoykiran.com.tr/en/posts/modern-web-ui-css-javascript/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/modern-web-ui-css-javascript/</guid><pubDate>Sat, 15 Nov 2025 11:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Backend</category><category>web</category><category>performance</category><category>tooling</category><description>A modern web UI without a framework: container queries, :has() and CSS nesting, popover and dialog, Web Components, islands, and measuring Core Web Vitals.</description><content:encoded><![CDATA[<p>You don't need React or Vue in every project to build a modern web UI. CSS and HTML features that shipped in browsers over the last few years now cover a lot of what used to take JavaScript or a library. For a dashboard, a landing page or the admin panel of a backend service, plain HTML, modern CSS and a little vanilla JavaScript are often enough.</p>
<p>I'm not against frameworks; I built my <a href="https://blog.furkankoykiran.com.tr/en/posts/awesome-trending-repos-web/" >awesome-trending-repos site</a> with React and Vite. This post is about what the browser gives you before you reach for one. Every CSS feature here works in current versions of Chrome, Firefox and Safari.</p>

<h2 class="relative group">Container queries: let components look at their own width
    <div id="container-queries-let-components-look-at-their-own-width" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#container-queries-let-components-look-at-their-own-width" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Media queries look at the viewport. A card component, though, sometimes sits in a wide main column and sometimes in a narrow sidebar, so the viewport width is the wrong question. With a container query, the component changes layout based on the width of the box it lives in:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-css" data-lang="css"><span class="line"><span class="cl"><span class="p">.</span><span class="nc">card-wrap</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="k">container-type</span><span class="p">:</span> <span class="kc">inline</span><span class="o">-</span><span class="k">size</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="p">.</span><span class="nc">card</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="k">display</span><span class="p">:</span> <span class="k">grid</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="k">gap</span><span class="p">:</span> <span class="mf">0.75</span><span class="kt">rem</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="k">padding</span><span class="p">:</span> <span class="mi">1</span><span class="kt">rem</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="p">@</span><span class="k">container</span> <span class="o">(</span><span class="nt">min-width</span><span class="o">:</span> <span class="nt">28rem</span><span class="o">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="p">.</span><span class="nc">card</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">grid-template-columns</span><span class="p">:</span> <span class="mi">8</span><span class="kt">rem</span> <span class="mi">1</span><span class="n">fr</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>The same <code>.card</code> now picks the right layout wherever you drop it. Here is the card CSS from this post rendered by Chrome in two containers of different widths:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/modern-web-ui-css-javascript/img/container-query-card-demo_hu_deb7a217f7bf9549.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/modern-web-ui-css-javascript/img/container-query-card-demo_hu_bc23c6cb8695231b.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/modern-web-ui-css-javascript/img/container-query-card-demo_hu_9c27b72b47f6efb3.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/modern-web-ui-css-javascript/img/container-query-card-demo_hu_d684404a7f36fb8f.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/modern-web-ui-css-javascript/img/container-query-card-demo_hu_27c1be6e9cb7c7be.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/modern-web-ui-css-javascript/img/container-query-card-demo_hu_c016d027c1d5b7fb.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/modern-web-ui-css-javascript/img/container-query-card-demo_hu_27c1be6e9cb7c7be.webp" alt="The same card drawn with the image on top in an 18rem sidebar and with the image on the left in a 40rem column" width="1600" height="608"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Same component, same CSS: the container's width decides the layout</figcaption></figure><p>The viewport is the same for both cards; only the container differs. That's why most component-level media queries become unnecessary.</p>

<h2 class="relative group">Nesting, cascade layers and theme variables
    <div id="nesting-cascade-layers-and-theme-variables" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#nesting-cascade-layers-and-theme-variables" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Nesting was the most used Sass feature, and CSS has it natively now. The same card with nesting, <code>@layer</code> and <code>light-dark()</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-css" data-lang="css"><span class="line"><span class="cl"><span class="p">@</span><span class="k">layer</span> <span class="nt">reset</span><span class="o">,</span> <span class="nt">base</span><span class="o">,</span> <span class="nt">components</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="p">:</span><span class="nd">root</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="k">color-scheme</span><span class="p">:</span> <span class="n">light</span> <span class="n">dark</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="nv">--bg</span><span class="p">:</span> <span class="nf">light-dark</span><span class="p">(</span><span class="mh">#ffffff</span><span class="p">,</span> <span class="mh">#0f1115</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="nv">--fg</span><span class="p">:</span> <span class="nf">light-dark</span><span class="p">(</span><span class="mh">#1b1f24</span><span class="p">,</span> <span class="mh">#e6e8eb</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="nv">--accent</span><span class="p">:</span> <span class="nf">light-dark</span><span class="p">(</span><span class="mh">#0b63ce</span><span class="p">,</span> <span class="mh">#5aa2ff</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="p">@</span><span class="k">layer</span> <span class="nt">base</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nt">body</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">background</span><span class="p">:</span> <span class="nf">var</span><span class="p">(</span><span class="o">--</span><span class="n">bg</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="k">color</span><span class="p">:</span> <span class="nf">var</span><span class="p">(</span><span class="o">--</span><span class="n">fg</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="k">font-family</span><span class="p">:</span> <span class="n">system-ui</span><span class="p">,</span> <span class="kc">sans-serif</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="p">@</span><span class="k">layer</span> <span class="nt">components</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="p">.</span><span class="nc">card</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">display</span><span class="p">:</span> <span class="k">grid</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="k">gap</span><span class="p">:</span> <span class="mf">0.75</span><span class="kt">rem</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="k">padding</span><span class="p">:</span> <span class="mi">1</span><span class="kt">rem</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="k">border</span><span class="p">:</span> <span class="mi">1</span><span class="kt">px</span> <span class="kc">solid</span> <span class="nf">color-mix</span><span class="p">(</span><span class="n">in</span> <span class="n">oklab</span><span class="p">,</span> <span class="nf">var</span><span class="p">(</span><span class="o">--</span><span class="n">fg</span><span class="p">)</span> <span class="mi">15</span><span class="kt">%</span><span class="p">,</span> <span class="kc">transparent</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="k">border-radius</span><span class="p">:</span> <span class="mf">0.75</span><span class="kt">rem</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="err">&amp;</span> <span class="err">h3</span> <span class="err">{</span>
</span></span><span class="line"><span class="cl">      <span class="k">margin</span><span class="p">:</span> <span class="mi">0</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="o">&amp;</span><span class="p">:</span><span class="nd">hover</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="k">border-color</span><span class="p">:</span> <span class="nf">var</span><span class="p">(</span><span class="o">--</span><span class="n">accent</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="p">@</span><span class="k">container</span> <span class="o">(</span><span class="nt">min-width</span><span class="o">:</span> <span class="nt">28rem</span><span class="o">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nt">grid-template-columns</span><span class="o">:</span> <span class="nt">8rem</span> <span class="nt">1fr</span><span class="o">;</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="err">}</span></span></span></code></pre></div></div>
<p>Three things are going on:</p>
<ul>
<li><code>color-scheme: light dark</code> with <code>light-dark()</code> picks the right color for the user's system theme, with no separate <code>prefers-color-scheme</code> block.</li>
<li><code>@layer</code> decides which styles win. A rule in the <code>components</code> layer beats a more specific selector in <code>base</code>. That ends most <code>!important</code> fights.</li>
<li><code>color-mix()</code> derives a translucent border color from the text color, so the design system doesn't need another variable.</li>
</ul>

<h2 class="relative group">Selecting parents with :has()
    <div id="selecting-parents-with-has" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#selecting-parents-with-has" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p><code>:has()</code> fills the gap that was known for years as &quot;CSS has no parent selector&quot;. You can style an element based on what it contains:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-css" data-lang="css"><span class="line"><span class="cl"><span class="c">/* Cards with an image get a different layout */</span>
</span></span><span class="line"><span class="cl"><span class="p">.</span><span class="nc">card</span><span class="p">:</span><span class="nd">has</span><span class="o">(</span><span class="nt">img</span><span class="o">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="k">grid-template-rows</span><span class="p">:</span> <span class="kc">auto</span> <span class="mi">1</span><span class="n">fr</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c">/* Dim the submit button while the form has an invalid field the user touched */</span>
</span></span><span class="line"><span class="cl"><span class="nt">form</span><span class="p">:</span><span class="nd">has</span><span class="o">(</span><span class="p">:</span><span class="nd">user-invalid</span><span class="o">)</span> <span class="nt">button</span><span class="o">[</span><span class="nt">type</span><span class="o">=</span><span class="s2">&#34;submit&#34;</span><span class="o">]</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="k">opacity</span><span class="p">:</span> <span class="mf">0.5</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c">/* The label of a checked checkbox */</span>
</span></span><span class="line"><span class="cl"><span class="nt">label</span><span class="p">:</span><span class="nd">has</span><span class="o">(</span><span class="nt">input</span><span class="p">:</span><span class="nd">checked</span><span class="o">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="k">font-weight</span><span class="p">:</span> <span class="mi">600</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>Each of these used to need an event listener toggling a class. And because <code>:user-invalid</code> only matches after the user has interacted with a field, the form doesn't greet people with red error boxes on page load.</p>

<h2 class="relative group">Interaction without JavaScript: dialog and popover
    <div id="interaction-without-javascript-dialog-and-popover" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#interaction-without-javascript-dialog-and-popover" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Modals and dropdown menus were two of the most common reasons to install a library. Both are in HTML now:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-html" data-lang="html"><span class="line"><span class="cl"><span class="p">&lt;</span><span class="nt">button</span> <span class="na">popovertarget</span><span class="o">=</span><span class="s">&#34;user-menu&#34;</span><span class="p">&gt;</span>Account<span class="p">&lt;/</span><span class="nt">button</span><span class="p">&gt;</span>
</span></span><span class="line"><span class="cl"><span class="p">&lt;</span><span class="nt">div</span> <span class="na">id</span><span class="o">=</span><span class="s">&#34;user-menu&#34;</span> <span class="na">popover</span><span class="p">&gt;</span>
</span></span><span class="line"><span class="cl">  <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">&#34;/profile&#34;</span><span class="p">&gt;</span>Profile<span class="p">&lt;/</span><span class="nt">a</span><span class="p">&gt;</span>
</span></span><span class="line"><span class="cl">  <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">&#34;/logout&#34;</span><span class="p">&gt;</span>Log out<span class="p">&lt;/</span><span class="nt">a</span><span class="p">&gt;</span>
</span></span><span class="line"><span class="cl"><span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="p">&lt;</span><span class="nt">dialog</span> <span class="na">id</span><span class="o">=</span><span class="s">&#34;confirm&#34;</span><span class="p">&gt;</span>
</span></span><span class="line"><span class="cl">  <span class="p">&lt;</span><span class="nt">form</span> <span class="na">method</span><span class="o">=</span><span class="s">&#34;dialog&#34;</span><span class="p">&gt;</span>
</span></span><span class="line"><span class="cl">    <span class="p">&lt;</span><span class="nt">p</span><span class="p">&gt;</span>Are you sure you want to delete this?<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span>
</span></span><span class="line"><span class="cl">    <span class="p">&lt;</span><span class="nt">button</span> <span class="na">value</span><span class="o">=</span><span class="s">&#34;cancel&#34;</span><span class="p">&gt;</span>Cancel<span class="p">&lt;/</span><span class="nt">button</span><span class="p">&gt;</span>
</span></span><span class="line"><span class="cl">    <span class="p">&lt;</span><span class="nt">button</span> <span class="na">value</span><span class="o">=</span><span class="s">&#34;ok&#34;</span><span class="p">&gt;</span>Delete<span class="p">&lt;/</span><span class="nt">button</span><span class="p">&gt;</span>
</span></span><span class="line"><span class="cl">  <span class="p">&lt;/</span><span class="nt">form</span><span class="p">&gt;</span>
</span></span><span class="line"><span class="cl"><span class="p">&lt;/</span><span class="nt">dialog</span><span class="p">&gt;</span>
</span></span><span class="line"><span class="cl"><span class="p">&lt;</span><span class="nt">button</span> <span class="na">onclick</span><span class="o">=</span><span class="s">&#34;document.getElementById(&#39;confirm&#39;).showModal()&#34;</span><span class="p">&gt;</span>Delete<span class="p">&lt;/</span><span class="nt">button</span><span class="p">&gt;</span></span></span></code></pre></div></div>
<p>A popover closes on outside click and on <code>Esc</code>, and renders in the top layer, so there's no <code>z-index</code> to tune. A dialog opened with <code>showModal()</code> traps focus, makes the background inert and closes on <code>Esc</code>. Those are exactly the accessibility details that hand-rolled versions tend to forget.</p>

<h2 class="relative group">Islands: JavaScript only where it's needed
    <div id="islands-javascript-only-where-its-needed" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#islands-javascript-only-where-its-needed" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>If most of a page is static content, there's no reason to turn the whole page into a JavaScript app. That's the &quot;islands&quot; idea behind tools like Astro: render HTML on the server and hydrate only the interactive parts. A simple version needs no build step:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-html" data-lang="html"><span class="line"><span class="cl"><span class="p">&lt;</span><span class="nt">div</span> <span class="na">data-island</span><span class="o">=</span><span class="s">&#34;price-chart&#34;</span> <span class="na">data-symbol</span><span class="o">=</span><span class="s">&#34;BTCUSDT&#34;</span><span class="p">&gt;&lt;/</span><span class="nt">div</span><span class="p">&gt;</span></span></span></code></pre></div></div>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-javascript" data-lang="javascript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">io</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">IntersectionObserver</span><span class="p">((</span><span class="nx">entries</span><span class="p">)</span> <span class="p">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="k">for</span> <span class="p">(</span><span class="kr">const</span> <span class="nx">entry</span> <span class="k">of</span> <span class="nx">entries</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">entry</span><span class="p">.</span><span class="nx">isIntersecting</span><span class="p">)</span> <span class="k">continue</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="nx">io</span><span class="p">.</span><span class="nx">unobserve</span><span class="p">(</span><span class="nx">entry</span><span class="p">.</span><span class="nx">target</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="kr">const</span> <span class="nx">el</span> <span class="o">=</span> <span class="nx">entry</span><span class="p">.</span><span class="nx">target</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="kr">import</span><span class="p">(</span><span class="sb">`./islands/</span><span class="si">${</span><span class="nx">el</span><span class="p">.</span><span class="nx">dataset</span><span class="p">.</span><span class="nx">island</span><span class="si">}</span><span class="sb">.js`</span><span class="p">).</span><span class="nx">then</span><span class="p">((</span><span class="nx">mod</span><span class="p">)</span> <span class="p">=&gt;</span> <span class="nx">mod</span><span class="p">.</span><span class="nx">mount</span><span class="p">(</span><span class="nx">el</span><span class="p">));</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">document</span><span class="p">.</span><span class="nx">querySelectorAll</span><span class="p">(</span><span class="s2">&#34;[data-island]&#34;</span><span class="p">).</span><span class="nx">forEach</span><span class="p">((</span><span class="nx">el</span><span class="p">)</span> <span class="p">=&gt;</span> <span class="nx">io</span><span class="p">.</span><span class="nx">observe</span><span class="p">(</span><span class="nx">el</span><span class="p">));</span></span></span></code></pre></div></div>
<p>Each island's code is fetched only when it scrolls into view. A chart at the bottom of the page costs no network or main-thread time until the user gets there. For an island that shows live data, feeding it from the server over <a href="https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/" >WebSockets</a> or SSE is a natural match.</p>

<h2 class="relative group">Web Components: framework-independent components
    <div id="web-components-framework-independent-components" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#web-components-framework-independent-components" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Components don't require a library either. Custom elements and Shadow DOM are browser standards and work inside any framework, which makes them a good fit for a UI kit shared across projects:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-javascript" data-lang="javascript"><span class="line"><span class="cl"><span class="kr">class</span> <span class="nx">ProductCard</span> <span class="kr">extends</span> <span class="nx">HTMLElement</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nx">connectedCallback</span><span class="p">()</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">shadowRoot</span><span class="p">)</span> <span class="k">return</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="kr">const</span> <span class="nx">root</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">attachShadow</span><span class="p">({</span> <span class="nx">mode</span><span class="o">:</span> <span class="s2">&#34;open&#34;</span> <span class="p">});</span>
</span></span><span class="line"><span class="cl">    <span class="nx">root</span><span class="p">.</span><span class="nx">innerHTML</span> <span class="o">=</span> <span class="sb">`
</span></span></span><span class="line"><span class="cl"><span class="sb">      &lt;style&gt;
</span></span></span><span class="line"><span class="cl"><span class="sb">        :host { display: block; container-type: inline-size; }
</span></span></span><span class="line"><span class="cl"><span class="sb">        .card { padding: 1rem; border: 1px solid #ccc; border-radius: 0.5rem; }
</span></span></span><span class="line"><span class="cl"><span class="sb">      &lt;/style&gt;
</span></span></span><span class="line"><span class="cl"><span class="sb">      &lt;div class=&#34;card&#34;&gt;
</span></span></span><span class="line"><span class="cl"><span class="sb">        &lt;h3&gt;&lt;/h3&gt;
</span></span></span><span class="line"><span class="cl"><span class="sb">        &lt;slot name=&#34;description&#34;&gt;&lt;/slot&gt;
</span></span></span><span class="line"><span class="cl"><span class="sb">      &lt;/div&gt;`</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="c1">// Don&#39;t interpolate the attribute into innerHTML; textContent can&#39;t inject markup
</span></span></span><span class="line"><span class="cl">    <span class="nx">root</span><span class="p">.</span><span class="nx">querySelector</span><span class="p">(</span><span class="s2">&#34;h3&#34;</span><span class="p">).</span><span class="nx">textContent</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">getAttribute</span><span class="p">(</span><span class="s2">&#34;name&#34;</span><span class="p">)</span> <span class="o">??</span> <span class="s2">&#34;&#34;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nx">customElements</span><span class="p">.</span><span class="nx">define</span><span class="p">(</span><span class="s2">&#34;product-card&#34;</span><span class="p">,</span> <span class="nx">ProductCard</span><span class="p">);</span></span></span></code></pre></div></div>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-html" data-lang="html"><span class="line"><span class="cl"><span class="p">&lt;</span><span class="nt">product-card</span> <span class="na">name</span><span class="o">=</span><span class="s">&#34;Running shoe&#34;</span><span class="p">&gt;</span>
</span></span><span class="line"><span class="cl">  <span class="p">&lt;</span><span class="nt">p</span> <span class="na">slot</span><span class="o">=</span><span class="s">&#34;description&#34;</span><span class="p">&gt;</span>Light, breathable upper.<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span>
</span></span><span class="line"><span class="cl"><span class="p">&lt;/</span><span class="nt">product-card</span><span class="p">&gt;</span></span></span></code></pre></div></div>
<p>Putting a value that might come from a user into <code>innerHTML</code> through a template string is a very common security bug in framework-free code. Write the static template with <code>innerHTML</code> and the dynamic values with <code>textContent</code>.</p>

<h2 class="relative group">State: events and a tiny signal
    <div id="state-events-and-a-tiny-signal" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#state-events-and-a-tiny-signal" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>On a medium-sized page you rarely need a global store for components to talk. The DOM's own event system is a pub/sub mechanism:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-javascript" data-lang="javascript"><span class="line"><span class="cl"><span class="nb">document</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="s2">&#34;click&#34;</span><span class="p">,</span> <span class="p">(</span><span class="nx">event</span><span class="p">)</span> <span class="p">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">button</span> <span class="o">=</span> <span class="nx">event</span><span class="p">.</span><span class="nx">target</span><span class="p">.</span><span class="nx">closest</span><span class="p">(</span><span class="s2">&#34;[data-action=&#39;add-to-cart&#39;]&#34;</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">button</span><span class="p">)</span> <span class="k">return</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">cart</span><span class="p">.</span><span class="nx">push</span><span class="p">(</span><span class="nx">button</span><span class="p">.</span><span class="nx">dataset</span><span class="p">.</span><span class="nx">sku</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="nb">document</span><span class="p">.</span><span class="nx">dispatchEvent</span><span class="p">(</span><span class="k">new</span> <span class="nx">CustomEvent</span><span class="p">(</span><span class="s2">&#34;cart:updated&#34;</span><span class="p">,</span> <span class="p">{</span> <span class="nx">detail</span><span class="o">:</span> <span class="p">{</span> <span class="nx">count</span><span class="o">:</span> <span class="nx">cart</span><span class="p">.</span><span class="nx">length</span> <span class="p">}</span> <span class="p">}));</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">document</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="s2">&#34;cart:updated&#34;</span><span class="p">,</span> <span class="p">(</span><span class="nx">event</span><span class="p">)</span> <span class="p">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nb">document</span><span class="p">.</span><span class="nx">querySelector</span><span class="p">(</span><span class="s2">&#34;#cart-count&#34;</span><span class="p">).</span><span class="nx">textContent</span> <span class="o">=</span> <span class="nx">event</span><span class="p">.</span><span class="nx">detail</span><span class="p">.</span><span class="nx">count</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span></span></span></code></pre></div></div>
<p>The first listener uses event delegation: instead of attaching a listener to hundreds of buttons, one listener checks where the click came from. Buttons added later work automatically.</p>
<p>For a value read in many places, a small signal does the job:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-javascript" data-lang="javascript"><span class="line"><span class="cl"><span class="kd">function</span> <span class="nx">signal</span><span class="p">(</span><span class="nx">initial</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kd">let</span> <span class="nx">value</span> <span class="o">=</span> <span class="nx">initial</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">subscribers</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Set</span><span class="p">();</span>
</span></span><span class="line"><span class="cl">  <span class="k">return</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nx">get</span><span class="o">:</span> <span class="p">()</span> <span class="p">=&gt;</span> <span class="nx">value</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nx">set</span><span class="p">(</span><span class="nx">next</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nx">value</span> <span class="o">=</span> <span class="nx">next</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">      <span class="nx">subscribers</span><span class="p">.</span><span class="nx">forEach</span><span class="p">((</span><span class="nx">fn</span><span class="p">)</span> <span class="p">=&gt;</span> <span class="nx">fn</span><span class="p">(</span><span class="nx">value</span><span class="p">));</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="nx">subscribe</span><span class="p">(</span><span class="nx">fn</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nx">subscribers</span><span class="p">.</span><span class="nx">add</span><span class="p">(</span><span class="nx">fn</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">      <span class="nx">fn</span><span class="p">(</span><span class="nx">value</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">      <span class="k">return</span> <span class="p">()</span> <span class="p">=&gt;</span> <span class="nx">subscribers</span><span class="p">.</span><span class="k">delete</span><span class="p">(</span><span class="nx">fn</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl">  <span class="p">};</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">count</span> <span class="o">=</span> <span class="nx">signal</span><span class="p">(</span><span class="mi">0</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="nx">count</span><span class="p">.</span><span class="nx">subscribe</span><span class="p">((</span><span class="nx">v</span><span class="p">)</span> <span class="p">=&gt;</span> <span class="p">(</span><span class="nb">document</span><span class="p">.</span><span class="nx">querySelector</span><span class="p">(</span><span class="s2">&#34;#cart-count&#34;</span><span class="p">).</span><span class="nx">textContent</span> <span class="o">=</span> <span class="nx">v</span><span class="p">));</span>
</span></span><span class="line"><span class="cl"><span class="nx">count</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="mi">3</span><span class="p">);</span></span></span></code></pre></div></div>
<p>Solid, Preact Signals, Vue and Angular all build on this idea, and there's a TC39 proposal to standardize it. When your state outgrows something like this, that's a good sign it's time for a framework.</p>

<h2 class="relative group">Measure performance
    <div id="measure-performance" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#measure-performance" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Google's Core Web Vitals are LCP (when the largest content appears), INP (how fast the page responds to interactions) and CLS (layout shift). INP replaced FID in March 2024 and measures every interaction across the page's lifetime, not just the first click. Large JavaScript bundles hurt it most, because clicks wait while the main thread is busy.</p>
<p>Measure instead of guessing:</p>
<ul>
<li>Record a trace in the Chrome DevTools Performance panel. Long tasks (over 50 ms) and forced reflow warnings from interleaved DOM reads and writes are the first things to look at.</li>
<li>For real-user data, the <code>web-vitals</code> library is a few KB and can send metrics to your own analytics endpoint.</li>
<li>Run Lighthouse CI and axe-core on every PR, and fail the build when the performance score or accessibility checks drop below your threshold.</li>
</ul>

<h2 class="relative group">Accessibility and the law
    <div id="accessibility-and-the-law" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#accessibility-and-the-law" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The European Accessibility Act has applied since June 2025 and makes accessibility a requirement for many digital products and services in the EU. Technically, most of it comes down to using basic HTML correctly:</p>
<ul>
<li>Use <code>header</code>, <code>nav</code>, <code>main</code> and <code>button</code> instead of <code>div</code> for everything. A click handler on a <code>div</code> means writing keyboard and screen reader support yourself.</li>
<li>Give icon-only buttons an <code>aria-label</code>.</li>
<li>Navigate the page with <code>Tab</code> only, no mouse. Is focus visible? Does the order make sense?</li>
</ul>
<p>That's also why <code>dialog</code> and <code>popover</code> are worth using: focus handling and keyboard behavior come built in.</p>

<h2 class="relative group">So when do you need a framework?
    <div id="so-when-do-you-need-a-framework" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#so-when-do-you-need-a-framework" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>My split: content-heavy pages, dashboards and small tools get plain HTML, modern CSS and islands. A multi-page app with heavy state and many people working in the same codebase gets a framework, because its structure and ecosystem pay for themselves there.</p>

<h2 class="relative group">Where to go next
    <div id="where-to-go-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-to-go-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Take a component in an existing project that uses media queries, switch it to a container query and drop it into columns of different widths; you'll see the difference right away. Track browser support on <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_containment/Container_queries"  target="_blank" rel="noreferrer">MDN</a> and <a href="https://web.dev/baseline"  target="_blank" rel="noreferrer">Baseline</a>. MDN shows the status as a badge at the top of each reference page:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/posts/modern-web-arayuzu-css-javascript/img/mdn-container-baseline_hu_3b5f4572c885c569.avif 480w, https://blog.furkankoykiran.com.tr/posts/modern-web-arayuzu-css-javascript/img/mdn-container-baseline_hu_bbfdf88a4cb2e3f1.avif 800w, https://blog.furkankoykiran.com.tr/posts/modern-web-arayuzu-css-javascript/img/mdn-container-baseline_hu_8eb59850b1999fd3.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/posts/modern-web-arayuzu-css-javascript/img/mdn-container-baseline_hu_bf88c6cf805d00a6.webp 480w, https://blog.furkankoykiran.com.tr/posts/modern-web-arayuzu-css-javascript/img/mdn-container-baseline_hu_251e1a7e08bfc591.webp 800w, https://blog.furkankoykiran.com.tr/posts/modern-web-arayuzu-css-javascript/img/mdn-container-baseline_hu_82b7cc630facf3ba.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/posts/modern-web-arayuzu-css-javascript/img/mdn-container-baseline_hu_251e1a7e08bfc591.webp" alt="MDN&#39;s Baseline badge on the @container page: widely available, supported in Chrome, Edge, Firefox and Safari" width="1426" height="188"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Baseline status for @container. Source: MDN Web Docs contributors, quotation</figcaption></figure><p>If you want to feed live data into pages like these, my <a href="https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/" >WebSocket post</a> covers the server side, and my <a href="https://blog.furkankoykiran.com.tr/en/posts/awesome-trending-repos-web/" >awesome-trending-repos post</a> shows a site built with a framework.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/modern-web-ui-css-javascript/feature_hu_e116be783c8f56e6.jpg" medium="image" type="image/jpeg"/></item><item><title>Python Email Automation: smtplib, aiosmtplib and SPF</title><link>https://blog.furkankoykiran.com.tr/en/posts/python-email-automation/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/python-email-automation/</guid><pubDate>Sat, 08 Nov 2025 12:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Python</category><category>python</category><category>automation</category><category>asyncio</category><category>security</category><description>Python email automation done right: EmailMessage, smtplib with app passwords, aiosmtplib for batches, Jinja2 templates, and SPF, DKIM and DMARC for delivery.</description><content:encoded><![CDATA[<p>Python email automation looks like a five-line job: connect with <code>smtplib</code>, send the message. Getting reports, alerts and notifications to actually land in the inbox is a different problem. Where you keep credentials, how you build the message and what your domain's DNS records say matter at least as much as the code.</p>
<p>This is the setup I use in my own automations: building messages with <code>EmailMessage</code>, <code>smtplib</code> with an app password, <code>aiosmtplib</code> for sending many messages, Jinja2 templates, and SPF, DKIM and DMARC for deliverability.</p>

<h2 class="relative group">Build the message with EmailMessage
    <div id="build-the-message-with-emailmessage" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#build-the-message-with-emailmessage" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Older examples glue parts together by hand with <code>MIMEMultipart</code> and <code>MIMEText</code>. Since Python 3.6, <code>email.message.EmailMessage</code> does the same with far less code:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">email.message</span> <span class="kn">import</span> <span class="n">EmailMessage</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">email.utils</span> <span class="kn">import</span> <span class="n">formataddr</span><span class="p">,</span> <span class="n">formatdate</span><span class="p">,</span> <span class="n">make_msgid</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">pathlib</span> <span class="kn">import</span> <span class="n">Path</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">build_message</span><span class="p">(</span><span class="n">to</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">subject</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">html</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">attachment</span><span class="p">:</span> <span class="n">Path</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">EmailMessage</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">msg</span> <span class="o">=</span> <span class="n">EmailMessage</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">msg</span><span class="p">[</span><span class="s2">&#34;From&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">formataddr</span><span class="p">((</span><span class="s2">&#34;Report Bot&#34;</span><span class="p">,</span> <span class="s2">&#34;reports@example.com&#34;</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="n">msg</span><span class="p">[</span><span class="s2">&#34;To&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">to</span>
</span></span><span class="line"><span class="cl">    <span class="n">msg</span><span class="p">[</span><span class="s2">&#34;Subject&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">subject</span>
</span></span><span class="line"><span class="cl">    <span class="n">msg</span><span class="p">[</span><span class="s2">&#34;Date&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">formatdate</span><span class="p">(</span><span class="n">localtime</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">msg</span><span class="p">[</span><span class="s2">&#34;Message-ID&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">make_msgid</span><span class="p">(</span><span class="n">domain</span><span class="o">=</span><span class="s2">&#34;example.com&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">msg</span><span class="o">.</span><span class="n">set_content</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>                    <span class="c1"># plain-text version</span>
</span></span><span class="line"><span class="cl">    <span class="n">msg</span><span class="o">.</span><span class="n">add_alternative</span><span class="p">(</span><span class="n">html</span><span class="p">,</span> <span class="n">subtype</span><span class="o">=</span><span class="s2">&#34;html&#34;</span><span class="p">)</span>  <span class="c1"># HTML version</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">attachment</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">msg</span><span class="o">.</span><span class="n">add_attachment</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="n">attachment</span><span class="o">.</span><span class="n">read_bytes</span><span class="p">(),</span>
</span></span><span class="line"><span class="cl">            <span class="n">maintype</span><span class="o">=</span><span class="s2">&#34;application&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">subtype</span><span class="o">=</span><span class="s2">&#34;pdf&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">filename</span><span class="o">=</span><span class="n">attachment</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">msg</span></span></span></code></pre></div></div>
<p>Include both a plain-text and an HTML version. Clients that don't render HTML and some corporate filters look at the text part, and HTML-only messages score worse with spam filters. <code>EmailMessage</code> encodes non-ASCII subjects and display names for you.</p>

<h2 class="relative group">App passwords, not your real password
    <div id="app-passwords-not-your-real-password" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#app-passwords-not-your-real-password" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Don't put your actual Gmail or Outlook password in a script. Google has removed the &quot;less secure apps&quot; option that allowed password logins over SMTP. To send from a personal Gmail account with a script:</p>
<ol>
<li>Turn on 2-Step Verification for the account.</li>
<li>Create an app password in your Google account settings. The 16-character password works only for this script and can be revoked any time.</li>
<li>Put it in an environment variable or a secrets manager, not in the code.</li>
</ol>
<p>Microsoft 365 and Outlook.com are retiring Basic Auth for SMTP, so there you need OAuth2 (XOAUTH2) or the Microsoft Graph API. For a long-running service, OAuth2 is the sturdier choice on Gmail as well; app passwords are a practical middle ground for personal automations.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">os</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">smtplib</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">ssl</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">SMTP_HOST</span> <span class="o">=</span> <span class="s2">&#34;smtp.gmail.com&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">SMTP_USER</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">&#34;SMTP_USER&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="n">SMTP_APP_PASSWORD</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">&#34;SMTP_APP_PASSWORD&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">send</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">context</span> <span class="o">=</span> <span class="n">ssl</span><span class="o">.</span><span class="n">create_default_context</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">smtplib</span><span class="o">.</span><span class="n">SMTP</span><span class="p">(</span><span class="n">SMTP_HOST</span><span class="p">,</span> <span class="mi">587</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">30</span><span class="p">)</span> <span class="k">as</span> <span class="n">smtp</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">smtp</span><span class="o">.</span><span class="n">starttls</span><span class="p">(</span><span class="n">context</span><span class="o">=</span><span class="n">context</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">smtp</span><span class="o">.</span><span class="n">login</span><span class="p">(</span><span class="n">SMTP_USER</span><span class="p">,</span> <span class="n">SMTP_APP_PASSWORD</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">smtp</span><span class="o">.</span><span class="n">send_message</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span></span></span></code></pre></div></div>
<p><code>ssl.create_default_context()</code> verifies the certificate and hostname. If you call <code>starttls()</code> without a context, <code>smtplib</code> falls back to a default that doesn't verify certificates: the connection is encrypted, but nothing checks that the server really is Gmail. Always pass the context explicitly.</p>

<h2 class="relative group">Port 587 or 465?
    <div id="port-587-or-465" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#port-587-or-465" aria-label="Anchor">#</a>
    </span>
    
</h2>
<ul>
<li>587 (STARTTLS): the connection starts in plain text and upgrades with the <code>STARTTLS</code> command. Use <code>smtplib.SMTP</code> plus <code>starttls()</code>.</li>
<li>465 (implicit TLS): TLS from the first byte. Use <code>smtplib.SMTP_SSL</code>. RFC 8314 recommends this for message submission, so it is not a legacy port.</li>
<li>25: server-to-server relay. Cloud providers like AWS and Google Cloud block outbound port 25 by default; don't use it from your app.</li>
</ul>
<p>Both 587 and 465 are secure. Use whichever your provider documents.</p>

<h2 class="relative group">Sending many messages: aiosmtplib
    <div id="sending-many-messages-aiosmtplib" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#sending-many-messages-aiosmtplib" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Opening a new connection per message is slow. For hundreds of notifications, reuse one connection. If your code already runs on asyncio, <code>aiosmtplib</code> sends without blocking the loop:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">asyncio</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">os</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">aiosmtplib</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">SMTP_USER</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">&#34;SMTP_USER&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="n">SMTP_APP_PASSWORD</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">&#34;SMTP_APP_PASSWORD&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">send_batch</span><span class="p">(</span><span class="n">messages</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">with</span> <span class="n">aiosmtplib</span><span class="o">.</span><span class="n">SMTP</span><span class="p">(</span><span class="n">hostname</span><span class="o">=</span><span class="s2">&#34;smtp.gmail.com&#34;</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="mi">587</span><span class="p">,</span> <span class="n">start_tls</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> <span class="k">as</span> <span class="n">smtp</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">await</span> <span class="n">smtp</span><span class="o">.</span><span class="n">login</span><span class="p">(</span><span class="n">SMTP_USER</span><span class="p">,</span> <span class="n">SMTP_APP_PASSWORD</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">for</span> <span class="n">msg</span> <span class="ow">in</span> <span class="n">messages</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">                <span class="k">await</span> <span class="n">smtp</span><span class="o">.</span><span class="n">send_message</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">except</span> <span class="n">aiosmtplib</span><span class="o">.</span><span class="n">SMTPRecipientsRefused</span> <span class="k">as</span> <span class="n">exc</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">                <span class="nb">print</span><span class="p">(</span><span class="s2">&#34;Refused:&#34;</span><span class="p">,</span> <span class="n">exc</span><span class="o">.</span><span class="n">recipients</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.5</span><span class="p">)</span>  <span class="c1"># be gentle with the provider</span></span></span></code></pre></div></div>
<p>One SMTP connection sends one message at a time, so the loop is sequential. If you need more throughput you can open a few connections and split the work, but check the provider's limits first. Personal Gmail accounts are capped at roughly 500 recipients a day, Google Workspace accounts at 2,000. Beyond that, use a transactional email service such as Amazon SES, Postmark or Mailgun instead of a mailbox.</p>
<p>For a one-off message there is a shortcut:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">await</span> <span class="n">aiosmtplib</span><span class="o">.</span><span class="n">send</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">msg</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">hostname</span><span class="o">=</span><span class="s2">&#34;smtp.gmail.com&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">port</span><span class="o">=</span><span class="mi">587</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">start_tls</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">username</span><span class="o">=</span><span class="n">SMTP_USER</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">password</span><span class="o">=</span><span class="n">SMTP_APP_PASSWORD</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span></span></span></code></pre></div></div>
<p>TaskGroup and timeouts on the asyncio side are covered in the <a href="https://blog.furkankoykiran.com.tr/en/posts/python-asyncio-concurrent-processing/" >asyncio post</a>.</p>

<h2 class="relative group">HTML templates with Jinja2
    <div id="html-templates-with-jinja2" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#html-templates-with-jinja2" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>HTML embedded in Python strings gets out of hand quickly. I keep templates in separate files and render them with Jinja2:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">jinja2</span> <span class="kn">import</span> <span class="n">Environment</span><span class="p">,</span> <span class="n">FileSystemLoader</span><span class="p">,</span> <span class="n">select_autoescape</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">env</span> <span class="o">=</span> <span class="n">Environment</span><span class="p">(</span><span class="n">loader</span><span class="o">=</span><span class="n">FileSystemLoader</span><span class="p">(</span><span class="s2">&#34;templates&#34;</span><span class="p">),</span> <span class="n">autoescape</span><span class="o">=</span><span class="n">select_autoescape</span><span class="p">())</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">render</span><span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="o">**</span><span class="n">context</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]:</span>
</span></span><span class="line"><span class="cl">    <span class="n">html</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">get_template</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">.html&#34;</span><span class="p">)</span><span class="o">.</span><span class="n">render</span><span class="p">(</span><span class="o">**</span><span class="n">context</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">text</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">get_template</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2">.txt&#34;</span><span class="p">)</span><span class="o">.</span><span class="n">render</span><span class="p">(</span><span class="o">**</span><span class="n">context</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">text</span><span class="p">,</span> <span class="n">html</span></span></span></code></pre></div></div>
<p>Keep <code>autoescape</code> on. If an outside value like a username lands in the HTML unescaped, someone can inject markup or links into your message.</p>
<p>HTML support in email clients is far behind browsers. Gmail supports <code>&lt;style&gt;</code> blocks only partly, and desktop Outlook still renders with Word's engine. Don't rely on flexbox or grid. Table-based layout with inline CSS is the safe route. If you'd rather not write tables by hand, design in MJML, compile to HTML, and move CSS inline with a library like <code>css-inline</code>.</p>

<h2 class="relative group">Why mail doesn't reach the inbox
    <div id="why-mail-doesnt-reach-the-inbox" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-mail-doesnt-reach-the-inbox" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>If the code runs cleanly and the message still lands in spam, the problem is most likely your domain's authentication. The receiving server checks three DNS records:</p>
<ul>
<li>SPF lists which servers may send mail for your domain, for example <code>v=spf1 include:_spf.google.com ~all</code>.</li>
<li>DKIM: the sending server signs the message with a private key, and the receiver verifies it with the public key in DNS. It proves the message wasn't modified on the way.</li>
<li>DMARC requires SPF or DKIM to pass <em>and</em> align with the domain in the <code>From</code> address, tells receivers what to do with failures (<code>p=none</code>, <code>quarantine</code>, <code>reject</code>), and includes an address for reports.</li>
</ul>
<p>The chart below is DMARC's decision in one step:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/python-email-automation/img/email-deliverability_hu_b1c43c9d50b8f671.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/python-email-automation/img/email-deliverability_hu_dfdcd39df0114b99.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/python-email-automation/img/email-deliverability_hu_639c41d1d71ad2f3.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/python-email-automation/img/email-deliverability_hu_d9f9fe410d1ba7c0.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/python-email-automation/img/email-deliverability_hu_3f5dbffbbdd071f.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/python-email-automation/img/email-deliverability_hu_4c91834861a972f6.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/python-email-automation/img/email-deliverability_hu_3f5dbffbbdd071f.webp" alt="DMARC decision: if SPF or DKIM passes and aligns with the From domain, DMARC passes; otherwise the none, quarantine or reject policy applies" width="1600" height="1261"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>The one question DMARC asks</figcaption></figure><p>Since February 2024, Gmail and Yahoo require senders of more than 5,000 messages a day to set up all three of SPF, DKIM and DMARC, support one-click unsubscribe, and keep spam complaints under 0.3 percent. Smaller senders still need at least SPF or DKIM. If you send from your own domain, start DMARC at <code>p=none</code>, read the reports, and tighten it once they look clean.</p>
<p>For marketing and bulk notification mail in that category, one-click unsubscribe headers are mandatory:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">msg</span><span class="p">[</span><span class="s2">&#34;List-Unsubscribe&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="s2">&#34;&lt;https://example.com/unsubscribe?token=abc123&gt;, &lt;mailto:unsubscribe@example.com&gt;&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">msg</span><span class="p">[</span><span class="s2">&#34;List-Unsubscribe-Post&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="s2">&#34;List-Unsubscribe=One-Click&#34;</span></span></span></code></pre></div></div>
<p>The <code>List-Unsubscribe-Post</code> header (RFC 8058) is what makes Gmail's &quot;Unsubscribe&quot; button at the top of the message work. The token in the URL should be unguessable and tied to one recipient.</p>
<p>The quickest way to see whether a message passed authentication: open it in Gmail and choose &quot;Show original&quot;. SPF, DKIM and DMARC results are listed at the top.</p>

<h2 class="relative group">Track bounces
    <div id="track-bounces" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#track-bounces" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Repeatedly sending to addresses that don't exist hurts your sender reputation. Remove addresses that hard bounce (<code>5xx</code>). Retry soft bounces (<code>4xx</code>, such as a full mailbox) a few times. Transactional services report bounces and spam complaints through webhooks, so the cleanest setup is an endpoint that receives those events, verifies their signature, and updates the recipient's status in your database.</p>

<h2 class="relative group">Test without touching real inboxes
    <div id="test-without-touching-real-inboxes" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#test-without-touching-real-inboxes" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Don't send to real addresses during development. Python 3.12 removed the <code>smtpd</code> module from the standard library; <code>aiosmtpd</code> replaces it. This runs a local server that prints every message to the terminal:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">pip install aiosmtpd
</span></span><span class="line"><span class="cl">python -m aiosmtpd -n -l localhost:1025</span></span></code></pre></div></div>
<p>Point your code at <code>localhost:1025</code> (this server has no TLS or login, so skip those steps in development). When I sent it a message built with <code>build_message</code>, the text and HTML versions arrived as two parts under <code>multipart/alternative</code>.</p>
<p>If you want to see how the HTML renders, a catcher with a web UI like Mailpit starts with a single Docker command. In unit tests, mocking the SMTP class and asserting that <code>send_message</code> was called with the right message is enough; mocking and fixtures are in the <a href="https://blog.furkankoykiran.com.tr/en/posts/pytest-advanced-testing-techniques/" >pytest post</a>.</p>

<h2 class="relative group">Checklist
    <div id="checklist" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#checklist" aria-label="Anchor">#</a>
    </span>
    
</h2>
<ul>
<li>Is the password in an environment variable, and is it an app password or OAuth2 rather than the real account password?</li>
<li>Is the connection TLS with certificate verification?</li>
<li>Does every message have both a text and an HTML part?</li>
<li>Are SPF, DKIM and DMARC records set on the domain?</li>
<li>Do bulk messages carry <code>List-Unsubscribe</code> and <code>List-Unsubscribe-Post</code>?</li>
<li>Are hard-bounced addresses removed from the list?</li>
<li>Is volume under the provider's daily limit?</li>
</ul>

<h2 class="relative group">Next step
    <div id="next-step" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#next-step" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Send yourself a test message and check SPF, DKIM and DMARC with &quot;Show original&quot; in Gmail. If any of them isn't <code>PASS</code>, fix DNS before touching the code. If you're attaching PDF reports, I covered generating them in the <a href="https://blog.furkankoykiran.com.tr/en/posts/python-pdf-processing/" >Python PDF processing post</a>.</p>
<p>Docs: <a href="https://docs.python.org/3/library/email.html"  target="_blank" rel="noreferrer">email package</a>, <a href="https://docs.python.org/3/library/smtplib.html"  target="_blank" rel="noreferrer">smtplib</a>, <a href="https://aiosmtplib.readthedocs.io/"  target="_blank" rel="noreferrer">aiosmtplib</a>, <a href="https://support.google.com/a/answer/81126"  target="_blank" rel="noreferrer">Google's sender guidelines</a>.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/python-email-automation/feature_hu_9561deb03dc46df4.jpg" medium="image" type="image/jpeg"/></item><item><title>Database Connection Pooling: SQLAlchemy and PgBouncer</title><link>https://blog.furkankoykiran.com.tr/en/posts/database-connection-pool-optimization/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/database-connection-pool-optimization/</guid><pubDate>Sat, 01 Nov 2025 10:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Backend</category><category>databases</category><category>python</category><category>performance</category><category>asyncio</category><description>Database connection pooling for PostgreSQL: SQLAlchemy 2 async pool settings, sizing limits per process, leaks, timeouts and PgBouncer transaction mode.</description><content:encoded><![CDATA[<p>Database connection pooling is one of the first things to check when a Python service falls over under load. If PostgreSQL logs <code>FATAL: sorry, too many clients already</code>, or SQLAlchemy raises <code>QueuePool limit of size 5 overflow 10 reached</code>, the problem is most likely how you manage connections, not your queries.</p>
<p>This post covers pool settings with SQLAlchemy 2's async engine, how to work out how many connections you can afford, the common mistakes, and PgBouncer for when the number of app processes grows.</p>

<h2 class="relative group">Why a connection is expensive
    <div id="why-a-connection-is-expensive" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-a-connection-is-expensive" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Opening a PostgreSQL connection is nothing like a function call:</p>
<ol>
<li>A TCP connection is set up, and in most environments a TLS handshake on top.</li>
<li>The client authenticates. SCRAM takes several round trips.</li>
<li>PostgreSQL forks a new backend process for the connection, with its own memory.</li>
</ol>
<p>On the same machine that takes a few milliseconds; across a network with TLS it takes longer. A simple indexed <code>SELECT</code> is often faster than that. If you open and close a connection per HTTP request, most of the time goes into handshakes. You also can't raise <code>max_connections</code> as far as you like, because every connection is a process with its own memory.</p>
<p>That's what the pool is for: open connections once, keep them, and lend them out between requests.</p>

<h2 class="relative group">An async pool with SQLAlchemy 2
    <div id="an-async-pool-with-sqlalchemy-2" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#an-async-pool-with-sqlalchemy-2" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p><code>create_async_engine</code> uses a queue-based pool by default, with <code>pool_size=5</code>, <code>max_overflow=10</code> and <code>pool_timeout=30</code>. These are my starting values:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">sqlalchemy</span> <span class="kn">import</span> <span class="n">text</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">sqlalchemy.ext.asyncio</span> <span class="kn">import</span> <span class="n">async_sessionmaker</span><span class="p">,</span> <span class="n">create_async_engine</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">engine</span> <span class="o">=</span> <span class="n">create_async_engine</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;postgresql+asyncpg://app:secret@db:5432/shop&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">pool_size</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span>        <span class="c1"># connections kept open</span>
</span></span><span class="line"><span class="cl">    <span class="n">max_overflow</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span>      <span class="c1"># extra connections allowed during bursts</span>
</span></span><span class="line"><span class="cl">    <span class="n">pool_timeout</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span>      <span class="c1"># wait at most 5 s for a free connection, then raise</span>
</span></span><span class="line"><span class="cl">    <span class="n">pool_pre_ping</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>  <span class="c1"># check the connection is alive before handing it out</span>
</span></span><span class="line"><span class="cl">    <span class="n">pool_recycle</span><span class="o">=</span><span class="mi">1800</span><span class="p">,</span>   <span class="c1"># replace connections older than 30 minutes</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">SessionLocal</span> <span class="o">=</span> <span class="n">async_sessionmaker</span><span class="p">(</span><span class="n">engine</span><span class="p">,</span> <span class="n">expire_on_commit</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">get_user</span><span class="p">(</span><span class="n">user_id</span><span class="p">:</span> <span class="nb">int</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">with</span> <span class="n">SessionLocal</span><span class="p">()</span> <span class="k">as</span> <span class="n">session</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">result</span> <span class="o">=</span> <span class="k">await</span> <span class="n">session</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="n">text</span><span class="p">(</span><span class="s2">&#34;SELECT id, email FROM users WHERE id = :id&#34;</span><span class="p">),</span> <span class="p">{</span><span class="s2">&#34;id&#34;</span><span class="p">:</span> <span class="n">user_id</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">result</span><span class="o">.</span><span class="n">one_or_none</span><span class="p">()</span></span></span></code></pre></div></div>
<p>What each setting buys you:</p>
<ul>
<li><code>pool_pre_ping</code> runs a cheap liveness check before lending a connection. If the database restarted, or a firewall silently dropped an idle connection, you get a fresh one instead of an error. The cost is small; I always turn it on.</li>
<li><code>pool_recycle</code> closes and replaces connections past a certain age. If a load balancer or proxy in between kills idle connections, keep it below that timeout.</li>
<li><code>pool_timeout</code> left at the default 30 seconds means requests hang for 30 seconds when the pool is exhausted. A short value fails fast so you can shed load.</li>
</ul>
<p>Create the <code>engine</code> once per application. A new engine per request is a new pool per request, which defeats the point.</p>

<h2 class="relative group">How many connections can you open?
    <div id="how-many-connections-can-you-open" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#how-many-connections-can-you-open" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>This is the calculation people skip:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">total connections = number of processes × (pool_size + max_overflow)</span></span></code></pre></div></div>
<p>&quot;Processes&quot; doesn't mean pods. If each pod runs several uvicorn or gunicorn workers, each worker has its own pool. With 4 pods, 4 workers per pod and <code>pool_size=10, max_overflow=5</code>, the worst case is 4 × 4 × 15 = 240 connections. PostgreSQL's default <code>max_connections</code> is 100, and a few of those are reserved for superusers.</p>
<p>Keep the total under <code>max_connections</code> and leave room for migrations, cron jobs and your own <code>psql</code> session. With autoscaling, do the math with the maximum replica count. When the total gets tight, the next step is PgBouncer.</p>

<h2 class="relative group">Leaks: connections that never come back
    <div id="leaks-connections-that-never-come-back" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#leaks-connections-that-never-come-back" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>When I see <code>QueuePool limit ... reached</code>, my first suspect isn't load but connections that never return to the pool. Open a session, forget to close it on an error path, and the connection stays checked out.</p>
<p>The rule is simple: always open sessions with a context manager. SQLAlchemy also has a shortcut that manages the transaction for you:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">transfer</span><span class="p">(</span><span class="n">from_id</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">to_id</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">amount</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">with</span> <span class="n">SessionLocal</span><span class="o">.</span><span class="n">begin</span><span class="p">()</span> <span class="k">as</span> <span class="n">session</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">await</span> <span class="n">session</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="n">text</span><span class="p">(</span><span class="s2">&#34;UPDATE accounts SET balance = balance - :a WHERE id = :id&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">            <span class="p">{</span><span class="s2">&#34;a&#34;</span><span class="p">:</span> <span class="n">amount</span><span class="p">,</span> <span class="s2">&#34;id&#34;</span><span class="p">:</span> <span class="n">from_id</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">await</span> <span class="n">session</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="n">text</span><span class="p">(</span><span class="s2">&#34;UPDATE accounts SET balance = balance + :a WHERE id = :id&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">            <span class="p">{</span><span class="s2">&#34;a&#34;</span><span class="p">:</span> <span class="n">amount</span><span class="p">,</span> <span class="s2">&#34;id&#34;</span><span class="p">:</span> <span class="n">to_id</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># commit if the block succeeds, rollback on an exception; the connection always goes back</span></span></span></code></pre></div></div>
<p>In FastAPI I write it as a dependency so every endpoint shares the same lifecycle:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">collections.abc</span> <span class="kn">import</span> <span class="n">AsyncIterator</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">sqlalchemy.ext.asyncio</span> <span class="kn">import</span> <span class="n">AsyncSession</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">get_session</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">AsyncIterator</span><span class="p">[</span><span class="n">AsyncSession</span><span class="p">]:</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">with</span> <span class="n">SessionLocal</span><span class="p">()</span> <span class="k">as</span> <span class="n">session</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">yield</span> <span class="n">session</span></span></span></code></pre></div></div>
<p>Don't count on the garbage collector to clean up. You don't know when it runs, and until then the connection is stuck.</p>

<h2 class="relative group">Stay inside the database during a transaction
    <div id="stay-inside-the-database-during-a-transaction" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#stay-inside-the-database-during-a-transaction" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Another way to drain the pool: open a transaction, make an HTTP call to an external API inside it, then commit. If that API takes 30 seconds, the connection sits <code>idle in transaction</code> for 30 seconds and holds its locks the whole time.</p>
<p>Keep transactions short. Make the external call first, then do the database work in one go. Add a safety net on the PostgreSQL side as well:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">ALTER</span><span class="w"> </span><span class="k">ROLE</span><span class="w"> </span><span class="n">app</span><span class="w"> </span><span class="k">SET</span><span class="w"> </span><span class="n">idle_in_transaction_session_timeout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;30s&#39;</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">ALTER</span><span class="w"> </span><span class="k">ROLE</span><span class="w"> </span><span class="n">app</span><span class="w"> </span><span class="k">SET</span><span class="w"> </span><span class="n">statement_timeout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;5s&#39;</span><span class="p">;</span></span></span></code></pre></div></div>
<p>Setting these on the role is more reliable than sending them on every connection, and it keeps working behind PgBouncer. Give long-running reporting queries their own role.</p>

<h2 class="relative group">When you scale out: PgBouncer
    <div id="when-you-scale-out-pgbouncer" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#when-you-scale-out-pgbouncer" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>As process counts grow, even a small pool per process pushes the total past <code>max_connections</code>. At that point I put a connection pooler like PgBouncer between the app and the database.</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/database-connection-pool-optimization/img/pgbouncer-scale-out_hu_f9f0062e9e705075.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/database-connection-pool-optimization/img/pgbouncer-scale-out_hu_f91be47bf2d1908b.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/database-connection-pool-optimization/img/pgbouncer-scale-out_hu_3163603421460039.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/database-connection-pool-optimization/img/pgbouncer-scale-out_hu_633115ef830d0ee4.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/database-connection-pool-optimization/img/pgbouncer-scale-out_hu_209fb21d7a0395f.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/database-connection-pool-optimization/img/pgbouncer-scale-out_hu_232bd20eb11ee844.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/database-connection-pool-optimization/img/pgbouncer-scale-out_hu_209fb21d7a0395f.webp" alt="Direct pools push the total past max_connections; through PgBouncer in transaction mode, hundreds of client connections become 20 to 50 real ones" width="1600" height="1198"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>PgBouncer sits between many app workers and a few dozen database connections</figcaption></figure><p>PgBouncer has two main modes:</p>
<ul>
<li>Session mode: a server connection stays with the client for as long as the client is connected. It barely reduces the total.</li>
<li>Transaction mode: a server connection is assigned only for the length of a transaction, then goes to another client. This is where the win is: hundreds of client connections served by a few dozen real ones.</li>
</ul>
<p>The price of transaction mode is losing per-connection state. Session settings made with <code>SET</code>, advisory locks, <code>LISTEN/NOTIFY</code> and temporary tables don't carry over between transactions. For a long time asyncpg's prepared statements were a problem too. Since 1.21, PgBouncer supports protocol-level prepared statements in transaction mode through <code>max_prepared_statements</code>; turn it on. If you're stuck on an older PgBouncer, disable both SQLAlchemy's and asyncpg's statement caches:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">engine</span> <span class="o">=</span> <span class="n">create_async_engine</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># older PgBouncer only: SQLAlchemy and asyncpg statement caches disabled</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;postgresql+asyncpg://app:secret@pgbouncer:6432/shop?prepared_statement_cache_size=0&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">connect_args</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;statement_cache_size&#34;</span><span class="p">:</span> <span class="mi">0</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="n">pool_size</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">max_overflow</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">pool_pre_ping</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span></span></span></code></pre></div></div>
<p>Behind PgBouncer I keep the application pool small. Some teams use <code>NullPool</code> in SQLAlchemy and leave all pooling to PgBouncer. That works, but every request then opens a new TCP connection to PgBouncer, so a small app-side pool usually performs better.</p>

<h2 class="relative group">Caching and pooling are different things
    <div id="caching-and-pooling-are-different-things" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#caching-and-pooling-are-different-things" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>People mix these up. A pool keeps the road to the database open; a cache lets you skip the trip. Keeping hot, rarely changing data in Redis takes pressure off the pool, but on a cache miss you still need a solid pool. I cover caching patterns in my <a href="https://blog.furkankoykiran.com.tr/en/posts/redis-caching-session-management/" >Redis post</a>.</p>

<h2 class="relative group">Monitoring
    <div id="monitoring" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#monitoring" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Tuning a pool you can't see is guessing. SQLAlchemy reports the pool's state:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">pool</span> <span class="o">=</span> <span class="n">engine</span><span class="o">.</span><span class="n">sync_engine</span><span class="o">.</span><span class="n">pool</span>
</span></span><span class="line"><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="n">pool</span><span class="o">.</span><span class="n">status</span><span class="p">())</span>  <span class="c1"># size, checked out, overflow and waiting</span>
</span></span><span class="line"><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="n">pool</span><span class="o">.</span><span class="n">checkedout</span><span class="p">(),</span> <span class="n">pool</span><span class="o">.</span><span class="n">overflow</span><span class="p">())</span></span></span></code></pre></div></div>
<p>Export these numbers to your metrics system, such as Prometheus, on a timer. On the database side, <code>pg_stat_activity</code> shows who is connected and what state they're in:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">SELECT</span><span class="w"> </span><span class="n">usename</span><span class="p">,</span><span class="w"> </span><span class="n">application_name</span><span class="p">,</span><span class="w"> </span><span class="k">state</span><span class="p">,</span><span class="w"> </span><span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">FROM</span><span class="w"> </span><span class="n">pg_stat_activity</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">GROUP</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="mi">3</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="k">DESC</span><span class="p">;</span></span></span></code></pre></div></div>
<p>Lots of <code>idle in transaction</code> rows point back to the external-call problem above.</p>

<h2 class="relative group">Before you go live
    <div id="before-you-go-live" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#before-you-go-live" aria-label="Anchor">#</a>
    </span>
    
</h2>
<ul>
<li>Is <code>processes × (pool_size + max_overflow)</code> below <code>max_connections</code>, even at the maximum replica count?</li>
<li>Is <code>pool_pre_ping</code> on, and is <code>pool_recycle</code> shorter than any proxy's idle timeout?</li>
<li>Is <code>pool_timeout</code> short enough that requests fail fast when the pool is full?</li>
<li>Does the role have <code>statement_timeout</code> and <code>idle_in_transaction_session_timeout</code>?</li>
<li>Is every session opened with a context manager?</li>
</ul>

<h2 class="relative group">What's next
    <div id="whats-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#whats-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Run the connection math for your own service first; it explains a lot of &quot;the database is slow&quot; complaints. For the general idea behind <code>with</code> and <code>async with</code>, see my <a href="https://blog.furkankoykiran.com.tr/en/posts/python-context-managers/" >context managers post</a>, and for SQLAlchemy basics my <a href="https://blog.furkankoykiran.com.tr/en/posts/mysql-sqlalchemy-database-management/" >SQLAlchemy post</a>. The <a href="https://docs.sqlalchemy.org/en/20/core/pooling.html"  target="_blank" rel="noreferrer">SQLAlchemy pooling docs</a> and the <a href="https://www.pgbouncer.org/config.html"  target="_blank" rel="noreferrer">PgBouncer config reference</a> have the details.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/database-connection-pool-optimization/feature_hu_7009cb4033bf032c.jpg" medium="image" type="image/jpeg"/></item><item><title>gRPC High-Performance APIs with Python asyncio</title><link>https://blog.furkankoykiran.com.tr/en/posts/grpc-high-performance-apis/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/grpc-high-performance-apis/</guid><pubDate>Wed, 22 Oct 2025 14:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Backend</category><category>api</category><category>python</category><category>asyncio</category><category>performance</category><description>Service-to-service APIs with gRPC and Protobuf: grpcio asyncio server and client, streaming, deadlines, interceptors, schema evolution, HTTP/2 load balancing.</description><content:encoded><![CDATA[<p>The pitch for gRPC high-performance APIs is short: binary Protobuf messages instead of JSON, HTTP/2 instead of HTTP/1.1, and typed code generated from a schema instead of a hand-written client. Between services, those three make a real difference. For an API that browsers call, REST is still usually the more practical choice.</p>
<p>In this post I write a service with the asyncio API of <code>grpcio</code> (<code>grpc.aio</code>), then cover what came up once it ran in production: deadlines, status codes, interceptors, schema evolution and load balancing. I ran the server and client examples with Python 3.12 and grpcio 1.84.</p>

<h2 class="relative group">When gRPC makes sense
    <div id="when-grpc-makes-sense" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#when-grpc-makes-sense" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>REST/JSON has known costs for internal traffic: field names travel with every message, parsing text burns CPU, and only the documentation tells you whether a field is a string or an int. In gRPC the contract is a <code>.proto</code> file, and client and server code for every language is generated from it.</p>
<p>How I choose:</p>
<div class="fk-table" role="region" tabindex="0" aria-label="Scrollable table">
<table>
  <thead>
    <tr>
      <th style="text-align:left"></th>
      <th style="text-align:left">REST (JSON)</th>
      <th style="text-align:left">gRPC (Protobuf)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align:left">Payload</td>
      <td style="text-align:left">Text, human-readable</td>
      <td style="text-align:left">Binary, compact</td>
    </tr>
    <tr>
      <td style="text-align:left">Transport</td>
      <td style="text-align:left">Usually HTTP/1.1</td>
      <td style="text-align:left">HTTP/2</td>
    </tr>
    <tr>
      <td style="text-align:left">Contract</td>
      <td style="text-align:left">OpenAPI (optional)</td>
      <td style="text-align:left"><code>.proto</code> (required)</td>
    </tr>
    <tr>
      <td style="text-align:left">Streaming</td>
      <td style="text-align:left">Separately, via SSE or WebSocket</td>
      <td style="text-align:left">Four call types built in</td>
    </tr>
    <tr>
      <td style="text-align:left">Browser</td>
      <td style="text-align:left">Direct</td>
      <td style="text-align:left">Needs gRPC-Web or Connect</td>
    </tr>
    <tr>
      <td style="text-align:left">Debugging</td>
      <td style="text-align:left"><code>curl</code></td>
      <td style="text-align:left"><code>grpcurl</code>, reflection</td>
    </tr>
  </tbody>
</table>
</div><p>I use gRPC for east-west traffic between services, especially when they're written in different languages or need streaming. For a public API, a backend the browser calls directly or a simple CRUD app, <a href="https://blog.furkankoykiran.com.tr/en/posts/python-fastapi-rest-api/" >REST with FastAPI</a> has less friction.</p>

<h2 class="relative group">The contract: a .proto file
    <div id="the-contract-a-proto-file" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-contract-a-proto-file" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Everything in gRPC starts from one <code>.proto</code> file. <code>protoc</code> generates the message classes and stubs from it, and both server and client build on that generated code.</p>
<p>A small price service with one unary and one server-streaming method:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-protobuf" data-lang="protobuf"><span class="line"><span class="cl"><span class="c1">// proto/stocks.proto
</span></span></span><span class="line"><span class="cl"><span class="k">syntax</span> <span class="o">=</span> <span class="s">&#34;proto3&#34;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">package</span> <span class="nn">stocks.v1</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kd">service</span> <span class="n">StockService</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kd">rpc</span> <span class="n">GetPrice</span><span class="p">(</span><span class="n">PriceRequest</span><span class="p">)</span> <span class="k">returns</span> <span class="p">(</span><span class="n">PriceReply</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="kd">rpc</span> <span class="n">StreamPrices</span><span class="p">(</span><span class="n">PriceRequest</span><span class="p">)</span> <span class="k">returns</span> <span class="p">(</span><span class="k">stream</span> <span class="n">PriceReply</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kd">message</span> <span class="nc">PriceRequest</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kt">string</span> <span class="n">symbol</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kd">message</span> <span class="nc">PriceReply</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kt">string</span> <span class="n">symbol</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="kt">double</span> <span class="n">price</span> <span class="o">=</span> <span class="mi">2</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="kt">int64</span> <span class="n">ts_ms</span> <span class="o">=</span> <span class="mi">3</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>Generate the code:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">pip install grpcio grpcio-tools
</span></span><span class="line"><span class="cl">python -m grpc_tools.protoc -I proto --python_out<span class="o">=</span>. --pyi_out<span class="o">=</span>. --grpc_python_out<span class="o">=</span>. proto/stocks.proto</span></span></code></pre></div></div>
<p>That produces <code>stocks_pb2.py</code> (messages), <code>stocks_pb2.pyi</code> (type hints for your editor) and <code>stocks_pb2_grpc.py</code> (stub and servicer).</p>

<h2 class="relative group">An asyncio server
    <div id="an-asyncio-server" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#an-asyncio-server" aria-label="Anchor">#</a>
    </span>
    
</h2>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># server.py</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">asyncio</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">random</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">time</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">grpc</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">stocks_pb2</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">stocks_pb2_grpc</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">quote</span><span class="p">(</span><span class="n">symbol</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">float</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="nb">round</span><span class="p">(</span><span class="mi">100</span> <span class="o">+</span> <span class="n">random</span><span class="o">.</span><span class="n">uniform</span><span class="p">(</span><span class="o">-</span><span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">),</span> <span class="mi">2</span><span class="p">)</span>  <span class="c1"># stand-in for a real feed</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">StockService</span><span class="p">(</span><span class="n">stocks_pb2_grpc</span><span class="o">.</span><span class="n">StockServiceServicer</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">def</span> <span class="nf">GetPrice</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="n">request</span><span class="o">.</span><span class="n">symbol</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">await</span> <span class="n">context</span><span class="o">.</span><span class="n">abort</span><span class="p">(</span><span class="n">grpc</span><span class="o">.</span><span class="n">StatusCode</span><span class="o">.</span><span class="n">INVALID_ARGUMENT</span><span class="p">,</span> <span class="s2">&#34;symbol is required&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">stocks_pb2</span><span class="o">.</span><span class="n">PriceReply</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="n">symbol</span><span class="o">=</span><span class="n">request</span><span class="o">.</span><span class="n">symbol</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">price</span><span class="o">=</span><span class="n">quote</span><span class="p">(</span><span class="n">request</span><span class="o">.</span><span class="n">symbol</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">            <span class="n">ts_ms</span><span class="o">=</span><span class="nb">int</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">*</span> <span class="mi">1000</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">def</span> <span class="nf">StreamPrices</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># Streams until the client cancels or the deadline expires</span>
</span></span><span class="line"><span class="cl">        <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">yield</span> <span class="n">stocks_pb2</span><span class="o">.</span><span class="n">PriceReply</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">                <span class="n">symbol</span><span class="o">=</span><span class="n">request</span><span class="o">.</span><span class="n">symbol</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                <span class="n">price</span><span class="o">=</span><span class="n">quote</span><span class="p">(</span><span class="n">request</span><span class="o">.</span><span class="n">symbol</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">                <span class="n">ts_ms</span><span class="o">=</span><span class="nb">int</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">*</span> <span class="mi">1000</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">            <span class="p">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">serve</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">server</span> <span class="o">=</span> <span class="n">grpc</span><span class="o">.</span><span class="n">aio</span><span class="o">.</span><span class="n">server</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">stocks_pb2_grpc</span><span class="o">.</span><span class="n">add_StockServiceServicer_to_server</span><span class="p">(</span><span class="n">StockService</span><span class="p">(),</span> <span class="n">server</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">server</span><span class="o">.</span><span class="n">add_insecure_port</span><span class="p">(</span><span class="s2">&#34;[::]:50051&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">await</span> <span class="n">server</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">await</span> <span class="n">server</span><span class="o">.</span><span class="n">wait_for_termination</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&#34;__main__&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">asyncio</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">serve</span><span class="p">())</span></span></span></code></pre></div></div>
<p>The synchronous <code>grpc.server()</code> runs on a thread pool, and every open stream holds a thread. With <code>grpc.aio</code>, one process can carry thousands of streams on the event loop. The usual asyncio rule still applies: a blocking call in a handler, like a synchronous database driver, stalls the whole server.</p>

<h2 class="relative group">The client and deadlines
    <div id="the-client-and-deadlines" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-client-and-deadlines" aria-label="Anchor">#</a>
    </span>
    
</h2>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># client.py</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">asyncio</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">grpc</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">stocks_pb2</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">stocks_pb2_grpc</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">main</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">with</span> <span class="n">grpc</span><span class="o">.</span><span class="n">aio</span><span class="o">.</span><span class="n">insecure_channel</span><span class="p">(</span><span class="s2">&#34;localhost:50051&#34;</span><span class="p">)</span> <span class="k">as</span> <span class="n">channel</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">stub</span> <span class="o">=</span> <span class="n">stocks_pb2_grpc</span><span class="o">.</span><span class="n">StockServiceStub</span><span class="p">(</span><span class="n">channel</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">reply</span> <span class="o">=</span> <span class="k">await</span> <span class="n">stub</span><span class="o">.</span><span class="n">GetPrice</span><span class="p">(</span><span class="n">stocks_pb2</span><span class="o">.</span><span class="n">PriceRequest</span><span class="p">(</span><span class="n">symbol</span><span class="o">=</span><span class="s2">&#34;AAPL&#34;</span><span class="p">),</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="n">reply</span><span class="o">.</span><span class="n">symbol</span><span class="p">,</span> <span class="n">reply</span><span class="o">.</span><span class="n">price</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">async</span> <span class="k">for</span> <span class="n">update</span> <span class="ow">in</span> <span class="n">stub</span><span class="o">.</span><span class="n">StreamPrices</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">                <span class="n">stocks_pb2</span><span class="o">.</span><span class="n">PriceRequest</span><span class="p">(</span><span class="n">symbol</span><span class="o">=</span><span class="s2">&#34;AAPL&#34;</span><span class="p">),</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">5</span>
</span></span><span class="line"><span class="cl">            <span class="p">):</span>
</span></span><span class="line"><span class="cl">                <span class="nb">print</span><span class="p">(</span><span class="s2">&#34;live:&#34;</span><span class="p">,</span> <span class="n">update</span><span class="o">.</span><span class="n">price</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">except</span> <span class="n">grpc</span><span class="o">.</span><span class="n">aio</span><span class="o">.</span><span class="n">AioRpcError</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">if</span> <span class="n">err</span><span class="o">.</span><span class="n">code</span><span class="p">()</span> <span class="o">!=</span> <span class="n">grpc</span><span class="o">.</span><span class="n">StatusCode</span><span class="o">.</span><span class="n">DEADLINE_EXCEEDED</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">                <span class="k">raise</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">asyncio</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">main</span><span class="p">())</span></span></span></code></pre></div></div>
<p>With the server running in one terminal and the client in another, this is what I got:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/posts/grpc-ile-yuksek-performansli-apiler/img/grpc-client-run-terminal_hu_f059fca24f94fad3.avif 480w, https://blog.furkankoykiran.com.tr/posts/grpc-ile-yuksek-performansli-apiler/img/grpc-client-run-terminal_hu_2e6a1d4fa924b471.avif 800w, https://blog.furkankoykiran.com.tr/posts/grpc-ile-yuksek-performansli-apiler/img/grpc-client-run-terminal_hu_707f081be2aad77.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/posts/grpc-ile-yuksek-performansli-apiler/img/grpc-client-run-terminal_hu_6790ac26e3c9ab5a.webp 480w, https://blog.furkankoykiran.com.tr/posts/grpc-ile-yuksek-performansli-apiler/img/grpc-client-run-terminal_hu_40fb1c296993fa01.webp 800w, https://blog.furkankoykiran.com.tr/posts/grpc-ile-yuksek-performansli-apiler/img/grpc-client-run-terminal_hu_a66d1294781d30c2.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/posts/grpc-ile-yuksek-performansli-apiler/img/grpc-client-run-terminal_hu_40fb1c296993fa01.webp" alt="Terminal showing protoc code generation, then client.py printing one AAPL price and five live prices in about 7 seconds" width="1600" height="626"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>A unary reply, then a stream that runs until the 5-second deadline</figcaption></figure><p>The unary <code>GetPrice</code> reply comes first, then the stream prints a price every second. At five seconds the deadline expires, and the client catches <code>DEADLINE_EXCEEDED</code> and exits cleanly.</p>
<p><code>timeout</code> sets the gRPC deadline. There is none by default, so if the server hangs, the client waits forever. Put a deadline on every call. It's propagated to the server, which can check <code>context.time_remaining()</code>.</p>
<p>A channel is expensive because it owns an HTTP/2 connection. Create one per target and reuse it for the life of the app.</p>

<h2 class="relative group">Status codes and interceptors
    <div id="status-codes-and-interceptors" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#status-codes-and-interceptors" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>gRPC has its own status codes instead of HTTP ones: <code>INVALID_ARGUMENT</code>, <code>NOT_FOUND</code>, <code>UNAVAILABLE</code>, <code>DEADLINE_EXCEEDED</code> and so on. I decide what the client retries from these codes: <code>UNAVAILABLE</code> is usually safe to retry, <code>INVALID_ARGUMENT</code> never is. If you want structured details with an error, such as a retry delay or per-field violations, the <code>grpcio-status</code> package supports the <code>google.rpc.Status</code> model.</p>
<p>For cross-cutting work like logging, auth and metrics, use an interceptor instead of repeating code in every method. A simple token check for the asyncio server:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">AuthInterceptor</span><span class="p">(</span><span class="n">grpc</span><span class="o">.</span><span class="n">aio</span><span class="o">.</span><span class="n">ServerInterceptor</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">token</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">_expected</span> <span class="o">=</span> <span class="p">(</span><span class="s2">&#34;authorization&#34;</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;Bearer </span><span class="si">{</span><span class="n">token</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="k">async</span> <span class="k">def</span> <span class="nf">deny</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">            <span class="k">await</span> <span class="n">context</span><span class="o">.</span><span class="n">abort</span><span class="p">(</span><span class="n">grpc</span><span class="o">.</span><span class="n">StatusCode</span><span class="o">.</span><span class="n">UNAUTHENTICATED</span><span class="p">,</span> <span class="s2">&#34;invalid token&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">_deny</span> <span class="o">=</span> <span class="n">grpc</span><span class="o">.</span><span class="n">unary_unary_rpc_method_handler</span><span class="p">(</span><span class="n">deny</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">def</span> <span class="nf">intercept_service</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">continuation</span><span class="p">,</span> <span class="n">handler_call_details</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_expected</span> <span class="ow">in</span> <span class="p">(</span><span class="n">handler_call_details</span><span class="o">.</span><span class="n">invocation_metadata</span> <span class="ow">or</span> <span class="p">()):</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="k">await</span> <span class="n">continuation</span><span class="p">(</span><span class="n">handler_call_details</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_deny</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">server</span> <span class="o">=</span> <span class="n">grpc</span><span class="o">.</span><span class="n">aio</span><span class="o">.</span><span class="n">server</span><span class="p">(</span><span class="n">interceptors</span><span class="o">=</span><span class="p">[</span><span class="n">AuthInterceptor</span><span class="p">(</span><span class="s2">&#34;s3cret&#34;</span><span class="p">)])</span></span></span></code></pre></div></div>
<p>The client sends the token as metadata: <code>await stub.GetPrice(req, metadata=[(&quot;authorization&quot;, &quot;Bearer s3cret&quot;)], timeout=2)</code>. This example rejects with a unary handler; to protect streaming methods, return a handler that matches the method type. I cover the token side in my <a href="https://blog.furkankoykiran.com.tr/en/posts/api-authentication-strategies/" >API authentication post</a>. In production, use <code>add_secure_port</code> with TLS instead of <code>add_insecure_port</code>. A bearer token over a plaintext channel protects nothing.</p>

<h2 class="relative group">Schema evolution: field numbers are forever
    <div id="schema-evolution-field-numbers-are-forever" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#schema-evolution-field-numbers-are-forever" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>On the wire, Protobuf sends field numbers, not names. So:</p>
<ul>
<li>Adding a field is safe. Old clients ignore fields they don't know.</li>
<li>Changing a field's number or type breaks compatibility.</li>
<li>When you delete a field, lock its number and name with <code>reserved</code>. Protobuf doesn't do this for you, and if someone later reuses the number for a different field, old clients will misread the data.</li>
</ul>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-protobuf" data-lang="protobuf"><span class="line"><span class="cl"><span class="kd">message</span> <span class="nc">PriceReply</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="k">reserved</span> <span class="mi">4</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="k">reserved</span> <span class="s">&#34;currency&#34;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="kt">string</span> <span class="n">symbol</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="kt">double</span> <span class="n">price</span> <span class="o">=</span> <span class="mi">2</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="kt">int64</span> <span class="n">ts_ms</span> <span class="o">=</span> <span class="mi">3</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>Renaming a field is fine for the binary format, but Protobuf's JSON mapping uses field names, so a gateway that converts to JSON will break. For a truly breaking change, I create a new package (<code>package stocks.v2;</code>) and run both versions side by side for a while.</p>

<h2 class="relative group">The HTTP/2 load balancing trap
    <div id="the-http2-load-balancing-trap" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-http2-load-balancing-trap" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Of all the gRPC pitfalls, this is the one I'd warn you about first. HTTP/2 multiplexes many requests over one TCP connection. A connection-level L4 load balancer, including a default Kubernetes <code>ClusterIP</code> service, pins that one connection to one pod, and every request from that client goes there. You run three replicas; one is on fire while two sit idle.</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/grpc-high-performance-apis/img/http2-balancing_hu_e1880ee70274b56e.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/grpc-high-performance-apis/img/http2-balancing_hu_d8ccbcbc93951fb1.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/grpc-high-performance-apis/img/http2-balancing_hu_f86723c078e9a977.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/grpc-high-performance-apis/img/http2-balancing_hu_93a4948e4b9c8dfc.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/grpc-high-performance-apis/img/http2-balancing_hu_eacc0fc16714a2e9.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/grpc-high-performance-apis/img/http2-balancing_hu_e1f0c0fe03bb5d39.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/grpc-high-performance-apis/img/http2-balancing_hu_eacc0fc16714a2e9.webp" alt="An L4 load balancer pins one HTTP/2 connection to one pod while two idle; an L7 proxy or client-side LB spreads requests across all three" width="1600" height="1130"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Per-connection versus per-request balancing</figcaption></figure><p>There are two fixes:</p>
<ol>
<li>An L7 proxy that balances per request: Envoy, a service mesh such as Istio or Linkerd, or an ingress with gRPC support.</li>
<li>Client-side load balancing. In Kubernetes, a headless service returns all pod IPs from DNS, and the gRPC client connects to each and spreads requests across them:</li>
</ol>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">channel</span> <span class="o">=</span> <span class="n">grpc</span><span class="o">.</span><span class="n">aio</span><span class="o">.</span><span class="n">insecure_channel</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;dns:///stocks-headless.default.svc.cluster.local:50051&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">options</span><span class="o">=</span><span class="p">[(</span><span class="s2">&#34;grpc.lb_policy_name&#34;</span><span class="p">,</span> <span class="s2">&#34;round_robin&#34;</span><span class="p">)],</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span></span></span></code></pre></div></div>
<p>With the second option the client has to re-resolve DNS when pods change. Setting <code>grpc.max_connection_age_ms</code> on the server forces connections to be recycled periodically, which helps. I cover the Kubernetes side in my <a href="https://blog.furkankoykiran.com.tr/en/posts/kubernetes-container-orchestration/" >Kubernetes post</a>.</p>

<h2 class="relative group">Debugging and health checks
    <div id="debugging-and-health-checks" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#debugging-and-health-checks" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>REST's best feature is that you can poke anything with <code>curl</code>. The gRPC equivalent is <code>grpcurl</code>, and with server reflection enabled it can list and call methods without the <code>.proto</code> file. The <code>grpcio-reflection</code> and <code>grpcio-health-checking</code> packages add reflection and the standard health service that Kubernetes probes can call. I'd set up both from the start.</p>
<p>Browsers can't call gRPC directly because browser APIs don't expose HTTP/2 frames and trailers. If you must, use gRPC-Web through a proxy, or the Connect protocol, which speaks both gRPC and JSON. For pushing live data to a browser, <a href="https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/" >WebSockets</a> are a more natural fit.</p>

<h2 class="relative group">Where to go next
    <div id="where-to-go-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-to-go-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Put the <code>.proto</code>, server and client from this post in one folder and run them. Then add <code>grpcio-reflection</code> to the server and explore it with <code>grpcurl -plaintext localhost:50051 list</code>. The official <a href="https://grpc.io/docs/languages/python/"  target="_blank" rel="noreferrer">gRPC Python docs</a> and the <a href="https://grpc.github.io/grpc/python/grpc_asyncio.html"  target="_blank" rel="noreferrer">asyncio API reference</a> are the sources I trust.</p>
<p>If you also need to expose the service over REST, my <a href="https://blog.furkankoykiran.com.tr/en/posts/python-fastapi-rest-api/" >FastAPI post</a> covers that side, and for moving events between services, see my <a href="https://blog.furkankoykiran.com.tr/en/posts/apache-kafka-event-streaming/" >Kafka post</a>.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/grpc-high-performance-apis/feature_hu_bf1880f0e34fd61d.jpg" medium="image" type="image/jpeg"/></item><item><title>Advanced pytest: Fixtures, Mocking and Hypothesis</title><link>https://blog.furkankoykiran.com.tr/en/posts/pytest-advanced-testing-techniques/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/pytest-advanced-testing-techniques/</guid><pubDate>Fri, 10 Oct 2025 12:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Python</category><category>python</category><category>testing</category><category>asyncio</category><category>ci-cd</category><description>Advanced testing with pytest 8: fixture scopes, parametrize, mocking with mocker and respx, async tests, Hypothesis property tests, snapshots and CI settings.</description><content:encoded><![CDATA[<p>Every Python library and service I write is tested with pytest. This post collects the advanced pytest techniques I actually use: fixture scopes, parametrize, mocking, async tests, property-based testing with Hypothesis, and CI settings. Everything here targets pytest 8, and the parametrize and <code>RaisesGroup</code> examples also run on pytest 9.1.</p>
<p>A note before starting: <code>unittest</code> is fine. It's in the standard library and it works. But writing classes, remembering <code>self.assertEqual</code>, and jumping between <code>setUp</code> and <code>tearDown</code> makes tests feel like a chore. pytest does the same job with plain functions and plain <code>assert</code>, and on failure it shows both sides of the expression in detail. It also runs existing <code>unittest</code> tests unchanged, so you can migrate gradually.</p>

<h2 class="relative group">Configure the project up front
    <div id="configure-the-project-up-front" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#configure-the-project-up-front" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>All the settings live in <code>pyproject.toml</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">tool</span><span class="p">.</span><span class="nx">pytest</span><span class="p">.</span><span class="nx">ini_options</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="nx">minversion</span> <span class="p">=</span> <span class="s2">&#34;8.0&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nx">testpaths</span> <span class="p">=</span> <span class="p">[</span><span class="s2">&#34;tests&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="nx">addopts</span> <span class="p">=</span> <span class="p">[</span><span class="s2">&#34;-ra&#34;</span><span class="p">,</span> <span class="s2">&#34;--strict-markers&#34;</span><span class="p">,</span> <span class="s2">&#34;--strict-config&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="nx">xfail_strict</span> <span class="p">=</span> <span class="kc">true</span>
</span></span><span class="line"><span class="cl"><span class="nx">markers</span> <span class="p">=</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;slow: long-running tests (run separately in CI)&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">]</span></span></span></code></pre></div></div>
<p><code>--strict-markers</code> turns a typo like <code>@pytest.mark.slwo</code> into an error instead of silently ignoring it. <code>xfail_strict</code> fails tests marked <code>xfail</code> that start passing, so fixed bugs don't stay marked as broken. <code>-ra</code> prints a summary of why tests were skipped or failed at the end of the run.</p>

<h2 class="relative group">Fixtures and scope
    <div id="fixtures-and-scope" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#fixtures-and-scope" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A fixture prepares what a test needs and cleans up afterwards: setup before <code>yield</code>, teardown after. Scope decides how long the fixture lives: <code>function</code> (the default), <code>class</code>, <code>module</code>, <code>package</code> or <code>session</code>. This diagram from the pytest docs shows that wider-scoped fixtures are set up first:</p>
<figure class="fk-figure"><img src="https://blog.furkankoykiran.com.tr/posts/pytest-ile-ileri-seviye-test-teknikleri/img/pytest-fixtures-scope-order.svg" alt="pytest fixture order: sess, pack, mod, cls and func fixtures set up from widest to narrowest scope, with test_order at the bottom" class="fk-svg my-0 rounded-md"
    loading="lazy" decoding="async"><figcaption>Wider-scoped fixtures are set up first. Source: pytest documentation, MIT license.</figcaption></figure><p>The pattern I get the most out of: open expensive resources (a database engine, a Docker container) once at <code>session</code> scope, and give each test a cheap, isolated slice of it. With SQLAlchemy 2, running every test inside a transaction that gets rolled back looks like this:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># tests/conftest.py</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">os</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">pytest</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">sqlalchemy</span> <span class="kn">import</span> <span class="n">create_engine</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">sqlalchemy.orm</span> <span class="kn">import</span> <span class="n">Session</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">app.models</span> <span class="kn">import</span> <span class="n">Base</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@pytest.fixture</span><span class="p">(</span><span class="n">scope</span><span class="o">=</span><span class="s2">&#34;session&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">engine</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">    <span class="n">engine</span> <span class="o">=</span> <span class="n">create_engine</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">&#34;TEST_DATABASE_URL&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">    <span class="n">Base</span><span class="o">.</span><span class="n">metadata</span><span class="o">.</span><span class="n">create_all</span><span class="p">(</span><span class="n">engine</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">yield</span> <span class="n">engine</span>
</span></span><span class="line"><span class="cl">    <span class="n">Base</span><span class="o">.</span><span class="n">metadata</span><span class="o">.</span><span class="n">drop_all</span><span class="p">(</span><span class="n">engine</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">engine</span><span class="o">.</span><span class="n">dispose</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@pytest.fixture</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">db_session</span><span class="p">(</span><span class="n">engine</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">engine</span><span class="o">.</span><span class="n">connect</span><span class="p">()</span> <span class="k">as</span> <span class="n">connection</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">transaction</span> <span class="o">=</span> <span class="n">connection</span><span class="o">.</span><span class="n">begin</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="n">session</span> <span class="o">=</span> <span class="n">Session</span><span class="p">(</span><span class="n">bind</span><span class="o">=</span><span class="n">connection</span><span class="p">,</span> <span class="n">join_transaction_mode</span><span class="o">=</span><span class="s2">&#34;create_savepoint&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">yield</span> <span class="n">session</span>
</span></span><span class="line"><span class="cl">        <span class="n">session</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="n">transaction</span><span class="o">.</span><span class="n">rollback</span><span class="p">()</span></span></span></code></pre></div></div>
<p>With <code>join_transaction_mode=&quot;create_savepoint&quot;</code>, the code under test can call <code>session.commit()</code> without breaking the outer transaction. When the test ends, everything is rolled back and the next test sees a clean database.</p>
<p>Fixtures in <code>conftest.py</code> are available to every test in that directory and below, with no import needed.</p>

<h3 class="relative group">Factory fixtures
    <div id="factory-fixtures" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#factory-fixtures" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Sometimes a test needs several objects of the same kind. Having the fixture return a function that builds them keeps things tidy:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="nd">@pytest.fixture</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">make_user</span><span class="p">(</span><span class="n">db_session</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">_make</span><span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="s2">&#34;furkan&#34;</span><span class="p">,</span> <span class="n">is_admin</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">User</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">user</span> <span class="o">=</span> <span class="n">User</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">name</span><span class="p">,</span> <span class="n">is_admin</span><span class="o">=</span><span class="n">is_admin</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">db_session</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">user</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">db_session</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">user</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">_make</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">test_admin_can_delete</span><span class="p">(</span><span class="n">make_user</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">admin</span> <span class="o">=</span> <span class="n">make_user</span><span class="p">(</span><span class="s2">&#34;admin&#34;</span><span class="p">,</span> <span class="n">is_admin</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">target</span> <span class="o">=</span> <span class="n">make_user</span><span class="p">(</span><span class="s2">&#34;target&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">assert</span> <span class="n">admin</span><span class="o">.</span><span class="n">can_delete</span><span class="p">(</span><span class="n">target</span><span class="p">)</span></span></span></code></pre></div></div>

<h3 class="relative group">Built-in fixtures
    <div id="built-in-fixtures" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#built-in-fixtures" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Check pytest's built-ins before writing your own. The ones I use most:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">test_writes_report</span><span class="p">(</span><span class="n">tmp_path</span><span class="p">,</span> <span class="n">monkeypatch</span><span class="p">,</span> <span class="n">caplog</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">monkeypatch</span><span class="o">.</span><span class="n">setenv</span><span class="p">(</span><span class="s2">&#34;REPORT_DIR&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">tmp_path</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="n">write_report</span><span class="p">({</span><span class="s2">&#34;ok&#34;</span><span class="p">:</span> <span class="kc">True</span><span class="p">})</span>
</span></span><span class="line"><span class="cl">    <span class="k">assert</span> <span class="p">(</span><span class="n">tmp_path</span> <span class="o">/</span> <span class="s2">&#34;report.json&#34;</span><span class="p">)</span><span class="o">.</span><span class="n">exists</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">assert</span> <span class="s2">&#34;report written&#34;</span> <span class="ow">in</span> <span class="n">caplog</span><span class="o">.</span><span class="n">text</span></span></span></code></pre></div></div>
<p><code>tmp_path</code> is a fresh temporary directory per test, <code>monkeypatch</code> changes environment variables and attributes and restores them afterwards, and <code>caplog</code> captures log output.</p>

<h2 class="relative group">One test, many cases with parametrize
    <div id="one-test-many-cases-with-parametrize" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#one-test-many-cases-with-parametrize" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>When testing one function with different inputs, use <code>parametrize</code> instead of copy-paste. <code>pytest.param</code> gives each case a readable id and, if needed, a marker:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">pytest</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@pytest.mark.parametrize</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="p">(</span><span class="s2">&#34;email&#34;</span><span class="p">,</span> <span class="s2">&#34;expected&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="p">[</span>
</span></span><span class="line"><span class="cl">        <span class="n">pytest</span><span class="o">.</span><span class="n">param</span><span class="p">(</span><span class="s2">&#34;test@example.com&#34;</span><span class="p">,</span> <span class="kc">True</span><span class="p">,</span> <span class="nb">id</span><span class="o">=</span><span class="s2">&#34;simple&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">        <span class="n">pytest</span><span class="o">.</span><span class="n">param</span><span class="p">(</span><span class="s2">&#34;user+tag@gmail.com&#34;</span><span class="p">,</span> <span class="kc">True</span><span class="p">,</span> <span class="nb">id</span><span class="o">=</span><span class="s2">&#34;plus-tag&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">        <span class="n">pytest</span><span class="o">.</span><span class="n">param</span><span class="p">(</span><span class="s2">&#34;plainaddress&#34;</span><span class="p">,</span> <span class="kc">False</span><span class="p">,</span> <span class="nb">id</span><span class="o">=</span><span class="s2">&#34;no-at&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">        <span class="n">pytest</span><span class="o">.</span><span class="n">param</span><span class="p">(</span><span class="s2">&#34;@example.com&#34;</span><span class="p">,</span> <span class="kc">False</span><span class="p">,</span> <span class="nb">id</span><span class="o">=</span><span class="s2">&#34;no-local-part&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">        <span class="n">pytest</span><span class="o">.</span><span class="n">param</span><span class="p">(</span><span class="s2">&#34;ü@exämple.com&#34;</span><span class="p">,</span> <span class="kc">True</span><span class="p">,</span> <span class="nb">id</span><span class="o">=</span><span class="s2">&#34;unicode&#34;</span><span class="p">,</span> <span class="n">marks</span><span class="o">=</span><span class="n">pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">xfail</span><span class="p">(</span><span class="n">reason</span><span class="o">=</span><span class="s2">&#34;no IDN support yet&#34;</span><span class="p">)),</span>
</span></span><span class="line"><span class="cl">    <span class="p">],</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">test_is_valid_email</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">expected</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">assert</span> <span class="n">is_valid_email</span><span class="p">(</span><span class="n">email</span><span class="p">)</span> <span class="ow">is</span> <span class="n">expected</span></span></span></code></pre></div></div>
<p>Failures show up as <code>test_is_valid_email[no-at]</code>, and <code>pytest -k unicode</code> runs a single case.</p>
<p>With a simple regex-based <code>is_valid_email</code>, the <code>RaisesGroup</code> test from further down and the <code>pyproject.toml</code> settings above, <code>pytest -v</code> prints this. The unicode case is an expected <code>XFAIL</code>, and <code>-ra</code> repeats the reason in the summary:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/pytest-advanced-testing-techniques/img/pytest-parametrize-run_hu_9304099b2eb87d6b.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/pytest-advanced-testing-techniques/img/pytest-parametrize-run_hu_839834dc0b964a34.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/pytest-advanced-testing-techniques/img/pytest-parametrize-run_hu_7bbc205f7028ee04.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/pytest-advanced-testing-techniques/img/pytest-parametrize-run_hu_3b47236c1bbbb607.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/pytest-advanced-testing-techniques/img/pytest-parametrize-run_hu_45d36a7fbbd36fc2.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/pytest-advanced-testing-techniques/img/pytest-parametrize-run_hu_659fa3ce7c3d7dd9.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/pytest-advanced-testing-techniques/img/pytest-parametrize-run_hu_45d36a7fbbd36fc2.webp" alt="pytest -v output: four parametrize cases and test_taskgroup_errors pass, the unicode case is XFAIL with &#39;no IDN support yet&#39;" width="1476" height="750"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>A real run on pytest 9.1</figcaption></figure>
<h2 class="relative group">Mocking: keep the outside world out
    <div id="mocking-keep-the-outside-world-out" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#mocking-keep-the-outside-world-out" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A unit test should not hit the internet. A test that really calls a payment API is slow, flaky and, in the worst case, expensive. The <code>mocker</code> fixture from <code>pytest-mock</code> wraps <code>unittest.mock</code> and undoes every patch automatically when the test ends:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># app/payments.py</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">httpx</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">charge</span><span class="p">(</span><span class="n">amount</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">resp</span> <span class="o">=</span> <span class="n">httpx</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="s2">&#34;https://api.payments.example/charge&#34;</span><span class="p">,</span> <span class="n">json</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;amount&#34;</span><span class="p">:</span> <span class="n">amount</span><span class="p">},</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">10</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">resp</span><span class="o">.</span><span class="n">raise_for_status</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">resp</span><span class="o">.</span><span class="n">json</span><span class="p">()[</span><span class="s2">&#34;status&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># tests/test_payments.py</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">app.payments</span> <span class="kn">import</span> <span class="n">charge</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">test_charge_success</span><span class="p">(</span><span class="n">mocker</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">post</span> <span class="o">=</span> <span class="n">mocker</span><span class="o">.</span><span class="n">patch</span><span class="p">(</span><span class="s2">&#34;app.payments.httpx.post&#34;</span><span class="p">,</span> <span class="n">autospec</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">post</span><span class="o">.</span><span class="n">return_value</span><span class="o">.</span><span class="n">json</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;status&#34;</span><span class="p">:</span> <span class="s2">&#34;success&#34;</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">assert</span> <span class="n">charge</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="o">==</span> <span class="s2">&#34;success&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">post</span><span class="o">.</span><span class="n">assert_called_once_with</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;https://api.payments.example/charge&#34;</span><span class="p">,</span> <span class="n">json</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;amount&#34;</span><span class="p">:</span> <span class="mi">100</span><span class="p">},</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">10</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span></span></span></code></pre></div></div>
<p>Patch where the name is used (<code>app.payments.httpx.post</code>), not where it's defined. <code>autospec=True</code> makes the fake match the real function's signature, so a call with wrong arguments fails the test.</p>
<p>If your HTTP client is <code>httpx</code>, mocking the network layer with <code>respx</code> is cleaner than patching functions. That's how the tests in <a href="https://furkankoykiran.com.tr/en/projects/speedapi_lib/"  target="_blank" rel="noreferrer">speedapi_lib</a>, a Python SDK I wrote, are set up: a fixture in <code>conftest.py</code> opens a <code>respx</code> router for the API's base URL, and each test defines its own responses:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">httpx</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">pytest</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">respx</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">BASE_URL</span> <span class="o">=</span> <span class="s2">&#34;https://api.example.com&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@pytest.fixture</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">mock_router</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">respx</span><span class="o">.</span><span class="n">mock</span><span class="p">(</span><span class="n">base_url</span><span class="o">=</span><span class="n">BASE_URL</span><span class="p">,</span> <span class="n">assert_all_called</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span> <span class="k">as</span> <span class="n">router</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">yield</span> <span class="n">router</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">test_get_balance</span><span class="p">(</span><span class="n">mock_router</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">mock_router</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;/balance&#34;</span><span class="p">)</span><span class="o">.</span><span class="n">mock</span><span class="p">(</span><span class="n">return_value</span><span class="o">=</span><span class="n">httpx</span><span class="o">.</span><span class="n">Response</span><span class="p">(</span><span class="mi">200</span><span class="p">,</span> <span class="n">json</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;amount&#34;</span><span class="p">:</span> <span class="mi">10</span><span class="p">}))</span>
</span></span><span class="line"><span class="cl">    <span class="n">resp</span> <span class="o">=</span> <span class="n">httpx</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">BASE_URL</span><span class="si">}</span><span class="s2">/balance&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">assert</span> <span class="n">resp</span><span class="o">.</span><span class="n">json</span><span class="p">()</span> <span class="o">==</span> <span class="p">{</span><span class="s2">&#34;amount&#34;</span><span class="p">:</span> <span class="mi">10</span><span class="p">}</span></span></span></code></pre></div></div>
<p>A request to a route you didn't mock raises an error, so no test can accidentally reach the real API.</p>

<h2 class="relative group">Testing async code
    <div id="testing-async-code" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#testing-async-code" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>For async tests I use <code>pytest-asyncio</code>. Both <a href="https://github.com/furkankoykiran/speedapi_lib"  target="_blank" rel="noreferrer">speedapi_lib</a> and <a href="https://github.com/furkankoykiran/Quorum"  target="_blank" rel="noreferrer">Quorum</a> use the same configuration:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">tool</span><span class="p">.</span><span class="nx">pytest</span><span class="p">.</span><span class="nx">ini_options</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="nx">asyncio_mode</span> <span class="p">=</span> <span class="s2">&#34;auto&#34;</span>
</span></span><span class="line"><span class="cl"><span class="nx">asyncio_default_fixture_loop_scope</span> <span class="p">=</span> <span class="s2">&#34;function&#34;</span></span></span></code></pre></div></div>
<p>In <code>auto</code> mode every <code>async def test_...</code> runs without a marker, and async fixtures are declared with the regular <code>@pytest.fixture</code>. pytest-asyncio 1.0 removed the old <code>event_loop</code> fixture; if you need a wider loop, use <code>@pytest.mark.asyncio(loop_scope=&quot;module&quot;)</code>.</p>
<p>Mock async functions with <code>AsyncMock</code>. For TaskGroup, timeouts and the rest, see the <a href="https://blog.furkankoykiran.com.tr/en/posts/python-asyncio-concurrent-processing/" >asyncio post</a>.</p>

<h2 class="relative group">What changed in pytest 8
    <div id="what-changed-in-pytest-8" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-changed-in-pytest-8" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A few changes in the pytest 8 series worth knowing about:</p>
<ul>
<li>Since 8.4, an async test function with no plugin to run it fails outright instead of warning and being skipped. If you had async tests that were silently skipped, CI may go red.</li>
<li>Also since 8.4, a test function that returns anything other than <code>None</code> fails. A test that returns a value has usually forgotten its <code>assert</code>.</li>
<li>8.4 added <code>pytest.RaisesGroup</code> for the <code>ExceptionGroup</code>s that TaskGroup raises:</li>
</ul>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">asyncio</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">pytest</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">test_taskgroup_errors</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">def</span> <span class="nf">boom</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">        <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">&#34;bad data&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">pytest</span><span class="o">.</span><span class="n">RaisesGroup</span><span class="p">(</span><span class="ne">ValueError</span><span class="p">,</span> <span class="ne">ValueError</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">async</span> <span class="k">with</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">TaskGroup</span><span class="p">()</span> <span class="k">as</span> <span class="n">tg</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">tg</span><span class="o">.</span><span class="n">create_task</span><span class="p">(</span><span class="n">boom</span><span class="p">())</span>
</span></span><span class="line"><span class="cl">            <span class="n">tg</span><span class="o">.</span><span class="n">create_task</span><span class="p">(</span><span class="n">boom</span><span class="p">())</span></span></span></code></pre></div></div>
<p>For a single exception, <code>pytest.raises(ValueError, match=&quot;bad&quot;)</code> is still the right tool. <code>match</code> takes a regex; use it to check the message too.</p>

<h2 class="relative group">Property-based testing with Hypothesis
    <div id="property-based-testing-with-hypothesis" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#property-based-testing-with-hypothesis" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Hand-written examples test the cases you thought of. Hypothesis asks you for a rule and then searches for input that breaks it: empty strings, zero, negative numbers, odd Unicode. When it finds a failure, it shrinks the input to the smallest example and shows you that.</p>
<p>The most useful rule is the round trip: serialize, deserialize, get the same thing back.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">hypothesis</span> <span class="kn">import</span> <span class="n">given</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">hypothesis</span> <span class="kn">import</span> <span class="n">strategies</span> <span class="k">as</span> <span class="n">st</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@given</span><span class="p">(</span><span class="n">st</span><span class="o">.</span><span class="n">dictionaries</span><span class="p">(</span><span class="n">st</span><span class="o">.</span><span class="n">text</span><span class="p">(),</span> <span class="n">st</span><span class="o">.</span><span class="n">integers</span><span class="p">()</span> <span class="o">|</span> <span class="n">st</span><span class="o">.</span><span class="n">text</span><span class="p">()</span> <span class="o">|</span> <span class="n">st</span><span class="o">.</span><span class="n">none</span><span class="p">()))</span>
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">test_json_roundtrip</span><span class="p">(</span><span class="n">data</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">assert</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">data</span><span class="p">))</span> <span class="o">==</span> <span class="n">data</span></span></span></code></pre></div></div>
<p>If your code has pairs like <code>encode</code>/<code>decode</code> or <code>to_dict</code>/<code>from_dict</code>, this pattern catches real bugs in a few lines. For pricing code you can write rules like &quot;the total is never negative&quot; or &quot;a discount never exceeds the price&quot;.</p>

<h2 class="relative group">Snapshot tests
    <div id="snapshot-tests" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#snapshot-tests" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Asserting a large JSON response or generated HTML line by line is tedious. <code>syrupy</code> stores the output in a file on the first run and compares against it afterwards:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">test_report_output</span><span class="p">(</span><span class="n">snapshot</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="k">assert</span> <span class="n">build_report</span><span class="p">(</span><span class="n">month</span><span class="o">=</span><span class="s2">&#34;2025-09&#34;</span><span class="p">)</span> <span class="o">==</span> <span class="n">snapshot</span></span></span></code></pre></div></div>
<p>When you change the output on purpose, <code>pytest --snapshot-update</code> rewrites the stored snapshots. The change shows up in the git diff, so it gets seen in review. The only real risk is someone running <code>--snapshot-update</code> without looking.</p>

<h2 class="relative group">Speed and coverage in CI
    <div id="speed-and-coverage-in-ci" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#speed-and-coverage-in-ci" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The plugins I use:</p>
<ul>
<li><code>pytest-xdist</code>: <code>-n auto</code> spreads tests across one process per core. Hidden dependencies between tests show up here, which is a good thing.</li>
<li><code>pytest-cov</code>: <code>--cov=src --cov-report=term-missing</code> shows which lines are untested. I look at missing lines in critical files rather than chasing a percentage.</li>
<li><code>pytest-rerunfailures</code>: retries a flaky test with <code>@pytest.mark.flaky(reruns=2)</code>. Use it as a stopgap and go find the root cause (a race, a fixed <code>sleep</code>, test order).</li>
</ul>
<p>The command I usually run in CI:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">pytest -n auto --cov<span class="o">=</span>src --cov-report<span class="o">=</span>term-missing --durations<span class="o">=</span><span class="m">10</span></span></span></code></pre></div></div>
<p><code>--durations=10</code> lists the ten slowest tests. A unit test that takes seconds usually means a forgotten <code>sleep</code> or a real network call. The GitHub Actions side is covered in the <a href="https://blog.furkankoykiran.com.tr/en/posts/github-actions-ci-cd-pipeline/" >CI/CD post</a>.</p>

<h2 class="relative group">Habits I avoid
    <div id="habits-i-avoid" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#habits-i-avoid" aria-label="Anchor">#</a>
    </span>
    
</h2>
<ul>
<li>Branching with <code>if</code> inside a test. If you need a branch, write two tests or parametrize.</li>
<li>Tests that depend on each other. If <code>test_2</code> can't pass without <code>test_1</code>, things break as soon as tests run in random order (<code>pytest-randomly</code>) or in parallel.</li>
<li>Waiting for something with <code>time.sleep()</code>. Poll for the condition in a short loop, or use an <code>Event</code> in async code.</li>
<li>Mocking everything. Once you're mocking your own internal functions, the test checks implementation details rather than behavior.</li>
</ul>

<h2 class="relative group">Where to start
    <div id="where-to-start" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-to-start" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Add the <code>pyproject.toml</code> settings above to an existing project and run <code>pytest -ra --durations=10</code>. You'll immediately see which tests are skipped and which are slow. Then find an <code>encode</code>/<code>decode</code> pair in your code and write one Hypothesis test for it.</p>
<p>Docs: <a href="https://docs.pytest.org/"  target="_blank" rel="noreferrer">pytest</a>, <a href="https://pytest-asyncio.readthedocs.io/"  target="_blank" rel="noreferrer">pytest-asyncio</a>, <a href="https://hypothesis.readthedocs.io/"  target="_blank" rel="noreferrer">Hypothesis</a>, <a href="https://lundberg.github.io/respx/"  target="_blank" rel="noreferrer">respx</a>, <a href="https://github.com/syrupy-project/syrupy"  target="_blank" rel="noreferrer">syrupy</a>.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/pytest-advanced-testing-techniques/feature_hu_35bc54c5acab5f01.jpg" medium="image" type="image/jpeg"/></item><item><title>MinIO Object Storage After the Community Edition Ended</title><link>https://blog.furkankoykiran.com.tr/en/posts/minio-object-storage/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/minio-object-storage/</guid><pubDate>Sun, 28 Sep 2025 11:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>DevOps</category><category>devops</category><category>docker</category><category>python</category><category>security</category><description>Object storage with MinIO: stateless apps, boto3 and presigned URLs, erasure coding, object lock, and what to run now that the community edition is archived.</description><content:encoded><![CDATA[<p>For a few years, MinIO was the shortest path to running S3 on your own server. That has changed: the open source community edition was wound down step by step, and its GitHub repository is now archived. This post covers why you should move files to object storage, how to work with the S3 API through boto3, concepts like erasure coding and object lock, and which software I'd pick for a new setup. The code doesn't depend on the storage backend, so it keeps working when you move off MinIO.</p>

<h2 class="relative group">The problem with files on the app server's disk
    <div id="the-problem-with-files-on-the-app-servers-disk" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-problem-with-files-on-the-app-servers-disk" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The classic upload flow: take the file from the form, save it as <code>/var/www/uploads/user_123.jpg</code>. On a single server that works. It breaks the day you add a second server behind a load balancer. A user uploads a photo to server 1, refreshes, lands on server 2 and gets a 404.</p>
<p>The fix is to make the app stateless. Files live in separate object storage, and the app only knows which file is where.</p>
<p>Object storage is flat. There are no real directories: each object sits in a bucket under a key, with metadata attached. A key like <code>photos/2026/a.jpg</code> looks like a path, but it's just a string.</p>

<h2 class="relative group">Direct uploads with presigned URLs
    <div id="direct-uploads-with-presigned-urls" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#direct-uploads-with-presigned-urls" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The main benefit is that large files never pass through your app servers. The app issues a short-lived signed URL, and the browser uploads to or downloads from storage directly. The diagram below shows the upload flow step by step.</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/minio-object-storage/img/presigned-upload-flow_hu_8d8228bc4d8a829d.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/minio-object-storage/img/presigned-upload-flow_hu_7bca1b01e1416e1f.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/minio-object-storage/img/presigned-upload-flow_hu_4a3268d539573d8a.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/minio-object-storage/img/presigned-upload-flow_hu_9560936140ee52f7.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/minio-object-storage/img/presigned-upload-flow_hu_42e7f184bea4aeb4.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/minio-object-storage/img/presigned-upload-flow_hu_84cfeaf69935ffa1.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/minio-object-storage/img/presigned-upload-flow_hu_42e7f184bea4aeb4.webp" alt="The browser gets a 10-minute presigned PUT URL from the API, uploads straight to storage, then reports the key back to the API" width="1600" height="1172"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>The file never passes through the app server; the API only grants access and records the key</figcaption></figure><p>Because the S3 API is the de facto standard, the code doesn't care which storage server sits behind it. This boto3 client works with AWS S3, MinIO, Garage, SeaweedFS, Ceph RGW and Cloudflare R2; only <code>endpoint_url</code> and the credentials change:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">os</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">boto3</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">botocore.config</span> <span class="kn">import</span> <span class="n">Config</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">s3</span> <span class="o">=</span> <span class="n">boto3</span><span class="o">.</span><span class="n">client</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;s3&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">endpoint_url</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">&#34;S3_ENDPOINT&#34;</span><span class="p">],</span>  <span class="c1"># e.g. http://localhost:9000</span>
</span></span><span class="line"><span class="cl">    <span class="n">aws_access_key_id</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">&#34;S3_ACCESS_KEY&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">    <span class="n">aws_secret_access_key</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">&#34;S3_SECRET_KEY&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">    <span class="n">region_name</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;S3_REGION&#34;</span><span class="p">,</span> <span class="s2">&#34;us-east-1&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="n">config</span><span class="o">=</span><span class="n">Config</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">signature_version</span><span class="o">=</span><span class="s2">&#34;s3v4&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">s3</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;addressing_style&#34;</span><span class="p">:</span> <span class="s2">&#34;path&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="c1"># boto3 1.36+ default checksums break uploads on some S3-compatible servers</span>
</span></span><span class="line"><span class="cl">        <span class="n">request_checksum_calculation</span><span class="o">=</span><span class="s2">&#34;when_required&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">response_checksum_validation</span><span class="o">=</span><span class="s2">&#34;when_required&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">),</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">upload_and_share</span><span class="p">(</span><span class="n">local_path</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">bucket</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">key</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Upload a file and return a download link valid for one hour.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">s3</span><span class="o">.</span><span class="n">upload_file</span><span class="p">(</span><span class="n">local_path</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span>  <span class="c1"># switches to multipart for large files</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">s3</span><span class="o">.</span><span class="n">generate_presigned_url</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;get_object&#34;</span><span class="p">,</span> <span class="n">Params</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;Bucket&#34;</span><span class="p">:</span> <span class="n">bucket</span><span class="p">,</span> <span class="s2">&#34;Key&#34;</span><span class="p">:</span> <span class="n">key</span><span class="p">},</span> <span class="n">ExpiresIn</span><span class="o">=</span><span class="mi">3600</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">presigned_upload_url</span><span class="p">(</span><span class="n">bucket</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">key</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">content_type</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;URL the browser can PUT to directly, valid for 10 minutes.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">s3</span><span class="o">.</span><span class="n">generate_presigned_url</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;put_object&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">Params</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;Bucket&#34;</span><span class="p">:</span> <span class="n">bucket</span><span class="p">,</span> <span class="s2">&#34;Key&#34;</span><span class="p">:</span> <span class="n">key</span><span class="p">,</span> <span class="s2">&#34;ContentType&#34;</span><span class="p">:</span> <span class="n">content_type</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="n">ExpiresIn</span><span class="o">=</span><span class="mi">600</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span></span></span></code></pre></div></div>
<p>Keep credentials out of the code; read them from the environment or a secret manager. The checksum settings fix a real trap: the new default integrity checks that shipped with boto3 1.36 break uploads on S3-compatible servers that don't support them. If your provider does support them, drop those two lines.</p>
<p>For direct browser uploads, also add a CORS rule to the bucket. Without it the browser blocks the PUT.</p>

<h2 class="relative group">What happened to MinIO
    <div id="what-happened-to-minio" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-happened-to-minio" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The short timeline:</p>
<ul>
<li>In spring 2025, admin features were removed from the community edition's web console, leaving mostly an object browser.</li>
<li>In October 2025, official Docker images and binaries stopped. The last community release, <code>RELEASE.2025-10-15T17-29-55Z</code>, shipped as source only.</li>
<li>In December 2025 the repository went into maintenance mode: no new features, no PRs accepted.</li>
<li>After that, the <a href="https://github.com/minio/minio"  target="_blank" rel="noreferrer">minio/minio</a> repository was archived and became read-only.</li>
</ul>
<p>On GitHub, the repository carries a &quot;Public archive&quot; badge next to its name; the license is still AGPLv3.</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/posts/minio-ile-object-storage-sistemi/img/minio-repo-public-archive_hu_d403d371ba52eb4.avif 480w, https://blog.furkankoykiran.com.tr/posts/minio-ile-object-storage-sistemi/img/minio-repo-public-archive_hu_3ec094cdd6c6f440.avif 800w, https://blog.furkankoykiran.com.tr/posts/minio-ile-object-storage-sistemi/img/minio-repo-public-archive_hu_8361a1237e44443c.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/posts/minio-ile-object-storage-sistemi/img/minio-repo-public-archive_hu_7a2b319f367f9f74.webp 480w, https://blog.furkankoykiran.com.tr/posts/minio-ile-object-storage-sistemi/img/minio-repo-public-archive_hu_14a36da17bcbf1b2.webp 800w, https://blog.furkankoykiran.com.tr/posts/minio-ile-object-storage-sistemi/img/minio-repo-public-archive_hu_99b6c4e9e9cfa240.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/posts/minio-ile-object-storage-sistemi/img/minio-repo-public-archive_hu_14a36da17bcbf1b2.webp" alt="Header of the minio/minio GitHub repository: Public archive badge, 61.3k stars and a last commit clarifying the state of the project" width="1600" height="372"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>The archived minio/minio repository. Source: GitHub</figcaption></figure><p>The company now focuses on its commercial product, AIStor. The code is still AGPLv3, so forking is allowed and community forks exist. What's gone is &quot;pull the official image and run it&quot;: the <code>minio/minio</code> image from older <code>docker-compose.yml</code> files is no longer on Docker Hub.</p>
<p>A running deployment won't break tomorrow. The real risk is security: when the next vulnerability appears, no official fix will ship. So my advice is:</p>
<ul>
<li>Don't rip out an existing MinIO setup overnight, but keep it off the public internet and plan a migration.</li>
<li>Don't start new projects on MinIO community.</li>
<li>If you need commercial support and MinIO's feature set, read AIStor's license terms.</li>
</ul>

<h2 class="relative group">What I'd pick today
    <div id="what-id-pick-today" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-id-pick-today" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>All of these speak the S3 API, so the boto3 code above runs unchanged. The differences are in operations:</p>
<div class="fk-table" role="region" tabindex="0" aria-label="Scrollable table">
<table>
  <thead>
    <tr>
      <th>Option</th>
      <th>When</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><a href="https://garagehq.deuxfleurs.fr/"  target="_blank" rel="noreferrer">Garage</a></td>
      <td>Small to medium scale, a few nodes, possibly in different locations. Written in Rust and lightweight.</td>
    </tr>
    <tr>
      <td><a href="https://github.com/seaweedfs/seaweedfs"  target="_blank" rel="noreferrer">SeaweedFS</a></td>
      <td>Lots of small files and high volume. Also offers a filer and FUSE besides S3.</td>
    </tr>
    <tr>
      <td>Ceph RGW</td>
      <td>You already run Ceph or you're at petabyte scale. The steepest learning curve of the lot.</td>
    </tr>
    <tr>
      <td>Managed S3 (AWS S3, R2, Backblaze B2)</td>
      <td>You don't want to run hardware. If your data may leave the country, this is the least work.</td>
    </tr>
  </tbody>
</table>
</div><p>When data has to stay in a specific jurisdiction (GDPR, financial or health data), self-hosting still makes sense, and Garage is the easiest one for a small team to start with. Its setup changes between versions, so I won't copy it here; the <a href="https://garagehq.deuxfleurs.fr/documentation/quick-start/"  target="_blank" rel="noreferrer">official quick start</a> has the current steps.</p>

<h2 class="relative group">Erasure coding instead of RAID
    <div id="erasure-coding-instead-of-raid" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#erasure-coding-instead-of-raid" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>This was the most useful concept I picked up from MinIO, and you'll meet it again in Ceph and SeaweedFS.</p>
<p>Erasure coding splits an object into data and parity shards and spreads them across drives (Reed-Solomon). In MinIO, a 4-drive set defaults to 2 parity shards: any two drives can fail completely and objects stay readable. Writes need more drives than reads, so with two drives lost in that set, reads continue but writes stop. Compared to RAID, protection works per object rather than per disk, and each shard is verified with a hash. When a drive silently corrupts data over time (bit rot), the bad shard is detected and rebuilt from parity.</p>
<p>Garage uses replication instead (typically three copies). It costs more disk but the model is simpler. Know this difference when you choose.</p>

<h2 class="relative group">Object lock and WORM
    <div id="object-lock-and-worm" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#object-lock-and-worm" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Finance, healthcare and legal work often come with rules like &quot;this document must not be deleted or modified for 5 years&quot;. S3 object lock covers that (WORM: write once, read many). It has two modes:</p>
<ul>
<li>Governance: users with a special permission can override the lock.</li>
<li>Compliance: nobody, root included, can delete the object or shorten the retention until it expires.</li>
</ul>
<p>Object lock has to be enabled when the bucket is created; you can't add it later. With boto3:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">s3</span><span class="o">.</span><span class="n">create_bucket</span><span class="p">(</span><span class="n">Bucket</span><span class="o">=</span><span class="s2">&#34;invoices&#34;</span><span class="p">,</span> <span class="n">ObjectLockEnabledForBucket</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">s3</span><span class="o">.</span><span class="n">put_object_lock_configuration</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">Bucket</span><span class="o">=</span><span class="s2">&#34;invoices&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">ObjectLockConfiguration</span><span class="o">=</span><span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;ObjectLockEnabled&#34;</span><span class="p">:</span> <span class="s2">&#34;Enabled&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;Rule&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;DefaultRetention&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;Mode&#34;</span><span class="p">:</span> <span class="s2">&#34;COMPLIANCE&#34;</span><span class="p">,</span> <span class="s2">&#34;Years&#34;</span><span class="p">:</span> <span class="mi">5</span><span class="p">}},</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span></span></span></code></pre></div></div>
<p>Try compliance mode on a test bucket before you turn it on in production. If you set the wrong retention, you can't undo it either. Also check that your storage supports object lock at all; not every S3-compatible server does.</p>

<h2 class="relative group">Lifecycle rules
    <div id="lifecycle-rules" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#lifecycle-rules" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Log and temp-file buckets that never clean themselves up will fill the disk. Lifecycle rules are part of the S3 API too:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">s3</span><span class="o">.</span><span class="n">put_bucket_lifecycle_configuration</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">Bucket</span><span class="o">=</span><span class="s2">&#34;logs&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">LifecycleConfiguration</span><span class="o">=</span><span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;Rules&#34;</span><span class="p">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">            <span class="p">{</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;ID&#34;</span><span class="p">:</span> <span class="s2">&#34;expire-30d&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;Filter&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;Prefix&#34;</span><span class="p">:</span> <span class="s2">&#34;&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;Status&#34;</span><span class="p">:</span> <span class="s2">&#34;Enabled&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;Expiration&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;Days&#34;</span><span class="p">:</span> <span class="mi">30</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;AbortIncompleteMultipartUpload&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;DaysAfterInitiation&#34;</span><span class="p">:</span> <span class="mi">7</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">            <span class="p">}</span>
</span></span><span class="line"><span class="cl">        <span class="p">]</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span></span></span></code></pre></div></div>
<p>People tend to forget the <code>AbortIncompleteMultipartUpload</code> line. Parts of abandoned multipart uploads are invisible in listings but still take up space.</p>

<h2 class="relative group">Encryption
    <div id="encryption" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#encryption" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>S3 has three models for encryption at rest:</p>
<ul>
<li>SSE-S3: the storage server manages the keys.</li>
<li>SSE-KMS: keys live in an external KMS such as Vault.</li>
<li>SSE-C: the client sends the key with every request.</li>
</ul>
<p>SSE-S3 looks like the easy option, but in MinIO even SSE-S3 needs a KMS connection (KES). On your own hardware, disk-level encryption (LUKS) plus TLS is often the most practical route.</p>

<h2 class="relative group">Common questions
    <div id="common-questions" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#common-questions" aria-label="Anchor">#</a>
    </span>
    
</h2>

<h3 class="relative group">Can I use object storage like a database?
    <div id="can-i-use-object-storage-like-a-database" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#can-i-use-object-storage-like-a-database" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>No. You can't search inside objects; you read them by key. Keep &quot;which user owns which file&quot; in your database.</p>

<h3 class="relative group">What hardware matters?
    <div id="what-hardware-matters" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-hardware-matters" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>The bottleneck is usually disk and network, not CPU. NVMe drives behind a 1 Gbps NIC will saturate the network long before the drives.</p>

<h3 class="relative group">How do I monitor it?
    <div id="how-do-i-monitor-it" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#how-do-i-monitor-it" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Scrape the storage server's Prometheus metrics: disk usage, request latency, error counts. Tools like Thanos also use object storage as long-term metric storage, which I cover in my <a href="https://blog.furkankoykiran.com.tr/en/posts/prometheus-grafana-monitoring/" >Prometheus and Grafana post</a>.</p>

<h2 class="relative group">What's next
    <div id="whats-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#whats-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>If your files still sit on the app server's disk, start by moving one upload flow to the S3 API with the boto3 code above. After that, swapping the storage behind it is as easy as changing <code>S3_ENDPOINT</code>.</p>
<p>For packaging the app side, see my <a href="https://blog.furkankoykiran.com.tr/en/posts/docker-python-app-containerization/" >Docker for Python apps post</a>. To watch the storage once it runs, the <a href="https://blog.furkankoykiran.com.tr/en/posts/prometheus-grafana-monitoring/" >Prometheus and Grafana post</a> covers the setup.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/minio-object-storage/feature_hu_60d9617d8d866c58.jpg" medium="image" type="image/jpeg"/></item><item><title>Real-Time WebSockets with FastAPI and Redis</title><link>https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/</guid><pubDate>Mon, 15 Sep 2025 15:30:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Backend</category><category>web</category><category>python</category><category>fastapi</category><category>redis</category><category>asyncio</category><description>Build a FastAPI WebSocket server, scale it across instances with Redis Pub/Sub, and handle pings, auth, Nginx proxying and client reconnects with backoff.</description><content:encoded><![CDATA[<p>Chat apps, live scores, market tickers and the dashboard of a trading bot all need the server to push data without being asked. Plain HTTP can't do that: the server only answers requests. Real-time WebSocket communication fixes it with one TCP connection that stays open, so either side can send a message at any time.</p>
<p>In this post I build a WebSocket server with FastAPI, scale it to several instances with Redis Pub/Sub, and go through the connection, auth and proxy problems that show up in production.</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/img/websocket-real-time-overview_hu_4c85937b04faaf8d.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/img/websocket-real-time-overview_hu_68aefac2116c19e4.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/img/websocket-real-time-overview_hu_b8787c21a7d20f4a.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/img/websocket-real-time-overview_hu_d54803a7025e7a72.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/img/websocket-real-time-overview_hu_b435a6f3b07cbb9c.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/img/websocket-real-time-overview_hu_6af451eff515d235.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/img/websocket-real-time-overview_hu_b435a6f3b07cbb9c.webp" alt="With HTTP the client sends a request, gets a response and the exchange ends; with WebSocket, after 101 Switching Protocols the server pushes price updates" width="1600" height="1214"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>An HTTP request ends, a WebSocket connection stays open</figcaption></figure>
<h2 class="relative group">Polling, SSE or WebSocket?
    <div id="polling-sse-or-websocket" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#polling-sse-or-websocket" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>WebSocket isn't always the right tool, so check the alternatives first:</p>
<ul>
<li>Polling: the client asks &quot;anything new?&quot; every few seconds. Simple, but most answers are empty and latency equals the polling interval.</li>
<li>Server-Sent Events (SSE): a one-way stream from server to client over plain HTTP. The browser reconnects on its own. Good enough for notifications, log tails and streaming LLM output.</li>
<li>WebSocket: two-way. Needed when the client also sends often, as in chat, games or live collaboration.</li>
</ul>
<p>My rule: if the client only listens, use SSE. If both sides talk, use WebSocket.</p>

<h2 class="relative group">A single FastAPI server
    <div id="a-single-fastapi-server" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#a-single-fastapi-server" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>FastAPI's WebSocket support (inherited from Starlette) fits naturally with async/await. Here's a simple hub that works on one server:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">fastapi</span> <span class="kn">import</span> <span class="n">FastAPI</span><span class="p">,</span> <span class="n">WebSocket</span><span class="p">,</span> <span class="n">WebSocketDisconnect</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">app</span> <span class="o">=</span> <span class="n">FastAPI</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">Hub</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">sockets</span><span class="p">:</span> <span class="nb">set</span><span class="p">[</span><span class="n">WebSocket</span><span class="p">]</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">def</span> <span class="nf">connect</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">ws</span><span class="p">:</span> <span class="n">WebSocket</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">await</span> <span class="n">ws</span><span class="o">.</span><span class="n">accept</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">sockets</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">ws</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">disconnect</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">ws</span><span class="p">:</span> <span class="n">WebSocket</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">sockets</span><span class="o">.</span><span class="n">discard</span><span class="p">(</span><span class="n">ws</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">def</span> <span class="nf">broadcast</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">for</span> <span class="n">ws</span> <span class="ow">in</span> <span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sockets</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">            <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">                <span class="k">await</span> <span class="n">ws</span><span class="o">.</span><span class="n">send_text</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">            <span class="k">except</span> <span class="ne">Exception</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">disconnect</span><span class="p">(</span><span class="n">ws</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">hub</span> <span class="o">=</span> <span class="n">Hub</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@app.websocket</span><span class="p">(</span><span class="s2">&#34;/ws/</span><span class="si">{user}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">chat</span><span class="p">(</span><span class="n">ws</span><span class="p">:</span> <span class="n">WebSocket</span><span class="p">,</span> <span class="n">user</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">await</span> <span class="n">hub</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">ws</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">text</span> <span class="o">=</span> <span class="k">await</span> <span class="n">ws</span><span class="o">.</span><span class="n">receive_text</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">            <span class="k">await</span> <span class="n">hub</span><span class="o">.</span><span class="n">broadcast</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">user</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="n">text</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="n">WebSocketDisconnect</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">hub</span><span class="o">.</span><span class="n">disconnect</span><span class="p">(</span><span class="n">ws</span><span class="p">)</span></span></span></code></pre></div></div>
<p>Run it with <code>uvicorn main:app</code>, connect from two browser tabs and messages go back and forth. The problem starts when you add a second server.</p>

<h2 class="relative group">Several servers: Redis Pub/Sub
    <div id="several-servers-redis-pubsub" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#several-servers-redis-pubsub" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A WebSocket connection is stateful. The connection object lives in the memory of the process that accepted it. With two servers behind a load balancer, Alice on server A and Bob on server B, A's <code>hub.broadcast</code> never reaches Bob.</p>
<p>You need a message bus between the servers. Redis Pub/Sub is the easiest option: each server publishes incoming messages to a Redis channel, and every server subscribes to that channel and forwards messages to its own connections.</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/img/redis-pub-sub-fan-out_hu_e97d0d5aa56ac596.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/img/redis-pub-sub-fan-out_hu_f07932af3603b3f6.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/img/redis-pub-sub-fan-out_hu_b6d032cdd7868a7d.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/img/redis-pub-sub-fan-out_hu_43ece874eda602a7.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/img/redis-pub-sub-fan-out_hu_b650987e5d8a3248.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/img/redis-pub-sub-fan-out_hu_91960453230836e1.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/img/redis-pub-sub-fan-out_hu_b650987e5d8a3248.webp" alt="Alice is on Server A and Bob on Server B; A publishes to the Redis chat channel and Redis delivers the message to both servers" width="1600" height="954"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Every server subscribes, so a message reaches everyone through Redis</figcaption></figure><p>With redis-py's asyncio API:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">asyncio</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">contextlib</span> <span class="kn">import</span> <span class="n">asynccontextmanager</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">redis.asyncio</span> <span class="k">as</span> <span class="nn">redis</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">fastapi</span> <span class="kn">import</span> <span class="n">FastAPI</span><span class="p">,</span> <span class="n">WebSocket</span><span class="p">,</span> <span class="n">WebSocketDisconnect</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">CHANNEL</span> <span class="o">=</span> <span class="s2">&#34;chat&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">r</span> <span class="o">=</span> <span class="n">redis</span><span class="o">.</span><span class="n">from_url</span><span class="p">(</span><span class="s2">&#34;redis://localhost:6379&#34;</span><span class="p">,</span> <span class="n">decode_responses</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">hub</span> <span class="o">=</span> <span class="n">Hub</span><span class="p">()</span>  <span class="c1"># the class from above</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">relay</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Listen on the Redis channel and forward every message to local sockets.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="n">pubsub</span> <span class="o">=</span> <span class="n">r</span><span class="o">.</span><span class="n">pubsub</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">await</span> <span class="n">pubsub</span><span class="o">.</span><span class="n">subscribe</span><span class="p">(</span><span class="n">CHANNEL</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">for</span> <span class="n">msg</span> <span class="ow">in</span> <span class="n">pubsub</span><span class="o">.</span><span class="n">listen</span><span class="p">():</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">msg</span><span class="p">[</span><span class="s2">&#34;type&#34;</span><span class="p">]</span> <span class="o">==</span> <span class="s2">&#34;message&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">await</span> <span class="n">hub</span><span class="o">.</span><span class="n">broadcast</span><span class="p">(</span><span class="n">msg</span><span class="p">[</span><span class="s2">&#34;data&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@asynccontextmanager</span>
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">lifespan</span><span class="p">(</span><span class="n">app</span><span class="p">:</span> <span class="n">FastAPI</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">task</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">create_task</span><span class="p">(</span><span class="n">relay</span><span class="p">())</span>
</span></span><span class="line"><span class="cl">    <span class="k">yield</span>
</span></span><span class="line"><span class="cl">    <span class="n">task</span><span class="o">.</span><span class="n">cancel</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="k">await</span> <span class="n">r</span><span class="o">.</span><span class="n">aclose</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">app</span> <span class="o">=</span> <span class="n">FastAPI</span><span class="p">(</span><span class="n">lifespan</span><span class="o">=</span><span class="n">lifespan</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@app.websocket</span><span class="p">(</span><span class="s2">&#34;/ws/</span><span class="si">{user}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">chat</span><span class="p">(</span><span class="n">ws</span><span class="p">:</span> <span class="n">WebSocket</span><span class="p">,</span> <span class="n">user</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">await</span> <span class="n">hub</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">ws</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="n">text</span> <span class="o">=</span> <span class="k">await</span> <span class="n">ws</span><span class="o">.</span><span class="n">receive_text</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">            <span class="k">await</span> <span class="n">r</span><span class="o">.</span><span class="n">publish</span><span class="p">(</span><span class="n">CHANNEL</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">user</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="n">text</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>  <span class="c1"># no local broadcast</span>
</span></span><span class="line"><span class="cl">    <span class="k">except</span> <span class="n">WebSocketDisconnect</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">hub</span><span class="o">.</span><span class="n">disconnect</span><span class="p">(</span><span class="n">ws</span><span class="p">)</span></span></span></code></pre></div></div>
<p>The endpoint no longer broadcasts locally; it publishes to Redis. The message comes back through Redis to every server, including the one that sent it.</p>
<p>Pub/Sub has a catch: it's fire and forget. A server that isn't subscribed at that moment (say, it's restarting) never sees the message, and Redis doesn't keep it. If you need chat history, also write messages to a database or to Redis Streams. I cover other Redis patterns in my <a href="https://blog.furkankoykiran.com.tr/en/posts/redis-caching-session-management/" >Redis post</a>.</p>
<p>The <code>broadcast</code> loop also sends to sockets one after another, so one very slow client can hold up the rest. With thousands of connections, give each socket its own send queue or wrap sends in <code>asyncio.wait_for</code> with a timeout.</p>

<h2 class="relative group">Dropped and zombie connections
    <div id="dropped-and-zombie-connections" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#dropped-and-zombie-connections" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Mobile users go through tunnels and switch networks. Worse, a client can vanish without its TCP close ever reaching the server, and the server keeps treating the connection as live.</p>
<p>Protocol-level ping/pong solves that, and uvicorn does it for you. By default it sends a ping every 20 seconds and closes the connection if no pong arrives within 20 seconds (<code>--ws-ping-interval</code> and <code>--ws-ping-timeout</code>). Browsers answer pings automatically. You don't write anything in JavaScript for this, and the browser WebSocket API doesn't let you send pings anyway.</p>
<p>You only need an application-level &quot;are you there?&quot; message if you want to know whether the user is active.</p>

<h2 class="relative group">Authentication
    <div id="authentication" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#authentication" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The browser <code>WebSocket</code> API can't set custom headers, so <code>Authorization: Bearer ...</code> is out. In practice you have three options:</p>
<ol>
<li>A cookie, if you're on the same site. The handshake is a normal HTTP request, so cookies are sent automatically.</li>
<li>A short-lived ticket in the query string. Don't put a long-lived JWT in the URL, since URLs end up in proxy and access logs. Fetch a single-use ticket valid for 30 seconds from a normal HTTP endpoint first, then send it when connecting.</li>
<li>Send the token in the first message after connecting and accept nothing else until it checks out.</li>
</ol>
<p>Whichever you pick, also check the <code>Origin</code> header. Browsers don't apply CORS to WebSockets, so with cookie auth another site can open a connection as your user (cross-site WebSocket hijacking).</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">fastapi</span> <span class="kn">import</span> <span class="n">Query</span><span class="p">,</span> <span class="n">status</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">ALLOWED_ORIGINS</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;https://app.example.com&#34;</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@app.websocket</span><span class="p">(</span><span class="s2">&#34;/ws&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">secure_ws</span><span class="p">(</span><span class="n">ws</span><span class="p">:</span> <span class="n">WebSocket</span><span class="p">,</span> <span class="n">ticket</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="n">Query</span><span class="p">(</span><span class="o">...</span><span class="p">))</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">ws</span><span class="o">.</span><span class="n">headers</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;origin&#34;</span><span class="p">)</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">ALLOWED_ORIGINS</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">await</span> <span class="n">ws</span><span class="o">.</span><span class="n">close</span><span class="p">(</span><span class="n">code</span><span class="o">=</span><span class="n">status</span><span class="o">.</span><span class="n">WS_1008_POLICY_VIOLATION</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span>
</span></span><span class="line"><span class="cl">    <span class="n">user</span> <span class="o">=</span> <span class="k">await</span> <span class="n">redeem_ticket</span><span class="p">(</span><span class="n">ticket</span><span class="p">)</span>  <span class="c1"># validate and delete the ticket</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">user</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">await</span> <span class="n">ws</span><span class="o">.</span><span class="n">close</span><span class="p">(</span><span class="n">code</span><span class="o">=</span><span class="n">status</span><span class="o">.</span><span class="n">WS_1008_POLICY_VIOLATION</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span>
</span></span><span class="line"><span class="cl">    <span class="k">await</span> <span class="n">hub</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">ws</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">...</span></span></span></code></pre></div></div>
<p><code>redeem_ticket</code> is yours to write; reading and deleting the ticket with Redis <code>GETDEL</code> is a simple way. Closing before <code>accept</code> makes the client's handshake fail with a 403.</p>

<h2 class="relative group">Nginx and load balancers
    <div id="nginx-and-load-balancers" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#nginx-and-load-balancers" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>WebSocket starts with the HTTP/1.1 <code>Upgrade</code> mechanism. Nginx doesn't forward those headers by default and closes idle proxied connections after 60 seconds:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-nginx" data-lang="nginx"><span class="line"><span class="cl"><span class="k">location</span> <span class="s">/ws</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="kn">proxy_pass</span> <span class="s">http://app_servers</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="kn">proxy_http_version</span> <span class="mi">1</span><span class="s">.1</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="kn">proxy_set_header</span> <span class="s">Upgrade</span> <span class="nv">$http_upgrade</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="kn">proxy_set_header</span> <span class="s">Connection</span> <span class="s">&#34;upgrade&#34;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="kn">proxy_set_header</span> <span class="s">Host</span> <span class="nv">$host</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="kn">proxy_read_timeout</span> <span class="s">1h</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>With plain WebSockets you don't need sticky sessions: the handshake is a single HTTP request and the connection stays on that server afterwards. Libraries like Socket.IO that can fall back to long polling are different; their requests must reach the same server, so you need <code>ip_hash</code> or cookie-based routing. More on that in my <a href="https://blog.furkankoykiran.com.tr/en/posts/nginx-load-balancing-caching/" >Nginx load balancing post</a>.</p>
<p>On capacity: an idle connection costs little CPU, and one async process can hold tens of thousands of them. File descriptor limits (<code>ulimit -n</code>) and memory run out first, so raise the limits and load-test to find your own number.</p>

<h2 class="relative group">Client: reconnect with exponential backoff
    <div id="client-reconnect-with-exponential-backoff" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#client-reconnect-with-exponential-backoff" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>However solid the server is, connections will drop. Retrying every 3 seconds means that when you redeploy, every client hits the server at the same moment. Double the delay each time and add jitter:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-javascript" data-lang="javascript"><span class="line"><span class="cl"><span class="kd">function</span> <span class="nx">connect</span><span class="p">(</span><span class="nx">url</span><span class="p">,</span> <span class="nx">onMessage</span><span class="p">,</span> <span class="nx">attempt</span> <span class="o">=</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">ws</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">WebSocket</span><span class="p">(</span><span class="nx">url</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="nx">ws</span><span class="p">.</span><span class="nx">onopen</span> <span class="o">=</span> <span class="p">()</span> <span class="p">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nx">attempt</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="p">};</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="nx">ws</span><span class="p">.</span><span class="nx">onmessage</span> <span class="o">=</span> <span class="p">(</span><span class="nx">event</span><span class="p">)</span> <span class="p">=&gt;</span> <span class="nx">onMessage</span><span class="p">(</span><span class="nx">event</span><span class="p">.</span><span class="nx">data</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="nx">ws</span><span class="p">.</span><span class="nx">onclose</span> <span class="o">=</span> <span class="p">(</span><span class="nx">event</span><span class="p">)</span> <span class="p">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="p">(</span><span class="nx">event</span><span class="p">.</span><span class="nx">code</span> <span class="o">===</span> <span class="mi">1008</span><span class="p">)</span> <span class="k">return</span><span class="p">;</span> <span class="c1">// auth failure: don&#39;t retry
</span></span></span><span class="line"><span class="cl">    <span class="kr">const</span> <span class="nx">base</span> <span class="o">=</span> <span class="nb">Math</span><span class="p">.</span><span class="nx">min</span><span class="p">(</span><span class="mi">30000</span><span class="p">,</span> <span class="mi">1000</span> <span class="o">*</span> <span class="mi">2</span> <span class="o">**</span> <span class="nx">attempt</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="kr">const</span> <span class="nx">delay</span> <span class="o">=</span> <span class="nx">base</span> <span class="o">/</span> <span class="mi">2</span> <span class="o">+</span> <span class="nb">Math</span><span class="p">.</span><span class="nx">random</span><span class="p">()</span> <span class="o">*</span> <span class="p">(</span><span class="nx">base</span> <span class="o">/</span> <span class="mi">2</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="nx">setTimeout</span><span class="p">(()</span> <span class="p">=&gt;</span> <span class="nx">connect</span><span class="p">(</span><span class="nx">url</span><span class="p">,</span> <span class="nx">onMessage</span><span class="p">,</span> <span class="nx">attempt</span> <span class="o">+</span> <span class="mi">1</span><span class="p">),</span> <span class="nx">delay</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="p">};</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">return</span> <span class="nx">ws</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nx">connect</span><span class="p">(</span><span class="s2">&#34;wss://app.example.com/ws?ticket=...&#34;</span><span class="p">,</span> <span class="p">(</span><span class="nx">data</span><span class="p">)</span> <span class="p">=&gt;</span> <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">data</span><span class="p">));</span></span></span></code></pre></div></div>
<p>In a real app you'd fetch a fresh ticket before each reconnect, since tickets are single-use. Also note that the browser only sees code 1008 if the server accepted the connection before closing it. A handshake rejected with 403 shows up as 1006, so put a cap on the number of attempts too.</p>

<h2 class="relative group">Where to go next
    <div id="where-to-go-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-to-go-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Run the Redis version on two ports (<code>uvicorn main:app --port 8001</code> and <code>--port 8002</code>), connect a tab to each, and watch messages cross between servers. A good next step is per-connection <a href="https://blog.furkankoykiran.com.tr/en/posts/python-rate-limiting-api-throttling/" >rate limiting</a> against buggy or abusive clients. The <a href="https://fastapi.tiangolo.com/advanced/websockets/"  target="_blank" rel="noreferrer">FastAPI WebSocket docs</a> and <a href="https://uvicorn.dev/settings/"  target="_blank" rel="noreferrer">uvicorn settings</a> cover the rest.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/websocket-real-time-communication/feature_hu_a0b7bae027917560.jpg" medium="image" type="image/jpeg"/></item><item><title>Prometheus and Grafana Monitoring: Production Notes</title><link>https://blog.furkankoykiran.com.tr/en/posts/prometheus-grafana-monitoring/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/prometheus-grafana-monitoring/</guid><pubDate>Mon, 01 Sep 2025 14:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>DevOps</category><category>monitoring</category><category>devops</category><category>kubernetes</category><category>python</category><description>Prometheus and Grafana monitoring in practice: avoiding cardinality blowups, PromQL for the golden signals, recording rules, Alertmanager routing, Thanos.</description><content:encoded><![CDATA[<p>Getting Prometheus and Grafana monitoring running takes a Helm chart and ten minutes. The hard part comes later: deciding which metrics to collect, which labels you must never add, and who gets paged at 3 a.m. I'm skipping the install steps here. These are the rules, PromQL patterns and failure modes I keep coming back to when I monitor my own services.</p>

<h2 class="relative group">How the pieces fit together
    <div id="how-the-pieces-fit-together" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#how-the-pieces-fit-together" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Prometheus pulls. It scrapes each target's <code>/metrics</code> endpoint on an interval, evaluates rules, and sends firing alerts to Alertmanager. Grafana is only the visualization layer: it reads from Prometheus by running PromQL queries.</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/prometheus-grafana-monitoring/img/pull-model_hu_617253ccdfa43417.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/prometheus-grafana-monitoring/img/pull-model_hu_3a8e6bee4efb4833.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/prometheus-grafana-monitoring/img/pull-model_hu_26344719af339d05.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/prometheus-grafana-monitoring/img/pull-model_hu_5ef04885fb8aa94a.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/prometheus-grafana-monitoring/img/pull-model_hu_f8600f62f2edcd0e.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/prometheus-grafana-monitoring/img/pull-model_hu_2aefcf41596656e6.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/prometheus-grafana-monitoring/img/pull-model_hu_f8600f62f2edcd0e.webp" alt="Three steps: Prometheus scrapes /metrics on an interval, evaluates rules and sends alerts to Alertmanager, and Grafana reads the data with PromQL" width="1600" height="986"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Grafana only reads; Prometheus collects the data and fires the alerts</figcaption></figure><p>Every step above is a place where something breaks in production. The rest of this post roughly follows it.</p>

<h2 class="relative group">Trap number one: cardinality explosions
    <div id="trap-number-one-cardinality-explosions" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#trap-number-one-cardinality-explosions" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>In Prometheus, every unique label combination is its own time series. <code>http_requests_total{method=&quot;GET&quot;, route=&quot;/users/:id&quot;, status=&quot;200&quot;}</code> is one series. Put an unbounded value into a label (a user ID, the raw URL path, an email, a User-Agent string) and your series count grows with your user base. Prometheus keeps active series in memory, so eventually it gets OOM-killed. Adding RAM only buys time.</p>
<p>The rule: label values must come from a bounded set. Use the route template, never the raw path. In FastAPI the template is already sitting in the request scope:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">time</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">fastapi</span> <span class="kn">import</span> <span class="n">FastAPI</span><span class="p">,</span> <span class="n">Request</span>
</span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">prometheus_client</span> <span class="kn">import</span> <span class="n">Counter</span><span class="p">,</span> <span class="n">Histogram</span><span class="p">,</span> <span class="n">make_asgi_app</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">REQUESTS</span> <span class="o">=</span> <span class="n">Counter</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;http_requests_total&#34;</span><span class="p">,</span> <span class="s2">&#34;HTTP requests&#34;</span><span class="p">,</span> <span class="p">[</span><span class="s2">&#34;method&#34;</span><span class="p">,</span> <span class="s2">&#34;route&#34;</span><span class="p">,</span> <span class="s2">&#34;status&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">LATENCY</span> <span class="o">=</span> <span class="n">Histogram</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;http_request_duration_seconds&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;Request duration&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">[</span><span class="s2">&#34;method&#34;</span><span class="p">,</span> <span class="s2">&#34;route&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">    <span class="n">buckets</span><span class="o">=</span><span class="p">(</span><span class="mf">0.01</span><span class="p">,</span> <span class="mf">0.025</span><span class="p">,</span> <span class="mf">0.05</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">,</span> <span class="mf">0.25</span><span class="p">,</span> <span class="mf">0.5</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mf">2.5</span><span class="p">,</span> <span class="mi">5</span><span class="p">),</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">app</span> <span class="o">=</span> <span class="n">FastAPI</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"><span class="n">app</span><span class="o">.</span><span class="n">mount</span><span class="p">(</span><span class="s2">&#34;/metrics&#34;</span><span class="p">,</span> <span class="n">make_asgi_app</span><span class="p">())</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nd">@app.middleware</span><span class="p">(</span><span class="s2">&#34;http&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">prometheus_middleware</span><span class="p">(</span><span class="n">request</span><span class="p">:</span> <span class="n">Request</span><span class="p">,</span> <span class="n">call_next</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">start</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">perf_counter</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">response</span> <span class="o">=</span> <span class="k">await</span> <span class="n">call_next</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">route</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">scope</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;route&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">path</span> <span class="o">=</span> <span class="n">route</span><span class="o">.</span><span class="n">path</span> <span class="k">if</span> <span class="n">route</span> <span class="k">else</span> <span class="s2">&#34;unmatched&#34;</span>  <span class="c1"># /users/{user_id}, not 42</span>
</span></span><span class="line"><span class="cl">    <span class="n">REQUESTS</span><span class="o">.</span><span class="n">labels</span><span class="p">(</span><span class="n">request</span><span class="o">.</span><span class="n">method</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">status_code</span><span class="p">))</span><span class="o">.</span><span class="n">inc</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="n">LATENCY</span><span class="o">.</span><span class="n">labels</span><span class="p">(</span><span class="n">request</span><span class="o">.</span><span class="n">method</span><span class="p">,</span> <span class="n">path</span><span class="p">)</span><span class="o">.</span><span class="n">observe</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">perf_counter</span><span class="p">()</span> <span class="o">-</span> <span class="n">start</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">response</span></span></span></code></pre></div></div>
<p>The <code>unmatched</code> bucket matters too. Without it, a bot scanning for 404s creates a new series for every URL it tries.</p>
<p>When a Prometheus server is already bloated, the fastest way to find the culprit is <strong>Status → TSDB Status</strong> in the UI. The capture below comes from the Prometheus project's public demo server. Histogram <code>_bucket</code> series top the list, because every bucket boundary (<code>le</code>) is a series of its own.</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/posts/prometheus-grafana-ile-izleme/img/prometheus-tsdb-status-demo_hu_5aed3844a506b447.avif 480w, https://blog.furkankoykiran.com.tr/posts/prometheus-grafana-ile-izleme/img/prometheus-tsdb-status-demo_hu_71e376c2207088dd.avif 800w, https://blog.furkankoykiran.com.tr/posts/prometheus-grafana-ile-izleme/img/prometheus-tsdb-status-demo_hu_78a7a1e60fc4beea.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/posts/prometheus-grafana-ile-izleme/img/prometheus-tsdb-status-demo_hu_3cd365c13f72b6b.webp 480w, https://blog.furkankoykiran.com.tr/posts/prometheus-grafana-ile-izleme/img/prometheus-tsdb-status-demo_hu_b86083f2b286020e.webp 800w, https://blog.furkankoykiran.com.tr/posts/prometheus-grafana-ile-izleme/img/prometheus-tsdb-status-demo_hu_c91d1a03dbc77599.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/posts/prometheus-grafana-ile-izleme/img/prometheus-tsdb-status-demo_hu_b86083f2b286020e.webp" alt="TSDB Status page of the Prometheus demo server: 16,949 series, with grafana_http_response_size_bytes_bucket at the top of the per-metric list" width="1300" height="1672"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>The TSDB Status page on the public Prometheus demo server</figcaption></figure><p>You can get the same answer with a query:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="k">topk</span><span class="o">(</span><span class="mi">10</span><span class="p">,</span><span class="w"> </span><span class="k">count</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="o">(</span><span class="nv">__name__</span><span class="o">)</span><span class="w"> </span><span class="o">(</span><span class="p">{</span><span class="nl">__name__</span><span class="o">=~</span><span class="p">&#34;</span><span class="s">.+</span><span class="p">&#34;}</span><span class="o">))</span></span></span></code></pre></div></div>
<p>That query is expensive. Don't run it on a large server at peak time.</p>

<h2 class="relative group">What to watch: the four golden signals
    <div id="what-to-watch-the-four-golden-signals" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-to-watch-the-four-golden-signals" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The <a href="https://sre.google/sre-book/monitoring-distributed-systems/"  target="_blank" rel="noreferrer">four golden signals from the Google SRE book</a> are a good default for every service. Get these right before you collect a hundred random metrics.</p>
<p>Traffic, the load on the system:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="k">sum</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="o">(</span><span class="nv">route</span><span class="o">)</span><span class="w"> </span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">http_requests_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span></span></span></code></pre></div></div>
<p>Error ratio. A 4xx is usually the client's fault, so alert on 5xx:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">http_requests_total</span><span class="p">{</span><span class="nl">status</span><span class="o">=~</span><span class="p">&#34;</span><span class="s">5..</span><span class="p">&#34;}[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="o">/</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">sum</span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">http_requests_total</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span></span></span></code></pre></div></div>
<p>Latency. Averages hide your slowest requests. Look at P95 and P99; P99 is what your unhappiest 1% of users experience:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="kr">histogram_quantile</span><span class="o">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="mf">0.99</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">sum</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="o">(</span><span class="nv">le</span><span class="p">,</span><span class="w"> </span><span class="nv">route</span><span class="o">)</span><span class="w"> </span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">http_request_duration_seconds_bucket</span><span class="p">[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="o">)</span></span></span></code></pre></div></div>
<p>Saturation, how full your resources are. CPU is only part of it: connection pools, thread pools, disk I/O and queue depth belong here too, and that's where the sneakiest problems hide.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-promql" data-lang="promql"><span class="line"><span class="cl"><span class="mi">1</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="k">avg</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="o">(</span><span class="nv">instance</span><span class="o">)</span><span class="w"> </span><span class="o">(</span><span class="kr">rate</span><span class="o">(</span><span class="nv">node_cpu_seconds_total</span><span class="p">{</span><span class="nl">mode</span><span class="o">=</span><span class="p">&#34;</span><span class="s">idle</span><span class="p">&#34;}[</span><span class="s">5m</span><span class="p">]</span><span class="o">))</span></span></span></code></pre></div></div>

<h2 class="relative group">rate vs irate vs increase
    <div id="rate-vs-irate-vs-increase" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#rate-vs-irate-vs-increase" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>All three work on counters and handle counter resets when a process restarts.</p>
<ul>
<li><code>rate()</code> gives the per-second average increase over the window. Use it by default for dashboards and alerts.</li>
<li><code>irate()</code> only looks at the last two samples. It shows short spikes, but the graph turns into a saw blade. Keep it out of alerts.</li>
<li><code>increase()</code> is the total increase over the window. It answers &quot;how many errors did we get in the last hour?&quot;</li>
</ul>
<p>Make the window at least four times the scrape interval. With a 30-second scrape interval, a <code>[1m]</code> window sometimes contains a single sample and <code>rate()</code> returns nothing.</p>
<p>If the same heavy expression runs in ten dashboard panels and three alerts, write a recording rule. Prometheus precomputes it and stores the result as a new series:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">groups</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">api-recording</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">rules</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">record</span><span class="p">:</span><span class="w"> </span><span class="l">route:http_requests:rate5m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="l">sum by (route) (rate(http_requests_total[5m]))</span></span></span></code></pre></div></div>

<h2 class="relative group">Alerting without burning out on-call
    <div id="alerting-without-burning-out-on-call" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#alerting-without-burning-out-on-call" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The most common mistake is alerting on everything: CPU at 80%, RAM at 70%, disk at 85%. After a hundred notifications a day, people treat all of them as noise, and the one &quot;database down&quot; alert gets lost.</p>
<p>The rules I follow:</p>
<ul>
<li>Alert on symptoms, not causes. CPU at 99% with 50 ms latency and no errors means users are fine. CPU at 10% with P99 over two seconds means they aren't.</li>
<li>Page a human only when something needs action now. A disk that fills up in three days can be a Slack message or a ticket.</li>
<li>Use <code>for:</code>. A spike seen in a single scrape shouldn't wake anyone.</li>
</ul>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">groups</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">api-alerts</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">rules</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">HighErrorRate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">expr</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">          sum(rate(http_requests_total{status=~&#34;5..&#34;}[5m]))
</span></span></span><span class="line"><span class="cl"><span class="sd">            / sum(rate(http_requests_total[5m])) &gt; 0.05</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">for</span><span class="p">:</span><span class="w"> </span><span class="l">10m</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">severity</span><span class="p">:</span><span class="w"> </span><span class="l">page</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">annotations</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">summary</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;5xx ratio above 5% for 10 minutes&#34;</span></span></span></code></pre></div></div>
<p>Routing and inhibition in Alertmanager are the other half. When the database is down, you don't need fifty &quot;can't connect&quot; alerts from the services behind it:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">route</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">receiver</span><span class="p">:</span><span class="w"> </span><span class="l">slack-warnings</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">group_by</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&#34;alertname&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;cluster&#34;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">routes</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">matchers</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s1">&#39;severity=&#34;page&#34;&#39;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">receiver</span><span class="p">:</span><span class="w"> </span><span class="l">pagerduty</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">inhibit_rules</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">source_matchers</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s1">&#39;alertname=&#34;DatabaseDown&#34;&#39;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">target_matchers</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s1">&#39;severity=&#34;warning&#34;&#39;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">equal</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&#34;cluster&#34;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c"># receivers section (slack_configs, pagerduty_configs) omitted</span></span></span></code></pre></div></div>
<p>Then there's the question of who monitors the monitoring. Define an always-firing <code>Watchdog</code> alert and send it to an external dead man's switch. If it stops arriving, Prometheus or Alertmanager is down.</p>

<h2 class="relative group">Service discovery on Kubernetes
    <div id="service-discovery-on-kubernetes" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#service-discovery-on-kubernetes" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Pods come and go, so hardcoding IPs is out. <code>kubernetes_sd_configs</code> watches the Kubernetes API and finds targets for you. You still don't want to scrape every pod in the cluster, so keep only the ones that opt in:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">scrape_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="l">kubernetes-pods</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">kubernetes_sd_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">role</span><span class="p">:</span><span class="w"> </span><span class="l">pod</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">relabel_configs</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span>- <span class="nt">source_labels</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">__meta_kubernetes_pod_annotation_prometheus_io_scrape]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">action</span><span class="p">:</span><span class="w"> </span><span class="l">keep</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">regex</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;true&#34;</span></span></span></code></pre></div></div>
<p>The <code>prometheus.io/scrape</code> annotation is a community convention, not a built-in Prometheus feature; the relabel rule above is what gives it meaning. If you run the Prometheus Operator, you do the same thing with <code>ServiceMonitor</code> and <code>PodMonitor</code> resources, which is what most teams pick today.</p>

<h2 class="relative group">Exporters worth installing
    <div id="exporters-worth-installing" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#exporters-worth-installing" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Don't stop at <code>node_exporter</code>. The ones I find most useful:</p>
<ul>
<li><code>blackbox_exporter</code> probes your service from the outside over HTTP, TCP, DNS or ICMP. It tells you whether the endpoint is reachable and how many days your TLS certificate has left.</li>
<li><code>postgres_exporter</code> exposes active connections, locks, cache hit ratio and dead tuples. Watching a database only as a process that uses CPU and RAM isn't enough.</li>
<li>The JMX exporter for JVM services: heap usage and GC pause times.</li>
</ul>
<p>For metrics inside your own app you don't need an exporter at all. The official client library for your language (such as <code>prometheus_client</code>) is enough.</p>

<h2 class="relative group">Long-term storage: Thanos, Mimir, VictoriaMetrics
    <div id="long-term-storage-thanos-mimir-victoriametrics" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#long-term-storage-thanos-mimir-victoriametrics" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Prometheus stores data on local disk and deletes it once retention runs out, usually after 15 to 30 days. On its own it can't answer &quot;compare this week to the same week last year&quot;.</p>
<p>The setup I see most often is Thanos in sidecar mode. The sidecar runs next to Prometheus and uploads each two-hour TSDB block to S3-compatible object storage. If you want to self-host that storage, my <a href="https://blog.furkankoykiran.com.tr/en/posts/minio-object-storage/" >MinIO and object storage post</a> covers it, including what changed with MinIO's community edition. Thanos gives you:</p>
<ul>
<li>Cheap, practically unlimited retention in object storage.</li>
<li>One query layer across Prometheus servers in several clusters.</li>
<li>Downsampling: old data kept at 5-minute or 1-hour resolution, so year-long queries stay fast.</li>
</ul>
<p>Grafana Mimir and VictoriaMetrics solve the same problem through remote write. For a small single-server setup I find VictoriaMetrics easier to run; for large multi-cluster setups Thanos or Mimir are more common.</p>
<p>The Prometheus 3.x line brought a new UI, UTF-8 metric and label names, OTLP ingestion and native histograms. A service instrumented with the OpenTelemetry SDK can now push metrics straight into Prometheus. Flags change between releases, so check the <a href="https://prometheus.io/docs/prometheus/latest/feature_flags/"  target="_blank" rel="noreferrer">feature flags page</a> for your version.</p>

<h2 class="relative group">Three problems you'll meet in the field
    <div id="three-problems-youll-meet-in-the-field" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#three-problems-youll-meet-in-the-field" aria-label="Anchor">#</a>
    </span>
    
</h2>

<h3 class="relative group">&quot;context deadline exceeded&quot;
    <div id="context-deadline-exceeded" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#context-deadline-exceeded" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Prometheus can't get a response from <code>/metrics</code> within the scrape timeout. The usual cause is a metrics endpoint that runs a database query on every scrape. Metrics should be read from memory. Raising the timeout just hides the problem.</p>

<h3 class="relative group">Target is &quot;down&quot; but the service is up
    <div id="target-is-down-but-the-service-is-up" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#target-is-down-but-the-service-is-up" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Usually a NetworkPolicy or a security group. Test it by hand from inside the Prometheus pod. The official Prometheus image is busybox-based, so there's no <code>curl</code>, but <code>wget</code> works:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">kubectl <span class="nb">exec</span> -it &lt;prometheus-pod&gt; -- wget -qO- http://my-app-service:8000/metrics</span></span></code></pre></div></div>
<p>If you can't reach it from there, neither can Prometheus.</p>

<h3 class="relative group">Gaps in graphs
    <div id="gaps-in-graphs" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#gaps-in-graphs" aria-label="Anchor">#</a>
    </span>
    
</h3>
<p>Either the Prometheus server is overloaded (CPU throttling) or scrapes take almost as long as the scrape interval. If <code>rate(prometheus_tsdb_head_samples_appended_total[5m])</code> is jumpy, it's time to scale Prometheus up or shard it.</p>

<h2 class="relative group">Grafana habits that pay off
    <div id="grafana-habits-that-pay-off" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#grafana-habits-that-pay-off" aria-label="Anchor">#</a>
    </span>
    
</h2>
<ul>
<li>Variables: don't draw one panel per server. With <code>instance</code>, <code>cluster</code> or <code>service</code> variables, a single dashboard covers hundreds of targets.</li>
<li>Annotations: mark deploys as vertical lines on your graphs. The answer to &quot;why did CPU jump?&quot; is often that line. Your CI can add one with a single call to the Grafana API.</li>
<li>Data links: clicking an error spike should open your logs filtered to the same time range. This works much better when logs are structured JSON, which I cover in my <a href="https://blog.furkankoykiran.com.tr/en/posts/python-logging-best-practices/" >Python logging post</a>.</li>
<li>Keep dashboards as JSON in git and load them through provisioning. A dashboard that only exists in the UI will get lost one day.</li>
</ul>

<h2 class="relative group">Pre-production checklist
    <div id="pre-production-checklist" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#pre-production-checklist" aria-label="Anchor">#</a>
    </span>
    
</h2>
<ul>
<li>Are <code>--storage.tsdb.retention.time</code> and <code>--storage.tsdb.retention.size</code> set for your disk? If the disk fills up, Prometheus stops.</li>
<li>Does Alertmanager run at least two replicas with <code>--cluster.peer</code>?</li>
<li>Do critical alerts page a phone while the rest go to a channel?</li>
<li>Is the scrape interval sensible? 15 or 30 seconds is enough for most workloads.</li>
<li>Is the Watchdog alert wired to something outside the cluster?</li>
<li>For services that use queues, are queue depth and wait time monitored? My <a href="https://blog.furkankoykiran.com.tr/en/posts/celery-arq-async-task-queue/" >Celery vs Arq post</a> lists the metrics I watch.</li>
</ul>

<h2 class="relative group">What's next
    <div id="whats-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#whats-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>To start, pick one service, add the middleware above, build one panel per golden signal and a single <code>HighErrorRate</code> alert. Tuning thresholds after a week of real alerts teaches you more than writing twenty alerts up front. The <a href="https://prometheus.io/docs/"  target="_blank" rel="noreferrer">Prometheus docs</a> and the <a href="https://prometheus.io/docs/alerting/latest/configuration/"  target="_blank" rel="noreferrer">Alertmanager configuration reference</a> are the places to go next.</p>
<p>If you want logs next to those metrics, the <a href="https://blog.furkankoykiran.com.tr/en/posts/python-logging-best-practices/" >Python logging post</a> is a good next step. For self-hosted long-term storage, see the <a href="https://blog.furkankoykiran.com.tr/en/posts/minio-object-storage/" >MinIO post</a>.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/prometheus-grafana-monitoring/feature_hu_1c8cfe5c0c9d1955.jpg" medium="image" type="image/jpeg"/></item><item><title>RabbitMQ Message Queues with aio-pika: Reliable Setup</title><link>https://blog.furkankoykiran.com.tr/en/posts/rabbitmq-message-queue/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/rabbitmq-message-queue/</guid><pubDate>Mon, 18 Aug 2025 10:30:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Backend</category><category>messaging</category><category>python</category><category>asyncio</category><category>monitoring</category><description>How to stop losing messages in RabbitMQ: prefetch, publisher confirms, quorum queues and dead letter exchanges, with working Python aio-pika examples.</description><content:encoded><![CDATA[<p>A RabbitMQ message queue looks simple: one service publishes, another consumes. The questions that matter in production are different. Does a message survive a broker restart? Who picks it up if a consumer crashes mid-task? Can one bad message block the queue forever?</p>
<p>This post covers the settings I use when I run RabbitMQ from Python with <code>aio-pika</code>. The examples target RabbitMQ 4.x and aio-pika 9.x.</p>

<h2 class="relative group">The shape: exchange, queue and DLX
    <div id="the-shape-exchange-queue-and-dlx" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-shape-exchange-queue-and-dlx" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Producers don't publish to queues. They publish to an exchange, which looks at the routing key and bindings and copies the message into one or more queues. Messages that can't be processed go through a dead letter exchange (DLX) into a separate queue.</p>
<p>Pick the exchange type for the job:</p>
<ul>
<li><code>direct</code>: the routing key must match exactly. Enough for plain work distribution.</li>
<li><code>fanout</code>: the routing key is ignored and every bound queue gets a copy. Useful when both the email service and analytics need &quot;user signed up&quot;.</li>
<li><code>topic</code>: matches patterns like <code>order.created</code> or <code>order.*.eu</code>, where <code>*</code> is one word and <code>#</code> is zero or more. I start most projects with this one.</li>
<li><code>headers</code>: routes on header values instead of the key. I've rarely needed it.</li>
</ul>

<h2 class="relative group">Four settings that stop message loss
    <div id="four-settings-that-stop-message-loss" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#four-settings-that-stop-message-loss" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>They only protect you when all four are on:</p>
<ol>
<li>Declare the queue with <code>durable=True</code> so its definition survives a broker restart.</li>
<li>Publish with <code>delivery_mode=PERSISTENT</code> so the message is written to disk. Quorum queues persist messages anyway, but classic queues need this.</li>
<li>Turn on publisher confirms. Without them, <code>publish</code> returns and you have no idea whether the broker actually took the message. In aio-pika, channels have confirms on by default and <code>await exchange.publish(...)</code> waits for the broker's ack.</li>
<li>Don't use automatic acks on the consumer. With <code>no_ack=True</code>, the message is deleted the moment it reaches the consumer, so a crash mid-task loses it. Ack after the work is done.</li>
</ol>

<h2 class="relative group">Publishing with aio-pika
    <div id="publishing-with-aio-pika" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#publishing-with-aio-pika" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Don't open a connection per message. Open one at startup and keep it; <code>connect_robust</code> reconnects on its own after a network drop.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">asyncio</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">aio_pika</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">AMQP_URL</span> <span class="o">=</span> <span class="s2">&#34;amqp://app:secret@localhost/&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">main</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">connection</span> <span class="o">=</span> <span class="k">await</span> <span class="n">aio_pika</span><span class="o">.</span><span class="n">connect_robust</span><span class="p">(</span><span class="n">AMQP_URL</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">with</span> <span class="n">connection</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">channel</span> <span class="o">=</span> <span class="k">await</span> <span class="n">connection</span><span class="o">.</span><span class="n">channel</span><span class="p">()</span>  <span class="c1"># publisher confirms are on by default</span>
</span></span><span class="line"><span class="cl">        <span class="n">exchange</span> <span class="o">=</span> <span class="k">await</span> <span class="n">channel</span><span class="o">.</span><span class="n">declare_exchange</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;orders&#34;</span><span class="p">,</span> <span class="n">aio_pika</span><span class="o">.</span><span class="n">ExchangeType</span><span class="o">.</span><span class="n">TOPIC</span><span class="p">,</span> <span class="n">durable</span><span class="o">=</span><span class="kc">True</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">order</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;id&#34;</span><span class="p">:</span> <span class="mi">42</span><span class="p">,</span> <span class="s2">&#34;total&#34;</span><span class="p">:</span> <span class="mf">199.90</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">        <span class="k">await</span> <span class="n">exchange</span><span class="o">.</span><span class="n">publish</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="n">aio_pika</span><span class="o">.</span><span class="n">Message</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">                <span class="n">body</span><span class="o">=</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">order</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(),</span>
</span></span><span class="line"><span class="cl">                <span class="n">content_type</span><span class="o">=</span><span class="s2">&#34;application/json&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                <span class="n">delivery_mode</span><span class="o">=</span><span class="n">aio_pika</span><span class="o">.</span><span class="n">DeliveryMode</span><span class="o">.</span><span class="n">PERSISTENT</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                <span class="n">message_id</span><span class="o">=</span><span class="nb">str</span><span class="p">(</span><span class="n">order</span><span class="p">[</span><span class="s2">&#34;id&#34;</span><span class="p">]),</span>
</span></span><span class="line"><span class="cl">            <span class="p">),</span>
</span></span><span class="line"><span class="cl">            <span class="n">routing_key</span><span class="o">=</span><span class="s2">&#34;order.created&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="s2">&#34;broker confirmed the message&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">asyncio</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">main</span><span class="p">())</span></span></span></code></pre></div></div>
<p>Keep message bodies small. Instead of a 10 MB JSON blob or a base64 image, upload the file to object storage and put its path in the message. RabbitMQ is comfortable with messages of a few KB.</p>

<h2 class="relative group">Consuming: prefetch, acks and the DLX
    <div id="consuming-prefetch-acks-and-the-dlx" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#consuming-prefetch-acks-and-the-dlx" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The most important consumer setting is <code>prefetch_count</code>: how many unacked messages a consumer may hold at once. Without it, RabbitMQ can push the whole backlog to the first consumer that connects, which then fills its memory while the others sit idle.</p>
<p>For slow, heavy jobs start at 1 so work is spread evenly. For small, fast messages, values between 20 and 100 cut network round trips and raise throughput. Measure to find the right number.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">asyncio</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">aio_pika</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">AMQP_URL</span> <span class="o">=</span> <span class="s2">&#34;amqp://app:secret@localhost/&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">handle</span><span class="p">(</span><span class="n">order</span><span class="p">:</span> <span class="nb">dict</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="s2">&#34;id&#34;</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">order</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">&#34;missing id&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>  <span class="c1"># real work goes here</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">async</span> <span class="k">def</span> <span class="nf">main</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">connection</span> <span class="o">=</span> <span class="k">await</span> <span class="n">aio_pika</span><span class="o">.</span><span class="n">connect_robust</span><span class="p">(</span><span class="n">AMQP_URL</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">async</span> <span class="k">with</span> <span class="n">connection</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">channel</span> <span class="o">=</span> <span class="k">await</span> <span class="n">connection</span><span class="o">.</span><span class="n">channel</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">        <span class="k">await</span> <span class="n">channel</span><span class="o">.</span><span class="n">set_qos</span><span class="p">(</span><span class="n">prefetch_count</span><span class="o">=</span><span class="mi">10</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">dlx</span> <span class="o">=</span> <span class="k">await</span> <span class="n">channel</span><span class="o">.</span><span class="n">declare_exchange</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;orders.dlx&#34;</span><span class="p">,</span> <span class="n">aio_pika</span><span class="o">.</span><span class="n">ExchangeType</span><span class="o">.</span><span class="n">FANOUT</span><span class="p">,</span> <span class="n">durable</span><span class="o">=</span><span class="kc">True</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">dead</span> <span class="o">=</span> <span class="k">await</span> <span class="n">channel</span><span class="o">.</span><span class="n">declare_queue</span><span class="p">(</span><span class="s2">&#34;orders.dead&#34;</span><span class="p">,</span> <span class="n">durable</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">await</span> <span class="n">dead</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span><span class="n">dlx</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="n">exchange</span> <span class="o">=</span> <span class="k">await</span> <span class="n">channel</span><span class="o">.</span><span class="n">declare_exchange</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;orders&#34;</span><span class="p">,</span> <span class="n">aio_pika</span><span class="o">.</span><span class="n">ExchangeType</span><span class="o">.</span><span class="n">TOPIC</span><span class="p">,</span> <span class="n">durable</span><span class="o">=</span><span class="kc">True</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="n">queue</span> <span class="o">=</span> <span class="k">await</span> <span class="n">channel</span><span class="o">.</span><span class="n">declare_queue</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;orders&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">durable</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="n">arguments</span><span class="o">=</span><span class="p">{</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;x-queue-type&#34;</span><span class="p">:</span> <span class="s2">&#34;quorum&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;x-dead-letter-exchange&#34;</span><span class="p">:</span> <span class="s2">&#34;orders.dlx&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;x-delivery-limit&#34;</span><span class="p">:</span> <span class="mi">5</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">await</span> <span class="n">queue</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span><span class="n">exchange</span><span class="p">,</span> <span class="n">routing_key</span><span class="o">=</span><span class="s2">&#34;order.*&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">        <span class="k">async</span> <span class="k">with</span> <span class="n">queue</span><span class="o">.</span><span class="n">iterator</span><span class="p">()</span> <span class="k">as</span> <span class="n">messages</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">async</span> <span class="k">for</span> <span class="n">message</span> <span class="ow">in</span> <span class="n">messages</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">                <span class="c1"># Ack if the block finishes, reject(requeue=False) on an exception</span>
</span></span><span class="line"><span class="cl">                <span class="k">async</span> <span class="k">with</span> <span class="n">message</span><span class="o">.</span><span class="n">process</span><span class="p">(</span><span class="n">requeue</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">                    <span class="k">await</span> <span class="n">handle</span><span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">asyncio</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">main</span><span class="p">())</span></span></span></code></pre></div></div>
<p><code>message.process(requeue=False)</code> does two jobs: it acks when the block completes and rejects without requeueing when it raises. Because of <code>x-dead-letter-exchange</code>, the rejected message lands in <code>orders.dead</code> instead of disappearing.</p>
<p>If a message crashes the consumer every time and you keep requeueing it, it loops forever and blocks everything behind it. With a DLX it's set aside, and you can inspect it, fix the bug and republish. For transient failures like a timeout you can reject with <code>requeue=True</code> and let the quorum queue's <code>x-delivery-limit</code> cap the retries; past the limit the message is dead-lettered. Since RabbitMQ 4.0 quorum queues default that limit to 20.</p>
<p>Here are both paths side by side:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/rabbitmq-message-queue/img/retry-vs-dead-letter_hu_d3e40a1178dde1fd.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/rabbitmq-message-queue/img/retry-vs-dead-letter_hu_42f5b2c9867649c1.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/rabbitmq-message-queue/img/retry-vs-dead-letter_hu_c341a60aa11927b7.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/rabbitmq-message-queue/img/retry-vs-dead-letter_hu_5497e045b8ffc91b.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/rabbitmq-message-queue/img/retry-vs-dead-letter_hu_3a6b0ea0ba946d0f.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/rabbitmq-message-queue/img/retry-vs-dead-letter_hu_24884c4df99f835b.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/rabbitmq-message-queue/img/retry-vs-dead-letter_hu_3a6b0ea0ba946d0f.webp" alt="Transient errors use reject(requeue=True) and retry until x-delivery-limit; permanent ones use reject(requeue=False) and go straight to orders.dead" width="1600" height="1260"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>Transient errors get retried, permanent ones are set aside at once</figcaption></figure>
<h2 class="relative group">Classic or quorum queues?
    <div id="classic-or-quorum-queues" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#classic-or-quorum-queues" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Older guides recommend mirrored classic queues for high availability. RabbitMQ 4.0 removed them. If you need a replicated, durable queue, use a quorum queue: it replicates through Raft, stays consistent when a node dies and gives you the delivery limit above.</p>
<p>Classic queues still make sense for transient data you can afford to lose and for single-node setups. Also skip <code>global=True</code> on <code>prefetch_count</code>. Quorum queues don't support global QoS, and a per-consumer limit is what you want anyway.</p>

<h2 class="relative group">Monitoring
    <div id="monitoring" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#monitoring" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The management UI shows the current state but keeps no history. Enable the built-in <code>rabbitmq_prometheus</code> plugin and scrape it with Prometheus; I covered that setup in my <a href="https://blog.furkankoykiran.com.tr/en/posts/prometheus-grafana-monitoring/" >Prometheus and Grafana post</a>. Per-queue metrics come from the <code>/metrics/per-object</code> endpoint. The ones I alert on:</p>
<ul>
<li><code>rabbitmq_queue_messages_ready</code>: the backlog. If it keeps growing, consumers can't keep up.</li>
<li><code>rabbitmq_queue_messages_unacked</code>: delivered but not acked. High values mean stuck consumers or missing acks, and the broker holds these messages in memory.</li>
<li><code>rabbitmq_queue_consumers</code>: if it drops below what you expect, a service may be down.</li>
<li>The depth of the dead letter queue: anything above zero needs a human.</li>
</ul>

<h2 class="relative group">Before you go live
    <div id="before-you-go-live" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#before-you-go-live" aria-label="Anchor">#</a>
    </span>
    
</h2>
<ul>
<li>Is the default <code>guest</code> user gone? It can only connect from localhost, but remove it anyway.</li>
<li>If several projects share a broker, does each have its own vhost and user?</li>
<li>Do queues have a ceiling via <code>x-max-length</code> or <code>x-message-ttl</code>, so a stalled consumer can't fill the disk?</li>
<li>Does the app use one long-lived connection and a channel per task?</li>
<li>In a cluster, did you choose a partition handling strategy (<code>cluster_partition_handling</code>, usually <code>pause_minority</code>) on purpose?</li>
</ul>

<h2 class="relative group">RabbitMQ or Kafka?
    <div id="rabbitmq-or-kafka" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#rabbitmq-or-kafka" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>They solve different problems. RabbitMQ is a smart broker: it routes messages by rules and deletes them once acked. It fits work queues, order processing and RPC, the &quot;someone please do this job&quot; cases. Kafka is an append-only log that doesn't delete on consumption: each consumer tracks its own offset, can replay history and handles very high volume. If you need event streams and replay, see my <a href="https://blog.furkankoykiran.com.tr/en/posts/apache-kafka-event-streaming/" >Kafka post</a>. RabbitMQ's own stream queues are a middle ground for smaller replay use cases.</p>

<h2 class="relative group">Where to go next
    <div id="where-to-go-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-to-go-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>For plain background jobs, a task queue library is less code than raw RabbitMQ; I compared two of them in my <a href="https://blog.furkankoykiran.com.tr/en/posts/celery-arq-async-task-queue/" >Celery and ARQ post</a>. If you use aio-pika directly, point the consumer above at a local RabbitMQ, publish a deliberately broken message and watch it land in <code>orders.dead</code>. The <a href="https://docs.aio-pika.com/"  target="_blank" rel="noreferrer">aio-pika docs</a> and RabbitMQ's <a href="https://www.rabbitmq.com/docs/reliability"  target="_blank" rel="noreferrer">reliability guide</a> are good next reads.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/rabbitmq-message-queue/feature_hu_708acc1d6b93254d.jpg" medium="image" type="image/jpeg"/></item><item><title>Elasticsearch Full-Text Search in Production</title><link>https://blog.furkankoykiran.com.tr/en/posts/elasticsearch-full-text-search/</link><guid isPermaLink="true">https://blog.furkankoykiran.com.tr/en/posts/elasticsearch-full-text-search/</guid><pubDate>Tue, 05 Aug 2025 09:00:00 +0300</pubDate><dc:creator>Furkan Köykıran</dc:creator><category>Backend</category><category>databases</category><category>python</category><category>performance</category><category>devops</category><description>Full-text search with Elasticsearch 8/9 and the Python client: strict mappings, language analyzers, zero-downtime reindexing with aliases, shards, pagination.</description><content:encoded><![CDATA[<p>Elasticsearch full-text search is easy on day one. You start the Docker image, index a few JSON documents and search works. The trouble starts later: you need to change a mapping, the cluster turns yellow, or a user tries to jump to page 10,000.</p>
<p>These are the things I watch for when I run Elasticsearch as a search engine next to PostgreSQL. The examples target Elasticsearch 8/9 and the official <code>elasticsearch</code> Python client.</p>

<h2 class="relative group">Rule zero: Elasticsearch is not your source of truth
    <div id="rule-zero-elasticsearch-is-not-your-source-of-truth" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#rule-zero-elasticsearch-is-not-your-source-of-truth" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Elasticsearch is a search and analytics engine. It has no transactions, a new document becomes searchable after the refresh interval (one second by default), and changing a field's mapping means reindexing. I keep the real data in a database like PostgreSQL and sync it to Elasticsearch asynchronously, through a queue or CDC. If the index breaks, you rebuild it from the database. The reverse may not be possible.</p>
<div class="fk-table" role="region" tabindex="0" aria-label="Scrollable table">
<table>
  <thead>
    <tr>
      <th style="text-align:left"></th>
      <th style="text-align:left">PostgreSQL</th>
      <th style="text-align:left">Elasticsearch</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align:left">Role</td>
      <td style="text-align:left">Source of truth</td>
      <td style="text-align:left">Search and analytics</td>
    </tr>
    <tr>
      <td style="text-align:left">Consistency</td>
      <td style="text-align:left">ACID</td>
      <td style="text-align:left">Near real time, eventual</td>
    </tr>
    <tr>
      <td style="text-align:left">Text search</td>
      <td style="text-align:left"><code>ILIKE '%x%'</code> is slow, <code>tsvector</code> is limited</td>
      <td style="text-align:left">Inverted index with relevance scoring</td>
    </tr>
    <tr>
      <td style="text-align:left">Schema change</td>
      <td style="text-align:left"><code>ALTER TABLE</code></td>
      <td style="text-align:left">New index plus reindex</td>
    </tr>
  </tbody>
</table>
</div><p>If you sync through a message broker, my posts on <a href="https://blog.furkankoykiran.com.tr/en/posts/rabbitmq-message-queue/" >RabbitMQ</a> and <a href="https://blog.furkankoykiran.com.tr/en/posts/apache-kafka-event-streaming/" >Kafka</a> cover the two common options.</p>

<h2 class="relative group">Connecting from Python
    <div id="connecting-from-python" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#connecting-from-python" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Since Elasticsearch 8, security is on by default: TLS and authentication are required. Match the client's major version to the server: <code>elasticsearch&gt;=8,&lt;9</code> for an 8.x cluster, <code>elasticsearch&gt;=9,&lt;10</code> for 9.x. The 9.x client dropped the old <code>body=</code> parameter entirely, so every request uses keyword arguments.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">elasticsearch</span> <span class="kn">import</span> <span class="n">Elasticsearch</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">es</span> <span class="o">=</span> <span class="n">Elasticsearch</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;https://localhost:9200&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">api_key</span><span class="o">=</span><span class="s2">&#34;BASE64_API_KEY&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">ca_certs</span><span class="o">=</span><span class="s2">&#34;http_ca.crt&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">request_timeout</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="n">es</span><span class="o">.</span><span class="n">info</span><span class="p">()[</span><span class="s2">&#34;version&#34;</span><span class="p">][</span><span class="s2">&#34;number&#34;</span><span class="p">])</span></span></span></code></pre></div></div>
<p>Give the application a scoped API key instead of a username and password. In async code such as a FastAPI service, <code>AsyncElasticsearch</code> has the same API.</p>

<h2 class="relative group">Turn off dynamic mapping in production
    <div id="turn-off-dynamic-mapping-in-production" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#turn-off-dynamic-mapping-in-production" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>When Elasticsearch sees an unknown field, it guesses the type. If the first document has <code>&quot;status_code&quot;: 200</code>, the field becomes <code>long</code>. The day another service sends <code>&quot;status_code&quot;: &quot;404 Not Found&quot;</code>, that document is rejected with a <code>mapper_parsing_exception</code>. There is a slower failure too: every new JSON key creates a new field and the mapping keeps growing.</p>
<p>I use <code>dynamic: strict</code> on production indices. An undefined field makes the write fail, so you find out right away.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">es</span><span class="o">.</span><span class="n">indices</span><span class="o">.</span><span class="n">create</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">index</span><span class="o">=</span><span class="s2">&#34;products-v1&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">settings</span><span class="o">=</span><span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;number_of_shards&#34;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;number_of_replicas&#34;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;analysis&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;analyzer&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;tr_text&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">                    <span class="s2">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;custom&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                    <span class="s2">&#34;tokenizer&#34;</span><span class="p">:</span> <span class="s2">&#34;standard&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                    <span class="s2">&#34;filter&#34;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&#34;apostrophe&#34;</span><span class="p">,</span> <span class="s2">&#34;turkish_lowercase&#34;</span><span class="p">,</span> <span class="s2">&#34;asciifolding&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">                <span class="p">}</span>
</span></span><span class="line"><span class="cl">            <span class="p">},</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;filter&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;turkish_lowercase&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;lowercase&#34;</span><span class="p">,</span> <span class="s2">&#34;language&#34;</span><span class="p">:</span> <span class="s2">&#34;turkish&#34;</span><span class="p">}</span>
</span></span><span class="line"><span class="cl">            <span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="n">mappings</span><span class="o">=</span><span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;dynamic&#34;</span><span class="p">:</span> <span class="s2">&#34;strict&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;properties&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;sku&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;keyword&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;name&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;text&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;analyzer&#34;</span><span class="p">:</span> <span class="s2">&#34;tr_text&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">                <span class="s2">&#34;fields&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;raw&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;keyword&#34;</span><span class="p">}},</span>
</span></span><span class="line"><span class="cl">            <span class="p">},</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;description&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;text&#34;</span><span class="p">,</span> <span class="s2">&#34;analyzer&#34;</span><span class="p">:</span> <span class="s2">&#34;tr_text&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;price&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;scaled_float&#34;</span><span class="p">,</span> <span class="s2">&#34;scaling_factor&#34;</span><span class="p">:</span> <span class="mi">100</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;created_at&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;date&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span></span></span></code></pre></div></div>
<p><code>keyword</code> is for exact matches, filters and sorting. <code>text</code> is analyzed into tokens. If you need both, define a multi-field like <code>name.raw</code> above.</p>

<h2 class="relative group">Analyzers decide what matches
    <div id="analyzers-decide-what-matches" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#analyzers-decide-what-matches" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The inverted index maps every term to the documents that contain it. That is where the speed comes from, but the analyzer decides the result: if indexing and searching don't split text into the same tokens, nothing matches.</p>
<p>My data is mostly Turkish, which is a good stress test for any analyzer:</p>
<ol>
<li>The default <code>lowercase</code> filter turns the capital dotted &quot;İ&quot; into &quot;i&quot; plus a combining dot. A lowercase filter with <code>language: turkish</code> handles it correctly.</li>
<li>Turkish attaches suffixes with an apostrophe (&quot;İstanbul'da&quot; means &quot;in Istanbul&quot;). The standard tokenizer keeps it as one token; the <code>apostrophe</code> filter drops everything after the apostrophe.</li>
</ol>
<p>The diagram below follows a sample string through the <code>tr_text</code> analyzer from the mapping, one step at a time:</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/elasticsearch-full-text-search/img/tr-text-analyzer-pipeline_hu_4773f7882f0b9f1d.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/elasticsearch-full-text-search/img/tr-text-analyzer-pipeline_hu_6776ace91c0a349e.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/elasticsearch-full-text-search/img/tr-text-analyzer-pipeline_hu_a4e6a72c2f2f918d.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/elasticsearch-full-text-search/img/tr-text-analyzer-pipeline_hu_cbf075634694d4ac.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/elasticsearch-full-text-search/img/tr-text-analyzer-pipeline_hu_c6f3abd48d8131e0.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/elasticsearch-full-text-search/img/tr-text-analyzer-pipeline_hu_cf32c01c389a84d5.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/elasticsearch-full-text-search/img/tr-text-analyzer-pipeline_hu_c6f3abd48d8131e0.webp" alt="The text İstanbul&#39;da KOŞU ayakkabısı passes through standard tokenizer, apostrophe, turkish_lowercase and asciifolding to become istanbul, kosu, ayakkabisi" width="1600" height="1310"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>What each filter in tr_text does to the tokens</figcaption></figure><p><code>tr_text</code> fixes both problems, and <code>asciifolding</code> maps letters like &quot;ç, ş, ğ&quot; to ASCII so a user typing on an English keyboard still gets hits. For English content you would start from the built-in <code>english</code> analyzer instead. Every language has quirks like these, so check with <code>_analyze</code> before you trust an analyzer:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">tokens</span> <span class="o">=</span> <span class="n">es</span><span class="o">.</span><span class="n">indices</span><span class="o">.</span><span class="n">analyze</span><span class="p">(</span><span class="n">index</span><span class="o">=</span><span class="s2">&#34;products-v1&#34;</span><span class="p">,</span> <span class="n">analyzer</span><span class="o">=</span><span class="s2">&#34;tr_text&#34;</span><span class="p">,</span> <span class="n">text</span><span class="o">=</span><span class="s2">&#34;İstanbul&#39;da KOŞU ayakkabısı&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="nb">print</span><span class="p">([</span><span class="n">t</span><span class="p">[</span><span class="s2">&#34;token&#34;</span><span class="p">]</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">tokens</span><span class="p">[</span><span class="s2">&#34;tokens&#34;</span><span class="p">]])</span>
</span></span><span class="line"><span class="cl"><span class="c1"># [&#39;istanbul&#39;, &#39;kosu&#39;, &#39;ayakkabisi&#39;]</span></span></span></code></pre></div></div>
<p>The built-in language analyzers also add stemming. On short text like product names, stemming can pull in unrelated results, so test it on real queries before turning it on.</p>
<p>For search I use <code>multi_match</code> across fields and boost the name:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">resp</span> <span class="o">=</span> <span class="n">es</span><span class="o">.</span><span class="n">search</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">index</span><span class="o">=</span><span class="s2">&#34;products&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">query</span><span class="o">=</span><span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;multi_match&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;query&#34;</span><span class="p">:</span> <span class="s2">&#34;koşu ayakkabısı&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;fields&#34;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&#34;name^3&#34;</span><span class="p">,</span> <span class="s2">&#34;description&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">            <span class="s2">&#34;fuzziness&#34;</span><span class="p">:</span> <span class="s2">&#34;AUTO&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="n">size</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="k">for</span> <span class="n">hit</span> <span class="ow">in</span> <span class="n">resp</span><span class="p">[</span><span class="s2">&#34;hits&#34;</span><span class="p">][</span><span class="s2">&#34;hits&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="cl">    <span class="nb">print</span><span class="p">(</span><span class="n">hit</span><span class="p">[</span><span class="s2">&#34;_score&#34;</span><span class="p">],</span> <span class="n">hit</span><span class="p">[</span><span class="s2">&#34;_source&#34;</span><span class="p">][</span><span class="s2">&#34;name&#34;</span><span class="p">])</span></span></span></code></pre></div></div>

<h2 class="relative group">Zero-downtime reindexing with aliases
    <div id="zero-downtime-reindexing-with-aliases" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#zero-downtime-reindexing-with-aliases" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>You can't change the type of an existing field. Switching a field from <code>text</code> to <code>keyword</code>, changing an analyzer or changing the shard count all mean creating a new index and copying the data. If the application talks to the physical index name, that's an outage.</p>
<p>The fix is to read and write through an alias. In the search above, <code>products</code> is an alias.</p>
<figure class="fk-figure"><picture>
    <source type="image/avif" srcset="https://blog.furkankoykiran.com.tr/en/posts/elasticsearch-full-text-search/img/zero-downtime-reindexing-aliases_hu_199d8de8e1437434.avif 480w, https://blog.furkankoykiran.com.tr/en/posts/elasticsearch-full-text-search/img/zero-downtime-reindexing-aliases_hu_4e97e1b238326c87.avif 800w, https://blog.furkankoykiran.com.tr/en/posts/elasticsearch-full-text-search/img/zero-downtime-reindexing-aliases_hu_b2f60d7c23c76fa9.avif 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <source type="image/webp" srcset="https://blog.furkankoykiran.com.tr/en/posts/elasticsearch-full-text-search/img/zero-downtime-reindexing-aliases_hu_f82d152740318202.webp 480w, https://blog.furkankoykiran.com.tr/en/posts/elasticsearch-full-text-search/img/zero-downtime-reindexing-aliases_hu_750a8bae9dda92c.webp 800w, https://blog.furkankoykiran.com.tr/en/posts/elasticsearch-full-text-search/img/zero-downtime-reindexing-aliases_hu_45b99719a184ebcc.webp 1200w" sizes="(min-width: 768px) 720px, 100vw">
    <img src="https://blog.furkankoykiran.com.tr/en/posts/elasticsearch-full-text-search/img/zero-downtime-reindexing-aliases_hu_750a8bae9dda92c.webp" alt="The application only knows the products alias; _reindex copies products-v1 into products-v2 and the alias moves to v2 atomically" width="1600" height="1015"
      class="my-0 rounded-md"
      loading="lazy" decoding="async">
  </picture><figcaption>The app stays on the alias while the physical index changes behind it</figcaption></figure><p>The steps:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># 1. Create products-v2 with the new mapping (same create call as above)</span>
</span></span><span class="line"><span class="cl"><span class="c1"># 2. Copy the data; run it as a background task on large indices</span>
</span></span><span class="line"><span class="cl"><span class="n">task</span> <span class="o">=</span> <span class="n">es</span><span class="o">.</span><span class="n">reindex</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">source</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;index&#34;</span><span class="p">:</span> <span class="s2">&#34;products-v1&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="n">dest</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;index&#34;</span><span class="p">:</span> <span class="s2">&#34;products-v2&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="n">wait_for_completion</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="n">task</span><span class="p">[</span><span class="s2">&#34;task&#34;</span><span class="p">])</span>  <span class="c1"># poll with es.tasks.get(task_id=...)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># 3. When it&#39;s done, move the alias in one atomic request</span>
</span></span><span class="line"><span class="cl"><span class="n">es</span><span class="o">.</span><span class="n">indices</span><span class="o">.</span><span class="n">update_aliases</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">actions</span><span class="o">=</span><span class="p">[</span>
</span></span><span class="line"><span class="cl">        <span class="p">{</span><span class="s2">&#34;remove&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;index&#34;</span><span class="p">:</span> <span class="s2">&#34;products-v1&#34;</span><span class="p">,</span> <span class="s2">&#34;alias&#34;</span><span class="p">:</span> <span class="s2">&#34;products&#34;</span><span class="p">}},</span>
</span></span><span class="line"><span class="cl">        <span class="p">{</span><span class="s2">&#34;add&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;index&#34;</span><span class="p">:</span> <span class="s2">&#34;products-v2&#34;</span><span class="p">,</span> <span class="s2">&#34;alias&#34;</span><span class="p">:</span> <span class="s2">&#34;products&#34;</span><span class="p">,</span> <span class="s2">&#34;is_write_index&#34;</span><span class="p">:</span> <span class="kc">True</span><span class="p">}},</span>
</span></span><span class="line"><span class="cl">    <span class="p">]</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span></span></span></code></pre></div></div>
<p>One catch: documents written to v1 while the reindex runs don't reach v2. Either dual-write during that window or run a second, smaller reindex filtered on an <code>updated_at</code> field. I find the second option simpler.</p>
<p>For time-series data like logs, skip manual aliases and use data streams with ILM. Elasticsearch then handles rollover for you.</p>

<h2 class="relative group">Sizing shards and the cluster
    <div id="sizing-shards-and-the-cluster" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#sizing-shards-and-the-cluster" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The shard count depends on data size. Elastic's current guidance is 10 to 50 GB per shard and fewer than 200 million documents. Tiny shards waste heap and file handles; huge ones take hours to move when a node dies. Since 8.x, new indices default to one primary shard, which is usually fine for a small catalog.</p>
<p>Keep at least one replica. On a single-node cluster, though, the replica has nowhere to go and the cluster stays yellow, so set <code>number_of_replicas: 0</code> in development.</p>
<p>Many older articles tell you to set <code>discovery.zen.minimum_master_nodes</code>. That setting was removed in Elasticsearch 7. The cluster now manages its voting configuration itself, and you only provide <code>cluster.initial_master_nodes</code> when bootstrapping a new cluster. Still run an odd number of master-eligible nodes (3 or 5): four voters tolerate no more failures than three.</p>
<p>As traffic grows, split roles. Master nodes manage cluster state, data nodes hold data and run queries, ingest nodes run pipelines before indexing. The main benefit: a heavy aggregation that chokes a data node doesn't take cluster management down with it.</p>

<h2 class="relative group">Deep pagination: use search_after, not from/size
    <div id="deep-pagination-use-search_after-not-fromsize" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#deep-pagination-use-search_after-not-fromsize" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A query with <code>from: 100000, size: 10</code> makes every shard collect and sort the top 100,010 hits. That's why Elasticsearch refuses to go past 10,000 results by default (<code>index.max_result_window</code>). To walk through everything, use a point in time (PIT) with <code>search_after</code>:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">pit</span> <span class="o">=</span> <span class="n">es</span><span class="o">.</span><span class="n">open_point_in_time</span><span class="p">(</span><span class="n">index</span><span class="o">=</span><span class="s2">&#34;products&#34;</span><span class="p">,</span> <span class="n">keep_alive</span><span class="o">=</span><span class="s2">&#34;1m&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="n">search_after</span> <span class="o">=</span> <span class="kc">None</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="n">resp</span> <span class="o">=</span> <span class="n">es</span><span class="o">.</span><span class="n">search</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">        <span class="n">pit</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;id&#34;</span><span class="p">:</span> <span class="n">pit</span><span class="p">[</span><span class="s2">&#34;id&#34;</span><span class="p">],</span> <span class="s2">&#34;keep_alive&#34;</span><span class="p">:</span> <span class="s2">&#34;1m&#34;</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">        <span class="n">query</span><span class="o">=</span><span class="p">{</span><span class="s2">&#34;match_all&#34;</span><span class="p">:</span> <span class="p">{}},</span>
</span></span><span class="line"><span class="cl">        <span class="n">sort</span><span class="o">=</span><span class="p">[{</span><span class="s2">&#34;created_at&#34;</span><span class="p">:</span> <span class="s2">&#34;asc&#34;</span><span class="p">}],</span>
</span></span><span class="line"><span class="cl">        <span class="n">size</span><span class="o">=</span><span class="mi">1000</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="n">search_after</span><span class="o">=</span><span class="n">search_after</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">hits</span> <span class="o">=</span> <span class="n">resp</span><span class="p">[</span><span class="s2">&#34;hits&#34;</span><span class="p">][</span><span class="s2">&#34;hits&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">hits</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">break</span>
</span></span><span class="line"><span class="cl">    <span class="k">for</span> <span class="n">hit</span> <span class="ow">in</span> <span class="n">hits</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="o">...</span>  <span class="c1"># process the document</span>
</span></span><span class="line"><span class="cl">    <span class="n">search_after</span> <span class="o">=</span> <span class="n">hits</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="s2">&#34;sort&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">es</span><span class="o">.</span><span class="n">close_point_in_time</span><span class="p">(</span><span class="nb">id</span><span class="o">=</span><span class="n">pit</span><span class="p">[</span><span class="s2">&#34;id&#34;</span><span class="p">])</span></span></span></code></pre></div></div>
<p>With a PIT, Elasticsearch adds an implicit <code>_shard_doc</code> tiebreaker to the sort, so documents with the same <code>created_at</code> aren't skipped. In a user interface, the best fix is not offering a &quot;page 10,000&quot; button at all.</p>

<h2 class="relative group">Yellow clusters and circuit breakers
    <div id="yellow-clusters-and-circuit-breakers" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#yellow-clusters-and-circuit-breakers" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A yellow cluster means all primaries are assigned but some replicas aren't. The two usual causes are a disk past the high watermark (90% by default) or no node available for the replica. Ask instead of guessing:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="n">es</span><span class="o">.</span><span class="n">cluster</span><span class="o">.</span><span class="n">allocation_explain</span><span class="p">())</span></span></span></code></pre></div></div>
<p>A <code>circuit_breaking_exception</code> means a request wanted so much heap that Elasticsearch rejected it instead of running out of memory. The usual causes are aggregating on a <code>text</code> field with <code>fielddata: true</code>, or oversized <code>terms</code> aggregations. Aggregate on <code>keyword</code> fields.</p>

<h2 class="relative group">Pre-launch checklist
    <div id="pre-launch-checklist" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#pre-launch-checklist" aria-label="Anchor">#</a>
    </span>
    
</h2>
<ul>
<li>Is swap disabled on the nodes, or <code>bootstrap.memory_lock: true</code> set?</li>
<li>If you set the heap by hand, is it at most half the RAM and under about 31 GB? Since 8.x Elasticsearch sizes the heap automatically, and in most cases you should leave it alone.</li>
<li>Have you blocked leading-wildcard queries (<code>*kiran</code>)? They scan the whole term dictionary. If you really need them, look at the <code>wildcard</code> field type.</li>
<li>Is port 9200 kept off the public internet, with the app using a scoped API key?</li>
<li>Are snapshots running on a schedule? Any S3-compatible store works; on your own servers, <a href="https://blog.furkankoykiran.com.tr/en/posts/minio-object-storage/" >MinIO</a> does the job.</li>
</ul>

<h2 class="relative group">Where to go next
    <div id="where-to-go-next" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-to-go-next" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Start a single-node Elasticsearch in Docker and run your own text through <code>_analyze</code> with the analyzer you plan to use. Getting the mapping, analyzer and alias right on day one saves most of the reindexing you'd otherwise do later. If you sync through an event stream, my <a href="https://blog.furkankoykiran.com.tr/en/posts/apache-kafka-event-streaming/" >Kafka post</a> is a good next read. The <a href="https://www.elastic.co/docs/reference/elasticsearch/clients/python"  target="_blank" rel="noreferrer">Python client docs</a> and the <a href="https://www.elastic.co/docs/reference/text-analysis/analyzer-reference"  target="_blank" rel="noreferrer">analyzer reference</a> are the references I use.</p>
]]></content:encoded><media:content url="https://blog.furkankoykiran.com.tr/en/posts/elasticsearch-full-text-search/feature_hu_2f203cd9fea75c11.jpg" medium="image" type="image/jpeg"/></item></channel></rss>