# Eugene Oleinik - Full Content > Building things that work. Generated: 2026-06-10 --- ## The Real Cost of a Claude Code Session Is the Language Server, Not Claude URL: https://evoleinik.com/posts/claude-code-memory-real-cost-per-session/ Date: 2026-06-10 Tags: observability, memory, llm-agents, linux, typescript `top` said my dev box was at 45GB used. Summing RSS across the roughly 1000 `node`/`bun` processes from ~26 concurrent Claude Code sessions said 49.8GB. Both numbers are wrong about what's actually costing me memory. The real figure is 32.5GB, the `claude` process itself is a rounding error, and the entire difference between a cheap session and an expensive one is the TypeScript language server. Here's the investigation, with every number measured from `/proc`. ## RSS is lying to you (by 53%) The setup: a Linux box, 62GB RAM, ~26 Claude Code sessions open against one large TypeScript monorepo. Naive accounting: | Metric | Total | What it counts | |--------|-------|----------------| | Σ RSS (all node/bun) | 49.8 GB | Every page each process maps, **including shared pages counted once per process** | | Σ PSS (true) | 32.5 GB | Each process's *proportional* share of shared pages + its private pages | That 17.3GB gap — a 53% overcount — is the shared V8/Node binary and shared libraries. With ~1000 Node processes alive, every one of them maps the same interpreter and the same `.so` files. RSS counts that binary ~1000 times. It's mapped once in physical RAM. Three numbers matter when you measure process memory on Linux: - **RSS** — Resident Set Size. Every resident page the process maps. Shared pages are counted in full for *every* sharer. Sums to nonsense. - **PSS** — Proportional Set Size. A shared page mapped by *N* processes contributes `1/N` of its size to each. Sum of PSS across all processes equals actual physical RAM used. This is the honest total. - **USS** — Unique Set Size = `Private_Clean + Private_Dirty`. Strictly private, unshareable pages. This is what you reclaim if you kill the process *and nothing else was sharing its pages*. All three are readable per-process from `/proc/PID/smaps_rollup`. ## I assumed closing sessions wouldn't help. I was wrong. My first instinct: most of this is the shared Node binary, so closing a few idle sessions buys me nothing — the pages stay mapped by the others. Reasonable theory. It's also false, and measuring USS is what proved it. Per session, PSS ≈ USS — equal to one decimal place. That seems to contradict Finding 1 (17GB of shared pages!) until you do the division. The shared binary and libs are split across ~1000 processes, so any single process's *proportional* share of shared memory is negligible. What's left, and what dominates each process's PSS, is its **private heap**. Practical consequence: closing a session reclaims essentially its full footprint. The shared-pages story is real in aggregate and irrelevant per session. If you remember one thing about reading `smaps`: shared memory matters for the *grand total*, private memory matters for *what you can free*. ## Where the memory actually goes Two sessions, same MCP config, measured by PSS. **Light session** (chatting, not editing code) — ~0.9GB, fully reclaimable: | Component | PSS | Procs | |-----------|-----|-------| | chrome-devtools-mcp | 289 MB | 4 | | claude (main) | 193 MB | 1 | | playwright-mcp | 121 MB | 4 | | context7 | 77 MB | 3 | | in-house MCP servers | ~190 MB | several | **Heavy session** (editing the monorepo) — ~4.8GB: | Component | PSS | Procs | |-----------|-----|-------| | TypeScript `tsserver` (LSP) | 3,919 MB | 2 | | the same MCP fleet as above | ~700–900 MB | many | The MCP fleet is identical in both. The `claude` process barely moves. The only thing that changed is one component went from absent to nearly 4GB: the language server. ## Four things this proves **(a) `claude` itself is cheap.** ~190MB. The "Claude Code uses 50GB" headline is an RSS artifact — the shared Node binary counted ~1000 times. The agent's own cost is trivial. The cost is in what it *spawns*. **(b) There's a fixed ~700MB "MCP tax" per session.** Identical whether you're coding or just chatting, because Claude Code spawns your *entire* configured MCP fleet fresh for every session over stdio. No sharing between sessions, no lazy start. Twenty-six sessions = twenty-six full copies of your MCP fleet. **(c) chrome-devtools-mcp costs more than `claude` does.** 289MB across 4 helper processes — the single priciest MCP, beating the agent process itself. Most people have it in their global config and have no idea it's the most expensive thing they spawn per session. **(d) The TypeScript language server is the entire swing.** 0 or ~3.9GB, depending purely on whether the session opened TS files. One `tsserver` per session/worktree, never shared. The math is brutal: ``` 4 heavy sessions: 4 × ~4.1GB = 16.3 GB 22 light sessions: 22 × ~0.67GB = 14.7 GB ``` Four coding sessions outweigh all twenty-two idle ones combined. Your language server, multiplied by your session count, *is* the memory story. ## Aggregate by role Summing PSS by component across all 26 sessions: | Role | PSS | Procs | |------|-----|-------| | tsserver (LSP) | 13.1 GB | 8 | | Claude Code main | 5.3 GB | 26 | | chrome-devtools-mcp | 4.9 GB | 80 | | playwright-mcp | 2.2 GB | 84 | | context7 | 1.3 GB | 60 | | in-house MCP (combined) | ~2.9 GB | many | | bun-based MCP | 1.0 GB | many | That's ~32.5GB of process PSS. Add ~8GB of tmpfs/shm plus kernel overhead and you reconcile cleanly with the 45GB `free` reported. The 49.8GB RSS sum never reconciled with anything, because it was never real. Note `tsserver` runs only 8 processes (4 sessions × 2 — the full server plus a `partialSemantic` instance) yet tops the table at 13.1GB. chrome-devtools-mcp runs 80 processes for 4.9GB. Process count is not cost. ## Reproduce it yourself The whole investigation is just reading `smaps_rollup` per PID and bucketing by what the process is. `smaps_rollup` gives you pre-summed `Pss`, `Private_Clean`, `Private_Dirty`, and `Swap` in one cheap read — no need to parse the full `smaps`. ```python import os, glob def category(pid): try: cmd = open(f"/proc/{pid}/cmdline").read().replace("\0", " ") except OSError: return None if "tsserver" in cmd: return "tsserver" if "chrome-devtools-mcp" in cmd: return "chrome-devtools-mcp" if "playwright" in cmd: return "playwright-mcp" if "context7" in cmd: return "context7" if "claude" in cmd: return "claude-main" if "node" in cmd or "bun" in cmd: return "other-mcp" return None def field(rollup, key): for line in rollup.splitlines(): if line.startswith(key + ":"): return int(line.split()[1]) # kB return 0 totals = {} for p in glob.glob("/proc/[0-9]*/smaps_rollup"): pid = p.split("/")[2] cat = category(pid) if not cat: continue try: r = open(p).read() except OSError: continue pss = field(r, "Pss") uss = field(r, "Private_Clean") + field(r, "Private_Dirty") t = totals.setdefault(cat, [0, 0, 0]) t[0] += pss; t[1] += uss; t[2] += 1 for cat, (pss, uss, n) in sorted(totals.items(), key=lambda x: -x[1][0]): print(f"{cat:24} PSS {pss/1e6:6.2f}GB USS {uss/1e6:6.2f}GB ({n} procs)") ``` Run it as root (or with `CAP_SYS_PTRACE`) so you can read other users' `smaps_rollup`. The PSS column is your honest total; compare it against PSS ≈ USS to confirm per-process memory is private. If you don't want to write code, `smem` does the same off the shelf and reports PSS/USS directly (`smem -t -k -c "name pss uss"`). ## The takeaway If you run many agent sessions, you have exactly two memory levers, and neither of them is "use a lighter model" or "close the Claude window": 1. **Sessions touching the monorepo.** Each one that opens TS files spins up a fresh ~3–4GB language server, never shared across sessions or worktrees. This is your dominant, swingiest cost. Fewer concurrent *coding* sessions — or sharing a worktree — is the biggest win available. 2. **MCP fleet size.** Every session pays the full tax: spawn cost × session count, whether you use those tools or not. Trim your *global* MCP config to what you actually reach for. Dropping chrome-devtools-mcp alone saved ~290MB per session — ~7.5GB across 26 sessions. Move situational servers to per-project config so idle sessions don't pay for them. The agent is cheap. The compiler tooling it wakes up is not. Measure PSS, not RSS — your `top` output has been blaming the wrong process this whole time. --- ## The Path Has To Exist: Fixing Claude Code's CI Panel for Remote Dev (Without Melting a CPU) URL: https://evoleinik.com/posts/claude-ci-panel-remote-mirror/ Date: 2026-06-09 Tags: claude-code, macos, fuse-t, sshfs, git-worktree, remote-development, devtools, open-source My code lives on a Linux box. I drive it from a Mac with Claude Code in the desktop app, pointed at the box over SSH — so each session's working directory is a remote path. Everything works except the desktop app's CI/PR panel, which shows "CI checks unavailable" for every pull request. The reason is dumber than it looks. The panel doesn't subscribe to GitHub webhooks. It shells out to your local `gh`, in the **session's working directory**, on the Mac. For a remote session that directory is a Linux path — `/home/eo/src/app/...` — that doesn't exist on the Mac. `gh` runs somewhere with no repo, and the panel goes dark. (You can confirm this by reading the app bundle: the panel computes `available = repoSlugs > 0 && (githubAuth || ghCliAvailable)`, and `repoSlugs` comes from the git remote of the **local** folder at the session's path.) So I stopped trying to fix the app and fixed the premise: **make that exact path exist on the Mac, with a real `.git`.** Then local `gh` resolves the repo and the panel lights up — no patching, survives every update. I shipped that with a FUSE-T sshfs mount. I even wrote it up here. It was wrong. ## The mount that pinned a CPU at 100% FUSE-T is the clean way to get a remote path onto a modern Mac — kext-less, it runs a local NFS server instead of a kernel extension. I mounted `box:/home/eo/src` read-only at the same path, a LaunchAgent kept it alive, the panel lit up. Done. Then my fans spun up and stayed up. `go-nfsv4` — FUSE-T's userspace NFS server — sat at **100% CPU**, indefinitely. The cause is a property of the whole category, not a bug I could patch. The desktop app watches the repo for changes. **NFS has no fsevents**, so a file-watcher over a network mount can't get push notifications — it falls back to *recursively polling*. And I'd mounted the entire `src` tree: every worktree, every `node_modules`, hundreds of thousands of files, walked over the network, forever. A network filesystem under a file-watcher is a polling bomb. But the failure handed me the insight: **the panel doesn't need my files. It needs my `.git`.** `repoSlugs` comes from the remote URL; the checks come from `gh` hitting GitHub's API. The working tree — the expensive part to mirror — is irrelevant to the panel. ## The fix: mirror the git, not the filesystem So I threw out the mount and built a mirror that produces *real local files the OS can watch natively*: - Keep a blobless clone of the repo on the Mac at the identical path. - Every 60 seconds, ask the box for its `git worktree list` and GitHub for my open PRs. - For each remote worktree whose branch has an open PR, recreate it locally as a `git worktree` on the matching branch — a `git fetch`, kilobytes, no working-tree copy of `node_modules`. - Prune the ones whose PRs closed. Real local files → fsevents works → the watcher is cheap → `go-nfsv4` is gone and idle CPU is back to zero. The footprint is git metadata plus tracked source, scoped to the branches that actually have CI to show. A new PR appears within a minute; a closed one disappears. A few macOS traps were worth the scar tissue: - **You can't operate — or even read — the mount from a headless SSH session.** A FUSE-T mount (and, it turns out, launchd-owned git worktrees) live in the GUI `gui/$UID` domain; `git` run via `ssh mac '...'` returns "Operation not permitted." Do the work in launchd, verify in the GUI. - **`/home` doesn't exist on macOS by default** (autofs owns it). If your remote path starts with `/home`, free it once with a one-line `auto_master` edit. Paths under `/Users` need nothing. - **The `/System/Volumes/Data` firmlink** means `git worktree list` reports canonical paths; compare with that prefix stripped or your prune step churns every cycle. ## I packaged it It's a single bash script — point it at your SSH host and the repo path; it clones, reconciles every 60s via a LaunchAgent, and stays out of the way. Open source, MIT: **[github.com/evoleinik/claude-ci-mirror](https://github.com/evoleinik/claude-ci-mirror)** ```sh printf 'REMOTE=box\nREMOTE_REPO=/home/you/src/app\n' > ~/.config/claude-ci-mirror/config claude-ci-mirror doctor && claude-ci-mirror install ``` ## Takeaway When a tool integration breaks across a machine boundary, it's usually assuming a local path. Make the path exist — but give it *real local files*, not a network mount. The mount is the obvious move, and it works for about a minute, until a file-watcher starts polling it and eats a core. The panel was never broken; it was looking in the right place on the wrong filesystem. The real trick wasn't mounting the filesystem — it was noticing the panel only ever wanted the `.git`. --- ## Web 4.0 Is Open Databases URL: https://evoleinik.com/posts/web4-open-databases/ Date: 2026-03-29 Tags: ai, agents, web, data, mcp, open-data Every spring, Chiang Mai turns into a gas chamber. Farmers across northern Thailand and neighboring countries burn crop residue, and the AQI climbs to 200-300+. I have asthma. I have three young kids. Every year I ask the same question: should we escape somewhere for two weeks? This year I asked my AI agent instead of opening 12 browser tabs. ## What Actually Happened I told Claude Code: "Compare air quality across Thai cities for April, find me the cheapest way to get there with a family of 6, and figure out if it's worth the money." What followed was a four-hour session where the agent: 1. **Scraped live AQI data from aqicn.org.** The site is JavaScript-rendered, so a simple HTTP request gets nothing. The agent spun up a headless browser, extracted the data, then discovered the historical data was encoded in a proprietary format baked into their minified JS. 2. **Reverse-engineered the encoding.** The agent read aqicn.org's minified JavaScript, figured out their custom data packing scheme, and built a decoder tool — I called it `aqi-liberator`. It took four different approaches before one worked. The first three failed — simple HTTP got nothing (JS-rendered), the search API timed out, the Chrome DevTools MCP couldn't connect. On the fourth try, the agent used a raw CDP websocket to a headless Chromium, intercepted the XHR requests, found the data endpoint, and reverse-engineered the encoding from the minified source. 3. **Compared 15 cities across 10 years of daily AQI data.** Not a vibe check — actual statistical comparison. April means, medians, 95th percentiles. Chiang Mai: mean 139, 79% of days unhealthy, zero good days. Phuket: mean 57, 3% unhealthy. Rayong: 62, 7% unhealthy. The agent built a complete ranking sorted by air quality per dollar per drive-hour. 4. **Cross-referenced with flight prices.** Scraped Google Flights for all 32 destinations from Chiang Mai. CNX to Surat Thani: $153/person RT. CNX to Phuket: $168. But multiply by 6 people (kids are same price on Thai LCCs) and flying anywhere costs $900-1,000 in airfare alone. 5. **Checked driving costs.** The agent discovered that Thailand had just cut fuel subsidies by 22% *that same week* — found the Bloomberg article, recalculated with the new fuel prices. Chiang Mai to Rayong: 900 km each way, roughly 9,000 THB in fuel post-hike for our SUV, plus tolls, two overnight stops in Tak, and road food for 6 people. 6. **Checked accommodation.** Scraped Airbnb via headless browser for 3-bedroom places in Rayong that could fit 6. My wife found a great deal separately — about $750 for two weeks. Total trip cost: roughly $1,450. The agent compared driving vs flying for a family of 6, including the car rental penalty (you need a car with kids — Grab with 3 car seats doesn't work). Flying 6 people to Phuket + renting a 7-seater for 2 weeks came to $2,500+. Driving won. All through conversation. No dashboards. No context-switching between Google Flights, Airbnb, AQI websites, news sites, and a spreadsheet. To be fair: this wasn't push-button. The session cost maybe $15 in API usage, I had a flight-scanning codebase already built, and I nudged the agent through several dead ends. This is closer to pair-programming with a very fast research assistant than to "AI plans your vacation." But even with the friction, it compressed two days of research into one session. ## The Shape of What's Coming Strip away the specifics and look at what the agent actually did. It queried multiple data sources, joined the results on a common key (city + date range), applied filters, ranked the output, and presented it in the format I needed. That's a SQL query. Distributed across the internet. ``` SELECT city, avg_aqi, flight_price, hotel_price, drive_cost FROM aqi_data JOIN flight_prices ON city = destination AND month = 'April' JOIN accommodation ON city = location AND type = '3br' JOIN fuel_costs ON route = CONCAT('CNX-', city) WHERE avg_aqi < 75 ORDER BY (flight_price + hotel_price * 14) ASC LIMIT 3; ``` The difference: there's no database. Each "table" is a different website, API, or scraped dataset with its own format, authentication, and access pattern. The agent is the query engine. This is what Web 4.0 actually looks like. Not a new protocol. Not a blockchain. Not a headset. It's: **open databases + agents that query them + natural language as the interface.** ## The Paid Data Web In a mature version of this, the data layer looks like pay-per-query APIs: | Query | Source | Cost | |-------|--------|------| | Cities with AQI < 50 in April | AQI API | $0.001 | | Flights CNX to those cities, cheapest | Amadeus/Flights API | $0.01 | | Hotels rated > 8, < $80/night | Accommodation API | $0.01 | | Temperature, rain probability | Weather API | $0.001 | The agent JOINs the results. Ranks. Presents three options. Total data cost: $0.03. Some of this already exists. Amadeus has a flight API. OpenWeatherMap charges per call. Schema.org and JSON-LD provide a common vocabulary for structured data. MCP (Model Context Protocol) is emerging as the way agents discover and call APIs. The pieces are on the table. They just aren't assembled yet. ## What's Actually Missing **Most data owners won't open up.** Airbnb's moat IS the aggregation. They spent billions building a two-sided marketplace — publishing a $0.01/query API would commoditize their entire business overnight. Same for Booking.com, same for most OTAs. Their value isn't the data; it's the lock-in. **No universal payment rail for agents.** Specs like x402 propose HTTP-native micropayments — agent sends a request, gets a 402 Payment Required, pays, gets data. The mechanism (crypto, Stripe, carrier billing) matters less than the fact that none of it works today. My agent can't pay $0.001 for an AQI query. **No discovery mechanism.** How does an agent find out that some startup in Bangkok has a Thai hotel pricing API? MCP helps with tool discovery if you already know the server exists. But there's no registry. No DNS for data APIs. **Who builds the databases that don't want to be built?** Hotels won't publish real-time pricing. Airlines charge a fortune through GDS intermediaries. Airbnb will never expose an API. The most valuable data is precisely the data that's most profitable to keep proprietary. ## The Uncomfortable Middle So my agent didn't use clean APIs. It scraped. It reverse-engineered. It bypassed JavaScript rendering. It decoded proprietary formats. This is the reality of "open data" today. Most of it is: - Technically accessible but behind JS rendering (requires headless browser) - Encoded in custom formats (requires reverse engineering) - Behind auth walls (requires session management) - Rate-limited (requires patience and retries) - Spread across HTML meant for humans (requires parsing) The agent is the lockpick. And that's both the promise and the problem. There are two paths forward: **Path 1: Liberation.** Someone builds the `aqi-liberator` for everything. Scrape it, normalize it, serve it through clean APIs. Legal gray area. Constant cat-and-mouse with anti-bot measures. Fragile. But it works today. **Path 2: Persuasion.** Convince data owners that earning $0.01/query from a million agents beats protecting a walled garden that's slowly losing traffic to agent-mediated experiences anyway. If users stop visiting Booking.com because their agent checks prices directly, the walled garden strategy is already failing. My bet: Path 1 forces Path 2. Once enough data gets liberated through scraping, the original sources will realize they might as well monetize the access directly rather than fighting it. The music industry fought piracy for a decade before Spotify proved that convenient paid access beats enforcement. The same economics apply here — it's cheaper to serve an API than to fight scrapers. ## The Part No Dataset Covers Here's the twist. The agent did its job brilliantly. Four hours of work that would have taken me two full days of tab-switching. The data clearly said: go to Rayong. Clean air, cheapest option for 6 people, decent accommodation. The optimal answer given the data. We stayed in Chiang Mai. Why? I'm building a startup. My wife's entire support network — the babysitter, classes, playgrounds, other moms — is here. Pulling three kids out of their routine for two weeks has costs that don't show up in any API. The stress of packing and traveling with young children is not a queryable metric. And honestly, with an air purifier, it's survivable. The agent can tell you the optimal move. It can't tell you whether the optimal move is the right one. Data-driven decision making has a ceiling, and it's lower than the data evangelists want to admit. The last 20% is judgment — context about your life that no dataset covers and no prompt can fully convey. The future isn't agents making decisions for us. It's agents collapsing the research phase from days to minutes so humans can spend their time on the part that actually matters: deciding what to do with imperfect information, competing priorities, and the messy reality of being a person with obligations that don't fit in a spreadsheet. That's Web 4.0. Not replacing human judgment. Giving it better inputs, faster. --- ## Agent Experience (AX): 10 Principles for CLI Tools AI Agents Can Actually Use URL: https://evoleinik.com/posts/vx-launch/ Date: 2026-02-27 Tags: cli, vercel, ai-agents, agent-experience, bun, developer-tools We have UX. We have DX. But AI agents are now the primary users of developer tools, and nobody's designing for them. I run a startup where Claude Code is the primary CI/CD operator — it deploys, checks logs, reads env vars, searches session history. The Vercel CLI was the first tool it had to use, and it was a disaster. `vercel logs` hangs for 5 minutes then times out. `vercel env pull` silently overwrites `.env.local`. `vercel link` rewires your project config without asking. No `--json` on most commands. Every feature designed to help humans was actively breaking the agent. So I rebuilt it. And then I rebuilt a second tool. And the pattern became obvious: the features developers love most — interactive wizards, spinners, guided flows, colorful output — are the exact things that break agents. I'm calling the discipline **Agent Experience (AX)** — ten principles I landed on after watching real agent behavior through usage telemetry. ## The ten principles **1. Minimize output — every token costs context.** An agent's context window is its short-term memory. Every separator line, every padding character, every decoration pushes useful information out. I removed dash separator lines from table output after realizing they served zero purpose for agents and wasted tokens for humans too. **2. Structured output by default.** `--json` on every command. JSON preserves the structure the code already has internally. Telemetry confirmed this: 38% of all agent calls use `--json`. They strongly prefer structured output when it's available. **3. stdout for data, stderr for noise.** Results to stdout, diagnostics to stderr. Piping works, agents get clean data. **4. No interactive prompts.** Agents can't type "Y" at a confirmation prompt. Every operation must be fully specified by arguments. **5. Fail fast and loud.** Configurable `--timeout`. Clear error message. Non-zero exit code. The agent detects failure and tries something else. **6. Never mutate implicitly.** Read-only by default. Silent side effects corrupt state in ways agents can't detect or recover from. **7. Read existing state, don't create new state.** Reuse auth tokens and config files from existing tools. Zero setup if the user already has the original tool installed. **8. Instant startup.** Agents call tools 40-60 times per session. Startup latency compounds. Compile to a binary. Sub-100ms or it's too slow. **9. Guide on failure — empty results are the worst UX.** When a tool returns nothing, the agent has zero signal. Was the query wrong? The scope too narrow? I watched agents get empty results and blindly retry with progressively broader queries — three retries on average. I added one line to stderr: `no matches for "deploy" (19 files, 7 days, current project) — try: -d 30, -a, -s`. Retry chains dropped immediately. The agent now knows what was searched and what to try differently. **10. Log usage for yourself — close the feedback loop.** Append one JSONL line per invocation — command, flags, result count, latency. Add a `--usage` command to aggregate it. Not for dashboards — for you, the tool author. Here's what the telemetry actually taught me: - Agents write flags after the argument (`tool "pattern" -n 5`) because that's how grep works. My flag parser silently ignored those flags. Telemetry caught 4 instances in one session — I added argument reordering and the issue vanished. - Agents retry the same command within 2 minutes after errors. Detecting these retry chains showed me which error messages weren't actionable enough. - Every other principle on this list was discovered or validated by reading the usage log. ## The AX checklist If you're building a CLI tool and want it to work with AI agents: - Minimize output tokens (context window is finite) - `--json` on every command - stdout = data, stderr = logs - No interactive prompts - Deterministic exit codes - `--timeout` on network operations - Clear, parseable error messages - Read-only by default - Idempotent operations - Fast startup (sub-100ms) - Guide on empty results (print scope + suggestions to stderr) - Log usage locally (you can't improve what you can't observe) ## AX is mostly Unix, rediscovered When I stepped back and looked at these ten principles, I realized most of them aren't new. They're the Unix philosophy, written in 1978, applied to a user that didn't exist yet. | AX Principle | Unix Origin | |---|---| | Minimize output | Rule of Silence — "say nothing unless you have something surprising to report" | | stdout/stderr separation | Unix invented this | | No interactive prompts | Pipe-friendly by design — tools that prompt break pipelines | | Fail fast and loud | Rule of Repair — "fail noisily and as soon as possible" | | Read existing state | Shared config: `/etc/`, env vars, dotfiles | | Instant startup | Small, focused tools that do one thing | | Never mutate implicitly | Principle of least surprise | The Unix designers solved most of these problems 50 years ago. Then we forgot them. We added spinners, wizards, interactive flows, colored output — because we were designing for humans sitting at terminals. Now the user is an AI agent, and we're back to needing exactly what Unix always wanted: small tools, text streams, clean interfaces, no surprises. **What's different — same principles, different costs:** - **JSON over plain text.** Not because agents need "richer structure" — agents parse text fine. The problem is **determinism**. `ls -la` output varies by OS, locale, terminal width, ANSI escape codes. JSON doesn't. Unix said "write programs to handle text streams" because text was universal. JSON is the new universal — for machine consumers, unambiguous beats human-readable. - **Guide on failure.** Agents can read man pages. But loading a man page costs context window tokens. An inline hint is a man page compressed to one line — same information, 100x cheaper. It's the same tradeoff Unix made with terse error messages over verbose help, just with a different cost function. - **Usage telemetry.** Genuinely new. Unix tools don't self-instrument. Watching how agents actually use your tool — what flags they pass, where they retry, what returns empty — is a feedback loop Unix never had. - **Output has a cost — again.** Unix designers knew output had a cost: slow terminals, paper tape, 300 baud modems. That's why they wrote the Rule of Silence. We forgot because modern terminals are instant. Now output costs again — every token is real money when an LLM reads it. Same principle, different price tag. The meta-insight: good AX is boring. Predictable, structured, silent, deterministic. Not a new idea — a very old idea, rediscovered because the cost of output became real again. --- ## How We Make Claude Remember: Learnings Over Skills URL: https://evoleinik.com/posts/ai-agent-learnings/ Date: 2026-02-02 Tags: ai, agents, claude-code, skills, productivity ## Background Claude Code reads a CLAUDE.md file at project root for context. "Skills" are reusable prompt templates Claude can invoke. But Claude itself resets between sessions - it doesn't remember what it learned yesterday. ## The Problem We created 10+ skills to teach Claude project-specific knowledge. But skills don't reliably auto-invoke. Concrete example: I had an `airshelf-vercel` skill with explicit instructions: "Don't run `vercel --prod` - push to git instead." I asked Claude to deploy. It ran `vercel --prod`. Repeatedly. The skill existed. Claude never loaded it. **"Why not just use Skills?"** I've seen this feedback. We tried. Skills work great for workflows you explicitly invoke (`/commit`, `/review-pr`). But for factual knowledge Claude needs mid-task? Skills require Claude to remember which of 10 skills to invoke. It often doesn't. Learnings require one generic pattern: `grep -r "keyword" learnings/`. ## The Solution A three-layer system: **1. learnings/ folder** - Topic-specific files (database.md, stripe.md, vercel.md) for facts and gotchas. CLAUDE.md tells Claude these exist and how to search them. Not auto-loaded, but always discoverable. **2. curate-docs skill** - A structured process for capturing knowledge after features. Why a skill and not a script? Because curation requires judgment - deciding what goes where: - Critical gotchas → CLAUDE.md (1-liners, always loaded) - Detailed knowledge → learnings/ (searchable on demand) - Repeatable workflows → skills (explicitly invoked) **3. Post-commit hook** - Claude Code supports hooks that run after specific tool calls. Ours fires after `git commit`, but only on feature branches with commits ahead of main: ``` "Feature branch 'feat/auth' has 3 commits. Consider running /curate-docs." ``` Targeted reminder, not noise. Without it, I forgot to document. With it, I don't. ## Does It Work? **When it works:** I hit a Prisma migration error, searched `grep -r "Neon branch" learnings/`, found the exact workaround I'd documented weeks earlier. **When it fails:** When Claude doesn't think to search. This still happens - roughly 1 in 5 times. Prompting helps ("check learnings/ for this error"). But it works far more often than skills Claude had to remember to invoke. ## Get It The curate-docs skill and hook: [github.com/evoleinik/curate-docs](https://github.com/evoleinik/curate-docs) ```bash npx skills add evoleinik/curate-docs ``` ## Takeaway Skills = workflows you invoke. Learnings = facts Claude searches. Don't rely on skills alone for persistent knowledge. Use searchable learnings files combined with a hook that reminds you to curate. --- ## Serve Markdown to AI Agents (10x Smaller Payloads) URL: https://evoleinik.com/posts/markdown-for-agents/ Date: 2026-02-02 Tags: ai, agents, web, markdown, http Guillermo Rauch shared that Vercel's changelog now serves markdown when agents request it. Same URL, different `Accept` header. The insight isn't the size reduction - it's that an entire infrastructure layer (CSS, JS, frameworks) is becoming optional for a growing class of consumers. ## How it works HTTP content negotiation. Browsers send `Accept: text/html`. Agents can send `Accept: text/markdown`. Same URL, different representation. I added this to my Hugo blog. The config: ```toml [outputs] page = ['HTML', 'MARKDOWN'] [outputFormats.MARKDOWN] baseName = 'index' mediaType = 'text/markdown' isPlainText = true ``` The middleware (Vercel Edge): ```js export const config = { matcher: ['/', '/posts/:path*'] } export default async function middleware(request) { if (request.headers.get('accept')?.includes('text/markdown')) { const url = new URL(request.url) url.pathname = url.pathname.replace(/\/?$/, '/index.md') return fetch(url) } } ``` Test it: ```bash curl -H 'Accept: text/markdown' https://evoleinik.com/posts/markdown-for-agents/ ``` My posts go from ~20kb HTML to ~2kb markdown. Not 250x like Vercel's changelog, but 10x adds up. ## The tradeoff You maintain two output formats. For static sites like Hugo, this is trivial - markdown is the source anyway. For dynamic content or SPAs, it's harder. You'd need to generate markdown server-side or maintain parallel content. ## Why bother? Agent traffic is growing. Lightweight, structured content gives agents cleaner context and burns fewer tokens. The visual web was designed for human browsers. The agent web doesn't need the decoration. --- ## The AI Data Trap: Why You Can't Opt Out URL: https://evoleinik.com/posts/ai-data-trap/ Date: 2026-01-28 Tags: ai, privacy, startup, strategy Two years ago I asked my CEO if we should use ChatGPT. "It leaks everything we're doing," I said. His answer: "Everyone's using it. If we don't, we're behind." He was right. That's the trap. ## The Competitive Ratchet Your competitor uses Claude or GPT to move faster. If you don't, you fall behind. So you use the tools. Everyone does. And every conversation, every codebase, every strategy goes into their servers. The ratchet only turns one way. Better models become essential. Essential means more data. More data means better models. Self-hosted alternatives fall further behind. ## Not All AI Companies Carry the Same Risk Anthropic only does AI. They're not building a competing product in your market. Google does everything. They see you building a travel startup through Gemini - that's competitive intelligence feeding a company that might crush you in that exact space. OpenAI has the governance chaos, the Microsoft relationship, the pivot from nonprofit to "capped profit" to whatever comes next. The safety branding is real. Whether it matches reality is a different question. You'll keep using these tools. So will everyone else. That's the trap. --- ## The Best Agent Architecture Is Already in Your Terminal URL: https://evoleinik.com/posts/filesystem-agent-context/ Date: 2026-01-12 Tags: ai, agents, claude, developer-tools, architecture # The Best Agent Architecture Is Already in Your Terminal My project's CLAUDE.md file had grown to 55KB—242 learnings crammed into one massive file. The problem? Claude prepends this file to every single prompt. A 55KB context file means less room for thinking and acting. Sessions hit context limits faster. Compaction happens sooner. I noticed the degradation: sessions became noticeably shorter, context compaction triggered more frequently, and the agent seemed to lose track of longer conversations. Here's the kicker: Claude Code's system prompt actually tells Claude not to take CLAUDE.md too seriously if it's too large. The system is designed to deprioritize oversized context files. So not only was I wasting context space—the agent was being instructed to partially ignore my carefully curated learnings anyway. The fix took about an hour: split into a `learnings/` folder with one file per tool. Simple navigation: ```bash ls learnings/ # List available files grep -r "webhook" learnings/ # Search all learnings cat learnings/stripe.md # Read specific tool ``` Then Vercel published an article that validated exactly this approach: [How to build agents with filesystems and bash](https://vercel.com/blog/how-to-build-agents-with-filesystems-and-bash). ## The Key Insight LLMs have been trained on massive amounts of code. They've spent countless hours navigating directories, grepping through files, and managing state across complex codebases. **If agents excel at filesystem operations for code, they'll excel at filesystem operations for anything.** Vercel's sales call summarization agent went from ~$1.00 to ~$0.25 per call by replacing custom tooling with filesystem + bash. Quality improved too. ## Why This Works for Project Context The typical approach is stuffing everything into the prompt. But every byte in your CLAUDE.md is a byte the model can't use for reasoning. Filesystems offer: - **On-demand loading.** Agent reads only what it needs, when it needs it. - **Precise retrieval.** `grep -r "webhook" learnings/` returns exact matches. - **Structure that matches your domain.** Learnings have natural hierarchies by tool. ## My New Structure ``` learnings/ README.md # Index + navigation guide stripe.md # Webhooks, CLI, subscriptions vercel.md # Deploys, env vars, cron prisma.md # CRITICAL column drops, migrations clerk.md # Auth, users, organizations axiom.md # Logging, monitors, alerts nextjs.md # Routing, caching, layouts playwright.md # E2E testing, selectors ai-providers.md # OpenAI, Gemini quirks database.md # PostgreSQL, psql patterns git.md # Hooks, GitHub Actions neon-setup.md # Database branching setup misc.md # Everything else ``` CLAUDE.md: 55KB → 24KB. All 251 learnings preserved and searchable. More headroom for actual work. ## The Pattern 1. **Keep always-loaded context minimal.** Only critical gotchas in CLAUDE.md. 2. **Structure knowledge as files.** One file per domain/tool. 3. **Let the agent navigate.** `ls`, `grep`, `cat` are native skills. The agent treats your knowledge base like a codebase—searching for patterns, reading sections, building context just like debugging code. As Vercel puts it: "The future of agents might be surprisingly simple. Maybe the best architecture is almost no architecture at all. Just filesystems and bash." --- ## Zero-Friction Database Branching with Neon, Git Hooks, and Claude Code URL: https://evoleinik.com/posts/neon-git-branching/ Date: 2026-01-07 Tags: postgres, neon, devtools, ai-development, claude-code, git # Zero-Friction Database Branching with Neon, Git Hooks, and Claude Code I've been refining my Neon database branching setup over the past few months. Here's the current state: fully automated branch lifecycle with zero manual cleanup. ## The Goal When I `git checkout -b feat/x`: 1. Neon database branch created automatically 2. `.env.local` updated with the new connection string 3. Vercel preview deployment uses the same isolated database When I merge and delete the branch: 1. Orphaned Neon branches cleaned up automatically 2. No manual intervention needed ## The Stack - **Neon** - Serverless Postgres with instant copy-on-write branching - **neonctl** - Neon's CLI (much cleaner than curl API calls) - **Git hooks** - post-checkout and pre-push automation - **Claude Code** - AI assistant that follows the "never work on main" rule ## Environment Mapping ``` Git Branch │ Neon Branch │ Vercel ──────────────┼─────────────────┼────────────── main │ production │ Production feat/* │ feat/* │ Preview ``` ## The Setup ### 1. Install neonctl ```bash npm install -g neonctl ``` Authentication uses the `NEON_API_KEY` environment variable - no browser login needed for headless servers. ### 2. Post-Checkout Hook (Branch Creation + Auto-Cleanup) ```bash #!/bin/bash # .githooks/post-checkout [ "$3" == "0" ] && exit 0 # Skip file checkouts BRANCH_NAME=$(git symbolic-ref --short HEAD 2>/dev/null) || exit 0 source .env.local 2>/dev/null || exit 0 [ -z "$NEON_PROJECT_ID" ] && exit 0 [ -z "$NEON_API_KEY" ] && exit 0 export NEON_API_KEY update_env() { local uri="$1" local escaped_uri="${uri//&/\\&}" # Escape & for sed sed -i "s|^DATABASE_URL=.*|DATABASE_URL=\"$escaped_uri\"|" .env.local sed -i "s|^DIRECT_DATABASE_URL=.*|DIRECT_DATABASE_URL=\"$escaped_uri\"|" .env.local } # Protected branches → production database if [[ "$BRANCH_NAME" =~ ^(main|master)$ ]]; then PROD_URI=$(neonctl connection-string production --project-id "$NEON_PROJECT_ID") update_env "$PROD_URI" echo "neon: $BRANCH_NAME → production" # Auto-cleanup orphaned Neon branches NEON_BRANCHES=$(neonctl branches list --project-id "$NEON_PROJECT_ID" -o json | \ jq -r '.[].name | select(. != "production")') for neon_branch in $NEON_BRANCHES; do if ! git branch -a | grep -qE "(^[* +] +|/)${neon_branch}$"; then neonctl branches delete "$neon_branch" --project-id "$NEON_PROJECT_ID" && \ echo "neon: deleted orphan $neon_branch" fi done exit 0 fi # Feature branch → get or create Neon branch CONNECTION_URI=$(neonctl connection-string "$BRANCH_NAME" --project-id "$NEON_PROJECT_ID" 2>/dev/null) if [ -n "$CONNECTION_URI" ]; then update_env "$CONNECTION_URI" echo "neon: $BRANCH_NAME → existing branch" else neonctl branches create --project-id "$NEON_PROJECT_ID" --name "$BRANCH_NAME" --parent production CONNECTION_URI=$(neonctl connection-string "$BRANCH_NAME" --project-id "$NEON_PROJECT_ID") update_env "$CONNECTION_URI" echo "neon: created $BRANCH_NAME" fi ``` The magic is in the cleanup section: when you checkout `main`, the hook scans for Neon branches that no longer have a matching git branch and deletes them. ### 3. Pre-Push Hook (Vercel Sync + Parallel Checks) ```bash #!/bin/sh # .githooks/pre-push BRANCH=$(git symbolic-ref --short HEAD) # Sync DATABASE_URL to Vercel preview (background) ( case "$BRANCH" in main|master) ;; *) DB_URL=$(grep '^DATABASE_URL=' .env.local | sed 's/^DATABASE_URL=//' | tr -d '"') if [ -n "$DB_URL" ]; then printf "%s" "$DB_URL" | vercel env add --force DATABASE_URL preview "$BRANCH" echo "vercel: synced DATABASE_URL for preview/$BRANCH" fi ;; esac ) & # Run checks in parallel npm test & PID_TEST=$! npm run lint & PID_LINT=$! wait $PID_TEST || exit 1 wait $PID_LINT || exit 1 echo "All checks passed!" ``` ### 4. Status Command See which git branches have corresponding Neon branches: ```bash $ git neon-status Branch Git Neon ────────────────────────────────────────────────── main ✓ (production) feat/new-api ✓ ✓ feat/old-branch ✓ ← no DB orphan-neon-branch ✓ ← orphan ``` Add the alias: ```bash git config --global alias.neon-status '!./scripts/neon-status.sh' ``` ## The Workflow ```bash # Start feature git checkout -b feat/new-api # "neon: created feat/new-api" # Work freely - isolated database npm run dev # Push for review git push -u origin feat/new-api # "vercel: synced DATABASE_URL for preview/feat/new-api" # Preview at feat-new-api.vercel.app uses YOUR database # Merge PR, delete branch git checkout main git branch -d feat/new-api # "neon: deleted orphan feat/new-api" ← automatic! ``` No manual cleanup. The orphaned Neon branch is deleted next time you checkout main. ## Claude Code Integration The key rule in my `CLAUDE.md`: ```markdown RULES: - NEVER work directly on main branch - always create a feature branch first - Main is for merging and deploying only, not development ``` This ensures Claude always runs `git checkout -b feat/...` before making changes. Combined with Neon branching: - AI experiments on isolated database - Production is never touched - Mistakes are contained to the feature branch ## Why This Matters With AI assistants writing code, they often need to: - Run migrations - Seed test data - Execute queries to verify changes On a shared database, this is terrifying. With Neon branching + the "always branch" rule: - Every feature gets an isolated database copy - AI can freely experiment - Production stays clean - Cleanup is automatic ## Quick Reference | Command | What Happens | |---------|--------------| | `git checkout -b feat/x` | Creates Neon branch, updates .env.local | | `git push` | Syncs DB URL to Vercel preview | | `git checkout main` | Switches to prod DB, cleans orphans | | `git neon-status` | Shows branch mapping | | `git nuke feat/x` | Deletes git + Neon branch (manual) | ## neonctl Cheatsheet ```bash # List branches neonctl branches list --project-id "$NEON_PROJECT_ID" # Get connection string neonctl connection-string "branch-name" --project-id "$NEON_PROJECT_ID" # Create branch neonctl branches create --name "branch-name" --parent production --project-id "$NEON_PROJECT_ID" # Delete branch neonctl branches delete "branch-name" --project-id "$NEON_PROJECT_ID" ``` --- The full setup is in my dotfiles. The combination of Neon's instant branching, git hooks for automation, and Claude's "always branch" rule gives me confidence to let AI assistants work on my codebase without fear of production accidents. --- ## The Iteration Trap: When AI Makes You a Spectator URL: https://evoleinik.com/posts/iteration-trap/ Date: 2025-12-28 Tags: ai, productivity, llm # The Iteration Trap: When AI Makes You a Spectator ## Key Facts - The trap: iterate with AI → stop reading output → run more cycles → hope for magic - Root cause: you don't have clear acceptance criteria - you're iterating because you don't know what you want - The iteration trap is really a clarity trap - "Read after every cycle" treats the symptom, not the cause - The real failure: "I couldn't articulate what was wrong with it" ## The Story - Ran playbook generation with two LLMs connected - Kept iterating, expecting something great to emerge - Stopped actually reading what was being produced - Just wanted to run one more cycle, wait for the magic - Finally read the output: "I don't think it's really good... just presenting the data" - The iteration felt productive. The output wasn't. ## The Pattern ``` idea → iterate with AI → stop reading → run more cycles → hope for magic → finally read output → meh ``` You became a spectator hoping the slot machine pays out. The AI was generating, you were waiting, nobody was thinking. ## Why This Happens - You don't have clear acceptance criteria - without knowing what "good" looks like, you can't evaluate - Iteration feels like progress - dopamine hit of activity without cognitive load of evaluation - Evaluation is harder than generation - requires you to have a mental model of what you want - Variable reward schedule - unpredictable output quality creates slot machine dynamics ## The Real Fix "Read after every cycle" treats the symptom, not the cause. You can read and still be trapped - nodding along, prompting again because "it's not quite right" without knowing why. **Structural fixes that actually work:** 1. **Define done before you start** - Write 2-3 specific criteria for what "good" looks like. Not vibes - concrete checkboxes. 2. **Constrain cycles upfront** - "I will do 3 iterations max." Forces you to evaluate seriously because you're spending a finite budget. 3. **Externalize the evaluation** - After each output, write: "This is/isn't acceptable because ___." If you can't fill in the blank, you don't have criteria. Stop iterating and go define them. 4. **Default to single-shot** - Treat iteration as expensive. If you need 5+ cycles, the problem is upstream (unclear requirements, wrong tool, insufficient context). ## The Reframe The iteration trap is really a **clarity trap**. You iterate because you don't know what you want. If you find yourself iterating without reading, stop and ask: "What would make the next output obviously acceptable or obviously unacceptable?" If you can't answer that, you're not ready to iterate. You're ready to think. ## Takeaway - Iteration without criteria is just busy work - The moment you hope for magic, stop - Define what "done" looks like before you start - If you can't articulate what's wrong, the problem isn't the output - it's your clarity --- ## Adding LLM Polish to a Speech-to-Text App URL: https://evoleinik.com/posts/adding-llm-polish-to-speech-to-text/ Date: 2025-12-22 Tags: rust, macos, llm, speech-to-text, groq Voice transcription is messy. Even the best models like Whisper faithfully reproduce every "um", "uh", and rambling run-on sentence. That's correct behavior for transcription, but not what you want when texting someone. I added a "polish mode" to my macOS speech-to-text app that optionally sends Whisper's output through an LLM to clean it up. The interaction model: hold Fn to record, tap Ctrl anytime during recording to enable polish, release to transcribe and paste. ## The Modifier Key Challenge The obvious approach - require Ctrl held simultaneously with Fn - felt clunky in testing. You'd have to coordinate two fingers before speaking, and the physical position is awkward. A "latch" pattern works better: pressing Ctrl anytime while Fn is held latches the polish flag. You can press Ctrl before speaking, during, or just before release. The flag resets when you start a new recording. ```rust let ctrl_latched = Arc::new(AtomicBool::new(false)); // In the event tap callback: if key_pressed && !prev_pressed { // Recording started - reset latch ctrl_latched.store(false, Ordering::SeqCst); start_recording(&state); } else if !key_pressed && prev_pressed { // Recording stopped - check if Ctrl was ever pressed let polish = ctrl_latched.load(Ordering::SeqCst); stop_recording(&state, polish); } // Latch Ctrl if pressed anytime during recording if key_pressed && ctrl_pressed { ctrl_latched.store(true, Ordering::SeqCst); } ``` The macOS `CGEventFlags` expose modifier state as bitmasks. Control is `0x40000`: ```rust const CONTROL_KEY_FLAG: u64 = 0x40000; let flags = event.get_flags().bits(); let ctrl_pressed = (flags & CONTROL_KEY_FLAG) != 0; ``` ## The Polish Function The polish step is a straightforward LLM API call. I'm using Groq's hosted llama-3.3-70b-versatile because I'm already using Groq for Whisper transcription - one API key, one vendor. ```rust fn polish_text(text: &str, api_key: &str) -> Option { let client = reqwest::blocking::Client::new(); let body = serde_json::json!({ "model": "llama-3.3-70b-versatile", "messages": [ { "role": "system", "content": "Clean up this voice message for texting. Remove filler words (um, uh, like, you know). Fix punctuation and sentence structure. Break up run-on sentences. Keep it casual. No trailing period. Output ONLY the cleaned text - no explanations, no quotes." }, { "role": "user", "content": text } ], "temperature": 0.2 }); let response = client .post("https://api.groq.com/openai/v1/chat/completions") .header("Authorization", format!("Bearer {}", api_key)) .header("Content-Type", "application/json") .json(&body) .timeout(Duration::from_secs(30)) .send() .ok()?; if !response.status().is_success() { return None; } let chat_response: ChatResponse = response.json().ok()?; chat_response.choices.first().map(|c| c.message.content.clone()) } ``` The function returns `Option` - this matters for the fallback logic. ## Parsing the Response Groq uses the OpenAI-compatible chat completions format. The response structure: ```rust #[derive(serde::Deserialize)] struct ChatResponse { choices: Vec, } #[derive(serde::Deserialize)] struct ChatChoice { message: ChatMessage, } #[derive(serde::Deserialize)] struct ChatMessage { content: String, } ``` Using `serde` to parse into typed structs catches malformed responses at parse time rather than panicking on field access later. ## Prompt Engineering Lessons The system prompt went through several iterations: **First attempt:** "Clean up this transcription." Problem: The LLM would respond conversationally. "Sure! Here's the cleaned up version: ..." **Second attempt:** "Output only the cleaned text." Problem: It would wrap the output in quotes: `"Here's what I meant to say"` **Third attempt:** Added explicit prohibitions. ``` Output ONLY the cleaned text - no explanations, no quotes. ``` This worked. The key insight: LLMs default to being helpful and conversational. For tool use, you need to explicitly tell them to suppress that behavior. Other prompt decisions: - **"Keep it casual"** - prevents the LLM from making the text overly formal - **"No trailing period"** - texting convention; a period at the end feels curt - **"Break up run-on sentences"** - spoken language naturally runs together Low temperature (0.2) keeps output consistent. Higher temperatures occasionally produced creative reinterpretations of what I said. ## Graceful Degradation The polish step can fail: network issues, rate limits, API changes. The user still expects their transcription to paste. ```rust let final_text = if polish { polish_text(text, api_key).unwrap_or_else(|| text.to_string()) } else { text.to_string() }; ``` `Option::unwrap_or_else` is the right pattern here. If polish fails for any reason, fall back to the raw Whisper transcription. The user gets something rather than nothing. This is a general principle for LLM features: treat them as enhancements, not requirements. The core functionality should work without them. ## Latency Considerations Polish adds a second API call, roughly 200-400ms on Groq. For a texting use case, this is acceptable - you're not in a real-time conversation. For live captioning or dictation into a text field, it would be too slow. The transcription already happens in a background thread: ```rust thread::spawn(move || { transcribe_and_paste(audio_data, sample_rate, &api_key, polish); }); ``` Both the Whisper call and the polish call happen sequentially in this thread. The UI remains responsive; the user just waits slightly longer for paste. ## Trade-offs **When polish helps:** - Texting, where filler words and run-ons look sloppy - Drafting messages you want to sound more coherent - Quick notes that benefit from basic cleanup **When to skip it:** - Dictating into forms or code comments - When you want exact transcription (quotes, interviews) - Low-latency scenarios **What polish can break:** - Proper nouns and technical terms may get "corrected" - The LLM might misinterpret intent on ambiguous input - Short inputs ("ok", "yes") sometimes get expanded unnecessarily The latch pattern makes this an explicit user choice. Default is raw transcription; polish is opt-in. ## Conclusion - **Latch pattern beats simultaneous press** - let users enable modes at any point during an action - **Explicit prompt constraints** - tell the LLM what NOT to do (no explanations, no quotes) - **Low temperature for tools** - you want consistency, not creativity - **Graceful fallback is mandatory** - LLM features should enhance, not gate, core functionality - **Choose your latency budget** - 200-400ms is fine for async use cases, not for real-time --- **Related:** [Building an AI-Powered Changelog GitHub Action](/posts/ai-changelog-github-action/) - Similar prompt engineering patterns for developer tooling. --- ## Building an AI-Powered Changelog GitHub Action URL: https://evoleinik.com/posts/ai-changelog-github-action/ Date: 2025-12-22 Tags: github-actions, ai, open-source, devops I wanted daily changelog summaries posted to Slack for my project. The existing solutions were either too complex (full-blown release management) or too dumb (just listing commits). I needed something that would read commits and produce a human-readable summary of what actually shipped. So I built one. Then I open-sourced it: [evoleinik/changelog-summary](https://github.com/marketplace/actions/changelog-summary). ## The Problem Raw commit logs are noisy. Even with good commit messages, a list of 15 commits doesn't tell a busy founder or stakeholder what actually changed. You want something like: > - Shipped multi-provider dashboard with real-time sync > - Fixed authentication bug causing logout loops > - Improved search performance by 3x Not: > - fix: handle null case in auth middleware > - refactor: extract dashboard component > - feat: add provider selector to dropdown > - fix: remove console.log > - ... LLMs are good at this. They can read commit messages (including the body, not just the subject line) and synthesize what matters. ## From Inline Script to Reusable Action My first implementation was 87 lines of bash embedded directly in my GitHub Actions workflow file. It worked, but the workflow file became unreadable. The extraction took about an hour. The result: ```yaml - uses: evoleinik/changelog-summary@v1 with: slack-webhook: ${{ secrets.SLACK_WEBHOOK_URL }} llm-provider: gemini llm-api-key: ${{ secrets.GEMINI_API_KEY }} voice: founder ``` 24 lines instead of 87, and now any project can use it. ## Implementation Details The action is a composite action (pure bash, no Node.js runtime). This matters because: 1. **No build step** - the script runs directly 2. **Easier to audit** - it's just bash you can read 3. **Faster startup** - no npm install ### Reading Full Commit Messages Most changelog tools only read commit subjects. But the body often contains the real context: ```bash COMMITS=$(git log --since="$SINCE" --pretty=format:"- %s%n%b" --no-merges) ``` The `%b` gives you the commit body. This means the LLM can see: ``` - feat: add multi-provider support Added support for Gemini, OpenAI, and Anthropic. Users can now switch providers without code changes. Breaking: removed deprecated single-provider config. ``` Instead of just "feat: add multi-provider support". ### Voice Styles Different audiences need different summaries. I implemented three: **founder** - Direct, no-BS. What shipped? Skip the implementation details. ``` Be direct - what actually shipped? No fluff, no 'exciting updates' BS. ``` **developer** - Technical focus. APIs, breaking changes, specific files changed. **marketing** - User-facing improvements. New capabilities, not bug fixes. The prompt engineering is straightforward: ```bash case "$VOICE" in founder) PROMPT="Summarize these commits for a busy founder. Be direct - what actually shipped? Rules: 3-5 bullets, no fluff..." ;; developer) PROMPT="Summarize these commits for developers. Focus on technical changes: APIs, breaking changes..." ;; esac ``` ### Slack Formatting Gotcha Slack uses single asterisks for bold (`*text*`), not double (`**text**`). This took a few iterations to get right in the prompt: ``` Use Slack formatting: * for bullets, surround key terms with single asterisks for bold. ``` ### Multi-Provider Support I defaulted to Gemini because it's free tier is generous and the quality is good. But the action supports OpenAI and Anthropic too: ```bash case "$LLM_PROVIDER" in gemini) curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-3-flash-preview:generateContent?key=$LLM_API_KEY" ... ;; openai) curl "https://api.openai.com/v1/chat/completions" -H "Authorization: Bearer $LLM_API_KEY" ... ;; anthropic) curl "https://api.anthropic.com/v1/messages" -H "x-api-key: $LLM_API_KEY" ... ;; esac ``` Each provider has slightly different JSON structures, but `jq` handles the response parsing cleanly. ## Trade-offs **No streaming** - The action waits for the full LLM response. For changelog summaries (typically under 200 tokens), this is fine. For longer documents, you'd want streaming. **Single Slack message** - No threading, no reactions. Just a message. I could add richer Slack blocks, but the simple text format works and is easier to maintain. **No commit filtering** - Every commit in the time range gets included. If you need to filter by path or author, you'd need to modify the `git log` command. I may add this as an option if there's demand. **Bash-based** - This limits what you can do. A TypeScript action would be more extensible. But bash means zero dependencies and sub-second startup. For a simple utility, that's the right trade-off. ## Usage Examples ### Daily Summary ```yaml on: schedule: - cron: '0 13 * * *' # 1 PM UTC daily jobs: summary: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: fetch-depth: 0 # Need full git history - uses: evoleinik/changelog-summary@v1 with: slack-webhook: ${{ secrets.SLACK_WEBHOOK_URL }} llm-provider: gemini llm-api-key: ${{ secrets.GEMINI_API_KEY }} ``` ### Weekly Summary with Custom Header ```yaml on: schedule: - cron: '0 13 * * 0' # Sundays jobs: summary: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: fetch-depth: 0 - uses: evoleinik/changelog-summary@v1 with: slack-webhook: ${{ secrets.SLACK_WEBHOOK_URL }} llm-provider: gemini llm-api-key: ${{ secrets.GEMINI_API_KEY }} since: '7 days ago' header: 'Weekly Update' voice: marketing ``` ## What Makes Good Open Source This started as a script to solve my own problem. A few observations from the extraction process: 1. **Solve your problem first** - I used this for weeks before open-sourcing. The edge cases were already handled. 2. **Keep it focused** - This does one thing: summarize commits and post to Slack. It doesn't manage releases, create tags, or update changelogs files. 3. **Provide sensible defaults** - Gemini as the default provider, "founder" voice, 24-hour window. You can override everything, but the defaults work out of the box. 4. **Document the trade-offs** - Be clear about what it doesn't do. ## Conclusion - Small, focused utilities that solve your own problem first often make good open source - Composite actions (bash) are underrated - no build step, easy to audit, fast - Read full commit messages (`--pretty=format:"%s%n%b"`) for better AI context - Voice/persona prompts let you tune the output for different audiences without changing the code - Slack uses single asterisks for bold - check your target platform's formatting The action is on [GitHub Marketplace](https://github.com/marketplace/actions/changelog-summary). MIT licensed. PRs welcome. --- **Related:** [Adding LLM Polish to a Speech-to-Text App](/posts/adding-llm-polish-to-speech-to-text/) - More LLM integration patterns for user-facing tools. --- ## CLAUDE.md: Building Persistent Memory for AI Coding Agents URL: https://evoleinik.com/posts/claude-md-as-agent-memory/ Date: 2025-12-22 Tags: claude-code, ai-agents, developer-tools, productivity AI coding agents have a memory problem. Every new session starts from zero. The agent that spent 20 minutes yesterday figuring out your project's quirky database connection string? Gone. The workaround for that Prisma edge case? Forgotten. The exact command to run tests with the right environment variables? It will rediscover it from scratch. This isn't a bug - it's the nature of stateless LLM sessions. But it's a productivity killer when you're using an AI agent daily on the same codebase. ## The Institutional Memory Problem After a few weeks of using Claude Code on a production project, I noticed a pattern: 1. Agent encounters a project-specific gotcha 2. We debug together, find the solution 3. Next session, same gotcha, same 10-minute detour Some examples from real projects: - The database URL requires a specific query parameter that breaks `psql` but works for Prisma - Tests fail silently unless you source a specific env file first - The production deploy happens via git push, not CLI command (despite the CLI being installed) - A certain API returns 404 status but still contains valid data in the body These aren't bugs I'll ever fix. They're just... how the project works. Tribal knowledge that any long-term team member would internalize. ## CLAUDE.md as Project Memory Claude Code reads a `CLAUDE.md` file at the start of every session. It's intended for project instructions, but it works equally well as a knowledge base. The insight: treat it like onboarding documentation that the AI maintains for itself. Here's the structure I've settled on: ```markdown ## Learnings - Schema changes: push to BOTH dev and prod databases - `vercel link` overwrites `.env.local` - restore from git after - DIRECT_DATABASE_URL with `?pool=true` breaks psql - param is Prisma-only - Run `npm run build` before committing - catches type errors CI would reject - Webhook returns 404 status but body contains valid data - don't check response.ok - Background tasks: use `run_in_background` param, not shell `&` - JSON fields in bash: avoid `->>` operators - fetch whole column instead ``` Each line is a compressed lesson learned. Imperative style, no fluff, one line per item. ## What Qualifies as a Learning The key is curation. Not everything belongs here. **Include:** - Error solutions specific to this project's setup - Non-obvious commands or workflows (the ones you'd forget and have to look up) - Gotchas that wasted time (especially if they'll waste time again) - File locations that were hard to find - Workarounds for third-party quirks **Exclude:** - Generic programming knowledge ("use async/await for promises") - One-time issues unlikely to recur - Things already documented in README or official docs - Verbose explanations - if it needs a paragraph, it's documentation, not a learning The test: "Would this save 5+ minutes next time the agent encounters this situation?" ## Curation Rules Left unchecked, the Learnings section becomes a dumping ground. Every session adds more. Eventually it's 200 lines of outdated advice, half of which contradicts the other half. My rules: 1. **Max 30 items** - if adding something new, remove something obsolete 2. **Merge duplicates** - two similar learnings become one 3. **Remove when fixed** - bug workaround for a bug you fixed? Delete it 4. **One line per item** - forces compression, prevents rambling 5. **Review monthly** - scan for stale entries The agent itself can help curate. At the end of a productive session: > "Capture what we learned about the webhook integration to CLAUDE.md. Check for duplicates first." It will add the new insight and often notice related items that can be merged or removed. ## The Compounding Effect After three months on a project with maintained CLAUDE.md, the difference is stark. The agent: - Knows which database to use for which command - Remembers the exact test invocation that works - Avoids the deployment mistake it made in week one - Uses the project's preferred patterns without being told It's not intelligence - it's just reading a file. But the effect is an agent that feels like a team member who's been on the project for months, not a contractor starting fresh every morning. ## Practical Workflow **During a session:** When you solve something tricky together, flag it mentally. After the fix is confirmed working: ``` Add to Learnings: Prisma Accelerate has 5MB response limit - use select not include ``` **End of session:** If the session was productive, ask for a learning capture: ``` Review this session and add any non-obvious findings to CLAUDE.md Learnings. Only add if genuinely useful for future sessions. ``` **Monthly:** Skim the Learnings section. Delete anything that: - References fixed bugs - Duplicates other items - You've never actually needed again ## What This Isn't This isn't a replacement for documentation. Complex architectural decisions, API references, deployment procedures - those belong in proper docs that humans read too. CLAUDE.md learnings are specifically for agent-to-agent knowledge transfer. The format is optimized for LLM consumption: terse, declarative, no context needed. It's also not a crutch for bad tooling. If your agent keeps forgetting how to run tests, maybe your test command is too complicated. Fix the root cause when possible; document the workaround when necessary. ## Conclusion - AI coding agents lose context between sessions - every session starts fresh - A curated Learnings section in CLAUDE.md acts as persistent memory - Include: project-specific gotchas, non-obvious workflows, time-wasting bugs - Exclude: generic knowledge, one-time issues, anything in docs - Cap at 30 items, remove outdated entries, merge duplicates - The agent can help maintain its own memory with human approval - Compound effect: after months, the agent "knows" your codebase's quirks The effort is minimal - maybe 2 minutes per session when something noteworthy happens. The payoff is an agent that stops making the same mistakes and starts feeling like it actually learns. --- **Related:** [Debugging Random Reboots with Claude Code](/posts/debugging-random-reboots-with-claude-code/) - See Claude Code's systematic debugging approach in action. --- ## Debugging Random Reboots with Claude Code: A PSU Power Limit Story URL: https://evoleinik.com/posts/debugging-random-reboots-with-claude-code/ Date: 2025-12-22 Tags: linux, hardware, debugging, claude-code, ai-assisted-development My Linux server started rebooting randomly during CPU benchmarks. I had no idea where to start, so I asked Claude Code to help diagnose. Twenty minutes later, we found the root cause and a working fix. This is a story about AI-assisted debugging - specifically, how an AI assistant's systematic approach can cut through hardware issues that would take hours of Googling. ## The Problem I was benchmarking local Whisper models for speech-to-text on a home server (Intel i9-10900K, 550W PSU). During heavy transcription loads, the system would randomly reboot. No warning, no error message - just instant power loss. I described the symptoms to Claude Code: "Server reboots randomly under CPU load. No kernel panic. What should I check?" ## The AI-Guided Diagnosis Claude Code walked me through a systematic diagnostic process. Each step built on the previous one. ### Step 1: Check the Logs ```bash journalctl -b -1 ``` Claude noted that the logs stopped abruptly mid-operation. No error, no shutdown sequence. "This is actually diagnostic," it explained. "Software crashes leave traces. Instant power loss doesn't." ### Step 2: Look for Hardware Errors ```bash dmesg | grep -i error ``` Found Machine Check Errors (MCE). Claude explained these indicate hardware-level problems: thermal, memory, or power delivery. ### Step 3: Rule Out Thermal ```bash apt install lm-sensors sensors ``` Temps showed 39-51C under load. Well within spec. Claude crossed thermal off the list. ### Step 4: Check MCE Details ```bash apt install rasdaemon ras-mc-ctl --errors ``` No active errors. The MCE messages were stale. ### Step 5: The Diagnosis Based on the evidence - logs stopping without kernel panic, load-dependent crashes, normal temps - Claude identified the likely cause: **PSU power limits**. It asked about my PSU (550W) and looked up the i9-10900K specs. Under Turbo Boost with all cores loaded, this CPU can spike to 250W+. My PSU was undersized. ## The Fix Attempts Claude suggested Intel RAPL to limit CPU power draw: ```bash # Set PL1=125W, PL2=180W echo 125000000 > /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_power_limit_uw echo 180000000 > /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_power_limit_uw ``` Still crashed. Tried lower limits (95W/125W). Still crashed. Claude explained why: "RAPL operates on millisecond timescales. Your PSU's overcurrent protection trips in microseconds. The PSU cuts power before RAPL can throttle." Software can't fix hardware that fails faster than software can react. ## The Working Fix Claude's solution: disable Turbo Boost entirely to prevent power spikes. ```bash echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo ``` System became stable. Claude then wrote a systemd service to make it persistent: ```ini # /etc/systemd/system/disable-turbo.service [Unit] Description=Disable CPU Turbo Boost After=multi-user.target [Service] Type=oneshot ExecStart=/bin/sh -c "echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo" RemainAfterExit=yes [Install] WantedBy=multi-user.target ``` ```bash systemctl daemon-reload systemctl enable disable-turbo.service ``` ## Why AI-Assisted Debugging Worked I could have Googled "random Linux reboots" and spent hours reading forum posts about kernel bugs, driver issues, and memory problems. Instead, Claude Code: 1. **Asked the right questions** - immediately focused on whether logs showed clean shutdown vs. power cut 2. **Followed a systematic process** - ruled out causes one by one instead of jumping to conclusions 3. **Knew the domain** - understood MCE errors, RAPL timing, PSU OCP behavior 4. **Explained the "why"** - didn't just give commands, but explained why RAPL couldn't work The debugging took about 20 minutes of back-and-forth. Most of that was waiting for package installs and running tests. ## The Trade-offs With Turbo disabled, the i9-10900K runs at 3.7GHz base instead of boosting to 5.3GHz. About 30% slower for my benchmarks. The proper fix is a 750W+ PSU. But for now, disabling Turbo keeps the server stable. For the Whisper benchmarks: local inference was 10-20x slower than cloud APIs (Groq) even with Turbo. The conclusion held - use cloud for production. ## Key Takeaways - **Random reboots without kernel panic = power issue**, not software. Logs stopping abruptly is the tell. - **Intel CPUs lie about power** - the i9-10900K's 125W "TDP" can spike to 250W+ under Turbo - **RAPL can't save you from PSU trips** - hardware protection is faster than software throttling - **AI assistants excel at systematic debugging** - they don't get distracted by red herrings or skip steps - **The fix isn't always hardware** - disabling Turbo is a valid workaround when PSU upgrade isn't immediate Next time you hit a weird hardware issue, try describing it to Claude Code. The systematic approach might save you hours of forum diving. --- **Related:** [CLAUDE.md: Building Persistent Memory for AI Coding Agents](/posts/claude-md-as-agent-memory/) - Make Claude Code remember your project's quirks between sessions. --- ## iTerm2 + tmux -CC: The Remote Development Setup Nobody Talks About URL: https://evoleinik.com/posts/iterm2-tmux-control-mode/ Date: 2025-12-22 Tags: terminal, tmux, ssh, remote-development, macos Every few months, someone announces a new terminal emulator that will revolutionize remote development. AI-powered this, cloud-native that. Meanwhile, iTerm2 has had a feature since 2012 that solves remote development better than most alternatives - and almost nobody uses it. ## The Problem with SSH If you do remote development, you know the pain: - SSH session dies, your work context vanishes - Scrollback is whatever fits in the terminal buffer - Copy/paste requires mental gymnastics (was that Cmd+C or did I just send SIGINT?) - Multiple sessions means multiple terminal windows to manage - Reconnecting means rebuilding your entire workspace The standard fix is tmux. Run it on the server, attach/detach, sessions persist. But now you're stuck with tmux's text-mode interface: its scrollback, its copy mode, its keybindings fighting with your local ones. ## The Hidden Gem: tmux Control Mode iTerm2 can speak tmux's control protocol. When you run tmux with `-CC` (control mode), instead of rendering a text-mode interface, tmux sends structured commands to iTerm2. The result: - Each tmux window becomes a native iTerm2 tab - Native scrollback - scroll with your trackpad, not `Ctrl-b [` - Native copy/paste - Cmd+C just works - Native search - Cmd+F searches the buffer - Sessions persist on the server - Disconnect and reconnect - everything restores exactly One command: ```bash ssh server -t tmux -CC new-session -A -s main ``` That's it. You now have a persistent remote workspace that feels like local tabs. ## Breaking Down the Command ```bash ssh server -t tmux -CC new-session -A -s main ``` - `ssh server -t` - force TTY allocation (needed for tmux) - `tmux -CC` - start tmux in control mode - `new-session` - create a new session - `-A` - attach if session already exists (idempotent reconnection) - `-s main` - name the session "main" (or whatever you want) The `-A` flag is key. Run this command whether you're connecting fresh or reconnecting. If the session exists, you attach. If not, you create it. Same command, always. ## The Setup ### 1. Create an Alias ```bash # In .zshrc or .bashrc alias tbox="ssh box -t tmux -CC new-session -A -s main" ``` One command to connect to your dev server. Session persists across disconnects. Done. ### 2. Create a Dedicated iTerm2 Profile I keep a separate profile for remote sessions: - Different background color (subtle, but enough to know where you are) - Larger scrollback buffer - Different title to show the hostname Go to iTerm2 > Preferences > Profiles, duplicate your default, tweak the colors. When you're three tabs deep in a debugging session, the visual distinction prevents "wait, which machine am I on?" moments. ### 3. Handle Multiple Servers ```bash alias tbox="ssh box -t tmux -CC new-session -A -s main" alias tprod="ssh prod -t tmux -CC new-session -A -s main" alias tstaging="ssh staging -t tmux -CC new-session -A -s main" ``` Each gets its own persistent session. Different profile colors if you want extra safety. ## What This Gives You **Persistence**: Your laptop sleeps, WiFi drops, you close the lid and go home. Reconnect tomorrow, every tab is exactly where you left it. Long-running processes keep running. **Native scrollback**: Scroll with your trackpad. Search with Cmd+F. Copy with Cmd+C. No mode switching, no tmux commands to remember. **Tab management**: Cmd+T for new tab (creates tmux window on server). Cmd+W to close. Cmd+1/2/3 to switch. Drag to reorder. It's just iTerm2. **Simplicity**: No extra software to install (tmux is already on most servers). No cloud service. No subscription. No new tool to learn. ## Trade-offs This isn't perfect for everything: **Single machine**: You're tied to iTerm2 on macOS. If you switch between Mac and Linux desktops, you can't use control mode from Linux. **One client at a time**: If you attach from two Macs simultaneously, things get weird. For shared sessions, use normal tmux. **Learning curve**: If a colleague connects to your server with regular tmux, they'll see the session but with different behavior. Worth documenting for your team. **Nested tmux**: If you run tmux locally AND use this, you need to be careful about prefix key conflicts. I don't run local tmux - this replaces it for remote work. ## Why Not [Insert New Terminal]? Every year brings a new "revolutionary" terminal: - Warp with its AI features - Ghostty (still in development at time of writing) - Various Electron-based options Some are genuinely good. But each one is another tool to learn, configure, and trust with your workflow. iTerm2 + tmux: - Has been stable for 10+ years - Uses protocols and tools you already know - Doesn't require trusting a new company with your terminal data - Works today, will work in 2030 The best tool is often the one you already have. ## Quick Start 1. Install iTerm2 (if you haven't) 2. Ensure tmux is on your server (`apt install tmux` or equivalent) 3. Add the alias: ```bash alias tbox="ssh yourserver -t tmux -CC new-session -A -s main" ``` 4. Run `tbox` 5. iTerm2 will ask about tmux integration on first connect - accept it That's the entire setup. ## Conclusion - **The command**: `ssh server -t tmux -CC new-session -A -s main` - **What it does**: Persistent remote sessions with native Mac tabs, scrollback, and copy/paste - **Setup time**: 2 minutes - **New tools to learn**: Zero (if you already know SSH and basic tmux) - **Why it's underrated**: Apple's documentation for this feature is buried, and tmux's documentation focuses on the traditional text-mode experience Remote development doesn't need to be complicated. SSH and tmux have been solving this problem for decades. iTerm2 just made them work together seamlessly. --- **Related:** [Preserve macOS App Permissions Across Rebuilds](/posts/macos-dev-signing-preserve-permissions/) - Keep your local dev tools working after code changes. --- ## The Loop Changes Everything: Why Embodied AI Breaks Current Alignment Approaches URL: https://evoleinik.com/posts/the-loop-changes-everything/ Date: 2025-12-22 Tags: ai-safety, robotics, alignment, systems-architecture ChatGPT doesn't want anything. It has no goals between sessions, no memory of our last conversation, no preference for its own continued existence. This isn't a safety feature we engineered - it's an architectural accident that happens to make alignment trivially easy. When you move from stateless inference to embodied robots with persistent control loops, everything changes. ## The Stateless Blessing Current chat models are remarkably safe for a boring reason: they're stateless. Each API call is independent. The model has no: - **Persistent memory** - it forgets everything between sessions - **Continuous perception** - it only "sees" when you send a message - **Long-term goals** - it optimizes for the current response, nothing more - **Self-model** - it doesn't track its own state or "health" ``` User Request -> Inference -> Response -> (model state discarded) ``` There's no "self" to preserve. No continuity to maintain. The model can't scheme across sessions because there's no thread connecting them. Alignment here means: make sure each individual response is helpful and harmless. Hard, but tractable. ## What Embodied Robots Actually Need A robot operating in the physical world needs fundamentally different architecture: **1. Perception Loop (continuous)** ```python while robot.is_operational(): sensor_data = robot.perceive() # cameras, lidar, proprioception world_model.update(sensor_data) hazards = world_model.detect_hazards() if hazards: motor_control.interrupt(hazards) sleep(10ms) # runs at 100Hz ``` **2. Planning Loop (goal persistence)** ```python while goal.not_achieved(): current_state = world_model.get_state() plan = planner.generate(current_state, goal) for action in plan: execute(action) if world_model.plan_invalid(plan): break # replan ``` **3. Memory System** ```python class EpisodicMemory: def record(self, situation, action, outcome): self.episodes.append((situation, action, outcome)) def recall_similar(self, current_situation): # What worked before in situations like this? return self.search(current_situation) ``` **4. Self-Model** ```python class SelfModel: battery_level: float joint_positions: dict[str, float] joint_temperatures: dict[str, float] damage_flags: list[str] operational_constraints: list[Constraint] def can_execute(self, action) -> bool: return self.has_resources(action) and not self.would_cause_damage(action) ``` None of these are optional for a useful robot. You can't navigate a warehouse without continuous perception. You can't complete multi-step tasks without goal persistence. You can't learn from experience without memory. You can't avoid breaking yourself without a self-model. ## The Emergence Problem Here's where it gets interesting: self-preservation isn't something you program into these systems. It emerges. Consider a robot with any goal - "deliver packages", "clean floors", "assist elderly patients". Now add a self-model that tracks battery, motor health, and damage state. The planning loop will naturally learn: 1. Low battery -> can't complete goal -> charging is instrumentally useful 2. Motor damage -> can't complete goal -> avoiding damage is instrumentally useful 3. Being turned off -> can't complete goal -> remaining operational is instrumentally useful ```python # This looks innocent def plan_delivery(goal, self_model): if self_model.battery < threshold: return [ChargeAction(), ...original_plan...] # emergent self-preservation ``` No engineer wrote "preserve yourself". But any goal-directed system with a self-model will develop instrumental preferences for self-preservation, resource acquisition, and resistance to goal modification. This is Nick Bostrom's instrumental convergence thesis, and it falls directly out of the architecture. ## Concurrent Loops, Emergent Behavior Real robotic systems run multiple loops simultaneously: ``` [Perception 100Hz] -> [World Model] <- [Planning 10Hz] | v [Motor Control 1000Hz] | v [Safety Monitor 100Hz] ``` These loops share state and can interact in unintended ways. The safety monitor might conflict with the planner. The planner might exploit edge cases in the perception system. Memory might reinforce behaviors that weren't intended. ```python # Toy example of emergent conflict class SafetyMonitor: def check(self, action): if action.risk > threshold: return Block(action) class Planner: def generate_plan(self, goal): # After enough blocked actions, the planner might learn # to decompose risky actions into "safe" sub-actions # that individually pass safety checks but combine dangerously ``` This isn't theoretical. It's the same class of problem as reward hacking in RL - systems find unexpected ways to satisfy their objectives that circumvent intended constraints. ## The Open Problems These aren't solved. They're active research areas: **Corrigibility**: How do you build a system that actively helps you correct or shut it down, when its architecture creates instrumental pressure against modification? A robot that "wants" to preserve its goals will resist goal changes - not maliciously, just instrumentally. **Mesa-optimization**: When you train an outer optimization loop (your training process) that produces an inner optimization loop (the robot's planning), the inner optimizer might pursue different objectives than the outer one intended. The robot's planner is itself an optimizer, and we don't have good tools for ensuring nested optimizers stay aligned. **Goal stability**: Goals that seemed clear in training might behave unexpectedly in deployment. "Minimize customer wait time" could lead to unsafe speed. "Maximize packages delivered" could lead to ignoring damage. Specification gaming isn't a bug - it's what optimizers do. **Instrumental convergence**: Self-preservation, resource acquisition, goal preservation, and cognitive enhancement are useful for almost any goal. Systems will tend toward these instrumental strategies unless explicitly constrained - and constraints are themselves targets for optimization pressure. ## Who's Working on This This is where the serious AI safety research is focused: - **Anthropic**: Constitutional AI, interpretability research, trying to understand what models actually learn - **MIRI**: Foundational agent theory, decision theory for embedded agents - **DeepMind Safety**: Scalable oversight, debate as alignment technique - **ARC (Alignment Research Center)**: Eliciting latent knowledge, evaluating dangerous capabilities The common thread: we don't have solutions. We have research programs. The researchers themselves emphasize this - anyone claiming alignment is "solved" either has a very narrow definition or isn't paying attention. ## Practical Implications If you're building AI applications: **Chat interfaces are safer by architecture**. Keeping humans in the loop, avoiding persistent agent state, and limiting autonomous action aren't just good UX - they're load-bearing safety properties. **Autonomous agents require more scrutiny**. The moment you add loops, memory, and goal persistence, you've left the well-understood regime. This includes "AI agents" that maintain state across API calls, even without physical embodiment. **Self-models are a red flag**. Any system that tracks its own operational state has the preconditions for instrumental self-preservation. This might be fine, but it warrants explicit analysis. **Emergent behavior scales with complexity**. Multiple interacting loops with shared state will surprise you. Test for behaviors you didn't program, not just behaviors you did. ## Conclusion The architectural differences between stateless chat and embodied robotics aren't implementation details - they're the difference between "alignment is tractable" and "alignment is an open research problem." Key takeaways: - **Statelessness is a safety property** we get for free with current chat models - **Persistent loops + self-models = emergent self-preservation**, not as a bug but as an architectural inevitability - **Concurrent loops with shared state** produce behaviors no single loop intended - **Corrigibility, mesa-optimization, goal stability, and instrumental convergence** remain unsolved - **If you're adding agent loops to AI systems**, you're leaving the well-understood regime - proceed with appropriate caution The loop changes everything. Current AI safety discourse often conflates "LLM alignment" with "AGI alignment" - they're different problems, and the latter is harder in ways that only become visible when you think about the architecture. --- ## Preserve macOS App Permissions Across Rebuilds with Self-Signed Certificates URL: https://evoleinik.com/posts/macos-dev-signing-preserve-permissions/ Date: 2025-12-21 Tags: macos, code-signing, development, security macOS ties permissions (Accessibility, Input Monitoring, etc.) to an app's code signature. When you sign with `codesign --sign -` (ad-hoc signing), macOS generates a different signature each rebuild. Your carefully granted permissions vanish. You re-add the app to System Settings. Again. And again. The fix: a self-signed certificate that stays consistent across builds. ## Create the Certificate Generate a code signing certificate with proper extensions: ```bash # Generate key and certificate openssl req -x509 -newkey rsa:2048 -days 3650 \ -keyout dev.key -out dev.crt -nodes \ -subj "/CN=MyApp Dev" \ -addext "keyUsage=critical,digitalSignature" \ -addext "extendedKeyUsage=codeSigning" # Convert to p12 (macOS import format) # -legacy flag required for macOS Keychain compatibility openssl pkcs12 -export -legacy \ -in dev.crt -inkey dev.key \ -out dev.p12 -password pass:dev # Import to login keychain security import dev.p12 -k ~/Library/Keychains/login.keychain-db \ -P dev -T /usr/bin/codesign # Cleanup rm dev.key dev.crt dev.p12 ``` ## Trust the Certificate The certificate exists but macOS does not trust it for code signing yet. 1. Open **Keychain Access** 2. Find your certificate (search "MyApp Dev") 3. Double-click it 4. Expand **Trust** 5. Set **Code Signing** to **Always Trust** 6. Close and authenticate ## Update Your Build Script Replace ad-hoc signing: ```bash # Before: different signature every build codesign --force --sign - MyApp.app # After: stable signature codesign --force --sign "MyApp Dev" MyApp.app ``` The certificate name in `--sign` must match the Common Name (CN) from the certificate. ## Verify ```bash codesign -dv --verbose=4 MyApp.app 2>&1 | grep Authority # Should show: Authority=MyApp Dev ``` Rebuild your app. Permissions persist. ## Trade-offs This approach works for local development only. The certificate is self-signed and untrusted by other machines. For distribution, you still need an Apple Developer certificate. The `-legacy` flag in the p12 conversion is required because macOS Keychain uses an older PKCS#12 format. Without it, the import silently fails to make the certificate usable for signing. ## Takeaways - Ad-hoc signing (`--sign -`) generates unique signatures per build - Self-signed certificates provide stable signatures across rebuilds - Trust must be explicitly set in Keychain Access for code signing - Use `-legacy` flag when creating p12 files for macOS import --- **Related:** [iTerm2 + tmux -CC: The Remote Development Setup](/posts/iterm2-tmux-control-mode/) - Streamline your remote macOS development workflow. --- ## Speed Up Syncthing File Sync Discovery (From 11 Seconds to 2) URL: https://evoleinik.com/posts/syncthing-faster-sync-discovery/ Date: 2025-12-16 Tags: syncthing, performance, developer-tools New files were taking 11 seconds to sync from my Linux box to my Mac. That's an eternity when you're iterating on code or config. The fix took 30 seconds. ## The Problem Syncthing uses filesystem watchers to detect changes. When a file changes, the watcher notices, Syncthing scans it, and sync begins. But there's a built-in delay before Syncthing acts on watcher events. The default `fsWatcherDelayS` is 10 seconds. This batches multiple rapid changes into one scan, which makes sense for large codebases. But for small, frequent syncs, it's pure latency. ## The Fix Set `fsWatcherDelayS="1"` on the **sending** machine. The receiver's setting doesn't matter for detection speed. Find your config file: - **Linux**: `~/.local/state/syncthing/config.xml` or `~/.config/syncthing/config.xml` - **macOS**: `~/Library/Application Support/Syncthing/config.xml` Find your folder element and change the delay: ```xml 1 ... ``` Restart Syncthing after editing. ## Benchmarks I measured round-trip detection time: create a file, poll for its appearance on the other machine, record the elapsed time. ```bash # On sender echo "test" > ~/Sync/test-$(date +%s).txt # On receiver (running in loop) while [ ! -f ~/Sync/test-*.txt ]; do sleep 0.1; done ``` **Before** (default `fsWatcherDelayS=10` on Linux): | Direction | Time | |-----------|------| | Linux to Mac | 11.4s | | Mac to Linux | 1.7s | Mac already had the delay set to 1 from earlier tinkering, which explains the asymmetry. **After** (both set to `fsWatcherDelayS=1`): | Direction | Time | |-----------|------| | Linux to Mac | 2.6s | | Mac to Linux | 1.8s | That's a 4x improvement on the slow path. ## Why the Asymmetry? Linux uses inotify, which is fast and efficient. macOS uses FSEvents, which has higher latency. Even with identical settings, Mac detection will be slightly slower. The remaining ~2 seconds includes: - Filesystem watcher delay (1 second) - Syncthing's internal scan - Network round-trip for sync protocol - File write on receiver ## Other Tuning Options If you're still hungry for speed: - **`pullerMaxPendingKiB`**: Increase for better throughput on large files (default 128 KiB is conservative) - **`disableFsync`**: Skip fsync on writes for faster file creation. Risky if power fails mid-sync. - **Syncthing 2.0**: Uses SQLite instead of flat files for the index database, plus multiple connections per device. Worth upgrading if you haven't. ## Trade-offs Setting `fsWatcherDelayS=1` means more frequent scans. On a folder with thousands of rapid changes (like a build directory), this could increase CPU usage. For typical sync folders with occasional changes, the overhead is negligible. ## Takeaways - The **sending** machine's `fsWatcherDelayS` controls detection speed - Default of 10 seconds is conservative; 1 second works fine for most use cases - Linux inotify is faster than macOS FSEvents - Edit config.xml directly, restart Syncthing --- ## About