What actually broke, and how it got fixed.

I deliver internal systems for companies, and I run the infrastructure they sit on. This is where the parts worth writing down end up — usually the ones that cost two hours to work out and take three sentences to explain.

date

2026-10-28

entry

055

topic

tooling

Inside an off-screen iframe, not one loading="lazy" image loaded; under --dump-dom they all did

Measuring layout in a same-origin iframe, every loading="lazy" image had a naturalWidth of 0. Rerun in Chrome 156, the server log had no request for any image that didn't load. Per the HTML spec, lazy loading measures distance against the top-level viewport: an iframe at left: -10000px or at top: 3000px loaded nothing within 5 seconds, while at the top left, making it 9000px tall or adding visibility: hidden still loaded the same three. The measured cutoff lies between 1247 and 1267px, consistent with the 1250px web.dev documents. In the other direction, headless --dump-dom loaded all six; I didn't find why. Setting loading to eager before measuring is enough; also switching each src to a new URL, as I did in the test, caused two requests per image.
date

2026-10-27

entry

054

topic

tooling

Remotion's <Freeze> freezes the clock, not just the picture: a 22-second hold left captions and highlights stuck on one line

In Remotion 4.0.536, <Freeze> replaces the frame its whole subtree reads. A screen-recording shot played 840 frames and held 671; with the whole shot in Freeze, captions and highlights stayed in their frame-839 state, and the last two captions and the Export highlight never appeared. Moving captions out was half a fix. The final version wraps only OffthreadVideo: overlays run on output time, the cursor and blur boxes on source time. Also measured: when the frozen frame plus the Sequence start lands past the composition end, a Freeze written directly in the Sequence reads a clamped value (49 instead of 80).
date

2026-10-18

entry

053

topic

tooling

In Claude Code on macOS, find is bfs: -newermt "-5 minutes" returns Invalid timestamp, and wc -l counts 0

In Claude Code 2.1.302's Bash tool (macOS), find is a shell function running the embedded bfs 4.1.1. Compared with macOS's /usr/bin/find: -newermt rejects "-5 minutes" and "yesterday" with Invalid timestamp and exit code 1; the error goes to stderr and wc -l counts 0. @seconds is the reverse, accepted only by bfs. -regex is GNU-style, with + and \| as operators; -printf works; output order differs. With this PATH, timeout and xargs run the system find.
date

2026-10-18

entry

052

topic

integration

Filtering CISA KEV by dateAdded for a weekly watch list misses changes to old entries: a field-by-field diff of 35 versions

From the git history of cisagov/kev-data I took 35 KEV versions, August 31 to October 16, and compared them field by field keyed on cveID: 61 added, 0 removed; 9 old entries flipped knownRansomwareCampaignUse from Unknown to Known, 2 dueDate values changed (one moved earlier), and on September 1 all 1,687 entries gained a forensicTriage field. A weekly report filtered by dateAdded can't see these changes. Same-day releases share a catalogVersion, so it can't key versions.
date

2026-10-18

entry

051

topic

integration

Counting a WordPress site's posts with the REST API: one HEAD request gives the total, and paging past the end of a non-empty result returns 400, not an empty array

Measured on wordpress.org/news: HEAD with per_page=1 gives 1,110 posts from X-WP-Total. per_page=101 returns 400; page 13 returns 400 rest_post_invalid_page_number rather than an empty array, so a loop that only stops on an empty array doesn't stop. Twelve pages return 1,110 rows with no duplicate IDs; _fields shrinks the first page from 2,341,785 bytes to 12,267. Category count values sum to 1,759, more than the posts, because 525 posts have two or more categories.
date

2026-10-18

entry

050

topic

tooling

The go-build cache is back at 8.1G within three days of a clean: Go only deletes entries unused for five days, and every new go run executable is stored too

This Mac's go-build was cleared with go clean -cache three times in 13 days; 68.5 hours after the latest clear it was 8.1G again. Go's source deletes entries unused for five days, with no size cap; since Go 1.24, go run executables are cached too. Measured with a separate GOCACHE: an edit plus go run adds 3.3 MiB, cross-compiling about 84 MiB, -race about 107 MiB. What each go clean flag deletes, and why rm -rf can't fully remove the module cache.
date

2026-10-18

entry

049

topic

infrastructure

Block site data in Chrome and most of the homepage goes blank: reading localStorage throws SecurityError, and nothing after it runs

When Chrome blocks site data, reading the window.localStorage property itself throws SecurityError; typeof doesn't guard it, and 'localStorage' in window returns true. The module stops on that line, the code further down that makes .reveal visible never runs, and CSS opacity:0 leaves the sections transparent: dark pixels in a 600×200 screenshot drop from 6,988 to 0. Reordering alone brings the sections back but the language setup still doesn't run; with try/catch, the later code runs too. Visitors with JavaScript off also see transparent sections.
date

2026-10-15

entry

048

topic

infrastructure

Finding what ate the disk: files modified since yesterday add up to 505 GB, more than the whole APFS container

Listing files modified since yesterday with find -newermt and summing their stat sizes gave 505.23 GB, more than the 494 GB APFS container: one sparse file (OrbStack's virtual disk) has a logical size of 494 GB and 19.46 GB allocated. Allocated blocks give 30.42 GB, but a modification time isn't new data, and APFS clones that share blocks make du overcount. Subtracting two du snapshots, plus df: over 27 minutes the home directory changed by −16 MiB net, while free space grew by 172 MiB, leaving about 156 MiB unexplained.
date

2026-10-15

entry

047

topic

tooling

A read-only SQLite query returns "database or disk is full": GROUP BY copies a whole column into a temp file

A GROUP BY with sum(length(large column)) via sqlite3 -readonly on a 2.69 GB database returned exit code 13 with about 2.1 GB of disk free. EXPLAIN shows the sorter carrying the whole large column into the temp B-tree, with length() computed after the sort; a 150.5 MB test database used about 158 MB more temp space. A RAM-disk reproduction of database or disk is full (13); with a MATERIALIZED CTE computing the length first, no increase was measurable.
date

2026-10-15

entry

046

topic

tooling

In macOS's bash 3.2, "$n個檔案" doesn't print 3: the Chinese character after a variable loses a byte

macOS's built-in /bin/bash 3.2.57, in a UTF-8 locale, takes the leading byte of a Chinese character or full-width punctuation right after $n as part of the variable name: the value vanishes and garbage may remain, and with set -u you get n�: unbound variable. The cause is a byte-by-byte isalnum(). zsh leaves full-width punctuation alone but takes a run of Chinese characters as the name; ShellCheck 0.11.0 is silent by default; four cases in this machine's logs. Write ${n}.
date

2026-10-15

entry

045

topic

tooling

headless Chrome without --user-data-dir: kill it, and the profile directory stays

With Chrome 155, --headless --screenshot without --user-data-dir keeps its temporary profile in ~/Library/Application Support/Google/Chrome-headless/: in my tests nothing was left after a normal exit, and every stop by SIGTERM, SIGKILL, SIGINT or timeout left one, about 47 MiB for runs of 20 to 25 seconds. The SIGTERM run exited 0 with no screenshot. When the page waits on an unanswered request, --virtual-time-budget doesn't stop it, --timeout does.
date

2026-10-15

entry

044

topic

tooling

codex exec generates a 1.7 MB image, and ~/.codex stores about 8 MB

By default codex exec writes a session file per new run. On this Mac, October's 1,162 exec sessions total 4.6 GB, median 4.43 MB, and about 86% of the bytes are base64 images: 842 of 865 generated images exist exactly four times, a PNG in ~/.codex/generated_images, two copies in the session file and one in thread_history_1.sqlite, about 8.4 MB in total on average. In two tests with codex-cli 0.159.2, --ephemeral left no session file and no thread record; whether the PNG is still written, I haven't tested.
date

2026-10-15

entry

043

topic

tooling

A browser extension typing mixed Chinese and English: ERP and the spaces around it vanished

Claude in Chrome's type sends mapped characters (letters, digits, space and so on) as key events and the rest as Input.insertText, without waiting for responses. On October 13, in a Threads reply box, "ERP" between Chinese text vanished twice, and clearing failed, leaving three segments. 120 local runs on three editors matched exactly; cause unknown. Switched to one execCommand('insertText') and a full-text comparison, noting that in a Chrome 155 test it fired no beforeinput.
date

2026-10-15

entry

042

topic

tooling

grep -r in Claude Code skips files listed in .gitignore: one string, 6 files versus 13

In Claude Code's (2.1.299) Bash tool, grep is a shell function calling an embedded ugrep with --ignore-files and -I: recursion applies .gitignore and ignores matches in binary files, so one string, measured before this post was added, returned 6 files in this repo against 13 from the system grep, and when only ignored files match the exit code is 1. xargs, timeout and newly started shells use the system grep; with no ripgrep on PATH, timeout rg can't find an executable.
date

2026-10-15

entry

041

topic

infrastructure

The sitemap check flagged an extra URL: the page wasn't in git, but had been live since the day before

Deploying my company site from a clean worktree, the sitemap check reported stale in sitemap: the committed sitemap listed a page git didn't have. The page had been deployed from the working directory the day before and submitted to Google and IndexNow, yet it was taken for a draft and dropped from the sitemap, and the next day's deploys uploaded a sitemap without it. The generator reads the disk, a commit reads the index; lastmod uses git log and lags too. Local reproduction, three fixes, and the same property in this site.
date

2026-10-15

entry

040

topic

tooling

git push says Everything up-to-date, and the remote is a commit behind

When an HTTPS push drops mid-upload, git push can still print Everything up-to-date after fatal: the remote end hung up unexpectedly (--porcelain prints Done), exiting 1 in the local reproduction; piped through | tail -2 without pipefail the pipeline returns 0 and the RPC failed line is cut. Two GitHub push records, a local fake-server reproduction, and the order in git v2.50.1's source: the helper reports an empty result, then exits 128. Confirm with pipefail and git ls-remote.
date

2026-10-15

entry

039

topic

tooling

codex exec with -i before the prompt: the prompt becomes a second image path

codex exec's -i, --image <FILE>... takes multiple values, so a prompt written after it is parsed as a second image path. With stdin from /dev/null it fails in 0.23 s; with content on stdin it uses that as the prompt, the original prompt shows up in the session log as a could-not-read-image notice, and it exits 0; with an open pipe it waits. Three orderings that don't lose the prompt, measured.
date

2026-10-13

entry

038

topic

infrastructure

Three IndexNow scripts send the whole sitemap by default, and exit 0 when rejected

Of my four sites' IndexNow scripts, three share one submission logic: with no arguments they send the entire sitemap (413, 1,920 and 15 URLs in the versions committed on 2026-10-13), while IndexNow's FAQ says it isn't designed for submitting a whole site at once. A 202 prints as "submitted", though the docs say 202 means the key is still being validated, and against a fake server returning 403, 422 or 429 the exit code is still 0. The fourth defaults to a dry run and takes URLs from git changes or by hand.
date

2026-10-13

entry

037

topic

tooling

On macOS, df / says 82% while the APFS container is 99.4% used

On macOS 26, df -h / showed Capacity 82% while /System/Volumes/Data showed 100% and the APFS container was 99.4% used. / is a sealed system snapshot with only 12 GiB Used on this machine, and Capacity is Used as a share of Used+Avail, so / exceeds 90% only once about 1.3 GiB is left. Monitor absolute Avail instead.
date

2026-10-13

entry

036

topic

tooling

aws login credential errors: the fix stays, but the cause I wrote in my commits doesn't match the API docs

After switching to aws login, deploy scripts running sequences of aws calls intermittently got an invalid grant from CreateOAuth2Token. Over two days the commits cycled through several hypotheses, ending with aws configure export-credentials to export one set of credentials and use it throughout. But the cause I wrote in the commits, that each call spends the refresh token, doesn't match AWS docs: credentials refresh every 15 minutes as needed, and refresh is idempotent within the validity window. The fix stays, the cause is relabelled as inference, and a guard for the 900-second lifetime is added.
date

2026-10-13

entry

035

topic

integration

Looking up Taiwan in UN Comtrade: code 158 is empty, and 490 isn't guaranteed to be only Taiwan

Taiwan's customs statistics sit behind a CAPTCHA, so I used UN Comtrade mirror data: what other countries report trading with Taiwan. The ISO code TWN maps to 158, and every query I tried there came back empty; Comtrade's documentation puts Taiwan under 490, "Other Asia, nes", but that code can also hold other unspecified Asian areas, and Saudi Arabia booked half its 2025 exports there. Viet Nam's 2024 and 2025 datasets weren't available on the day, and reporters with export records to 490 went from 145 to 112, so fix the country set before summing across years.
date

2026-10-13

entry

034

topic

integration

EUR-Lex returns 202 and zero bytes, and curl calls it success: fetching EU law from Cellar instead

Fetching an EUR-Lex page with curl returned 202 Accepted and 0 bytes; curl -f exited 0 and Python urllib raised nothing. It was an AWS WAF challenge. Fetching from Cellar via content negotiation instead: in requests for three documents, text/html and application/pdf returned 404 while application/xhtml+xml got the full text. Read the branch notice first to see which formats exist.
date

2026-10-13

entry

033

topic

infrastructure

Google re-reads sitemaps without a resubmit: two timestamps in Search Console

My rule was "Google won't come back to a sitemap on its own, so resubmit after every deploy." Pulling lastSubmitted and lastDownloaded for 23 sitemaps: 12 not submitted for months were still downloaded within 9 days, and of 9 I resubmitted, 7 had a download about half an hour later while 2 were still waiting a day on. The indexed field is deprecated, the ping endpoint returns 404, and Google isn't on the IndexNow list.
date

2026-10-13

entry

032

topic

infrastructure

Deploying from a clean worktree, sync --delete removed 32 images that weren't in git

I deploy from a clean git worktree so that only committed content goes live. The last step, aws s3 sync --delete, compares against that local directory, where 32 uncommitted social images don't exist, so their copies on S3 were deleted, twice. An offline reproduction with a thirty-line fake S3, four fixes, and a dry-run guard before anything is deleted.
date

2026-10-13

entry

031

topic

infrastructure

GitHub repos created after July 15, 2026 get two extra IDs in the OIDC sub

For repos created after July 15, 2026, the GitHub Actions OIDC token's sub reads repo:owner@ID/repo@ID:…. An AWS trust policy in the old format only returns Not authorized, after 12 attempts per run, and repo:owner/repo:* doesn't match either. The claim as printed, the changelog, and the API to check the prefix before writing the policy.
date

2026-10-13

entry

030

topic

infrastructure

A CloudFront Function that returns its own 301 can drop the query string in two places

When the www redirect and the directory index live in one CloudFront Function, the 301's URL is whatever your code builds. Reading only value drops the second value of a repeated parameter, and the trailing-slash redirect leaves the query behind, so a utm tag is gone after two hops. Live curl, running the handler locally in Node, and the fixed function.
date

2026-10-13

entry

029

topic

tooling

Pasting into a ProseMirror editor from a script: blank lines vanish, and the paste lands after the old text

LinkedIn's post composer is Tiptap (ProseMirror). Paste plain text with a synthetic ClipboardEvent and any run of newlines becomes one paragraph break, so every blank line disappears. Select all from a script and paste immediately, and the editor's selection hasn't synced yet, so the new text lands after the old. Both reproduced locally, plus how to capture a file input that never enters the DOM.
date

2026-10-12

entry

028

topic

integration

Taiwan's company registry API: $filter vanishes in the shell, and a bad $format returns 200

Company lookups belong on the official open-data API, not the search pages behind a Cloudflare block and a CAPTCHA. But $format and $filter inside double quotes expand to empty strings; a bad $format returns HTTP 200 with one line of Chinese, and not found returns 200 with 0 bytes. Real responses, and a layered check.
date

2026-10-12

entry

027

topic

tooling

Headless Chrome at 375px wide lays the page out at 500px

Below a --window-size width of 500, headless Chrome lays the page out at 500px and crops the screenshot to the width you asked for, so the PNG size is exactly right. innerHeight read at load is also 87px short of the window. Measurements at six widths, the 500 in Chromium's source, and setting the viewport through the DevTools Protocol instead.
date

2026-10-12

entry

026

topic

infrastructure

Commas in AWS CLI shorthand: an error if you're lucky, two values if you're not

AWS CLI shorthand splits on commas, including the ones inside values. The Comment in --function-config only accepts a string, so it errors; Values in ec2 --filters is already a list, so it quietly becomes two values and goes out. Reproduced with a local endpoint and example keys, touching no AWS account, plus why my scripts always use JSON.
date

2026-10-12

entry

025

topic

tooling

codex exec hangs: it is waiting for EOF on stdin

Whenever stdin isn't a terminal, codex exec reads it to EOF and appends it to the prompt. Spawned from Node.js with default settings, it was still waiting after 60 seconds; inside a while read loop, it ate the rest of the list and the loop ran once. Timings for four kinds of stdin, and the < /dev/null fix.
date

2026-10-12

entry

024

topic

tooling

The tool called Bash runs zsh: set -- $pair yields one argument

set -- $pair in Claude Code produced one argument: on my Mac the tool called Bash runs commands in zsh. zsh doesn't split unquoted variables by default, with no error; an unmatched glob means that command never runs, and in a for list it ends the whole script. Side-by-side output from three shells, and a form that works in both.
date

2026-09-07

entry

023

topic

tooling

Your guardrail has never actually fired

An identity gate only executes when the identity is wrong, and nobody walks that path. Running it wrong on purpose surfaced two problems in two minutes: comparing the account id lets root through (root shares your account), and the error message itself crashed under set -u. The two acceptance criteria for a negative test, and what else only ever runs on failure.
date

2026-09-06

entry

022

topic

infrastructure

A content site needs a real 404, not an SPA fallback

The directory key trap from the first entry traces back to SPA error handling applied to a site with no client-side routing. Three costs of calling a 404 a 200 (soft 404s, blind monitoring, deceiving yourself), what the error response should actually say, and why a private S3 bucket answers 403 rather than 404.
date

2026-09-06

entry

021

topic

tooling

Your build config should not contain a file list

Multi-page bundlers want every HTML entry point enumerated. That list is hand-maintained and missing an entry raises no error — the build succeeds one page short, while the dev server still shows it, so only production output reveals the gap. Scanning the filesystem instead, three implementation details, and why drafts belong behind a directory convention rather than an exclusion list.
date

2026-09-06

entry

020

topic

tooling

Do not use set -e in a deploy script

set -e assumes stopping beats continuing, and a deploy script is where that fails — half-deployed is broken and silent. It caught me once: sync finished, the session expired midway through rewriting directory keys, and both the invalidation and the whole-site verification were skipped. Where to draw the line at the first write to production, and why -u and pipefail stay.
date

2026-09-06

entry

019

topic

infrastructure

A static site needs three cache policies, not one

One max-age for everything ends two ways: too long and changes never show, too short and the CDN buys nothing. The test — does the filename change with the content — splits a static site into exactly three groups. What immutable adds beyond a long max-age, why max-age=0 is not "do not cache", and why invalidation clears the CDN but never the browser.
date

2026-09-06

entry

018

topic

performance

Do not webfont your Chinese: one stylesheet cost 800ms

Lighthouse measured one Google Fonts stylesheet costing close to 800ms of first paint. Where that time goes (two origins, two rounds of DNS and TLS, with a sequential dependency between them), how the preload-then-swap-rel trick works and what it costs, and why the tens of thousands of characters in a Chinese face make webfonts a bad trade from the start.
date

2026-09-06

entry

017

topic

tooling

Verify a deploy by content, not status codes

The script I gave three sentences in the first entry, written out properly. Both common static-site failures — a missing directory key and a deploy that never landed — return 200, so only content comparison catches them. Why <title> makes the right fingerprint, why set -e is deliberately absent, and why it also belongs on a daily schedule.
date

2026-09-06

entry

016

topic

tooling

A bilingual static site: canonical points at itself, hreflang states the rest

Pointing the English page's canonical at the Chinese one tells search engines the English page need not be indexed — and it duly disappears. On self-referencing canonicals, reciprocal hreflang sets, changing <html lang> too, why feeds come in pairs, and scripting a check across the seven places one article touches.
date

2026-09-05

entry

015

topic

delivery

Your access log cannot answer "why"

An access line is emitted by middleware after the handler returns, so all it sees is the response — the error detail died at the return. Pairing by timestamp works at low traffic and collapses under load. On threading a request id through everything and handing it back to the client, how far to structure logs, and the three categories that count as leaked the moment they are written.
date

2026-09-05

entry

014

topic

integration

The Threads API's two-phase publish: why a thread cannot be batched

Container creation and publishing are separate calls with a wait between them, and reply_to_id takes a published post id — three constraints that make a ten-post thread strictly serial. The real problem is that the chain has no transaction: when the sixth fails, the first five are already public. How to make it resumable.
date

2026-09-05

entry

013

topic

tooling

Ask "who am I" before running any aws command

Nothing found, no error, so you conclude it does not exist — when in fact an export from half an hour ago is still live and you queried a different account. Credential precedence, the aws sts get-caller-identity gate, and why a read against the wrong account is harder to catch than a write.
date

2026-09-05

entry

012

topic

infrastructure

Seven projects on one box: what is isolated and what is not

Putting several clients' backends on one machine is a sound call for a small team; the risk is that everyone then assumes they cannot affect each other. Vhosts, containers and log groups are isolated. Disk, the Docker daemon, nginx reload, the kernel and the egress IP are not — those five are your blast radius. With the disk-incident commands and four reasons to split a project out.
date

2026-09-05

entry

011

topic

tooling

Every service needs /version: confirm which commit is running

You fixed it, you pushed it, the problem persists — and "the fix is wrong" looks exactly like "the fix is not running". Baking the git sha in with -ldflags -X, why the frontend needs this even more, and how to turn a deploy script's last step from hope into verification.
date

2026-09-05

entry

010

topic

tooling

Reviewing code an agent wrote: the report is intent, not fact

Delegate an implementation and the completion report describes what the agent meant to do — producing the report and executing the work are separate paths. Five ways report and diff diverge, a sign-off procedure (read the diff yourself, rerun tests with -count=1, inspect the test diff for false greens), and why review cost is fixed the moment you write the brief.
date

2026-09-05

entry

009

topic

integration

A payment callback is a write endpoint, not a notification

The POST your payment provider sends is public, unauthenticated, forgeable and repeated — and what it changes is a money field. Working through ECPay's AIO: recomputing CheckMacValue, the two callbacks async payments produce, idempotency on MerchantTradeNo, and why the wrong response body makes the provider keep calling.
date

2026-09-05

entry

008

topic

integration

Three ways MQTT fails to connect, all with the same symptom

Six hops in the chain and the phone always says the same thing. Three unrelated failures: a client library's if (secure) undoing your WebSocket config, EMQX reading an empty 200 as a denial, and Docker's default bridge not resolving service names. What they share is that the broken hop reports itself healthy.
date

2026-09-05

entry

007

topic

delivery

Schema drift: why prod is missing a column and nobody knows

The log says column "invoice_no" does not exist. The code is right and the migration is in the repo — prod just never ran it. Detecting drift left by hand-edited production databases, why prod is the baseline when cleaning up, and why the fix is permissions rather than discipline.
date

2026-09-05

entry

006

topic

tooling

Put the API token in SSM Parameter Store, not .env

The problem with a .env file is not that it is insecure — it is that it gets copied, into backups, build contexts and downloads folders. A workflow that keeps credentials off disk, why zone ids should not be stored either, and when to skip the judgement call and just rotate.
date

2026-09-05

entry

005

topic

infrastructure

Why sync --delete blanks the screen mid-deploy

The browser blames MIME types; the real cause is a deploy script deleting the chunk a user was about to fetch. Load the page after the deploy and all is well — have it open during the deploy and you get a white screen. Plus why the fix is dropping --delete.
date

2026-09-05

entry

004

topic

infrastructure

The nginx error_page that overwrites your error body

The backend returned a 500 with a real message; the client received {"status":"ok","message":"Request processed"}. Right status, right JSON shape, wrong content — courtesy of proxy_intercept_errors and one error_page line. Plus how to peel the layers to find the rewrite.
date

2026-08-01

entry

003

topic

reading

Failures that talk back

An eight-week incident study of a production LLM agent runtime. What 4,286 unit tests and 827 governance checks did not catch was the model rewriting a tool error into a fluent, plausible completion report.
date

2026-08-01

entry

002

topic

method

The fake trend my own page cap produced

A safety cap on a paginated fetcher never errored and never warned. It quietly grew a trend that did not exist, pointing the opposite way from reality. Plus why cat: matches more than you think.
date

2026-07-31

entry

001

topic

infrastructure

The S3 + CloudFront directory key trap: why /blog/ serves your homepage

With a REST origin, CloudFront will not resolve /blog/ to blog/index.html — and an SPA fallback turns that 404 into a 200 serving the homepage. Plus why sync --delete keeps undoing the fix.