The sitemap check flagged an extra URL: the page wasn't in git, but had been live since the day before

An earlier note covered the time I started deploying from a clean worktree so that only committed content would ship, and --delete removed 32 images that existed only in my working directory. The lesson then: things in the working directory that never made it into git may already be live.

This is the other side of the same thing. On October 10, deploying my company site from a clean worktree, the pre-deploy sitemap check stopped with stale in sitemap: /services/partner-services/. The committed sitemap listed a URL whose page wasn't in git. The session at the time took it for "someone else's unfinished draft" and removed it from the sitemap. But the page had gone live the day before and had been submitted to Google and IndexNow. This note is about how the mismatch arose, why it surfaced only in a clean worktree this time, and why "not in git" can't be read as "not live". In order: the timeline, why the check passes in the working directory, a lastmod lag found along the way, a local reproduction, and what I'd change.

Timeline

Times are +08:00, from git log, the file's creation time, session records and deploy logs:

Time	What happened
Oct 9 21:19–21:20	A session creates `services/partner-services/index.html` (a new "partner services" page), builds locally and regenerates the sitemap: "1 added, 378 URLs"
Oct 9 21:23	After I confirm, the site is deployed from the working directory; the sitemap check says "consistent (378 URLs)". After deploying, the session confirms the page returns 200 and the live sitemap lists it, then resubmits the sitemap and pings IndexNow. The page is not committed at this point
Oct 9 21:46	Commit `26c2805`, for another batch of articles, also commits the working directory's sitemap: 377 → 385 URLs. Two of those URLs have no page in that commit: this page, and an article that was missed by `git add` and added a minute later in `acefa73`
Oct 10 19:17	Another session deploys from a clean sparse worktree; the check stops: `stale in sitemap`
Oct 10 19:18	Finding the page untracked in the main working directory, it decides it's someone else's draft, regenerates the sitemap in the clean worktree and commits it (`f2e260e`, whose message says "unpublished"). The deploy that follows uploads nothing because the AWS account is wrong
Oct 11 23:27, 23:33	Two deploys from a clean worktree upload a 402-URL sitemap without the page
Oct 13 01:42	Commit `eae52b3` finally adds the page itself, and the sitemap entry comes back

The site's deploy script only uploads HTML and never deletes it (--delete applies only to assets/), so after the two October 11 deploys the page should still have been live, just missing from the sitemap. That's inferred from the script; I have no record of fetching the page during those two days.

Why the check passes in the working directory

The site's scripts/gen-sitemap.py walks the on-disk tree with os.walk; every directory with an index.html is a URL (excluding dist, node_modules and similar, and pages whose first 4,000 characters match its noindex rule). Inclusion ignores git; git is only used for lastmod. In --check mode it recomputes the URL set, compares it with public/sitemap.xml on disk, and exits 1 on any addition or removal; it doesn't compare lastmod.

On October 9, the session that created the page first ran the check and got MISSING from sitemap; after regenerating, it was consistent. From then on, as long as the URL set didn't change, the check passed in that working directory however often it ran, untracked page included: generation and checking use the same rules on the same disk.

A commit records what's in the index. That time the index had the sitemap and not the page, so they disagreed inside the commit. In the working directory, as long as the page is on disk, the existing checker can't see that mismatch; the clean worktree on October 10 had no untracked files, which is what exposed it. Comparing the committed sitemap with the pages git ls-tree lists would show it too.

The check was right: the sitemap in the commit really didn't match the pages in the commit. What went wrong was the next judgement. "Not in git" only says it wasn't committed, not that it isn't live; the page had been deployed from the working directory, so the fix should have been to commit the page, not to drop the URL from the sitemap.

Found along the way: lastmod stuck at the previous commit

f2e260e's diff didn't just remove one URL. It also changed the lastmod of 7 URLs to 2026-10-10. Those 7 pages (the blog index and 6 existing articles) had all been edited in commit 96dc792 at 19:17 on October 10.

The cause is in lastmod(): it takes the date of the last commit from git log -1 --format=%cs -- <file>, falling back to the file's modification time when that returns nothing or fails. My routine is to regenerate the sitemap and then commit everything together, so at generation time the pages edited in this round aren't committed yet and git log returns the previous commit's date until the next regeneration (not necessarily a day off; it depends on how far apart the commits are). --check compares only the URL set, so the lag never fails a check.

Reproducing it locally

I copied the site's gen-sitemap.py unchanged into a toy repo: a home page and blog/a/ committed with an October 5 date; then blog/a/ edited, an uncommitted services/draft/ added, the sitemap regenerated, and blog/ and public/ committed with an October 9 date (set with GIT_COMMITTER_DATE and GIT_AUTHOR_DATE; the reproduction ran on October 15):

$ python3 scripts/gen-sitemap.py
  +  /services/draft/
寫入：3 個 URL（新增 1、移除 0）          # "wrote 3 URLs (1 added, 0 removed)"
$ python3 scripts/gen-sitemap.py --check
sitemap 一致（3 個 URL）                  # "sitemap consistent (3 URLs)"

# the committed sitemap afterwards:
/                 lastmod 2026-10-05
/blog/a/          lastmod 2026-10-05     ← edited in this commit, dated the previous one
/services/draft/  lastmod 2026-10-15     ← not in git, dated by file mtime

$ git clone -q . ../clean && cd ../clean && python3 scripts/gen-sitemap.py --check
stale in sitemap:    /services/draft/       # excerpt
$ echo $?
1

Regenerating in the clean clone gives a diff of the same shape as f2e260e: the /services/draft/ entry disappears and /blog/a/'s lastmod moves from 10-05 to 10-09.

What I'd change

The root is that "what's live" and "what's in git" can drift apart: deploying from the working directory puts untracked pages online, and deploying from a clean worktree then treats them as nonexistent. Fixing only the sitemap doesn't address that; both need to go down one path. Three things:

Ask git before deploying. When deploying from the working directory, stop if the site's directories have uncommitted changes, or always deploy from a clean worktree. In the toy repo:
```
$ cat gate.sh
dirty=$(git status --porcelain --untracked-files=all -- blog services index.html) || exit 1
[ -z "$dirty" ] || { echo "uncommitted pages in the working directory, not deploying:" >&2; echo "$dirty" >&2; exit 1; }
$ bash gate.sh; echo "exit=$?"
uncommitted pages in the working directory, not deploying:
 M blog/a/index.html
?? services/draft/index.html
exit=1
```
The real path list has to follow the site's directory layout. It stops uncommitted parts of what's about to be deployed; files excluded by .gitignore aren't covered, and pages already live aren't removed by it.
When the check says stale, look at the live site first. If curl -s -o /dev/null -w '%{http_code}' https://…/services/partner-services/ returns 200, check the content too; it's most likely live, so first ask whether the page should be committed. On a 404, check the deploy records and whether it was meant to be published before deciding between committing it and dropping it from the sitemap. The status code is a starting point, not a verdict.
Base lastmod on what git knows. The generator includes only pages git ls-files lists, and pages that differ from HEAD get today's date:
```
def git_set(*args):
    out = subprocess.run(["git", *args, "-z", "--", "."], cwd=ROOT,
                         capture_output=True, text=True, check=True).stdout
    return {os.path.join(ROOT, f) for f in out.split("\0") if f}

TRACKED = git_set("ls-files")                                   # files currently in the index
CHANGED = git_set("diff", "--name-only", "--relative", "HEAD")  # content differs from HEAD
```
git diff --name-only prints paths relative to the repo root by default, and the site's script lives under the repo's official_website/, hence --relative. Run in the toy repo and in a second toy repo with the site in a subdirectory: the untracked services/draft/ is skipped and /blog/a/ gets that day's date. CHANGED also covers edits not yet git added, and the generator still reads file contents from the working directory, not an index snapshot. On its own this backfires: without item 1, an untracked page still deployed from the working directory ends up live but missing from the sitemap, just like October 11. So it goes together with item 1. Also, "today" is the regeneration date, which may differ from the commit date.

I haven't merged any of the three into the site's repo yet.

This site has the same property

Writing this, I went back to baobaolin.com's own build. vite.config.mjs collects every post under posts/ with readdirSync, the sitemap is generated from the build output, and scripts/deploy.sh runs npm run build right in the repo directory without checking git status. Going by the code, an uncommitted post in posts/ would be built and deployed (and, for the Chinese version, listed in the sitemap), the same path the October 9 page took. This site doesn't commit its sitemap, so "inconsistent inside a commit" can't happen here, but "live but not in git" can.

If you remember one thing

"Not in git" doesn't mean "not live". The stale in sitemap at the top was right: it caught a commit whose sitemap and pages disagreed. The mistake was reading it as "this page isn't published yet". For anything that may have been deployed from a working directory, only the live site can tell you whether it's there.