The fake trend my own page cap produced

I wanted to know how much recent research addressed a particular topic, so I wrote a paginated fetch against the arXiv API for a few categories over three months. To stop the loop running away if my logic was wrong, I added a cap:

MAXPAGE = 60   # 60 * 100 = 6000 per category, a runaway guard

for page in range(MAXPAGE):
    start = page * 100
    url = (f"...search_query=cat:{cat}+AND+submittedDate:[{A}+TO+{B}]"
           f"&start={start}&max_results=100"
           f"&sortBy=submittedDate&sortOrder=descending")
    ...

An ordinary guard rail. The whole point is that you never hit it in normal operation, which is exactly why I wrote no alert for it.

The data looked like a finding

Counting the results by month:

2026-05:  4,902
2026-06:  7,671
2026-07:  8,648

Nearly a doubling in three months. Tempting, because it happened to support the story I already wanted to tell: the field is heating up fast. I nearly wrote it into the conclusion.

The data that gets you into trouble is rarely the obviously broken kind. It is the kind that matches what you expected.

What stopped me was simple arithmetic. arXiv submissions do not triple in a quarter. The magnitude was not credible.

Ask the source, not your own data

To check a sampling bias you need a question whose answer does not depend on your own fetch. The arXiv API returns <opensearch:totalResults>, so you can request a single result and simply ask how many exist:

url = (f"...search_query=cat:{cat}"
       f"+AND+submittedDate:[{a}0000+TO+{b}2359]"
       f"&start=0&max_results=1")
total = int(re.search(r'<opensearch:totalResults[^>]*>(\d+)<', s).group(1))

Asked across three separate weeks:

Category	May 1–7	Jun 1–7	Jul 24–30
cs.AI	1,331	1,444	944
cs.LG	1,350	1,325	835
cs.CL	536	731	376

The first week of May was busier than the last week of July. Reality ran the other way. The growth was entirely my own manufacture.

Why the truncation lands on the old end

The cause is sortOrder=descending. Results arrive newest first, so stopping at 6,000 leaves you holding the most recent 6,000 and discarding the older tail.

Three large categories each had between ten and twenty thousand papers in the window and all hit the cap. The six smaller ones finished cleanly. So the finished dataset was well covered near the present and increasingly sparse going back. Plot that by month and you get something indistinguishable from a healthy growth curve.

Two properties make this class of bug hard to catch:

• Nothing fails. The loop terminates normally. No exception, no retry, clean logs.
• It produces a plausible shape. Random truncation would look like noise and invite suspicion. Truncation along a sort key produces a monotonic trend, and a monotonic trend reads as knowledge.

The fix: make the fetch prove its own completeness

The fix is not a bigger cap. A bigger cap only moves the problem: the next time the data grows, it comes back, just as silently. The fix is to turn "I got everything" from an assumption into a checked assertion — ask for the total first, compare after.

for c in CATS:
    exp = total(c)                    # ask the source first
    got = 0
    for start in range(0, exp + 100, 100):   # bound comes from exp, not a constant
        ...
        got += n
    verify[c] = {'official_total': exp, 'fetched': got}
    print(f"{c} official {exp} / fetched {got} {'ok' if got >= exp else 'SHORT'}")

Two changes. The loop bound now comes from the data source rather than a number I invented. And every category leaves behind a line recording what the source claimed and what I actually got — stored alongside the data, so anyone later (including me) who wants to question whether the set is complete has something to look at.

On the rerun all nine categories matched. Only then were the numbers worth publishing.

An aside: cat: is not the cat you think

The same exercise hid a second trap. search_query=cat:cs.AI matches any paper tagged cs.AI, including cross-listings — not papers whose primary category is cs.AI.

So querying nine categories and deduplicating is a different quantity from "how many distinct papers are in these nine categories," and a single category's totalResults is not its primary-category count either. I only went back to check this after noticing the deduplicated total was far below the sum of the parts.

If you remember one thing

A safety limit you set yourself is invisible in the output. The program will not tell you it hit the ceiling unless you make it say so. And truncation along a sorted axis does not produce noise. It produces a trend.

So when you see a clean monotonic curve, suspect your sampling before you believe the world — particularly when the curve happens to support the point you were already planning to make.