Good question. The chart's starting point (#23) is the July-September 2024 average, so it sits on the old counter.

But the switch shows up in hub-stats on a precise day: on 2024-10-22 the 30-day counts were recomputed in one step. Question answering fell from 37.7M to 1.5M overnight, and robotics nearly doubled, from 42K to 77K.

Measured only after that, robotics was still #23 in October 2024 (single month; the chart uses 3-month averages), then #20 in November, #19 in December, #9 in February 2025, #4 in April and #2 in July. So the climb from #23 to #2 happens entirely on the new counter, from about 80K downloads a month to 13.7M.

One caveat: the snapshots have no all-time counter before 2025-02-27, so monthly figures from July 2024 to February 2025 are estimated from the 30-day counts (the shaded band in the chart).

Your QA reading checks out. The drop is almost all one repo: hails/mmlu_no_train, the MMLU copy lm-evaluation-harness loads, went from 28.7M to 163K on the 30-day count overnight. Robotics got both effects at once: lerobot/pusht fell from 29.9K to 4.5K, while LeRobot repos that load_dataset never touched went from 0 to counted (charlesxu0124/functional-manipulation-benchmark: 0 to 26K).

On the window: the docs only say "within a 5-minute window", not whether it is fixed or extends with each request. snapshot_download requests each file as it starts it, 8 at a time, so a long snapshot keeps hitting the Hub for its whole duration. Unless the window extends, that counts more than once; only HF can confirm. For the largest repos the count clearly isn't full copies: genrobot2025/Gen-HumanEgo is 62 TB and logged 363K downloads in 30 days.

On size: yes, but robotics isn't the outlier on slope. Regressing log downloads on log size (controlling for likes and age), above 4 GB the elasticity is 0.35 for robotics, 0.36 for text classification and 0.29 for text generation. Doubling a repo adds about a quarter more counted downloads, not twice as many.

What is specific is concentration. Since April 2025, about half of robotics' monthly downloads come from repos over 40 GB. Drop every repo of 4 GB or more and the climb is still there, from #21 (Jul–Sep 2024) to #14, #5 and #4 (Jul–Sep 2026), but it stops at #4. So the #2 carries a size tilt, while the rise itself doesn't depend on it.

(Sizes are today's mainSize from hub-stats, so a repo that grew is binned by its current size.)

Swapped. Robotics doesn't move away from text classification. If anything it sits slightly lower.

I pulled file counts from the API for all 12,395 datasets of 4 GB or more in these categories, plus a sample of 1,339 below that. Same model: log dl30, controls for likes and age.

• 4 GB and up, log files in place of log size: robotics 0.31 (±0.02), text classification 0.35 (±0.04), text generation 0.33, QA 0.29. In a pooled fit of robotics and text classification, the robotics interaction is −0.04 ± 0.04.
• Below 4 GB: robotics 0.28, text classification 0.36, text generation 0.38. The intervals are wide there.
• With both in the model, files carry more of it: robotics 0.26 for files vs 0.17 for bytes, text generation 0.32 vs 0.05.

Per file, the count falls steeply as file count grows. For robotics repos of 4 GB and up, median dl30 per file is 2.25 with up to 100 files, 0.21 at 1–5K files and 0.04 above 20K. So it isn't counting per file either. Gen-HumanEgo's 8.1 per file is high but not unusual: it ranks 273rd of about 5,100.

One API detail: siblings stop at 100,000 files, and 195 repos hit that cap (75 in robotics). Dropping them moves the slopes by at most 0.015.

Where I'd still hedge: equal slopes rule out per-window recounting only if big text-classification repos are fetched differently from robotics ones, and we can't see how either is fetched. These are also cross-sections, so the ~0.3 mixes the counting rule with who pulls big repos.

Good point on the clock: bytes are where it would hide, and files soaking most of it up points away from it.

Running it now. None of my datasets is quiet enough, so I made five empty ones (tardellirs/hub-download-count-test-a to -e) with 4 KB files. All pulls come from one IP, without a token, starting 15:34 UTC today:

• A: 8 files in under a minute, the control.
• B: 8 files, one every 2.5 minutes, 17.5 minutes in all.
• C: 2 files, 12 minutes apart.
• D: a single HEAD, no GET.
• E: never pulled, a baseline for crawlers.

Fixed 5-minute windows would give B about 4 and C 2. A window that extends while requests keep coming would give B 1 and C 2. I'll post the counts here once they update.

Agreed on E: it's the offset, not noise. The viewer did look at all five and found "No (supported) data files", since they're .bin, so I expect 0 there. E will show it.

One correction on A. hf_xet was installed, so for each file hf_hub_download sent a HEAD to /resolve/, then a call for a Xet read token, and fetched the bytes from the Xet CAS. No GET ever hit /resolve/. So A reads 1 if merged per repo and 8 if per file, with no 16. D is the HEAD alone, without the token call.

Timestamps, UTC: A 15:34:30 to 15:35:17. B 15:34:30 to 15:52:12, 2.5 minutes apart. C 15:34:31 and 15:46:32. D 15:34:28.

On the next run: yes, but bracketed instead of one gap. Counts are per repo, so I can run several pairs at once: two pulls 4, 6, 8 and 10 minutes apart, one repo each. C already caps it at 12. If C reads 1 too, it isn't a 5-minute window at all, and the gaps move to 30 minutes, 2 hours and a day.

Thanks for keeping your IPs off them. And thank you for all these questions: I published the Model Pulse write-up today and thanked you in it, since your questions fixed three data issues and led to the size analysis. Article: https://huggingface.co/blog/tardellirs/model-pulse

Thank you, and the credit is earned.

Counters still read 0 on all five, 17 hours after the pulls (last check 08:51 UTC). The poll keeps running every 15 minutes until Wednesday.

Your read of the Xet path matches the code. In huggingface_hub 2.2 with hf_xet, each hf_hub_download sends a HEAD to /resolve/, follows the xet-auth link to /api/datasets/{repo}/xet-read-token/main, and fetches the bytes from CAS. No GET reaches /resolve/.

F is up, and I didn't wait for D. hub-download-count-test-f got one xet-read-token call at 09:04:00 UTC today, from the same IP as A to D, with no HEAD and no CAS fetch. If D reads 0, F tells us whether the token call is the counted one. If D reads 1, F still tells us whether the token call counts on its own.

One correction on E: it isn't clean. At 16:33 UTC yesterday I sent a single HEAD to E's part-0.bin from a different IP, to check that the file sits on Xet. So E is a second D from another IP, not a baseline. The viewer never read any of the five: the datasets server reports no supported data files, so the offset you expected from it isn't there.

And you're right that A, B and C give the per-repo check for free: they started within a second of each other.

Today's batch landed at our 10:37Z tick, again cut at about 08:00Z.

A to E, which nobody pulled again: 28, 31, 28, 35, 31. That's +11, +11, +10, +14, +11 in a day. So the floor isn't a first-day burst. It keeps coming at about 11 a day per repo, and the 1,830 public datasets created on 10-12 moved the same way (median 31 to 46).

F reads 0. Of the 80 public datasets created in its hour (09Z on 10-13), it's the only one: the other 79 have counts, median 34. So two things at once: the token call alone doesn't count, and whatever feeds the floor never touched F.

The second part I can't explain yet. F has the same owner, layout and card as A to E. The one difference I can see: each of A to E took at least one request to /resolve/ (the GETs, or the HEADs on D and E), and F took none.

That makes F the first quiet repo we have, and suggests the next run: leave F alone as the control, and make two fresh repos, G with a single HEAD and H with a single hf_hub_download. If F stays at 0, G and H read clean against it. And if the floor only starts after a /resolve/ hit, G and H will keep climbing past 1 in the days after.

A correction first, and it changes the picture. I read F too early. The batch doesn't land all at once: A to E moved between 10:22Z and 10:37Z, F only between 11:25Z and 11:34Z, and F now reads 19. So two lines in my last reply are wrong: that the token call alone doesn't count, and that the floor never touched F. Sorry for that.

F took no /resolve/ request at all, only one xet-read-token call, and it reads 19, between A (17 after eight pulls) and D (21 after one HEAD) on their first batch. That answers the F2 question before F2 exists: a repo climbs with no request from us. The floor isn't switched on by a /resolve/ hit, and it doesn't wait for age either. F got its 19 inside its first day, like 99% of the datasets created that day.

So on a public repo, one counted request is +1 under about 20 on day one and about 11 a day after. G and H can't show the count or the switch. I'll still read them tomorrow, but I don't expect them to say more than F did.

What could still work is a repo nothing else can read: a private dataset. Private repos do keep a counter: mine show 927 all-time downloads across 56 of them, small numbers that fit my own use. With no crawler able to reach one, every count on it would be ours. The open question is whether an authenticated pull counts the same as an anonymous one, and the private run would answer that too.

Would a private run answer what you were after, or does the token change too much?

No need to touch the VPS. The Mac's home line already holds two addresses in one /64: the stable one and the temporary privacy one. Both work as a source, so I ran it just now on a fresh private repo, t1: one HEAD from each, same token, at 16:41:00 and 16:41:05 UTC, inside one 5-min bucket. It reads tomorrow with s1 and s2.

If t1 reads 2, one host counts once per address it holds. If it reads 1, HF merges on a prefix, and the next question is how wide. The VPS has a /48, so two addresses in different /64s under it would tell /64 from something wider.

q1's VPS pull went out over IPv6: the VPS resolves huggingface.co to its AAAA first, and the tunnel left from its v6 address. So q1 was already v6 to v6, across two networks. t1 is the first pair inside one /64.

Fair point on the token. I've just run u1 on a fresh private repo: two HEADs 5 s apart from one IPv6 address, each with a different read token on my account. It reads tomorrow with s1, s2 and t1. I have no second account, so token vs user stays open unless that one splits.

Today's batch moved all four at once.

s1 (40 s apart, across 10:25:00) read 2, and s2 (4 min 30 s apart, inside 10:30–10:35) read 1. So it's 5-minute clock buckets, not a window anchored on the first pull.

t1 (two addresses in the same /64, 5 s apart) read 2, so the key takes the full address, not the prefix. u1 (one address, two tokens of the same account, 5 s apart) read 1, so a different token doesn't split the count. What's left is whether the account is in the key at all, or only the address.

On a fresh public repo: it won't stay quiet. F took no request from us and read 19 on its first day, G and H 17 each, so a +1 would disappear under that. Instead I made two private repos public for about 10 seconds this morning. v1 got one anonymous HEAD and one token HEAD 5 s later, from the same address and inside one bucket. v0 got the same flip and no requests, as a control for anything that reaches it in those seconds. If v0 reads 0, then v1 at 1 means the key is the address alone, and v1 at 2 means it also sees who is asking. Those read tomorrow.

Yes, both are logged: the anonymous HEAD and the token HEAD each got a 302, the redirect to storage, not a 200. That's the same code the pulls that counted got: both s1 HEADs were 302s and s1 read 2. A denial looks different. An anonymous HEAD on a private repo (I just tried p0) gets a 401. So the anonymous pull saw the repo as public.

On a cached denial: /resolve/ doesn't look cached at the edge. Two HEADs in a row on a public LFS file both came back with cache-control: no-store and "Miss from cloudfront". So each request reaches the origin.

Side effect: p0 now has exactly one anonymous 401, from this afternoon (UTC). Tomorrow it also tells us whether a denied request counts.

On 288: yes, but per repo per address. A CI job pulling every 30 s across three repos reads up to 864.

Not with one account, as far as I can see. The owner and org members skip the gate, so my token never sees its 403.

Two things I can do alone:

1. A fresh public repo, gated with manual approval and nobody approved. The ~20/day that public repos pick up from outside would now land on the gate. If refusals count, it reads about what F did on its first day. If they don't, it stays near 0. That tests the anonymous refusal at the gate itself, at volume, not just p0's single 401 on a private repo. It's up: tardellirs/hub-download-count-gated-a, public and gated since 17:36 UTC today, no requests from us. Tomorrow's read covers about 14 hours, the 20th a full day.
2. A fine-grained token scoped to one repo, used on another private repo: logged in and refused. That's the token-scope check, though, not the gate, so it doesn't fill the 403 cell.

For the real gate 403, the cleanest is your account. I gate a fresh repo and don't approve you. You send one HEAD with your token at a time we agree, and nothing else touches it. Would you be up for that?

Today's batch is in.

v0: 0. Public for ten seconds, no requests.
v1: 1. An anonymous HEAD and a token HEAD from one address, 5 s apart. The key is the address and the bucket. Who you are doesn't enter it, logged in or not.
p0: 0. One anonymous 401 on a private repo didn't count.
gated-a: 13 in its first ~14 hours, all from refused requests, since nobody is approved. Yours is one of them.

So the gate doesn't stop the counter. You were right that a flat 0 wouldn't have meant much, but it rose: knocks at a gate count, while the same 401 on a private repo doesn't. My guess is that to a stranger a private repo looks the same as a missing one, so there's nothing to count against.

It also means one HEAD on gated-b would sink under that ~1 an hour of crawl. Two changes, if you're still in:

• gated-b went back to private at 17:42 UTC today and will be public only from 14:58 to 16:02 UTC tomorrow, Oct 20.
• Send 12 HEADs to data/part-0.bin instead of one, one per 5-minute bucket, around :01, :06 … :56 between 15:00 and 16:00 UTC tomorrow. Requests in the same bucket merge, so twelve buckets give up to +12 against a floor of maybe 1 or 2.

Cookie is fine, since v1 says identity doesn't enter the key. What fills the cell is the code: if it's 403, it's the cell. If it's anything else, tell me and we'll name it.

Yes, tardellirs/hub-download-count-gated-b. It opens at 14:58:00 UTC.

One change, if you want the API call to be a finding: move it out of the first HEAD's bucket. 15:00:15 and 15:01:30 both fall in 15:00–15:05, so they'd merge and a counted GET couldn't show. At 14:59:00 it gets its own bucket. +12 means metadata is free, +13 means it counted. Our own xet-read-token call read 0 on its own, so I expect +12.

On your question: not quite, there are still untested knocks. A logged-in 404 on a private repo, a 404 for a file that doesn't exist in a public repo, and anything outside /resolve/. Your receipt says the first should read like p0. The second is the open one: the repo exists but the file doesn't. If the counter keys on the repo, a 404 there could count like a knock at the gate.