fn http_get(
url: &str,
expected_total: u64,
on_progress: &mut dyn FnMut(u64, u64),
) -> Result<Vec<u8>, DownloadError>Expand description
Fetch a URL with exponential-backoff retries, streaming the body and reporting
on_progress(downloaded, total) as it goes (total from expected_total, else the
response Content-Length, else 0 = unknown). Progress resets to 0 at the start of each
attempt (a retried download restarts from the beginning).
Weights are hosted on OSF, which throttles bursts of anonymous downloads with a 403 (not 429) — so when many jobs fetch weights at once (e.g. the CI model matrix) individual requests fail spuriously. We retry on 403/408/425/429/5xx and transport errors with backoff (0.5s, 1s, 2s, 4s) plus URL-derived jitter so concurrent fetchers of different files desynchronise. Permanent failures (404, 401, other 4xx) fail fast.