Skip to main content

http_get

Function http_get 

Source
fn http_get(
    url: &str,
    expected_total: u64,
    on_progress: &mut dyn FnMut(u64, u64),
) -> Result<Vec<u8>, DownloadError>
Expand description

Fetch a URL with exponential-backoff retries, streaming the body and reporting on_progress(downloaded, total) as it goes (total from expected_total, else the response Content-Length, else 0 = unknown). Progress resets to 0 at the start of each attempt (a retried download restarts from the beginning).

Weights are hosted on OSF, which throttles bursts of anonymous downloads with a 403 (not 429) — so when many jobs fetch weights at once (e.g. the CI model matrix) individual requests fail spuriously. We retry on 403/408/425/429/5xx and transport errors with backoff (0.5s, 1s, 2s, 4s) plus URL-derived jitter so concurrent fetchers of different files desynchronise. Permanent failures (404, 401, other 4xx) fail fast.