New Streamed Downloader:
Tiny chunks of files are directly downloaded and streamed to the Minerva Server. Files are never kept on your machine.
Leaderboard: https://firehose.minerva-archive.org/
Note: The backend is still very much a work-in-progress, expect downtime like you did before.
Commits
- 2b42421: Fix link to the correct pypi project page (Citrodata) #23
- 0d60201: Added configuration for setting the auth callback host and port (SpoopyTim) #22
- e39604a: Linted with ruff (SpoopyTim) #22
- 06e1335: WIP merge with main code of hyperscraper (rlaphoenix)
- d77fbe5: Fully utilize multithreading, more error handling (rlaphoenix)
- 6776282: Remove unused worker_thread and relevant code (rlaphoenix)
- 387c1c9: Correct the chunk_size and job size values (rlaphoenix)
- d72706b: Remove Job Cache system, new Hyperscraper cant rlly cache (rlaphoenix)
- a6000fa: Fix is_downloaded check (rlaphoenix)
- 8175a88: Update ws_message to communicate be crafted byte payloads (rlaphoenix)
- 866eca9: Create new data classes for Websocket Messages, Bump Server version (rlaphoenix)
- 1b70cb8: Create Job State dataclass, use ChunkInfo across code (rlaphoenix)
- dffc1cd: Remove --max-cache-size (rlaphoenix)
- 24b4173: Exit the download when its finished (i need sleep) (rlaphoenix)
- d1fe520: maybe put the break in the right place (im going to bed rn) (rlaphoenix)
- fb3aa95: Stop freezing on reconnect please thank you (rlaphoenix)
- fa90207: Clear the event thing after a reconnect, so it requests new tasks (turai)
- 4f7b669: Improve CI/CD (rlaphoenix)
- 9e714dc: Add job size environment variables, make retries name consistent (rlaphoenix)
- 824ed2f: Update readme to be relevant to current codebase, add Troubleshooting (rlaphoenix)
- dc94f7e: Remove unnecessary lock (rlaphoenix)
- adb4fd8: Separate Upload and Download Speed (rlaphoenix)
- 251fd56: Improve looping code visually, some small tweaks (rlaphoenix)
- c710a9d: Normalize URL strings because Myrient expects them in a weird way (rlaphoenix)
- f930101: State in stats bar if worker is down and for how long (rlaphoenix)
- 0f86c3c: Separate Uptime from other columns, improve UI (rlaphoenix)
- eb7cc0a: Remove unused packages (rlaphoenix)
- f61702c: Add leaderboard stats, fix down check if down on startup (rlaphoenix)
- 8fd960e: Heavily improve the async threading, add logging (rlaphoenix)
- f8a0051: Update Dockerfile to use 'uv sync' instead of 'uv venv' (Oleksandr Kalko) #26
- f03195c: No need for the extra groups, those are only used in development (rlaphoenix) #26
- c6974e3: Bump to v2.0.0 (rlaphoenix)
