How grabnr downloads one file over every network connection

A browser downloads a file down one route: whichever network interface the operating system picks. If you are on Wi-Fi with an Ethernet cable and a phone on USB, two of those three links sit idle. grabnr is a download manager I wrote to use all of them for the same file at once.

The idea: one queue, many workers

grabnr asks the server for the file size, splits the file into chunks and puts them in a shared queue. Each network link gets a worker. A worker takes the next chunk, downloads it over its own link and goes back for another. Nobody decides in advance how much each link should carry. A fast link comes back to the queue more often, so it takes more chunks. A slow link takes fewer. If a link vanishes, its unfinished chunk goes back in the queue.

The nice part of this design is that it needs no speed estimate. It adapts as conditions change, which matters on Wi-Fi and phone tethering where speed moves around.

Making a connection use a specific link

The hard part is not the queue. It is making a socket leave through a particular interface. A normal connection follows the default route, so every worker would end up on the same link. grabnr binds each worker socket to its interface: IP_BOUND_IF on macOS and SO_BINDTODEVICE on Linux. On Windows it binds by source address, which works when each adapter has its own router.

There is a small command to check this on your own machine. grabnr spike reports the public IP each link appears as. If two links report different addresses, the bind worked and they really are separate routes.

Details that took the most thought

• HTTP/2 is avoided on purpose. HTTP/2 multiplexes many requests over one TCP connection, which would put all the workers back on a single link. grabnr uses HTTP/1.1 so each worker owns its own connection.
• Tail racing. The last few chunks are the slowest part of a multi-link download, because one slow link can hold up the finish. Near the end, the remaining chunks are raced on several links and whichever finishes first wins.
• Links come and go. Plug in a phone in the middle of a download and it joins the queue. Unplug one and its work moves to the others. A changed IP address restarts that link.
• Resume that survives a crash. Progress inside a chunk is saved, and finished chunks are checked by checksum when a download resumes, so a power cut cannot leave silent holes in the file.
• Not only HTTP. FTP, SFTP, BitTorrent and magnet links, and HLS streams are split over the links the same way.

The browser extension and the local API

Most downloads start in a browser, so grabnr ships an extension for Chrome, Brave, Edge, Firefox and Safari. When you start a download, the extension hands it to the app and cancels the browser copy. If grabnr is not running, the browser downloads normally, so the extension can never leave you without a download.

The app talks to the extension over a local API on 127.0.0.1:17653. That address is only reachable from your own machine, and the app refuses requests that come from web pages. Adding a download is only accepted from the official extension's fixed ID or with a token. This matters because a download manager that any website can command would be a security hole.

A security bug I fixed

grabnr supports sign-in for downloads (Basic and Bearer headers, and cookies). An earlier version sent those credentials to the host a download was redirected to, and to mirrors on other hosts. That is wrong: a redirect to another site should never receive your login. Credentials now go only to the host you gave, over the same scheme and port. It was a good reminder that the dangerous bugs in a networking tool are the quiet ones, where everything still appears to work.

Try it

# see your connections and which ones can be used
grabnr links

# download over every link
grabnr get https://proof.ovh.net/files/100Mb.dat -o ~/Downloads

# only some links, with a speed cap and a checksum
grabnr get URL --only en0,en5 --limit 20 --checksum sha256:<hex>

# check that traffic really leaves through each link
grabnr spike --secs 8

What it cannot do

I also want to be plain about the state of the testing. The engine is tested against a local range server and the interface binding is checked by grabnr spike, but I have not yet measured the speed-up on two physical connections with different gateways. That measurement is next on my list, and I will publish the numbers when I have them.

grabnr is written in Rust with a Tauri and React desktop app, a command-line tool on the same engine, and a browser extension for Chrome, Brave, Edge, Firefox and Safari. It is MIT licensed. You can read the code on GitHub or see the project site.