Eric Seidel
463ace0c56
* feat: add resumable downloads with streaming to disk
Previously, failed patch downloads were lost entirely — the updater
would re-download from scratch on every retry, creating a doom loop for
users on poor networks. Downloads were also fully buffered in memory.
This changes the download abstraction from returning bytes in memory
(`DownloadFileFn`) to streaming directly to disk with resume support
(`DownloadToPathFn`). The new signature accepts a `resume_from` byte
offset and the default implementation uses HTTP Range headers.
Key changes:
- DownloadToPathFn streams to file, sends Range header when resuming
- DownloadResult returns total_bytes and content_length from server
- DownloadState sidecar JSON tracks URL/patch/size for resume decisions
- compute_resume_offset detects valid partial downloads to resume
- Post-download size validation catches truncated downloads
- cleanup_download_artifacts removes compressed files + sidecars after
successful install (fixes pre-existing leak of download artifacts)
The fn-pointer signature maps directly to a future C callback for
platform-native download backends (iOS NSURLSession, Android
DownloadManager).
* fix: address self-review issues in download implementation
- Parse Content-Range header for 206 responses to get total file size
(reqwest's content_length() only returns the partial body size)
- Only append to existing file when server actually returns 206, not
just when resume_from > 0 (handles servers that ignore Range header)
- Remove incorrect resume_from offset addition to expected_size
- Clean up download artifacts on size mismatch before bailing
- Restore parent directory creation in download_to_path wrapper
* refactor: remove expected_hash from DownloadState
The hash stored in the sidecar was the *inflated* file hash from the
server response — it couldn't validate the compressed partial download
and was never used for resume decisions. The URL match already
determines whether to resume, and the real hash check happens after
inflate using the fresh server response. Simplifies the sidecar to
just url, patch_number, and expected_size.
* test: add coverage for resumable downloads + review fixes
- Add 12 new tests covering:
- compute_resume_offset: no sidecar, matching sidecar, mismatched URL,
empty file
- cleanup_download_artifacts: removes file+sidecar, noop when missing
- Integration: successful update cleans up artifacts
- Integration: partial download resumes via 206 with mockito
- Integration: URL change triggers fresh download
- parse_content_range_total: valid, missing header, unknown size (*)
- Extract parse_content_range_total as testable helper (was inline chain)
- Add expected_hash back to DownloadState — catches the case where a
patch is deleted and re-added with same number but different content
- Add "rsplit" to cspell config
* feat: add orphan cleanup for download directory + WriteFile comment
Scan the download directory before each download and remove any files
that don't belong to the current patch number. We own this directory
entirely, so anything from a prior patch, a crashed inflate (.full),
or an unrecognized file is safe to delete. This prevents gradual
accumulation of orphaned partial downloads over time.
Also adds a TODO comment explaining the reuse of WriteFile context for
seek operations (FileOperation lacks a SeekFile variant).
* test: add coverage for hash mismatch, patch number mismatch, corrupt sidecar
- compute_resume_offset_mismatched_hash: same URL but hash changed
(patch deleted and re-added), verifies fresh download
- compute_resume_offset_mismatched_patch_number: sidecar for different
patch number, verifies fresh download
- compute_resume_offset_corrupt_sidecar: garbage JSON in sidecar,
verifies graceful fallback to fresh download
* test: cover download size mismatch and unknown content-length paths
- update_fails_on_download_size_mismatch: mock returns content_length
that doesn't match total_bytes, verifies error + artifact cleanup
- update_succeeds_when_content_length_unknown: verifies the size
validation is skipped when content_length is None
* fix: replace fake hash strings to pass cspell
* chore: remove unnecessary TODO comment about FileOperation::SeekFile
* refactor: panic in test download mocks that should never be called
Tests where the download is never reached (patch check fails or no
patch available) now panic instead of returning dummy data, making it
explicit that the mock shouldn't be invoked.
* refactor: add UNEXPECTED_DOWNLOAD/UNEXPECTED_REPORT test constants
Shared panicking constants for test mocks that should never be called.
Tests use these by name instead of writing inline panic closures,
making intent clearer and avoiding uncoverable dead code in closures.
* test: add coverage for handle_download_result
Tests for the download-specific HTTP response handler:
- 200 OK: accepted
- 206 Partial Content: accepted (for resumed downloads)
- 500: rejected with error message
* fix: update missed download fn signatures in tests
Two test sites in updater.rs were not updated to the new
DownloadToPathFn 3-argument signature:
- set_noop_network_hooks in multi_engine_tests used old 1-arg closure
- update_starts_fresh_when_url_changes had unused patch_bytes variable
* test: add coverage for handle_download_result error branches
Mirror the existing handle_network_result_no_internet and
handle_network_result_unknown_error tests for the download variant.
These exercise the connection error and builder error paths in
handle_download_result that were previously uncovered.
* fix: adapt resumable downloads to ureq (post-rebase cleanup)
- Replace reqwest with ureq for download_to_path_default
- Remove handle_download_result (ureq's handle_network_result handles 206)
- Fix TempDir::new("prefix") → TempDir::new() for tempfile crate
- Remove unused Read/Write imports
Updater library
This is the Rust side of the Shorebird code push system. This is built
in Rust with a C API for easy calling from other languages, most notably
for linking into libflutter.so.
The primary modification Shorebird makes to the stock Flutter engine is adding support for the updater library (this repo). The updater library is written in Rust and is used to update the code running in the Flutter app. The updater library is built as a static library and is linked into the Flutter engine during build time.
Parts
library: Runtime library linked into Flutter Engine to unpack and apply updates. Build into libupdater.a and linked into libflutter.so.patch: Developer tooling to package Shorebird updates ("patches"). Built intopatch.exeand downloaded and run byshorebirdcommand line tool: https://github.com/shorebirdtech/shorebird/tree/main/packages/shorebird_cli.shorebird_code_push: The Dart bindings for communicating with the updater library from within a Flutter app. Published to pub.dev, usage is optional by developers.
Most interesting code is in the library directory. There is also
a README.md in that directory explaining the design.
Developing
It's best to edit this repository from within an engine checkout. See BUILDING_ENGINE.md for instructions on how to set up an engine checkout.
The workflow I use involves 2 to 3 VSC windows:
- Opening the engine
src.
In that terminal I:
cd third_party/updater
- To build the updater as part of the engine:
cargo ndk --target aarch64-linux-android build --release && \
ninja -C ../../out/android_release_arm64 && say "done"
The cargo part should not be needed, but I haven't yet done the work to integrate the Rust code into the gn files for the Flutter engine yet.
I add say "done" to the end as linking can take several minutes for release
Android builds.
-
In a second window, I open
code third_party/updater. I do this because otherwise therust_analyzercan't seem to find the rust code. We could fix this by adding the directory to the VSC workspace, but I'm not sure where we would put the workspace file in the first place.srcis actuallyshorebirdtech/buildrootand is controlled viagclientbyshorebirdtech/engine/DEPS. -
In a third window I open my test app. e.g.:
flutter create test_app
cd test_app
shorebird init
shorebird release
code .
- To run the test app with my local engine I use:
shorebird run --local-engine-src-path $HOME/Documents/GitHub/engine/src \
--local-engine android_release_arm64
You may also need to build out/host_release once as flutter build looks for
some Dart .dill files in host_release.
Coverage
We'd like to get to 100% coverage but aren't there yet.
https://github.com/taiki-e/cargo-llvm-cov is the best tool I've found for generating coverage reports.
Install: https://github.com/taiki-e/cargo-llvm-cov#installation
cargo llvm-cov will then generate the report.