Brandon DeRosier
fe3733374c
* fix(inflate): preserve real upstream cause + add diagnostic byte counters
The current `inflate()` always reports "Decompression of patch failed"
when the decompression thread's join returns Err -- but that masks the
common in-the-wild failure shape where bipatch (or the output write)
errors first, `fresh_r` gets dropped, and the decompression thread's
next write returns BrokenPipe purely as downstream noise. The misleading
message has made hard-to-reproduce field reports very hard to diagnose.
Two changes:
1. Error attribution. When both `patch_result` and `decompress_result`
error, `patch_result` is now reported as the primary cause (the
upstream one in the BrokenPipe shape) and the decompression error
is folded into the context chain so a genuine zstd corruption error
doesn't disappear. The `(Ok, Err)` case (decompression trailing
error after bipatch already finished, e.g. extra zstd frame bytes)
is now treated as success with an info-level log -- the output was
produced and will be hash-checked downstream.
2. Diagnostic counters. Every error path now carries a one-line
`InflateDiagnostics` snapshot in its context. The fields cover the
whole pipeline:
patch_file=<size> -- on-disk compressed patch size
base_size=<size> -- base reader length (probed at start)
decompressed_into_pipe=<bytes> -- bytes the zstd thread wrote
decompressed_out_of_pipe=<bytes> -- bytes bipatch consumed
base_read=<bytes> -- bytes bipatch read from base
base_last_seek_pos=<offset> -- where base reader last seeked
base_seeks=<count> -- how many times bipatch seeked the base
output_written=<bytes> -- bytes copied to output
Concretely: a field crash where `base_read=0` immediately implicates
the base reader; `decompressed_into_pipe=8 output_written=0`
implicates bipatch parsing right after the header; a `base_size`
that disagrees with what the host extracted while building the patch
implicates iOS install-time munging of the snapshot bytes.
These counters are tracked via Arc<AtomicU64> shared between the
inflate body and the decompression thread, wrapped around each
stream end with three small Read/Seek/Write adapter structs.
Bipatch initialization failure also drains and surfaces any
concurrent decompression error (previously the thread leaked
detached on this path).
Tests updated/added:
- `inflate_fails_with_corrupt_zstd_data_reports_bipatch_init_failure`
(replaces `inflate_fails_with_corrupt_zstd_data`): garbage zstd
body now surfaces as bipatch-init failure with full counters.
- `inflate_when_both_streams_error_reports_patch_side_primary`
(replaces `inflate_reports_decompression_error_as_primary`): valid
bipatch header + corrupt second zstd frame -- patch_result wins,
decompression error is in the context chain.
- New `inflate_diagnostics_include_all_counters`: pins the diagnostic
format contract so future refactors can't accidentally drop a
field that in-the-wild bug reports depend on.
Motivation: digging into a reproducible iOS-standalone patch failure
where the only on-device log line was "Update failed: Decompression of
patch failed" -- which turned out to be downstream noise. With these
counters the next investigator can read the failing pipeline state
directly from the device log.
* fix(inflate): keep success-path logging at original chattiness
Move the success-case diagnostic counters from `shorebird_info!` to
`shorebird_debug!` so production doesn't grow a ~200-character
diagnostic line on every successful patch install. The happy path now
emits the same single short "Patch successfully applied to X" info line
as before; counters land at debug level for development / opt-in
production diagnostics. Also demoted the "Base file size: X bytes"
line that I'd added at info level back to debug. Error-path diagnostics
are unchanged — those fire only on failure where the byte counts are
exactly what the next investigator wants.
Net effect on production log volume: zero new info-level lines per
inflate (matches pre-PR shape). Failure paths retain full diagnostic
counters.
* fix(inflate): restore success-path diagnostic blob at info level
Verified offline that the concern about "hundreds of lines of noise"
isn't the right shape here: it's a single info-level line per inflate
(rare event, maybe daily per user), just with the byte counters
appended. Same line count as before this PR; only the line length
grows. Keeping the success blob at info enables cross-run forensic
comparison — e.g. "did base_size change between this user's last
working patch and the broken one?" — which is exactly the kind of
question the in-the-wild bug report is going to want answered.
* fix(inflate): cargo fmt
Updater library
This is the Rust side of the Shorebird code push system. This is built
in Rust with a C API for easy calling from other languages, most notably
for linking into libflutter.so.
The primary modification Shorebird makes to the stock Flutter engine is adding support for the updater library (this repo). The updater library is written in Rust and is used to update the code running in the Flutter app. The updater library is built as a static library and is linked into the Flutter engine during build time.
Parts
library: Runtime library linked into Flutter Engine to unpack and apply updates. Build into libupdater.a and linked into libflutter.so.patch: Developer tooling to package Shorebird updates ("patches"). Built intopatch.exeand downloaded and run byshorebirdcommand line tool: https://github.com/shorebirdtech/shorebird/tree/main/packages/shorebird_cli.shorebird_code_push: The Dart bindings for communicating with the updater library from within a Flutter app. Published to pub.dev, usage is optional by developers.
Most interesting code is in the library directory. There is also
a README.md in that directory explaining the design.
Developing
It's best to edit this repository from within an engine checkout. See BUILDING_ENGINE.md for instructions on how to set up an engine checkout.
The workflow I use involves 2 to 3 VSC windows:
- Opening the engine
src.
In that terminal I:
cd third_party/updater
- To build the updater as part of the engine:
cargo ndk --target aarch64-linux-android build --release && \
ninja -C ../../out/android_release_arm64 && say "done"
The cargo part should not be needed, but I haven't yet done the work to integrate the Rust code into the gn files for the Flutter engine yet.
I add say "done" to the end as linking can take several minutes for release
Android builds.
-
In a second window, I open
code third_party/updater. I do this because otherwise therust_analyzercan't seem to find the rust code. We could fix this by adding the directory to the VSC workspace, but I'm not sure where we would put the workspace file in the first place.srcis actuallyshorebirdtech/buildrootand is controlled viagclientbyshorebirdtech/engine/DEPS. -
In a third window I open my test app. e.g.:
flutter create test_app
cd test_app
shorebird init
shorebird release
code .
- To run the test app with my local engine I use:
shorebird run --local-engine-src-path $HOME/Documents/GitHub/engine/src \
--local-engine android_release_arm64
You may also need to build out/host_release once as flutter build looks for
some Dart .dill files in host_release.
Coverage
We'd like to get to 100% coverage but aren't there yet.
https://github.com/taiki-e/cargo-llvm-cov is the best tool I've found for generating coverage reports.
Install: https://github.com/taiki-e/cargo-llvm-cov#installation
cargo llvm-cov will then generate the report.