* perf: mmap libapp.so out of the APK instead of buffering in RAM
`open_base_lib` previously read the entire decompressed libapp.so into a
Vec<u8> via `read_to_end` and handed bipatch a Cursor over that buffer.
For large apps libapp.so can be tens of megabytes, and that allocation
happens immediately after the patch download is buffered to disk — a
plausible OOM trigger on memory-constrained devices (we have at least
one customer report on OnePlus where the patch install appears to halt
silently right after the download completes).
Modern AGP (3.6+) defaults to extractNativeLibs=false, which keeps
libapp.so STORED uncompressed inside the APK so the dynamic linker can
mmap it directly. When that's the case, do the same: find the entry's
data offset via the zip crate, drop the archive, reopen the APK, and
mmap the entry's slice. Cursor<Mmap> implements Read + Seek, which is
what bipatch's `Reader::new(patch, base)` requires.
When the entry isn't stored uncompressed (older builds, or builds that
explicitly compress native libs), fall back to the previous buffered
read so we always succeed.
Mmap doesn't change the peak working set when bipatch traverses the
whole base linearly, but file-backed mappings are clean and reclaimable
under memory pressure where an anonymous Vec is not, and we avoid the
~2x transient allocation peak from `read_to_end` growing the buffer.
Tests cover both paths (stored → mmap, deflated → buffered) on the host.
* ci: add mmap, memmap, SIGBUS to cspell dictionary
* docs: integration tests design proposal
Adds docs/integration_tests.md proposing a desktop-only integration suite
that drives the Dart `ShorebirdUpdater` API against the real Rust core via
FFI, with a fake HTTP server and per-test tempdir state. Describes a new
`library_test_hooks` cdylib that reaches into `updater` via a `test-hooks`
Cargo feature, so production C API stays clean. Status: design exploration,
not a committed plan.
* feat: stage 1 integration test harness for shorebird_code_push
Lands the test_hooks crate, the Dart-side loader, and one trivial
end-to-end test, per docs/integration_tests.md stage 1.
Pieces:
- New `library_test_hooks` workspace crate (cdylib). Depends on `updater`
with a new `test-hooks` Cargo feature that widens the visibility of
internal items (currently `testing_reset_config`) for sibling-crate
access only — production builds do not enable it. The crate also
re-exports `updater::c_api::dart::*` and `updater::c_api::engine::*`,
which keeps the rlib's `#[no_mangle]` symbols out of DCE so the
resulting cdylib carries both the production C API and the new
`shorebird_test_*` hooks.
- Workspace `default-members` excludes `library_test_hooks` so plain
`cargo build` / `cargo test` invocations don't unify the `test-hooks`
feature into production builds.
- `shorebird_code_push/test/integration/all_test.dart` (single file,
with a header comment explaining why) loads the cdylib via
`DynamicLibrary.open`, reassigns the existing `@visibleForTesting`
`Updater.bindings` setter, and exercises both surfaces. The build
helper shells out to `cargo build -p library_test_hooks`; if it
fails, `markTestSkipped` keeps the suite green on machines without
a working Rust toolchain.
- ffigen config (`ffigen_test_hooks.yaml`) generates the test-only
Dart bindings under `test/integration/generated/`, not `lib/`.
- CI: `library_test_hooks` is added to the rust_crate matrix, and the
shorebird_code_push job triggers on `library/**` and
`library_test_hooks/**` so cdylib changes can't break the Dart-side
integration suite without CI noticing.
Verified locally: 232 Rust unit tests + 42 Dart tests (41 existing + 1
integration) green; clippy/fmt/cspell clean; production cdylib does
not contain `shorebird_test_reset` (`nm` confirms feature isolation).
Stages 2 (FakePatchServer + golden path) and 3 (adversarial scenarios)
land separately.
* fix(integration test): early-return on skip and bump per-test timeout
Two issues caught by Shorebird CI on PR #353:
1. `markTestSkipped` does not abort test execution — it only flags the
test as skipped on its way out. The body kept running and crashed on
the `late testHooks` field when the cdylib build had failed in
setUpAll. Move the skip check into the test body itself with an
early return; drop the (no-op) skip in `setUp`.
2. Default per-test timeout (30s) also covers `setUpAll`. A cold
`cargo build -p library_test_hooks` compiles `updater` and ~100
transitive deps, which can run minutes on CI. Bump to 10 minutes
via `@Timeout` on the library.
Verified locally: passes when cargo is on PATH (1 passed), reports a
clean skip and exit 0 when cargo is removed from PATH (1 skipped).
* refactor(integration test): drop ! by promoting testHooks to late final
`testHooks` was nullable so accessing it after the skipReason check
required `!`, and `markTestSkipped(skipReason!)` had the same smell.
Make `testHooks` `late final` (non-nullable, throws if read before
setUpAll assigns) and pull `skipReason` into a non-null local before
use. Same control flow, no bang operators.
Reduces binary size of the updater staticlib when linked into the
Flutter engine. On host macOS arm64 the dylib shrinks from 3.07 MB
to 2.56 MB; the biggest iOS savings come from dropping the DWARF
unwind tables (__eh_frame, __gcc_except_tab, __unwind_info) that
panic=unwind generates.
- panic = "abort" in [profile.release]: kills unwind-table generation
and eliminates dead panic-unwinding code paths. We don't use
catch_unwind anywhere in the library. log-panics still runs its
panic hook before abort, so panic messages continue to surface to
logcat/oslog during development.
- Drop `backtrace` feature from anyhow and `with-backtrace` from
log-panics: removes gimli + addr2line + rustc_demangle (~65 KB of
symbolication machinery) that customers can't read anyway. A
future in-engine crash reporter will symbolize native stacks,
making these features redundant.
Run `cargo update` to bump transitive dependencies, fixing 10 of 11
alerts (h2, ring, idna, mio, tokio, bytes, time, quinn-proto,
rustls-webpki, unsafe-libyaml).
Replace deprecated `tempdir` dev-dependency with `tempfile` to
eliminate the `remove_dir_all` vulnerability (the last alert).
* refactor: replace reqwest with ureq to reduce binary size
reqwest's blocking API is built on top of its async implementation,
pulling in tokio, hyper, futures, and ~84 other transitive dependencies
even though we only make simple synchronous HTTP calls.
ureq is a synchronous-only HTTP client that eliminates the async
runtime entirely. This reduces transitive dependencies from 227 to 143
and the linked dylib from 4.5 MB to 3.7 MB (-18%). The .a archive
drops from 28 MB to 26 MB, but real savings will be larger once
linked into libflutter with dead code stripping.
The network API surface is unchanged — three functions (patch check,
file download, event reporting) using POST/GET with JSON.
* chore: add ureq to spell check dictionary
* refactor: use into_body() instead of body_mut() where response is consumed
* fix: simplify network error matching to avoid fragile string checks
Consolidate HostNotFound, ConnectionFailed, and all Io errors into
a single network-error arm instead of pattern-matching on error
message strings that could change across OS versions or locales.
* chore: add TODO for misleading network error message
I just ran `cargo clippy -- -W clippy::pedantic` and fixed things.
These are more invasive that the default set and the remaining
warnings are mostly about our (abysmal) public docs missing
Error and Panic sections to explain errors and panicks.
This is all for shorebirdtech/shorebird#695
I ended up fixing a lot of things in our Rust code while I was in there, including:
* Using PathBuf instead of Sting wherever we are holding onto Path objects (this removed a bunch of code which was converting in and out of these).
* Made a variety of functions private (the default) that were previously mistakenly pub.
* Moved the android hacks around libapp_paths closer to the C++ code. Previously we were storing all of the libapp_paths on ResolvedConfig and handling the android hacks during patch install. Now we're doing the android path hacks on init, eventually we'll remove them entirely by making the C++ code pass us in the android app_dir or even the resolved .apk we need instead of the libapp_paths.
* Fixed several tests to use /dir/lib/arch/libapp.so paths instead of 'libapp.so' now that the init code applies the android hacks an expects the deeper paths. Again, this will go away when we move the android hacks into c++.
* Added a test which confirms that if we init twice we log instead of trying to init again (this was one behavior which FCM was triggering as part of fix: ANR when using Firebase Cloud Messaging with Shorebird shorebird#695
* Add a test which confirms that further calls to update while an update is ongoing just fail out quickly instead of hanging.
* Moved to serial_test instead of a separate per-thread config system for testing. This makes our tests operate more like the code does in the wild, at the cost of needing to annotate a test as #[serial] any time it needs to use shorebird_init.
* Removed ResolvedConfig and now call it UpdateConfig. ResolvedConfig previously had a bool on it to tell if it was initialized, that's now represented as Option instead which is more rusty.
* Changed how we handle NetworkHooks from being compile-time test switched to using function pointers held off of UpdateConfig.network_hooks. I think this makes more sense? I did this to remove the special ThreadConfig object which was used during unit tests, but didn't exist in production. I think the way we mock networking could still be improved.
Fixesshorebirdtech/shorebird#235🤞
This moves us away from depending on libapp.so having been extracted from the APK and instead we always extract it ourselves.
This was needed so that it could be called from Dart as well as flutter_main/C++.
It turns out flutter_main does not run on the "ui thread", so when Dart was calling
into the updater it would panic due to thinking the updater (which was using a thread local) was not yet initialized.
Also added the log-panics crate on Android so that panics appear in adb logcat.
dependabot[bot]