shorebird-updater/library

Shorebird CodePush Updater

The rust library that does the actual update work.

Design

The updater library is built in Rust for safety (and modernity). It's built as a C-compatible library, so it can be used from any language.

The library is thread-safe, as it needs to be called both from the flutter_main thread (during initialization) and then later from the Dart/UI thread (from application Dart code) in Flutter.

The overarching principle with the Updater is "first, do no harm". The updater should "fail open", terms of continuing to work with the currently installed or active version of the application even when the network is unavailable.

The updater also needs to handle error cases conservatively, such as partial downloads from a server, or malformed responses (e.g. a proxy interfering) and not crash the application or leave the application in a broken state.

Every time the updater runs it needs to verify that the currently installed patch is compatible with the currently installed base version. If it is not, it should refuse to return paths to incompatible patches.

The updater also needs to regularly verify that the current state directory is in a consistent state. If it is not, it should invalidate any installed patches and return to a clean state.

Not all of the above is implemented yet, but such is the intent.

Architecture

The updater is split into separate layers. The top layer is the C-compatible API, which is used by all consumers of the updater. The C-compatible API is a thin wrapper around the Rust API, which is the main implementation but only used directly for testing (see the cli directory).

Thread safety is handled by a global configuration object that is locked when accessed. It's possible I've missed cases where this is not sufficient, and there could be thread safety issues in the library.

• c_api (module) - C-compatible API
  • c_file.rs - a read-seek interface usable by the engine (used to provide access to iOS patch files)
  • mod.rs - the implementation of the C API.
• src/lib.rs - Rust API (and crate root)
• src/updater.rs - Core updater logic
• cache (module) - On-disk state management
  • disk-io.rs - Manages reading and writing serializable state to disk
  • mod.rs - Cache management
  • patch_manager.rs - Patch file state management. Owned by UpdaterState.
  • updater_state.rs - Public API for this module. Provides functions that trigger updates to the internal Boot State Machine (detailed below).
• src/config.rs - In memory configuration and thread locking
• src/cache.rs - On-disk state management
• src/logging.rs - Logging configuration (for platforms that need it)
• src/network.rs - Logic dealing with network requests and updater server

Rust

We use normal rust idioms (e.g. Result) inside the library and then bridge those to C via an explicit stable C API (explicit enums, null pointers for optional arguments, etc). This lets the Rust code feel natural and also gives us maximum flexibility in the future for exposing more in the C API without having to refactor the internals of the library.

https://docs.rust-embedded.org/book/interoperability/rust-with-c.html are docs on how to use Rust from C (what we're doing).

https://github.com/RubberDuckEng/safe_wren has an example of building in Rust and exposing it with a C api.

Integration

The updater library is built as a static library, and is linked into the libflutter.so as part of a custom build of Flutter. We also link libflutter.so with the correct flags such that updater symbols are exposed to Dart.

Building for Android

The best way I found was to install: https://github.com/bbqsrc/cargo-ndk

cargo install cargo-ndk
rustup target add \
    aarch64-linux-android \
    armv7-linux-androideabi \
    x86_64-linux-android \
    i686-linux-android
cargo ndk -t armeabi-v7a -t arm64-v8a build --release

When building to include with libflutter.so, you need to build with the same version of the ndk as Flutter is using:

You'll need to have a Flutter engine checkout already setup and synced. As part of gclient sync the Flutter engine repo will pull down a copy of the ndk into src/third_party/android_tools/ndk.

Then you can set the NDK_HOME environment variable to point to that directory. e.g.:

NDK_HOME=$HOME/Documents/GitHub/engine/src/third_party/android_tools/ndk

Then you can build the updater library as above. If you don't want to change your NDK_HOME, you can also set the environment variable for just the one call:

NDK_HOME=$HOME/Documents/GitHub/engine/src/third_party/android_tools/ndk cargo ndk -t armeabi-v7a -t arm64-v8a build --release

Imagined Architecture (not all implemented)

Assumptions (not all enforced yet)

• Updater library is never allowed to crash, except on bad parameters from C.
• Network and Disk are untrusted.
• Running code is trusted.
• Store-installed bundle is trusted (e.g. APK).
• Updates are signed by a trusted key.
• Updates must be applied in order.
• Updates are applied in a single transaction.

State Machines

Boot State Machine

This state machine tracks the process of the engine starting up, with or without a patch. These state transitions happen whether or not a new patch is available, but in the context of the updater, we only care about the case where we are booting from a patch.

The patch boot state is internal to the PatchManager and stored on disk in patches_state.json. It contains three fields, all of which are Optional PatchMetadata structs:

• last_boot_patch: The last patch that was successfully booted.
• last_attempted_patch: The last patch that we attempted to boot.
• next_boot_patch: The next patch that we will attempt to boot.

This state machine has the following states. It can only move forward through them.

1. Ready - The engine is initialized but has not started booting yet.
2. Booting - The engine has started booting.
3. Success - The engine successfully booted.
4. Failure - The engine failed to boot.

State is advanced through calls to the following methods of the ManagePatches trait, which PatchManager implements:

• record_launch_start: Moves from Ready to Booting
  • next_boot_patch is the patch that we will attempt to boot, i.e., the "current" patch.
  • last_attempted_patch is set to next_boot_patch.
• record_launch_success: Moves from Booting to Success
  • last_boot_patch is set to next_boot_patch.
  • Artifacts for patches older than last_boot_patch are deleted.
• record_launch_failure: Moves from Booting to Failure
  • next_boot_patch artifacts are deleted.
  • next_boot_patch is set to either:
    • last_boot_patch if it is still valid, or
    • None, if last_boot_patch is None or invalid.

These are effectively no-ops if we are not booting from a patch.

Assumptions (not currently enforced, but should as possible):

• This state machine will have advanced at least as far as the Booting state before the Patch Check State Machine (below) is started.
• Calls to mutate state will not come out-of-order. For example, record_launch_failure will not be called before record_launch_start. This is important because PatchManager state is implicit - it does not track which state it is in.

Patch Check/Update State Machine

This state machine tracks the process of checking for new patches. It is managed by the code in updater.rs and does not have any on-disk state. It has the following states:

1. Ready - Ready to check for updates.
2. Send queued events (e.g., report that a patch succeeded or failed to boot) a. Move to checking once events, if any, have been reported.
3. Checking for new patches - A PatchCheckRequest is issued but not completed. a. If no patch is available, move back to Ready. b. If a new patch is is available, move to Downloading Patch.
4. Downloading Patch - A patch is available, we're downloading it. a. If the download fails, move back to Ready. b. If the download succeeds, move to Inflating Patch.
5. Inflating Patch - A patch has been downloaded, we're inflating it. a. Attempt to inflate the patch (apply a bidiff to the current release). b. If the patch is valid, queue a PatchInstallSuccess event and move to Ready. c. If the patch is invalid, queue a PatchInstallFailure event and move to Ready.

Changes in this state can be triggered by:

1. The engine via the C API.
2. The user via the C API (using the shorebird_code_push package).
3. Network activity.

• Server is authoritative, regarding current update/patch state. Client can cache state in memory. Not written to disk.
• Patches are downloaded to a temporary location on disk.
• Client keeps on disk (in state.json):
  • Current release version. Set when the app launches. If the app is updated to a new release version, all state is invalidated.
  • Queue of PatchEvents. This is cleared once events are sent to our servers.

Trust model

• Network and Disk are untrusted.
• Running software (including apk service) is trusted.
• Patch contents are signed, public key is included in the APK.

Patch Verification Modes

The updater supports two patch verification modes, configured via patch_verification in shorebird.yaml. Both modes require a patch_public_key to be configured for signature verification to occur.

Strict Mode (default)

patch_verification: strict

In Strict mode, patch signature verification happens at boot time. This provides the strongest security guarantee because it detects any potential on-disk tampering to the patch file after installation (e.g., if an attacker were to modify the patch file on disk between app launches). However the practical risk to such an attack is very low, since patches are stored within the app's protected storage. An attacker in this case would need to have already compromised the app itself, or the system (e.g. via a rooted device). However if an attacker has compromised the system (rooted) they could already modify the APK/IPA internals directly. The on-boot protection here is for cases where developers are concerned that their app might be compromised and they wish to ensure that such a compromise could not theoretically persist itself via editing an installed patch file. Such a case is impractical, but we default to the strongest-possible security stance regardless.

Strict mode is currently default for Shorebird, however some of our large customers requested that we add an install_only mode, since their applications were so large (many hundreds of mb) that the hash-verification during boot was showing up on profiles from older devices.

Install flow:

1. Download patch from server
2. Inflate patch (apply bidiff to base release)
3. check_hash(): Compute SHA256 of inflated file, verify it matches server-provided hash
4. Store patch file, hash, and signature to disk

Boot flow:

1. Verify patch file exists and size matches stored metadata
2. hash_file(): Re-compute SHA256 of patch file on disk
3. check_signature(): Verify the computed hash has a valid signature using the public key
4. If verification fails, fall back to last known good patch or base release

Install Only Mode

patch_verification: install_only

In Install Only mode, patch signature verification happens at install time only. This provides faster boot times but does not protect against post-install tampering (extremely uncommon). The only case that this does not protect against is if your app itself were to accidentally (or through some other malicious exploit of your app) modify its own data directory and modify the patch files within such.

Install flow:

1. Download patch from server
2. Inflate patch (apply bidiff to base release)
3. check_hash(): Compute SHA256 of inflated file, verify it matches server-provided hash
4. check_signature(): Verify the server-provided hash has a valid signature using the public key
5. Store patch file, hash, and signature to disk

Boot flow:

1. Verify patch file exists and size matches stored metadata
2. (No signature verification - trusted from install time)

Without a Public Key

If no patch_public_key is configured, signature verification is skipped in both modes. The check_hash() step still runs during install to detect download corruption, but there is no cryptographic verification that the patch came from a trusted source.

TODO:

• Add an async API.
• Write tests for state management.
• Make state management/filesystem management atomic (and tested).

Later-stage update system design docs

• https://theupdateframework.io/
• https://fuchsia.dev/fuchsia-src/concepts/packages/software_update_system