shorebird-updater/docs/boot_state_machine.md

Boot State Machine

Overview

The Shorebird updater manages over-the-air code updates for Flutter applications. A critical part of this is the boot state machine, which tracks:

• Which patch should be loaded on app start
• Whether a patch booted successfully
• Automatic rollback if a patch crashes during boot

State Variables

Persisted to Disk (patches_state.json)

Variable	Type	Description
next_boot_patch	Option<PatchMetadata>	The patch to boot on next app start. Set when a patch is downloaded/installed.
last_booted_patch	Option<PatchMetadata>	The patch that last completed a successful boot cycle. Our "known good" state.
currently_booting_patch	Option<PatchMetadata>	Transient flag: set when boot starts, cleared on success/failure. If set on init, indicates crash.
known_bad_patches	HashSet<usize>	Patches that have failed to boot. Never attempt these again for this release.

In-Memory (Config)

Variable	Description
UpdateConfig	Global config set once via init(). Contains app ID, paths, network hooks, etc.

Boot Lifecycle

Happy Path

[Process Start]
       │
       ▼
    init()
       │
       ├─► Check currently_booting_patch
       │   └─► If set: previous boot crashed → mark patch as bad, fall back
       │
       ▼
    Engine gets next_boot_patch path
       │
       ▼
    TryLoadFromPatch() loads patch snapshot
       │
       └─► report_launch_start()  [called once via std::once_flag]
       │   └─► currently_booting_patch = next_boot_patch
       │
       ▼
    Shell::Shell() constructor completes
       │
       └─► report_launch_success()
       │   ├─► last_booted_patch = currently_booting_patch
       │   └─► currently_booting_patch = None
       │
       ▼
    [App Running - Dart code executing]

Crash Recovery

If the app crashes between report_launch_start() and report_launch_success():

1. Process dies with currently_booting_patch still set on disk
2. New process starts, calls init()
3. handle_prior_boot_failure_if_necessary() sees currently_booting_patch is set
4. Marks that patch as failed (adds to known_bad_patches)
5. Falls back to last_booted_patch or base release

Implementation Details

Where Boot Lifecycle Calls Are Made

report_launch_start() is called from TryLoadFromPatch() in runtime/shorebird/patch_cache.cc:

std::shared_ptr<const fml::Mapping> TryLoadFromPatch(...) {
  // ... validation and patch loading ...

  // Only report launch_start when we're actually about to use a patch,
  // and only once per process (for the first symbol, isolate data)
  static std::once_flag launch_start_flag;
  if (symbol == kIsolateDataSymbol) {
    std::call_once(launch_start_flag, []() {
      shorebird_report_launch_start();
    });
  }

  // Return the mapping
  // ...
}

report_launch_success() is called from Shell::Shell() constructor in shell/common/shorebird/shorebird.cc, after the Dart VM is created successfully.

Why This Placement Matters

The boot lifecycle calls are placed at specific points for good reason:

1. report_launch_start() in TryLoadFromPatch(): Called right before the patched Dart snapshot is actually loaded. This ensures:

   • FlutterEngineGroup warmup (which doesn't load patches) doesn't trigger false crash detection
   • The call only happens when we're actually about to use a patch
   • std::once_flag ensures exactly one boot cycle per process

2. report_launch_success() in Shell constructor: Called after the Dart VM is created, indicating the patch loaded successfully.

3. Crash on patch load failure: If TryLoadFromPatch() fails to load the patch, it calls FML_LOG(FATAL) which crashes the process. On the next launch, crash recovery naturally handles it.

Invariants

1. Config is set once per process: set_config() returns error if already set
2. Crash recovery runs once per process: Only on first successful init()
3. One boot cycle per process: std::once_flag ensures report_launch_start() is called at most once
4. State is persisted atomically: Each state change writes to disk immediately
5. Bad patches are permanent: Once in known_bad_patches, never tried again for this release

API Behavior

init()

• Sets global config (once per process, subsequent calls return AlreadyInitialized)
• Calls handle_prior_boot_failure_if_necessary() only on first init
• Does NOT run crash recovery if config already initialized

report_launch_start()

• If next_boot_patch exists, sets currently_booting_patch
• In production, called only once per process due to std::once_flag in C++

report_launch_success()

• Clears currently_booting_patch
• Sets last_booted_patch
• Subsequent calls are no-ops if currently_booting_patch is None

report_launch_failure()

• Marks currently_booting_patch as bad
• Falls back to previous good state
• Queues failure event for server

Historical Context: FlutterEngineGroup Bug

Prior to the current implementation, report_launch_start() was called from ConfigureShorebird() during FlutterMain::Init(). This caused a bug:

Problem: FlutterEngineGroup's constructor calls ensureInitializationComplete() which triggered report_launch_start(), but does NOT create a Shell (no report_launch_success()). If the app was killed before createAndRunEngine() was called, crash recovery incorrectly marked the patch as bad.

Solution: Move report_launch_start() to TryLoadFromPatch(), which is only called when actually loading a patch. FlutterEngineGroup never calls TryLoadFromPatch() (no Shell created), so no false positives occur.

Multiple Processes (Android)

If multiple processes access the same state file:

• Each process has its own Java initCalled flag
• Each process has its own Rust global config
• BUT they share the same on-disk state files
• No cross-process locking exists

This could cause issues if:

1. Process A sets currently_booting_patch
2. Process B also sets currently_booting_patch (it's the first engine in THAT process)
3. Process A completes, clears flag
4. Process B is killed before completing
5. On restart: crash recovery sees flag set (by Process B) and marks patch as bad

Testing

Unit tests in library/src/updater.rs (multi_engine_tests module) verify the boot state machine behavior:

• multi_engine_false_positive_rollback: Demonstrates the historical bug where multiple report_launch_start() calls caused false positive rollbacks
• interleaved_boot_calls_success_clears_flag: Verifies that report_launch_success() properly clears the booting flag

Note: In production, the C++ std::once_flag prevents multiple report_launch_start() calls, so the Rust-level tests demonstrate behavior that can only occur if the C++ guard is bypassed.