feat: add tests and docs for boot state machine (#309)
* test: test api calls * chore: add docs * chore: fix cspell * fix: use no-op network hooks in multi_engine tests Set no-op network hooks after init_for_testing so the fire-and-forget thread spawned by report_launch_success completes instantly without network I/O, preventing leaked threads from interfering with subsequent serial tests that use mock servers.
This commit is contained in:
Eric Seidel
@@ -0,0 +1,199 @@
|
||||
# Boot State Machine
|
||||
|
||||
## Overview
|
||||
|
||||
The Shorebird updater manages over-the-air code updates for Flutter
|
||||
applications. A critical part of this is the **boot state machine**, which
|
||||
tracks:
|
||||
- Which patch should be loaded on app start
|
||||
- Whether a patch booted successfully
|
||||
- Automatic rollback if a patch crashes during boot
|
||||
|
||||
## State Variables
|
||||
|
||||
### Persisted to Disk (`patches_state.json`)
|
||||
|
||||
| Variable | Type | Description |
|
||||
|----------|------|-------------|
|
||||
| `next_boot_patch` | `Option<PatchMetadata>` | The patch to boot on next app start. Set when a patch is downloaded/installed. |
|
||||
| `last_booted_patch` | `Option<PatchMetadata>` | The patch that last completed a successful boot cycle. Our "known good" state. |
|
||||
| `currently_booting_patch` | `Option<PatchMetadata>` | Transient flag: set when boot starts, cleared on success/failure. If set on init, indicates crash. |
|
||||
| `known_bad_patches` | `HashSet<usize>` | Patches that have failed to boot. Never attempt these again for this release. |
|
||||
|
||||
### In-Memory (Config)
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `UpdateConfig` | Global config set once via `init()`. Contains app ID, paths, network hooks, etc. |
|
||||
|
||||
## Boot Lifecycle
|
||||
|
||||
### Happy Path
|
||||
|
||||
```
|
||||
[Process Start]
|
||||
│
|
||||
▼
|
||||
init()
|
||||
│
|
||||
├─► Check currently_booting_patch
|
||||
│ └─► If set: previous boot crashed → mark patch as bad, fall back
|
||||
│
|
||||
▼
|
||||
Engine gets next_boot_patch path
|
||||
│
|
||||
▼
|
||||
TryLoadFromPatch() loads patch snapshot
|
||||
│
|
||||
└─► report_launch_start() [called once via std::once_flag]
|
||||
│ └─► currently_booting_patch = next_boot_patch
|
||||
│
|
||||
▼
|
||||
Shell::Shell() constructor completes
|
||||
│
|
||||
└─► report_launch_success()
|
||||
│ ├─► last_booted_patch = currently_booting_patch
|
||||
│ └─► currently_booting_patch = None
|
||||
│
|
||||
▼
|
||||
[App Running - Dart code executing]
|
||||
```
|
||||
|
||||
### Crash Recovery
|
||||
|
||||
If the app crashes between `report_launch_start()` and
|
||||
`report_launch_success()`:
|
||||
|
||||
1. Process dies with `currently_booting_patch` still set on disk
|
||||
2. New process starts, calls `init()`
|
||||
3. `handle_prior_boot_failure_if_necessary()` sees `currently_booting_patch` is
|
||||
set
|
||||
4. Marks that patch as failed (adds to `known_bad_patches`)
|
||||
5. Falls back to `last_booted_patch` or base release
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Where Boot Lifecycle Calls Are Made
|
||||
|
||||
**`report_launch_start()`** is called from `TryLoadFromPatch()` in
|
||||
`runtime/shorebird/patch_cache.cc`:
|
||||
|
||||
```cpp
|
||||
std::shared_ptr<const fml::Mapping> TryLoadFromPatch(...) {
|
||||
// ... validation and patch loading ...
|
||||
|
||||
// Only report launch_start when we're actually about to use a patch,
|
||||
// and only once per process (for the first symbol, isolate data)
|
||||
static std::once_flag launch_start_flag;
|
||||
if (symbol == kIsolateDataSymbol) {
|
||||
std::call_once(launch_start_flag, []() {
|
||||
shorebird_report_launch_start();
|
||||
});
|
||||
}
|
||||
|
||||
// Return the mapping
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
**`report_launch_success()`** is called from `Shell::Shell()` constructor in
|
||||
`shell/common/shorebird/shorebird.cc`, after the Dart VM is created
|
||||
successfully.
|
||||
|
||||
### Why This Placement Matters
|
||||
|
||||
The boot lifecycle calls are placed at specific points for good reason:
|
||||
|
||||
1. **`report_launch_start()` in `TryLoadFromPatch()`**: Called right before the
|
||||
patched Dart snapshot is actually loaded. This ensures:
|
||||
- FlutterEngineGroup warmup (which doesn't load patches) doesn't trigger
|
||||
false crash detection
|
||||
- The call only happens when we're actually about to use a patch
|
||||
- `std::once_flag` ensures exactly one boot cycle per process
|
||||
|
||||
2. **`report_launch_success()` in Shell constructor**: Called after the Dart VM
|
||||
is created, indicating the patch loaded successfully.
|
||||
|
||||
3. **Crash on patch load failure**: If `TryLoadFromPatch()` fails to load the
|
||||
patch, it calls `FML_LOG(FATAL)` which crashes the process. On the next
|
||||
launch, crash recovery naturally handles it.
|
||||
|
||||
### Invariants
|
||||
|
||||
1. **Config is set once per process**: `set_config()` returns error if already
|
||||
set
|
||||
2. **Crash recovery runs once per process**: Only on first successful `init()`
|
||||
3. **One boot cycle per process**: `std::once_flag` ensures
|
||||
`report_launch_start()` is called at most once
|
||||
4. **State is persisted atomically**: Each state change writes to disk
|
||||
immediately
|
||||
5. **Bad patches are permanent**: Once in `known_bad_patches`, never tried again
|
||||
for this release
|
||||
|
||||
## API Behavior
|
||||
|
||||
### `init()`
|
||||
- Sets global config (once per process, subsequent calls return
|
||||
`AlreadyInitialized`)
|
||||
- Calls `handle_prior_boot_failure_if_necessary()` only on first init
|
||||
- Does NOT run crash recovery if config already initialized
|
||||
|
||||
### `report_launch_start()`
|
||||
- If `next_boot_patch` exists, sets `currently_booting_patch`
|
||||
- In production, called only once per process due to `std::once_flag` in C++
|
||||
|
||||
### `report_launch_success()`
|
||||
- Clears `currently_booting_patch`
|
||||
- Sets `last_booted_patch`
|
||||
- Subsequent calls are no-ops if `currently_booting_patch` is None
|
||||
|
||||
### `report_launch_failure()`
|
||||
- Marks `currently_booting_patch` as bad
|
||||
- Falls back to previous good state
|
||||
- Queues failure event for server
|
||||
|
||||
## Historical Context: FlutterEngineGroup Bug
|
||||
|
||||
Prior to the current implementation, `report_launch_start()` was called from
|
||||
`ConfigureShorebird()` during `FlutterMain::Init()`. This caused a bug:
|
||||
|
||||
**Problem**: `FlutterEngineGroup`'s constructor calls
|
||||
`ensureInitializationComplete()` which triggered `report_launch_start()`, but
|
||||
does NOT create a Shell (no `report_launch_success()`). If the app was killed
|
||||
before `createAndRunEngine()` was called, crash recovery incorrectly marked the
|
||||
patch as bad.
|
||||
|
||||
**Solution**: Move `report_launch_start()` to `TryLoadFromPatch()`, which is
|
||||
only called when actually loading a patch. FlutterEngineGroup never calls
|
||||
`TryLoadFromPatch()` (no Shell created), so no false positives occur.
|
||||
|
||||
## Multiple Processes (Android)
|
||||
|
||||
If multiple processes access the same state file:
|
||||
- Each process has its own Java `initCalled` flag
|
||||
- Each process has its own Rust global config
|
||||
- BUT they share the same on-disk state files
|
||||
- No cross-process locking exists
|
||||
|
||||
This could cause issues if:
|
||||
1. Process A sets `currently_booting_patch`
|
||||
2. Process B also sets `currently_booting_patch` (it's the first engine in THAT
|
||||
process)
|
||||
3. Process A completes, clears flag
|
||||
4. Process B is killed before completing
|
||||
5. On restart: crash recovery sees flag set (by Process B) and marks patch as
|
||||
bad
|
||||
|
||||
## Testing
|
||||
|
||||
Unit tests in `library/src/updater.rs` (`multi_engine_tests` module) verify the
|
||||
boot state machine behavior:
|
||||
|
||||
- `multi_engine_false_positive_rollback`: Demonstrates the historical bug where
|
||||
multiple `report_launch_start()` calls caused false positive rollbacks
|
||||
- `interleaved_boot_calls_success_clears_flag`: Verifies that
|
||||
`report_launch_success()` properly clears the booting flag
|
||||
|
||||
Note: In production, the C++ `std::once_flag` prevents multiple
|
||||
`report_launch_start()` calls, so the Rust-level tests demonstrate behavior that
|
||||
can only occur if the C++ guard is bypassed.
|
||||
Reference in New Issue
Block a user