Aart Bik
80a6f4b5dd
Rationale: Refactor DartFuzz so that long lists of values appear in separate file, making the actual fuzzing code easier to read. Also, this prepares generating some of the lists automaticaly (such a library methods). Note that *no* version bump is required, since this refactoring does not change behavior. Change-Id: Ic58eb7bc1aa63f48ec0d37e3c2399917786086bd Reviewed-on: https://dart-review.googlesource.com/c/87266 Reviewed-by: Ryan Macnak <rmacnak@google.com> Commit-Queue: Aart Bik <ajcbik@google.com>
DartFuzz
DartFuzz is a tool for generating random programs with the objective of fuzz testing the Dart project. Each randomly generated program can be run under various modes of execution, such as using JIT, using AOT, using JavaScript after dart2js, and using various target architectures (x86, arm, etc.). Any difference between the outputs (divergence) may indicate a bug in one of the execution modes.
How to run DartFuzz
To generate a single random Dart program, run
dart dartfuzz.dart [--help] [--seed SEED] FILENAME
where
--help : prints help and exits
--seed : defines random seed (system-set by default)
The tool provides a runnable main isolate. A typical single test run looks as:
dart dartfuzz.dart fuzz.dart
dart fuzz.dart
How to start DartFuzz testing
To start a fuzz testing session, run
dart dartfuzz_test.dart
run_dartfuzz_test.py [--help]
[--isolates ISOLATES ]
[--repeat REPEAT]
[--time TIME]
[--true_divergence]
[--mode1 MODE]
[--mode2 MODE]
where
--help : prints help and exits
--isolates : number of isolates in the session (1 by default)
--repeat : number of tests to run (1000 by default)
--time : time limit in seconds (none by default)
--show-stats : show statistics during session (true by default)
--true-divergence : only report true divergences (true by default)
--dart-top : sets DART_TOP explicitly through command line
--mode1 : m1
--mode2 : m2, and values one of
jit-[stress-][debug-]ia32 = Dart JIT (ia32)
jit-[stress-][debug-]x64 = Dart JIT (x64)
jit-[stress-][debug-]arm32 = Dart JIT (simarm)
jit-[stress-][debug-]arm64 = Dart JIT (simarm64)
jit-[stress-][debug-]dbc = Dart JIT (simdbc)
jit-[stress-][debug-]dbc64 = Dart JIT (simdbc64)
aot-[debug-]x64 = Dart AOT (x64)
aot-[debug-]arm32 = Dart AOT (simarm)
aot-[debug-]arm64 = Dart AOT (simarm64)
kbc-int-[debug-]x64 = Dart KBC (interpreted bytecode)
kbc-mix-[debug-]x64 = Dart KBC (mixed-mode bytecode)
kbc-cmp-[debug-]x64 = Dart KBC (compiled bytecode)
js-x64 = dart2js + Node.JS
If no modes are given, a random JIT and/or AOT combination is used.
This fuzz testing tool must have access to the top of a Dart SDK development tree (DART_TOP) in which all proper binaries have been built already (for example, testing jit-ia32 will invoke the binary ${DART_TOP}/out/ReleaseIA32/dart to start the Dart VM). The DART_TOP can be provided through the --dart-top option, as an environment variable, or, by default, as the current directory by invoking the fuzz testing tool from the Dart SDK top.
Background
Although test suites are extremely useful to validate the correctness of a system and to ensure that no regressions occur, any test suite is necessarily finite in size and scope. Tests typically focus on validating particular features by means of code sequences most programmers would expect. Regression tests often use slightly less idiomatic code sequences, since they reflect problems that were not anticipated originally, but occurred “in the field”. Still, any test suite leaves the developer wondering whether undetected bugs and flaws still linger in the system.
Over the years, fuzz testing has gained popularity as a testing technique for discovering such lingering bugs, including bugs that can bring down a system in an unexpected way. Fuzzing refers to feeding a large amount of random data as input to a system in an attempt to find bugs or make it crash. Generation- based fuzz testing constructs random, but properly formatted input data. Mutation-based fuzz testing applies small random changes to existing inputs in order to detect shortcomings in a system. Profile-guided or coverage-guided fuzz testing adds a direction to the way these random changes are applied. Multi-layered approaches generate random inputs that are subsequently mutated at various stages of execution.
The randomness of fuzz testing implies that the size and scope of testing is no longer bounded. Every new run can potentially discover bugs and crashes that were hereto undetected.