DartFuzz
========

DartFuzz is a tool for generating random programs with the objective
of fuzz testing the Dart project. Each randomly generated program
can be run under various modes of execution, such as using JIT,
using AOT, using JavaScript after dart2js, and using various target
architectures (x86, arm, etc.). Any difference between the outputs
(**divergence**) may indicate a bug in one of the execution modes.

How to run DartFuzz
===================
To generate a single random Dart program, run

    dart dartfuzz.dart [--help] [--seed SEED] [--[no-]fp] FILENAME

where

    --help      : prints help and exits
    --seed      : defines random seed (system-set by default)
    --[no-]fp   : enables/disables floating-point operations (default: on)
    --[no-]ffi  : enables/disables FFI method calls (default: off)
    --[no-]flat : enables/disables flat types (default: off)
    --[no-]mini : enables minimization mode (default: off)
    --smask     : bitmask indicating which statements to omit (Bit=1 omits, defaults to "0")
    --emask     : bitmask indicating which expressions to omit (Bit=1 omits, defaults to "0")

The tool provides a runnable main isolate. A typical single
test run looks as:

    dart dartfuzz.dart fuzz.dart
    dart fuzz.dart

How to start DartFuzz testing
=============================
To start a fuzz testing session, run

    dart dartfuzz_test.dart [--help]
                            [--isolates ISOLATES ]
                            [--repeat REPEAT]
                            [--time TIME]
                            [--num-output-lines NUMOUTPUTLINES]
                            [--true_divergence]
                            [--show-stats]
                            [--dart-top DARTTOP]
                            [--mode1 MODE]
                            [--mode2 MODE]
                            [--[no-]rerun]

where

    --help             : prints help and exits
    --isolates         : number of isolates in the session (1 by default)
    --repeat           : number of tests to run (1000 by default)
    --time             : time limit in seconds (none by default)
    --num-output-lines : number of output lines to be printed in the case of a divergence (200 by default)
    --true-divergence  : only report true divergences (true by default)
    --show-stats       : show statistics during session (true by default)
    --dart-top         : sets DART_TOP explicitly through command line
    --mode1            : m1
    --mode2            : m2, and values one of
        jit-[debug-][x64|arm32|arm64]                    = Dart JIT
        aot-[debug-][x64|arm32|arm64]                    = Dart AOT
        djs-x64                                          = dart2js + Node.JS
    --[no-]rerun       : re-run a testcase if there is only a divergence in
                         the return codes outside the range [-255,+255];
                         if the second run produces no divergence the previous
                         one will be ignored (true by default)

If no modes are given, a random combination is used.

This fuzz testing tool must have access to the top of a Dart SDK
development tree (DART_TOP) in which all proper binaries have been
built already (for example, testing jit-x64 will invoke the binary
${DART_TOP}/out/ReleaseX64/dart to start the Dart VM). The DART_TOP
can be provided through the --dart-top option, as an environment
variable, or, by default, as the current directory by invoking the
fuzz testing tool from the Dart SDK top.

Background
==========

Although test suites are extremely useful to validate the correctness of a
system and to ensure that no regressions occur, any test suite is necessarily
finite in size and scope. Tests typically focus on validating particular
features by means of code sequences most programmers would expect. Regression
tests often use slightly less idiomatic code sequences, since they reflect
problems that were not anticipated originally, but occurred “in the field”.
Still, any test suite leaves the developer wondering whether undetected bugs
and flaws still linger in the system.

Over the years, fuzz testing has gained popularity as a testing technique for
discovering such lingering bugs, including bugs that can bring down a system
in an unexpected way. Fuzzing refers to feeding a large amount of random data
as input to a system in an attempt to find bugs or make it crash.
Generation-based fuzz testing constructs random, but properly formatted input
data. Mutation-based fuzz testing applies small random changes to existing
inputs in order to detect shortcomings in a system. Profile-guided or
coverage-guided fuzz testing adds a direction to the way these random changes
are applied. Multi-layered approaches generate random inputs that are
subsequently mutated at various stages of execution.

The randomness of fuzz testing implies that the size and scope of testing is
no longer bounded. Every new run can potentially discover bugs and crashes
that were hereto undetected.

Links
=====

* [Dart bugs found with fuzzing](https://github.com/dart-lang/sdk/issues?utf8=%E2%9C%93&q=label%3Adartfuzz+)
* [DartFuzz](https://github.com/dart-lang/sdk/tree/master/runtime/tools/dartfuzz)
* [DartLibFuzzer](https://github.com/dart-lang/sdk/tree/master/runtime/vm/libfuzzer)
* [Dust](https://pub.dev/packages/dust)
* [LibFuzzer](https://llvm.org/docs/LibFuzzer.html)
