I don't like having a large volume of Dart code sitting under tools/
where it is hard to analyze, lint, test, and reuse. Also, eventually
we want to merge test.dart and test.py. This seems like an easy mostly
mechanical first step.
All I did was:
1. Move the contents of tools/test.dart to
pkg/test_runner/lib/test_runner.dart. (That's not a great file name
since we already have pkg/test_runner/bin/test_runner.dart, but it
was the best I could come up with.
2. Copy tools/bots/results to pkg/test_runner/bot_results.dart. I
don't like duplicating this, but there are other scripts under tools
that import the old location. Eventually, we should have those
scripts import it from package:test_runner/bot_results.dart, but I
didn't want to do that here since I'm not familiar with those other
scripts.
3. Make tools/test.dart import and forward to
pkg/test_runner/lib/test_runner.dart.
4. Fix any linter and type errors. The test_runner package has a bunch
of strictness checks and lints enable to keep it cleaner.
5. Run dartfmt --fix to format and get rid of "new", etc.
Change-Id: Ifc89817508d3fc147fa78dbc6744d547aeaf4c55
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/155240
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Jonas Termansen <sortie@google.com>
It was using the wrong 'expectation' field instead of the correct 'expected'
field. This change uses the right field. However as the matches field has
now been initialized, this change forcefully recomputes it instead of
politely using putIfAbsent. The matches result for the current result will
still take precedence.
Change-Id: I0775c1ff43b8f6691e457b5155ac9335338b209d
Reviewed-on: https://dart-review.googlesource.com/c/95182
Reviewed-by: William Hesse <whesse@google.com>
The expected field can be used to be displayed in the flakiness dashboard.
The matches fields can be used in the flakiness dashboards to decide whether
to include links to logs for a given outcome.
Change-Id: I19454e7cc8fb49871dbb0e7a11478d8888b56662
Reviewed-on: https://dart-review.googlesource.com/c/95024
Commit-Queue: Jonas Termansen <sortie@google.com>
Reviewed-by: William Hesse <whesse@google.com>
This change makes update_flakiness count how many times each outcome has
happened. If the same outcome happens more than a 100 times in a row, the
test is forgiven. If that was a mistake, and it happens again, the flakiness
frequency is less than 3.65% with 95% certainty, and the probability of
deflaking not catching it is astronomical.
The number of occurences of each kind of outcome mode is tracked. This lets
us track how probable each flakiness mode is. It also lets us answer
important questions such as how many flakes only happen a single time (not
fault of test) versus multiple times (something to do with the test).
Bug: https://github.com/dart-lang/sdk/issues/35318
Change-Id: Ieccd4afb96ada8053f59fbdc005fd853f0d1f996
Reviewed-on: https://dart-review.googlesource.com/c/85952
Reviewed-by: William Hesse <whesse@google.com>
tools/bots/compare_results.dart compares the previous and current test
results in the results.json format and lists the differences, taking the
flakiness data into account.
tools/bots/update_flakiness.dart reads new result.json files and updates the
flakiness data in the flaky.json format file. The updated flakiness data
contains the list of tests that were already known to be flaky, plus any
new tests with multiple different outcomes in the provided results.json
files.
For instance, after running the tests, to find the list of tests that
changed result and needs to be deflaked, excluding tests that are already
known to be flaky:
compare_results.dart \
--flakiness-data flaky.json
--changed \
--passing \
--failing \
previous.json results.json
After the tests needing deflakinghas been run again, the flakiness data can
be updated:
update_flakiness.dart -i flaky.json -o flaky.json results.json more.json
Finally a human readable report can explain what happened, exiting 1 if any
tests started failing in a non-flaky manner:
compare_results.dart \
--flakiness-data flaky.json \
--judgement \
--human \
--verbose \
--changed \
--failing \
--flaky \
previous.json results.json
Bug: https://github.com/dart-lang/sdk/issues/34517
Bug: https://github.com/dart-lang/sdk/issues/34518
Change-Id: I156a8a49b8df09c0aebcb77376b69d365d0aa2ac
Reviewed-on: https://dart-review.googlesource.com/75540
Reviewed-by: William Hesse <whesse@google.com>
William Hesse