This script takes a list of test results and identifies blamelists
in the result feed data that include the commit of the test results
and tries to narrow the blamelist, if possible.
This CL also adds a small library to use the firestore REST API,
which contains mostly the functionality used in the script, but
should be easy enough to extend for other scripts.
Change-Id: If3c8272438e2a9bbf24891d9f5b62c342ea77cc6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/153966
Commit-Queue: Karl Klose <karlklose@google.com>
Reviewed-by: William Hesse <whesse@google.com>
I don't like having a large volume of Dart code sitting under tools/
where it is hard to analyze, lint, test, and reuse. Also, eventually
we want to merge test.dart and test.py. This seems like an easy mostly
mechanical first step.
All I did was:
1. Move the contents of tools/test.dart to
pkg/test_runner/lib/test_runner.dart. (That's not a great file name
since we already have pkg/test_runner/bin/test_runner.dart, but it
was the best I could come up with.
2. Copy tools/bots/results to pkg/test_runner/bot_results.dart. I
don't like duplicating this, but there are other scripts under tools
that import the old location. Eventually, we should have those
scripts import it from package:test_runner/bot_results.dart, but I
didn't want to do that here since I'm not familiar with those other
scripts.
3. Make tools/test.dart import and forward to
pkg/test_runner/lib/test_runner.dart.
4. Fix any linter and type errors. The test_runner package has a bunch
of strictness checks and lints enable to keep it cleaner.
5. Run dartfmt --fix to format and get rid of "new", etc.
Change-Id: Ifc89817508d3fc147fa78dbc6744d547aeaf4c55
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/155240
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Jonas Termansen <sortie@google.com>
This change adds a 'preapprovals' field to approved_results.json, which is
a map from gerrit Change-Id values to a pre-approval record, containing:
* 'from': What the previous approval was.
* 'result': What the new approval is.
* 'matches': Whether the new approval matches the expectation.
* 'expected': The expectation for the new approval.
* 'preapprover': Who did the preapproval.
* 'preapproved_at': When the preapproval occurred.
tools/approve_results.dart now produces pre-approval records instead of
updating the actual approvals. This ensures pre-approvals don't take
effect immediately and fixes the problem of pre-approving one kind of
failure becoming another kind of failure. The script needed to query the
gerrit API in order to determine the Change-Id of changelists. This query
also lets approve_results know what the latest changelist and lifts the
restriction that the patchset must be supplied when pre-approving.
The innards of approve_results has been refactored so it better can handle
selective modification of approve_results.dart.
approve_results now also checks for race conditions with any other
concurrent approval in interactive mode. This should catch most cases where
people leave approve_results running for long. There still is a small race
condition, so there is no check done in the non-interactive mode as the
results would just have been downloaded anyway.
The new tools/bots/apply_preapprovals.dart script will be used by the
recipe. It locates the outstanding pre-approved changelists and checks
the git history for whether they have landed by searching for their
Change-Id. It then applies their pre-approvals in order, warning about any
merge conflicts. Optionally it uploads the pre-approvals. This will work
for the commit queue where the pre-approvals can be applied temporarily,
and in the continuous integration where the pre-approvals become permanent
approvals.
This change allows empty approval records that only contain preapprovals but
with no actual base approval. tools/bots/compare_results.dart has been
updared to handle that case.
This changelist addresses https://github.com/dart-lang/sdk/issues/36279 by
ensuring only verified fields make it into new approved_results.json records
and adds a temporary workaround for 14 days to remove existing needless
existing fields.
Change-Id: I7b3ff59756dc0d28721c6db480782b4ea983b55d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/97308
Reviewed-by: William Hesse <whesse@google.com>
The output of a compare-results run showing logs will enclose
the failing test name in a box, as in:
/=======================================================================================\
| dart2js_extra/no_such_method_test is new and failed (CompileTimeError, expected Pass) |
\=======================================================================================/
--- Command "dart2js" (took 244ms):
DART_CONFIGURATION=ReleaseX64 out/ReleaseX64/dart-sdk/bin/dart2js --generate-code-with-compile-time-errors --test-mode --packages=/b/s/w/ir/cache/builder/sdk/.packages --out=/b/s/w/ir/cache/builder/sdk/out/ReleaseX64/generated_compilations/dart2js-sdk/tests_compiler_dart2js_extra_no_such_method_test/test.js /b/s/w/ir/cache/builder/sdk/tests/compiler/dart2js_extra/no_such_method_test.dart --preview-dart-2
exit code:
1
Change-Id: Ic63fa41381bab320636340c1e7919e22cd424c1d
Reviewed-on: https://dart-review.googlesource.com/c/86925
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Reviewed-by: Jonas Termansen <sortie@google.com>
tools/bots/compare_results.dart compares the previous and current test
results in the results.json format and lists the differences, taking the
flakiness data into account.
tools/bots/update_flakiness.dart reads new result.json files and updates the
flakiness data in the flaky.json format file. The updated flakiness data
contains the list of tests that were already known to be flaky, plus any
new tests with multiple different outcomes in the provided results.json
files.
For instance, after running the tests, to find the list of tests that
changed result and needs to be deflaked, excluding tests that are already
known to be flaky:
compare_results.dart \
--flakiness-data flaky.json
--changed \
--passing \
--failing \
previous.json results.json
After the tests needing deflakinghas been run again, the flakiness data can
be updated:
update_flakiness.dart -i flaky.json -o flaky.json results.json more.json
Finally a human readable report can explain what happened, exiting 1 if any
tests started failing in a non-flaky manner:
compare_results.dart \
--flakiness-data flaky.json \
--judgement \
--human \
--verbose \
--changed \
--failing \
--flaky \
previous.json results.json
Bug: https://github.com/dart-lang/sdk/issues/34517
Bug: https://github.com/dart-lang/sdk/issues/34518
Change-Id: I156a8a49b8df09c0aebcb77376b69d365d0aa2ac
Reviewed-on: https://dart-review.googlesource.com/75540
Reviewed-by: William Hesse <whesse@google.com>
Alexander Thomas