test.dart locates where the current branch branched off master and compares
the local testing results with the appropriate mainline builder results,
letting you know how the current change compares without the need for status
files.
Bug: https://github.com/dart-lang/sdk/issues/35086
Change-Id: Ib79479b867c5ac131302fea1bdf7effd0422a83a
Reviewed-on: https://dart-review.googlesource.com/c/83281
Reviewed-by: Alexander Thomas <athom@google.com>
The approve_results program downloads the current results, the approved
results, and flakiness information for the specified bots. It presents
tables of the tests that have been fixed and the ones that have broken since
the last approval. Finally, it provides the option to approve the new
changes.
The approve_results program is part of the new workflow and only affects the
greenness of bots enrolled in it.
You can list the available bots by running:
dart tools/approve_results.dart -l
You can approve the results on a bot by running:
dart tools/approve_results.dart -b vm-kernel-linux-release-x64
Globs are supported, e.g. to approve all vm linux bots on ia32 and x64:
dart tools/approve_results.dart -b 'vm-*linux*{ia32,x64}*'
By default the tool is interactive and asks whether the changes should be
approved. Approving the results turns the bots green on the next commit.
It's assumed that bugs have been filed prior to approving the test changes.
The -y option approves the changes non-interactively while the -n option
does a try run and shows what needs to be approved.
This tool is meant to be used early and often to turn the bot green. Emails
are not set about additional failures until the regressions has been
approved.
Bug: https://github.com/dart-lang/sdk/issues/35000
Change-Id: I6479edaa76556aa5ef499509c04b92ae770d1727
Reviewed-on: https://dart-review.googlesource.com/c/82222
Reviewed-by: William Hesse <whesse@google.com>
tools/bots/compare_results.dart compares the previous and current test
results in the results.json format and lists the differences, taking the
flakiness data into account.
tools/bots/update_flakiness.dart reads new result.json files and updates the
flakiness data in the flaky.json format file. The updated flakiness data
contains the list of tests that were already known to be flaky, plus any
new tests with multiple different outcomes in the provided results.json
files.
For instance, after running the tests, to find the list of tests that
changed result and needs to be deflaked, excluding tests that are already
known to be flaky:
compare_results.dart \
--flakiness-data flaky.json
--changed \
--passing \
--failing \
previous.json results.json
After the tests needing deflakinghas been run again, the flakiness data can
be updated:
update_flakiness.dart -i flaky.json -o flaky.json results.json more.json
Finally a human readable report can explain what happened, exiting 1 if any
tests started failing in a non-flaky manner:
compare_results.dart \
--flakiness-data flaky.json \
--judgement \
--human \
--verbose \
--changed \
--failing \
--flaky \
previous.json results.json
Bug: https://github.com/dart-lang/sdk/issues/34517
Bug: https://github.com/dart-lang/sdk/issues/34518
Change-Id: I156a8a49b8df09c0aebcb77376b69d365d0aa2ac
Reviewed-on: https://dart-review.googlesource.com/75540
Reviewed-by: William Hesse <whesse@google.com>
Jonas Termansen