This change takes advantage of the new --clean-exit and --silent-failures
options to test.py, which means the test.py native progress bar can be used.
test.py exiting 0 if results were written out, no matter the results, lets
us properly handle if the program failed.
The output is now sectioned to make it more readable, and if no tests
failed, then the program says so.
This change also fixes logs which were broken because the logs for the steps
weren't merged, and the main test.py run wasn't even being passed the
--write-logs option.
This change also fixes directly starting dart scripts as an executable
rather than using the platform resolved executable.
TBR=whesse@google.com
Bug: https://github.com/dart-lang/sdk/issues/35359
Change-Id: I7bed009475df3ffea01c9e805513d0e04e77427c
Reviewed-on: https://dart-review.googlesource.com/c/86568
Reviewed-by: Jonas Termansen <sortie@google.com>
This change renames the --no-status option to --clean-exit, as that better
describes what it does, which is useful. We'll want to add a proper option
--no-status-files in the future that ignores status file entries except skip
and slow, which will be useful for the reproduction commands provided by
test.dart in the transition period until the status files are deleted.
Bug: https://github.com/dart-lang/sdk/issues/35358
Change-Id: Idc22a6c1b109508dadafb8a225cb91ad06287c0b
Reviewed-on: https://dart-review.googlesource.com/c/86565
Reviewed-by: William Hesse <whesse@google.com>
Currently reproduction commands often look like this:
tools/test.py -m release -n dartk-linux-release-x64 language_2/...
This has duplicate information in it, namely "--mode=release".
This CL
a) validates that the implied options are not passed in addition to -n
b) the reproduction commands will no longer print the redundant information.
Change-Id: I1a70ad021d5dcfb7a514cd9bbefbb8d0f77edb0a
Reviewed-on: https://dart-review.googlesource.com/c/85944
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
This change makes update_flakiness count how many times each outcome has
happened. If the same outcome happens more than a 100 times in a row, the
test is forgiven. If that was a mistake, and it happens again, the flakiness
frequency is less than 3.65% with 95% certainty, and the probability of
deflaking not catching it is astronomical.
The number of occurences of each kind of outcome mode is tracked. This lets
us track how probable each flakiness mode is. It also lets us answer
important questions such as how many flakes only happen a single time (not
fault of test) versus multiple times (something to do with the test).
Bug: https://github.com/dart-lang/sdk/issues/35318
Change-Id: Ieccd4afb96ada8053f59fbdc005fd853f0d1f996
Reviewed-on: https://dart-review.googlesource.com/c/85952
Reviewed-by: William Hesse <whesse@google.com>
This change ads two new fields to approved_results.json:
* approver: The username of the person who approved the new results.
* approved_at: The ISO 8601 UTC time of when the approval occured.
This data provides a history for approvals as a replacement for the git
commit history for status files. Each build makes a copy of the
approved_results.json file at the time, effectively acting as a commit.
Bug: https://github.com/dart-lang/sdk/issues/35315
Change-Id: I64430a53820a0cffeb277012d5318b50999a9df5
Reviewed-on: https://dart-review.googlesource.com/c/85940
Reviewed-by: William Hesse <whesse@google.com>
This includes:
* Selecting front-end target.
* Support for multi-root virtual file system.
* Not linking platform into resulting kernel file.
* Specifying input as URI (instead of file path) on command line.
* Automatically converting input script URI to package URI.
* Writing ninja dependencies file.
* Writing package-split kernel binaries.
After this change Fuchsia's compiler.dart will become a small wrapper
over pkg/vm, sharing most logic and even most command line options
with pkg/vm gen_kernel tool.
Also, this CL attempts to share some pieces of code between frontend
server and gen_kernel.
In addition, seperate bytecode generation for package-split binaries
is implemented (needed for https://dart-review.googlesource.com/c/sdk/+/85469).
Corresponding Fuchsia CL: https://fuchsia-review.googlesource.com/c/topaz/+/229964
Change-Id: I12d7b2f6401357b3c9df2e31bc736af5a9dc5fd2
Reviewed-on: https://dart-review.googlesource.com/c/85721
Reviewed-by: Alexander Aprelev <aam@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Makes the --keep-generated-files flag an option for the test.py command
only, not part of a named configuration passed in with the -n flag
and defined in tools/bots/test_matrix.json.
Change-Id: If67c2848cf6fc15e4cb7167ce23eeb3eb9bd1684
Reviewed-on: https://dart-review.googlesource.com/c/85705
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: William Hesse <whesse@google.com>
Switch the Dart SDK builders to use the dart_sdk.py script to build
and upload the SDK.
Change-Id: I8d8fc41e8d90d14507d5337bd6dcc573ee09bb6a
Reviewed-on: https://dart-review.googlesource.com/c/85701
Reviewed-by: Jonas Termansen <sortie@google.com>
This change lets committers preapprove new failing tests based on commit
queue / dry run results for CQ builders, such that the CQ builder stays
green when the change is submitted.
The current implementation does not support changing one test from a failing
mode to another failing mode. It will work, but if the corresponding CI
builder cycles before the changelist is submitted, it will become red until
the changelist is submitted. This will be addressed in the future.
Likewise approving a builder after a preapproval will undo the preapproval
unless the changelist has been submitted and the corresponding CI builder
has cycled.
Bug: https://github.com/dart-lang/sdk/issues/35289
Change-Id: Ie43df4439a27a3472c5178cb7c6af69292cf70e4
Reviewed-on: https://dart-review.googlesource.com/c/85601
Reviewed-by: William Hesse <whesse@google.com>
We do not archive core dumps generated by tests that are expected
to crash - so there is no need to generate those dumps.
It seems that generating core dumps is very slow on Mac and causes
timeouts.
Fixes#35275
Remove Timeout expectations from Crashing tests
Change-Id: I36c89fd583a6a94521560544163f02414cbbf41a
Reviewed-on: https://dart-review.googlesource.com/c/85680
Commit-Queue: Vyacheslav Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
This should fix some trybot flakiness we've seen today.
Original commit message:
[infra] Update Clang toolchain and GN
We no longer rely on Fuchsia's buildtools, instead we directly fetch
Clang and GN packages from CIPD using gclient support.
Change-Id: Ib0412072a80590afe407e9b4c2906d0f3a4a7d85
Reviewed-on: https://dart-review.googlesource.com/c/85642
Commit-Queue: Paul Berry <paulberry@google.com>
Commit-Queue: Konstantin Shcheglov <scheglov@google.com>
Auto-Submit: Paul Berry <paulberry@google.com>
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
We no longer rely on Fuchsia's buildtools, instead we directly fetch
Clang and GN packages from CIPD using gclient support.
Change-Id: I8995d89babf9c3c6b30214a1133e7b0a1cb9a796
Reviewed-on: https://dart-review.googlesource.com/c/85416
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Zach Anderson <zra@google.com>
Removes the flaky log, result.log result format, test outcome log,
and --append-logs option from test.py. They are replaced by the
new --write-results and --write-logs options.
The debug log is now always appended to, if it is written to a file.
Change-Id: I20dbbc1db86ef23a19ac4ea02067f18ae4906b3d
Reviewed-on: https://dart-review.googlesource.com/c/85440
Reviewed-by: Jonas Termansen <sortie@google.com>
This reverts commit ad69aaef62.
Reason for revert: <INSERT REASONING HERE>
Original change's description:
> Reland: [infra] Roll buildtools to c72a1c5aadde06505b7bb4641720880f3db28ff9
>
> Tries to work around update.py race by setting up symlinks for 'git cl format'
> instead of copying.
>
> Change-Id: I4ed9b99254803feea5ea0e8d30db23fc982e9562
> Reviewed-on: https://dart-review.googlesource.com/c/85427
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
> Commit-Queue: Zach Anderson <zra@google.com>
TBR=rmacnak@google.com,zra@google.com
Change-Id: Ida584adda37cd3e2dac16654eb1e81e92f123f88
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/85467
Reviewed-by: Zach Anderson <zra@google.com>
Commit-Queue: Zach Anderson <zra@google.com>
Jonas Termansen