This change makes named configurations central in test.dart. Previously
test.dart would attempt to emulate the steps taken by a builder. Instead it
now runs a single named configuration as a single step and invokes test.py
directly without any other arguments from the test steps. This better
matches the behavior of test.py and what the developers expect.
Either a named configuration or a builder must be specified. If both are
provided, then the results for that builder is downloaded and used to
compare with when running the named configuration. Otherwise if the only the
named configuration is provided, then it downloads results from all the
builders using the named configuration. Finally if only the builder is
provided, then the named configuration defaults to the one tested by the
builder. If the builder has multiple named configuration, the user is
asked to clarify which one should be tested.
This change adds support for branches to the test matrix. test.dart needs to
know which builders are on which branch, so it can download the relevant
results, and not look on builders that won't be building the right commit.
Fixes https://github.com/dart-lang/sdk/issues/35873
Change-Id: Ie7b75445b954250493528299a0b45eca4e0bb2e5
Reviewed-on: https://dart-review.googlesource.com/c/92780
Reviewed-by: William Hesse <whesse@google.com>
This change assigns systematic names to the builders for branches, where
those builders end in their branch name as the suffix. This will be used to
identify to which branch the builders belong.
Change-Id: Ifa3be5dbbd45d4f623f70dd36d12426d360b0c30
Reviewed-on: https://dart-review.googlesource.com/c/92741
Reviewed-by: Alexander Thomas <athom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
The shards are currently defaulting to use j4 because the docker host
has 4 cores. 6 devices share one host and are therefore trying to run 20
processes competing for the only 4 physical cores. The docker hosts will
soon be upgraded to have 8 cores. j2 will result in 10 processes
competing for those 8 cores which should result in a better balance
between utilization and latency.
Change-Id: Ieb67cb92f17317108d1a1d3da710c58400484a7c
Reviewed-on: https://dart-review.googlesource.com/c/91540
Reviewed-by: William Hesse <whesse@google.com>
Context: we finally removed the old emitter and fast startup is on by default.
This CL removes from the test matrix duplicate steps that now are doing the same
thing and the faststartup builders that are no longer needed.
Change-Id: Id65ea3178bcd7a9554bfa127a09938ca910c57e8
Reviewed-on: https://dart-review.googlesource.com/c/89181
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Sigmund Cherem <sigmund@google.com>
The buildbucket API is not always reliable and responds within 30 seconds.
Rather than spuriously failing, try again. This should improve the
reliability of the base branch update bot.
Change-Id: Iade2d78372743b8d843c0dead591c28f8ee8f3ae
Reviewed-on: https://dart-review.googlesource.com/c/88445
Reviewed-by: William Hesse <whesse@google.com>
The output of a compare-results run showing logs will enclose
the failing test name in a box, as in:
/=======================================================================================\
| dart2js_extra/no_such_method_test is new and failed (CompileTimeError, expected Pass) |
\=======================================================================================/
--- Command "dart2js" (took 244ms):
DART_CONFIGURATION=ReleaseX64 out/ReleaseX64/dart-sdk/bin/dart2js --generate-code-with-compile-time-errors --test-mode --packages=/b/s/w/ir/cache/builder/sdk/.packages --out=/b/s/w/ir/cache/builder/sdk/out/ReleaseX64/generated_compilations/dart2js-sdk/tests_compiler_dart2js_extra_no_such_method_test/test.js /b/s/w/ir/cache/builder/sdk/tests/compiler/dart2js_extra/no_such_method_test.dart --preview-dart-2
exit code:
1
Change-Id: Ic63fa41381bab320636340c1e7919e22cd424c1d
Reviewed-on: https://dart-review.googlesource.com/c/86925
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Reviewed-by: Jonas Termansen <sortie@google.com>
Rationale:
With 100 machines, 8 isolates each, we get a theoretical
peak of 100,000 fuzzing tests per nightly run (actual
number depends on time-based execution mode run).
Change-Id: I5872f681a636e6b5369f684ceef0f37f2311c8d7
Reviewed-on: https://dart-review.googlesource.com/c/84664
Reviewed-by: Alexander Thomas <athom@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Commit-Queue: Aart Bik <ajcbik@google.com>
This change makes update_flakiness count how many times each outcome has
happened. If the same outcome happens more than a 100 times in a row, the
test is forgiven. If that was a mistake, and it happens again, the flakiness
frequency is less than 3.65% with 95% certainty, and the probability of
deflaking not catching it is astronomical.
The number of occurences of each kind of outcome mode is tracked. This lets
us track how probable each flakiness mode is. It also lets us answer
important questions such as how many flakes only happen a single time (not
fault of test) versus multiple times (something to do with the test).
Bug: https://github.com/dart-lang/sdk/issues/35318
Change-Id: Ieccd4afb96ada8053f59fbdc005fd853f0d1f996
Reviewed-on: https://dart-review.googlesource.com/c/85952
Reviewed-by: William Hesse <whesse@google.com>
This includes:
* Selecting front-end target.
* Support for multi-root virtual file system.
* Not linking platform into resulting kernel file.
* Specifying input as URI (instead of file path) on command line.
* Automatically converting input script URI to package URI.
* Writing ninja dependencies file.
* Writing package-split kernel binaries.
After this change Fuchsia's compiler.dart will become a small wrapper
over pkg/vm, sharing most logic and even most command line options
with pkg/vm gen_kernel tool.
Also, this CL attempts to share some pieces of code between frontend
server and gen_kernel.
In addition, seperate bytecode generation for package-split binaries
is implemented (needed for https://dart-review.googlesource.com/c/sdk/+/85469).
Corresponding Fuchsia CL: https://fuchsia-review.googlesource.com/c/topaz/+/229964
Change-Id: I12d7b2f6401357b3c9df2e31bc736af5a9dc5fd2
Reviewed-on: https://dart-review.googlesource.com/c/85721
Reviewed-by: Alexander Aprelev <aam@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Switch the Dart SDK builders to use the dart_sdk.py script to build
and upload the SDK.
Change-Id: I8d8fc41e8d90d14507d5337bd6dcc573ee09bb6a
Reviewed-on: https://dart-review.googlesource.com/c/85701
Reviewed-by: Jonas Termansen <sortie@google.com>
Jonas Termansen