The first test being compiled on a batch compiler pays for the cost of
the batch compiler startup, meaning it is more likely to time out, e.g.
see https://github.com/dart-lang/sdk/issues/53672.
E.g via
`tools/test.py '--progress=status' --report --time --write-results --write-logs --clean-exit -ncfe-strong-linux service/next_...`
I get output like this:
```
0:00:37.713771 - fasta - fasta-none release_x64/service/next_through_simple_linear_test/service
0:00:37.682524 - fasta - fasta-none release_x64/service/next_through_for_loop_with_break_and_continue_test/dds
0:00:36.805734 - fasta - fasta-none release_x64/service/next_through_create_list_and_map_test/service
0:00:36.589329 - fasta - fasta-none release_x64/service/next_through_simple_linear_2_test/dds
0:00:36.586428 - fasta - fasta-none release_x64/service/next_through_simple_linear_2_test/service
0:00:36.518832 - fasta - fasta-none release_x64/service/next_through_for_each_loop_test/dds
0:00:36.407107 - fasta - fasta-none release_x64/service/next_through_create_list_and_map_test/dds
0:00:36.259945 - fasta - fasta-none release_x64/service/next_through_for_each_loop_test/service
0:00:36.211573 - fasta - fasta-none release_x64/service/next_through_call_on_field_in_class_test/service
0:00:36.189456 - fasta - fasta-none release_x64/service/next_through_call_on_field_in_class_test/dds
0:00:35.974007 - fasta - fasta-none release_x64/service/next_through_for_loop_with_break_and_continue_test/service
0:00:35.921138 - fasta - fasta-none release_x64/service/next_through_simple_linear_test/dds
0:00:09.690812 - fasta - fasta-none release_x64/service/next_through_multi_catch_test/service
0:00:09.562068 - fasta - fasta-none release_x64/service/async_next_test/service
0:00:09.530882 - fasta - fasta-none release_x64/service/next_through_operator_bracket_test/dds
0:00:09.407169 - fasta - fasta-none release_x64/service/next_through_function_expression_test/dds
0:00:09.366673 - fasta - fasta-none release_x64/service/next_through_multi_catch_test/dds
0:00:09.327895 - fasta - fasta-none release_x64/service/next_through_function_expression_test/service
0:00:09.303926 - fasta - fasta-none release_x64/service/next_through_await_for_test/service
0:00:09.236295 - fasta - fasta-none release_x64/service/next_through_simple_async_with_returns_test/dds
```
where the first 12 (the really slow ones) each is the first test being
run on that particular batch compiler, where they both pay for the
startup cost of the batch compiler (~12 seconds on my machine when
launching 12 simultaneously which is the default on my machine -- it's
probably more on the bots) and for the jit not having warmed up yet:
E.g. the first test compiles in ~22 seconds, the second in ~9 seconds,
the third in ~6 seconds on my machine (again with 12 simultaneous runs).
To make it less likely that the first compile times out this CL adds
some extra time allowance for the first run after the bulk compiler is
started. For now it's 30 seconds, but it can be adjusted if needed.
Longer term we should probably start a batch compilation test by
aot-compilng the batch compiler (which, at least currently,
doesn't support asserts
(https://github.com/dart-lang/sdk/issues/53343)
which is unfortunate).
I'll leave that for a future CL.
Bug: https://github.com/dart-lang/sdk/issues/53672
Change-Id: Id3427223c46c1e7a34b401097a3f7d5f1321ad93
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/332220
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: William Hesse <whesse@google.com>
This reverts commit f827eb3a78.
Reason for revert: The tests being run, and the results being compared to, are changing dramatically, causing all sorts of test results going from [status] -> Skipped and from New Test -> [status].
Something is really going wrong.
Original change's description:
> Balance tests equally across shards.
>
> The sharded test runner invocations are now passed the previous results
> which contains the test timing, which are used to simulate how long each
> shard would take to run. The shards are now balanced as evenly as
> possible on a test level, taking multiple cores into account.
>
> Sharded tests are now run starting with the slowest test first, such
> that extremely long running tests finish as early as possible. This
> behavior ensures the cores are saturated and can be padded with fast
> tests near the end, rather than waiting for a few slow tests to complete
> while the rest of the system is idle.
>
> The algorithm works very well whenever it's able to accurately predict
> the time to run shards. In a number of cases, the model doesn't quite
> reflect reality and the data, which makes it fairly imperfect but still
> reasonably good. I think a second order feedback loop might kick in once
> it reorders the tests across shards and the test timing data reflects
> the new test timings.
>
> Multitests are no longer always sent to the same shard, since the data
> isn't available at the moment, and the change as-is speeds up the test
> running considerably.
>
> The front end unit test suites currently ignore the feature as there are
> no benefits yet to improving those quick shards.
>
> Upgrade the language version to 3.0.0 so patterns can be used and fix
> a mixin not being a mixin.
>
> Fixes: b/291585137
> Change-Id: I3cc1b1d96038d5b46e836b091e299097717c226c
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/314081
> Reviewed-by: William Hesse <whesse@google.com>
> Commit-Queue: Jonas Termansen <sortie@google.com>
Change-Id: I233e4bfa6d6ecf0cea4f97c1e47f1635f7b9040c
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/315060
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: William Hesse <whesse@google.com>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
The sharded test runner invocations are now passed the previous results
which contains the test timing, which are used to simulate how long each
shard would take to run. The shards are now balanced as evenly as
possible on a test level, taking multiple cores into account.
Sharded tests are now run starting with the slowest test first, such
that extremely long running tests finish as early as possible. This
behavior ensures the cores are saturated and can be padded with fast
tests near the end, rather than waiting for a few slow tests to complete
while the rest of the system is idle.
The algorithm works very well whenever it's able to accurately predict
the time to run shards. In a number of cases, the model doesn't quite
reflect reality and the data, which makes it fairly imperfect but still
reasonably good. I think a second order feedback loop might kick in once
it reorders the tests across shards and the test timing data reflects
the new test timings.
Multitests are no longer always sent to the same shard, since the data
isn't available at the moment, and the change as-is speeds up the test
running considerably.
The front end unit test suites currently ignore the feature as there are
no benefits yet to improving those quick shards.
Upgrade the language version to 3.0.0 so patterns can be used and fix
a mixin not being a mixin.
Fixes: b/291585137
Change-Id: I3cc1b1d96038d5b46e836b091e299097717c226c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/314081
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Jonas Termansen <sortie@google.com>
The whole directory is already copied to the device. This makes
it consistent with tests running on the host and allows us to
access various artefacts (e.g. SO files containing DWARF
information).
Change-Id: I9fe20fdbb48ad5a3742286a84c87f87f580f2c09
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/310600
Reviewed-by: Tess Strickland <sstrickl@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
Any adb failures when test_runner runs tests on android devices,
except an failure of the actual precompiled test when it is run,
should be reported as an infra failure and should shut down the
test runner.
After change https://dart-review.googlesource.com/c/sdk/+/291304
these types of failure were reported as test failures instead.
Fixes: b/281492587
Change-Id: Iea47ccffac81b0b92f3a3fb3780d2181d08fc1d1
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/310260
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: William Hesse <whesse@google.com>
* The new compiler name "ddc" will be used for all configurations
in an upcoming change.
* Aliases "dartdevc" and "dartdevk" will be removed in the future.
Change-Id: I2fa48e0ec043b1ba7fb70f883b39e40f338b6928
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/280862
Commit-Queue: Nicholas Shahan <nshahan@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
For some reason, we launched Safari from a file, which since macOS 10.14
requires the user to confirm the operation in a modal dialog which
caused timeouts.
* Removed unused http test driver port option.
* Added some basic tests for test.py's browser interactions.
Bug: b/208186791
Change-Id: I070529148d37bf312f219c576abca3da972f73e6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/204202
Commit-Queue: Alexander Thomas <athom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
* Remove all uses of `--dart2js-batch`.
* Enable batch mode for dart2js by default.
* Rename `--noBatch` to `--no-batch`.
* Make `--no-batch` affect dart2js configurations.
Change-Id: I33bdb5cfe8a82ccfc6ce9e43843f1b7a34ff1fbf
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/217006
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
There was a race between the subprocess listening on the socket and
the test trying to connect.
Also the Android bits of the test harness needed to be taught about
the abstract_socket_test program, similar to how process_test is
handled.
TEST=Fixes tests
Change-Id: I5e1c5d2e23e4bff2a01cd5c491a45e1c58d2578e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/201861
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Zach Anderson <zra@google.com>
I'm not sure if the output truncation was ever practically useful. But
it causes problems now that we have a CFE test that wants to validate
that the CFE can produce many errors in one go.
I thought about allowing a test to extend the truncation size, but I
figured the simplest fix was to just not truncate at all and make the
test runner a little simpler. Also did a little clean up on this
ancient code while I was at it.
Fix#44493.
Change-Id: Ie678c80e7114e0fb7f68931335075e3d319fb53a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/178940
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
Emphasize that the operation is going away,
and mark constructor as deprecated.
TEST= Refactoring+deprecation only, covered by existing tests.
Change-Id: I82aa044cd2cf7bf347b624371399f44bda8f4a07
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/173261
Commit-Queue: Lasse R.H. Nielsen <lrn@google.com>
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Reviewed-by: Nate Bosch <nbosch@google.com>
- Make it synchronous. I did some benchmarking and the async didn't seem
to make a measurable difference and made the code harder to follow.
- Fix a bunch of small-scale idiom things: "var", names, etc.
- Move TestCase.hash into TestFile since that lets us access it before
we've created a TestCase.
- Remove dead code.
- Make it clearer which functions work with TestFiles and which with
TestCases.
- Clarify the code that determines whether or not to enqueue a test.
The last point is the motivating one. Soon, I'll be extending this code
to take NNBD into account when determining which tests to skip, so I
wanted to clean it up some first.
Change-Id: I488ed6c7d2453535968ac43389a2fd8370ead57d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/116662
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Mayank Patke <fishythefish@google.com>
- Remove the pointless static factory methods. Most of them only had a
single callsite and in either case, they accomplished nothing useful
since the underlying classes are already public.
- Make index final.
- Make indexedCopy() and remove the unused (and wrong!) base
implementation.
- Define an abstract createOutput() method on Command and have all
Command classes implement that to return an output for themselves.
This is a better object-oriented design than a monolithic
createCommandOutput() function full of "is ___" checks.
- Remove CleanDirectoryCopyCommand and MakeSymlinkCommand. They were
unused.
- Change VmBatchCommand to VMBatchCommand to follow the style guide.
- Capitalize "L" in JSCommandLineCommand.
There are no behavioral changes. This stuff was just bugging me.
Change-Id: I2493f6afe0ee129e9c24e3f8289d37fe6dd4f802
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/114465
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Previously we would not have any VM logs because they are were sent to the Android
system logs, which are not captured by the bots.
Also added a flag --android-log-to-stderr, which is useful for local testing
and debugging.
Change-Id: I1968b4c230d70b5546d2dd48e40fe3cfe5196def
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/112250
Commit-Queue: Samir Jindel <sjindel@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
I pulled this list from the build package and turned on the ones that
only had a handful of violations. Left the others that I eventually want
to enable commented out.
Change-Id: I09ec037982e29d3682bc0eaa6af28adfad91b941
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/108563
Reviewed-by: Nate Bosch <nbosch@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
If a test has the static error markers and is run on analyzer, the
test runner verifies that the reported errors exactly match the expected
ones. A test with error markers is skipped on all other configurations.
Change-Id: Ib921acde517688f5b47937a5b100c0507e0e9138
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/107823
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
Move generation of Function objects for native trampolines to the Precompiler, so they can be generated during AOT and tree-shaken if possible.
Issue dartbug.com/35765
Change-Id: I0e69b7e0b22db73e3a40f2fe445660e57ddb6fa9
Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-bare-linux-release-simarm64-try, vm-kernel-precomp-bare-linux-release-x64-try, vm-kernel-precomp-linux-debug-x64-try, vm-dartkb-linux-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/107407
Commit-Queue: Samir Jindel <sjindel@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
There's a bunch of information that gets pulled from the contents of a
test file itself: whether it's a multitest, VMOptions, etc. Previously,
that lived in some combination of TestInformation objects and a big
stringly-typed "optionsFromFile" map that got passed everywhere. Also,
the map got mutated in a couple of choice places.
This replaces all of that with a single TestFile class that represents
all of the metadata gleaned from a given test file. This cleans up the
code base and also should pave the way for supporting static error
tests using that same class.
There should be no behavioral changes in this patch.
I didn't remove the packageRoot stuff in this change since there's
another change in flight for that. I'll merge those two together once
one of them lands.
Change-Id: Ia6d4c926afb342b71cb0041db7219586a792ac80
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/106581
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
- Run dartfmt --fix. This converts JavaDoc comments to "///", removes
"new" and extraneous "const", and a couple of other things.
- Fix SCREAMING_CAPS constants to lowerCamelCase.
- Use collection literals where possible.
- Use UI-as-code in a couple of places where it seemed obvious.
- Use "var" for more local variables.
- Use "const" instead of "final" when possible.
- Make members private when possible. Deleted a few that then became
obviously unused.
- ".length > 0" -> ".isNotEmpty".
There are no meaningful changes.
Change-Id: Ic6c5a74b2af9b3ebcbe881dbed69f65488bdef09
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/105880
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
Jens Johansen