In for instance https://dart-review.googlesource.com/c/sdk/+/193961/5
we had an instance of something good happening (a crashing-ish test
that started to pass) but where the author was told no (red bot) and
not given a path to make the bot happy (bad messaging).
Specifically tests failed and the best log one got was
```
To re-run this test, run:
dart pkg/front_end/test/unit_test_suites.dart -p pkg/front_end/test/fasta/textual_outline/general/constants/js_semantics/issue45376
```
which isn't very helpful.
If running it locally a little more information is given via stdout,
but having to do that isn't great (and the little extra information
still wasn't great).
This CL updates the messaging to - hopefully - give a path to make the
bot happy. The messaging will now - directly from the log
on the bot - be
```
FAILED: pkg/front_end/test/fasta/textual_outline/general/constants/js_semantics/issue45376: Pass
To re-run this test, run:
dart pkg/front_end/test/unit_test_suites.dart -p pkg/front_end/test/fasta/textual_outline/general/constants/js_semantics/issue45376
The test passed, but wasn't expected to. You should update the status file for this test.
There's a status entry looking something like
general/constants/js_semantics/issue45376: FormatterCrash
which should be removed.
The status file is file:///[wherever]/sdk/pkg/front_end/testcases/textual_outline.status.
```
which
a) gives the extra information previously only available via stdout when
running locally (the first line).
b) Gives an explanation as to why it failed
("The test passed, but wasn't expected to.").
c) A path to make the bot green ("You should update the status file for
this test. [...] should be removed").
Hopefully this will make it easier to deal with in the future when
something good happens.
Change-Id: I838699abce580334c956d8b4753884fd600802ae
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/194242
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Johnni Winther <johnniwinther@google.com>
This changes the CFE expectation test suites to use unsound null safety
as default to avoid passing --enable-experiments=no-non-nullable to opt
out old test folders.
This change removes most test folders from the 'strong' tester, since
this now only supports sound null safety, which requires all libraries
to be opted in. Instead, the 'weak' tester now runs all test folders.
Additionally the 'outline' tester is changed to use unsound null safety
so that it call be used on all test folders.
The 'fast_strong' tester is removed since it should just be run locally
and it can be run by passing an option to the other testers.
References to non-existing 'shaker' folder has been removed from
testing.json.
Change-Id: Ibcc306f7b27d06b9637c205238bc408194f1d062
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/184787
Commit-Queue: Johnni Winther <johnniwinther@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
This CL shards the CFE try bot tests better.
* co19_2 runs was previously made faster, so using 10 shards on that is
way too much.
* sdk_test runs was previously made faster, so using 5 shard on that is
way too much.
* unit_tests was taking ~10 minutes and wasn't sharded. This CL shards
it. Note that this sharding is semi-complicated because a few tests
use git which is not usable on shard-runs. To overcome this a previous
CL renamed the tests that use git and here the sharded runs only run
the tests that doesn't require git. Then the "main bot" runs the tests
that require git.
* unit tests suites was taking ~12 minutes and wasn't sharded. This CL
shards it. Note that this sharding is semi-complicated because a few
of the suites use git which is not usable on shard-runs. Also a single
suite uses many more files that it is reasonable to include in the
fileset for the shards. In both cases these are run on the "main bot"
instead. The suit runnier was already "threaded", but simply launched
all threads (say around 20 or something like that) at the same time.
That's not ideal if having, say, 8 cores to work with (which is what
the bots seem to have) - or 12 (which is what my computer has).
Now only 'cores - 1' "threads" are run at any one time, and most
"sub-suites" are sharded, so that if one finishes early that core
can start another "sub-suite" - hopefully utilizing more of the
resources avaiable on the computer running it.
In total - in my 'benchmarks' (i.e. try bot runs) - the
front-end-linux-release-x64-try bot previously rook around 35 minutes
to complete, and with these changes it takes around 15 minutes to
complete. The biggest entry in the timeline is now building dart which
takes almost half that time. It does so with fewer shards than before,
but each might be utilized for longer.
Change-Id: Ie034058d8f33aafd21bc49f2bc878484563ba01c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/181383
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Johnni Winther <johnniwinther@google.com>
This CL is a warm-up to a follow-up change that will make running the
CFE trybot faster.
This CL:
* Rename tests that require git (because when working with filesets
and shards in the test system, the "checkout" is not a git checkout
and git commands thus doesn't work properly.
* Filters the front-end unit test setup on the front-end bot the way
it was probably intended, i.e. run tests in pkg/kernel, pkg/front_end
and pkg/fasta (actually, the last one doesn't exist, but still)
instead of just in folders inside the "suite" 'pkg' that somewhere in
the path has something called 'kernel', 'front_end' or 'fasta'.
* Split unit_test_suites.dart into a "forwarding shell" and a impl.
In a follow-up CL the impl will be converted to nnbd to allow for
using 'required' on named parameters, but if the entry point was
nnbd it would run in sound nnbd mode and nothing would compile
because all imports are not nnbd.
Overall this CL should change very little, mostly just run a few less
tests, i.e. for instance skip tests that live inside a folder called
'fasta' somewhere inside the analyzer directory path.
Change-Id: I3226c7261cff8b68cc287cff07dc1715dfd85159
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/181381
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
Jens Johansen