This captures failures of broken android runners, and does not report
them as new failing results on all tests.
Change-Id: I83c4613d2ce7939c0b2b4cd2fb85f8191f01184b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/138923
Reviewed-by: Alexander Thomas <athom@google.com>
There's a bundle of improvements here:
- Show the stack trace of the failure when a test using async_minitest
fails. The trade-off is that it now only reports the first failure.
Collecting all of them didn't play nice with the stack trace
deobfuscation the test runner does.
- Report the group and name of the test that failed when known.
- Fix as many double-reporting test result bugs as I could find. Also,
in case that still happens, don't spew out a pile of useless JSON.
- Correctly wait for a test to complete even if it schedules its own
asynchronous operations without returning futures.
- Generally simplify and clean some stuff up.
Change-Id: Ie020ae0b80a11764c455cf0ce24dfea09ca0adce
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/138903
Reviewed-by: Srujan Gaddam <srujzs@google.com>
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Running tools/test.py locally behaves differently than on the bots
because the bots will have special environment variables set.
=> We want also local runs of tools/test.py to report ASAN errors, so
this CL moves the setting of environment variables to tools/test.py
Furthermore we enable "additional check_initialization_order=true" asan
option to detect ordering issues when initializing global state.
Make the IsolateGroup::isolate_group_random_ heap allocated on VM
startup to avoid initialization ordering issue (turns out our `Random`
can depend on embedder entropy source callback being available)
Also add locking around assigning of isolate group ids -- without it we
could have two threads racing and possibly getting the same id.
See b/149978682
Change-Id: I41b9a34c66934dcc0d3804233f5acf09004ba3bd
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/136630
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
Original change is in Patchset 1. I erroneously assumed DART_PRECOMPILER
was only used AOT compilation (it is also used for JIT snapshots).
Change-Id: I77f6c463db7f2b482c9b098f353dc28f0676ea0d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/135900
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Samir Jindel <sjindel@google.com>
When the test runner runs a legacy test with the non-nullable experiment
enabled, this passes an extra command line flag to tell analyzer to
treat the test library as opted out of NNBD.
This CL also removes the guard that made the test runner skip all legacy
tests when running analyzer with the non-nullable experiment enabled.
We want to run those tests now and (when analyzer supports the new
command line flag) they should be correctly treated as opted out.
This change also removes the tests package config file since that no
longer needs to be used to set the language version. It also removes the
code to generate a package config for the multitests. This CL replaces
both of those mechanisms.
Change-Id: Iedc1e6111bc0623ac21263499f3f5d11bd6fdf7c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/135346
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Closes: https://github.com/dart-lang/sdk/issues/40233
This CL creates nnbd versions of the tests and runs them on the nnbd sdk.
This CL does not (1) migrate sdk_nnbd/lib/ffi fully yet, and does not (2) fix all the tests/ffi (which is NNBD tests) yet.
Uncovered new issues:
Issue: https://github.com/dart-lang/sdk/issues/40234 nullptr should have type Pointer<Never>.
Issue: https://github.com/dart-lang/sdk/issues/40247 Structs need external fields.
Issue: https://github.com/dart-lang/sdk/issues/40271 Callbacks hit assert in debug.
Change-Id: Icb1b83577e03ed283165eb17703fc8dfc7fa5960
Cq-Include-Trybots: luci.dart.try:vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64-try,app-kernel-linux-debug-x64-try,vm-kernel-linux-debug-ia32-try,vm-kernel-win-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-kernel-precomp-linux-debug-x64-try,vm-dartkb-linux-release-x64-abi-try,vm-kernel-precomp-android-release-arm64-try,vm-kernel-asan-linux-release-x64-try,vm-kernel-linux-release-simarm-try,vm-kernel-linux-release-simarm64-try,vm-kernel-precomp-android-release-arm_x64-try,vm-kernel-precomp-obfuscate-linux-release-x64-try,dart-sdk-linux-try,analyzer-analysis-server-linux-try,analyzer-linux-release-try,front-end-linux-release-x64-try,vm-kernel-precomp-win-release-x64-try,analyzer-nnbd-linux-release-try,dart2js-nnbd-linux-x64-chrome-try,ddc-nnbd-linux-release-chrome-try,front-end-nnbd-linux-release-x64-try,vm-kernel-nnbd-linux-debug-x64-try,vm-kernel-nnbd-linux-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/132604
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
This works on all browsers, in d8, and supports both dartdevk and
dart2js. In the case of dart2js, this also expands minified names and
inlined data.
The result is presented to users as a separate section, so the original
JS stack trace is printed as is, and a second deobfuscated stack trace
is printed afterwards.
Change-Id: I10af5e0241a8c87ad8db48234ce52cfb51a27411
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/132966
Commit-Queue: Sigmund Cherem <sigmund@google.com>
Reviewed-by: Vijay Menon <vsm@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Sometimes, the batch runners need the experiment flags set. This is
especially true when running in NNBD mode which needs to be enabled in
the VM running the batch runner as well as in the test code.
Previously this was only done for the CFE batch runner, this CL moves
the logic to apply to all compilation commands using batch runners.
Change-Id: I9358ae7cfed4f082075ef91b59408edcb2912361
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/131831
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
* Correct mistake in the named configuration.
* Add missing test directories to the fileset.
* Add configurations for all operating systems (for local testing).
* Avoid duplicate experiments specified by both configuration and test.
* Pass experiment flags also to VM running the batch runner.
Change-Id: I47f2c1b07e91e2f8120a2712ce362689b68e9ce4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/130900
Reviewed-by: Bob Nystrom <rnystrom@google.com>
It seems that we need to pass this flag to the VM now based on the
following message that appears when trying to run tests with the
experiment enabled:
```
VM initialization failed: Snapshot not compatible with the current VM
configuration: the snapshot requires 'release no-dwarf_stack_traces
use_bare_instructions nnbd-experiment' but the VM has
'release no-dwarf_stack_traces use_bare_instructions'
```
Alternatively, we could use the script from the sdk_nnbd/bin/ directory
where the experiment is hardcoded on, but that could be more confusing
when we unfork. WDYT?
Bug: #38665
Change-Id: I033e9e2e4a38f04c5cb8227efeee5a66826b46e5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/128570
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Nicholas Shahan <nshahan@google.com>
Allows us to more easily start running tests that use the NNBD sdk and do not
depend on these packages. We can add them back when they are migrated or can
easily be opted out and they build without errors.
Also cleans up a little of the ddk/ddc split and the location of the package
.dill files when running without --use-sdk.
Change-Id: I12fcbf18b47ad3189040f7e42c3ce9a4a06ed838
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/127966
Commit-Queue: Nicholas Shahan <nshahan@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
The script verifies that the multi-test only contains static annotations
and that converting it to a single test produces the same error messages
as testing the sub-tests generated by the multi-test.
If that is the case, it uses
pkg/test_runner/tool/update_static_error_tests.dart
to insert the expected errors and removes the multi-test markers.
Additionally, a copy of the '/none' test and of all the sub-tests with an
'OK' marker are created to preserve the testof runtime behavior.
Change-Id: I6eb67246f209dfbdf9b4ba8705c368c6f1978938
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/124128
Commit-Queue: Karl Klose <karlklose@google.com>
Reviewed-by: Jonas Termansen <sortie@google.com>
It doesn't do anything useful yet because the forked co19 tests aren't
getting pulled into the repo, but once gclient syncs them to
tests/co19, this should start doing something.
Change-Id: I5ee54f9afc19353c7a7a379178df1084c6f2e8d9
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/125681
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
- Copies corelib_2/a* -> corelib/
- Copies language_2/ab* -> language/
- Copies lib_2/math/ -> lib/math/
- Copies standalone_2/a* -> standalone/
And also copies over and renames all of the status files in those
directories.
Then it migrates those tests to be static error free in NNBD.
Finally, adds support to the test_runner for the new suites.
Note that this review is split into multiple patchsets. The first
patchset is a straight copy of the existing files. Then the later
patchsets have the interesting changes.
Change-Id: Icec2ff850a3aee30b653066ac184495d1e3814d0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/125467
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Leaf Petersen <leafp@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
This change allows to run test.dart and pkg/test_runner with multiple arguments
for the -n option to run tests for multiple configurations with one invocation.
Change-Id: If62e0bfc364460fa415c7f700f7e449b0de56987
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/122395
Commit-Queue: Karl Klose <karlklose@google.com>
Reviewed-by: William Hesse <whesse@google.com>
Jonas Termansen