Sometimes, the batch runners need the experiment flags set. This is
especially true when running in NNBD mode which needs to be enabled in
the VM running the batch runner as well as in the test code.
Previously this was only done for the CFE batch runner, this CL moves
the logic to apply to all compilation commands using batch runners.
Change-Id: I9358ae7cfed4f082075ef91b59408edcb2912361
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/131831
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
* Correct mistake in the named configuration.
* Add missing test directories to the fileset.
* Add configurations for all operating systems (for local testing).
* Avoid duplicate experiments specified by both configuration and test.
* Pass experiment flags also to VM running the batch runner.
Change-Id: I47f2c1b07e91e2f8120a2712ce362689b68e9ce4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/130900
Reviewed-by: Bob Nystrom <rnystrom@google.com>
It seems that we need to pass this flag to the VM now based on the
following message that appears when trying to run tests with the
experiment enabled:
```
VM initialization failed: Snapshot not compatible with the current VM
configuration: the snapshot requires 'release no-dwarf_stack_traces
use_bare_instructions nnbd-experiment' but the VM has
'release no-dwarf_stack_traces use_bare_instructions'
```
Alternatively, we could use the script from the sdk_nnbd/bin/ directory
where the experiment is hardcoded on, but that could be more confusing
when we unfork. WDYT?
Bug: #38665
Change-Id: I033e9e2e4a38f04c5cb8227efeee5a66826b46e5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/128570
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Nicholas Shahan <nshahan@google.com>
Allows us to more easily start running tests that use the NNBD sdk and do not
depend on these packages. We can add them back when they are migrated or can
easily be opted out and they build without errors.
Also cleans up a little of the ddk/ddc split and the location of the package
.dill files when running without --use-sdk.
Change-Id: I12fcbf18b47ad3189040f7e42c3ce9a4a06ed838
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/127966
Commit-Queue: Nicholas Shahan <nshahan@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
The script verifies that the multi-test only contains static annotations
and that converting it to a single test produces the same error messages
as testing the sub-tests generated by the multi-test.
If that is the case, it uses
pkg/test_runner/tool/update_static_error_tests.dart
to insert the expected errors and removes the multi-test markers.
Additionally, a copy of the '/none' test and of all the sub-tests with an
'OK' marker are created to preserve the testof runtime behavior.
Change-Id: I6eb67246f209dfbdf9b4ba8705c368c6f1978938
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/124128
Commit-Queue: Karl Klose <karlklose@google.com>
Reviewed-by: Jonas Termansen <sortie@google.com>
It doesn't do anything useful yet because the forked co19 tests aren't
getting pulled into the repo, but once gclient syncs them to
tests/co19, this should start doing something.
Change-Id: I5ee54f9afc19353c7a7a379178df1084c6f2e8d9
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/125681
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
- Copies corelib_2/a* -> corelib/
- Copies language_2/ab* -> language/
- Copies lib_2/math/ -> lib/math/
- Copies standalone_2/a* -> standalone/
And also copies over and renames all of the status files in those
directories.
Then it migrates those tests to be static error free in NNBD.
Finally, adds support to the test_runner for the new suites.
Note that this review is split into multiple patchsets. The first
patchset is a straight copy of the existing files. Then the later
patchsets have the interesting changes.
Change-Id: Icec2ff850a3aee30b653066ac184495d1e3814d0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/125467
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Leaf Petersen <leafp@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
This change allows to run test.dart and pkg/test_runner with multiple arguments
for the -n option to run tests for multiple configurations with one invocation.
Change-Id: If62e0bfc364460fa415c7f700f7e449b0de56987
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/122395
Commit-Queue: Karl Klose <karlklose@google.com>
Reviewed-by: William Hesse <whesse@google.com>
Both a test file and now a test configuration can specify that
experiment flags should be enabled. Since they both can, passing those
to the tools is a little tricky because we need to merge the two
command line arguments. It's made worse by the fact that this corner of
test_runner is riddled with poorly factored code.
This doesn't fix the redundant code, but does merge and pass the flags
through. Right now, it parses them from the test file by looking for
them in the SharedOptions line, which is where they currently appear.
That's pretty hacky. Eventually, we should change these tests to use:
// Requirements=
Fixes#38386, #38387, #38388, and #38389.
Change-Id: I1aea1482b30b9ba19c427319f895822d81ee12ff
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/121262
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
If the current test configuration enables the "non-nullable" experiment,
the test runner will read from the NNBD build directory instead of the
normal one.
This only kicks in when the experiment is enabled by the
*configuration*. If the test file enables it using:
// SharedOptions=--enable-experiment=non-nullable
It has no effect on which directory is used. I think the right fix for
this is to use a different mechanism for these tests to specify that
they require NNBD. I'll send an email about that.
Change-Id: I4be2de8d0f7301c93dfb3f7e81c12372fc5d154a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/121143
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Nicholas Shahan <nshahan@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
This gives us a way to specify that a test is only meaningful for
certain NNBD modes and should be skipped on other configurations. My
hope is we can extend this for other platform capabilities and then
eventually use this to express "skips" instead of relying on status
files for them.
Change-Id: I99548c326ee6fbe6db64e28e7f7f07d2fc1fd23c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/117080
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Mayank Patke <fishythefish@google.com>
- Make it synchronous. I did some benchmarking and the async didn't seem
to make a measurable difference and made the code harder to follow.
- Fix a bunch of small-scale idiom things: "var", names, etc.
- Move TestCase.hash into TestFile since that lets us access it before
we've created a TestCase.
- Remove dead code.
- Make it clearer which functions work with TestFiles and which with
TestCases.
- Clarify the code that determines whether or not to enqueue a test.
The last point is the motivating one. Soon, I'll be extending this code
to take NNBD into account when determining which tests to skip, so I
wanted to clean it up some first.
Change-Id: I488ed6c7d2453535968ac43389a2fd8370ead57d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/116662
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Mayank Patke <fishythefish@google.com>
- Better name than "optedIn".
- Actually hook up the command-line option to the configuration.
- Add tests, which would have caught the previous mistake.
- Don't allow comma-separated values for "--progress" and "--nnbd".
- Remove unused dead "--strong" option.
Change-Id: I57d7cb0d81af50d662dcf3f7f4c9ca1f2b102f2f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/116544
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Mayank Patke <fishythefish@google.com>
The CFE currently does not report the length in an error location, just
the starting position. When the static error update tool inserts an
error marker for an error reported only by the CFE, it doesn't know
what length to use.
It used to deal with this by always writing an explicit error location
like:
// [error line 1, column 9]
Those are kind of ugly and brittle, though. This changes it to treat
the error as implicitly having length 1. This way, it can just output:
// ^
When validating an error expectation against a report CFE error, the
length is ignored anyway (since the CFE doesn't report it). In order to
ensure that the parsed output of the tool matches the reported data that
produced it, the parser also ignores the length when parsing an error
expectation for a CFE-only error with length one.
Fix#37991.
Change-Id: I20e109142546b7e82a5f796a1a40613b90dc89bd
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/114745
Reviewed-by: Leaf Petersen <leafp@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
This rarely comes into play because most static error markers don't
have an explicit line number. (This is by design so that shifts don't
cause diff churn.) But in some cases, the line number is written. When
that happens, the number may be wrong because previous added or removed
errors can shift the output line from where the original error was
reported.
Compensate for that. Fix#37990.
Change-Id: Ie306465c11d3b6a1de5ed158ad63167b4a5d4797
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/114594
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Leaf Petersen <leafp@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
- Remove the pointless static factory methods. Most of them only had a
single callsite and in either case, they accomplished nothing useful
since the underlying classes are already public.
- Make index final.
- Make indexedCopy() and remove the unused (and wrong!) base
implementation.
- Define an abstract createOutput() method on Command and have all
Command classes implement that to return an output for themselves.
This is a better object-oriented design than a monolithic
createCommandOutput() function full of "is ___" checks.
- Remove CleanDirectoryCopyCommand and MakeSymlinkCommand. They were
unused.
- Change VmBatchCommand to VMBatchCommand to follow the style guide.
- Capitalize "L" in JSCommandLineCommand.
There are no behavioral changes. This stuff was just bugging me.
Change-Id: I2493f6afe0ee129e9c24e3f8289d37fe6dd4f802
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/114465
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
In order to test debugger functionality on vm-dartkb-*-simarm64 bots
we need to pass 'debugger-stops' bytecode generation option to kernel
compiler.
We don't have debugger functionality in PRODUCT mode, so 'debugger-stops'
(and 'local-var-info') are omitted.
Change-Id: I6453f9277862816d127e68f802cfe8db1e4b6c64
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/114720
Reviewed-by: Régis Crelier <regis@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
The old code to report the difference between a tests's expected and
actual errors was easily confused by missing or unexpected errors.
Those would misalign the lists of errors causing all subsequent errors
to be reported as wrong. This fixes that. Before, this test containing
a single unexpected error:
main() {
int a = "s";
// ^^^
// [analyzer] STATIC_TYPE_WARNING.INVALID_ASSIGNMENT
unknownFunction();
int b = "s";
// ^^^
// [analyzer] STATIC_TYPE_WARNING.INVALID_ASSIGNMENT
int c = "s";
// ^^^
// [analyzer] STATIC_TYPE_WARNING.INVALID_ASSIGNMENT
}
Would cause the test runner to output:
incorrect static errors:
Error at line 6, column 3, length 15
- Expected on line 8 but was on 6.
- Expected on column 11 but was on 3.
- Expected length 3 but was 15.
- Expected error code STATIC_TYPE_WARNING.INVALID_ASSIGNMENT but was STATIC_TYPE_WARNING.UNDEFINED_FUNCTION.
Error at line 8, column 11, length 3
- Expected on line 12 but was on 8.
reported unexpected static errors:
Error at line 12, column 11, length 3
STATIC_TYPE_WARNING.INVALID_ASSIGNMENT
Oof. Now it prints:
static error failures:
Unexpected static error at line 6, column 3, length 15:
- Had error code STATIC_TYPE_WARNING.UNDEFINED_FUNCTION.
Change-Id: I1b22b2a542c6f8049f92fda3627ae69144bfd52a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/114240
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Leaf Petersen <leafp@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Alexander Thomas