In slow configurations, the test subprocesses could
take more than 10 seconds. Increase their timeout,
and use a smaller timeout for the timing-out test.
Change-Id: I218a3b18b763f4ea9c4868b2841b2519ef3b9746
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/306360
Reviewed-by: Tess Strickland <sstrickl@google.com>
Commit-Queue: William Hesse <whesse@google.com>
These were added during the Dart 2 test migration as a way to indicate
why a test was expected to fail. But, like negative tests, they have
very poorly granularity.
Static error tests are strictly superior.
The co19 tests have been migrated off of these markers for a couple of
years, so our own language tests were the only holdouts. I see no uses
of "@syntax-error", "@runtime-error", or "@static-warning". I have now
migrated all of the tests that contained "@compile-error" to be proper
static error tests.
This simplifies the test runner and makes our tests more precise.
Fix#45634.
Change-Id: I0f46d110b6f322d98187e734195ecba7524574af
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/296720
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Use "ddcOptions" in test files to closer resemble the name 'ddc-options'
from the test matrix or command line option.
I don't like that this option is in camelCase when written in a test
file but uses a dash when passed on the command line but that matches
all the other implementations (dart2jsOptions, sharedOptionn, etc).
Hopefully it is the most discoverable or least surprising choice.
Change-Id: I8f0d53d50416b952c9ea74406c18bb9c700f98dd
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/281820
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Nicholas Shahan <nshahan@google.com>
Make test.py pick migrated suites by default when the `NnbdMode` is not `NnbdMode.legacy`.
* Remove non-existing benchmark_smoke suite.
* Remove broken analyzer_library suite from default suites.
* Deprecated observatory_ui is not added to the default migrated suites.
* Remove remaining references to samples-dev.
* Remove unnecessary suite specifications from the test_matrix.json.
* Remove broken references to observatory_ui from the test_matrix.json.
* Remove defunct observatory_ui hack from test.py.
Fixes: b/268474066
Change-Id: I6c3635bcb396021ca86498d60c9efc77ab39589e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/281701
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
"vm-aot" is more user friendly than dart_precompiled-dartkp. But, "vm" implied --runtime="vm" which is incompatible with --compiler="dartkp".
Bug: b/201271877
Change-Id: I19e6ccf77d2f56df5f3e27586c022a1810f7ef6e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/280094
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
* Use lowercase-with-hyphens for option names
* Add backwards compatible aliases_with_underscores.
* Deny list `reset-browser-configuration`.
* Add more tests.
Bug: b/232495224
Change-Id: Ife56c48450b5e15577c4aec859355e80c2f57bdc
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/245366
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
This changes safaridriver to be running whenever test.py runs any safari
configuration. The service exists when all tests are done. The process
can only be used by a single browser but supports many consecutive
sessions.
Bug: b/208186791
Cq-Include-Trybots: luci.dart.try:dart2js-strong-mac-x64-safari-try
Change-Id: I7c1d5910f8c97ae97c1806a07251df918f139e0a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/243720
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
For some reason, we launched Safari from a file, which since macOS 10.14
requires the user to confirm the operation in a modal dialog which
caused timeouts.
* Removed unused http test driver port option.
* Added some basic tests for test.py's browser interactions.
Bug: b/208186791
Change-Id: I070529148d37bf312f219c576abca3da972f73e6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/204202
Commit-Queue: Alexander Thomas <athom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
* Remove all uses of `--dart2js-batch`.
* Enable batch mode for dart2js by default.
* Rename `--noBatch` to `--no-batch`.
* Make `--no-batch` affect dart2js configurations.
Change-Id: I33bdb5cfe8a82ccfc6ce9e43843f1b7a34ff1fbf
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/217006
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
* Move compare_results.dart to pkg/test_runner.
* Clean up analyzer issues in compare_results.dart.
* Add a test for compare_results.dart.
* Fix compare and extend results to detect expectation changes.
Change-Id: I451ddc2d0b01a7e5fd7c9d791b000c786c2f2c93
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/214805
Commit-Queue: Alexander Thomas <athom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
Error tests with explicit line/column markers, like
[error line 123, column 456]
currently require both "line" and "column".
All existing occurrences of `[error line 123, column 456]` always
refer to the most recent non-comment line *just like* the `// ^^` lines,
so omitting the line, and having the most recent non-comment line
as the *default* line, seems reasonable.
This CL makes the line optional, the new format without a "line" entry
will be just `[error column 456]`.
This allows files to change the number of lines in the code without
invalidating any later expectations.
(Or, say, remove a comment line at the top of the file!)
Omits the line entry by default when writing/updating expectations.
Change-Id: Ia0c95bf804a1c6b11c0c1621bfd481e43f8fa0c0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/200429
Commit-Queue: Lasse R.H. Nielsen <lrn@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Currently only CFE ("Fasta") tests have their context message output
parsed. It should be easy to extend that to dart2js and DDC if that's
useful. Analyzer might be more work.
This also adds support to the test updater for inserting context
messages when updating tests. By default, that flag is off, so the
existing behavior is preserved where context messages are ignnored. If
you want them, pass "-c" when updating a test.
When validating test output, if the test file contains context messages,
then they are validated. Otherwise, any context messages in the CFE
output are ignored. This way existing tests still pass.
Change StaticError to represent a single error for a single front end.
Before, the data model collapsed errors for different front-ends at the
same location into a single StaticError object which tracked different
messages for each front end. The idea was to move towards a world where
they really are the "same" error with eventually the same message.
But this adds a lot of complexity with things like merging errors and
doesn't reflect the reality that each error from each front end is
basically its own thing. Also, critically, it makes it much harder to
attach context messages to a specific front end's error object.
This changes it so that an instance of StaticError represents a single
error for a single front end. The test file syntax is unchanged and the
updated tool behaves the same. In a static error test, multiple
expectations can still share the same "// ^^^" marker line. They are
just expanded to multiple StaticError objects at parse time.
This eliminates all of the complexity around merging and simplifying
errors.
Change-Id: Ida1736bfcde436fc2d1ce2963d91fa9cb154afa8
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/193281
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Paul Berry <paulberry@google.com>
Before, the data model collapsed errors for different front-ends at the
same location into a single StaticError object which tracked different
messages for each front end. The idea was to move towards a world where
they really are the "same" error with eventually the same message.
But this adds a lot of complexity with things like merging errors and
doesn't reflect the reality that each error from each front end is
basically its own thing. Also, critically, it makes it much harder to
attach context messages to a specific front end's error object.
This changes it so that an instance of StaticError represents a single
error for a single front end. The test file syntax is unchanged and the
updated tool behaves the same. In a static error test, multiple
expectations can still share the same "// ^^^" marker line. They are
just expanded to multiple StaticError objects at parse time.
This eliminates all of the complexity around merging and simplifying
errors.
Change-Id: I1d55a6e885e12cc9c438f928297fc0db7dd5ce85
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/193280
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
Reviewed-by: Paul Berry <paulberry@google.com>
This fixes an issue introduced when DDS testing support was added to the
test suite. Previously, tests with multiple VMOptions lines would get
the same name for both DDS and service variants. This caused incorrect
behavior in the results processing that broke deflaking, but also
caused one of the results to be ignored by the infrastructure.
Also adds some tests for VMOptions handling in test_suite.dart.
Fixes https://github.com/dart-lang/sdk/issues/43768
Change-Id: I3c0f9cbc1807fe814aed5ecb7531ef4289e95683
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/166858
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
If a static error test only has expectations for *warnings* then it
still has well-defined runtime semantics that we want to test.
This distinguishes those tests and runs them on non-front-end
configurations.
Change-Id: I41b8d84a229ba53ad0db0271b28a9b9482ad582d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/155305
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Karl Klose <karlklose@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
This doesn't actually run DDC to generate the web errors yet, but it
changes the CLI in anticipation of that, and adds tests to verify that
once web errors are reported that the updater handles them correctly.
Change-Id: I31264e3d468969b07f9eb60353a9b02a93bec7ea
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/155102
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Srujan Gaddam <srujzs@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Previously it tested if it was at the end of file (defined as
_currentLine >= _lines.length), if it was not it would try to read the
_next_ line (_peek(1)), meaning that if _currentLine = _lines.length - 1
it would _peek(1) and try to read _lines[lines.length] and crash with a
range error. This CL fixes it by actually checking if it can safely peek
the amount it wants to.
Change-Id: I2e66c539e823f0b39bd0da39e2b2277a9e6c9501
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/150624
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
This enables parsing and validation for tests that contain:
// ^^^
// [web] Some web-specific error.
This doesn't run those tests on DDC and dart2js yet, and the test
updating tool doesn't handle web tests yet. I'll do those in follow-up
patches.
Change-Id: Id06397a20a06c00e48801a16c01d5878e0264f6b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/146462
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Srujan Gaddam <srujzs@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Both a test file and now a test configuration can specify that
experiment flags should be enabled. Since they both can, passing those
to the tools is a little tricky because we need to merge the two
command line arguments. It's made worse by the fact that this corner of
test_runner is riddled with poorly factored code.
This doesn't fix the redundant code, but does merge and pass the flags
through. Right now, it parses them from the test file by looking for
them in the SharedOptions line, which is where they currently appear.
That's pretty hacky. Eventually, we should change these tests to use:
// Requirements=
Fixes#38386, #38387, #38388, and #38389.
Change-Id: I1aea1482b30b9ba19c427319f895822d81ee12ff
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/121262
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
This gives us a way to specify that a test is only meaningful for
certain NNBD modes and should be skipped on other configurations. My
hope is we can extend this for other platform capabilities and then
eventually use this to express "skips" instead of relying on status
files for them.
Change-Id: I99548c326ee6fbe6db64e28e7f7f07d2fc1fd23c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/117080
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Mayank Patke <fishythefish@google.com>
- Make it synchronous. I did some benchmarking and the async didn't seem
to make a measurable difference and made the code harder to follow.
- Fix a bunch of small-scale idiom things: "var", names, etc.
- Move TestCase.hash into TestFile since that lets us access it before
we've created a TestCase.
- Remove dead code.
- Make it clearer which functions work with TestFiles and which with
TestCases.
- Clarify the code that determines whether or not to enqueue a test.
The last point is the motivating one. Soon, I'll be extending this code
to take NNBD into account when determining which tests to skip, so I
wanted to clean it up some first.
Change-Id: I488ed6c7d2453535968ac43389a2fd8370ead57d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/116662
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Mayank Patke <fishythefish@google.com>
- Better name than "optedIn".
- Actually hook up the command-line option to the configuration.
- Add tests, which would have caught the previous mistake.
- Don't allow comma-separated values for "--progress" and "--nnbd".
- Remove unused dead "--strong" option.
Change-Id: I57d7cb0d81af50d662dcf3f7f4c9ca1f2b102f2f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/116544
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Mayank Patke <fishythefish@google.com>
The CFE currently does not report the length in an error location, just
the starting position. When the static error update tool inserts an
error marker for an error reported only by the CFE, it doesn't know
what length to use.
It used to deal with this by always writing an explicit error location
like:
// [error line 1, column 9]
Those are kind of ugly and brittle, though. This changes it to treat
the error as implicitly having length 1. This way, it can just output:
// ^
When validating an error expectation against a report CFE error, the
length is ignored anyway (since the CFE doesn't report it). In order to
ensure that the parsed output of the tool matches the reported data that
produced it, the parser also ignores the length when parsing an error
expectation for a CFE-only error with length one.
Fix#37991.
Change-Id: I20e109142546b7e82a5f796a1a40613b90dc89bd
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/114745
Reviewed-by: Leaf Petersen <leafp@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
This rarely comes into play because most static error markers don't
have an explicit line number. (This is by design so that shifts don't
cause diff churn.) But in some cases, the line number is written. When
that happens, the number may be wrong because previous added or removed
errors can shift the output line from where the original error was
reported.
Compensate for that. Fix#37990.
Change-Id: Ie306465c11d3b6a1de5ed158ad63167b4a5d4797
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/114594
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Leaf Petersen <leafp@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
- Remove the pointless static factory methods. Most of them only had a
single callsite and in either case, they accomplished nothing useful
since the underlying classes are already public.
- Make index final.
- Make indexedCopy() and remove the unused (and wrong!) base
implementation.
- Define an abstract createOutput() method on Command and have all
Command classes implement that to return an output for themselves.
This is a better object-oriented design than a monolithic
createCommandOutput() function full of "is ___" checks.
- Remove CleanDirectoryCopyCommand and MakeSymlinkCommand. They were
unused.
- Change VmBatchCommand to VMBatchCommand to follow the style guide.
- Capitalize "L" in JSCommandLineCommand.
There are no behavioral changes. This stuff was just bugging me.
Change-Id: I2493f6afe0ee129e9c24e3f8289d37fe6dd4f802
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/114465
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Ben Konyi