With this change, the test runner correctly recognizes
`pkg/test_runner/test/update_errors_test.dart` as a file that's
expected _not_ to contain errors. Previously, it mistook it as a test
that was expected to contain errors, so it skipped testing it
entirely.
Change-Id: I4613215b38406b1765cb008eb33e8b076a6a6964
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/491340
Auto-Submit: Paul Berry <paulberry@google.com>
Reviewed-by: Ivan Inozemtsev <iinozemtsev@google.com>
Commit-Queue: Ivan Inozemtsev <iinozemtsev@google.com>
This update computation of implicit field types to return the inferred initializer. The ensure that we only compute the initializer once, and allows us to merge two pipelines for compute the field type and initializer.
Change-Id: I6005e19cc3dec9c717e426f06d817e2d45edcff1
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/451480
Reviewed-by: Chloe Stefantsova <cstefantsova@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
Reviewed-by: Lasse Nielsen <lrn@google.com>
Commit-Queue: Johnni Winther <johnniwinther@google.com>
This reverts commit f95a84da5c.
Reason for revert: Testing if CL caused increased meta-flakiness of non-timeout failures.
Bug: https://github.com/dart-lang/sdk/issues/55044
Original change's description:
> [test_runner] Improve timeout deflaking
>
> The usual deflaking logic is to simply rerun the test 5 times, but if the failure was a timeout this can be very expensive. Instead, only rerun timeout failures twice, and use the last successful run time to set a tighter timeout.
>
> Repeat counts and timeouts are now per-test. The name, repeat count, and timeout make a `DeflakeInfo`. This object is constructed in compare_results.dart, and passed to test.py's `--tests` flag. The recipe was already passing the output of compare_results directly to that flag, so no change is needed to the recipies.
>
> Backwards compatibility:
>
> If you want the old behavior of compare_results.dart, use the `--name-only` flag.
>
> test.py only uses this new logic if it detects that the `--tests` flag is JSON (if it starts with a `{`).
>
> Change-Id: I4113b68c54bfb7fd9a5e8fc9dab7a265807f3e77
> Bug: https://github.com/dart-lang/sdk/issues/55044
> Fixes: https://github.com/dart-lang/sdk/issues/55044
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/435760
> Commit-Queue: Liam Appelbe <liama@google.com>
> Reviewed-by: Alexander Thomas <athom@google.com>
Bug: https://github.com/dart-lang/sdk/issues/55044
Change-Id: I68268dcaffbe857ccec20e211c34325c03643b37
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/439180
Commit-Queue: Liam Appelbe <liama@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
The usual deflaking logic is to simply rerun the test 5 times, but if the failure was a timeout this can be very expensive. Instead, only rerun timeout failures twice, and use the last successful run time to set a tighter timeout.
Repeat counts and timeouts are now per-test. The name, repeat count, and timeout make a `DeflakeInfo`. This object is constructed in compare_results.dart, and passed to test.py's `--tests` flag. The recipe was already passing the output of compare_results directly to that flag, so no change is needed to the recipies.
Backwards compatibility:
If you want the old behavior of compare_results.dart, use the `--name-only` flag.
test.py only uses this new logic if it detects that the `--tests` flag is JSON (if it starts with a `{`).
Change-Id: I4113b68c54bfb7fd9a5e8fc9dab7a265807f3e77
Bug: https://github.com/dart-lang/sdk/issues/55044
Fixes: https://github.com/dart-lang/sdk/issues/55044
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/435760
Commit-Queue: Liam Appelbe <liama@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
When the test runner output is piped to a file, 'compact' progress
indicator prints carriage returns to the file.
To avoid adding carriage returns to text files, only use 'compact'
indicator when stdout is a terminal. Otherwise use 'line' indicator.
Change-Id: Ic8ca111059799876caf401d3b040a5d3b5a7c731
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/431861
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Ömer Ağacan <omersa@google.com>
The static error test format allows an expected error message to contain newlines like:
```dart
some bad code;
// [cfe] The error message
// is two lines long.
```
Here, the expected error is "The error message\nistwo lines long."
This has an unfortunate side effect in that if you have a comment
immediately after a static error expectation, it will be treated as part
of the expected error:
```dart
some bad code;
// [cfe] Error.
// Unrelated comment.
```
This will expect the CFE to report "Error.\nUnrelated comment."
I looked at every static error test and there are no intentional uses
of multi-line expectations. The only two I found were both bugs.
Since this functionality is unused and error-prone, this CL removes it.
Error expectations can only be one line.
Fix#60137.
Change-Id: I7edb1f60a0c87d4160b93810806de49bc8f55b15
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/412185
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Commit-Queue: Johnni Winther <johnniwinther@google.com>
* Simplify runCommand using async/await.
* Remove outdated memory leak hack for DDC.
* Gather batch runner functionality in BatchRunnerProcess.
* Fix process leak that likely caused a hang on test.py shutdown.
* Add a test.
Change-Id: I87584e4c951ad0fe1650c67db090910acf0f37e4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/408700
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
The old formatter had a special rule that if a line comment as at the
left edge of the page, it would stay there regardless of the surrounding
indentation. So if you had:
```
class C {
m() {
// comment
}
}
```
After formatting, the comment would still be there instead of being
indented. The intent of that was to not shift over code that had been
commented out.
But all of the IDEs I tested don't actually work that way. When they
comment out code, they tend to put the `//` at the indentation of the
surrounding code. So the new formatter doesn't have this special rule
and always indents line comments following the surrounding indentation.
This is good because it also means that code generators that don't write
any leading whitespace will still get nicely formatted comments.
However, the static error test updater took advantage of this rule and
would write static error marker comments at column zero if needed to
get the carets to align with the code on the previous line and assume
that the formatter wouldn't move the comment.
This fixes the static error test updater. It always writes comments
using indentation from the previous line of code and if the caret
doesn't fit that way, it uses an explicit column marker.
Fix#57042.
Change-Id: I40fd7cd19d08dc228b6a6797e6a26965d1343d32
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/394363
Reviewed-by: Nate Bosch <nbosch@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Nate Bosch <nbosch@google.com>
* tools/test.py used testCase.unexpectedOutcome
* tools/test.dart used testCase.realExpected
This CL purely changes local usage of `tools/test.py` and does
not affect CI infrastucture or `tools/test.dart` runs
This almost gets rid of `expectedOutcomes` but it's still used
- for Summary reporting
- for Skip, SkipByDesign
- for Crash
Issue https://github.com/dart-lang/sdk/issues/56660
Change-Id: Ib52bd03b884e004da579b9318f1074f76bd4b43e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/383662
Reviewed-by: Alexander Thomas <athom@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
Based on abandoned change #264261.
Changes to that PR: compare and show file paths, improve part parsing, include analyzer expectations from other files as well as CFE expectations, update unit tests.
Add end to end tests. The simplest case "library_failure_test.dart" passes before this PR, all others require the change in this PR to pick up expectations in other files.
Change-Id: Ia71b78a8f0dced83f603309877132f261b47c5a3
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/345541
Reviewed-by: Erik Ernst <eernst@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Morgan :) <davidmorgan@google.com>
In slow configurations, the test subprocesses could
take more than 10 seconds. Increase their timeout,
and use a smaller timeout for the timing-out test.
Change-Id: I218a3b18b763f4ea9c4868b2841b2519ef3b9746
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/306360
Reviewed-by: Tess Strickland <sstrickl@google.com>
Commit-Queue: William Hesse <whesse@google.com>
These were added during the Dart 2 test migration as a way to indicate
why a test was expected to fail. But, like negative tests, they have
very poorly granularity.
Static error tests are strictly superior.
The co19 tests have been migrated off of these markers for a couple of
years, so our own language tests were the only holdouts. I see no uses
of "@syntax-error", "@runtime-error", or "@static-warning". I have now
migrated all of the tests that contained "@compile-error" to be proper
static error tests.
This simplifies the test runner and makes our tests more precise.
Fix#45634.
Change-Id: I0f46d110b6f322d98187e734195ecba7524574af
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/296720
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Use "ddcOptions" in test files to closer resemble the name 'ddc-options'
from the test matrix or command line option.
I don't like that this option is in camelCase when written in a test
file but uses a dash when passed on the command line but that matches
all the other implementations (dart2jsOptions, sharedOptionn, etc).
Hopefully it is the most discoverable or least surprising choice.
Change-Id: I8f0d53d50416b952c9ea74406c18bb9c700f98dd
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/281820
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Nicholas Shahan <nshahan@google.com>
Make test.py pick migrated suites by default when the `NnbdMode` is not `NnbdMode.legacy`.
* Remove non-existing benchmark_smoke suite.
* Remove broken analyzer_library suite from default suites.
* Deprecated observatory_ui is not added to the default migrated suites.
* Remove remaining references to samples-dev.
* Remove unnecessary suite specifications from the test_matrix.json.
* Remove broken references to observatory_ui from the test_matrix.json.
* Remove defunct observatory_ui hack from test.py.
Fixes: b/268474066
Change-Id: I6c3635bcb396021ca86498d60c9efc77ab39589e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/281701
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
"vm-aot" is more user friendly than dart_precompiled-dartkp. But, "vm" implied --runtime="vm" which is incompatible with --compiler="dartkp".
Bug: b/201271877
Change-Id: I19e6ccf77d2f56df5f3e27586c022a1810f7ef6e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/280094
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
* Use lowercase-with-hyphens for option names
* Add backwards compatible aliases_with_underscores.
* Deny list `reset-browser-configuration`.
* Add more tests.
Bug: b/232495224
Change-Id: Ife56c48450b5e15577c4aec859355e80c2f57bdc
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/245366
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
This changes safaridriver to be running whenever test.py runs any safari
configuration. The service exists when all tests are done. The process
can only be used by a single browser but supports many consecutive
sessions.
Bug: b/208186791
Cq-Include-Trybots: luci.dart.try:dart2js-strong-mac-x64-safari-try
Change-Id: I7c1d5910f8c97ae97c1806a07251df918f139e0a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/243720
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
For some reason, we launched Safari from a file, which since macOS 10.14
requires the user to confirm the operation in a modal dialog which
caused timeouts.
* Removed unused http test driver port option.
* Added some basic tests for test.py's browser interactions.
Bug: b/208186791
Change-Id: I070529148d37bf312f219c576abca3da972f73e6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/204202
Commit-Queue: Alexander Thomas <athom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
* Remove all uses of `--dart2js-batch`.
* Enable batch mode for dart2js by default.
* Rename `--noBatch` to `--no-batch`.
* Make `--no-batch` affect dart2js configurations.
Change-Id: I33bdb5cfe8a82ccfc6ce9e43843f1b7a34ff1fbf
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/217006
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
* Move compare_results.dart to pkg/test_runner.
* Clean up analyzer issues in compare_results.dart.
* Add a test for compare_results.dart.
* Fix compare and extend results to detect expectation changes.
Change-Id: I451ddc2d0b01a7e5fd7c9d791b000c786c2f2c93
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/214805
Commit-Queue: Alexander Thomas <athom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
Konstantin Shcheglov