This makes tests more reproducible, and makes it easier to copy commands between workspaces, or between a failing bot and a local workspace.
Change-Id: Ic8dd10a3540f314a406e5c5b0a23d97032e5d01d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/508364
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
This CL adds support to the test runner for the test outcome
expectation syntax that uses `// [...]` to indicate which diagnostic
we should expect with the given tool. Currently, `...` is `analyzer`
when testing the analyzer and `cfe` when testing the common front end.
After landing this CL, it also supports `// [spec_parser]`.
This change would allow us to migrate several tests from the multi-test
format (like `code; //# 01: compile-time error`) to the new format, in
particular tests with intentional syntax errors.
Change-Id: I5dd2560a873eb7433d3e07aa8f393e83c51b28e7
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/501161
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Erik Ernst <eernst@google.com>
Change the `test_runner` package's use of `DiagnosticCode.name` to
`DiagnosticCode.lowerCaseName`. There is no functional change since
all diagnostic codes generated by the analyzer are already in lower
case.
This helps pave the way for eventually deprecating and removing
`DiagnosticCode.name`, which will help ensure that analyzer clients
treat diagnostic codes in a case-insensitive fashion.
Change-Id: I6a6a69644898dff67a1f53dbe8c8a8cb757d2c96
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/468121
Commit-Queue: Paul Berry <paulberry@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Prior to this change, the static error test updater would spawn a new
analyzer CLI for each updated test file. This was incredibly slow, like
5-10 seconds for each file.
This change instead uses the analyzer as a library. It creates a single
analysis context collection for all of the files to be updated and then
gets the errors from that.
On my machine, it's about 100x faster. It's fast enough to routinely
update *all* of the test expectations in a batch. (Or, at least, it will
be if we can make a similar change for CFE.)
Change-Id: I39059489e966abf040e9f995505c7013b18d4930
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/396574
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Reviewed-by: Paul Berry <paulberry@google.com>
Commit-Queue: Konstantin Shcheglov <scheglov@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
This warning is similar to the existing `UNREACHABLE_SWITCH_CASE`
warning, except that it warns if the `default` clause of a switch
statement is unreachable due to all the `case` clasuses fully
exhausting the switched type.
To make the implementation easier, I changed the API for the
`reportExhaustiveness` method in `_fe_analyzer_shared` (which is the
primary entry point to the shared exhaustiveness checker). Previously,
this method returned a list of `ExhaustivenessError`, where each list
element was either an `UnreachableCaseError` (indicating that a
certain case was unreachable) or a `NonExhaustiveError` (indicating
that the entire switch statement was not exhaustive). If the caller
passed in `false` for `computeUnreachable`, `UnreachableCaseError`s
would not be returned, so the returned list would either be empty or
contain a single `NonExhaustiveError`.
The new API renames the types for clarity:
- `NonExhaustiveError` becomes `NonExhaustiveness`, to highlight the
fact that it's not necessarily an error for the switch's cases to be
non-exhaustive; it's only an error if the scrutinee's static type is
an "always exhaustive" type and there is no `default` clause.
- `UnreachableCaseError` becomes `CaseUnreachability`, to highlight
the fact that it's not an error for a case to be unreachable; it's a
warning.
Also, the new API adds instances of `CaseUnreachability` to an
optional user-provided list instead of returning a newly created list;
this allows callers to communicate that they don't need to see
`CaseUnreachability` information by passing `null`. This frees up the
return type to simply be an instance of `NonExhaustiveness` (if the
cases are not exhaustive) or `null` (if they are exhaustive). This
makes it easier for the analyzer to decide whether to issue the new
warning, because it doesn't have to dig around the list looking for an
instance of `NonExhaustiveness`.
The new warning has an associated quick fix (remove the unreachable
`default` clause). This quick fix uses the same `RemoveDeadCode` logic
in the analysis server that the existing `UNREACHABLE_SWITCH_CASE`
warning uses.
Fixes https://github.com/dart-lang/sdk/issues/54575.
Bug: https://github.com/dart-lang/sdk/issues/54575
Change-Id: I18b6b7c5249d77d28ead7488b4aae4ea65c4b664
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/378960
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
Reviewed-by: Samuel Rawlins <srawlins@google.com>
Reviewed-by: Erik Ernst <eernst@google.com>
Clean up and optimize some RegExps, and fix uses of `.group`.
Switch to a newer language version, to be able to use newer features.
Add a little documentation about why some RegExps are as they are.
Add (tentative) warning for multitests.
Change-Id: I59f73b87ce30caaeca1c0e0aa7954af1b97abd1b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/382620
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Samuel Rawlins <srawlins@google.com>
Commit-Queue: Lasse Nielsen <lrn@google.com>
Aligns exit codes of
* `dart compile wasm` and
* `pkg/dart2wasm/tool/compile_benchmark`
Also make them use exit codes recognized by the test runner to
distinguish CFE crashes, CFE compile-time-errors and other failures.
Also update status file entries to from D8 specific entries
to JS commandline shell entries
=> Step towards aligning D8 & JSC test results
Change-Id: I1acb8803f5db7c732ad546d5989b1c555583e7c5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/383660
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ömer Ağacan <omersa@google.com>
There should be only one result of running a test. Right now there's
confusion in the code base as some parts of the code use `result` and
others use `realResult` which can lead to inconsistent behavior between
what tests outcomes are and what are printed in the test progress
reporters. It can currently also result in differences in what
tools/test.py and tools/test.dart consider as test outcomes.
Change-Id: Ie7227e15ed9820432a766d6285091ed3efc0bb33
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/383841
Reviewed-by: Alexander Thomas <athom@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
Previously, `HintCode.UNREACHABLE_SWITCH_CASE` was marked as
deprecated, and `WarningCode.UNREACHABLE_SWITCH_CASE` was an alias to
it. This created a slightly confusing situation, because it meant that
all code referring to the diagnostic had to refer to it as
`WarningCode.UNREACHABLE_SWITCH_CASE` (to avoid a deprecation lint),
but the diagnostic still _behaved_ like it was a hint, and therefore
test runner expectations still had to treat it as a hint.
It turns out that it's not really necessary to go through the
deprecation dance when changing the kind of a diagnostic, since (a)
members of `HintCode` and `WarningCode` aren't exposed through the
analyzer public API, and (b) ignore comments don't have to specify
whether something is a hint or a warning.
So the easiest way to clear up the confusion is to just remove
`HintCode.UNREACHABLE_SWITCH_CASE` entirely, and move its implemention
into `WarningCode.UNREACHABLE_SWITCH_CASE` (so that the latter is no
longer an alias).
Change-Id: I9ff7901ad38a2c168c5e54cbe0c1c52bf7c50186
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/381103
Reviewed-by: Brian Wilkerson <brianwilkerson@google.com>
Reviewed-by: Samuel Rawlins <srawlins@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
Based on abandoned change #264261.
Changes to that PR: compare and show file paths, improve part parsing, include analyzer expectations from other files as well as CFE expectations, update unit tests.
Add end to end tests. The simplest case "library_failure_test.dart" passes before this PR, all others require the change in this PR to pick up expectations in other files.
Change-Id: Ia71b78a8f0dced83f603309877132f261b47c5a3
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/345541
Reviewed-by: Erik Ernst <eernst@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Morgan :) <davidmorgan@google.com>
Build one package with three elf_test_runner components: one for each of the binaries that the test harness invokes on the target. (Though currently only tests that use "dart" are setup.)
Create includable CML shards for the capabilites required by the AOT and JIT VMs.
Fold test_runner's fuchsia_cfv2.dart back into fuchsia.dart.
TEST=ci
Bug: https://github.com/dart-lang/sdk/issues/38752
Change-Id: I7bd8d43d184cbcb11903c7aed77ce31fb30cb894
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/332451
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Zijie He <zijiehe@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
This logic is responsible for creating a readable deobfuscated stack
from ddc and dart2js JavaScript stack traces. The logic was removing the
query parameter of URLs, but accidentally matched multi-line strings and
applied the regexp in invalid contexts (like the error message itself).
Excluding `\n` from the regexp pattern prevents this issue.
Example input (obfuscated error):
```
window.onerror called:
http://127.0.0.1:35679/root_build/gen/utils/ddc/stable/sdk/amd/dart_sdk.js:5082:
Uncaught Error
Error: Expected a value of type 'FutureOr<FileSystem>?', but got one of type 'LegacyJavaScriptObject'
at Object.throw_ [as throw] (http://127.0.0.1:35679/root_build/gen/utils/ddc/stable/sdk/amd/dart_sdk.js:5082:11)
at Object.castError (http://127.0.0.1:35679/root_build/gen/utils/ddc/stable/sdk/amd/dart_sdk.js:5041:15)
at Object.cast [as as] (http://127.0.0.1:35679/root_build/gen/utils/ddc/stable/sdk/amd/dart_sdk.js:5385:19)
at dart.NullableType.new.as (http://127.0.0.1:35679/root_build/gen/utils/ddc/stable/sdk/amd/dart_sdk.js:7243:60)
at _AsyncCompleter.new.complete (http://127.0.0.1:35679/root_build/gen/utils/ddc/stable/sdk/amd/dart_sdk.js:31216:28)
at http://127.0.0.1:35679/root_build/gen/utils/ddc/stable/sdk/amd/dart_sdk.js:99073:19
```
Example deobfuscation before (see truncated error and first frame was dropped accidentally):
```
window.onerror called:
out/ReleaseX64/gen/utils/ddc/stable/sdk/amd/dart_sdk.js:5082:
Uncaught Error
Error: Expected a value of type 'FutureOr<FileSystem>:5082:11)
at Object.castError ../../../../../../dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/errors.dart 127:3
at Object.cast ../../../../../../dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/operations.dart 818:12
at dart.NullableType.new.as ../../../../../../dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/types.dart 425:9
at _AsyncCompleter.new.complete ../../../../../../dart-sdk/lib/async/future_impl.dart 42:31
at <fn> ../../../../../../dart-sdk/lib/html/dart2js/html_dart2js.dart 33424:16
```
but after (error and first frame recovered):
```
window.onerror called:
out/ReleaseX64/gen/utils/ddc/stable/sdk/amd/dart_sdk.js:5082:
Uncaught Error
Error: Expected a value of type 'FutureOr<FileSystem>?', but got one of type 'LegacyJavaScriptObject'
at Object.throw_ out/ReleaseX64/gen/utils/ddc/stable/sdk/amd/dart_sdk.js 5082:11
at Object.castError ../../../../../../dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/errors.dart 127:3
at Object.cast ../../../../../../dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/operations.dart 818:12
at dart.NullableType.new.as ../../../../../../dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/types.dart 425:9
at _AsyncCompleter.new.complete ../../../../../../dart-sdk/lib/async/future_impl.dart 42:31
at <fn> ../../../../../../dart-sdk/lib/html/dart2js/html_dart2js.dart 33424:16
```
Change-Id: Id9f37efe9bdacc63aae3b24ebe0602a8ea7cf8fd
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/328765
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Sigmund Cherem <sigmund@google.com>
Sometimes failing tests crash or terminate with arbitrary exit codes.
For example, MSAN can abort a test with exit code 1.
This should not be treated as an infra failure by the test harness,
and should not prevent running other tests.
Change-Id: I97450d9cfb1e117451c35ab8a8cd01074fbef8db
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/291304
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Eventually, these hints should probably be moved over to warnings. But
for now, this makes it possible to write static error language tests
that validate that analyzer produces unreachable case warnings/hints
where expected.
Also updated the patterns/ and switch/ tests now that those errors must
be expected by the test.
Change-Id: If1fb92602c4bde2819b9eec73598033009054947
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/291967
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Reviewed-by: Samuel Rawlins <srawlins@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
* The new compiler name "ddc" will be used for all configurations
in an upcoming change.
* Aliases "dartdevc" and "dartdevk" will be removed in the future.
Change-Id: I2fa48e0ec043b1ba7fb70f883b39e40f338b6928
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/280862
Commit-Queue: Nicholas Shahan <nshahan@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
* ARGUMENT_TYPE_NOT_ASSIGNABLE_TO_ERROR_HANDLER
* BODY_MIGHT_COMPLETE_NORMALLY_CATCH_ERROR
* BODY_MIGHT_COMPLETE_NORMALLY_NULLABLE
* DEPRECATED_EXTENDS_FUNCTION
* DEPRECATED_IMPLEMENTS_FUNCTION
* DEPRECATED_MIXIN_FUNCTION
* DEPRECATED_NEW_IN_COMMENT_REFERENCE
This requires allowing these warnings to be reported in language tests without being expected.
Also, observatory now must comply with BODY_MIGHT_COMPLETE_NORMALLY_NULLABLE (or we could ignore the diagnostic).
TESTED=unit tests
Change-Id: I8facfa48a8645bac5bf8037163076a8f0a0ad05e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/275740
Reviewed-by: Ben Konyi <bkonyi@google.com>
Reviewed-by: Phil Quitslund <pquitslund@google.com>
Reviewed-by: Brian Wilkerson <brianwilkerson@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Samuel Rawlins <srawlins@google.com>
The test runner expects VM tests to exit with a compile error or
an uncaught exception, which have special exit codes.
Some front end tests fail by explicitly exiting with exit code 1.
Recognize these tests as test failures, and not as failures
of the test infrastructure.
Change-Id: Ib89d89c69f0e4f289fde0f762a7dd47dad446953
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/249820
Commit-Queue: William Hesse <whesse@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
There are two ways the test runner might want to parse and show errors
from analyzer:
1. When a static error test's output doesn't match the expected errors.
2. When a normal runtime test produces unexpected errors.
The former were correctly migrated over to parse and read the analyzer's
JSON output. The latter wasn't. It was still trying to parse the old
"|"-separated machine format.
Worse, when migrating to the new JSON output, a try-catch-block was
added to handle any unexpected errors. That had the unintended side
effect of silently catching this parse error and then continuing. The
end result is that if an analyzer test failed on a normal runtime test,
the test runner didn't print any of the analyzer errors. :(
This fixes that. It uses a single code path to parse the JSON output to
AnalyzerError. Then, for static error tests, it converts those to
StaticErrors.
Change-Id: I51c1277f0ed04faead82f579d668f8897f379a49
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/201382
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Paul Berry <paulberry@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
The CFE implementation of "why not promoted" functionality for
non-promotion of `this` doesn't associate the context message with any
location information, because there is no relevant location to cite.
For example, the output can look like this:
tests/language/why_not_promoted/this_error_test.dart:16:10: Error: Property 'isEven' cannot be accessed on 'int?' because it is potentially null.
Try accessing using ?. instead.
this.isEven;
^^^^^^
Context: 'this' can't be promoted.
See http://dart.dev/go/non-promo-this
The test runner assumes that all messages have a location, so it
wasn't picking up on this context message at all. This CL avoids the
problem by having the test runner associate any location-less context
message with the error above it.
(Note that the analyzer doesn't have this problem; all of its context
messages have locations).
Change-Id: Ied52daa8b0090f28617e7d3784233aa44dcc897a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/195301
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
I'm not sure if the output truncation was ever practically useful. But
it causes problems now that we have a CFE test that wants to validate
that the CFE can produce many errors in one go.
I thought about allowing a test to extend the truncation size, but I
figured the simplest fix was to just not truncate at all and make the
test runner a little simpler. Also did a little clean up on this
ancient code while I was at it.
Fix#44493.
Change-Id: Ie678c80e7114e0fb7f68931335075e3d319fb53a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/178940
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
Alexander Markov