The static error test updater needs to ask analyzer, CFE, and dart2js
for the errors for each updated test file. Previously, it would invoke
each of those as a separate process, one at a time, for each test file.
This was comically slow.
I recently updated it to invoke analyzer as a library and analyze all
the files at once, which made that part >100x faster.
This CL does essentially the same thing for CFE. It's still invoking CFE
as a process, but it does so with a batch of files. It's not as fast as
analyzer is, but it's much better.
It's still calling dart2js once per file but, strangely, that isn't too
slow. Also, web static error tests aren't very common, so this isn't as
important.
Change-Id: I6756901bb761579dc90f8af4d9774014b04bb009
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/399885
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Paul Berry <paulberry@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Prior to this change, the static error test updater would spawn a new
analyzer CLI for each updated test file. This was incredibly slow, like
5-10 seconds for each file.
This change instead uses the analyzer as a library. It creates a single
analysis context collection for all of the files to be updated and then
gets the errors from that.
On my machine, it's about 100x faster. It's fast enough to routinely
update *all* of the test expectations in a batch. (Or, at least, it will
be if we can make a similar change for CFE.)
Change-Id: I39059489e966abf040e9f995505c7013b18d4930
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/396574
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Reviewed-by: Paul Berry <paulberry@google.com>
Commit-Queue: Konstantin Shcheglov <scheglov@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
I'm slowly going through and reformatting the tests. That also means
regenerating the static error expectations because of #57042. But I
don't need to generate static error markers for files that don't already
have them.
So this adds a flag to the test updater to let it skip over any test
file that isn't already known to be a static error test.
Change-Id: I61f10d29924f1f9d2dabf61bb604e53a6622017f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/394940
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Based on abandoned change #264261.
Changes to that PR: compare and show file paths, improve part parsing, include analyzer expectations from other files as well as CFE expectations, update unit tests.
Add end to end tests. The simplest case "library_failure_test.dart" passes before this PR, all others require the change in this PR to pick up expectations in other files.
Change-Id: Ia71b78a8f0dced83f603309877132f261b47c5a3
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/345541
Reviewed-by: Erik Ernst <eernst@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Morgan :) <davidmorgan@google.com>
Multitests aren't valid Dart files that can be processed by a Dart
implementation so the updater generally does a poor job if it tries to
update one.
It's probably not worth supporting because, in practice, a test should
either be a multitest or a static error test, but not both.
Change the tool to skip over any multitests it encounters. If this
results in it doing nothing at all, it reports that as an error.
Otherwise, it just lists the multitests it didn't process.
Close#37721.
Change-Id: Icfb1ff9fe63f2c249b3ccfba65166b97654a9918
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/296760
Commit-Queue: Leaf Petersen <leafp@google.com>
Reviewed-by: Leaf Petersen <leafp@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
We no longer build separate binaries for `dartanalyzer` and `dart2js`,
so we need to do something different.
For `dartanalyzer`, we just run the `dartanalyzer` script directly
(because the features we need for testing aren't available directly
from `dart analyze`). For `dart2js`, we run in the new way (`dart
compile js`).
Change-Id: I7e34052a9e6ad62596279ca32604e32291a45953
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/251842
Commit-Queue: Paul Berry <paulberry@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
The vendored package was just used in some tooling that should be easy
to update to any breaking changes in args. Language versioning ensures
we can handle changes in the language as well.
Bug: b/232495224
Change-Id: I15223187f19b6fe2229dd4ac05e51fd794cbfbd4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/244627
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Auto-Submit: Alexander Thomas <athom@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Currently only CFE ("Fasta") tests have their context message output
parsed. It should be easy to extend that to dart2js and DDC if that's
useful. Analyzer might be more work.
This also adds support to the test updater for inserting context
messages when updating tests. By default, that flag is off, so the
existing behavior is preserved where context messages are ignnored. If
you want them, pass "-c" when updating a test.
When validating test output, if the test file contains context messages,
then they are validated. Otherwise, any context messages in the CFE
output are ignored. This way existing tests still pass.
Change StaticError to represent a single error for a single front end.
Before, the data model collapsed errors for different front-ends at the
same location into a single StaticError object which tracked different
messages for each front end. The idea was to move towards a world where
they really are the "same" error with eventually the same message.
But this adds a lot of complexity with things like merging errors and
doesn't reflect the reality that each error from each front end is
basically its own thing. Also, critically, it makes it much harder to
attach context messages to a specific front end's error object.
This changes it so that an instance of StaticError represents a single
error for a single front end. The test file syntax is unchanged and the
updated tool behaves the same. In a static error test, multiple
expectations can still share the same "// ^^^" marker line. They are
just expanded to multiple StaticError objects at parse time.
This eliminates all of the complexity around merging and simplifying
errors.
Change-Id: Ida1736bfcde436fc2d1ce2963d91fa9cb154afa8
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/193281
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Paul Berry <paulberry@google.com>
Before, the data model collapsed errors for different front-ends at the
same location into a single StaticError object which tracked different
messages for each front end. The idea was to move towards a world where
they really are the "same" error with eventually the same message.
But this adds a lot of complexity with things like merging errors and
doesn't reflect the reality that each error from each front end is
basically its own thing. Also, critically, it makes it much harder to
attach context messages to a specific front end's error object.
This changes it so that an instance of StaticError represents a single
error for a single front end. The test file syntax is unchanged and the
updated tool behaves the same. In a static error test, multiple
expectations can still share the same "// ^^^" marker line. They are
just expanded to multiple StaticError objects at parse time.
This eliminates all of the complexity around merging and simplifying
errors.
Change-Id: I1d55a6e885e12cc9c438f928297fc0db7dd5ce85
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/193280
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
Reviewed-by: Paul Berry <paulberry@google.com>
This change adds support for the 'enable-experiments' option to control
the experiments used in the analyzer and CFE.
It also adds support for multiple input files.
Change-Id: I15b9082e96f276249165efc66734ac9506b93ca1
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/178560
Commit-Queue: Karl Klose <karlklose@google.com>
Reviewed-by: Jonas Termansen <sortie@google.com>
The static error updater used to run analyzer using the scripts in
sdk/bin/, but those are no longer reliable now that analyzer looks for
a nearby VERSION file. Instead, look for the most recently built
analyzer in the build directory.
Change-Id: If2b437b12d036e15f5503aba816dc2a007e39cfe
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/160943
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Paul Berry <paulberry@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
This doesn't actually run DDC to generate the web errors yet, but it
changes the CLI in anticipation of that, and adds tests to verify that
once web errors are reported that the updater handles them correctly.
Change-Id: I31264e3d468969b07f9eb60353a9b02a93bec7ea
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/155102
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Srujan Gaddam <srujzs@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
A while back, we changed the test runner to validate warnings in static
error tests, so this gets the updater tool to parity with that.
Also, when printing analyzer errors, sort them by location before
severity.
Change-Id: Ia07e79adb6d7ca19a50210a2813d7f2f7e60f7e1
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/148285
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Leaf Petersen <leafp@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
The script verifies that the multi-test only contains static annotations
and that converting it to a single test produces the same error messages
as testing the sub-tests generated by the multi-test.
If that is the case, it uses
pkg/test_runner/tool/update_static_error_tests.dart
to insert the expected errors and removes the multi-test markers.
Additionally, a copy of the '/none' test and of all the sub-tests with an
'OK' marker are created to preserve the testof runtime behavior.
Change-Id: I6eb67246f209dfbdf9b4ba8705c368c6f1978938
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/124128
Commit-Queue: Karl Klose <karlklose@google.com>
Reviewed-by: Jonas Termansen <sortie@google.com>
Both a test file and now a test configuration can specify that
experiment flags should be enabled. Since they both can, passing those
to the tools is a little tricky because we need to merge the two
command line arguments. It's made worse by the fact that this corner of
test_runner is riddled with poorly factored code.
This doesn't fix the redundant code, but does merge and pass the flags
through. Right now, it parses them from the test file by looking for
them in the SharedOptions line, which is where they currently appear.
That's pretty hacky. Eventually, we should change these tests to use:
// Requirements=
Fixes#38386, #38387, #38388, and #38389.
Change-Id: I1aea1482b30b9ba19c427319f895822d81ee12ff
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/121262
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Sam Rawlins