Remove the `reportScannerErrors` option from `Scanner.tokenize`. The
parser handles scanner error tokens directly, so the scanner no longer
needs a mode that translates and reports them before parsing.
Update all callers to use the simplified `tokenize()` API. Adjust parser
recovery expectations where diagnostics are now reported at the
synthetic token location chosen by parser.
Change-Id: Ia2d479ba57b3156f1a4bab47c32444892c9d0aa4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/500361
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Reviewed-by: Samuel Rawlins <srawlins@google.com>
No more flag, always parse into the new AST, always visit new AST nodes,
always return them as child entities, parent-child structure reflects
the new AST.
So, use `namePart` and `body` where possible. Deprecate previous
properties.
This is still de jure a breaking change, because `parent` of deprecated
properties changes. De facto this required very few changes in google3.
Once this CL lands, I will publish `analyzer 10.0.0`, migrate everything
to new properties, delete deprecated properties, and publish `analyzer
11.0.0`.
Maybe deprecate `NamedCompilationUnitMember.name` and migrate to
subclass specific `name` or `namePart` properties before publishing
`analyzer 10.0.0`. This part is not breaking per se.
* Deprecations in `ClassDeclaration`:
* Properties `leftBracket`, `members`, `rightBracket` are deprecated, use `body` instead.
* Properties `name`, `typeParameters` are deprecated, use `namePart` instead.
* Deprecations in `EnumDeclaration`:
* Properties `leftBracket`, `constants`, `members`, `rightBracket` are deprecated, use `body` instead.
* Properties `name`, `typeParameters` are deprecated, use `namePart` instead.
* Deprecations in `ExtensionDeclaration`:
* Properties `leftBracket`, `members`, `rightBracket` are deprecated, use `body` instead.
* Deprecations in `ExtensionTypeDeclaration`:
* Properties `leftBracket`, `constants`, `members`, `rightBracket` are deprecated, use `body` instead.
* Properties `constKeyword`, `name`, `representation`, `typeParameters` are deprecated,
use `primaryConstructor` instead.
* **Breaking Change:** While the deprecated members mentioned above still exist in the AST,
their parent nodes have changed. This means that code relying on specific parent-child
relationships for these nodes might break.
Bug: https://github.com/dart-lang/sdk/issues/61701
Change-Id: Ic48104da8b029c9b454bbd2336574b7823025565
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/461841
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Reviewed-by: Samuel Rawlins <srawlins@google.com>
Reviewed-by: Paul Berry <paulberry@google.com>
Commit-Queue: Konstantin Shcheglov <scheglov@google.com>
Changes the analyzer and related packages so that when they refer to
diagnostic constants, they do so via the import prefix `diag`, which
refers to the appropriate `diagnostic.dart` file containing the top
level diagnostic constant declarations, rather than the static
declarations inside `DiagnosticCode`-derived classes (which will soon
be removed).
This CL was created by the following steps:
- Run the script
`pkg/analyzer_utilities/tool/messages/switch_to_toplevel_diagnostics.dart`.
- Execute `dart fix --apply --code=unused_import,unnecessary_import`
on the following directories (this removes imports that are no
longer necessary due to the change):
- `pkg/analysis_server`
- `pkg/analyzer`
- `pkg/linter`
- `pkg/analysis_server_plugin`
- `pkg/analyzer_plugin`
- `pkg/analyzer_testing`
- `pkg/front_end`
- `pkg/analyzer_cli`
- Execute `dart format` on the following files and directories:
- `pkg/analysis_server`
- `pkg/analyzer`
- `pkg/linter`
- `pkg/analysis_server_plugin`
- `pkg/analyzer_plugin`
- `pkg/analyzer_testing`
- `pkg/front_end/test/scanner_test.dart`
(Note that `pkg/front_end` and `pkg/analyzer_cli` are not
re-formatted as whole directories because they contain `.dart` files
that are test cases rather than source code, and reformatting those
files might change test expectations.)
- Manually add `diag` to
pkg/front_end/test/spell_checking_list_tests.txt.
- Manually fix the ignore comment in
`pkg/analyzer_testing/lib/src/analysis_rule/pub_package_resolution.dart`. (The
script `switch_to_toplevel_diagnostics.dart` automatically adds it
after `import 'package:analyzer/src/diagnostic/diagnostic.dart' as
diag;`, but then executing `dart format` bumps the ignore comment to
the following line, where it has no effect.)
Change-Id: I6a6a69643022aab2b5a6224fb4124eead243260d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/461521
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
Reviewed-by: Brian Wilkerson <brianwilkerson@google.com>
This also includes some small refactorings for parameter names to start using the diagnostic wording instead of error. As well as a small change to `contextMessage` and related to use a list of `Pattern`s instead of a single `String` for better matching.
In future CLs, the TODOs added in this one will be addressed so we can fully migrate this. Added them because of the size of the CL.
Bug: https://github.com/dart-lang/sdk/issues/61557
Change-Id: I5557668a1baed98faff977255c9f22e0d9d96008
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/452180
Commit-Queue: Paul Berry <paulberry@google.com>
Auto-Submit: Felipe Morschel <git@fmorschel.dev>
Reviewed-by: Paul Berry <paulberry@google.com>
Reviewed-by: Brian Wilkerson <brianwilkerson@google.com>
Removes the analyzer error code `ScannerErrorCode.expectedToken` and
replaces usages of it with `ParserErrorCode.expectedToken`. This
avoids an ambiguity between the two messages.
Avoiding ambiguities like these is important, because in many cases
the user only sees the diagnostic name; they don't see the class it's
in. For example, `ignore:` comments just give the diagnostic name, and
the web page https://dart.dev/tools/diagnostics only shows diagnostic
names.
In the future I intend to add an error check to the analyzer
diagnostic code generator, to ensure that there are no ambiguities
like these. This CL is a prerequisite for adding the error check.
Change-Id: I6a6a696491a41a9d220a4502bfa9b96410580a19
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/455561
Reviewed-by: Brian Wilkerson <brianwilkerson@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
Reviewed-by: Johnni Winther <johnniwinther@google.com>
This change was generated by the following process:
- The script `pkg/analyzer/tool/messages/rename_error_constants.dart`
was run. This generated the vast majority of the diffs.
- Then all modified files were reformatted using
`tools/sdk/dart-sdk/bin/dart/format`.
- Finally, the script `pkg/analyzer/tool/messages/generate.dart` was
run, to rebuild generated code.
Change-Id: I6a6a69644ed8740ad6269d98cb169076151824ed
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/444921
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Reviewed-by: Samuel Rawlins <srawlins@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
Sometimes when the parser is performing error recovery it skips tokens
until it finds the next `,` or `;`, to try to get back on track. This
is handled by the `Parser.findNextCommaOrSemicolon` method.
When this method encounters a `BeginToken` (i.e., `(`, `[`, `{`, or
`<`), it needs to skip to the matching end token (using the `endToken`
getter), so that it doesn't try to resume parsing inside some nested
structure.
However, sometimes there is no matching end token (either because the
user has failed to properly matched `()`, `[]`, or `{}`, or because
the `BeginToken` is `<`, which doesn't always have a matching `>`). To
avoid a crash when this happens, `Parser.findNextCommaOrSemicolon`
needs to treat an unmatched `BeginToken` like an ordinary token, and
just advance to the next token.
Fixes https://github.com/dart-lang/sdk/issues/54236.
Bug: https://github.com/dart-lang/sdk/issues/54236
Change-Id: Id208c7a46c9c00b69f7f460a638d59486ebaffea
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/339980
Reviewed-by: Jens Johansen <jensj@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
The analyzer/CFE shared error template framework requires name
substitutions to be non-empty. However, sometimes, due to parser error
recovery, an identifier token can be an empty string.
One of the circumstances in which this occurs is when parsing text
like the following:
final b = (final g, final ) = 55;
In this example, there is a parse error due to the fact that the user
is trying to declare variables inside a pattern assignment. But one of
those variables doesn't have a name yet, so the parser has created a
synthetic token for it whose name is an empty string.
To avoid a crash, we must supply some name to the error message
template, so the name we supply is `(unnamed)`.
Fixes https://github.com/dart-lang/sdk/issues/54178.
Bug: https://github.com/dart-lang/sdk/issues/54178
Change-Id: Iabf3263ee1f56a89d1a69bcd74a17a296d470b9c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/339661
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
Two of these tests (`recordPattern_nullable_beforeAs` and
`recordPattern_nullable_beforeWhen`) verify that the fix for #52439
(Destructuring with explicit type of nullable record is a parsing
error), which causes `(...)? identifier` to be recognized as a
variable pattern, doesn't get confused by the pseudo-identifiers `as`
and `when`, and so it continues to correctly parse `(...)? as ...` as a
cast pattern and `(...)? when ...` as a guarded pattern.
The other two tests (`recordPattern_nonNullable_beforeAs` and
`recordPattern_nonNullable_beforeWhen`) verify that `(...) as ...` and
`(...) when ...` are parsed correctly. These forms were never broken,
but they were not well tested either.
Bug: https://github.com/dart-lang/sdk/issues/52439
Change-Id: I866e1c7a6a8e47e0cd91a1a77654405f395b50da
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/305844
Reviewed-by: Jens Johansen <jensj@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
Previously, if a record-typed variable pattern lacked a `var` or
`final` keyword, and was followed by one of the tokens `||`, `&&`,
`as`, `?`, `!`, `when`, or `=>`, the parser would fail to recognize
it. This happened because the call to `computeVariablePatternType`
wasn't passing `true` for the optional parameter `required`; that in
turn placed `computeType` in a mode where it believed it was parsing a
potentially ambiguous construct in a top-level declaration, and hence
it would only accept the record type if the variable name was followed
by something that looked like part of a correct declaration (e.g. a
comma).
The fix is to pass `true` for the optional parameter `required` of
`computeVariablePatternType`. This places `computeType` in a mode
where it accepts the record type regardless of what follows it. This
is correct behavior since at the point where a variable pattern is
being parsed, ambiguities have already been taken care of and the
construct being parsed is most definitely a variable pattern.
Fixes#52521.
Change-Id: If20772ad914827a29df45c27eefb382ec1f470d2
Bug: https://github.com/dart-lang/sdk/issues/52521
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/305848
Reviewed-by: Jens Johansen <jensj@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
In rare circumstances involving syntax errors, the `parsePattern`
method inserts a synthetic token but does not consume any
tokens. Usually when this happens it's not a problem, because whatever
method is calling `parsePattern` consumes some tokens, so the parser
always makes progress. However, when parsing list patterns, after
calling `parsePattern`, the parser would look for a `,`, and if it
didn't find one, it would supply a synthetic `,` and call
`parsePattern` again, resulting in an infinite loop. A similar
situation happened with map patterns, though the situation was more
complex because in between the calls to `parsePattern`, the parser
would also create synthetic key expressions and `:`s.
To fix the problem, when parsing a list or map pattern, after the call
to `parsePattern`, the parser checks whether any tokens were
consumed. If no tokens were consumed, it ignores the next token from
the input stream in order to make progress.
I also investigated whether there were similar issues with
parenthesized/record patterns and switch expressions, since those
constructs also consist of a sequence of patterns separated by tokens
and other things that could in principle be supplied
synthetically. Fortunately, parser recovery doesn't get into an
infinite loop in those cases, so I didn't make any further
changes. But I did include test cases to make sure.
Fixes#52352.
Bug: https://github.com/dart-lang/sdk/issues/52352
Change-Id: Idc8140236f6054deb1fd3c862036fe47dd84f30b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/302803
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
In https://dart-review.googlesource.com/c/sdk/+/299400, the parser was
adjusted so that it no longer accepts `when` and `as` as the names for
variable patterns in cases where there is a possible ambiguity
(e.g. `int when` is not accepted as a pattern because `int` is a
legitimate pattern, therefore `when` could introduce a guard
clause). This change further prohibits `when` and `as` from being the
names of variable patterns or identifier patterns even in the case
where there is no ambiguity. This is in line with the discussion at
https://github.com/dart-lang/sdk/issues/52199#issuecomment-1526297771,
and the spec change at
https://github.com/dart-lang/language/pull/3033.
Three new error codes are introduced, to cover the three circumstances
in which `when` or `as` might be used illegally: in a declared
variable pattern, in an assigned variable pattern, or in an identifier
pattern. I've also added analyzer tests to ensure that the parser
recovers from these errors nicely. Unfortunately, nice error recovery
is only feasible in the non-ambiguous cases.
I've also updated the language test expectations in
`tests/language/patterns/version_2_32_changes_error_test.dart` to
reflect the new error messages, and added a few more examples of uses
of `when` and `as` that are still permitted.
Fixes#52260.
Bug: https://github.com/dart-lang/sdk/issues/52260
Change-Id: I229f627aa639659c30b83c74895759207da279f7
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/301482
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
This change fixes parsing of case clauses such as:
case foo when !flag:
Constructions like this require some lookahead in order to parse
correctly, because the token `when` is valid both as an identifier and
as a part of the grammar for a case clause. Therefore, at the time
`foo` is encountered, the parser must decide whether it is looking at
a variable pattern (`foo when`, where `when` is the name of the
variable) or an identifier pattern (`foo`, where `when` begins the
case's guard clause). Previous to this fix, the algorithm for
disambiguating these two choices was as follows:
- If the token sequence starting at `foo` looked like a type, and the
token that follows was an identifier, the parser assumed it was
looking at a variable pattern with a type; otherwise it assumed it
was looking at an identifier pattern.
- EXCEPT that if the token that followed the supposed type was `when`
or `as` (both of which are valid identifiers), then it probed
further:
- If the token that followed `when` or `as` was a token that could
legitimately follow a pattern, then it assumed that it was looking
at a variable pattern with a type. (The tokens that could
legitimately follow a pattern are `,`, `:`, `||`, `&&`, `)`, `}`,
`]`, `as`, `when`, `?`, `!`).
- Otherwise it assumed that it was looking at an identifier pattern.
This didn't fully disambiguate, because the third bullet didn't
account for the fact that the tokens `as`, `when`, and `!` could
_either_ legitimately follow a pattern _or_ legitimately begin an
expression (or, in the case of `when`, a type), therefore constructs
like the following were incorrectly parsed:
- `case foo when as:` (where `as` is a local boolean variable)
- `case foo when when:` (where `when` is a local boolean variable)
- `case foo when !flag:` (where `flag` is a local boolean variable)
- `case foo as when:` (where `when` is the name of a type)
The solution is to simplify the disambiguation logic so that if if the
token that follows the supposed type is `when` or `as`, then the
parser assumes that it's looking at an identifier pattern, _not_ a
typed variable pattern.
The consequence of this is that the above four constructions are
parsed correctly; however it is no longer possible for a typed
variable pattern to name a variable `when` or `as`.
For consistency we would like to prohibit _any_ variable pattern from
naming a variable `when` or `as`, however to keep this change as small
as possible (and reduce the risk involved in a possible cherry-pick)
that will be postponed until a later CL.
Fixes#52199.
Bug: https://github.com/dart-lang/sdk/issues/52199
Change-Id: Ibab9b92f01e3e4020d7d64f1ff000a9b964a4564
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/299400
Commit-Queue: Paul Berry <paulberry@google.com>
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
Previously, if a pattern variable declaration appeared outside of a
function or method, the parser would get very confused. Now it
recognizes the situation, issues a comprehensible error message, and
replaces the pattern with a synthetic identifier token so that the
analyzer AST is somewhat sensible.
Fixes#51322.
Bug: https://github.com/dart-lang/sdk/issues/51322
Change-Id: Ica5f060cb483a39c4e50942d639939b1fab40bea
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/293087
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
The type name in an object pattern is a `typeIdentifier`; in the spec
grammar, `typeIdentifier` includes `OTHER_IDENTIFIER`, which is defined as:
OTHER_IDENTIFIER ::=
`async` | `hide` | `of` | `on` | `show` | `sync` | `await` | `yield`
This adds tests to verify that all these identifiers are accepted as
type names in an object pattern.
Bug: https://github.com/dart-lang/sdk/issues/50035
Change-Id: I6b9d752e1b34f7bc93dedc4304f695e7cb42df48
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/292542
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
The precedence-based pattern parser can understand a unary pattern
inside another unary pattern (e.g. `_ as int as num`) or a relational
pattern inside a unary pattern (e.g. `> 1?`), but the specification
prohibits these constructions because they're difficult to read and
not very useful.
This change updates the implementation to match the spec, by producing
the appropriate error. The offset and length of the error cover the
inner pattern, so it should be easy to construct an analysis server
quick fix that inserts the necessary parentheses.
Bug: https://github.com/dart-lang/sdk/issues/50035
Change-Id: I33e74d6d1f863e7162851d26fefbacd4fd17277c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/292120
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
The logic for parsing types has special disambiguation rules for
deciding whether a trailing `?` should be included in the type, based
on what token(s) follow the `?`. In the case where the token that
follows the `?` is `when`, we need to look further ahead to
disambiguate, to distinguish `PATTERN as T? when guard` from something
like `EXPRESSION is T ? when : otherwise`.
(Note: an alternative implementation would be to disambiguate based on
whether we're parsing a pattern or an expression. But in the future I
want to move toward an architecture where expression parsing and
pattern parsing are combined, so that if the parser makes the wrong
decision about whether it's looking at a pattern or an expression,
error recovery will do a better job. So I'm disambiguating based
solely on what follows the `?`.)
Bug: https://github.com/dart-lang/sdk/issues/50035
Change-Id: Idbc780b7b54fecc7fd01cae868c34771564dd804
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/292282
Commit-Queue: Paul Berry <paulberry@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
The listener API for variable patterns is split into three separate
functions, to handle the three separate behaviors:
- `handleAssignedVariablePattern` for variable names appearing in an
assignment context (these assign to an existing variable upon a
successful match).
- `handleDeclaredVariablePattern` for variable declarations appearing
in a declaration or matching context (these cause a new variable
name to come into scope).
- `handleWildcardPattern` for wildcards in any context (these don't
capture the matched value).
Also, responsibility is shifted to the parser for reporting the
following error conditions:
- VariablePatternKeywordInDeclarationContext (e.g.
`var (var x) = ...;`)
- PatternAssignmentDeclaresVariable (e.g. `[x, var y] = ...;`)
Previously these errors were detected by the implementations, and
weren't fully covering all possible error scenarios.
In the case of VariablePatternKeywordInDeclarationContext, the
listener method `handleDeclaredVariablePattern` is called instead of
`handleAssignedVariablePattern`. This ensures that no tokens are
dropped from the analyzer AST. The CFE uses the `inAssignmentPattern`
argument of `handleDeclaredVariablePattern` to distinguish this error
recovery case from a legitimate declared variable pattern.
Fixes#51868.
Bug: https://github.com/dart-lang/sdk/issues/51868
Change-Id: I28ec679b73d64033166721c6460be35f15e23171
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/291583
Reviewed-by: Jens Johansen <jensj@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
Expressions can also occur in patterns, either after an equality or
relational operator (e.g. `== e`), or after `const` (though what's
allowed after `const` is heavily restricted). We need to make sure
that the expression parser doesn't greedily treat `=>` as introducing
a function expression when parsing these constructs inside a switch
expression.
But it's ok to allow function expressions inside list patterns, map
patterns, parenthesized patterns, and in the argument part of object
patterns. (These will be rejected by a later stage of analysis
because expressions inside of patterns must be const, and a function
expression can't be const. But the parser should still accept them so
that we can give useful error messages).
Fixes#50591.
Bug: https://github.com/dart-lang/sdk/issues/50591
Change-Id: I828555782f5bc8cb8aae8a3948849b7a75bdec57
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/273286
Reviewed-by: Jens Johansen <jensj@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
The parser now handles maps that contain rest patterns with
subpatterns (e.g. `{'foo': _, ...var rest}`). Even though this is
invalid Dart, it still makes sense for the parser to handle it so that
we can generate higher quality error messages.
The shared type analyzer now generates an error,
`restPatternWithSubPatternInMap`, if it encounters this error
condition.
Change-Id: I7f447bde28e646593aa432e3e5ad1f5e415bdd30
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/273283
Commit-Queue: Paul Berry <paulberry@google.com>
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
Konstantin Shcheglov