Context:
A while back, I inserted a synthetic token in the parser to allow us to parse without crashing the analyzer while I worked on the CFE. Now that both implementations are up and working, we can remove this synthetic token and allow the parser to parse as if we had enabled dot shorthands by default, and produce an error if the experiment isn't enabled.
This change allows a bunch of different language tests to start passing since they were blocked on the weird synthetic token messing up the parsing stream.
Bug: https://github.com/dart-lang/sdk/issues/59758
Change-Id: I792e3b917a76241b04ee708de06f670bbde64036
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/423563
Reviewed-by: Chloe Stefantsova <cstefantsova@google.com>
Commit-Queue: Kallen Tu <kallentu@google.com>
*TL;DR*
This improves scanner recovery for a missing `}` in certain situations,
reducing the risk of an in-body change causing a (temporary) outline
change (which in turn could result in the analyzer becoming unresponsive
for "no reason").
*Details*
The behavior of IntelliJ is that when typing `{` it only inserts a
matching end brace `}` when hitting enter.
Imagine you are typing an if: `if (1 + 1 == 2) {`, where you don't hit
enter quickly enough and you trigger a re-analysis at this point.
What happens then is that every method below where you are typing looks
to be local function declarations and thus the outline change. When the
outline change the analyzer has to do a lot of work: everything
(transitively) depending on the file has to be recompiled, and every
strongly connected component is compiled "in one go" where the analyzer
can't respond to queries. So if you have one or more large strongly
connected components depending on the file, or the file itself is part
of such a chain, you will (or at least might) experience that the
analyzer is slow to respond, and it will be extra puzzling because
logically you're just doing an in-body change.
For some code the user might not even naturally hit enter, e.g. `var foo
= {"I'm", "a", "set"};`.
The recovery in the scanner has always been that - upon reaching the end
of the file - it sees that we're missing a `}` and it inserts it at the
end. This CL instead tries to figure out a better place to insert it,
and if successful, will rerun the scanner, instructing it to insert it
at the better place and (hopefully) avoiding a subsequent outline
change.
It does this by looking at the indentation - which is new for recovery -
and under the assumption that the indentation was correct before, will
find the position where the start curly brace was inserted. Note that if
it finds a position it will always be between the start curly brace (the
one missing the end curly brace) and the end of file, and inserting the
missing curly end brace there can't really be "more wrong" than
inserting it at the end (if the new place is not correct it's just
"still wrong").
In the benchmark added we see how quickly we can get completion after
having typed `if (1+1==2) {`, then adding `\n ge\n}` and requesting
completion on the `ge` part, i.e. a simulation of typing
```
if (1+1==2) {
ge
}
```
and asking for completion at the `ge`.
The change in this CL - on cycles of size 1024 - caused the time to
completion response to come in between ~5 times faster (going from ~10.2
to ~2.1 seconds) to ~18 times faster (going from ~10.3 seconds to ~0.56
seconds):
`CodeType.ImportExportCycle` goes from:
```
+------+-----------+------------+
| Size | Initial | Completion |
+------+-----------+------------+
| 16 | 2.019581 | 0.97504 |
| 32 | 3.028976 | 1.031008 |
| 64 | 4.422884 | 1.198383 |
| 128 | 7.612125 | 1.597091 |
| 256 | 12.860864 | 2.906553 |
| 512 | 24.391894 | 5.017093 |
| 1024 | 48.390993 | 10.243085 |
+------+-----------+------------+
```
to
```
+------+-----------+------------+
| Size | Initial | Completion |
+------+-----------+------------+
| 16 | 2.107213 | 0.661066 |
| 32 | 3.012952 | 0.70554 |
| 64 | 4.682508 | 0.731176 |
| 128 | 7.508434 | 0.745501 |
| 256 | 13.105477 | 0.852413 |
| 512 | 24.520184 | 1.278403 |
| 1024 | 48.804348 | 2.11903 |
+------+-----------+------------+
```
and `CodeType.ImportExportChain` goes from:
```
+------+-----------+------------+
| Size | Initial | Completion |
+------+-----------+------------+
| 16 | 2.059196 | 0.892082 |
| 32 | 3.080717 | 0.93232 |
| 64 | 4.647163 | 1.240303 |
| 128 | 7.377035 | 1.674859 |
| 256 | 12.939432 | 2.705483 |
| 512 | 24.529501 | 5.02689 |
| 1024 | 47.713553 | 10.385469 |
+------+-----------+------------+
```
to
```
+------+-----------+------------+
| Size | Initial | Completion |
+------+-----------+------------+
| 16 | 2.020809 | 0.709643 |
| 32 | 3.106856 | 0.648818 |
| 64 | 4.503067 | 0.593152 |
| 128 | 7.45692 | 0.622423 |
| 256 | 13.140592 | 0.606948 |
| 512 | 24.933216 | 0.612687 |
| 1024 | 50.167541 | 0.567544 |
+------+-----------+------------+
```
Change-Id: I8dbefe215162d00a209206ae3db83b2b17505853
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/415581
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Brian Wilkerson <brianwilkerson@google.com>
Reviewed-by: Phil Quitslund <pquitslund@google.com>
The parser was still WIP and was emitting an extra error, so this CL removes that error so tests can start passing and so I can properly test the new implementation e2e.
It still needs some fixes after the analyzer implementation is complete. The current extra synthetic tokens added in the parser are temporary to prevent crashes on the analyzer side when the experiment flag is turned off. But it works fine for now, and I'll clean it up in a CL when the feature is complete.
Bug: https://github.com/dart-lang/sdk/issues/59758
Change-Id: I83d8c840f66bedff4aa27a737a3f57e77d3973b8
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/412344
Reviewed-by: Paul Berry <paulberry@google.com>
Reviewed-by: Chloe Stefantsova <cstefantsova@google.com>
Commit-Queue: Kallen Tu <kallentu@google.com>
Before, when parsing a new style typedef without an identifier (e.g.
`typedef = whatnot`) it concluded that the token after the equal-sign
wasn't an equal sign an that it thus had to be an old style typedef,
followed by recovery there.
This CL makes the recovery insert an identifier after `typedef` (what
the recovery of the old style actually did too), and then parse it as a
new style typedef.
This causes the previous many errors to just be `Expected an identifier,
but got '='.`
It is furthermore verified that the parsing of such a case is ~the same
as when having an indentifier.
Fixes https://github.com/dart-lang/sdk/issues/56912
Change-Id: I5cde1f29839555b1d6027a7d040dc6f60ac614a5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/392560
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Reviewed-by: Brian Wilkerson <brianwilkerson@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
The entire scanner/parser now no longer uses `optional` or `optional2`,
the latter being completely removed.
Furthermore, when compiling a fixed version of the CFE with the CFE
this CL reduces the instructions used by ~13.6 mio.
```
page-faults:u: -0.3695% +/- 0.0676% (-384.20 +/- 70.23)
instructions:u: -0.0645% +/- 0.0007% (-13671097.80 +/- 153289.78)
```
Change-Id: Ib93dd2983aa1b4df0610d099010ac10f55f357f8
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/388323
Reviewed-by: Brian Wilkerson <brianwilkerson@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Johnni Winther <johnniwinther@google.com>
This adds the enclosingDeclarationName to the beginMethod of the
parser listener.
This enables the removal BuilderFactory.currentTypeParameterScopeBuilder.
Change-Id: Ie2bec9432c20b8bdbd62a14e8a65c272179d4698
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/383182
Reviewed-by: Chloe Stefantsova <cstefantsova@google.com>
Reviewed-by: Paul Berry <paulberry@google.com>
This changes the BodyBuilder to create FunctionTypeParameter instead
of FormalParameterBuilder for "parameters" in function types. This
avoids the creation of unnecessary VariableDeclaration nodes
function types.
The types of these VariableDeclarations where created before the type
variable scope was completed, introducing an artificial dependency on
the pending nullability computation.
Change-Id: Ie1203fa4c78a27f3e7a0dfad16725cbbde24a6a1
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/381143
Reviewed-by: Chloe Stefantsova <cstefantsova@google.com>
Reviewed-by: Paul Berry <paulberry@google.com>
When the parser sends events to the listener some events has a pair of
parameters, `beginToken` and `endToken`. Most of these are constructed
in such a way that the `beginToken` is the first token and the
`endToken` is the last token in that construct. This is for instance the
case with `endClassDeclaration`.
It was, however, not the case for `endMetadata` where `endToken` instead
was the next token *not* in the metadata.
In this CL I've found a changed the following to point to the last token
in the construct instead of the next token not in the construct and
renamed the parameter where it made sense:
* `endAssert` --- and renamed `semicolonToken` (which only pointed to
a semicolon for statements) to `endToken`.
* `endAwaitExpression`
* `endConstLiteral` --- and renamed `token` to `endToken`
* `endConstructorReference`
* `endFieldInitializer` --- and renamed `token` to `endToken`
* `endForIn`
* `endForInBody` --- and renamed `token` to `endToken`
* `endForStatement`
* `endForStatementBody` --- and renamed `token` to `endToken`
* `endFunctionExpression` --- and renamed `token` to `endToken`
* `endInitializer` --- and renamed `token` to `endToken`
* `endInitializers`
* `endInvalidAwaitExpression`
* `endMetadata`
* `endSwitchCase`
* `endTopLevelDeclaration` --- and renamed `nextToken` to `endToken`
* `endWhileStatement`
* `endWhileStatementBody` --- and renamed `token` to `endToken`
* `handleNoConstructorReferenceContinuationAfterTypeArguments`
In the few places in listeners where these values were used I've mostly
updated to do e.g. `endToken.text!` to retain the current behavior.
Change-Id: I25495e160d1eec5c75bcf1313b512cd04bcb1533
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/364322
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Reviewed-by: Brian Wilkerson <brianwilkerson@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
Sometimes when the parser is performing error recovery it skips tokens
until it finds the next `,` or `;`, to try to get back on track. This
is handled by the `Parser.findNextCommaOrSemicolon` method.
When this method encounters a `BeginToken` (i.e., `(`, `[`, `{`, or
`<`), it needs to skip to the matching end token (using the `endToken`
getter), so that it doesn't try to resume parsing inside some nested
structure.
However, sometimes there is no matching end token (either because the
user has failed to properly matched `()`, `[]`, or `{}`, or because
the `BeginToken` is `<`, which doesn't always have a matching `>`). To
avoid a crash when this happens, `Parser.findNextCommaOrSemicolon`
needs to treat an unmatched `BeginToken` like an ordinary token, and
just advance to the next token.
Fixes https://github.com/dart-lang/sdk/issues/54236.
Bug: https://github.com/dart-lang/sdk/issues/54236
Change-Id: Id208c7a46c9c00b69f7f460a638d59486ebaffea
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/339980
Reviewed-by: Jens Johansen <jensj@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
Includes a cleanup of the errors reported for duplicate instance field.
Previously, in addition to reporting the duplicate, we would report that
the field cannot be used because it is a duplicate and that the
duplicate field hasn't been initialized.
Closes#53467Closes#53320
Change-Id: Ifd94223500809f3be36a1345d9d0c409778eed8a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/336885
Reviewed-by: Paul Berry <paulberry@google.com>
Reviewed-by: Samuel Rawlins <srawlins@google.com>
Commit-Queue: Johnni Winther <johnniwinther@google.com>
Reviewed-by: Chloe Stefantsova <cstefantsova@google.com>
This passes the first token of these top level declaration directly
to the endX listener method.
In the CFE this is used to ensure that can handle the new modifiers in
the textual outline. Furthermore, support for extension types is added
and having an "unknown chunk" now results in an error. The latter should
help us keep the textual outline up-to-date wrt new features.
Change-Id: I813d6162b6cba0a2bf550ed33a6091abf9bf49f8
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/324702
Commit-Queue: Johnni Winther <johnniwinther@google.com>
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
This changes the parser to always interpret `extension type` as the
start of an extension type declaration.
Recovery is added to handle missing identifier, primary constructor,
parameter constructor parameters, and extension type declaration body.
Change-Id: I98cac0b2641167c4708fa20b22f0de0a70336457
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/321704
Reviewed-by: Brian Wilkerson <brianwilkerson@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
Commit-Queue: Johnni Winther <johnniwinther@google.com>
This is not the flag you're looking for.
The 'extension-types' flag was used for an early experiment that is
not directly related to the Extension Types feature currently being
developed. The current feature uses the 'inline-class' flag.
Change-Id: Icbb6c3828c41e743e726161b17da4c7784a2c677
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/316380
Reviewed-by: William Hesse <whesse@google.com>
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Reviewed-by: Chloe Stefantsova <cstefantsova@google.com>
Commit-Queue: Johnni Winther <johnniwinther@google.com>
This adds parser support for the optional 'const' keyword in extension
type declaration. The token is passed on through the endPrimaryConstructor
listener call since it is used to make the primary constructor constant.
Change-Id: I518a8e397fb62272002c424e7b69affe6123f4af
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/316221
Commit-Queue: Johnni Winther <johnniwinther@google.com>
Reviewed-by: Chloe Stefantsova <cstefantsova@google.com>
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
This changes the parser to call endInvalidYieldStatement for
yield statement both in sync and async methods. This ensures that
the CFE creates an invalid expression for yield in async methods,
making the generated AST verifiable.
Change-Id: I1bfe922878fcf3dc19c823cb89ee869af3dd2ce3
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/309200
Commit-Queue: Johnni Winther <johnniwinther@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
Kallen Tu