This CL revises the scanner API so that tokens are always prepended
to the beginning of the token stream. This allows the parser handling
of error tokens to be simplified. Any clients using the scanner directly
rather than through the scanner API should call scannerRecovery
if the scanner has detected errors in the content.
Change-Id: I32510da10205bd964f80898a238489d1508733e6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/102680
Reviewed-by: Brian Wilkerson <brianwilkerson@google.com>
Commit-Queue: Dan Rubel <danrubel@google.com>
This improves recovery when parsing conditional import expressions
by repositioning the synthetic closing ')' that was inserted by the
scanner in a less than optimal location.
This approach relies on the beforeSynthetic field being set correctly.
While the scanner still sets the 'next' and 'beforeSynthetic' fields
directly for efficiency, most other functions which update an existing
token stream have been revised to call 'setNext()' rather than setting
the 'next' field directly. This ensures that the 'beforeSynthetic' field
will be correctly updated.
Change-Id: Id631fd600c64d1feaf00593acb74a7070e354f07
Reviewed-on: https://dart-review.googlesource.com/52120
Reviewed-by: Brian Wilkerson <brianwilkerson@google.com>
Commit-Queue: Dan Rubel <danrubel@google.com>
This includes Fasta, tools and observatory, so the checked-in SDK must
have the lower-case constants.
Change-Id: I8380ad041ad058f7d02ae19caccfecd434d13d75
Reviewed-on: https://dart-review.googlesource.com/50201
Commit-Queue: Lasse R.H. Nielsen <lrn@google.com>
Reviewed-by: Leaf Petersen <leafp@google.com>
This updates the fasta scanner to translate the remaining
previously untranslated error codes so that they will be seen
by the analyzer when the fasta scanner is used by analyzer.
R=ahe@google.com
Review-Url: https://codereview.chromium.org/2923113002 .
* improve fasta interpolation recovery by adding closing brace
for unterminated string
* update analyzer scanner test because fasta does not support
does not support \r line endings
R=scheglov@google.com
Review-Url: https://codereview.chromium.org/2910583002 .
* move isPseudo getter out of Token into TokenType
* move special case charCount logic out of fasta.Token into fasta.ErrorToken
* move more accessors from fasta.Token into analyzer.Token
* add @overrides to denote which methods have been declared in analyzer.Token
* make fasta LazySubstring and its subclasses private
* rename fasta.previousToken to previous to match analyzer.Token
* flatten fasta.Token.preceedingCommentTokens into preceedingComments
* normalize fasta.Token.next and analyzer.Token.next
R=scheglov@google.com
Review-Url: https://codereview.chromium.org/2872433005 .
* add new SyntheticSymbolToken class
* whenever fasta inserts a closer ( '}', ']', ')' ) that is not part
of the original source, ensure that the inserted closer is
synthetic (isSynthetic returns true) and in the token stream
* make EOF a synthetic token
R=ahe@google.com, scheglov@google.com
Review-Url: https://codereview.chromium.org/2777153002 .
* cleanup defaultRecoveryStrategy next/previous and eof
* add SymbolToken.eof() constructor to ensure that EOF tokens
are correctly self referencing
* add fasta.scanString method to match scan method
* add fasta.ScannerResult.hasErrors field
so that scan and scanString callers can quickly check for errors
* new scanner replacement test
which scans using fasta then asserts as if it were from analyzer
* improve SymbolToken.toString() for EOF
* update analyzer test :: fasta scanner considers "<" to be an opener
* extract and move translateErrorToken into front_end
* address comments in https://codereview.chromium.org/2763833002R=ahe@google.com, paulberry@google.com
Review-Url: https://codereview.chromium.org/2767083002 .
Johnni Winther