Previously we had several places where different uriToSource was used.
This was both weird and led to errors when something refering to
some state was used to look up in another state.
This CL makes it so we only have one source of truth.
To not include sdk sources when serializing a component that only mixes
something from the sdk in, (or extends it or...), the serialization is
changed slightly to keep track of which uris come from actual
implementation. Before the sdk sources was explicitly removed in the
incremental compiler, but we want to limit those kinds of things,
which is why we're doing it differently here.
Fixes#35215.
Bug: 35215
Change-Id: Iaa5618fcb0ea42b13aba7720f34a87a85144e047
Reviewed-on: https://dart-review.googlesource.com/c/85175
Reviewed-by: Peter von der Ahé <ahe@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
For all other "debugPath.add"s we've wrapped it in assert to avoid
doing it in the general case.
One apparently have slipped through.
Change-Id: Ib0dfc4a671428993e393d57a75db6612785e37b4
Reviewed-on: https://dart-review.googlesource.com/c/85161
Reviewed-by: Dmitry Stefantsov <dmitryas@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
Before this CL we could get a dill file out of compiling
tests/language_2/import_nonexisting_dart_uri_test.dart
but we couldn't load it back in again.
Change-Id: I05b6b8d6368c7ed03dbb798535ea03e0493df780
Reviewed-on: https://dart-review.googlesource.com/c/79220
Reviewed-by: Kevin Millikin <kmillikin@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
Previously, we only stored a reference to the originating Typedef declaration
and is thus loosing the originating type arguments. We need these type arguments
in various places, including diagnostics and bounds checks.
Note: this CL maintains backwards compatibility wrt to the Dart API of
FunctionType. I'll update the API in a follow-up CL.
Change-Id: I896ce3c3b4522d542a82c5036f8a17ee098143dc
Reviewed-on: https://dart-review.googlesource.com/c/81367
Commit-Queue: Peter von der Ahé <ahe@google.com>
Auto-Submit: Peter von der Ahé <ahe@google.com>
Reviewed-by: Kevin Millikin <kmillikin@google.com>
* Ensure that we have a .dill file and it has the correct version before
we start decoding it in dump.dart.
* Throw errors, not strings. Ensure that they have a useful
toString().
* Do not print usage except when the command is invoked
incorrectly (wrong number of arguments). Once we get into the
command it's less likely that it's been invoked incorrectly and more
likely that something else has gone wrong.
Because these utilities are invoked from other scripts (like the
fasta command), printing their usage for problems other than
invoking them correctly doesn't match the way that they were invoked
and it's confusing.
Change-Id: I7832383594d2b3719a0a7a7392ba4685717a79d2
Reviewed-on: https://dart-review.googlesource.com/c/78206
Commit-Queue: Kevin Millikin <kmillikin@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
The canonicalization of symbols needs to take library privacy into
account. Since the Symbol class itself only has a [_name] field but does
not reference in which library the symbol came from, the [_name] must be
mangled.
Mangling is done by backends and so we make a new [SymbolConstant] which
the backends can desugar by doing appropriate mangling and construction
of a [Symbol] instance.
Fixes https://github.com/dart-lang/sdk/issues/34396
Change-Id: I2e13288483c35f875d39eefdb73677b2cc03527a
Reviewed-on: https://dart-review.googlesource.com/74360
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
The canonicalization of symbols needs to take library privacy into
account. Since the Symbol class itself only has a [_name] field but does
not reference in which library the symbol came from, the [_name] must be
mangled.
Mangling is done by backends and so we make a new [SymbolConstant] which
the backends can desugar by doing appropriate mangling and construction
of a [Symbol] instance.
Fixes https://github.com/dart-lang/sdk/issues/34396
Change-Id: I5ddb5331ce79a0b942807929d4b8f1050a9899e7
Reviewed-on: https://dart-review.googlesource.com/73883
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Use positions already available in for instance AsExpressions,
also adds new offsets in the kernel format to
SuperInitializer and RedirectingInitializer.
Bug: #33216
Change-Id: I542967ddc6ec782e6513d62fce038a49239e1622
Reviewed-on: https://dart-review.googlesource.com/63883
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Previously, if we didn't include the source code, we wrote the url as
null. This for instance made it impossible to step through mixed in code
(at least when mixed in from the sdk).
This CL includes all used urls. If there's no source, the source is empty,
but the VM then tries to find the proper source to be able to display it
(e.g. the VM already has the sdk source).
This CL further more adds a service test that tests that we can actually
step into mixin in code from the sdk.
Change-Id: Ied9569723e23928769ebc980410aed60be6eaa22
Reviewed-on: https://dart-review.googlesource.com/51621
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Prior to this change we only had fileOffset which points to the name.
This CL adds a field - startFileOffset - that points to the start
(e.g. 'static', 'factory', the return type or any presiding annotation
(including '@')), as well as fixing `Function::GetSource` in the VM to
fix a crash when using mirrors to ask for the source of a function or
constructor.
The field is added to procedures, constructors and classes.
The latter changes the location of classes with annotations to be
the position of the annotation. This is on-par with --no-preview-dart-2.
The change in regards to procedures and constructors mean, that both
the location and the source (from .location and .source on a
MethodMirror) will change to include any annotations. This is a
different behavior than --no-preview-dart-2.
Closes#33271.
Change-Id: I90f1232c5ec2d01e60e0bab070d44c37232b2730
Reviewed-on: https://dart-review.googlesource.com/60560
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Kevin Millikin <kmillikin@google.com>
This CL adds an option to the dill loader (i.e. BinaryBuilder) to
enable performing a check of all canonical names. The theory is,
that if a dill is self-contained "every" (1) canonical name should
contain a reference to a node.
This check is (currently) disabled by default, but enabled
excplicitly in the incremental compiler when initializing from dill.
If the check fails a warning will be issued (and the dill wont be
used).
On my machine the check seems to take ~15 ms in all tested cases
(sdk, flutter test).
(1): Except for extra layer of URI of a library if the qualified
name is private.
Closes#32449.
Change-Id: I65c8fba647be3d5ba47f7ee16caef78cb2cfae07
Reviewed-on: https://dart-review.googlesource.com/51462
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Aske Simon Christensen <askesc@google.com>
Kernel mixin transformation desugars mixin applications into normal
classes. Mixed-in type is pulled into interfaces list.
However, dart:mirrors needs to know the original mixed-in type of
a mixin application.
This change solves this problem by propagating a 'isTransformedMixinApplication'
attribute of a class through kernel AST, kernel binary and VM objects
into dart:mirrors implementation.
Fixes: https://github.com/dart-lang/sdk/issues/33240
Change-Id: I98ca69294e1ad445402a5ca91d90c30447aabcb2
Reviewed-on: https://dart-review.googlesource.com/56721
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
Reviewed-by: Kevin Millikin <kmillikin@google.com>
Summary:
The constants table contains a concatenation of the variable-length encodings of
each constant used in the serialized component. Previously, we referenced the
constants in the code by index into the constants table, which requires reading
potentially the entire constants table to dereference. Now, we refer to the
constant by the binary offset into the constants table, which allows us to peek
into it in constant time.
Test Plan:
The constants table is used by all the precompiler modes, so the new encoding
will inherit the coverage of the existing encoding.
Results from debug and release precompiler bots are available under the
"cl-linux-try" button.
Change-Id: I4782bc0035d0b0cbd5bd6b9d086561ea601286c8
Reviewed-on: https://dart-review.googlesource.com/55470
Commit-Queue: Samir Jindel <sjindel@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
This CL updates the frontend_server to allow specifying what dill
file to initialize from.
Normally, it tries to initialize from the same dill file as is the
output. This is fine in many cases (e.g. flutter run always says it
should output to build/app.dill): The first time it compiles
everything, subsequent times (across restarts) it only recompiles
what's needed.
When running tests, however, the output is in a temporary directory,
so it doesn't work across restarts. The startup time for tests is
always rather high, because it always has to recompile everything.
This CL updates the frontend_server to allow specifying what dill
file to initialize from. Flutter can thus save the compiled test
file in a centralized location (e.g. build/test.dill) and load from
that across restarts.
Plumbing this into flutter yields the following result, when running
a single test (the one automatically created when creating a new
flutter project):
Before: ~5.5 seconds
Now, first run: ~5.5 second (unchanged)
Now, subsequent runs: ~3.4 seconds
Approximate breakdown:
Startup cost (e.g. if there's no test directory): ~ 800 ms
Starting up the actual frontend server: ~150 ms
Reading source from old dill, loading all relevant files
and invalidating source: ~ 300 ms
Compiling everything (or, rather, nothing as it's all
from the dill at this point): ~ 650 ms
Serializing: ~ 250 ms
Copying files: ~ 50 ms
Running the actual test: ~1100 ms
Total: ~3300 ms
`time flutter test --local-engine=host_debug --preview-dart-2`
says ~3.4 seconds...
To compare, --no-preview-dart-2 takes ~2.2 seconds,
so dart2 still has a (significantly) higher startup cost.
Moves the needle on https://github.com/flutter/flutter/issues/15767.
Change-Id: I36a2d05bc76e0193d369df42eb3b9f08416dd78f
Reviewed-on: https://dart-review.googlesource.com/51820
Reviewed-by: Kevin Millikin <kmillikin@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
The addition of recently added [PartialInstantiationConstant] caused an
issue during constant table writing, becaused we the table writing code
wasn't updated.
This change uses visitChildren to guard against such changes in the
future.
Closes https://github.com/dart-lang/sdk/issues/33095
Change-Id: I3c6b19e1383c6825f11120b2d6255b8b747d0063
Reviewed-on: https://dart-review.googlesource.com/55161
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
Metadata is no longer written ahead of all nodes. Instead, metadata for
each node is written in the same context as the node itself (into a separate
buffer). This allows metadata to contain (serialize) arbitrary nodes
(for example, arbitrary DartTypes) and use serialization context of parent
nodes (such as declared type parameters).
However, with this change metadata looses the ability to reference
arbitrary AST nodes. This ability was overly restricted and had no
practical uses. (It was not possible to reference nodes which are not
reachable from root Component. As a consequence, it was not possible to
write references to arbitrary DartTypes.)
This change aligns the serialization capabilities of metadata with
how kernel AST nodes are serialized.
Change-Id: I027299a33b599b62572eccd4aa7083ad1dd2b3b3
Reviewed-on: https://dart-review.googlesource.com/54481
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
Second try. First try was reverted because a previously empty URL (the
default 'always there' one) is now a null url instead, which wasn't
handled properly in the frontend_server.
See 47e9039512 for original details.
This reverts commit 4c9b712052.
Change-Id: I717eb927cbe29b6388e72f7e270428d234d44e4d
Reviewed-on: https://dart-review.googlesource.com/53580
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Peter von der Ahé <ahe@google.com>
This is the second try. First commit failed because another reading of
the binary was added after the change was originally made and before
actually landing it (and wasn't noticed when rebasing).
This reverts commit b298fc6d8f.
See 6e2536f585 for more information.
Change-Id: Ia0e7f8921de2cec8088654fa24950df13d846237
Reviewed-on: https://dart-review.googlesource.com/53560
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
Change kernel serialization so that
a) We don't UTF8 encode strings up front, but only as needed
(if nothing else it makes it more obvious where the cost of stuff is)
b) Don't call toString on URIs to save them in a StringIndexer, only to
parse the string later to get a URI back again.
This shaves a bit of the serialization time.
Timings from running `flutter test` in flutter/packages/flutter:
Without change:
```
02:14 +2438 ~18: All tests passed!
02:12 +2438 ~18: All tests passed!
02:14 +2438 ~18: All tests passed!
```
with change:
```
02:10 +2438 ~18: All tests passed!
02:11 +2438 ~18: All tests passed!
02:09 +2438 ~18: All tests passed!
```
=> -2.5% +/- 1.83615%
Change-Id: I03e651bbef23445fb1901452a4efcc732789a168
Reviewed-on: https://dart-review.googlesource.com/51302
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Kevin Millikin <kmillikin@google.com>
Previously doubles was saved as strings in the string table,
with a DoubleLiteral holding a StringReference.
This can cause overhead in both computation time
(converting the double to and from string) as well as size
(e.g. a single usage of the previously unused double 1000000.42
would use (at least)
* 10 bytes for the characters
* 1 byte for the size
* 1 byte for the reference to the string
whereas saving it as a double would simply save the 8 bytes.
On the other hand the string table doesn't contain duplicates so
many usages of the same double will use more space.
The SDK dill file size decreases slightly (< 1 KB).
On a Dart file with 1M different doubles (0.42, 1.42, ..., 999999.42)
added to a list:
Before:
compile and write via fasta (non-strong-mode): 0:12.18
Reading (via dart, eager): 2500-2600 ms
Writing (to null sink) (after reading): 1600-1800 ms
Output dill file (via fasta compile): ~62 MB
After:
compile and write via fasta (non-strong-mode): 0:11.76
Reading (via dart, eager): 2050-2350 ms
Writing (to null sink) (after reading): 400-550 ms
Output dill file (via fasta compile): ~54 MB
Running the dill file is ~the same time, but "Maximum resident set size
(kbytes)" (from /usr/bin/time -v) decreases with ~4%.
On the other side, if it's 1M of the same doubles (0.42), while
compiling is ~the same speed, the output dill goes from 43MB to 50MB.
Surprisingly the "Maximum resident set size (kbytes)" still decreases
though (~3%).
Running flutter test in flutter/packages/flutter:
Before:
```
02:33 +2425 ~18: All tests passed!
02:28 +2425 ~18: All tests passed!
02:28 +2425 ~18: All tests passed!
```
After:
```
02:12 +2425 ~18: All tests passed!
02:11 +2425 ~18: All tests passed!
02:12 +2425 ~18: All tests passed!
```
So that's -12.0267% +/- 3.15253%
File size of a dill file of an arbitrary test using flutter is reduced by ~44 KB (~0.3%).
Change-Id: I64151376cde1dae6f0d02b3d96991bc432a994ae
Reviewed-on: https://dart-review.googlesource.com/41660
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Kevin Millikin <kmillikin@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Currently, kernel metadata does not support references to nodes which
are not reachable from root Component. This CL adds an error if such
reference exists, instead of silently writing zero offset.
Change-Id: I6de886296bea66bd732f379cc99e0e3693ea79af
Reviewed-on: https://dart-review.googlesource.com/52527
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Kevin Millikin <kmillikin@google.com>
Make all lists returned by ast_from_binary growable so the ast is more consistent with the ast generated by parsing dart code which generally contains growable lists as opposed to fixed-length lists.
This fixes a bug in the Flutter --track-widget-creation transform
where it would fail on an AST if it was deserialized from a binary file
rather than parsed from source code.
Change-Id: I4e6b199ed98df1b72d8d6844829341975a2c29c9
Reviewed-on: https://dart-review.googlesource.com/51780
Commit-Queue: Jacob Richman <jacobr@google.com>
Reviewed-by: Peter von der Ahé <ahe@google.com>
It complicates the intermediate language, none of the back ends are
using it, and it's not something that we want transformation writers
and code generators to deal with.
Change-Id: Ic79f7935dd8619bd233346bb25947e864f38a104
Reviewed-on: https://dart-review.googlesource.com/50440
Commit-Queue: Kevin Millikin <kmillikin@google.com>
Reviewed-by: Samir Jindel <sjindel@google.com>
Before this CL, if initializing from a dill file, and then,
at a subsequent call to computeDelta asks for the full Component,
the sources from the originally loaded libraries was missing.
This also meant that some fileUris were missing.
This can basically break Flutter.
This CL introduces tests that catch this, fixes the issue and bumps the
kernel version to force recompilation in Flutter.
Change-Id: I9f62962db017cec232855377835103ffc324820b
Reviewed-on: https://dart-review.googlesource.com/49642
Reviewed-by: Peter von der Ahé <ahe@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
Before, the part-URI was serialized as a file-URI and conflated with
source location information. The part-URI is the URI in the following
declaration in a Dart source file:
part "URI";
This is different from what we normally call a file-URI. A file URI
is used to point to the source location of the part declaration, not
the URI in the part declaration.
Furthermore, the field was serialized using writeUriReference which
only works for URIs that are in the uriToSource map on a Component.
Although this might seem like a safe optimization, it doesn't work
if the uriToSource map is omitted or if a part declaration refers to
a missing file.
Finally, due to the confusing use of fileUri, LibraryPart was
mistakenly implementing FileUriNode and annotations were stripped
of source locations if the source for the part were omitted from
uriToSource.
The partUri field is now an unresolved string that can be resolved
against either the parent library's import- or file-URI to obtain
either version as needed.
Change-Id: I255cb4eeaf89928292ab32a2f6be9ead6cc8cee1
Reviewed-on: https://dart-review.googlesource.com/49500
Commit-Queue: Peter von der Ahé <ahe@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
Extending visitor makes it impossible to get a static error when new
syntax is added to Kernel. Also, add @override annotations so we can
get a hint when syntax is removed from Kernel.
Change-Id: I9f3111e406e9994e76a637d2782e24cde264fd31
Reviewed-on: https://dart-review.googlesource.com/46721
Commit-Queue: Kevin Millikin <kmillikin@google.com>
Reviewed-by: Dmitry Stefantsov <dmitryas@google.com>
Translate assert statements in async functions. The translation of
the condition subexpression can produce a sequence of statements that
have been hoisted out of it. These need to be guarded so they are
executed only when asserts are enabled.
For this purpose, we introduce an AssertBlock. The semantics of the
assert block:
assert { s0; ...; sn; }
is the same as:
if (assertsEnabled) { s0; ...; sn; }
where assertsEnabled is a more primitive nullary expression that is
true iff. assertions are enabled. We chose not to encode this using a
construct like assertsEnabled because (1) we would have to support it
appearing as an arbitrary expression which we don't currently need
and (2) it requires deeper pattern matching to detect and skip the
guarded code when desired.
The translation is more complicated because if there is a message
subexpression in an assert, it is conditionally evaluated only if the
assert's condition is false.
Fixes https://github.com/dart-lang/sdk/issues/28498
Change-Id: I0912a57104ede3160533e49f65b6fb79b76f1500
Reviewed-on: https://dart-review.googlesource.com/46442
Commit-Queue: Kevin Millikin <kmillikin@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
This CL changes:
* package:kernel/clone.dart: To not clone file offsets if the
file-uri we copy from is unknown.
* package:kernel/binary/ast_to_binary.dart: To not write out file
offsets if the file-uri is not written/indexed (which happens if
the file uri is not in the `Program.uriToSource` map).
* It reverts the file renaming part of 60a2cfa219 which
fixed the name collision problem only partially (it missed to
rename `runtime/lib/double.dart`, which is not possible because
there's already a `double_patch.dart` file).
Instead of renaming, we solve the problem by mangling the names as follows:
.../sdk/lib/core/double.dart -> dart:core/double.dart
.../runtime/lib/double.dart -> dart:core/runtime/lib/double.dart
.../runtime/lib/double_patch.dart -> dart:core/runtime/lib/double_patch.dart
* It changes the test to ignore any negative token positions. The vm
service does not just expose `-1` token positions for places where
there is no source, but also e.g. `-9` (which means TokenPosition::kMethodExtractor),
this was one reason why the appjit test failed.
* It changes the test to try to force-compile everything (unsure if that works).
Issue https://github.com/dart-lang/sdk/issues/32489
Change-Id: Ia9f42ca9d56d987e3041175add549bb7a133f269
Reviewed-on: https://dart-review.googlesource.com/46341
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
This CL adds two new kernel tools which can answer questions about
a dill file:
* count_breakdown.dart
Enumerates the different node types in the provided dill file and
counts them.
This has proved useful in the past to e.g. see how many doubles are
included in Flutter.
* size_breakdown.dart
Gives an overview of which parts of the dill file contributes how many
bytes.
This has proved useful in the past to see that the string table gets
huge when we have many doubles (because we currently save doubles in
kernel as strings).
It also adds error handling and usage information to existing tools.
Change-Id: Ie2ce9e4b8806d5975dba8f57445705a840be3474
Reviewed-on: https://dart-review.googlesource.com/42660
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Kevin Millikin <kmillikin@google.com>
Jens Johansen