We change the type of FileUriNode.fileUri from String to Uri, which in principle
doesn't change the binary format. However, we did notice that LibraryParts
weren't serialized as specified in binary.md, so we also fixed that.
Since fileUris are stored as strings in a separate table, Uri.parse is only called
once per unique URI.
Fasta only uses relativizeUri when printing diagnostics, and URIs stored in
expectation files (golden files) are relativized using String.replaceAll.
Change-Id: Ib2dc1b80c03a0cdaf84e48b8b3ba73b16bdf8a40
Reviewed-on: https://dart-review.googlesource.com/25421
Commit-Queue: Peter von der Ahé <ahe@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Assert initializers in Dart may be compiled directly to this form of initializer,
rather than through local initializers as is done now.
Bug:
Change-Id: Ia149ea3d1df5d1dc18be5636801604ffaf7ca7d8
Reviewed-on: https://dart-review.googlesource.com/14760
Commit-Queue: Samir Jindel <sjindel@google.com>
Reviewed-by: Kevin Millikin <kmillikin@google.com>
New type of AST nodes RedirectingFactoryConstructor is added to Kernel.
The goal is to keep the information about redirecting factory
constructors in the class for the linking phase.
Change-Id: I7477c617fa7fd4935ceb23098ec7c056d1f286ca
Reviewed-on: https://dart-review.googlesource.com/14740
Commit-Queue: Dmitry Stefantsov <dmitryas@google.com>
Reviewed-by: Peter von der Ahé <ahe@google.com>
This fixes some incorrect asserts that were breaking the debug bots.
The original revision is available in Patchset 1.
This reverts commit 26735519cb.
Bug:
Change-Id: Ifa599b7bff752dec4c505e10fd6db206e1abd977
Reviewed-on: https://dart-review.googlesource.com/23820
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Commit-Queue: Samir Jindel <sjindel@google.com>
This reverts commit f13f772bb2.
Reason for revert: dartk(p)-strong debug mode is completely broken
because we convert int values non-representable as Smi to Smi, which
triggers that assert. The code that does tagging checks that
Untag(Tag(x)) == x, however for our purposes this identity does not
need to hold because we only care about lower bits and ignore the sign.
Reverting to restore green-ness of DEBUG builds.
TBR=sjindel@google.com
Bug:
Change-Id: Id436cbe000d6dec8db3469070ed531327cc82d89
Reviewed-on: https://dart-review.googlesource.com/23661
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
The code for reading dill files via dart has a "debugPath" list that is
continously added and removed to in order to be able to debug what went
wrong if something goes wrong (e.g. if the dill file is invalid or the
reading code is wrong etc).
This CL wraps the updates of this list in assert so that we don't pay
for what we don't use in the general case. In a debug setting we can get
the functionality back via --checked.
On a benchmark of 10 runs, the time it takes to read vm_outline.dill 100
times after a 2 second warmup changes by -2.77% +/- 1.21%.
Bug:
Change-Id: I643b8dd778972621046fe76b536fd95e9bb66d1c
Reviewed-on: https://dart-review.googlesource.com/18820
Reviewed-by: Peter von der Ahé <ahe@google.com>
Reviewed-by: Dmitry Stefantsov <dmitryas@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
The introduced "constants" transformation can evaluate constant expressions. The
original use-sites of constant expressions are replaced by a new [ConstantExpression]
node, which points to a subclass of a new [Constant] class hierarchy. Constant
[Field]s and [VariableDeclarations]s will be removed, since all use-sites are
re-written.
The [Constant] class hierarchy is, similarly to the [DartType] class hierarchy, not
part of the AST tree (also has no parent pointer). The constants form a
DAG (directed acyclic graph).
There is no canonicalization requirement of the [Constant] objects referenced by the
AST (via [ConstantExpression]). Although it is beneficial to canonicalize them during
construction, since it reduces time spent in operator==/hashCode.
This CL furthermore adds support for a constant table in the binary format. Similarly
to [String]s, we canonicalize the constants before writing the table to the binary.
The constant table entries in the binary are written in a post-order way, to ensure
easy construction on the backend side.
The text format will be augmented with a "constants { ... }" section at the end,
which lists the constants in the same order as in the binary format.
The transformation can be used by those backends who choose to do so. It is not
enabled by default atm. It should therefore not affect analyzer, fasta or other
components.
Change-Id: I57cd9624fedcf537ab6870db76246149647bed21
Reviewed-on: https://dart-review.googlesource.com/14382
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Kevin Millikin <kmillikin@google.com>
The VM creates context objects even though it's only applicable in a
non-taken branch. This CL moves such a branch into its own function
and thus avoids the context object creation when loading a dill file
eagerly.
This CL further more reduces the number of context creations by not
lazy loading basically non-existing function node bodies (e.g. the body
of a FunctionNode in the outline).
The change has the following effect when loading platform/outline
100 times after 2 seconds warmup (statistics on 5 runs):
Outline, lazy loading disabled: -12.8133% +/- 2.26214%
Outline, lazy loading enabled: -40.9197% +/- 1.50042%
Platform, lazy loading disabled: No difference proven at 95.0% confidence
Platform, lazy loading enabled: -44.3347% +/- 0.613235%
Bug:
Change-Id: I9634e0a81f43efeb4e2524edb765d36072074f1e
Reviewed-on: https://dart-review.googlesource.com/18220
Reviewed-by: Peter von der Ahé <ahe@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
Loading vm_platform.dill 100 times after 2 seconds warmup:
Statistics on 5 runs:
It varies to much to tell on the runs I did.
Loading vm_outline.dill 100 times after 2 seconds warmup:
Statistics on 5 runs:
-7.84428% +/- 1.21335% at 95.0% confidence
Bug:
Change-Id: I34bbdca555ec50e1d16a5994c5cb46f845b4a4a3
Reviewed-on: https://dart-review.googlesource.com/18183
Reviewed-by: Dmitry Stefantsov <dmitryas@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
Changes in runtime:
Running python tools/test.py -m release -cdartk language -j6
Statistics on 3 runs:
real -1.21% +/- 1% at 95.0% confidence
user -2.01% +/- 0.29% at 95.0% confidence
sys No difference at 95%
Loading vm_platform.dill 100 times after 2 seconds warmup:
Statistics on 5 runs:
-15.3209% +/- 1.26028% at 95.0% confidence
Loading vm_outlin.dill 100 times after 2 seconds warmup:
Statistics on 5 runs:
-21.9672% +/- 0.48754% at 95.0% confidence
Bug:
Change-Id: I16474e32715df57922376d88baddd17a1cf73663
Reviewed-on: https://dart-review.googlesource.com/17788
Reviewed-by: Peter von der Ahé <ahe@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
The kernel format has been designed so that one can concatenate several
dill files into one file and then load it. For instance the dart
function BinaryBuilder.readProgram supports this.
Currently a dill file contains one or more programs.
In the VM each of these programs are called either program or subprogram.
Technically a dill "program" isn't necessarily a program at all (e.g. it
could be missing a library).
This naming snafu should probably be cleaned up at some point, but that's
for another CL.
When loading a dill file via BinaryBuilder.readProgram what happens is
this:
- Each program in the dill file ends in 4 bytes that indicates the size
of the program.
- Reading the input from the end one can then read the size, skip back
that amount of bytes, if we have more data (i.e. there's another
program), read another size and so on, and continue until we have
accounted for all bytes in the input.
- We then read each program from the start, and basically overwrite any
library, class, procedure etc. we find.
The first main reference found is the one used though.
(Saying that we overwrite is not completely true, but when the library
is a non-external library that's basically what happens).
This CL introduces (some) support on the C++ side for the same thing.
So far the C++ side could only handle single-program-dills, and trying
to load anything else would probably crash the VM.
The support added is this:
- Assume the SDK (i.e. vm_platform.dill) is not a concatenated file
(error out if it is).
- For user provided input, loop over each contained program one-by-one,
for each individual one behave as normal.
- The way LibraryLoad is implemented (i.e. it skips if the library is
already loaded) this means that it currently would behave differently
than the dart version (i.e. the first one is used, not the last one).
For now it is assumed that that's not a problem.
- There is a possibly snafu if the same script is included several times.
This could probably mostly be remedied by not creating scripts up front,
but only as needed. By the "keep only one" (and fixing the above point,
probably by simply loading in the opposite order, i.e. last program
in the binary first) the (theoretical) problem would probably do away.
Note that we will have separate string tables, canonical name tables etc
per "sub program" and that there might be some duplication.
The implementation was tested as indicated below, but introduces no tests.
$ cat test_lib1.dart
import "test_lib2.dart" as lib2;
String lib1field = "lib #1 field!!";
main() {
foo();
lib2.foo();
print("From lib2: ${lib2.lib2field}");
}
foo() {
print("Hello, Foo, from test_lib1!");
var x = 42;
print(x);
}
$ cat test_lib2.dart
String lib2field = "Lib #2 field!!!!";
foo() {
print("Hello, Foo, from test_lib2!");
var y = 34;
print(y);
}
$ out/ReleaseX64/dart pkg/front_end/tool/_fasta/compile.dart --packages=.packages --platform=out/ReleaseX64/vm_platform.dill test_lib1.dart
$ ls -lha test_lib1.dart.dill
[...] 4.2M Oct 26 14:42 test_lib1.dart.dill
$ dart pkg/kernel/bin/split.dart test_lib1.dart.dill
Wrote test_lib1.dart.dill.part1.dill
Wrote test_lib1.dart.dill.part2.dill
$ ls -lha test_lib1.dart.dill.part{1,2}.dill
[...] 811 Oct 26 14:42 test_lib1.dart.dill.part1.dill
[...] 582 Oct 26 14:42 test_lib1.dart.dill.part2.dill
$ cat test_lib1.dart.dill.part1.dill test_lib1.dart.dill.part2.dill > test_lib1.dart.dill.concat.dill
$ ls -lha test_lib1.dart.dill.concat.dill
[...] 1.4K Oct 26 14:44 test_lib1.dart.dill.concat.dill
$ out/ReleaseX64/dart --kernel-binaries=out/ReleaseX64 --packages=.packages test_lib1.dart.dill.concat.dill
Hello, Foo, from test_lib1!
42
Hello, Foo, from test_lib2!
34
From lib2: Lib #2 field!!!!
Change-Id: I233a033aa3042b202dd4708908a5be3089474588
Reviewed-on: https://dart-review.googlesource.com/16820
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Dmitry Stefantsov <dmitryas@google.com>
Previously lazy loading was done in Procedure by lazy loading functionNode,
now lazy loading is done (only through Procedure) in the functionNode by
lazy loading the body.
Bug:
Change-Id: I25cc86d038123ed4992162b65aa95781ea2c56e8
Reviewed-on: https://dart-review.googlesource.com/15560
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Peter von der Ahé <ahe@google.com>
This flag indicates when a procedure's return type makes contravariant
use of a class type parameter. The front end will have to insert "as"
checks at any call sites that refer to such a procedure as their
interface target. Placing the flag on the procedure will allow the
front end to determine whether the check is needed once, at the same
time as it's determining the need for covariant parameter checks,
rather than when compiling individual call sites.
Once this lands I will follow up with CLs that cause the front end to
set the flag appropriately, and to generate the necessary "as" checks.
Change-Id: I989fc702bc233384eb5ea8cd630c8efd384ab248
Reviewed-on: https://dart-review.googlesource.com/14365
Reviewed-by: Samir Jindel <sjindel@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
This CL adds internal binary format version to kernel binaries.
Kernel readers in the front-end and VM verify that the reader's format
version matches the format version in the binary file.
This improves detection and diagnostics of stale kernel binary files.
Change-Id: Ic69b16057397a84627040bdd6420ffa1852a4b3f
Reviewed-on: https://dart-review.googlesource.com/13280
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
Reviewed-by: Kevin Millikin <kmillikin@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
This section contains a linear mapping between offsets of serialized nodes
and associated opaque binary metadata.
Note: this CL does not yet update C++ reader and only implements metadata in Kernel package.
C++ implementation will be updated if we agree that format seems flexible enough.
Bug:
Change-Id: Id433458afc6c2ea76c72f6a1901b9dc55b8f1696
Reviewed-on: https://dart-review.googlesource.com/9340
Reviewed-by: Jens Johansen <jensj@google.com>
Commit-Queue: Vyacheslav Egorov <vegorov@google.com>
Before this CL we skipped procedure bodies in kernel_loader.cc by
parsing the body (but not storing anything).
With this CL we now skip them directly (i.e. don't read them at all)
in kernel_loader.cc by using the newly available extra indexes in kernel.
Change-Id: I48cf0599b2a85102c9008ff7c455785151ef3c9c
Reviewed-on: https://dart-review.googlesource.com/5764
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Samir Jindel <sjindel@google.com>
This adds more indexes to the kernel format so we know where classes
and procedures starts and stops. This allows for more random access.
E.g. one could now read the program index and jump directly to
library $i_1$, then read the library index and jump directly to class $i_2$,
read the class index and jump directly to procedure $i_3$.
The utilization (in this CL) is to not (always) read the procedure body
up front when loading kernel code on the dart side (ast_from_binary).
The observation is that - when running through the VM - almost none
of the bodies from the platform file are actually used.
This lowers the start-up cost which is noticeable for small programs
(e.g. hello world, or tests).
In this CL this is only done on the dart side and not on the C++ side,
that's for another CL.
Startup time:
dart2js: -1.84253% +/- 1.22157%
hello world: -11.1188% +/- 5.57892%
Running "time python tools/test.py -m release -cdartk language -j6":
real: -11.72% +/- 0.35%
user: -14.59% +/- 0.24%
sys: -12.71% +/- 0.61%
File size change (compiling with fasta to dill file incl. platform):
hello world: 0.88% (35,934 bytes).
dart2js: 0.97% (200,967 bytes).
Change-Id: I1f0ec121bc75bb17f11d3fade03da9815037d0bb
Reviewed-on: https://dart-review.googlesource.com/5262
Reviewed-by: Kevin Millikin <kmillikin@google.com>
This should take care of some or all flaky tests like:
TypePropagationTest_Kernel | test_forEach_async_inheritedStream
What was happening is that every tests adds /test.dart to AnalysisDriver,
which means that this file is scheduled for analysis at some point,
and then it also calls getResult() to get the resolved unit. When we
resolve the file for the first time, the ByteStore is empty, so we
build the corresponding Kernel file from scratch, and it has the offset.
But the second time we read the Kernel file from ByteStore. So, if we
manage to process the file as added first, and then as getResult(),
we fail because we cannot resolve the import directive. But if we
were not able to process the file as added, and just do getResult()
first (which also marks the file as added as ready), we succeed.
So, it was flaky.
R=ahe@google.com, kmillikin@google.com, paulberry@google.com, sigmund@google.com
Bug: https://github.com/dart-lang/sdk/issues/30863
Change-Id: I96151e3ebefcd212f2a7a1b2b22abb7d87ed4781
Reviewed-on: https://dart-review.googlesource.com/7782
Reviewed-by: Paul Berry <paulberry@google.com>
Commit-Queue: Konstantin Shcheglov <scheglov@google.com>
This reverts commit 9a8621b60a.
Revert "Rework getElement() in resynthesizer."
This reverts commit e4fa080f69.
Revert "Create (empty) initializers for parameters resynthesized from Kernel."
This reverts commit 8df6c79b9d.
Revert "Run NonErrorResolverTest in strong mode. Extract not strong tests."
This reverts commit 9bdda4b1d3.
Revert "Move TypeProvider creation into KernelResynthesizer and create loadLibrary functions."
This reverts commit c59eaf1788.
Revert "Return SimpleIdentifier or PrefixedIdentifier from _buildIdentifier()."
This reverts commit 6d0515f9ca.
Change-Id: I1099ca715ce6287ab56808b7cc3abe0589e939c1
Reviewed-on: https://dart-review.googlesource.com/7550
Reviewed-by: Peter von der Ahé <ahe@google.com>
While writeByte(0) can be read with readUint() (and writeUInt30(0) can be
read correctly with readByte()) it's probably better to use
writeUInt30/readUInt as a pair and writeByte/readByte as a pair.
Change-Id: I3e638c1de0bd66b112cfa1370a54412e876dca5a
Reviewed-on: https://dart-review.googlesource.com/4720
Reviewed-by: Samir Jindel <sjindel@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
Before this CL to read the source (and line endings etc) for a specific
file index, one had to read at least part of the data for all previous
file indexes (e.g. read all the line endings).
This CL introduces an index to the included sources meaning that we have
random access based on file id, i.e. can go to the data concerning a
specific file id in constant time.
Benchmarks run with "time python tools/test.py -m release -cdartk language -j6"
shows that - of 5 runs - the runtime has changed as follows:
real: -3.93% +/- 1.03%
user: -3.41% +/- 0.54%
sys: No difference at 95%
(statistics by math stolen from ministat)
So it is ~4% faster to run the language tests (with above command),
shaving approximately 9 seconds off the real runtime.
Change-Id: I9e60a16958356b16b3da0bf6c01ffc5619deb976
Reviewed-on: https://dart-review.googlesource.com/3180
Reviewed-by: Samir Jindel <sjindel@google.com>
Previously the VM couldn't handle external libraries, but that was
fixed in 2f49198520.
Part of the CL was reverted though because the compilatin was changed
to using an outline instead of the platform which doesn't work.
What does work though, is not including the external libraries in the
output.
This CL makes the following changes:
* Don't include external libraries in the output (by not setting all
libraries to be non-external).
* Only writes the sources actually used to the binary (i.e. whatever
libraries left out because they were external will not contribute
source code either).
* Cleanup of now unused code.
Timings (only run once though):
Without this CL (but with the CL it's based on):
$ time python tools/test.py -m release -cdartk language -j6
Test configuration: dartk_vm_release_x64
[05:43 | 100% | + 3504 | - 0]
real 5m43.597s
user 33m48.152s
sys 9m34.140s
Only the "utils/kernel-service/kernel-service.dart" part of this CL:
$ time python tools/test.py -m release -cdartk language -j6
Test configuration: dartk_vm_release_x64
[04:55 | 100% | + 3504 | - 0]
real 4m55.684s
user 29m54.360s
sys 8m7.408s
Entire CL:
$ time python tools/test.py -m release -cdartk language -j6
Test configuration: dartk_vm_release_x64
[04:20 | 100% | + 3504 | - 0]
real 4m20.416s
user 27m17.320s
sys 6m53.472s
Change-Id: Ie9c5bfa958e558a5007784e821a0b58d417bae55
Reviewed-on: https://dart-review.googlesource.com/3161
Reviewed-by: Samir Jindel <sjindel@google.com>
Currently serializing the ast for kernel is done in two passe:
1) Scan the program to find and index all strings. These are then
sorted based on frequency and assigned an id. All string-
references are refering to that id. As small numbers use less
space in the binary than big numbers, sorting the numbers by
frequency saves a certain amount of space.
In addition the string indexing is "hijacked" for the
"LimitedBinaryPrinter" to also perform some CanonicalName
re-indexing.
2) We then serialize the entire thing.
This CL gets rid of a pass by not indexing the strings up-front.
Whenever it is asked to serialize a string it adds it to the index
(if not already there). The serialization is otherwise the same.
This means that:
1) Strings are not sorted by frequency, i.e. the binary output size
can by bigger (numbers below).
2) The stringindex and canonical names are moved to the end of the
binary instead of the front. As we still need it up front for
deserialization some additional data is added to the
ProgramIndex.
3) The "hijacking" done in "LimitedBinaryPrinter" is replaced by
an alternative.
4) We don't spend time on walking the tree twice.
The cost is the binary size. Compiling helloworld with fasta,
as well as looking at outline.dill, platform.dill and
vmservice_io.dill reveals these numbers:
* helloworld.dill is 0.657248732% bigger (26573 bytes)
* outline.dill is 1.686911399% bigger (9395 bytes)
* platform.dill is 0.657062238% bigger (26565 bytes)
* vmservice_io.dill is 0.44991899% bigger (19147 bytes)
The cost does thus not appear to be very big.
The gain is the serialization time.
From 20 runs of an instrumented VM/serialization, running numbers
through calculations stolens from ministat
(https://www.freebsd.org/cgi/man.cgi?query=ministat) reveals the
following:
* Serialization time: -21.69% +/- 1.44%
* Total time spend in relevant parts of bootstrap_nocore.cc,
dart_api_impl.cc (Dart_LoadKernel), bootstrap_nocore.cc,
dart_api_impl.cc (LoadKernelProgram) as well as serialization:
-14.01% +/- 1.58%
From 5 runs of
"time python tools/test.py -m release -cdartk language -j6"
(again run through ministat calculations):
* real: -4.18% +/- 0.5%
* user: -4.2% +/- 0.29%
* sys: No difference at 95%
* user+sys: -3.3% +/- 0.36%
Change-Id: I1c220eac083496994f0a9f1e2a2445b3707c9a93
Reviewed-on: https://dart-review.googlesource.com/2880
Reviewed-by: Samir Jindel <sjindel@google.com>
Peter von der Ahé