TL;DR: Unbind canonical names doesn't do what you think it does and
probably shouldn't (ever) be used. This CL stops using it in a few places.
Longer version:
When loading a dill file it:
- First loads the table of canonical names. These have no references yet.
- When a canonical name is asked for its reference it creates one if it
doesn't yet have one.
- When loading, for instance, a library, it asks for the reference.
When unbinding a canonical name:
- It removes itself (the canonical name) from the reference
- It removes the reference in itself
- Note: Whatever has a pointer to the reference keeps it, and the
reference points to whatever node it already pointed to.
This also means, that if we have a dill file that's split in two and:
- Load #1
- Load #2
that works fine, but if we
- Load #1
- Unbind canonical names
- Binds canonical names
- Load #2
stuff is not bound correctly (and an error is thrown).
And - the cause of this bug:
- Load #1
- Load #2
everything is fine
- Unbind canonical names
- Binds canonical names
- Load #2'
stuff is not bound correctly --- references points to stuff loaded as #2,
not as #2'. On top of being weird, wrong and confusing it also caused wrong
things to be but into the class hierarchy which ultimatly caused the crash.
This CL fixes it by not calling unbind and force loading of dill files
(at specific call sites) to create new libraries
(and in the process overwriting references ".node").
Revert "[dartdevc] Retry ddc incremental compile on crash"
This reverts commit ecdbdf00b8.
Revert "[kernel_worker] retry on failure"
This reverts commit 43eebea5a3.
Fixes#36644
Bug: #36644
Change-Id: Id8f548179e6a409b01f2ebfa3219f94cb64b1c05
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/100380
Reviewed-by: Kevin Millikin <kmillikin@google.com>
Reviewed-by: Vijay Menon <vsm@google.com>
Reviewed-by: Jake Macdonald <jakemac@google.com>
Reviewed-by: Jenny Messerly <jmesserly@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
This CL adds an option to the dill loader (i.e. BinaryBuilder) to
enable performing a check of all canonical names. The theory is,
that if a dill is self-contained "every" (1) canonical name should
contain a reference to a node.
This check is (currently) disabled by default, but enabled
excplicitly in the incremental compiler when initializing from dill.
If the check fails a warning will be issued (and the dill wont be
used).
On my machine the check seems to take ~15 ms in all tested cases
(sdk, flutter test).
(1): Except for extra layer of URI of a library if the qualified
name is private.
Closes#32449.
Change-Id: I65c8fba647be3d5ba47f7ee16caef78cb2cfae07
Reviewed-on: https://dart-review.googlesource.com/51462
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Aske Simon Christensen <askesc@google.com>
New type of AST nodes RedirectingFactoryConstructor is added to Kernel.
The goal is to keep the information about redirecting factory
constructors in the class for the linking phase.
Change-Id: I7477c617fa7fd4935ceb23098ec7c056d1f286ca
Reviewed-on: https://dart-review.googlesource.com/14740
Commit-Queue: Dmitry Stefantsov <dmitryas@google.com>
Reviewed-by: Peter von der Ahé <ahe@google.com>
Before: ownership of some or all of the libraries from one Kernel
program (P1) would be transferred to another program (P2). All the
canonical names in P1 would be unbound from their references and the
canonical names would eventually be recreated for the libraries that
were transferred.
After: when ownership of a library is transferred, the ownership of
the canonical name subtree rooted at the library's name is also
transferred. This allows the 1:1 relationship between Canonicalname
and Reference to be maintained which will enable lazy deserialization
of procedure bodies (because the mapping from CanonicalNames to
References in the corresponding link table will be persistent).
Change-Id: I98f975d6ba5804f975c30528a484756b39f09d2a
Reviewed-on: https://dart-review.googlesource.com/7549
Commit-Queue: Kevin Millikin <kmillikin@google.com>
Reviewed-by: Peter von der Ahé <ahe@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
Currently serializing the ast for kernel is done in two passe:
1) Scan the program to find and index all strings. These are then
sorted based on frequency and assigned an id. All string-
references are refering to that id. As small numbers use less
space in the binary than big numbers, sorting the numbers by
frequency saves a certain amount of space.
In addition the string indexing is "hijacked" for the
"LimitedBinaryPrinter" to also perform some CanonicalName
re-indexing.
2) We then serialize the entire thing.
This CL gets rid of a pass by not indexing the strings up-front.
Whenever it is asked to serialize a string it adds it to the index
(if not already there). The serialization is otherwise the same.
This means that:
1) Strings are not sorted by frequency, i.e. the binary output size
can by bigger (numbers below).
2) The stringindex and canonical names are moved to the end of the
binary instead of the front. As we still need it up front for
deserialization some additional data is added to the
ProgramIndex.
3) The "hijacking" done in "LimitedBinaryPrinter" is replaced by
an alternative.
4) We don't spend time on walking the tree twice.
The cost is the binary size. Compiling helloworld with fasta,
as well as looking at outline.dill, platform.dill and
vmservice_io.dill reveals these numbers:
* helloworld.dill is 0.657248732% bigger (26573 bytes)
* outline.dill is 1.686911399% bigger (9395 bytes)
* platform.dill is 0.657062238% bigger (26565 bytes)
* vmservice_io.dill is 0.44991899% bigger (19147 bytes)
The cost does thus not appear to be very big.
The gain is the serialization time.
From 20 runs of an instrumented VM/serialization, running numbers
through calculations stolens from ministat
(https://www.freebsd.org/cgi/man.cgi?query=ministat) reveals the
following:
* Serialization time: -21.69% +/- 1.44%
* Total time spend in relevant parts of bootstrap_nocore.cc,
dart_api_impl.cc (Dart_LoadKernel), bootstrap_nocore.cc,
dart_api_impl.cc (LoadKernelProgram) as well as serialization:
-14.01% +/- 1.58%
From 5 runs of
"time python tools/test.py -m release -cdartk language -j6"
(again run through ministat calculations):
* real: -4.18% +/- 0.5%
* user: -4.2% +/- 0.29%
* sys: No difference at 95%
* user+sys: -3.3% +/- 0.36%
Change-Id: I1c220eac083496994f0a9f1e2a2445b3707c9a93
Reviewed-on: https://dart-review.googlesource.com/2880
Reviewed-by: Samir Jindel <sjindel@google.com>
Summary:
We use a modest set of annotations in ast.dart to describe how the Kernel AST
should be converted into Coq definitions.
We define a Kernel transformation that converts the kernel tree of ast.dart into
a valid Coq file containing the corresponding definitions.
Currently generating the Coq file is not done in the build system because
compiling it requires having Coq installed, and I don't want to introduce a
depedency on Coq into the build system.
Some parts of the AST are not represented because they don't significantly
contribute to the typing semantics:
- asserts
- typedefs
- most literals/basic types (excl. bool, which is needed for "is" tests)
- switch
- for-in
- parts
- yield/await
Test Plan:
Ran the output KernelSyntax.v file through "coqc".
Change-Id: Ic573163a017eaaf3759b741b9eec5ce3ce19225c
Reviewed-on: https://dart-review.googlesource.com/2960
Reviewed-by: Dmitry Stefantsov <dmitryas@google.com>
Commit-Queue: Dmitry Stefantsov <dmitryas@google.com>
Before: Canonical names contained pointers to the corresponding Kernel
tree which assumed that the whole tree was in memory whenever the
canonical names were.
Now: Canonical names do not contain these pointers. They were only
really used to perform name-based lookup in the VM's heap so the
canonical name itself is enough.
If we later find that we need to get from a canonical name to its
Kernel tree we can add an offset in the binary (for instance) to the
canonical name or in a separate mapping on the side.
BUG=
R=asgerf@google.com, jensj@google.com, vegorov@google.com
Committed: https://github.com/dart-lang/sdk/commit/ed77783cd32d55fdad61bf9bc749030847ba9384
Review-Url: https://codereview.chromium.org/2781893004 .
Before: Canonical names contained pointers to the corresponding Kernel
tree which assumed that the whole tree was in memory whenever the
canonical names were.
Now: Canonical names do not contain these pointers. They were only
really used to perform name-based lookup in the VM's heap so the
canonical name itself is enough.
If we later find that we need to get from a canonical name to its
Kernel tree we can add an offset in the binary (for instance) to the
canonical name or in a separate mapping on the side.
BUG=
R=asgerf@google.com, vegorov@google.com
Review-Url: https://codereview.chromium.org/2781893004 .
This adds a class CanonicalName that can represent a library, class,
or member. All references now go through a Reference object, which is
linked to both the AST node and its CanonicalName, so either can be
created first.
dartk now accepts multiple input files:
- If multiple dart files are given, they are all compiled.
- If multiple binaries are given, they are linked together.
Mixed dart and binary input is not supported by dartk.
dartk now has a flag --include-sdk which includes the entire SDK in
the output. This is so the SDK can be compiled alone and then linked.
Example of compiling separately and then linking:
dartk foo.dart -o foo.dill
dartk main.dart -o main.dill
dartk --include-sdk -o sdk.dill
dartk main.dill foo.dill sdk.dill --target=vm --link -o program.dill
dartk still has incredibly slow cold start due to the analyzer loading
the dart sdk, so this does not actually speed things up at the moment.
BUG=
R=ahe@google.com, kmillikin@google.com, kustermann@google.com, sigmund@google.com
Review-Url: https://codereview.chromium.org/2665723002 .
Jens Johansen