Before this CL to read the source (and line endings etc) for a specific
file index, one had to read at least part of the data for all previous
file indexes (e.g. read all the line endings).
This CL introduces an index to the included sources meaning that we have
random access based on file id, i.e. can go to the data concerning a
specific file id in constant time.
Benchmarks run with "time python tools/test.py -m release -cdartk language -j6"
shows that - of 5 runs - the runtime has changed as follows:
real: -3.93% +/- 1.03%
user: -3.41% +/- 0.54%
sys: No difference at 95%
(statistics by math stolen from ministat)
So it is ~4% faster to run the language tests (with above command),
shaving approximately 9 seconds off the real runtime.
Change-Id: I9e60a16958356b16b3da0bf6c01ffc5619deb976
Reviewed-on: https://dart-review.googlesource.com/3180
Reviewed-by: Samir Jindel <sjindel@google.com>
Previously the VM couldn't handle external libraries, but that was
fixed in 2f49198520.
Part of the CL was reverted though because the compilatin was changed
to using an outline instead of the platform which doesn't work.
What does work though, is not including the external libraries in the
output.
This CL makes the following changes:
* Don't include external libraries in the output (by not setting all
libraries to be non-external).
* Only writes the sources actually used to the binary (i.e. whatever
libraries left out because they were external will not contribute
source code either).
* Cleanup of now unused code.
Timings (only run once though):
Without this CL (but with the CL it's based on):
$ time python tools/test.py -m release -cdartk language -j6
Test configuration: dartk_vm_release_x64
[05:43 | 100% | + 3504 | - 0]
real 5m43.597s
user 33m48.152s
sys 9m34.140s
Only the "utils/kernel-service/kernel-service.dart" part of this CL:
$ time python tools/test.py -m release -cdartk language -j6
Test configuration: dartk_vm_release_x64
[04:55 | 100% | + 3504 | - 0]
real 4m55.684s
user 29m54.360s
sys 8m7.408s
Entire CL:
$ time python tools/test.py -m release -cdartk language -j6
Test configuration: dartk_vm_release_x64
[04:20 | 100% | + 3504 | - 0]
real 4m20.416s
user 27m17.320s
sys 6m53.472s
Change-Id: Ie9c5bfa958e558a5007784e821a0b58d417bae55
Reviewed-on: https://dart-review.googlesource.com/3161
Reviewed-by: Samir Jindel <sjindel@google.com>
Currently serializing the ast for kernel is done in two passe:
1) Scan the program to find and index all strings. These are then
sorted based on frequency and assigned an id. All string-
references are refering to that id. As small numbers use less
space in the binary than big numbers, sorting the numbers by
frequency saves a certain amount of space.
In addition the string indexing is "hijacked" for the
"LimitedBinaryPrinter" to also perform some CanonicalName
re-indexing.
2) We then serialize the entire thing.
This CL gets rid of a pass by not indexing the strings up-front.
Whenever it is asked to serialize a string it adds it to the index
(if not already there). The serialization is otherwise the same.
This means that:
1) Strings are not sorted by frequency, i.e. the binary output size
can by bigger (numbers below).
2) The stringindex and canonical names are moved to the end of the
binary instead of the front. As we still need it up front for
deserialization some additional data is added to the
ProgramIndex.
3) The "hijacking" done in "LimitedBinaryPrinter" is replaced by
an alternative.
4) We don't spend time on walking the tree twice.
The cost is the binary size. Compiling helloworld with fasta,
as well as looking at outline.dill, platform.dill and
vmservice_io.dill reveals these numbers:
* helloworld.dill is 0.657248732% bigger (26573 bytes)
* outline.dill is 1.686911399% bigger (9395 bytes)
* platform.dill is 0.657062238% bigger (26565 bytes)
* vmservice_io.dill is 0.44991899% bigger (19147 bytes)
The cost does thus not appear to be very big.
The gain is the serialization time.
From 20 runs of an instrumented VM/serialization, running numbers
through calculations stolens from ministat
(https://www.freebsd.org/cgi/man.cgi?query=ministat) reveals the
following:
* Serialization time: -21.69% +/- 1.44%
* Total time spend in relevant parts of bootstrap_nocore.cc,
dart_api_impl.cc (Dart_LoadKernel), bootstrap_nocore.cc,
dart_api_impl.cc (LoadKernelProgram) as well as serialization:
-14.01% +/- 1.58%
From 5 runs of
"time python tools/test.py -m release -cdartk language -j6"
(again run through ministat calculations):
* real: -4.18% +/- 0.5%
* user: -4.2% +/- 0.29%
* sys: No difference at 95%
* user+sys: -3.3% +/- 0.36%
Change-Id: I1c220eac083496994f0a9f1e2a2445b3707c9a93
Reviewed-on: https://dart-review.googlesource.com/2880
Reviewed-by: Samir Jindel <sjindel@google.com>
This CL copies the kernel bodies for all functions and
fields into the VM heap. The function bodies in the VM
heap are then used when compiling the flowgraphs.
This theoretically means that the malloc'd data can be
freed and that snapshotting from kernel could possibly
work, though it hasn't been tested.
R=kmillikin@google.com
Review-Url: https://codereview.chromium.org/2972343002 .
Prior to this CL we carried around information about the containing class
and member, both of which was fetched by reading out-of-line in the binary
(i.e. while reading the current member, start reading something from the
parent member etc).
It had also required the introduction of extra fields in the kernel
binary file (dill file).
This CL cleans that up, by
a) Setting type parameters on functions as needed (in kernel_reader.cc)
b) Using the VM Class and VM Function to get the required information
(with a above the information is all available).
(in kernel_binary_flowgraph.cc.) This means that
c) We don't have to read the binary out-of-line (for TypeParameterType
to work at least), and that
d) We can remove the previously introduced extra fields from the
kernel binary file (dill file).
R=dmitryas@google.com, kmillikin@google.com
Review-Url: https://codereview.chromium.org/2973633002 .
Summary:
Previously, we filled in all occurrences of captured type variables with either
"dynamic" or their bound, if they had one.
Now, we add extra type parameters to the top-level function corresponding to the
closure, and pass in the corresponding arguments as type arguments to the
"MakeClosure" operation.
Test Plan:
Updated [type_variables.dart] and added a new test case to it.
R=dmitryas@google.com
Review-Url: https://codereview.chromium.org/2989563002 .
Original CL had a bug that wasn't visible unless you delete your
out/ReleaseX64/patched_sdk folder.
Patchset #1 is the original CL, patchset #2 shows the fix.
This reverts commit 4aadfe09df.
BUG=
Review-Url: https://codereview.chromium.org/2976543002 .
This CL tweaks the public APIs in package:front_end, and
starts using those APIs outside the package. For example, this
removes 9 uses of DillTarget, so it is not longer mentioned
outside pkg/front_end and the analyzer_target.
Actual changes:
- in package:front_end
* added kernel_generator_impl: new file contains code that
used to be in kernel_generator. Code has some modifications:
it uses a single canonical-root when loading summaries, and
it supports generating both outlines and kernel in one go.
* removed code that didn't belong here:
a. most of calculating deps for .GN moved to patch_sdk
b. vm-specific outcomes moved to kernel-service
* updated how `native` is implemented, so we can more easily
support dart2js and ddc
* updated how we check where `int`, `bool`, etc can be implemented.
* added support "hermetic mode" in modular builds
('chaseDependencies = false' option)
* moved `trim` step out of fasta, and for now call it only within
the public API. This is not yet exposed, and I stopped covering it in
most tests (now only covered in shaker tests). The plan is to add
tests for the public API covering this in the future.
* removed `uriToSource` when serializing outlines
* added unit tests for public APIs
- patch_sdk
* use the public API to craete platform.dill, outline.dill (now
500K insted of 3Mb because it excludes sources), and vmservice_io.dill
* moved here logic internal to .GN
- kernel service
* use the public API
* moved here logic that depends on VM internals (e.g. status enum,
compilation results)
- package:compiler
* use the public API in tools and unit tests
* simplified patched-sdk generation: no more extending fasta's internals
- package:kernel
* fix bug in deserialization: initializers and other lists were
overwritten accidentally with external definitions.
* updated unit tests, moved shared logic to frontend/src/fasta/testing
R=johnniwinther@google.com, paulberry@google.com
Review-Url: https://codereview.chromium.org/2953703002 .
I'm about to start introducing usages of TypeParameterType.bound, and
nearly all of the usage sites will need to fall back on
TypeParameter.bound if there is no bound stored in the
TypeParameterType. This CL places the fallback behavior in the
TypeParameterType.bound getter so that we won't have to duplicate it
at every usage site (and risk making mistakes). The few call sites
that don't want the fallback behavior (such as the one in
ast_to_binary.dart) can avoid it by referring to
TypeParameterType.promotedBound directly.
R=ahe@google.com
Review-Url: https://codereview.chromium.org/2952883003 .
Mostly stream kernel_reader, i.e. the code that sets up the libraries,
classes, methods etc.
Mostly because it still takes a "Program" ast node, and looks at the
"Library" ast nodes to get their kernel offset in the binary.
Currently the scripts (containing breakable points etc) are also created
from the ast nodes.
The rest is now streamed.
This also means that more ast visitors could be deleted.
R=kmillikin@google.com
Review-Url: https://codereview.chromium.org/2931813002 .
This mimics the behaviour of the source-based pipeline,
i.e. instead of "manually" calling _AssertionError._create and giving
the correct parameters (wrong parameters, actually), use the helper
method _AssertionError.ThrowNew.
BUG=
R=ahe@google.com, vegorov@google.com
Review-Url: https://codereview.chromium.org/2940283002 .
- Put pointer to kernel data into Script.
- Replace function.kernel_function pointer to AstNode with
kernel_offset():
- Replace field.kernel_field pointer to AstNode with kernel_offest().
- Stream the previously unstreamed AstNodes: FunctionDeclaration and
FunctionExpression.
- Move special handling for _buildin.getMainClosure into the streaming
flowgraph builder.
- Delete big parts of kernel_to_il.
R=kmillikin@google.com
Review-Url: https://codereview.chromium.org/2901533002 .
This CL allows for streaming big parts of the binary,
i.e. without using the AST nodes.
It is thus a stepping-stone in getting rid of the AST nodes in the VM.
Generally, all Expressions except "FunctionExpression",
and all Statements except "FunctionDeclaration" can be streamed.
There are currently not streamed because they create new functions,
which has a pointer to an AstNode (which we don't have when streaming).
Once we no longer need AstNodes at all these can be streamed as well.
This is, I think, mostly a matter of streaming the ScopeBuilder as well,
something that is not currently done.
The way the streaming is build, one has to stream an entire subtree.
That means, that if an expression (or statement), A, that is generally
streamable contains an expression or a statement, B, that is not streamable,
A cannot be streamed.
The way this is build is by marking AstNodes as streamable or not
("cannot_stream_" field). That way we know up front whether we can stream
a subtree or not.
The streaming is done via "kernel_binary_flowgraph".
In this file there are many obvious comments, e.g.
```
TokenPosition position = ReadPosition(); // read position.
```
This has been done in an attempt to add a comment to everything that
reads from the binary to make it stand out more.
All changes from kernel_to_il up to and including May 2nd 2017
should be included.
R=kmillikin@google.com
Review-Url: https://codereview.chromium.org/2854393002 .
Jens Johansen