The first compilation result is leaked in two ways:
1) Directly by saving the component in a variable; and
2) Via an unfortunate context thing, probably a variation of
http://dartbug.com/36983. I will update that bug with a reproduction
example later.
The reason this creates a (big) leak is illustrated with an example:
* Say the first component (A) has 10 libraries in it. Each of these
libraries has parent pointers and points (currently) to A, which again
points to all of the 10 libraries.
* We then do a recompilation, say 5 libraries are reused and 5 are new.
They are put into a component (B). We really should have 10 libraries,
the 5 old ones and the 5 new ones (and for simplicity lets say these are
the ones in B). Notice that the 5 old ones will have their parent
pointers updated and also still be in the list of libraries in A.
We keep 15 libraries alive because we have the 10 original ones saved via
A and the 10 (where 5 is new) saved via B.
* We then do a recompilation, say 2 of the same libraries as was also
recompiled before, these end up in compnent C which has 5 libraries from A,
3 libraries from B and the 2 new ones. All of these libraries will have
their parent pointers updated to point to C.
* Because we saved A we keep all the 10 libraries in A though.
Because we saved A and some of the libraries in A had parent pointers
updated to point to B we also keep B and all libraries in B.
Because we keep B and some of the libraries in B had parent pointers
updated to point to C we also keep C and all libraries in C.
So instead of only having the 10 "live" libraries, we have 10 + 5 + 2 = 17
libraries, a leak of 7. With more compilations this keeps happening and
the leak keeps growing.
This CL stops the leak by not holding on to A (which, in turn, stops holding
on to B etc.)
Change-Id: If4f8b1e240b7c39f084df9cb2690570ff26fa9b3
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/149280
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Whenever the VM sends a request to evaluate an expression, it sends over
the list of alive dills (in order) and a sequence number (number of
reloads).
This can then either be
a) ignored if there's already a compiler around that has compiled
everything itself (e.g. if launched from source with observatory
enabled).
b) used to find out that we need to create a new compiler especially
for expression compilation. Either because we don't have any old
compiler or because a reload or GC changed the data available.
c) used to find out that the compiler we created previously is still in
a good state.
Tests for expression compilation has been added, and with the "framework"
added it should be easy to add new ones.
This CL leaves a few things to possibly be improved later:
* It doesn't support if one tries to do a mixed mode, where one
starts from source and reloads from dill. The compiler will not be
updated and the expression compilation will work on a old view of the
world.
* If - in any of the reloads - the main entry point changes library,
the initialization might not work correctly.
Fixes#34736.
Change-Id: Icae009c581e69b810ee19d2a5e2d0a803fd9f7c8
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/135683
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
This is a reland of 344c584ec4
First attempt failed due to lack of implementation of the new
API from SDK in Flutter/engine that this change introduces.
Fixed by creating dummy implementation in flutter/engine:
https://github.com/flutter/engine/pull/16309
Original change's description:
> Implementation of evaluation to JS in frontend server
>
> - made ProgramCompiler somewhat incremental by letting the user
> set the current library and class
> - fixed a bug in procedure_builder where extension method wrappers
> did not have correct fileEndOffset set, which prevented finding
> dart scope at a line when extension methods are used
> - added new compileExpressionToJs API to frontend server and
> frontend compiler interface
> - added ExpressionCompier class that combines IncrementalCompiler
> and ProgramCompiler to compile expression at given dart location
> to JavaScript
> - in JavascriptBundle, set the module name for library tracking in
> JavaScript and saved program compilers to a table for incremental
> reuse in expression compiler
> - Exposed generator from IncrementalCompiler for use in
> ExpressionCompiler
> - added tests for expression compiler
> - added test for compileExpressionToJs API in frontend server tests
>
>
> Change-Id: Ic5e7829e07030b8ad044da7d35bcf27e9fba81c8
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/132701
> Commit-Queue: Anna Gringauze <annagrin@google.com>
> Reviewed-by: Alexander Aprelev <aam@google.com>
> Reviewed-by: Sigmund Cherem <sigmund@google.com>
> Reviewed-by: Gary Roumanis <grouma@google.com>
> Reviewed-by: Jake Macdonald <jakemac@google.com>
> Reviewed-by: Johnni Winther <johnniwinther@google.com>
Change-Id: I56426d2b6d831b2f8d8c57d468d0b54833a0062e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/134561
Commit-Queue: Anna Gringauze <annagrin@google.com>
Reviewed-by: Sigmund Cherem <sigmund@google.com>
- made ProgramCompiler somewhat incremental by letting the user set
the current library and class
- fixed a bug in procedure_builder where extension method wrappers
did not have correct fileEndOffset set, which prevented finding
dart scope at a line when extension methods are used
- added new compileExpressionToJs API to frontend server and
frontend compiler interface
- added ExpressionCompier class that combines IncrementalCompiler
and ProgramCompiler to compile expression at given dart location
to JavaScript
- in JavascriptBundle, set the module name for library tracking in
JavaScript and saved program compilers to a table for incremental
reuse in expression compiler
- Exposed generator from IncrementalCompiler for use in
ExpressionCompiler
- added tests for expression compiler
- added test for compileExpressionToJs API in frontend server tests
Change-Id: Ic5e7829e07030b8ad044da7d35bcf27e9fba81c8
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/132701
Commit-Queue: Anna Gringauze <annagrin@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Reviewed-by: Gary Roumanis <grouma@google.com>
Reviewed-by: Jake Macdonald <jakemac@google.com>
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Trying it out in flutter (for flutter test runs):
Running "flutter test" in flutter/packages/flutter:
BEFORE ENABLING
===============
$ flutter test --local-engine=host_debug
[...]
03:38 +5149 ~27: All tests passed!
$ flutter test --local-engine=host_debug
[...]
03:11 +5149 ~27: All tests passed!
$ flutter test --local-engine=host_debug
[...]
03:10 +5149 ~27: All tests passed!
With coverage:
$ flutter test --local-engine=host_debug --coverage
[...]
08:37 +5149 ~27: All tests passed!
$ flutter test --local-engine=host_debug --coverage
[...]
08:21 +5149 ~27: All tests passed!
WITH THIS ENABLED
=================
$ flutter test --local-engine=host_debug
[...]
01:39 +5149 ~27: All tests passed!
$ flutter test --local-engine=host_debug
[...]
01:32 +5149 ~27: All tests passed!
$ flutter test --local-engine=host_debug
[...]
01:31 +5149 ~27: All tests passed!
With coverage:
$ flutter test --local-engine=host_debug --coverage
[...]
08:33 +5149 ~27: All tests passed!
$ flutter test --local-engine=host_debug --coverage
[...]
08:34 +5149 ~27: All tests passed!
So, without coverage more than twice as fast!
With coverage ~no change (which makes sense considering it's the VM thats spending extra time collecting coverage, so the compilation is not the bottleneck).
Running "flutter test" in a new project created with "flutter create" (i.e. with a single test):
BEFORE ENABLING
===============
$ time flutter test --local-engine=host_debug
00:05 +1: All tests passed!
real 0m6.360s
user 0m8.783s
sys 0m0.987s
$ time flutter test --local-engine=host_debug
00:02 +1: All tests passed!
real 0m2.790s
user 0m3.089s
sys 0m0.962s
$ time flutter test --local-engine=host_debug
00:02 +1: All tests passed!
real 0m2.693s
user 0m2.872s
sys 0m0.933s
WITH THIS ENABLED
=================
$ time flutter test --local-engine=host_debug
00:05 +1: All tests passed!
real 0m6.356s
user 0m8.253s
sys 0m1.161s
$ time flutter test --local-engine=host_debug
00:02 +1: All tests passed!
real 0m3.035s
user 0m3.449s
sys 0m0.880s
$ time flutter test --local-engine=host_debug
00:02 +1: All tests passed!
real 0m3.055s
user 0m3.534s
sys 0m0.885s
So maybe ~10% improvement on non-first runs (where it reuses the serialization from the dill it initialized from).
~no change on a clean run.
Change-Id: I2817fba9ca60190966194aa0b1ae8e16701137a0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/121121
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
This change adds support for bytecode generation in incremental compiler in
kernel service. Bytecode generation is performed as a post-processing step,
after delta components are merged.
Bytecode generation requires CoreTypes and ClassHierarchy, which cannot be
constructed using delta component from incremental compilation (as it doesn't
contain platform or previously compiled libraries). The solution is to obtain
CoreTypes and ClassHierarchy directly from incremental compiler.
Now the following tests work:
vm/cc/IsolateReload_KernelIncrementalCompile
vm/cc/IsolateReload_KernelIncrementalCompileAppAndLib
Change-Id: I34c5870b01850c057ace71a63f8effa5b010cfcf
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/108682
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
For initializeFromComponent to work correctly the platform has to be
provided (as the other libraries provided are linked to some platform,
and loading another one is thus no good).
This CL changes it so we don't load another one, and checks that the
component we're trying to initialize from actually contains dart:core.
Change-Id: I88d30436c101c589b0555ff70ae21297ed665d7b
Reviewed-on: https://dart-review.googlesource.com/c/93435
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
Original PR was reverted because it broke hot-reload bots. This CL adds steps to initialize kernel_service compiler for child isolate. Compiler is needed to support hot-reload request. Child isolate's compiler is created from full kernel file produced by main isolate's compiler. Changes since original PR are pkg/vm/bin/kernel_service.dart in pkg/vm/lib/incremental_compiler.dart.
Further this CL changes kernel fingerprint calculation for interface types so it calculates the hash of the canonical names themselves, rather than indices(that might change from one compilation to another).
This reverts commit 63fd8f63e6.
Change-Id: I6fe5b2ef99f209b32cd4087dfd1c8cac229c2d8b
Reviewed-on: https://dart-review.googlesource.com/c/87265
Commit-Queue: Alexander Aprelev <aam@google.com>
Reviewed-by: Liam Appelbe <liama@google.com>
Also, convert package:vm to use onDiagnostic.
We're consolidating all dependencies on package:front_end in one file per tool.
The idea is, if you want to depend on something in the package:front_end, you modify pkg/front_end/lib/src/api_unstable/vm.dart and consult with the front-end team.
These consolidated files will help us when designing a public API in the future.
Change-Id: I66bafdd0ae29605fd12f8d6c589dbc761e5c7a97
Reviewed-on: https://dart-review.googlesource.com/c/77581
Auto-Submit: Peter von der Ahé <ahe@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Peter von der Ahé <ahe@google.com>
This CL updates the frontend_server to allow specifying what dill
file to initialize from.
Normally, it tries to initialize from the same dill file as is the
output. This is fine in many cases (e.g. flutter run always says it
should output to build/app.dill): The first time it compiles
everything, subsequent times (across restarts) it only recompiles
what's needed.
When running tests, however, the output is in a temporary directory,
so it doesn't work across restarts. The startup time for tests is
always rather high, because it always has to recompile everything.
This CL updates the frontend_server to allow specifying what dill
file to initialize from. Flutter can thus save the compiled test
file in a centralized location (e.g. build/test.dill) and load from
that across restarts.
Plumbing this into flutter yields the following result, when running
a single test (the one automatically created when creating a new
flutter project):
Before: ~5.5 seconds
Now, first run: ~5.5 second (unchanged)
Now, subsequent runs: ~3.4 seconds
Approximate breakdown:
Startup cost (e.g. if there's no test directory): ~ 800 ms
Starting up the actual frontend server: ~150 ms
Reading source from old dill, loading all relevant files
and invalidating source: ~ 300 ms
Compiling everything (or, rather, nothing as it's all
from the dill at this point): ~ 650 ms
Serializing: ~ 250 ms
Copying files: ~ 50 ms
Running the actual test: ~1100 ms
Total: ~3300 ms
`time flutter test --local-engine=host_debug --preview-dart-2`
says ~3.4 seconds...
To compare, --no-preview-dart-2 takes ~2.2 seconds,
so dart2 still has a (significantly) higher startup cost.
Moves the needle on https://github.com/flutter/flutter/issues/15767.
Change-Id: I36a2d05bc76e0193d369df42eb3b9f08416dd78f
Reviewed-on: https://dart-review.googlesource.com/51820
Reviewed-by: Kevin Millikin <kmillikin@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
This is the second attempt at this change.
The original try was committed as f9c7a4a22d.
The revert of that was committed as 2f68e82526.
This CL allows users of the incremental compiler to ask for a computeDelta
call to include the full Component in the output,
basically turning the incremental compiler into a batch compiler.
This CL furthermore changes the frontend_server (in pkg/vm) to use this
functionality.
This change - used in flutter - takes the time it takes to run
`flutter test` in `packages/flutter` from ~8 minutes 30 seconds to
~2 minutes 30 seconds (on my machine).
Fixes#32663.
Change-Id: I073e77b3596b33a24cabcfbcedf97bb0be7aa450
Reviewed-on: https://dart-review.googlesource.com/49125
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Peter von der Ahé <ahe@google.com>
This reverts commit f9c7a4a22d.
Reason for revert: This CL breaks the restart functionality in Flutter, when we restart we end up with an empty app.dill file (60 bytes).
Original change's description:
> Allow to output full component via computeDelta
>
> This CL allows users of the incremental compiler to ask for a computeDelta
> call to include the full Component in the output,
> basically turning the incremental compiler into a batch compiler.
>
> This CL furthermore changes the frontend_server (in pkg/vm) to use this
> functionality.
>
> This change - used in flutter - takes the time it takes to run
> `flutter test` in `packages/flutter` from ~8 minutes 30 seconds to
> ~2 minutes 30 seconds (on my machine).
>
> Change-Id: Ia185d28da1f97b0be5dad8814cd01a8f13482724
> Reviewed-on: https://dart-review.googlesource.com/47020
> Commit-Queue: Jens Johansen <jensj@google.com>
> Reviewed-by: Peter von der Ahé <ahe@google.com>
> Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
TBR=ahe@google.com,vegorov@google.com,jensj@google.com
Change-Id: I5ff39f9d78319413f4bbb7800e24bf27ab4ada9c
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/48240
Reviewed-by: Siva Annamalai <asiva@google.com>
Commit-Queue: Siva Annamalai <asiva@google.com>
This CL allows users of the incremental compiler to ask for a computeDelta
call to include the full Component in the output,
basically turning the incremental compiler into a batch compiler.
This CL furthermore changes the frontend_server (in pkg/vm) to use this
functionality.
This change - used in flutter - takes the time it takes to run
`flutter test` in `packages/flutter` from ~8 minutes 30 seconds to
~2 minutes 30 seconds (on my machine).
Change-Id: Ia185d28da1f97b0be5dad8814cd01a8f13482724
Reviewed-on: https://dart-review.googlesource.com/47020
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Peter von der Ahé <ahe@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Corresponding parameter for IKG was added recently, IncrementalCompiler
needs to pipe it through.
Change-Id: I44a55bd56f8d5389720fee2ec58e5315c13df6c9
Reviewed-on: https://dart-review.googlesource.com/40511
Reviewed-by: Siva Annamalai <asiva@google.com>
Commit-Queue: Alexander Aprelev <aam@google.com>
This reverts commit 2d4c606e63.
The fix for the debug buildbot breakage now consists of skipping wide to utf8 conversion
altogether, instead move uri to path conversion directly into newly added File::OpenUri method.
Change-Id: Id3d7457d6a6b5045e5f38678548f4d7bed6d2c43
Reviewed-on: https://dart-review.googlesource.com/39801
Commit-Queue: Alexander Aprelev <aam@google.com>
Reviewed-by: Zach Anderson <zra@google.com>
Jens Johansen