ExpressionCompiler crashes while detecting current scope
if there are any scopes in the library with valid fileOffset
and invalid fileEndOffset.
Make ExpressionCompiler more tolerant to incomplete offset
information by just skipping scopes where fileEndScope is not set.
Fixes: https://github.com/dart-lang/sdk/issues/41241
Change-Id: I357017be1cb641053f23144edd5b74f30a9240ba
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/142080
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Commit-Queue: Anna Gringauze <annagrin@google.com>
Module names and import module names generated by the frontend server
are currently breaking debugger's assumptions on module names. This
change fixes it to make debugger works with various compiler
configurations, such as build_runner, frontend server with dwds tests,
and unblocks flutter tools to continue debugger and frontend server
integration.
JavaScriptBundle
- Change module import names to module names instead of file paths,
omit leading '/' as it cannot be in a module name in requireJS
- Module name changes are hidden under "debugger-module-names" flag
to prevent breaking flutter tools.
FrontendServer
- Always call exit after listening to commands is finished,
otherwise process running frontend server does not exit on 'quit'
command.
- Call computeCanonicalNames after initial compilation to support
subsequent module importing in compileExpressioToJs calls
- Add error handling to usage of kernelToJs compilers
ExpressionCompiler
- Separate library variables from library fields for loading modules,
as they can be different, for example
```main = require('web/main.dart').web__main```
Closes https://github.com/dart-lang/sdk/issues/40832
Change-Id: Ic103c22932c7181ee325f239193d32361d057ed0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/138010
Commit-Queue: Anna Gringauze <annagrin@google.com>
Reviewed-by: Jonah Williams <jonahwilliams@google.com>
Reviewed-by: Gary Roumanis <grouma@google.com>
Reviewed-by: Jake Macdonald <jakemac@google.com>
This is a reland of 344c584ec4
First attempt failed due to lack of implementation of the new
API from SDK in Flutter/engine that this change introduces.
Fixed by creating dummy implementation in flutter/engine:
https://github.com/flutter/engine/pull/16309
Original change's description:
> Implementation of evaluation to JS in frontend server
>
> - made ProgramCompiler somewhat incremental by letting the user
> set the current library and class
> - fixed a bug in procedure_builder where extension method wrappers
> did not have correct fileEndOffset set, which prevented finding
> dart scope at a line when extension methods are used
> - added new compileExpressionToJs API to frontend server and
> frontend compiler interface
> - added ExpressionCompier class that combines IncrementalCompiler
> and ProgramCompiler to compile expression at given dart location
> to JavaScript
> - in JavascriptBundle, set the module name for library tracking in
> JavaScript and saved program compilers to a table for incremental
> reuse in expression compiler
> - Exposed generator from IncrementalCompiler for use in
> ExpressionCompiler
> - added tests for expression compiler
> - added test for compileExpressionToJs API in frontend server tests
>
>
> Change-Id: Ic5e7829e07030b8ad044da7d35bcf27e9fba81c8
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/132701
> Commit-Queue: Anna Gringauze <annagrin@google.com>
> Reviewed-by: Alexander Aprelev <aam@google.com>
> Reviewed-by: Sigmund Cherem <sigmund@google.com>
> Reviewed-by: Gary Roumanis <grouma@google.com>
> Reviewed-by: Jake Macdonald <jakemac@google.com>
> Reviewed-by: Johnni Winther <johnniwinther@google.com>
Change-Id: I56426d2b6d831b2f8d8c57d468d0b54833a0062e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/134561
Commit-Queue: Anna Gringauze <annagrin@google.com>
Reviewed-by: Sigmund Cherem <sigmund@google.com>
- made ProgramCompiler somewhat incremental by letting the user set
the current library and class
- fixed a bug in procedure_builder where extension method wrappers
did not have correct fileEndOffset set, which prevented finding
dart scope at a line when extension methods are used
- added new compileExpressionToJs API to frontend server and
frontend compiler interface
- added ExpressionCompier class that combines IncrementalCompiler
and ProgramCompiler to compile expression at given dart location
to JavaScript
- in JavascriptBundle, set the module name for library tracking in
JavaScript and saved program compilers to a table for incremental
reuse in expression compiler
- Exposed generator from IncrementalCompiler for use in
ExpressionCompiler
- added tests for expression compiler
- added test for compileExpressionToJs API in frontend server tests
Change-Id: Ic5e7829e07030b8ad044da7d35bcf27e9fba81c8
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/132701
Commit-Queue: Anna Gringauze <annagrin@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Reviewed-by: Gary Roumanis <grouma@google.com>
Reviewed-by: Jake Macdonald <jakemac@google.com>
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Also updates StrongComponents to prefer the main component as the root if it is in a cycle. This makes bootstrapping more predictable.
This enables a clean integration of build_runner or ddr (internal) and the frontend_server by allowing it to work with any dev server.
All that is needed is a custom .packages file with `http` uris pointing at a dev server which does all code generation, etc.
Bug: https://github.com/dart-lang/webdev/issues/865
Change-Id: I14533d71e5f7ddac58d98073cf016c2589165e9f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/132962
Commit-Queue: Sigmund Cherem <sigmund@google.com>
Auto-Submit: Jake Macdonald <jakemac@google.com>
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Converts `package:` import uris into `/packages/` modules.
Also renames the output modules to append `.lib.js` instead of just `.js`. This allows us to distinguish between modules and applications based on extension.
Updates DDC source map code to be able to convert absolute file uris in sources so that they are relative to the source map.
Bug: https://github.com/dart-lang/webdev/issues/865
Change-Id: I55d70aa3761f10cc8bd7e92f5b567478040660de
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/132300
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Reviewed-by: Jonah Williams <jonahwilliams@google.com>
Commit-Queue: Jake Macdonald <jakemac@google.com>
Previously, generated bytecode was cached using metadata in kernel AST.
When serializaing kernel AST (for subsequent compilations) incremental
serialization was not used, so AST serialization was considerably
slower in bytecode mode compared to non-bytecode mode.
Apparently, metadata is not compatible with incremental serialization
(https://github.com/dart-lang/sdk/issues/39302).
This change reimplements caching of generated bytecode using a separate
'foo.dill.cache' file near the generated 'foo.dill' and 'foo.dill.ast'.
AST is now written using incremental serialization both in bytecode
and non-bytecode modes.
flutter_test_performance benchmark:
AST mode:
"without_change_elapsed_time_ms": 2615-2652,
"implementation_change_elapsed_time_ms": 6292-6394,
"interface_change_elapsed_time_ms": 6183-6484,
"with_coverage_time_ms": 2723-2834
Bytecode mode, before this change:
"without_change_elapsed_time_ms": 3246-3295,
"implementation_change_elapsed_time_ms": 7998-8068,
"interface_change_elapsed_time_ms": 7899-8029,
"with_coverage_time_ms": 3316-3378
Bytecode mode, after this change:
"without_change_elapsed_time_ms": 2689-2737,
"implementation_change_elapsed_time_ms": 7630-7677,
"interface_change_elapsed_time_ms": 7724-7917,
"with_coverage_time_ms": 2841-2949
Change-Id: I603d33f49949146cf6620226d810f2f3cb9853a9
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/124592
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Add checks that --aot cannot be used with the following options:
--no-link-platform (front-end server, gen_kernel and Fuchsia kernel compiler).
--split-output-by-packages (front-end server, gen_kernel and Fuchsia kernel compiler).
--incremental (front-end server).
--import-dill (front-end server).
Change-Id: I0a38ab8b3a31ec497d1dc05ace0c254455879504
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/123820
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Write JavaScript source map contents as part of the incremental compile. places them in a single file adjacent to the source and offset manifests.
This file contains a map where the module name corresponds to the key and the value is the source map object.
Change-Id: Ice8ac910a5d7119a51d567b8f14cef560cfe8e2e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/123360
Reviewed-by: Jake Macdonald <jakemac@google.com>
Commit-Queue: Jonah Williams <jonahwilliams@google.com>
If the entrypoint is not a file URI, we'll crash as the strong components implementation is unable to find it. Fallback to the first library in the component
Change-Id: Ibd1650978ebc904463ca2a97bd219a4069524b60
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/122560
Commit-Queue: Jonah Williams <jonahwilliams@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
This is a reland of 56991ad94b
Original change's description:
> [vm,aot] Simple unreachable code elimination before type-flow analysis, take 2
>
> This is the relanding of https://dart-review.googlesource.com/c/sdk/+/121500
> with fixes.
>
> Original CL description:
>
> This change adds transformation for very early cleanup of unreachable
> code such as code guarded by if statements with constant conditions or
> code used in assert statements when assertions are disabled.
>
> The advantage of cleaning such code early is that type-flow analysis
> won't be looking at it and TFA-based tree shaker is able to remove
> more code.
>
> flutter_gallery_total_size -0.5663% (arm), -0.5409% (arm64)
> build_bench_total_size -2.533% (arm), -2.449% (arm64)
> gesture_detector_total_size -4.183% (arm), -4.072% (arm64)
>
> Change-Id: Ief8ebd0cd828c0e7a847a824f44d2d97a3595b87
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/121901
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
> Reviewed-by: Martin Kustermann <kustermann@google.com>
> Commit-Queue: Alexander Markov <alexmarkov@google.com>
Change-Id: I82ea0f70845851f85ec159c366f408aa7f9e5788
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/122166
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Adds support to the frontend server for producing javascript when
compiling with a target model of dartdevc. This uses the ProgramCompiler
and associated types from the dev_compiler package to output the kernel
AST into a "bundle" format.
dev_compiler:
* Exposes additional types needed for JavaScript compilation.
* Allows overriding CoreTypes instead of forcing it to be created from
Component. This is require to support compilation of the
partial Components created by the incremental compiler.
* provides extraIndexedLibraries in Target class so the CoreTypes produced by the incremental compiler have indexed them.
frontend_server:
* When the target model is dartdev, JavaScript is output instead of
kernel. For each Component we compile, we first compute the strongly
connected components and produce a List of new Components to interop
with the dev compiler. The file uri is used as module name.
Change-Id: I4cc117b20671ffd48dd43f9786961c016a02d056
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/121400
Reviewed-by: Alexander Aprelev <aam@google.com>
Reviewed-by: Jake Macdonald <jakemac@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Jonah Williams <jonahwilliams@google.com>
This reverts commit 56991ad94b.
Reason for revert: broke building Flutter apps in release mode
Original change's description:
> [vm,aot] Simple unreachable code elimination before type-flow analysis, take 2
>
> This is the relanding of https://dart-review.googlesource.com/c/sdk/+/121500
> with fixes.
>
> Original CL description:
>
> This change adds transformation for very early cleanup of unreachable
> code such as code guarded by if statements with constant conditions or
> code used in assert statements when assertions are disabled.
>
> The advantage of cleaning such code early is that type-flow analysis
> won't be looking at it and TFA-based tree shaker is able to remove
> more code.
>
> flutter_gallery_total_size -0.5663% (arm), -0.5409% (arm64)
> build_bench_total_size -2.533% (arm), -2.449% (arm64)
> gesture_detector_total_size -4.183% (arm), -4.072% (arm64)
>
> Change-Id: Ief8ebd0cd828c0e7a847a824f44d2d97a3595b87
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/121901
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
> Reviewed-by: Martin Kustermann <kustermann@google.com>
> Commit-Queue: Alexander Markov <alexmarkov@google.com>
TBR=vegorov@google.com,kustermann@google.com,rmacnak@google.com,alexmarkov@google.com
Change-Id: I25fe5d4457bfcc5ee25506810aa7eee01ca1b28a
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/122023
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
This is the relanding of https://dart-review.googlesource.com/c/sdk/+/121500
with fixes.
Original CL description:
This change adds transformation for very early cleanup of unreachable
code such as code guarded by if statements with constant conditions or
code used in assert statements when assertions are disabled.
The advantage of cleaning such code early is that type-flow analysis
won't be looking at it and TFA-based tree shaker is able to remove
more code.
flutter_gallery_total_size -0.5663% (arm), -0.5409% (arm64)
build_bench_total_size -2.533% (arm), -2.449% (arm64)
gesture_detector_total_size -4.183% (arm), -4.072% (arm64)
Change-Id: Ief8ebd0cd828c0e7a847a824f44d2d97a3595b87
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/121901
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
This reverts commit 41b65b27c2.
Reason for revert: broke protobuf-aot benchmarks.
Original change's description:
> [vm,aot] Simple unreachable code elimination before type-flow analysis
>
> This change adds transformation for very early cleanup of unreachable
> code such as code guarded by if statements with constant conditions or
> code used in assert statements when assertions are disabled.
>
> The advantage of cleaning such code early is that type-flow analysis
> won't be looking at it and TFA-based tree shaker is able to remove
> more code.
>
> flutter_gallery_total_size -0.5663% (arm), -0.5409% (arm64)
> build_bench_total_size -2.533% (arm), -2.449% (arm64)
> gesture_detector_total_size -4.183% (arm), -4.072% (arm64)
>
> Change-Id: Ic764c056b6594164dc32eed4fd5679a7077ea2f9
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/121500
> Commit-Queue: Alexander Markov <alexmarkov@google.com>
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
> Reviewed-by: Martin Kustermann <kustermann@google.com>
TBR=vegorov@google.com,kustermann@google.com,rmacnak@google.com,alexmarkov@google.com
Change-Id: I3e5668176a7a90d2891126bf79139ac5859bc630
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/121824
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
This change adds transformation for very early cleanup of unreachable
code such as code guarded by if statements with constant conditions or
code used in assert statements when assertions are disabled.
The advantage of cleaning such code early is that type-flow analysis
won't be looking at it and TFA-based tree shaker is able to remove
more code.
flutter_gallery_total_size -0.5663% (arm), -0.5409% (arm64)
build_bench_total_size -2.533% (arm), -2.449% (arm64)
gesture_detector_total_size -4.183% (arm), -4.072% (arm64)
Change-Id: Ic764c056b6594164dc32eed4fd5679a7077ea2f9
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/121500
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Trying it out in flutter (for flutter test runs):
Running "flutter test" in flutter/packages/flutter:
BEFORE ENABLING
===============
$ flutter test --local-engine=host_debug
[...]
03:38 +5149 ~27: All tests passed!
$ flutter test --local-engine=host_debug
[...]
03:11 +5149 ~27: All tests passed!
$ flutter test --local-engine=host_debug
[...]
03:10 +5149 ~27: All tests passed!
With coverage:
$ flutter test --local-engine=host_debug --coverage
[...]
08:37 +5149 ~27: All tests passed!
$ flutter test --local-engine=host_debug --coverage
[...]
08:21 +5149 ~27: All tests passed!
WITH THIS ENABLED
=================
$ flutter test --local-engine=host_debug
[...]
01:39 +5149 ~27: All tests passed!
$ flutter test --local-engine=host_debug
[...]
01:32 +5149 ~27: All tests passed!
$ flutter test --local-engine=host_debug
[...]
01:31 +5149 ~27: All tests passed!
With coverage:
$ flutter test --local-engine=host_debug --coverage
[...]
08:33 +5149 ~27: All tests passed!
$ flutter test --local-engine=host_debug --coverage
[...]
08:34 +5149 ~27: All tests passed!
So, without coverage more than twice as fast!
With coverage ~no change (which makes sense considering it's the VM thats spending extra time collecting coverage, so the compilation is not the bottleneck).
Running "flutter test" in a new project created with "flutter create" (i.e. with a single test):
BEFORE ENABLING
===============
$ time flutter test --local-engine=host_debug
00:05 +1: All tests passed!
real 0m6.360s
user 0m8.783s
sys 0m0.987s
$ time flutter test --local-engine=host_debug
00:02 +1: All tests passed!
real 0m2.790s
user 0m3.089s
sys 0m0.962s
$ time flutter test --local-engine=host_debug
00:02 +1: All tests passed!
real 0m2.693s
user 0m2.872s
sys 0m0.933s
WITH THIS ENABLED
=================
$ time flutter test --local-engine=host_debug
00:05 +1: All tests passed!
real 0m6.356s
user 0m8.253s
sys 0m1.161s
$ time flutter test --local-engine=host_debug
00:02 +1: All tests passed!
real 0m3.035s
user 0m3.449s
sys 0m0.880s
$ time flutter test --local-engine=host_debug
00:02 +1: All tests passed!
real 0m3.055s
user 0m3.534s
sys 0m0.885s
So maybe ~10% improvement on non-first runs (where it reuses the serialization from the dill it initialized from).
~no change on a clean run.
Change-Id: I2817fba9ca60190966194aa0b1ae8e16701137a0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/121121
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
Anna Gringauze