This reverts commit ab6aeaa106.
Revert "[vm/compiler] Speed up the compiler part which deals with kernel reading up in DEBUG mode"
This reverts commit b316210d94.
Reason for revert: regression of snapshot sizes (DNO-599).
Original change's description:
> [vm/kernel] Use GC-tracked ExternalTypedData/TypedDataView for kernel buffers
>
> Until now we often leaked kernel buffers (e.g. hot reload buffers) because various
> objects were referencing ExternalTypedData objects pointing into the middle of
> c-allocated memory. This made it impossible for the GC to determine when the last
> reference is gone.
>
> This CL ensures that the actual buffers are *always* made available via
> ExternalTypedData and any inner pointers into it are created via TypedDataViews.
>
> The embedder guarantees to the free kernel buffers it has provided to:
> - Dart_CreateIsolateFromKernel
> - Dart_LoadScriptFromKernel
> - Dart_LoadLibraryFromKernel
> - Dart_SetDartLibrarySourcesKernel
> on isolate shutdown.
>
> All other kernel buffers will get a finalizer attached, which ensures the
> kernel buffers get freed by the GC once they are no longer referenced:
> - Kernel blobs for expression evaluation
> - Kernel blobs for Hot-Reload
> - Kernel blobs for cc tests
>
> Fixes https://github.com/dart-lang/sdk/issues/33973
> Fixes https://github.com/dart-lang/sdk/issues/36857
> Issue https://github.com/dart-lang/sdk/issues/37030
>
> Change-Id: I1cc410c94c0f4b229413e793728a261afcb10aaf
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/103130
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
> Commit-Queue: Martin Kustermann <kustermann@google.com>
TBR=kustermann@google.com,rmacnak@google.com
# Not skipping CQ checks because original CL landed > 1 day ago.
Change-Id: I49715d2400f4a5c8806b7d6a2912b7258f671a0a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/104343
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
Auto-Submit: Alexander Markov <alexmarkov@google.com>
Until now we often leaked kernel buffers (e.g. hot reload buffers) because various
objects were referencing ExternalTypedData objects pointing into the middle of
c-allocated memory. This made it impossible for the GC to determine when the last
reference is gone.
This CL ensures that the actual buffers are *always* made available via
ExternalTypedData and any inner pointers into it are created via TypedDataViews.
The embedder guarantees to the free kernel buffers it has provided to:
- Dart_CreateIsolateFromKernel
- Dart_LoadScriptFromKernel
- Dart_LoadLibraryFromKernel
- Dart_SetDartLibrarySourcesKernel
on isolate shutdown.
All other kernel buffers will get a finalizer attached, which ensures the
kernel buffers get freed by the GC once they are no longer referenced:
- Kernel blobs for expression evaluation
- Kernel blobs for Hot-Reload
- Kernel blobs for cc tests
Fixes https://github.com/dart-lang/sdk/issues/33973
Fixes https://github.com/dart-lang/sdk/issues/36857
Issue https://github.com/dart-lang/sdk/issues/37030
Change-Id: I1cc410c94c0f4b229413e793728a261afcb10aaf
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/103130
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
This for instance allows the VM to include all patches
(e.g. the previously missing `runtime/lib/date_patch.dart`)
in what's returned from `LoadedScripts()`.
Over time this also allows for `LoadedScripts()` to be simplified and
become faster (it's currently something like O(m + m*n) where m is the
number of results in DictionaryIterator(*this) and n is the number of
returned scripts).
Change-Id: I4b5a6a0fe666b774fc0987d099ed02e81ac97b43
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/98660
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Kevin Millikin <kmillikin@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
By indexing all sources in a concatenated dill file and then using that
index to find the sources and line starts we can fix missing information
(leading to crashes when collecting coverage) in for instance circular
instances.
This reverts commit 57321c1590 and adds a
fix.
Change-Id: I7a91801fb318cad0218cf3101d6a15f1ec929175
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/98006
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
This reverts commit a7ab99dd84.
Reason for revert: <INSERT REASONING HERE>
Original change's description:
> [vm/kernel] Index all source and line starts before using them
>
> By indexing all sources in a concatenated dill file and then using that
> index to find the sources and line starts we can fix missing information
> (leading to crashes when collecting coverage) in for instance circular
> instances.
>
> Change-Id: I454cff57f84f21de72f6486e1cecf15d1389a498
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/96901
> Commit-Queue: Jens Johansen <jensj@google.com>
> Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
TBR=vegorov@google.com,jensj@google.com
# Not skipping CQ checks because original CL landed > 1 day ago.
Change-Id: Iea75c9dd56e74fa0e98c90ea6841ed66b39d40cc
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/97941
Reviewed-by: Keerti Parthasarathy <keertip@google.com>
Commit-Queue: Keerti Parthasarathy <keertip@google.com>
This change replaces kernel AST declarations of fields and functions
with bytecode declarations.
Size of dilp files is reduced by 11-12%.
Startup latency:
Time to the first full frame: 1.945s -> 1.687s
FinalizeClass: 554ms -> 277ms
FinishClassLoading: 296ms -> 156ms
There are following regressions in bytecode mode, which will be fixed
in future:
* dart:mirrors are not supported yet (implementation of mirrors relies
on reading kernel AST in certain cases).
As the result, lib_2/mirrors/* tests fail.
* native extensions are not supported yet (annotations on libraries
and classes in AST are cleaned up as they could reference members
which are now removed from AST).
As the result, standalone_2/entrypoints_verification_test test fails.
* language_2/spread_collections/const_error_test/* tests fail
due to https://github.com/dart-lang/sdk/issues/36286.
Change-Id: I5130f401fd7b84038b136136e7ccc1a6e51b6cea
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/97561
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
By indexing all sources in a concatenated dill file and then using that
index to find the sources and line starts we can fix missing information
(leading to crashes when collecting coverage) in for instance circular
instances.
Change-Id: I454cff57f84f21de72f6486e1cecf15d1389a498
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/96901
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
- Added `Dart_SetDartLibrarySourcesKernel` to the Dart embedding API.
- vm_platform_strong.dill now contains the Dart SDK sources
- If vm_platform_strong.dill is linked into the Dart binary, the first
request for a Dart SDK Script object from the VM service will load the
sources for the script from the linked in kernel buffer.
Change-Id: I664abe31f9378d25ec79c21edce0b237a278495d
Reviewed-on: https://dart-review.googlesource.com/c/93375
Commit-Queue: Ben Konyi <bkonyi@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
The script URLs for core libraries were being recorded as follows
- dart:core for org-dartlang-sdk:///sdk/lib/core/core.dart
- dart:core/runtime/libcore_patch.dart for org-dartlang-sdk:///runtime/lib/core_patch.dart
- dart:io/runtime/binfilter_patch.dart for org-dartlang-sdk:///runtime/bin/filter_patch.dart
Now we record them as
- dart:core/core.dart for org-dartlang-sdk:///sdk/lib/core/core.dart
- dart:core/runtime/lib/core_patch.dart for org-dartlang-sdk:///runtime/lib/core_patch.dart
- dart:io/runtime/bin/filter_patch.dart for org-dartlang-sdk:///runtime/bin/filter_patch.dart
Change-Id: I645939e7ba937034471044fb492a5ecdb9a50bf5
Reviewed-on: https://dart-review.googlesource.com/c/92922
Reviewed-by: Ben Konyi <bkonyi@google.com>
Commit-Queue: Siva Annamalai <asiva@google.com>
In --compile-all mode with CL [1] when NoSuchMethodError was finalized
the following happened:
ClassFinalizer::FinalizeClass calls KernelLoader::FinishLoading,
which calls KernelLoader::LoadProcedure to load precedures,
which calls Library::GetMetadata, which runs Dart code and triggers
background compilation; BackgroundCompiler::Start calls finalization of
NoSuchMethodError class recursively.
FinalizeClass calls FinishLoading for the second time, but this time
background compilation is not invoked, FinishLoading completes
and all members are finalized. After returning from recursive finalization,
the first FinishLoading proceeds and overwrites already finalized
functions with freshly loaded non-finalized functions.
As class is already marked as finalized, functions are not finalized
for the second time, causing crash later.
This CL fixes 2 problems related to recursive class loading:
* When fields are loaded, they should replace previous fields
rather than appending them. This ensures that recursive loading
won't add several duplicate fields (potential problem).
* IsLoaded flag is added to a class and checked after loading each
procedure. If class happens to be loaded already, then FinishLoading
should return immediately, discarding its loaded functions.
This prevents overwriting finalized functions with freshly loaded.
Also, this change adds guard against repeated class loading (may happen
during recursive attempts of class finalization).
[1] https://dart-review.googlesource.com/c/sdk/+/91108
Change-Id: I5279a73f56fde26a8d9068912b3e08829ceaa4a8
Reviewed-on: https://dart-review.googlesource.com/c/91981
Auto-Submit: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Régis Crelier <regis@google.com>
Commit-Queue: Régis Crelier <regis@google.com>
This reverts commit c1e6140323.
Reason for revert:
On Windows
.\out\DebugX64\run_vm_tests.exe --dfe=out\DebugX64\gen\kernel-service.dart.snapshot IsolateReload_TearOff_List_Set
fails with:
sdk\runtime\vm\raw_object.cc: 231: error: Size mismatch: -2459565876494606883 from class vs 112 from tags c0718
Original change's description:
> Reland: [vm] Add a library and class cache to KernelProgramInfo
>
> Relands with fixed locking to protect the caches.
>
> Original message:
> Also create fewer handles in the KernelLoader.
>
> GenKernelKernelReadAllBytecode improves a further
> 15-20% after removing the timeline event argument
> in the bytecode reader.
>
> fixes#34939
>
> Change-Id: I88c57bdb673392a7677c5c6bbb16cfed8c95c0d7
> Reviewed-on: https://dart-review.googlesource.com/c/81620
> Commit-Queue: Zach Anderson <zra@google.com>
> Reviewed-by: Alexander Markov <alexmarkov@google.com>
TBR=rmacnak@google.com,alexmarkov@google.com,zra@google.com,asiva@google.com
Change-Id: I4f3ff1866db2990b512b3ee7e59bf66bdc480dcc
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/81745
Reviewed-by: Zach Anderson <zra@google.com>
Commit-Queue: Zach Anderson <zra@google.com>
Relands with fixed locking to protect the caches.
Original message:
Also create fewer handles in the KernelLoader.
GenKernelKernelReadAllBytecode improves a further
15-20% after removing the timeline event argument
in the bytecode reader.
fixes#34939
Change-Id: I88c57bdb673392a7677c5c6bbb16cfed8c95c0d7
Reviewed-on: https://dart-review.googlesource.com/c/81620
Commit-Queue: Zach Anderson <zra@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Relands with locks to protect the caches.
Original message:
Also create fewer handles in the KernelLoader.
GenKernelKernelReadAllBytecode improves a further
15-20% after removing the timeline event argument
in the bytecode reader.
Change-Id: I6d796986a7773938e528271033385465e4f13468
Reviewed-on: https://dart-review.googlesource.com/c/79380
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Zach Anderson <zra@google.com>
Also create fewer handles in the KernelLoader.
GenKernelKernelReadAllBytecode improves a further
15-20% after removing the timeline event argument
in the bytecode reader.
Change-Id: Ida7e74129ccb4c997bb74e7bb51cdae053587254
Reviewed-on: https://dart-review.googlesource.com/c/78710
Commit-Queue: Zach Anderson <zra@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
This mainly prevents Strings that will eventually be turned into
symbols anyway from being allocated in new space.
Picks up a couple percent during kernel loading.
Possibly avoids triggering a scavenge?
Change-Id: Ide3dca13306981131e6152004e068c2154f9fc4c
Reviewed-on: https://dart-review.googlesource.com/77484
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Zach Anderson <zra@google.com>
We need to block interrupts while evaluating pragmas to prevert reentrant class finalization.
Original revision is in patchset 0.
Change-Id: I872cec4eaf4ca85567c9657c458ed39c8b2e30de
Cq-Include-Trybots: luci.dart.try:vm-kernel-win-release-x64-try, vm-kernel-optcounter-threshold-linux-release-x64-try, vm-kernel-precomp-linux-debug-x64-try, vm-kernel-precomp-linux-release-simarm-try, vm-kernel-precomp-linux-release-simarm64-try, vm-kernel-precomp-linux-release-x64-try, vm-kernel-precomp-win-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/73160
Commit-Queue: Samir Jindel <sjindel@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Until now the expression evaluation implementation was using normal kernel loader
functionality, which registered a new class with the isolate. Then the cid of that
class was set to kIllegalCid.
This caused the direct_subclasses/direct_implementors CHA information to contain a
class with kIllegalCid.
This CL fixes this by ensuring we never register the libraries/classes
created for expression evaluation (which are not even used, they are an artifact
of how the expressions are encoded in kernel).
Issue https://github.com/flutter/flutter/issues/20255
Issue https://github.com/flutter/flutter/issues/20307
Change-Id: Ie6dd76c7ff696cd8adf4f27e9a072274afd90136
Reviewed-on: https://dart-review.googlesource.com/68681
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
This reverts commit 08f59e5de3.
There are no changes to take 3 except fixing merge conflicts.
Change-Id: I45d5a16274c80f469827c19f4e4f4ca724e85a67
Cq-Include-Trybots: luci.dart.try:vm-kernel-win-release-x64-try,vm-kernel-optcounter-threshold-linux-release-x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-linux-release-simarm-try,vm-kernel-precomp-linux-release-simarm64-try,vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-win-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/68363
Commit-Queue: Samir Jindel <sjindel@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
which was blocking the Dart SDK roll.
Revert "Take 3 for "[vm/kernel/precomp] Remove procedures from entry points files.""
This reverts commit 567109df7f.
Revert "[vm/precomp] Extend @pragma entry-points to classes."
This reverts commit 232698047c.
Change-Id: Ib63d1afb8a1c978be7ddf282af0e7d5547111cc3
Reviewed-on: https://dart-review.googlesource.com/67300
Reviewed-by: Alexander Markov <alexmarkov@google.com>
In certain cases StreamingFlowGraphBuilder was used not for building
flow graph, but as an advanced kernel reader. This CL extracts all such
functionality from StreamingFlowGraphBuilder and revises all such uses.
StreamingFlowGraphBuilder constructors without FlowGraphBuilder are
removed. After this CL StreamingFlowGraphBuilder is only used to build
flow graph, as intended.
Change-Id: I69b08e24d37f8f2f336bee85334af11be1639b24
Reviewed-on: https://dart-review.googlesource.com/64821
Reviewed-by: Régis Crelier <regis@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Zach Anderson <zra@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
This CL continues refactoring aimed at isolating StreamingFlowGraphBuilder.
StreamingFlowGraphBuilder dependency is removed from classes
* StreamingConstantEvaluator (it still depends on FlowGraphBuilder)
* ConstantHelper
* SimpleExpressionConverter
and several methods in object.cc.
StreamingConstantEvaluator and ConstantHelper are moved from
kernel_binary_flowgraph{.h, .cc} to a new source file
constant_evaluator{.h, .cc}. StreamingConstantEvaluator is renamed
to ConstantEvaluator.
KernelFingerprintHelper and KernelSourceFingerprintHelper are moved to
a new source file kernel_fingerprints{.h, .cc}.
Instances of kernel::FlowGraphBuilder no longer contain back reference
to a StreamingFlowGraphBuilder. In order to drop this circular dependency
TranslateFinallyFinalizers() is moved from FlowGraphBuilder to
StreamingFlowGraphBuilder.
Change-Id: Id550d22b3567dea9512328a900935bd6145a8107
Reviewed-on: https://dart-review.googlesource.com/64463
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Zach Anderson <zra@google.com>
Reviewed-by: Régis Crelier <regis@google.com>
After previous refactoring, StreamingScopeBuilder no longer depends on
StreamingFlowGraphBuilder.
This CL moves StreamingScopeBuilder to a separate file
vm/compiler/frontend/scope_builder{.h, .cc}.
Also:
* StreamingScopeBuilder is renamed to ScopeBuilder.
* IntMap is moved to hash_map.h.
* Minor cleanup of unused forward declarations and friend classes.
Change-Id: I5e780a53f37935c0a1d563d4fbaec3a92ae13d6a
Reviewed-on: https://dart-review.googlesource.com/63692
Reviewed-by: Zach Anderson <zra@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
When the constant transformation is enabled on annotations, we need to fix handling
of @ExternalName annotations in the kernel_loader to ensure that we are setting
is_external = false on native methods.
The original revision is in patchset 1.
# Test Plan
The only regression was on benchmarks, Golem results are pending.
Change-Id: Ib80bb9f532299056e770a3b378cc5ad9ee451f57
Reviewed-on: https://dart-review.googlesource.com/56960
Commit-Queue: Samir Jindel <sjindel@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
This reverts commit 3e50ea32b5.
Reason for revert: Severe performance regressions on many aot-v2 benchmarks.
Original change's description:
> [vm] Support definition of entry-points via @pragma('vm.extern') annotations.
>
> The `@pragma` annotations are evaluated by the constants transformation and
> visible to TFA and the precompiler, which match on the "options" field of the
> annotation to determine whether to mark the class/procedure as a root.
>
> This required enabling the transformation of annotation constants by default.
>
> # Test Plan
>
> The "vmservice_io.main" entry-point is removed from `main.cc` and annotated with
> `@pragma`. All precompiler tests will crash if "vmservice_io.main" is not
> available at runtime.
>
> Debug/release precompiler bots are visible in "cl-linux" button.
>
> Change-Id: I03c5d6ba7918672ed9905fcaee8dabe675a93a5d
> Reviewed-on: https://dart-review.googlesource.com/56660
> Commit-Queue: Samir Jindel <sjindel@google.com>
> Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
TBR=vegorov@google.com,alexmarkov@google.com,sjindel@google.com
Change-Id: I779c17d003659129a4b3fcf284423104948f60e2
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/56820
Reviewed-by: Samir Jindel <sjindel@google.com>
Commit-Queue: Samir Jindel <sjindel@google.com>
The `@pragma` annotations are evaluated by the constants transformation and
visible to TFA and the precompiler, which match on the "options" field of the
annotation to determine whether to mark the class/procedure as a root.
This required enabling the transformation of annotation constants by default.
# Test Plan
The "vmservice_io.main" entry-point is removed from `main.cc` and annotated with
`@pragma`. All precompiler tests will crash if "vmservice_io.main" is not
available at runtime.
Debug/release precompiler bots are visible in "cl-linux" button.
Change-Id: I03c5d6ba7918672ed9905fcaee8dabe675a93a5d
Reviewed-on: https://dart-review.googlesource.com/56660
Commit-Queue: Samir Jindel <sjindel@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Dart_LoadKernel is only used in Flutter engine and dart_runner, where it is always used as Dart_LoadKernel(Dart_ReadKernelBinary). These uses should be replaced with Dart_LoadScriptFromKernel.
Dart_LoadLibraryFromKernel is needed for loading split kernel files and implementing IsolateMirror.loadUri.
Change-Id: Ib505350eff53ec889406747f8f99393ebbdb4c7d
Reviewed-on: https://dart-review.googlesource.com/46220
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Zach Anderson <zra@google.com>
While working on #32274 I noticed that #32087 also impacted debugging
experience in the external debuggers like VS Code - IDE attempts to
load non-existent local sources for core libraries, instead of
asking Service for their source.
To resolve this I moved work-around we placed in stack trace formatting
futher up the flow into kernel loader, which will now rewrite urls
for all core library scripts as classes are being created.
Bug: 32087
Change-Id: I2175a041e5fdcc9958bfccb2af724c37c2435f94
Reviewed-on: https://dart-review.googlesource.com/43021
Reviewed-by: Jens Johansen <jensj@google.com>
Commit-Queue: Vyacheslav Egorov <vegorov@google.com>
Alexander Markov