This is the beginning of allowing us to - wherever we process kernel
data - make decisions based on the kernel version. Said another way,
we can actually start to change the kernel format without breaking
backwards compatibility in the VM.
Change-Id: Ied7b3d9fe72976c4f3c0895d02b6ab70e0f5dba9
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/111642
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Rationale:
Previously all constants were read from the constants table,
whether they were needed (at that point or at all) or not. This
CL replaces that eager reading with a lazy reading that makes
the raw bytes of the constants table available through
KernelProgramInfo and attaches a hashmap to KernelProgramInfo,
which maps offsets into the constant table to evaluated constants.
The maps starts empty and on a miss, the constant is evaluated,
but no sooner than when it it needed.
https://github.com/dart-lang/sdk/issues/36220
Change-Id: Ief4df3d70c950c4beb61d896632ce06cfd12d525
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/99088
Commit-Queue: Aart Bik <ajcbik@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Size of a large application:
Before: 26628600
After: 21480120 (-19.3%)
Size of snapshots:
isolate_snapshot_framework.bin
Before: 9322496
After: 6782976 (-27.2%)
isolate_snapshot_product_framework.bin
Before: 9166848
After: 6602752 (-27.9%)
Regressions in tests:
1) Test language_2/type_alias_equality_test/04 fails similarly to default mode, as VM
does not implement comparison of function types according to the specification.
Previously this test was passing as function types were canonicalized in bytecode,
which was not always correct. This CL fixes the problem with canonicalization of
function types in bytecode and the test starts failing again.
2) Tests standalone_2/entrypoints_verification_test, standalone_2/io/test_extension_test,
standalone_2/io/test_extension_fail_test fail as native extensions are not supported
in bytecode yet. These tests start passing after df5e7aac17,
which switched bytecode tests to kernel service (on x64), because kernel service doesn't
drop ASTs. This CL switches from reading AST library declarations to bytecode even if
AST is not removed, so tests fail again.
Change-Id: I8b7ba44bfa49d0b1599b2509553ff7c831a4e244
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/104700
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Régis Crelier <regis@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
This change renames Class::is_cycle_free to Class::is_declaration_loaded
and under the hood replaces cycle_free and type_finalized bits with
class loading state enum.
Also, added new assertions to check is_declaration_loaded().
Change-Id: Ib43e12731d0dc782e273be8e55912494e102cd79
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/104920
Reviewed-by: Régis Crelier <regis@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
This reverts commit ab6aeaa106.
Revert "[vm/compiler] Speed up the compiler part which deals with kernel reading up in DEBUG mode"
This reverts commit b316210d94.
Reason for revert: regression of snapshot sizes (DNO-599).
Original change's description:
> [vm/kernel] Use GC-tracked ExternalTypedData/TypedDataView for kernel buffers
>
> Until now we often leaked kernel buffers (e.g. hot reload buffers) because various
> objects were referencing ExternalTypedData objects pointing into the middle of
> c-allocated memory. This made it impossible for the GC to determine when the last
> reference is gone.
>
> This CL ensures that the actual buffers are *always* made available via
> ExternalTypedData and any inner pointers into it are created via TypedDataViews.
>
> The embedder guarantees to the free kernel buffers it has provided to:
> - Dart_CreateIsolateFromKernel
> - Dart_LoadScriptFromKernel
> - Dart_LoadLibraryFromKernel
> - Dart_SetDartLibrarySourcesKernel
> on isolate shutdown.
>
> All other kernel buffers will get a finalizer attached, which ensures the
> kernel buffers get freed by the GC once they are no longer referenced:
> - Kernel blobs for expression evaluation
> - Kernel blobs for Hot-Reload
> - Kernel blobs for cc tests
>
> Fixes https://github.com/dart-lang/sdk/issues/33973
> Fixes https://github.com/dart-lang/sdk/issues/36857
> Issue https://github.com/dart-lang/sdk/issues/37030
>
> Change-Id: I1cc410c94c0f4b229413e793728a261afcb10aaf
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/103130
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
> Commit-Queue: Martin Kustermann <kustermann@google.com>
TBR=kustermann@google.com,rmacnak@google.com
# Not skipping CQ checks because original CL landed > 1 day ago.
Change-Id: I49715d2400f4a5c8806b7d6a2912b7258f671a0a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/104343
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
Auto-Submit: Alexander Markov <alexmarkov@google.com>
In DEBUG mode during AOT compilation MetadataHelper::SetMetadataMappings
may spend considerable time verifying that node offsets are sorted
in kernel metadata. The problem is that there are a lot of MetadataHelpers
created (several per each compiled function).
This change moves verification of kernel metadata mappings to
the kernel loader, so it is performed only once per kernel program.
AOT gen_snapshot time on a small test:
Before: 11.680s
After: 9.041s
Change-Id: I1c0a676b0ed28fc5cfa756ca60159f914190fac0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/103486
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Aart Bik <ajcbik@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Until now we often leaked kernel buffers (e.g. hot reload buffers) because various
objects were referencing ExternalTypedData objects pointing into the middle of
c-allocated memory. This made it impossible for the GC to determine when the last
reference is gone.
This CL ensures that the actual buffers are *always* made available via
ExternalTypedData and any inner pointers into it are created via TypedDataViews.
The embedder guarantees to the free kernel buffers it has provided to:
- Dart_CreateIsolateFromKernel
- Dart_LoadScriptFromKernel
- Dart_LoadLibraryFromKernel
- Dart_SetDartLibrarySourcesKernel
on isolate shutdown.
All other kernel buffers will get a finalizer attached, which ensures the
kernel buffers get freed by the GC once they are no longer referenced:
- Kernel blobs for expression evaluation
- Kernel blobs for Hot-Reload
- Kernel blobs for cc tests
Fixes https://github.com/dart-lang/sdk/issues/33973
Fixes https://github.com/dart-lang/sdk/issues/36857
Issue https://github.com/dart-lang/sdk/issues/37030
Change-Id: I1cc410c94c0f4b229413e793728a261afcb10aaf
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/103130
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
This for instance allows the VM to include all patches
(e.g. the previously missing `runtime/lib/date_patch.dart`)
in what's returned from `LoadedScripts()`.
Over time this also allows for `LoadedScripts()` to be simplified and
become faster (it's currently something like O(m + m*n) where m is the
number of results in DictionaryIterator(*this) and n is the number of
returned scripts).
Change-Id: I4b5a6a0fe666b774fc0987d099ed02e81ac97b43
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/98660
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Kevin Millikin <kmillikin@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
By indexing all sources in a concatenated dill file and then using that
index to find the sources and line starts we can fix missing information
(leading to crashes when collecting coverage) in for instance circular
instances.
This reverts commit 57321c1590 and adds a
fix.
Change-Id: I7a91801fb318cad0218cf3101d6a15f1ec929175
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/98006
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
This reverts commit a7ab99dd84.
Reason for revert: <INSERT REASONING HERE>
Original change's description:
> [vm/kernel] Index all source and line starts before using them
>
> By indexing all sources in a concatenated dill file and then using that
> index to find the sources and line starts we can fix missing information
> (leading to crashes when collecting coverage) in for instance circular
> instances.
>
> Change-Id: I454cff57f84f21de72f6486e1cecf15d1389a498
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/96901
> Commit-Queue: Jens Johansen <jensj@google.com>
> Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
TBR=vegorov@google.com,jensj@google.com
# Not skipping CQ checks because original CL landed > 1 day ago.
Change-Id: Iea75c9dd56e74fa0e98c90ea6841ed66b39d40cc
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/97941
Reviewed-by: Keerti Parthasarathy <keertip@google.com>
Commit-Queue: Keerti Parthasarathy <keertip@google.com>
This change replaces kernel AST declarations of fields and functions
with bytecode declarations.
Size of dilp files is reduced by 11-12%.
Startup latency:
Time to the first full frame: 1.945s -> 1.687s
FinalizeClass: 554ms -> 277ms
FinishClassLoading: 296ms -> 156ms
There are following regressions in bytecode mode, which will be fixed
in future:
* dart:mirrors are not supported yet (implementation of mirrors relies
on reading kernel AST in certain cases).
As the result, lib_2/mirrors/* tests fail.
* native extensions are not supported yet (annotations on libraries
and classes in AST are cleaned up as they could reference members
which are now removed from AST).
As the result, standalone_2/entrypoints_verification_test test fails.
* language_2/spread_collections/const_error_test/* tests fail
due to https://github.com/dart-lang/sdk/issues/36286.
Change-Id: I5130f401fd7b84038b136136e7ccc1a6e51b6cea
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/97561
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
By indexing all sources in a concatenated dill file and then using that
index to find the sources and line starts we can fix missing information
(leading to crashes when collecting coverage) in for instance circular
instances.
Change-Id: I454cff57f84f21de72f6486e1cecf15d1389a498
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/96901
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Rationale:
The problem was that we were evaluating field
initializers while still loading the constant
table. This hit two asserts (1) having pending
potential_natives() when entering codegen and
(2) not being able to evaluate constants
since the table has not been loaded yet. The
solution is to simply postpone enum field
initialization (forced to avoid hot reload
issues) while loading the constant table
until right after the table is ready.
https://github.com/dart-lang/sdk/issues/36153
Change-Id: I5060476e5b3b49e555cfc507fb398ef0144b44d0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/96844
Commit-Queue: Aart Bik <ajcbik@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
- Added `Dart_SetDartLibrarySourcesKernel` to the Dart embedding API.
- vm_platform_strong.dill now contains the Dart SDK sources
- If vm_platform_strong.dill is linked into the Dart binary, the first
request for a Dart SDK Script object from the VM service will load the
sources for the script from the linked in kernel buffer.
Change-Id: I664abe31f9378d25ec79c21edce0b237a278495d
Reviewed-on: https://dart-review.googlesource.com/c/93375
Commit-Queue: Ben Konyi <bkonyi@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
This is a re-land of "[vm/extensions] Fix Loader::ReloadNativeExtensions(...)."
The native extension test was failing in hot-reload: this has been fixed.
Change-Id: Ic18da8cfbc6a3dba8aac09d654dce616487a67d7
Reviewed-on: https://dart-review.googlesource.com/c/92800
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Auto-Submit: Samir Jindel <sjindel@google.com>
The script URLs for core libraries were being recorded as follows
- dart:core for org-dartlang-sdk:///sdk/lib/core/core.dart
- dart:core/runtime/libcore_patch.dart for org-dartlang-sdk:///runtime/lib/core_patch.dart
- dart:io/runtime/binfilter_patch.dart for org-dartlang-sdk:///runtime/bin/filter_patch.dart
Now we record them as
- dart:core/core.dart for org-dartlang-sdk:///sdk/lib/core/core.dart
- dart:core/runtime/lib/core_patch.dart for org-dartlang-sdk:///runtime/lib/core_patch.dart
- dart:io/runtime/bin/filter_patch.dart for org-dartlang-sdk:///runtime/bin/filter_patch.dart
Change-Id: I645939e7ba937034471044fb492a5ecdb9a50bf5
Reviewed-on: https://dart-review.googlesource.com/c/92922
Reviewed-by: Ben Konyi <bkonyi@google.com>
Commit-Queue: Siva Annamalai <asiva@google.com>
Prototype for `dart:ffi` on Linux/MacOS x64 in JIT mode.
`dart:ffi` is experimental and its API is likely to change in the future.
Progress and design decisions are tracked in https://github.com/dart-lang/sdk/projects/13
issue: https://github.com/dart-lang/sdk/issues/34452
Change-Id: Ifa4566388e42c8757f154741d11e303465ef305d
Cq-Include-Trybots: luci.dart.try:vm-kernel-optcounter-threshold-linux-release-x64-try, vm-kernel-precomp-linux-debug-x64-try, vm-kernel-precomp-linux-release-simarm-try, vm-kernel-precomp-linux-release-simarm64-try, vm-kernel-precomp-linux-release-x64-try, vm-kernel-precomp-mac-release-simarm64-try, vm-kernel-precomp-win-release-x64-try, vm-kernel-mac-debug-x64-try, vm-kernel-asan-linux-release-x64
Reviewed-on: https://dart-review.googlesource.com/c/80124
Reviewed-by: Samir Jindel <sjindel@google.com>
Auto-Submit: Daco Harkes <dacoharkes@google.com>
In --compile-all mode with CL [1] when NoSuchMethodError was finalized
the following happened:
ClassFinalizer::FinalizeClass calls KernelLoader::FinishLoading,
which calls KernelLoader::LoadProcedure to load precedures,
which calls Library::GetMetadata, which runs Dart code and triggers
background compilation; BackgroundCompiler::Start calls finalization of
NoSuchMethodError class recursively.
FinalizeClass calls FinishLoading for the second time, but this time
background compilation is not invoked, FinishLoading completes
and all members are finalized. After returning from recursive finalization,
the first FinishLoading proceeds and overwrites already finalized
functions with freshly loaded non-finalized functions.
As class is already marked as finalized, functions are not finalized
for the second time, causing crash later.
This CL fixes 2 problems related to recursive class loading:
* When fields are loaded, they should replace previous fields
rather than appending them. This ensures that recursive loading
won't add several duplicate fields (potential problem).
* IsLoaded flag is added to a class and checked after loading each
procedure. If class happens to be loaded already, then FinishLoading
should return immediately, discarding its loaded functions.
This prevents overwriting finalized functions with freshly loaded.
Also, this change adds guard against repeated class loading (may happen
during recursive attempts of class finalization).
[1] https://dart-review.googlesource.com/c/sdk/+/91108
Change-Id: I5279a73f56fde26a8d9068912b3e08829ceaa4a8
Reviewed-on: https://dart-review.googlesource.com/c/91981
Auto-Submit: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Régis Crelier <regis@google.com>
Commit-Queue: Régis Crelier <regis@google.com>
- Move compiler pass events to a new CompilerVerbose stream
- Remove serialization phase events
- Add class name to class finalization event (lost along with finalization events in recent cleanups of finalization)
- Add event for kernel loading
Change-Id: Ie72bced978400ea174c1551c961baa55c691b019
Reviewed-on: https://dart-review.googlesource.com/c/90883
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Zach Anderson <zra@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Jens Johansen