As part of deprecating support for native extensions we are also
migrating away from legacy VM-specific `native 'name'` syntax
towards metadata based encoding which does not require any special
syntax.
This CL is a step 1 in migration:
- introduces support for `@pragma('vm:external-name', 'name')`
which serves as a direct replacement for `native 'name'`;
- all core libraries and tests are migrated to use the annotation;
Once this CL lands and rolls we will edit internal and external embedders
to eliminate uses of the native keyword (step 2) and finally remove
support for native keyword across our parsers (step 3).
TEST=ci
Bug: https://github.com/dart-lang/sdk/issues/28791
Change-Id: Id6dea878db82dd4fd81149243c425b5c5dc6df86
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/212461
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Lasse R.H. Nielsen <lrn@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Now that support for native extensions (aka dart-ext:*) has been
removed, the kernel loader is guaranteed not to call out to embedder
anymore (it did so before for loading shared libraries).
This allos us to clean up some code as well as extend the scope of our
NoActiveIsolateScope.
TEST=ci
Change-Id: I5c8c9ee47298fdd316583ba3aed74772c1c7783f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/212296
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
In the codebase, we have several different interfaces for instance
methods that return a hash:
- uword Hash() const;
- intptr_t Hash() const;
- uint32_t Hash() const;
- intptr_t Hashcode() const;
This CL standardizes on `uword Hash() const` and adjusts any related
functions to match.
TEST=Existing test suites, as this is an internal refactoring.
Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-linux-debug-x64-try
Change-Id: If2cbce57f3fae0f0d24031b6e324f0323c965f41
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/195067
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Since loading kernel will affect all isolates within a group, there
should be nothing isolate-specific inside the kernel loader. To ensure
we don't have (or introduce) any accidental uses, we add a
NoActiveIsolateScope to the main parts of kernel loading.
Only loading of native extensions - done by the kernel loader -
currently requires an active isolate. The reason for this is that
loading native extensions happen by calling out to embedder, which calls
back into the VM using the our embedding API (which currently requires
an active isolate)
Issue https://github.com/dart-lang/sdk/issues/36097
TEST=Refactoring of existing implementation.
Change-Id: I96e64dbfe7148b76b8fa006fe5dbde8c1f904504
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/184269
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
When handling hot reload via library tag handler's Kernel tag
VM must guarantee that external typed data it receives from the
embedder stays alive for as long as VM has any structures
referencing this kernel binary. This is achieved by attaching
original typed data to KernelProgramInfo objects containing
the views into it.
Unfortunately kernel::Reader APIs were somewhat unsafe: allowing
to construct reader object from Program's raw buffer and forget to
set the link betwen the reader and original typed data.
This created a situation where reloading using a multicomponent
Kernel binary would result in KernelProgramInfo objects without
link to the original typed data, which in turn leads to premature
finalization of external typed data and subsequent crashes
when trying to use delete kernel binary.
This CL reworks kernel::Reader API (by introducing kernel::ProgramBinary
wrapper) to make it safer and make sure that connection is preserved.
TEST=vm/cc/IsolateReload_RegressB179030011
Bug: b/179030011
Change-Id: I05f8c31c3cb8e67de6e94a20d9501a5f476b7e27
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/182280
Commit-Queue: Vyacheslav Egorov <vegorov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Currently we have things called XPtr which are not what you get from ptr().
Old world:
handle->raw() returns RawObject* (tagged)
raw_obj->ptr() returns RawObject* (untagged)
After 6fe15f6df9:
handle->raw() returns ObjectPtr
obj_ptr->ptr() returns ObjectLayout*
New world:
handle->ptr() returns ObjectPtr
obj_ptr->untag() returns UntaggedObject*
TEST=ci
Change-Id: I6c7f34014cf20737607caaf84979838300d12df2
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/149367
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
As part of making the compiler and other subsystems independent
of `Isolate` we have to move various state from `Isolate` to
`IsolateGroup`.
The class_table and object_store were already moved to `IsolateGroup`.
This CL only replaces usages of `Isolate::{object_store,class_table}`
with the equivalent in `IsolateGroup`.
Issue https://github.com/dart-lang/sdk/issues/36097
TEST=Pure refactoring - relying on existing test coverage.
Change-Id: I34a0682d715b054d6c5faff077a513980f59a348
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/177126
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
As part of making lightweight isolates work in JIT, we will need to
ensure that whenever a new static field is loaded, it will get a new
slot on every isolate and the possibly initial static value is set in
all isolates's field state.
In order to achieve this we first move field registration out of
Field::New*() - to the place where the field is created and the initial
static field value is known.
In a future CL we will make a field registration be performed on all
isolates in a group.
TEST=CL is internal refactoring, existing test coverage.
Issue https://github.com/dart-lang/sdk/issues/36097
Change-Id: I802ae55b841be173733dbd0ad3dbffeebd5c1889
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/173260
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Loading unit metadata is expanded to create an object representing each loading unit, and each library to reference the loading unit it is a part of.
To be used at compile-time to help assign objects to snapshots.
To be used at runtime to track which units have been loading and the base objects for cross-snapshot references.
Bug: https://github.com/dart-lang/sdk/issues/41974
Change-Id: Icd3fa95ffd5949bd8f7f84a6f750c4c515de0dba
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/151865
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Static methods and constructors may receive unboxed parameters.
This change adds a bitmap to the RawFunction object indicating
which parameters are unboxed and their types. Also, the stack frame
bitmap was updated to include new bits for those parameters.
The ArgumentsDescriptor object was updated to store not only the number
of arguments, but also their size, since 32-bit platforms would use two
words for each int64 and double value.
In addition, static methods that return non-nullable integers
and doubles may return these values unboxed.
Dart AOT x64:
TypedData.Uint8ClampedListViewVarBench +11.73%
DeltaBlue +9.32%
Meteor (Intel Xeon) +4.64%
Flutter release android-armv8:
velocity_tracker_bench -2.32% (less is better)
flutter_gallery_readonlydata_size +0.18%
flutter_gallery_total_size +0.13%
flutter_gallery_instructions_size +0.12%
Issue https://github.com/dart-lang/sdk/issues/33549
Change-Id: I97e310f8f30213920af2333fbdedf193d03db5c3
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/135313
Commit-Queue: Victor Agnez Lima <victoragnez@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
This paves the way for removing the old constant evaluator from the VM,
by no longer supporting versions without front-end constant evaluation.
For the front-end test that tests old kernel versions on the newest VM,
the dill files for versions before 29 are deleted.
Version 29 is from 2019-08-16: eae781c73e
Change-Id: Icb472bffc764d6e37ed1349a0cb4ff281e0e1b21
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/123739
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Aske Simon Christensen <askesc@google.com>
This is the beginning of allowing us to - wherever we process kernel
data - make decisions based on the kernel version. Said another way,
we can actually start to change the kernel format without breaking
backwards compatibility in the VM.
Change-Id: Ied7b3d9fe72976c4f3c0895d02b6ab70e0f5dba9
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/111642
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Rationale:
Previously all constants were read from the constants table,
whether they were needed (at that point or at all) or not. This
CL replaces that eager reading with a lazy reading that makes
the raw bytes of the constants table available through
KernelProgramInfo and attaches a hashmap to KernelProgramInfo,
which maps offsets into the constant table to evaluated constants.
The maps starts empty and on a miss, the constant is evaluated,
but no sooner than when it it needed.
https://github.com/dart-lang/sdk/issues/36220
Change-Id: Ief4df3d70c950c4beb61d896632ce06cfd12d525
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/99088
Commit-Queue: Aart Bik <ajcbik@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Size of a large application:
Before: 26628600
After: 21480120 (-19.3%)
Size of snapshots:
isolate_snapshot_framework.bin
Before: 9322496
After: 6782976 (-27.2%)
isolate_snapshot_product_framework.bin
Before: 9166848
After: 6602752 (-27.9%)
Regressions in tests:
1) Test language_2/type_alias_equality_test/04 fails similarly to default mode, as VM
does not implement comparison of function types according to the specification.
Previously this test was passing as function types were canonicalized in bytecode,
which was not always correct. This CL fixes the problem with canonicalization of
function types in bytecode and the test starts failing again.
2) Tests standalone_2/entrypoints_verification_test, standalone_2/io/test_extension_test,
standalone_2/io/test_extension_fail_test fail as native extensions are not supported
in bytecode yet. These tests start passing after df5e7aac17,
which switched bytecode tests to kernel service (on x64), because kernel service doesn't
drop ASTs. This CL switches from reading AST library declarations to bytecode even if
AST is not removed, so tests fail again.
Change-Id: I8b7ba44bfa49d0b1599b2509553ff7c831a4e244
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/104700
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Régis Crelier <regis@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
This reverts commit ab6aeaa106.
Revert "[vm/compiler] Speed up the compiler part which deals with kernel reading up in DEBUG mode"
This reverts commit b316210d94.
Reason for revert: regression of snapshot sizes (DNO-599).
Original change's description:
> [vm/kernel] Use GC-tracked ExternalTypedData/TypedDataView for kernel buffers
>
> Until now we often leaked kernel buffers (e.g. hot reload buffers) because various
> objects were referencing ExternalTypedData objects pointing into the middle of
> c-allocated memory. This made it impossible for the GC to determine when the last
> reference is gone.
>
> This CL ensures that the actual buffers are *always* made available via
> ExternalTypedData and any inner pointers into it are created via TypedDataViews.
>
> The embedder guarantees to the free kernel buffers it has provided to:
> - Dart_CreateIsolateFromKernel
> - Dart_LoadScriptFromKernel
> - Dart_LoadLibraryFromKernel
> - Dart_SetDartLibrarySourcesKernel
> on isolate shutdown.
>
> All other kernel buffers will get a finalizer attached, which ensures the
> kernel buffers get freed by the GC once they are no longer referenced:
> - Kernel blobs for expression evaluation
> - Kernel blobs for Hot-Reload
> - Kernel blobs for cc tests
>
> Fixes https://github.com/dart-lang/sdk/issues/33973
> Fixes https://github.com/dart-lang/sdk/issues/36857
> Issue https://github.com/dart-lang/sdk/issues/37030
>
> Change-Id: I1cc410c94c0f4b229413e793728a261afcb10aaf
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/103130
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
> Commit-Queue: Martin Kustermann <kustermann@google.com>
TBR=kustermann@google.com,rmacnak@google.com
# Not skipping CQ checks because original CL landed > 1 day ago.
Change-Id: I49715d2400f4a5c8806b7d6a2912b7258f671a0a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/104343
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
Auto-Submit: Alexander Markov <alexmarkov@google.com>
Until now we often leaked kernel buffers (e.g. hot reload buffers) because various
objects were referencing ExternalTypedData objects pointing into the middle of
c-allocated memory. This made it impossible for the GC to determine when the last
reference is gone.
This CL ensures that the actual buffers are *always* made available via
ExternalTypedData and any inner pointers into it are created via TypedDataViews.
The embedder guarantees to the free kernel buffers it has provided to:
- Dart_CreateIsolateFromKernel
- Dart_LoadScriptFromKernel
- Dart_LoadLibraryFromKernel
- Dart_SetDartLibrarySourcesKernel
on isolate shutdown.
All other kernel buffers will get a finalizer attached, which ensures the
kernel buffers get freed by the GC once they are no longer referenced:
- Kernel blobs for expression evaluation
- Kernel blobs for Hot-Reload
- Kernel blobs for cc tests
Fixes https://github.com/dart-lang/sdk/issues/33973
Fixes https://github.com/dart-lang/sdk/issues/36857
Issue https://github.com/dart-lang/sdk/issues/37030
Change-Id: I1cc410c94c0f4b229413e793728a261afcb10aaf
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/103130
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
This for instance allows the VM to include all patches
(e.g. the previously missing `runtime/lib/date_patch.dart`)
in what's returned from `LoadedScripts()`.
Over time this also allows for `LoadedScripts()` to be simplified and
become faster (it's currently something like O(m + m*n) where m is the
number of results in DictionaryIterator(*this) and n is the number of
returned scripts).
Change-Id: I4b5a6a0fe666b774fc0987d099ed02e81ac97b43
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/98660
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Kevin Millikin <kmillikin@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
By indexing all sources in a concatenated dill file and then using that
index to find the sources and line starts we can fix missing information
(leading to crashes when collecting coverage) in for instance circular
instances.
This reverts commit 57321c1590 and adds a
fix.
Change-Id: I7a91801fb318cad0218cf3101d6a15f1ec929175
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/98006
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
This reverts commit a7ab99dd84.
Reason for revert: <INSERT REASONING HERE>
Original change's description:
> [vm/kernel] Index all source and line starts before using them
>
> By indexing all sources in a concatenated dill file and then using that
> index to find the sources and line starts we can fix missing information
> (leading to crashes when collecting coverage) in for instance circular
> instances.
>
> Change-Id: I454cff57f84f21de72f6486e1cecf15d1389a498
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/96901
> Commit-Queue: Jens Johansen <jensj@google.com>
> Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
TBR=vegorov@google.com,jensj@google.com
# Not skipping CQ checks because original CL landed > 1 day ago.
Change-Id: Iea75c9dd56e74fa0e98c90ea6841ed66b39d40cc
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/97941
Reviewed-by: Keerti Parthasarathy <keertip@google.com>
Commit-Queue: Keerti Parthasarathy <keertip@google.com>
This change replaces kernel AST declarations of fields and functions
with bytecode declarations.
Size of dilp files is reduced by 11-12%.
Startup latency:
Time to the first full frame: 1.945s -> 1.687s
FinalizeClass: 554ms -> 277ms
FinishClassLoading: 296ms -> 156ms
There are following regressions in bytecode mode, which will be fixed
in future:
* dart:mirrors are not supported yet (implementation of mirrors relies
on reading kernel AST in certain cases).
As the result, lib_2/mirrors/* tests fail.
* native extensions are not supported yet (annotations on libraries
and classes in AST are cleaned up as they could reference members
which are now removed from AST).
As the result, standalone_2/entrypoints_verification_test test fails.
* language_2/spread_collections/const_error_test/* tests fail
due to https://github.com/dart-lang/sdk/issues/36286.
Change-Id: I5130f401fd7b84038b136136e7ccc1a6e51b6cea
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/97561
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
By indexing all sources in a concatenated dill file and then using that
index to find the sources and line starts we can fix missing information
(leading to crashes when collecting coverage) in for instance circular
instances.
Change-Id: I454cff57f84f21de72f6486e1cecf15d1389a498
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/96901
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
- Added `Dart_SetDartLibrarySourcesKernel` to the Dart embedding API.
- vm_platform_strong.dill now contains the Dart SDK sources
- If vm_platform_strong.dill is linked into the Dart binary, the first
request for a Dart SDK Script object from the VM service will load the
sources for the script from the linked in kernel buffer.
Change-Id: I664abe31f9378d25ec79c21edce0b237a278495d
Reviewed-on: https://dart-review.googlesource.com/c/93375
Commit-Queue: Ben Konyi <bkonyi@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
The script URLs for core libraries were being recorded as follows
- dart:core for org-dartlang-sdk:///sdk/lib/core/core.dart
- dart:core/runtime/libcore_patch.dart for org-dartlang-sdk:///runtime/lib/core_patch.dart
- dart:io/runtime/binfilter_patch.dart for org-dartlang-sdk:///runtime/bin/filter_patch.dart
Now we record them as
- dart:core/core.dart for org-dartlang-sdk:///sdk/lib/core/core.dart
- dart:core/runtime/lib/core_patch.dart for org-dartlang-sdk:///runtime/lib/core_patch.dart
- dart:io/runtime/bin/filter_patch.dart for org-dartlang-sdk:///runtime/bin/filter_patch.dart
Change-Id: I645939e7ba937034471044fb492a5ecdb9a50bf5
Reviewed-on: https://dart-review.googlesource.com/c/92922
Reviewed-by: Ben Konyi <bkonyi@google.com>
Commit-Queue: Siva Annamalai <asiva@google.com>
In --compile-all mode with CL [1] when NoSuchMethodError was finalized
the following happened:
ClassFinalizer::FinalizeClass calls KernelLoader::FinishLoading,
which calls KernelLoader::LoadProcedure to load precedures,
which calls Library::GetMetadata, which runs Dart code and triggers
background compilation; BackgroundCompiler::Start calls finalization of
NoSuchMethodError class recursively.
FinalizeClass calls FinishLoading for the second time, but this time
background compilation is not invoked, FinishLoading completes
and all members are finalized. After returning from recursive finalization,
the first FinishLoading proceeds and overwrites already finalized
functions with freshly loaded non-finalized functions.
As class is already marked as finalized, functions are not finalized
for the second time, causing crash later.
This CL fixes 2 problems related to recursive class loading:
* When fields are loaded, they should replace previous fields
rather than appending them. This ensures that recursive loading
won't add several duplicate fields (potential problem).
* IsLoaded flag is added to a class and checked after loading each
procedure. If class happens to be loaded already, then FinishLoading
should return immediately, discarding its loaded functions.
This prevents overwriting finalized functions with freshly loaded.
Also, this change adds guard against repeated class loading (may happen
during recursive attempts of class finalization).
[1] https://dart-review.googlesource.com/c/sdk/+/91108
Change-Id: I5279a73f56fde26a8d9068912b3e08829ceaa4a8
Reviewed-on: https://dart-review.googlesource.com/c/91981
Auto-Submit: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Régis Crelier <regis@google.com>
Commit-Queue: Régis Crelier <regis@google.com>
This reverts commit c1e6140323.
Reason for revert:
On Windows
.\out\DebugX64\run_vm_tests.exe --dfe=out\DebugX64\gen\kernel-service.dart.snapshot IsolateReload_TearOff_List_Set
fails with:
sdk\runtime\vm\raw_object.cc: 231: error: Size mismatch: -2459565876494606883 from class vs 112 from tags c0718
Original change's description:
> Reland: [vm] Add a library and class cache to KernelProgramInfo
>
> Relands with fixed locking to protect the caches.
>
> Original message:
> Also create fewer handles in the KernelLoader.
>
> GenKernelKernelReadAllBytecode improves a further
> 15-20% after removing the timeline event argument
> in the bytecode reader.
>
> fixes#34939
>
> Change-Id: I88c57bdb673392a7677c5c6bbb16cfed8c95c0d7
> Reviewed-on: https://dart-review.googlesource.com/c/81620
> Commit-Queue: Zach Anderson <zra@google.com>
> Reviewed-by: Alexander Markov <alexmarkov@google.com>
TBR=rmacnak@google.com,alexmarkov@google.com,zra@google.com,asiva@google.com
Change-Id: I4f3ff1866db2990b512b3ee7e59bf66bdc480dcc
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/81745
Reviewed-by: Zach Anderson <zra@google.com>
Commit-Queue: Zach Anderson <zra@google.com>
Relands with fixed locking to protect the caches.
Original message:
Also create fewer handles in the KernelLoader.
GenKernelKernelReadAllBytecode improves a further
15-20% after removing the timeline event argument
in the bytecode reader.
fixes#34939
Change-Id: I88c57bdb673392a7677c5c6bbb16cfed8c95c0d7
Reviewed-on: https://dart-review.googlesource.com/c/81620
Commit-Queue: Zach Anderson <zra@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Relands with locks to protect the caches.
Original message:
Also create fewer handles in the KernelLoader.
GenKernelKernelReadAllBytecode improves a further
15-20% after removing the timeline event argument
in the bytecode reader.
Change-Id: I6d796986a7773938e528271033385465e4f13468
Reviewed-on: https://dart-review.googlesource.com/c/79380
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Zach Anderson <zra@google.com>
Vyacheslav Egorov