Commit Graph

132 Commits

Author SHA1 Message Date
Ryan Macnak d36adbacaf [vm] Remove the VM isolate.
The former contents of the VM isolate are now included into each isolate group. This makes each isolate group's heap independent, and in particular allows each heap to be allocated to a separate pointer cage (not done in this CL).

The duplicated stubs that allowed PC relative calls are removed, since the originals can now be the target of PC relative calls.

The bootstrapping needing to load an AppJIT or AppAOT snapshot is reduced to allocating the oddballs. The code is entirely dropped in the AOT runtime, but the JIT runtime still has it to allow for flags to affect the compilation of the stub code. Further refactoring might be able to remove this for the JIT runtime too, with only gen_snapshot knowing how to bootstrap.

Class serialization no longer distinguishes predefined classes.

The page containing null is marked as never-evacuate. null, false and true must not move because the compiler relies on their low bits having certain patterns for some optimizations. (Previously, the entire VM isolate heap never moved.)

Compaction is disabled for IA32. Due to register pressure, some stub calls must not use a scratch register and embed the address of Code.

The page containing the call-through-safepoint stub is frozen when running with --write-protect-code and the stub is created at runtime (instead of loaded from an AppJIT or AppAOT snapshot). This stub must remain executable even during a safepoint, as a foreign call might during return during a safepoint and only block after the stub directs it to the runtime.

The snapshot symbols are renamed to kDartSnapshotData and kDartSnapshotText. There is no need to distinguish the VM isolate's snapshot, and snaphots are per isolate group not per isolate. Aliases with the old names are added to ease migration.

Some global flags that were automatically set based on the VM isolate's snapshot are now isolate group flags and automatically set by the isolate group's snapshot.

TEST=ci
Change-Id: Iee82016057d609112e9b021d178fc3d4d18b5044
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/500621
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Tess Strickland <sstrickl@google.com>
SLSA-Policy-Verified: SLSA Policy Verification Service <devtools-gerritcodereview-exitgate@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
2026-05-18 11:35:03 -07:00
Ryan Macnak 9e28e19002 [test] Synthetic versions of the mutator-marker race.
TEST=vm/cc/MutatorMarkerRace_*
Bug: https://github.com/dart-lang/sdk/issues/56895
Change-Id: I3e3f25b2c43aec09a409377f77c6d2fec287bccb
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/390263
Reviewed-by: Siva Annamalai <asiva@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
2024-10-24 03:24:08 +00:00
Ryan Macnak ea9eb67d54 Roll Clang to 3809e20afc68d7d03821f0ec59b928dcf9befbf4.
Re-run clang-format.

TEST=ci
Change-Id: Ic277ea6baefa42bcc49ebce4f1099b910066cd4e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/370680
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
2024-06-10 20:38:30 +00:00
Martin Kustermann 9bdd2ff1de [vm] Cleanup unused include, unnecessary fields on [Thread], rename mutator thread, ensure setjmp drains sticky error
Some cleanups factored out of a larger CL (which refactors enter/exit of threads):

  * remove unused `#include "vm/thread_registry.h"`
  * remove unused/unnecessary fields from [Thread] object
  * rename IsMutator() -> IsDartMutator()
  * make tests using setjmp() drain the sticky error
    => to ensure there's no sticky error on isolate shutdown

TEST=ci

Change-Id: I53935e8bd0628ab3768627d6d5e01c3f0e3a57ad
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/296582
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
2023-04-20 17:42:06 +00:00
Martin Kustermann fb43937e20 [vm] Make Thread::IsAtSafepoint() only be true if thread is blocked at a "safe" point
During safepoint we can distinguish between

  * owner of the safepoint operation (which is running code)
  * everyone else (which are all blocked

Currently `Thread::IsAtSafepoint()` will return true for both. Since the
thread owning the safepoint operation is running, it's not actually
guaranteed that it's at "safe" point (e.g. to GC or to deopt) - it
really depends on what it's doing.

=> This CL will change it so that only actually parked threads will
have `Thread::IsAtSafepoint()`.

In order to do that we change varrious usages of `IsAtSafepoint()` to be
more precise:

  * `Thread::OwnsSafepoint()`: True if this thread owns the
    active safepoint. The thread is running.

  * `Thread::OwnsGCSafepoint()`: True if the active safepoint is a GC
    (or Deopt) safepoint and this thread owns it. The thread is running.

  * `Thread::OwnsDeoptSafepoint()`: True if the active safepoint is a
    Deopt safepoint and this thread owns it. The thread is running.

  * `Thread::CanAcquireSafepointLocks()`: True if the thread is allowed
    to acquire safepoint locks.

  * `Thread::IsAtSafepoint()`: true if this thread is parked at a
    safepoint

TEST=ci

Change-Id: I1a5a6727e84843ae79e0a344c438da19b7d6d916
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/295781
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
2023-04-19 08:47:31 +00:00
Martin Kustermann abeb88a038 [vm] Refactoring to existing safepoint mechanism
This CL makes some refactorings to the current safepoint mechanism:

  * When owning safepoint level L we set current thread to be only at
    safepoint level L (not Thread::Current()->current_safepoint_level()).
    This ensures that after [WaitUntilThreadsReachedSafepointLevel] all
    other threads are actually parked.

  * When having nested safepoint scopes we increase operation count on
    the level we own and all nested levels. This will (in later CL) allow
    detection which closest scope we're in.

TEST=ci

Change-Id: Iffb2e9f4eea817a381acbd7a771bc75f5a89877b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/295541
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
2023-04-19 07:29:51 +00:00
Ryan Macnak f9a6a5bdd2 [vm] Update NULL to nullptr in runtime/vm.
TEST=build
Change-Id: I2834ef7cf7cb7c8770f8167a2438cbedcee5c623
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/292063
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
2023-04-10 18:15:12 +00:00
Ben Konyi c927840920 [ VM ] Add support for samping heap profiler in PRODUCT mode
Requires INCLUDE_SAMPLING_HEAP_PROFILER to be defined.

TEST=DartAPI_HeapSampling_*

Change-Id: I1c95be4747b295823a8fae1f369f9dc5d95a274e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/290620
Commit-Queue: Ben Konyi <bkonyi@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
2023-03-24 14:02:14 +00:00
Ben Konyi 6f633c364a Reland "[ VM ] Add support for sampling old space allocations"
This reverts commit 0bd5a2b05e.

Fixes https://github.com/dart-lang/sdk/issues/51023

TEST=CQ

Change-Id: I4b643976a59adba8bd228a96ac75ae25ef8e206c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/279120
Commit-Queue: Ben Konyi <bkonyi@google.com>
Reviewed-by: Derek Xu <derekx@google.com>
2023-01-16 16:28:43 +00:00
Tess Strickland 0bd5a2b05e Revert "[ VM ] Add support for sampling old space allocations"
This reverts commit 6e4c0c2196.

Reason for revert: Broke ASAN builds

See https://github.com/dart-lang/sdk/issues/51023

Original change's description:
> [ VM ] Add support for sampling old space allocations
>
> TEST=DartAPI_HeapSampling
>
> Change-Id: Idf26fca6ae59720557d73627f727b00229dd1e9c
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/273860
> Reviewed-by: Siva Annamalai <asiva@google.com>
> Commit-Queue: Ben Konyi <bkonyi@google.com>

# Not skipping CQ checks because original CL landed > 1 day ago.

Change-Id: Id704f9070c430234b7cd78d8c2b67a5ed3e503fe
Cq-Include-Trybots: luci.dart.try:vm-kernel-asan-linux-release-x64-try,vm-kernel-precomp-asan-linux-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/279100
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Tess Strickland <sstrickl@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
2023-01-16 11:42:45 +00:00
Ben Konyi 6e4c0c2196 [ VM ] Add support for sampling old space allocations
TEST=DartAPI_HeapSampling

Change-Id: Idf26fca6ae59720557d73627f727b00229dd1e9c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/273860
Reviewed-by: Siva Annamalai <asiva@google.com>
Commit-Queue: Ben Konyi <bkonyi@google.com>
2023-01-13 19:06:37 +00:00
Vyacheslav Egorov 3aa7a9e2b4 [vm] Detect when class table overflows
We only have 16bits for cids so class table
can't contain more than 65535 classes.

TEST=ci

Bug: b/255934984
Change-Id: I79d45f4f28399b0a61293ace40b8718f2569ef3c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/266320
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
2022-10-27 22:15:32 +00:00
Vyacheslav Egorov 4a4eedd860 [vm] Clean up ClassTable
* Merge ClassTable and SharedClassTable back together;
* Simplify handling of multiple arrays growing in sync;
* Refactor how reload deals with ClassTable.

The last change is the most important because it makes it
much easier to reason about the code. We move away from
copying bits and pieces of the class table and shared
class table into reload contexts.

Having two class table fields in the isolate group makes
it easier to reason about. One field contains program
class table (one modified by kernel loader and accessed
by various program structure cid lookups) and heap
walk class table (used by GC visitors). Normally these
two fields point to the same class table, but during
hot reload we temporary split them apart: original
class table is kept as a heap walk class table, while
program class table is replaced by a clone and updated
by reload.

If reload succeeds we drop original class table and
set program class table as heap walk one.

If reload fails we drop the program class table and
restore original one from heap walk table.

TEST=ci

Cq-Include-Trybots: luci.dart.try:vm-kernel-reload-linux-release-x64-try,vm-kernel-reload-linux-debug-x64-try,vm-kernel-reload-rollback-linux-debug-x64-try,vm-kernel-reload-rollback-linux-release-x64-try,vm-kernel-linux-debug-x64-try,vm-kernel-precomp-tsan-linux-release-x64-try,vm-kernel-tsan-linux-release-x64-try,vm-kernel-precomp-asan-linux-release-x64-try,vm-kernel-asan-linux-release-x64-try
Change-Id: I8b66259fcc474dea7dd2af063e4772df99be06c4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/258361
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
2022-09-10 15:12:35 +00:00
Alexander Markov 591b2bd724 [vm] Add --print-object-layout-to option to gen_snapshot
This option dumps offsets of Dart fields and sizes of Dart instances
into the specified file in JSON format.
Could be useful for debugging or memory usage investigations.

Typical output:

{"class":"_JsonUtf8Parser","size":80,"fields":[{"field":"emptyChunk","static":true},{"field":"decoder","offset":56},{"field":"chunk","offset":64},{"field":"chunkEnd","offset":72}]}

TEST=runtime/tests/vm/dart/print_object_layout_test.dart
Fixes https://github.com/dart-lang/sdk/issues/47892

Change-Id: I0b2f7ed9e4991e5fcd18517d5a06aa826c6d7125
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/223662
Reviewed-by: Dan Field <dnfield@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
2021-12-20 21:40:33 +00:00
Tess Strickland cd6fb67103 [vm] Clean up changes around internal-only class ids.
* IsInternalOnlyId -> IsInternalOnlyClassId

* Removes constexpr predicate ClassComesBeforeOtherInternalOnlyClasses()
  used in only two locations and inlines the returned condition.

* Change IsErrorClassId to check <= kUnwindErrorCid instead of
  <= kLastInternalOnlyCid.

TEST=Refactoring, so existing tests.

Change-Id: Ib7dfc89aba16d52733de05687f0ca7055c16e7a3
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/210126
Auto-Submit: Tess Strickland <sstrickl@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
2021-08-13 20:00:41 +00:00
Tess Strickland e8ddc0219f [vm] Abstract out internal-only cid ordering.
* Adds IsInternalOnlyCid, which replaces the old `cid < kInstanceCid`
  check.

* Adds kFirstInternalOnlyCid and kLastInternalOnlyCid, which should be
  used in places that loop over the internal-only cids.

* Adds ObjectComesBeforeOtherInternalOnlyClasses(), which can be used
  with COMPILE_ASSERT in places that assume kObjectCid immediately
  precedes the other internal-only cids.

* Replaces assumption in IsErrorId() that Instance immediately follows
  the internal-only cids with an assumption that the current last error
  cid is the same as kLastInternalOnlyCid.

* Separates out CLASS_LIST_NO_OBJECT_NOR_STRING_... into two pieces to
  make it clearer where to add new internal-only cids and where to add
  new Instance cids:

    * CLASS_LIST_INTERNAL_ONLY, which contains class ids for real
      classes that are VM internal, except for Object.

    * CLASS_LIST_INSTANCE_SINGLETONS, which contains class ids for
      subclasses of Instance that are not part of a separately handled
      group of cids.

TEST=Refactoring, so existing tests.

Change-Id: I9905d5928a592164d58de1f121b9ce669f2b3ce1
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/210122
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2021-08-13 13:27:41 +00:00
Alexander Aprelev aef297df72 [vm/concurrency] Ensure class table entry is not updated if only class size needs updating.
When size get updated, class table update is not needed, causes tsan data race warnings.

Fixes https://github.com/dart-lang/sdk/issues/46538
TEST=IsolateSpawn on tsan

Change-Id: I8dac5faa413b0581e42d6b33adeef8e94e492870
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/206329
Commit-Queue: Alexander Aprelev <aam@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2021-07-13 19:28:50 +00:00
Martin Kustermann 3c81d992ef [vm/concurrency] Distinguish "gc safepoint operations" from "deopt safepoint operations"
This extends the existing safepoint operation mechanism by allowing to
perform two different operations:

  * "gc safepoint operations": All mutators are stopped at places where
    it's safe to GC. It therefore requires stackmaps to be available for
    all optimized mutator frames.

  * "deopt safepoint operations": All mutators are stopped at places
    where it's safe to GC, but also safe to lazy-deopt mutator frames.
    It therefore requires deopt-id/deopt-info to be available for all
    optimized mutator frames.

Mutators can be asked to block for any of those two safepoint operations.
If a mutator is at a place where its safe to GC it will respond to "gc
safepoint operations" requests, if a mutator is additionally at a place
where it's also safe to lazy-deopt it will respond to "deopt safepoint
operation" requests.

Depending on how the runtime was entered (which is tracked via the
[Thread::runtime_call_deopt_ability_] value) - the mutator might
participate in both or only in gc safepoint operations.

During the start of a "deopt safepoint operation", the safepoint handler
will request all threads to stop at a "deopt safepoint". Some threads
might first want to initiate their own "gc safepoint operation"
(e.g. due to allocation failure) before they reach a "deopt safepoint".

We do allow this by letting the safepoint handler own a "deopt safepoint
operation" but still participate in other thread's "gc safepoint
operation" requests until all mutators are checked into places where
it's safe to lazy-deopt at which point the "deopt safepoint operation"
also owns a "gc safepoint operation".

In order to facilitate this, the Thread's safepoint_state will be
extended to consist of the following bits:

  * AtSafepoint
  * SafepointRequested
  * AtDeoptSafepoint
  * DeoptSafepointRequested
  * BlockedForSafepoint

Issue https://github.com/dart-lang/sdk/issues/45213

TEST=vm/cc/SafepointOperation_*

Change-Id: Icdc2827718f6780818f99b829a5e806d6bb5b130
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/196927
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
2021-05-10 09:13:09 +00:00
Tess Strickland d288c95de4 [vm] Remove unused functions from class_id.h/class_table.h.
TEST=Build succeeds.

Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-linux-debug-x64-try
Change-Id: I3f6a7a01eb3eabd7bbe735d3889510bc3144942b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/194766
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
2021-04-13 09:53:07 +00:00
Martin Kustermann b80ab44a53 [vm/concurrency] Move class registration from Isolate to IsolateGroup
We share the program across all isolates within a group. So hot-reload
should not depend on a particular isolate. This CL moves class
registration (which hot reload triggers) from Isolate to IsolateGroup.

Issue https://github.com/dart-lang/sdk/issues/36097

TEST=Refactoring of existing implementation.

Change-Id: Ibdb4e67a095c7f2f3e3c1a8f8194bd0ca936b140
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/184266
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
2021-02-12 07:53:46 +00:00
Alexander Markov 97e2f3ddc5 [vm/aot] Avoid retaining inlined functions with --dwarf_stack_traces
When --dwarf-stack-trace option is used, we don't use nor write inlined
functions information to the snapshot. However, we were retaining
Function objects corresponding to inlined functions.

On a large app, AOT-compiled in PRODUCT mode with
--dwarf-stack-traces on arm64:

Function objects in AOT snapshot -19.4%
Heap size used by objects in snapshot -4.45%
Total AOT snapshot size -2.35%

Flutter gallery AOT snapshot size
(release mode with --dwarf-stack-traces):
arm64 -1.1% (gzip -1.3%, brotli -1.4%)
arm -0.95% (gzip -1.2%, brotli -1.4%)

TEST=ci

Change-Id: Id2d7a2d6b6bff1a5929be97477b0e17a9afae475
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/181700
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Reviewed-by: Tess Strickland <sstrickl@google.com>
2021-02-01 18:26:29 +00:00
Ben Konyi fc48e553d5 [ Service / package:vm_service ] Allow for Sentinel value in Field.staticValue, fix null values in ClassList
Field.staticValue has always been able to take a Sentinel as a value but
was not correctly specified in the service spec. Updates the spec and
package:vm_service to allow for this case.

Also fixed an issue where FreeListElement and ForwardingCorpse were
being returned from getClassList.

Fixes https://github.com/dart-lang/sdk/issues/44588

TEST=pkg/vm_service/test/regress_44588_test.dart

Change-Id: I059717657533ff623a70d159a918d63e9e4b6cc7
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/179020
Commit-Queue: Ben Konyi <bkonyi@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
2021-01-20 19:25:46 +00:00
Ryan Macnak b1c09ecd8f [vm] Make naming more consistent when converting between handles, tagged and untagged pointers.
Currently we have things called XPtr which are not what you get from ptr().

Old world:
handle->raw() returns RawObject* (tagged)
raw_obj->ptr() returns RawObject* (untagged)

After 6fe15f6df9:
handle->raw() returns ObjectPtr
obj_ptr->ptr() returns ObjectLayout*

New world:
handle->ptr() returns ObjectPtr
obj_ptr->untag() returns UntaggedObject*

TEST=ci
Change-Id: I6c7f34014cf20737607caaf84979838300d12df2
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/149367
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
2021-01-15 23:32:02 +00:00
Martin Kustermann 0f76981bb1 [vm/concurrency] Change references to class_table/object_store/heap from Isolate to IsolateGroup
As part of making the compiler and other subsystems independent
of `Isolate` we have to move various state from `Isolate` to
`IsolateGroup`.

The class_table and object_store were already moved to `IsolateGroup`.

This CL only replaces usages of `Isolate::{object_store,class_table}`
with the equivalent in `IsolateGroup`.

Issue https://github.com/dart-lang/sdk/issues/36097

TEST=Pure refactoring - relying on existing test coverage.

Change-Id: I34a0682d715b054d6c5faff077a513980f59a348
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/177126
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
2021-01-06 15:22:11 +00:00
Martin Kustermann d215ab6d73 [vm] Assign top-level classes cids outside 16-bit range
Right now we assign class ids to top-level classes, abstract classes as
well as concrete classes. All of them have allocated from a 16-bit pool
of ids. The VM FATAL()s once it hits that limit.

Customers who run very large programs (significant amount of generated
code) on the Dart VM have started to hit this 16-bit class limit.

Concrete classes can have instances in the heap. Our current heap layout
only allows 16-bit class ids to be encoded in the header word. To avoid
increasing the size of heap objects or shrinking the size of the identity
hash code to 16-bit we keep class ids in object headers to be 16-bit.

Abstract classes cannot have instances in the heap. Though their class
ids are encoded in type objects. Furthermore we sort classes in
AOT/AppJIT mode to perform fast class-id range checks. To avoid impacting
this optimization we treat abstract classes the same way as concrete
classes.

Top-level classes cannot have instances in the heap. Their class ids are
only used in the runtime code, for example for hot-reload as well as
part of the service protocol.

=> We can allocate class ids outside the 16-bit range for top-level
classes, thereby freeing a significant amount of space in the 16-bit
range.

This CL does exactly that: We change classid_t to be int32_t. The
ClassLayout::id_ can now be assigned ids outside 16-bit range for
top-level classes. To do this we keep dart classes and top level classes
as separate arrays in the ClassTable.

Issue https://github.com/dart-lang/sdk/issues/42533

See also b/160229360

Change-Id: I6710a644e7b0ab2d4f4c792bef8e1f91cb117421
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/153607
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
2020-07-09 18:33:32 +00:00
Ryan Macnak 6fe15f6df9 [vm] Represent tagged pointers as C++ value types instead of C++ pointer types.
This works around bugs in UndefinedBehaviorSanitizer and Clang.

Bug: b/28638298
Change-Id: I6be595f9664516019d28017d24559583a1ae3a21
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/144354
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
2020-04-25 05:21:27 +00:00
Alexander Markov 53860a5822 [vm/nnbd] Take Never type out of the VM isolate
Never type in the VM isolate causes bootstrapping problems when
null-safety mode is auto-detected. We cannot initialize its type
testing stub without knowing null-safety mode and we cannot patch it
after figuring out null-safety mode because VM isolate is read-only.

The solution is to move Never type (and Never class) out of the VM
isolate and make them per isolate.

Issue: https://github.com/dart-lang/sdk/issues/41206
Change-Id: Iede937aa5becd5f1f7b029db836c25a1b20fbc5c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/144488
Reviewed-by: Régis Crelier <regis@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
2020-04-23 01:31:57 +00:00
Ryan Macnak 160046f44c [vm] Move CID predicates out of RawObject.
When using C++ value types to represent tagged pointers, we cannot use forward declarations for tagged pointers. This helps to break include cycles when attempting to keep IsXYZ predicates as tagged pointer member functions.

class_id.h also seems like a more natural place for these predicates, which were written before there was a class_id.h.

Change-Id: I0677560a794ed084d10f844606e202feb0c3820a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/144321
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
2020-04-21 20:09:55 +00:00
Alexander Aprelev 9c4a322b08 Reland "[vm/isolates] Introduce fast isolate spawn in AOT."
This reverts commit 922ea3e9b6 in patchset 1, fix for assertion triggered in https://ci.chromium.org/b/8883214567628884960 in patchset 2, fix for deadlock around symbols table mutex in patchset 4.

Original commit description:

Speed up is achieved by sharing most of the dart code, object store
and class table between isolates in single isolate group. So
instead of bootstrapping isolate from the snapshot, isolate is
initialized by setting pointers to existing data structures already
set up for first isolate, and only few isolate-specific structures (moved
to newly introducted isolate_object_store) are created.

To allow for safe cross-isolate switchable call site, type test cache
mutations additional synchronization via RunWithStoppedMutators(that
relies on safepoints) was added.
Besides switchable call sites, no other mutation to the dart code is
done in AOT, which allows such sharing.

Bug: https://github.com/dart-lang/sdk/issues/37835
Bug: https://github.com/dart-lang/sdk/issues/36097
Change-Id: I655e337198214c9dfacbe76f7852b941b5a7e910
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/143462
Commit-Queue: Alexander Aprelev <aam@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
2020-04-17 03:02:27 +00:00
Alexander Aprelev 922ea3e9b6 Revert "Reland "[vm/isolates] Introduce fast isolate spawn in AOT.""
This reverts commit fdd7a2c616 as there
seem to be sporadic dartkp crashes like https://ci.chromium.org/p/dart/builders/ci.sandbox/vm-kernel-precomp-linux-product-x64/7454

Change-Id: I4218b8ff629630991ad1ff94f217489bb16228d3
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/143383
Reviewed-by: Alexander Aprelev <aam@google.com>
2020-04-14 20:37:29 +00:00
Alexander Aprelev fdd7a2c616 Reland "[vm/isolates] Introduce fast isolate spawn in AOT."
Original revert in patchset 1, fix for deadlock issue in patchset 2: avoid reentrant calls to RunWithStoppedMutator.
Further review comments addressed in successive patchsets.

This reverts commit 8ef508ba36.

Original commit description:

Speed up is achieved by sharing most of the dart code, object store
and class table between isolates in single isolate group. So
instead of bootstrapping isolate from the snapshot, isolate is
initialized by setting pointers to existing data structures already
set up for first isolate, and only few isolate-specific structures (moved
to newly introducted isolate_object_store) are created.

To allow for safe cross-isolate switchable call site, type test cache
mutations additional synchronization via RunWithStoppedMutators(that
relies on safepoints) was added.
Besides switchable call sites, no other mutation to the dart code is
done in AOT, which allows such sharing.

Change-Id: I6a0279d9812020ad7a5c2b7851980b6a29b95b9a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/143327
Commit-Queue: Alexander Aprelev <aam@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2020-04-14 18:42:45 +00:00
Alexander Aprelev 8ef508ba36 Revert "[vm/isolates] Introduce fast isolate spawn in AOT."
This reverts commit 61c0960a8b as it broke
vm-kernel-win-release-ia32 buildbot test https://ci.chromium.org/b/8883214567628884960

Change-Id: I6377fd622a3c7d808f46fd9c541c64d343ecc269
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/143260
Reviewed-by: Alexander Aprelev <aam@google.com>
2020-04-12 15:59:41 +00:00
Alexander Aprelev 61c0960a8b [vm/isolates] Introduce fast isolate spawn in AOT.
Speed up is achieved by sharing most of the dart code, object store
and class table between isolates in single isolate group. So
instead of bootstrapping isolate from the snapshot, isolate is
initialized by setting pointers to existing data structures already
set up for first isolate, and only few isolate-specific structures (moved
to newly introducted isolate_object_store) are created.

To allow for safe cross-isolate switchable call site, type test cache
mutations additional synchronization via RunWithStoppedMutators(that
relies on safepoints) was added.
Besides switchable call sites, no other mutation to the dart code is
done in AOT, which allows such sharing.

Bug: https://github.com/dart-lang/sdk/issues/37835
Bug: https://github.com/dart-lang/sdk/issues/36097
Change-Id: I64c86525f4ef9cb30567a49a106bfe700355942b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/136780
Commit-Queue: Alexander Aprelev <aam@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
2020-04-12 14:57:09 +00:00
Ryan Macnak 1e34ad08ee [vm] Remove use of memmove/memset for class tables.
Using memmove/memset bypasses non-trivial assignment. In particular, it changes relaxed atomic operations into non-atomic operations.

Fixes -Werror=class-memaccess errors in gcc.

Change-Id: Ic4589fe7a564a268a578a856058c2d2183961b46
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/142549
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2020-04-08 23:49:16 +00:00
Ryan Macnak 23792e7816 [vm] Require explicit loads and stores when using AcqRelAtomic.
Change-Id: I19d64667cea7dd735c3bf7194285383289019aaa
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/139942
Reviewed-by: Alexander Aprelev <aam@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
2020-03-19 16:19:59 +00:00
Ryan Macnak 9a140bab87 [vm] Use acquire-release ordering to publish and consume the class table backing arrays.
Change-Id: I57a3109b3ba42e1addf97c15c5d8108cd0338266
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/139860
Reviewed-by: Alexander Aprelev <aam@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
2020-03-18 17:18:55 +00:00
Ryan Macnak 4ef342680b [vm] Fix data race installing into the SharedClassTable.
Bug: https://github.com/dart-lang/sdk/issues/39611
Change-Id: I391717a8f175f16a8286e6e8928aa5240bb28a90
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/139828
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
2020-03-17 21:59:15 +00:00
Martin Kustermann b4d4eb00df [vm/gc] Ensure we keep old trace allocations tables around until its safe to delete them (during GC)
We do this already for the class table, size table and unboxed field
table. This does the same for the trace allocation table.

Issue https://github.com/dart-lang/sdk/issues/40749

Change-Id: I417b7daf529967a5a496cc4ecc15cee20371ee9e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/136963
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
2020-02-24 20:03:58 +00:00
Martin Kustermann dae308461c [vm/concurrency] Share [Heap] and [SharedClassTable] between all isolates within one isolate group
This CL:

  * Moves [Heap]/[SharedClassTable] from [Isolate] to [IsolateGroup], which
    will make all isolates in the group use the same heap. The GC will use
    the shared class table for object size information.

  * Adds support for entering/leaving an isolate group as a helper thread
    (e.g. via [Thread::EnterIsolateGroupAsHelper]). The current active
    isolate group can be accessed via TLS `IsolateGroup::Current()` or
    `Thread::isolate_group_`. When entering as a helper thread there will be
    no current isolate.

  * Changes the GC to use the above mechanism and ensures GC works without
    a currently active isolate. The GC will use information purely available via
    [IsolateGroup]. The GC will iterate all isolates within an isolate
    group e.g. for scanning roots.

  * Makes spawning of new isolates start in their own isolate group.
    Once the isolate is fully functional it's heap will be merged into
    the original isolate group

  * Moves ApiState, containing persistent and weak persistent handles,
    from [Isolate] to [IsolateGroup], plus adds appropriate locking.

Issue https://github.com/dart-lang/sdk/issues/36097

Change-Id: Ia8e1d8aa78750e8400864200f4825395a182c004
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/126646
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
2020-02-20 21:08:35 +00:00
Ryan Macnak 68d178f223 [vm] Remove FLAG_support_service and FLAG_support_reload, superseded by PRODUCT define.
When these were originally added, we thought we would be able to use them in place of ifdefs and rely on the compiler optimizations and linker GC to remove things. This turned out not to reliably remove what we wanted removed, so we ended up with the ifdefs anyway.

Change-Id: I62e74d60d92b18a688b9dffaf77b1440c10a07ff
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/134402
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Ben Konyi <bkonyi@google.com>
2020-02-04 23:31:17 +00:00
Victor Lima b68d95ec9e [vm] Reland support for real unboxed floating point fields in AOT
Issue https://github.com/dart-lang/sdk/issues/40404

Change-Id: Icfa801ff0640a6b27bb3c13d0b737c40452cbf7d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/133983
Commit-Queue: Victor Agnez Lima <victoragnez@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2020-01-31 12:56:31 +00:00
Zichang Guo 162d6c5634 Revert "[vm] Add support for real unboxed floating point fields in AOT"
This reverts commit 9eb531bde4.

Reason for revert: Bots are red. Some tests are failing.

https://ci.chromium.org/p/dart/builders/ci.sandbox/vm-kernel-precomp-obfuscate-linux-release-x64/6039

https://ci.chromium.org/p/dart/builders/ci.sandbox/vm-kernel-precomp-android-release-arm_x64/957

Original change's description:
> [vm] Add support for real unboxed floating point fields in AOT
> 
> Non-nullable floating point fields (double, Float32x4, Float64x2)
> are fully unboxed in their classes.
> 
> A bitmap for each class was added to the shared class table in order to keep
> track of the pointers of the classes. Since all classes in Flutter Gallery
> have less than 64 fields, the bitmap is represented by a 64 bit integer and
> fields whose offset is more than 64 words are not unboxed.
> 
> The instance sizes and field offsets might change between target and host
> in cross-compilation, since the number of words used to store unboxed fields
> may differ.
> 
> dart-aot Xeon
> 
>   SplayLatency               -4.62%
>   SplayHarderLatency         -4.17%
>   NavierStokes               -2.20%
>   Tracer                      8.72%
>   ParticleSystemPaint         2.90%
>   NBodySIMD                   8.35%
>   NBody                      25.59%
> 
> With hack TFA to make doubles in Rect/Offset/Size classes in flutter non-nullable:
> 
> flutter arm-v8:
> 
>   gallery total size: -1%
> 
>   matrix_utils_transform_rect_perspective   -16.70% (less is better)
>   matrix_utils_transform_rect_affine        -31.82% (less is better)
>   matrix_utils_transform_point_perspective  -24.90% (less is better)
>   matrix_utils_transform_point_affine)      -27.26% (less is better)
>   rrect_contains_bench                      -4.719% (less is better)
> 
> Change-Id: I9ae09c9c3167d99f9efd071a92937aa51093fd1d
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/131824
> Commit-Queue: Victor Agnez Lima <victoragnez@google.com>
> Reviewed-by: Martin Kustermann <kustermann@google.com>
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
> Reviewed-by: Samir Jindel <sjindel@google.com>

TBR=kustermann@google.com,rmacnak@google.com,sjindel@google.com,victoragnez@google.com

Change-Id: Ic73858f6adb7f55c4129d4f46ff4731b378cb634
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/134020
Reviewed-by: Zichang Guo <zichangguo@google.com>
Commit-Queue: Zichang Guo <zichangguo@google.com>
2020-01-30 18:13:28 +00:00
Victor Lima 9eb531bde4 [vm] Add support for real unboxed floating point fields in AOT
Non-nullable floating point fields (double, Float32x4, Float64x2)
are fully unboxed in their classes.

A bitmap for each class was added to the shared class table in order to keep
track of the pointers of the classes. Since all classes in Flutter Gallery
have less than 64 fields, the bitmap is represented by a 64 bit integer and
fields whose offset is more than 64 words are not unboxed.

The instance sizes and field offsets might change between target and host
in cross-compilation, since the number of words used to store unboxed fields
may differ.

dart-aot Xeon

  SplayLatency               -4.62%
  SplayHarderLatency         -4.17%
  NavierStokes               -2.20%
  Tracer                      8.72%
  ParticleSystemPaint         2.90%
  NBodySIMD                   8.35%
  NBody                      25.59%

With hack TFA to make doubles in Rect/Offset/Size classes in flutter non-nullable:

flutter arm-v8:

  gallery total size: -1%

  matrix_utils_transform_rect_perspective   -16.70% (less is better)
  matrix_utils_transform_rect_affine        -31.82% (less is better)
  matrix_utils_transform_point_perspective  -24.90% (less is better)
  matrix_utils_transform_point_affine)      -27.26% (less is better)
  rrect_contains_bench                      -4.719% (less is better)

Change-Id: I9ae09c9c3167d99f9efd071a92937aa51093fd1d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/131824
Commit-Queue: Victor Agnez Lima <victoragnez@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Samir Jindel <sjindel@google.com>
2020-01-30 13:45:08 +00:00
Victor Lima afa98b977a [vm] Ensure abstract classes are marked as such and get a size of 0 in shared class table.
Change-Id: If19444e702a2b73a829bd9f581ff36ad041ecf41
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/131801
Commit-Queue: Victor Agnez Lima <victoragnez@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2020-01-16 10:34:53 +00:00
Ryan Macnak 9c9c283c78 [vm, gc] Remove dead allocation stats code.
This should have been removed in 545b10389d.

Bug: https://github.com/dart-lang/sdk/issues/37678
Change-Id: Ie679e68b76e339843fcfe00e866be6478b9cc98b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/131201
Reviewed-by: Siva Annamalai <asiva@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
2020-01-13 17:21:16 +00:00
Ryan Macnak e29a6d6f8d [vm] Initialize the handle vtable table during VM initialization instead of during class registration in each isolate's initialization.
Removes a (harmless) race initializing this table to appease TSAN.

Bug: https://github.com/dart-lang/sdk/issues/39611
Change-Id: I9f7bdd0edf2be5e87e56e453601641a9550d157a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/127965
Reviewed-by: Siva Annamalai <asiva@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
2019-12-11 18:54:08 +00:00
Alexander Aprelev f6cefb35c0 [vm/gardening] Clean up no longer used ClassAndSize class.
This was made redundant by https://dart-review.googlesource.com/c/sdk/+/116064.

Change-Id: I325a6faa257a1fc635a3265bf9e1b14096fc8f25
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/126000
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Alexander Aprelev <aam@google.com>
2019-11-21 19:37:26 +00:00
Ryan Macnak 545b10389d [vm, service] Compute allocation stats on demand, instead of during allocation and GC.
Removes support for accumulators, which now repeat the current live values.
Decreases performance difference between release and product modes.
Fixes inaccuracy when counters are queried before the first full GC.

Golem geomean   x64 4.687%
Golem geomean arm64 5.770%

Bug: https://github.com/dart-lang/sdk/issues/37678
Change-Id: I12b26a9834b0f0f911ddcc6e8e5ff4573272607d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/116885
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
2019-11-16 00:22:56 +00:00
Martin Kustermann 20962e656b [vm/concurrency] Split up IsolateReloadContext into IsolateReloadContext/IsolateGroupReloadContext
Similar to the split of ClassTable into ClassTable/SharedClassTable,
this CL splits up the IsolateReloadContext into:

  * IsolateGroupReloadContext: Consists of reload-related information across all
    isolates. The [Reload()] method is split up in phases that are
    performed on all isolates before the next phase is started.

    => This allows each isolate to add reasons for rolling back, if no
       reasons are found the reload will be accepted atomically.

  * IsolateReloadContext: Constists of reload-related information for a
    particular isolate (e.g. mappings of old to new classes)

The assumption is that all isolates have the same source (and therefore
the same libraries). For certain things, e.g. discovering which libraries
changed, it is necessary to examine the object store. We use the first
isolate in a group (but could use any of them) to do so, since the
isolate group does not have this information atm.

This is a preparation CL for supporting hot-reloading multiple isolates
within one isolate group. Though the support in this CL stays at having
only a single isolate in a group.

  => This CL turns off FLAG_enable_isolate_groups in JIT mode.

Issue https://github.com/dart-lang/sdk/issues/36097

Change-Id: I7f4d536d4f5ab4a2a73fb0c7618ba967c9b77234
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/123254
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
2019-11-15 14:08:45 +00:00
Régis Crelier df678dbec0 [VM/nnbd] Introduce type Never in VM and bytecode.
Introduce enum NNBDMode for subtype test checking mode.

Change-Id: Id9c39ee8c975c664ac58e313b7dd05e6e219dfc1
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/123557
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Régis Crelier <regis@google.com>
2019-11-01 00:43:59 +00:00