Remove header access from typed data case in RawObject::HeapSizeFromClass.
Address data race between visiting remembered cards and allocating new large pages for promotion.
Change-Id: I92cc4604ed38ab8f42b87b140a26a8118e7030d4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/142565
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
This reverts commit 9209f522d6.
Reason for revert: Flaky analyzer crashes
Original change's description:
> [vm, gc] Parallel scavenge.
>
> Run N tasks that compete to copy or promote survivors. Installation of the forwarding pointer uses a CAS.
> Tasks do not share the list of copied objects to be processed.
> Tasks do share the list of promoted objects to be processed.
> Divide the freelist in N copies. Each scavenger task uses one without locking. The sweeper inserts into the freelists round-robin.
>
> Change-Id: I54c7839ac5f15829462b3be7cf8c9635c01b785f
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/135962
> Commit-Queue: Ryan Macnak <rmacnak@google.com>
> Reviewed-by: Martin Kustermann <kustermann@google.com>
TBR=kustermann@google.com,aam@google.com,rmacnak@google.com,asiva@google.com
Change-Id: I0d97b8d9148b2a81cabafde4602512e423eba223
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/142554
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Run N tasks that compete to copy or promote survivors. Installation of the forwarding pointer uses a CAS.
Tasks do not share the list of copied objects to be processed.
Tasks do share the list of promoted objects to be processed.
Divide the freelist in N copies. Each scavenger task uses one without locking. The sweeper inserts into the freelists round-robin.
Change-Id: I54c7839ac5f15829462b3be7cf8c9635c01b785f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/135962
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
These changes were originally submitted separately on different days,
and a major performance regression was seen after the first change
when creating snapshots that led to both being reverted. However,
that performance regression should be addressed by the followup.
First change:
"[vm] Treat the dispatch table as a root in the snapshot.
Additional changes:
* Only serialize a dispatch table in precompiled snapshots.
* Add information in v8 snapshot profiles for the dispatch table.
* Fix a typo in a field name.
* Print the number of Instructions objects (or payloads, for
precompiled bare instructions mode) in the fake cluster for
the data section.
* Fix v8 snapshots profiles so objects in memory mapped segments
and only those are prefixed with "(RO) ".
* Add names for Instructions objects in v8 snapshot profiles
when we can use the assembly namer.
* Add command line flag for old #define'd false flag."
Second change:
"[vm/aot] Keep GC-visible references to dispatch table Code entries.
This change splits dispatch table handling into four distinct
parts:
* The dispatch table generator does not make a dispatch table
directly, but rather creates an Array that contains the Code
objects for dispatch table entries.
* The precompiler takes this Array and puts it in the object
store, which makes it a new GC root.
* The serializer takes this information and serializes the
dispatch table information in the same form as before.
* The deserializer creates a DispatchTable object and populates
it using the serialized information.
The change in the precompiler ensures that the Code objects
used in the dispatch table have GC-visible references. Thus,
even if all other references to them from the other GC roots
were removed, they would be accessible in the serializer in
the case of a GC pass between the precompiler and serializer.
This change also means that the serializer can retrieve and
trace the Code objects directly rather than first looking up
the Code objects by their entry point."
Bug: https://github.com/dart-lang/sdk/issues/41022
Change-Id: I52c83b0536fc588da0bef9aed1f0c72e8ee4663f
Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-linux-release-simarm-try,vm-kernel-precomp-linux-release-simarm64-try,vm-kernel-precomp-linux-release-simarm_x64-try,vm-kernel-precomp-android-release-arm64-try,vm-kernel-precomp-android-release-arm_x64-try,vm-kernel-precomp-mac-release-simarm64-try,vm-kernel-precomp-win-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/139285
Commit-Queue: Teagan Strickland <sstrickl@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
This change splits dispatch table handling into four distinct
parts:
* The dispatch table generator does not make a dispatch table
directly, but rather creates an Array that contains the Code
objects for dispatch table entries.
* The precompiler takes this Array and puts it in the object
store, which makes it a new GC root.
* The serializer takes this information and serializes the
dispatch table information in the same form as before.
* The deserializer creates a DispatchTable object and populates
it using the serialized information.
The change in the precompiler ensures that the Code objects
used in the dispatch table have GC-visible references. Thus,
even if all other references to them from the other GC roots
were removed, they would be accessible in the serializer in
the case of a GC pass between the precompiler and serializer.
This change also means that the serializer can retrieve and
trace the Code objects directly rather than first looking up
the Code objects by their entry point.
Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-linux-release-simarm-try,vm-kernel-precomp-linux-release-simarm64-try,vm-kernel-precomp-linux-release-simarm_x64-try,vm-kernel-precomp-android-release-arm64-try,vm-kernel-precomp-android-release-arm_x64-try,vm-kernel-precomp-mac-release-simarm64-try,vm-kernel-precomp-win-release-x64-try
Change-Id: Ic9ba29dfd64945a6e9a51f9de73f74d9202b58b7
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/138288
Commit-Queue: Teagan Strickland <sstrickl@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Aske Simon Christensen <askesc@google.com>
Three bugs were fixed:
1. BitVector::Equals was not fully fixed by the original CL.
2. We need to add old objects to the deferred marking queue during
RememberLiveTemporaries().
3. The thread being scanned in RestoreWriteBarrierInvariant may not
be scheduled, so we cannot use its store buffer block.
In addition, this changed uncovered another bug fixed in:
https://dart-review.googlesource.com/c/sdk/+/138960.
Original CL is in patchset 3.
This reverts commit 30a12a349e.
Change-Id: I36169b09563998ed5b3c3eac70ee0ebe78853e62
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/138920
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Samir Jindel <sjindel@google.com>
This reverts commit eff1a9ff97.
Reason for revert:
Causes flaky hits of RELEASE_ASSERT in marker.cc, see b/151131634.
Original change's description:
> Re-land "[vm] Aggressive write-barrier elimination."
>
> The original revision is in Patchset 3.
>
> Four bugs were fixed:
>
> 1. JoinEntryInstr::SuccessorCount() is not the correct way to get the
> number of successor blocks from the Join block;
> JoinEntryInstr::last_instruction()->SuccessorCount() must be used
> instead.
>
> 2. BitVector::Equals() was non-deterministically returning 'false'
> for equal vectors.
>
> 3. All blocks need to be processed at least once during the Analysis
> phase (not only in the SaveResults phase).
>
> 4. We were not removing write barriers from StoreIndexed instructions,
> even though we had support for it.
>
> This reverts commit 7fd8ad5a2d.
>
> Fixes https://github.com/dart-lang/sdk/issues/40780
>
> Change-Id: I9650ec2c547ec49cf88ca0524e14f6c245621f6a
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/138086
> Commit-Queue: Samir Jindel <sjindel@google.com>
> Reviewed-by: Martin Kustermann <kustermann@google.com>
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
TBR=kustermann@google.com,rmacnak@google.com,sjindel@google.com
Change-Id: If9afd84465175fad2431405a97e9293c8bd5e476
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/138808
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
The original revision is in Patchset 3.
Four bugs were fixed:
1. JoinEntryInstr::SuccessorCount() is not the correct way to get the
number of successor blocks from the Join block;
JoinEntryInstr::last_instruction()->SuccessorCount() must be used
instead.
2. BitVector::Equals() was non-deterministically returning 'false'
for equal vectors.
3. All blocks need to be processed at least once during the Analysis
phase (not only in the SaveResults phase).
4. We were not removing write barriers from StoreIndexed instructions,
even though we had support for it.
This reverts commit 7fd8ad5a2d.
Fixes https://github.com/dart-lang/sdk/issues/40780
Change-Id: I9650ec2c547ec49cf88ca0524e14f6c245621f6a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/138086
Commit-Queue: Samir Jindel <sjindel@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
A thread may enter a safepoint using a simple CAS. This can happen before a safepoint is requested, in which case the thread won't need to check into the safepoint using the monitor. However, we still need at least a release-acquire pair so the safepoint operation can observe memory effects up to the CAS.
Bug: https://github.com/dart-lang/sdk/issues/39611
Change-Id: Ied1b6c8c8e03bc17d86d848eccf874f24f5d0df4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/138051
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
Some embedder code relies on the presence of a current isolate equal to the isolate that created the handle. This code will be need to be updated to be isolate-group aware before we can run finalizers only in the context of the isolate group instead of an isolate.
Bug: https://github.com/dart-lang/sdk/issues/40836
Change-Id: I38956c556df551e48ee7e4d0822aa5def8dc4fc5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/138013
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
When moving the newly spawned isolate from the spawnee group to the main
group we have to ensure to update the cached
`Isolate::mutator_thread_->isolate_group_` pointer (which we do not
reset when descheduling mutator threads (for other threads we do)).
This will fix the ASAN crash in this test:
tools/test.py -n dartk-asan-linux-release-x64 lib_2/isolate/mandel_isolate_test
Fixes https://github.com/dart-lang/sdk/issues/40756
Change-Id: I5a897098106e6748cf4e58bd5d2799ce28b130a8
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/136971
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
It incorrectly assumed that all stores in Dart code write to Instances.
There is actually one exception, Contexts, which do not inherit from Instance.
I've added asserts to ensure this kind of bug cannot resurface.
The original change is in patchset 4.
Change-Id: Ic2d8d05e70a4de738eb9fb5980487b4f27111b8c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/136221
Commit-Queue: Samir Jindel <sjindel@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
This CL:
* Moves [Heap]/[SharedClassTable] from [Isolate] to [IsolateGroup], which
will make all isolates in the group use the same heap. The GC will use
the shared class table for object size information.
* Adds support for entering/leaving an isolate group as a helper thread
(e.g. via [Thread::EnterIsolateGroupAsHelper]). The current active
isolate group can be accessed via TLS `IsolateGroup::Current()` or
`Thread::isolate_group_`. When entering as a helper thread there will be
no current isolate.
* Changes the GC to use the above mechanism and ensures GC works without
a currently active isolate. The GC will use information purely available via
[IsolateGroup]. The GC will iterate all isolates within an isolate
group e.g. for scanning roots.
* Makes spawning of new isolates start in their own isolate group.
Once the isolate is fully functional it's heap will be merged into
the original isolate group
* Moves ApiState, containing persistent and weak persistent handles,
from [Isolate] to [IsolateGroup], plus adds appropriate locking.
Issue https://github.com/dart-lang/sdk/issues/36097
Change-Id: Ia8e1d8aa78750e8400864200f4825395a182c004
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/126646
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
This scope can sneak objects larger than kAllocatablePageSize directly into non-large heap pages.
This scope also holds a heap lock outside of the GC, which is likely to create issues when multiple isolates run on the same heap.
Change-Id: Ic56277c83ffccfe2e1396ed773dd30b8d172bbba
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/134040
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
This is a Dart-tailored implementation of the "row-displacement
dispatch table" technique for closed-world instance calls:
All interface targets in the program are grouped into selectors such
that all targets that could potentially be called from the same call
site have the same selector (currently just grouped by name).
Each selector is assigned a selector offset such that offset + classid
is unique for all selector/classid combinations where the class
implements the selector.
At every instance call site that has an interface target (i.e. where
the static type of the receiver is not dynamic), the selector offset +
receiver classid is computed and used as index into a global table of
entry points.
If the receiver can be null (as determined by the front-end TFA and the
VM type propagation), a null check is inserted before the call.
An arguments descriptor is provided (only) for selectors that need it
(those which have type parameters or optional/named parameters).
The dispatch table calls don't need the monomorphic entry code, so for
functions that are only called via dispatch table calls (i.e. never
called dynamically), the monomorphic entry code is left out.
Some future improvements to the table dispatch implementation are
mentioned in https://github.com/dart-lang/sdk/issues/40188
The table dispatch flag is disabled by default in this commit. A
separate commit enables the flag.
Change-Id: Ic2911742b4a2c9a8d3bc7df60605454cbe4c0714
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/126648
Commit-Queue: Aske Simon Christensen <askesc@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
This CL is responsible for inlining Float32x4 and Float64x2 arithmetic
operations in order to reduce their run-time cost.
Null-checks were required in order to allow the unbox instructions to
be non-speculative.
CheckNullInstr was modified to throw ArgumentError when checking
arguments for those operations.
Change-Id: I18b36d96a36011647480f301eb6dc6bb478f59a1
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/128589
Commit-Queue: Victor Agnez Lima <victoragnez@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Samir Jindel <sjindel@google.com>
Fuchsia is about to turn on ShadowCallStack for ARM64. Once this is enabled, we need to treat R18 like a preserved register. Generated Dart has not accessed this register because it is reserved on iOS, and in the absence of Dart exceptions this would be sufficient for us to be ShadowCallStack compatible. However, our exception handling mechanism jumps past all the C++ frames between the Dart exit frame and Exceptions::JumpToFrame, skipping code that would pop from R18.
Add save/restore of R18 in the invocation stubs, and restore of R18 in the jump stub. The latter prevents the ShadowCallStack from overflowing for code that has lots of exceptions without a native call.
Bug: https://bugs.fuchsia.dev/p/fuchsia/issues/detail?id=37449
Change-Id: I2ce6e46624c8d72507e7afa7a44839b1f0def556
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/119481
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Fixes the changes to assembly snapshot writer in SIMARM_X64 mode. This unfortunately can't be tested until the ELF loader is available.
Take 2 is in patchset 1.
Change-Id: Ib8b067dd4f09dcba3b142705e9cb4dfbf4e3eb53
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/117726
Commit-Queue: Samir Jindel <sjindel@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Ryan Macnak