Currently we have things called XPtr which are not what you get from ptr().
Old world:
handle->raw() returns RawObject* (tagged)
raw_obj->ptr() returns RawObject* (untagged)
After 6fe15f6df9:
handle->raw() returns ObjectPtr
obj_ptr->ptr() returns ObjectLayout*
New world:
handle->ptr() returns ObjectPtr
obj_ptr->untag() returns UntaggedObject*
TEST=ci
Change-Id: I6c7f34014cf20737607caaf84979838300d12df2
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/149367
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
The layout of WeakProperty is defined in C++ instead of Dart. Add missing recognition of its accessors so the compiler can use direct field access instead of always going through natives. Compare LinkedHashMap.
Change the sentinel for WeakProperty lists to object null so that no special constructor is needed.
TEST=ci
Bug: https://github.com/dart-lang/sdk/issues/44333
Change-Id: I8686383053c2c345c972b6615b514a1381e79fda
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/175001
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
An address containing an object header is sometimes interpreted as a object slot due to array truncation and concurrent marking/sweeping. Always accessing it as a word assuages TSAN that it won't be subject to tearing, and allows for the removal of TSAN suppressions from header access. In turn, this will allow TSAN to verify some acquire/release header accesses.
TEST=tsan
Bug: https://github.com/dart-lang/sdk/issues/44091
Change-Id: I7bf6449bf7dfeabd9db76e4bb37456e3be2d5124
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/172861
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Allow weak tables to contain immediate objects (Smis) by adjusting the sentinel value for an empty entry to a value which is never a valid Smi nor heap object. Immediates go into the old-space table, since they also don't need to be updated after a scavenge.
Remove the clustered serializer's separate Smi id table.
Change-Id: Ia34ebe92be17f6568f0d4e5090e9abf5e77dadb3
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/168620
Reviewed-by: Régis Crelier <regis@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
This reverts commit 922ea3e9b6 in patchset 1, fix for assertion triggered in https://ci.chromium.org/b/8883214567628884960 in patchset 2, fix for deadlock around symbols table mutex in patchset 4.
Original commit description:
Speed up is achieved by sharing most of the dart code, object store
and class table between isolates in single isolate group. So
instead of bootstrapping isolate from the snapshot, isolate is
initialized by setting pointers to existing data structures already
set up for first isolate, and only few isolate-specific structures (moved
to newly introducted isolate_object_store) are created.
To allow for safe cross-isolate switchable call site, type test cache
mutations additional synchronization via RunWithStoppedMutators(that
relies on safepoints) was added.
Besides switchable call sites, no other mutation to the dart code is
done in AOT, which allows such sharing.
Bug: https://github.com/dart-lang/sdk/issues/37835
Bug: https://github.com/dart-lang/sdk/issues/36097
Change-Id: I655e337198214c9dfacbe76f7852b941b5a7e910
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/143462
Commit-Queue: Alexander Aprelev <aam@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Original revert in patchset 1, fix for deadlock issue in patchset 2: avoid reentrant calls to RunWithStoppedMutator.
Further review comments addressed in successive patchsets.
This reverts commit 8ef508ba36.
Original commit description:
Speed up is achieved by sharing most of the dart code, object store
and class table between isolates in single isolate group. So
instead of bootstrapping isolate from the snapshot, isolate is
initialized by setting pointers to existing data structures already
set up for first isolate, and only few isolate-specific structures (moved
to newly introducted isolate_object_store) are created.
To allow for safe cross-isolate switchable call site, type test cache
mutations additional synchronization via RunWithStoppedMutators(that
relies on safepoints) was added.
Besides switchable call sites, no other mutation to the dart code is
done in AOT, which allows such sharing.
Change-Id: I6a0279d9812020ad7a5c2b7851980b6a29b95b9a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/143327
Commit-Queue: Alexander Aprelev <aam@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Speed up is achieved by sharing most of the dart code, object store
and class table between isolates in single isolate group. So
instead of bootstrapping isolate from the snapshot, isolate is
initialized by setting pointers to existing data structures already
set up for first isolate, and only few isolate-specific structures (moved
to newly introducted isolate_object_store) are created.
To allow for safe cross-isolate switchable call site, type test cache
mutations additional synchronization via RunWithStoppedMutators(that
relies on safepoints) was added.
Besides switchable call sites, no other mutation to the dart code is
done in AOT, which allows such sharing.
Bug: https://github.com/dart-lang/sdk/issues/37835
Bug: https://github.com/dart-lang/sdk/issues/36097
Change-Id: I64c86525f4ef9cb30567a49a106bfe700355942b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/136780
Commit-Queue: Alexander Aprelev <aam@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Remove header access from typed data case in RawObject::HeapSizeFromClass.
Address data race between visiting remembered cards and allocating new large pages for promotion.
Change-Id: I92cc4604ed38ab8f42b87b140a26a8118e7030d4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/142565
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
This reverts commit 9209f522d6.
Reason for revert: Flaky analyzer crashes
Original change's description:
> [vm, gc] Parallel scavenge.
>
> Run N tasks that compete to copy or promote survivors. Installation of the forwarding pointer uses a CAS.
> Tasks do not share the list of copied objects to be processed.
> Tasks do share the list of promoted objects to be processed.
> Divide the freelist in N copies. Each scavenger task uses one without locking. The sweeper inserts into the freelists round-robin.
>
> Change-Id: I54c7839ac5f15829462b3be7cf8c9635c01b785f
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/135962
> Commit-Queue: Ryan Macnak <rmacnak@google.com>
> Reviewed-by: Martin Kustermann <kustermann@google.com>
TBR=kustermann@google.com,aam@google.com,rmacnak@google.com,asiva@google.com
Change-Id: I0d97b8d9148b2a81cabafde4602512e423eba223
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/142554
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Run N tasks that compete to copy or promote survivors. Installation of the forwarding pointer uses a CAS.
Tasks do not share the list of copied objects to be processed.
Tasks do share the list of promoted objects to be processed.
Divide the freelist in N copies. Each scavenger task uses one without locking. The sweeper inserts into the freelists round-robin.
Change-Id: I54c7839ac5f15829462b3be7cf8c9635c01b785f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/135962
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Three bugs were fixed:
1. BitVector::Equals was not fully fixed by the original CL.
2. We need to add old objects to the deferred marking queue during
RememberLiveTemporaries().
3. The thread being scanned in RestoreWriteBarrierInvariant may not
be scheduled, so we cannot use its store buffer block.
In addition, this changed uncovered another bug fixed in:
https://dart-review.googlesource.com/c/sdk/+/138960.
Original CL is in patchset 3.
This reverts commit 30a12a349e.
Change-Id: I36169b09563998ed5b3c3eac70ee0ebe78853e62
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/138920
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Samir Jindel <sjindel@google.com>
This reverts commit eff1a9ff97.
Reason for revert:
Causes flaky hits of RELEASE_ASSERT in marker.cc, see b/151131634.
Original change's description:
> Re-land "[vm] Aggressive write-barrier elimination."
>
> The original revision is in Patchset 3.
>
> Four bugs were fixed:
>
> 1. JoinEntryInstr::SuccessorCount() is not the correct way to get the
> number of successor blocks from the Join block;
> JoinEntryInstr::last_instruction()->SuccessorCount() must be used
> instead.
>
> 2. BitVector::Equals() was non-deterministically returning 'false'
> for equal vectors.
>
> 3. All blocks need to be processed at least once during the Analysis
> phase (not only in the SaveResults phase).
>
> 4. We were not removing write barriers from StoreIndexed instructions,
> even though we had support for it.
>
> This reverts commit 7fd8ad5a2d.
>
> Fixes https://github.com/dart-lang/sdk/issues/40780
>
> Change-Id: I9650ec2c547ec49cf88ca0524e14f6c245621f6a
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/138086
> Commit-Queue: Samir Jindel <sjindel@google.com>
> Reviewed-by: Martin Kustermann <kustermann@google.com>
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
TBR=kustermann@google.com,rmacnak@google.com,sjindel@google.com
Change-Id: If9afd84465175fad2431405a97e9293c8bd5e476
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/138808
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
The original revision is in Patchset 3.
Four bugs were fixed:
1. JoinEntryInstr::SuccessorCount() is not the correct way to get the
number of successor blocks from the Join block;
JoinEntryInstr::last_instruction()->SuccessorCount() must be used
instead.
2. BitVector::Equals() was non-deterministically returning 'false'
for equal vectors.
3. All blocks need to be processed at least once during the Analysis
phase (not only in the SaveResults phase).
4. We were not removing write barriers from StoreIndexed instructions,
even though we had support for it.
This reverts commit 7fd8ad5a2d.
Fixes https://github.com/dart-lang/sdk/issues/40780
Change-Id: I9650ec2c547ec49cf88ca0524e14f6c245621f6a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/138086
Commit-Queue: Samir Jindel <sjindel@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Some embedder code relies on the presence of a current isolate equal to the isolate that created the handle. This code will be need to be updated to be isolate-group aware before we can run finalizers only in the context of the isolate group instead of an isolate.
Bug: https://github.com/dart-lang/sdk/issues/40836
Change-Id: I38956c556df551e48ee7e4d0822aa5def8dc4fc5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/138013
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
It incorrectly assumed that all stores in Dart code write to Instances.
There is actually one exception, Contexts, which do not inherit from Instance.
I've added asserts to ensure this kind of bug cannot resurface.
The original change is in patchset 4.
Change-Id: Ic2d8d05e70a4de738eb9fb5980487b4f27111b8c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/136221
Commit-Queue: Samir Jindel <sjindel@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
This CL:
* Moves [Heap]/[SharedClassTable] from [Isolate] to [IsolateGroup], which
will make all isolates in the group use the same heap. The GC will use
the shared class table for object size information.
* Adds support for entering/leaving an isolate group as a helper thread
(e.g. via [Thread::EnterIsolateGroupAsHelper]). The current active
isolate group can be accessed via TLS `IsolateGroup::Current()` or
`Thread::isolate_group_`. When entering as a helper thread there will be
no current isolate.
* Changes the GC to use the above mechanism and ensures GC works without
a currently active isolate. The GC will use information purely available via
[IsolateGroup]. The GC will iterate all isolates within an isolate
group e.g. for scanning roots.
* Makes spawning of new isolates start in their own isolate group.
Once the isolate is fully functional it's heap will be merged into
the original isolate group
* Moves ApiState, containing persistent and weak persistent handles,
from [Isolate] to [IsolateGroup], plus adds appropriate locking.
Issue https://github.com/dart-lang/sdk/issues/36097
Change-Id: Ia8e1d8aa78750e8400864200f4825395a182c004
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/126646
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
When these were originally added, we thought we would be able to use them in place of ifdefs and rely on the compiler optimizations and linker GC to remove things. This turned out not to reliably remove what we wanted removed, so we ended up with the ifdefs anyway.
Change-Id: I62e74d60d92b18a688b9dffaf77b1440c10a07ff
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/134402
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Ben Konyi <bkonyi@google.com>
- Ignore all data races on object headers. These are harmless because of higher-level constraints in the GC.
- Make all concurrent marker tasks check-in with the monitor to indicate they are done marking roots, not just the last one. This establishes synchronization between the which ever task read the handle blocks and the mutator resuming and writing to the handle blocks.
- When a helper thread exits the isolate, perform clearing of the reusable handles under the thread registry's lock so it doesn't race with the GC visiting thread roots.
- Disable background compilation when running under TSAN. There are lots of races here. Many are harmless. It's hard to annotate these without effectively disabling TSAN checks for the entire Dart heap, so simply disabling the background compiler will give us the most TSAN coverage for the time being.
- Fix race in abandoning TLABs during helper thread exit.
- Remove harmless race between concurrent sweeper and allocation of new heap pages.
- Suppress TSAN on known race in array truncation.
- Remove racy access FLAG_stacktrace_every, which is no longer needed as compilation can no longer trigger execution to evaluate constants.
Bug: https://github.com/dart-lang/sdk/issues/39611
Change-Id: Ibaa101b4f12ff5fa30678471fb37e63608ba7f8b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/128305
Reviewed-by: Siva Annamalai <asiva@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Don't trigger an interrupt when concurrent marking finishes, which could trigger cause synchronous finalization in the middle of a frame; instead wait for the next allocation failure or idle notification.
Change-Id: I969c7e8de05ee1eb70103eda2df98cb2320085b8
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/125082
Reviewed-by: Siva Annamalai <asiva@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Right now there is only one mutator working on a heap and it can
therefore access the heap's weak maps without locking. The GC is the
only other user of the weak maps and it accesses it within a safepoint
operation scope.
Once we move the heap to the isolate group there can be concurrent
accesses to the weak maps. As a preparation step this CL adds locking
around the non-GC API.
Issue https://github.com/dart-lang/sdk/issues/36097
Change-Id: I84acce24612b12a7393154cab816f0eff9c7589a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/116201
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
This CL moves heap related information (namely instance sizes and
allocation stats) out of the [ClassTable] into a [SharedClassTable].
Both classes are always in sync (i.e. they have the same number of entries).
This CL also changes GC related code to start using the size information
from the new [SharedClassTable].
In a futher step we will move the heap as well as this shared class
table out of the [Isolate] and into [IsolateGroup].
Issue https://github.com/dart-lang/sdk/issues/36097
Change-Id: Id54a89c9251ad3bbc13e60d32dc4f7bcc7f1d805
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/116064
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Ryan Macnak