Instead of handling FFI related execution state and safepoint
in assembly handle it in runtime code.
The transition needs to be done before JumpToFrame unwinds
stack because unwinding destroys exit frame and this can't
be done at safepoint as GC might be traversing the stack.
An incorrect order of operation was manifesting as crashes in
GC when one isolate in a group was encountering a lot of
exceptions thrown from an FFI call and another isolate is
triggering GCs.
To catch this in the future added a bit of validation to
ExitSafepoint runtime call which triggers when --use-slow-path
is enabled. Though after refactoring this code does not
trigger this code path anymore because it was completely
removed - but it is better than nothing.
This CL also removes a lot of unnecessary complexity which
was associated with handling this transition in the stub
itself.
TEST=ffi/vmspecific_handle_test
Bug: b/408377905
Cq-Include-Trybots: dart/try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-asan-linux-release-x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-msan-linux-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-tsan-linux-release-x64-try,vm-aot-ubsan-linux-release-x64-try,vm-aot-win-debug-x64-try,vm-aot-win-debug-x64c-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-arm64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-mac-debug-simarm64_arm64-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-fuchsia-release-x64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-arm64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-tsan-linux-release-arm64-try,vm-tsan-linux-release-x64-try,vm-ubsan-linux-release-arm64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-x64-try,vm-win-debug-x64c-try,vm-win-release-ia32-try
Change-Id: Ia073cb6bb9e1b5a0ea8514c7e048cee6019b84d6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/420324
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
By time we return from DLRT_ExitTemporaryIsolate, VM shutdown may have completed, including deleting the VM isolate heap containing the stub. Instead tail-call to DLRT_ExitTemporaryIsolate, so it may return directly to the foreign caller.
TEST=ffi/async_void_function_callbacks_test
Bug: https://github.com/dart-lang/sdk/issues/53248
Change-Id: I674ed672ea4a105ad6f72a86f6aedae7db1040dc
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/417963
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Before this change, an error reaching an FFI callback would attempt to execute the normal invocation stub from the beginning in the FFI callback's frame, which quickly crashes. After this change, the runtime recognizes this marker use of the invocation stub and returns to the FFI callback function instead.
TEST=ffi/unwind
Bug: https://github.com/dart-lang/sdk/issues/39487
Change-Id: I477cfcfc236e6cf518ebfe52860ba49e466ebf8b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/409562
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
The transitions in C++ have asserts which run in debug mode to validate
that the transitions are happening in the right order.
The transitions in the generated code had no equivalent. This CL
introduces the equivalent for the generated code.
The transitions to generated code are misnamed currently. They come
from either VM or Native, not only Native. For example the FFI
callbacks and call return sequence can already transition from native
into the VM when entering the safepoint.
These checks don't catch the invalid FFI callbacks.
Bug: https://github.com/dart-lang/sdk/issues/60021
These are caught when looking up the FFI metadata:
https://dart-review.googlesource.com/c/sdk/+/407023
This CL omits to implement the check in ia32, since ia32 is scheduled
for removal.
The check is only enabled in debug mode. The asserts for the C++ are
also only run in debug mode. And enabling this check in release for
AOT regresses the parent CL benchmark by ~30% on Mac Arm64.
TEST=test/ffi
Change-Id: Ieaf1e43533baae294af37b2e171b68bd314fdbe0
Cq-Include-Trybots: dart/try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-asan-linux-release-x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-msan-linux-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-tsan-linux-release-x64-try,vm-aot-ubsan-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-aot-win-debug-x64-try,vm-aot-win-debug-x64c-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-arm64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-fuchsia-release-arm64-try,vm-fuchsia-release-x64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-arm64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-tsan-linux-release-arm64-try,vm-tsan-linux-release-x64-try,vm-ubsan-linux-release-arm64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-arm64-try,vm-win-debug-x64-try,vm-win-debug-x64c-try,vm-win-release-ia32-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/407021
Reviewed-by: Liam Appelbe <liama@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
On architectures with a weak memory model, there is the possibility that the concurrent marker can see the publishing store of a new object before it sees the initializing store of that object's header. On an M1, the barrier to prevent reordering these stores is fairly cheap, so we emit this barrier on Mac/iOS ARM64. Otherwise, this barrier is very expensive (or at least expensive for some hardware within the ABI), so instead we avoid the race by deferring marking of objects inside an active TLAB.
Disable TSAN instrumentation on the marker setting the mark bit, as TSAN does not understand fences.
TEST=ooo arm64 machines
Bug: https://github.com/dart-lang/sdk/issues/56845
Change-Id: I0676661a7cf941fdc6b451e516d890c26826bb3b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/389265
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
No client of the VM uses this flag, only tests, and this flag was always
set to false in AOT mode. Thus, remove uses of this flag and instead
always lazily create dispatchers as needed when resolving method names
in JIT mode.
Remove the implicit value of `allow_add` for some Resolver
static methods. For callers that previously depended on the implicit
`true` value (which includes the AOT precompilier), pass `true` for
uses in the compiler and pass `!FLAG_precompiled_mode` for uses in the
runtime. Assert that `allow_add` is false when these methods are invoked
from the precompiled runtime.
Remove Resolver static methods that are no longer used.
TEST=ci
Change-Id: Ib6a7354f7a859e86743c381513a4129c14895753
Cq-Include-Trybots: luci.dart.try:vm-linux-debug-x64-try,vm-linux-release-x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-release-x64-try,vm-aot-mac-release-arm64-try,vm-mac-debug-arm64-try,vm-mac-release-arm64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/366668
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
When uploading CLs, the presubmit checks verify that the lines in the
diff are formatted correctly according to `git cl format runtime`.
However, when `buildtools/<os>-<arch>/clang/bin/clang-format` is
updated, it does not force reformatting of files that would be
reformatted.
This leads to two issues:
* Inconsistent style within the code base and within a single file.
* Spurious reformatting in CLs when (1) clang-format is used on the
whole file, or (2) the diff lines overlap.
`clang-format` doesn't change that frequently, so in general this is
not a large issue, but I've seen a bit too many "spurious formatting,
please revert" comments on CLs recently.
This CL formats the runtime to be in line with the current pinned
`clang-format`:
```
$ find runtime/ -iname *.h -o -iname *.cc | xargs buildtools/mac-arm64/clang/bin/clang-format -i
```
`git cl format` (which only formats changed lines, and does so with
`clang-format`) seems to not agree with itself, or clang-format, or
cpplint in a handful of places. This CL adds `// clang-format off`
for these. (See previous patchsets for the specific instances.)
TEST=A variety of bots including GCC, MacOS and Windows.
Change-Id: I470892e898971899fda14bb3b8f2c8efefd67686
Cq-Include-Trybots: luci.dart.try:vm-gcc-linux-try,vm-ffi-qemu-linux-release-riscv64-try,vm-ffi-qemu-linux-release-arm-try,vm-aot-win-debug-x64-try,vm-win-debug-x64c-try,vm-mac-debug-x64-try,vm-mac-debug-arm64-try,vm-aot-linux-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/362780
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Instead of having special support for targeting catch entry as a
lazy deoptimization target, split this into two independent
operations:
* Perform lazy deoptimization at call.
* Rethrow the exception once deoptimization is complete.
TEST=ci
Change-Id: Ib275d615bc6717e1184e88602ae5459d5b771f2b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/361580
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
Remove CompareWithCompressedFieldFromOffset, which has no uses.
Rename the LoadFromOffset and StoreFromOffset methods that took
Addresses to Load and Store, respectively. This makes the names
of the Assembler methods more uniform:
* Takes an address: Load, Store, LoadField, LoadCompressedField,
StoreIntoObject, StoreCompressedIntoObject, LoadSmi,
LoadCompressedSmi, etc.
* Takes a base register and an offset: LoadFromOffset, StoreToOffset,
LoadFieldFromOffset, LoadCompressedFieldFromOffset,
StoreIntoObjectOffset, StoreCompressedIntoObjectOffset,
LoadSmiFromOffset, LoadCompressedSmiFromOffset, etc.
Create AssemblerBase methods for loading and storing compressed
pointers that weren't already there, as well as the corresponding
methods for loading and storing uncompressed values.
Make non-virtual methods that load and store uncompressed fields
that call the corresponding method for loading from and storing to
memory regions, adjusting the address or offset accordingly. This
avoids needing per-architecture overrides for these.
Make non-virtual methods that load compressed fields, calling the
corresponding method for loading a compressed value from a memory
region. (Since compressed pointers are only stored in Dart objects,
and stores into a Dart object may require a barrier, there is no
method for storing a compressed value into an arbitrary memory region.)
Create pure virtual methods for loading from or storing to an Address
or any method that does not have both an Address-taking and a
base register and offset pair-taking version (e.g., LoadAcquire).
Create methods for loading from or storing to a base register
and an offset. The base implementation takes the base register and
offset and creates an Address from it, then calls the Address-taking
equivalent. These methods are non-virtual when the implementation is
the same on all architectures and virtual to allow overriding when
necessary.
Make a non-virtual method for loading uncompressed Smis, since all
architectures have the same code for this, including the DEBUG check.
If compressed pointers are not being used, all the methods for
compressed pointers are non-virtual methods that call the
corresponding method for uncompressed values.
If compressed pointers are being used:
* Install pure virtual methods for loading compressed values from
and storing compressed values to an Address or any method that does
not have both an Address-taking and a base register and offset
pair-taking version (e.g., LoadAcquireCompressed).
* Install virtual methods for loading compressed values from and
storing compressed values to a base register and offset. Like the
uncompressed implementation, the base implementation of these
create an Address and call the Address-taking equivalent, and these
implementations are overridden on ARM64.
* Install a non-virtual method for loading compressed Smis, since the
only difference is that it loads a zero-extended 32-bit value, which
AssemblerBase can do.
TEST=ci (refactoring only)
Change-Id: I934791d26a6e2cdaa6ac5f188b0fd89dbdc491d1
Cq-Include-Trybots: luci.dart.try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-arm64-try,vm-win-debug-x64-try,vm-win-release-ia32-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/359861
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
This CL
* adds a `kind` to `CheckWritableInstr`,
* starts using `CheckWritableInstr` in JIT mode, and
* adds support for `CheckWritableInstr` ia32.
TEST=tests/ffi/vmspecific_native_finalizer_deeply_immutable_test.dart
Closes: https://github.com/dart-lang/sdk/issues/55067
Change-Id: I0b397daba12cfc8b885401169889f7cd7c040166
Cq-Include-Trybots: dart-internal/g3.dart-internal.try:g3-cbuild-try
Cq-Include-Trybots: dart/try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-aot-win-debug-x64c-try,vm-aot-win-release-x64-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-arm64-try,vm-win-debug-x64-try,vm-win-debug-x64c-try,vm-win-release-ia32-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/359223
Auto-Submit: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Note this changes the error in AOT to tell what is the typed data that
is not writable.
This CL eases the follow up CL which starts reusing the
`CheckWritableInstr` and accompanying slow path and runtime entry for
other messages.
TEST=tests/lib/typed_data/polymorphic_unmodifiable_typed_data_test.dart
Bug: https://github.com/dart-lang/sdk/issues/55067
Change-Id: Icf26d119501059e2ea4e885897539ebcb1063da0
Cq-Include-Trybots: dart-internal/g3.dart-internal.try:g3-cbuild-try
Cq-Include-Trybots: dart/try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-linux-debug-simarm_x64-try,vm-aot-linux-debug-simriscv64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-aot-win-debug-x64c-try,vm-aot-win-release-x64-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-linux-debug-ia32-try,vm-linux-debug-simriscv64-try,vm-linux-debug-x64-try,vm-linux-debug-x64c-try,vm-linux-release-simarm-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-arm64-try,vm-win-debug-x64-try,vm-win-debug-x64c-try,vm-win-release-ia32-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/359063
Reviewed-by: Ilya Yanok <yanok@google.com>
Previously, method extractors (getter functions which return
tear-offs) where implemented in the following way:
Method extractor body is an asm intrinsic which loads a few registers
and jumps to BuildMethodExtractor stubs.
BuildMethodExtractor stub loads more registers, calls AllocateClosure
stub and initializes a few fields of a closure (this stub also used
to call AllocateContext but it is no longer needed for tear-offs).
AllocateClosure allocates a closure object and initializes closure
fields.
Instead of doing 2 hops to the object allocation, method extractor
body now calls AllocateClosure stubs directly which initializes
all closure fields. This saves a jump and a handful of instructions
to load certain things on registers and set closure fields.
Assembler intrinsic for method extractor body and
architecture-specific BuildMethodExtractor stubs are now removed.
Also, as method extractors are really tiny, they are now always
inlined.
TEST=ci
Change-Id: Icfeed18414659a76c7d24d3f6f53a76c91e5bbd3
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/356302
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Previously, when taking a tear-off, a separate Context object was
allocated in order to capture receiver. Now, receiver is stored directly
in the Closure object in the 'context' field. This saves 1 object
allocation per tear-off and makes tear-offs cheaper compared to
explicit closures which can share context with other closures.
Benchmarks in AOT mode:
x64:
TearOff.NotInlined +40%
TearOff.NotInlined.InTry +43%
TearOff.Inlined.InTry +47%
arm64:
TearOff.NotInlined +27-43%
TearOff.NotInlined.InTry +29-43%
TearOff.Inlined.InTry +58-94%
arm64c:
TearOff.NotInlined +71%
TearOff.NotInlined.InTry +72%
TearOff.Inlined.InTry +96%
TEST=ci
Issue: https://github.com/dart-lang/sdk/issues/54808
Change-Id: I3ad95e8a8a4fc23f856bbc0fe238da58a9d25b8d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/350945
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
- Initial and final marking no longer visit all of new-space, reducing the STW pause for major GC.
- A scavenge during concurrent marking must forward / filter objects in the marking worklist that are moved / collected, increasing the STW pause for minor GC.
- Unreachable intergenerational cycles and weak references are collected in the next mark-sweep instead of first requiring enough scavenges to promote the whole cycle or weak target into old-space.
- Artificial minor GCs are no longer needed to avoid memory leaks from back-to-back major GCs.
- reachabilityBarrier is now just a count of major GCs.
TEST=ci
Change-Id: I1e653c9b5d3e02e45b280302c832157a75788db6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/345350
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
This reverts commit a3b7c323b0.
Reason for revert: We are seeing crashes in some internal testing
si_signo=Segmentation fault(11), si_code=SEGV_ACCERR(2), si_addr=0x7f1cadb85bc6
version=3.3.0-edge+google3-v2 (google3) (google3) on "linux_x64"
pid=8967, thread=8969, isolate_group=main(0x5583571aaaf0), isolate=(nil)((nil))
os=linux, arch=x64, comp=no, sim=no
isolate_instructions=7f1cad8b7e40, vm_instructions=7f1cad8b1000
fp=7f1cacc7ebc0, sp=7f1cacc7eb90, pc=5583568155bc
pc 0x00005583568155bc fp 0x00007f1cacc7ebc0 dart::MarkingVisitorBase<true>::VisitPointers(dart::ObjectPtr*, dart::ObjectPtr*)+0x15c
pc 0x0000558356975a1d fp 0x00007f1cacc7ec60 dart::StackFrame::VisitObjectPointers(dart::ObjectPointerVisitor*)+0x1bd
pc 0x0000558356935c29 fp 0x00007f1cacc7ecc0 dart::UntaggedSuspendState::VisitSuspendStatePointers(dart::SuspendStatePtr, dart::ObjectPointerVisitor*)+0x89
pc 0x0000558356815948 fp 0x00007f1cacc7ed10 dart::MarkingVisitorBase<true>::DrainMarkingStackWithPauseChecks()+0x128
pc 0x0000558356815749 fp 0x00007f1cacc7ed80 dart::ConcurrentMarkTask::Run()+0x69
Original change's description:
> [vm, gc] Mark through new-space.
>
> - Initial and final marking no longer visit all of new-space, reducing the STW pause for major GC.
> - A scavenge during concurrent marking must forward / filter objects in the marking worklist that are moved / collected, increasing the STW pause for minor GC.
> - Unreachable intergenerational cycles and weak references are collected in the next mark-sweep instead of first requiring enough scavenges to promote the whole cycle or weak target into old-space.
> - Artificial minor GCs are no longer needed to avoid memory leaks from back-to-back major GCs.
> - reachabilityBarrier is now just a count of major GCs.
>
> TEST=ci
> Change-Id: Ic7754e8d972763654eae2b7faa8670735d9cda3f
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/340644
> Reviewed-by: Siva Annamalai <asiva@google.com>
> Commit-Queue: Ryan Macnak <rmacnak@google.com>
Change-Id: Ice07b4eb5bef3b41c9618ef0ca7759de006ffe00
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/343060
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Siva Annamalai <asiva@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
- Initial and final marking no longer visit all of new-space, reducing the STW pause for major GC.
- A scavenge during concurrent marking must forward / filter objects in the marking worklist that are moved / collected, increasing the STW pause for minor GC.
- Unreachable intergenerational cycles and weak references are collected in the next mark-sweep instead of first requiring enough scavenges to promote the whole cycle or weak target into old-space.
- Artificial minor GCs are no longer needed to avoid memory leaks from back-to-back major GCs.
- reachabilityBarrier is now just a count of major GCs.
TEST=ci
Change-Id: Ic7754e8d972763654eae2b7faa8670735d9cda3f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/340644
Reviewed-by: Siva Annamalai <asiva@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
This reverts commit 5daaa7d9eb.
Reason for revert: internal crashes
Original change's description:
> [vm, gc] Mark through new-space.
>
> - Initial and final marking no longer visit all of new-space, reducing the STW pause for major GC.
> - A scavenge during concurrent marking must forward / filter objects in the marking worklist that are moved / collected, increasing the STW pause for minor GC.
> - Unreachable intergenerational cycles and weak references are collected in the next mark-sweep instead of first requiring enough scavenges to promote the whole cycle or weak target into old-space.
> - Artificial minor GCs are no longer needed to avoid memory leaks from back-to-back major GCs.
> - reachabilityBarrier is now just a count of major GCs.
>
> TEST=ci
> Change-Id: I4a6a23273d8ecb78c640f054731d4ceb737bfc4d
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/325840
> Reviewed-by: Siva Annamalai <asiva@google.com>
> Commit-Queue: Ryan Macnak <rmacnak@google.com>
Change-Id: I8a50074db343c63c14f0487ae8b4f5fee2c4ae76
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/330720
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
- Initial and final marking no longer visit all of new-space, reducing the STW pause for major GC.
- A scavenge during concurrent marking must forward / filter objects in the marking worklist that are moved / collected, increasing the STW pause for minor GC.
- Unreachable intergenerational cycles and weak references are collected in the next mark-sweep instead of first requiring enough scavenges to promote the whole cycle or weak target into old-space.
- Artificial minor GCs are no longer needed to avoid memory leaks from back-to-back major GCs.
- reachabilityBarrier is now just a count of major GCs.
TEST=ci
Change-Id: I4a6a23273d8ecb78c640f054731d4ceb737bfc4d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/325840
Reviewed-by: Siva Annamalai <asiva@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
This reverts commit 3fb88e4c66.
Reason for revert: b/297175670
Original change's description:
> [vm, gc] Mark through new-space.
>
> - Initial and final marking no longer visit all of new-space, reducing the STW pause for major GC.
> - A scavenge during concurrent marking must forward / filter objects in the marking worklist that are moved / collected, increasing the STW pause for minor GC.
> - Unreachable intergenerational cycles and weak references are collected in the next mark-sweep instead of first requiring enough scavenges to promote the whole cycle or weak target into old-space.
> - Artificial minor GCs are no longer needed to avoid memory leaks from back-to-back major GCs.
> - reachabilityBarrier is now just a count of major GCs.
>
> TEST=ci
> Change-Id: I6362802cd93ba5ba9c39f116ddff82e4feb4c312
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/321304
> Commit-Queue: Ryan Macnak <rmacnak@google.com>
> Reviewed-by: Siva Annamalai <asiva@google.com>
Change-Id: I33075156160dc35861355d738a5776b74dce88b9
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/322344
Reviewed-by: Martin Kustermann <kustermann@google.com>
Auto-Submit: Ivan Inozemtsev <iinozemtsev@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
- Initial and final marking no longer visit all of new-space, reducing the STW pause for major GC.
- A scavenge during concurrent marking must forward / filter objects in the marking worklist that are moved / collected, increasing the STW pause for minor GC.
- Unreachable intergenerational cycles and weak references are collected in the next mark-sweep instead of first requiring enough scavenges to promote the whole cycle or weak target into old-space.
- Artificial minor GCs are no longer needed to avoid memory leaks from back-to-back major GCs.
- reachabilityBarrier is now just a count of major GCs.
TEST=ci
Change-Id: I6362802cd93ba5ba9c39f116ddff82e4feb4c312
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/321304
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
This reverts commit e95e7b8e96.
Reason for revert: suspected cause to non-deterministic AOT binaries crashes, b/296014654
Original change's description:
> [vm, gc] Mark through new-space.
>
> - Initial and final marking no longer visit all of new-space, reducing the STW pause for major GC.
> - A scavenge during concurrent marking must forward / filter objects in the marking worklist that are moved / collected, increasing the STW pause for minor GC.
> - Unreachable intergenerational cycles and weak references are collected in the next mark-sweep instead of first requiring enough scavenges to promote the whole cycle or weak target into old-space.
> - Artificial minor GCs are no longer needed to avoid memory leaks from back-to-back major GCs.
> - reachabilityBarrier is now just a count of major GCs.
>
> TEST=ci
> Change-Id: I8c2c64b120766571b62d3bd8dab37ae81c2dca98
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/319583
> Commit-Queue: Ryan Macnak <rmacnak@google.com>
> Reviewed-by: Siva Annamalai <asiva@google.com>
Change-Id: Idda542c7c657d4f14c836423b173c9b067132212
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/320820
Reviewed-by: Martin Kustermann <kustermann@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Ilya Yanok <yanok@google.com>
- Initial and final marking no longer visit all of new-space, reducing the STW pause for major GC.
- A scavenge during concurrent marking must forward / filter objects in the marking worklist that are moved / collected, increasing the STW pause for minor GC.
- Unreachable intergenerational cycles and weak references are collected in the next mark-sweep instead of first requiring enough scavenges to promote the whole cycle or weak target into old-space.
- Artificial minor GCs are no longer needed to avoid memory leaks from back-to-back major GCs.
- reachabilityBarrier is now just a count of major GCs.
TEST=ci
Change-Id: I8c2c64b120766571b62d3bd8dab37ae81c2dca98
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/319583
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
This reverts commit 6194209b28.
Reason for revert: issues on arm32
Original change's description:
> [vm, gc] Mark through new-space.
>
> - Initial and final marking no longer visit all of new-space, reducing the STW pause for major GC.
> - A scavenge during concurrent marking must forward / filter objects in the marking worklist that are moved / collected, increasing the STW pause for minor GC.
> - Unreachable intergenerational cycles and weak references are collected in the next mark-sweep instead of first requiring enough scavenges to promote the whole cycle or weak target into old-space.
> - Artificial minor GCs are no longer needed to avoid memory leaks from back-to-back major GCs.
> - reachabilityBarrier is now just a count of major GCs.
>
> TEST=ci
> Change-Id: I3668a2e56821f9eadf96e38c228dab27be656016
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/309826
> Reviewed-by: Siva Annamalai <asiva@google.com>
> Commit-Queue: Ryan Macnak <rmacnak@google.com>
Change-Id: I434eb595c9e7858efc8c9b07cbca954e5649f506
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/319321
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
- Initial and final marking no longer visit all of new-space, reducing the STW pause for major GC.
- A scavenge during concurrent marking must forward / filter objects in the marking worklist that are moved / collected, increasing the STW pause for minor GC.
- Unreachable intergenerational cycles and weak references are collected in the next mark-sweep instead of first requiring enough scavenges to promote the whole cycle or weak target into old-space.
- Artificial minor GCs are no longer needed to avoid memory leaks from back-to-back major GCs.
- reachabilityBarrier is now just a count of major GCs.
TEST=ci
Change-Id: I3668a2e56821f9eadf96e38c228dab27be656016
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/309826
Reviewed-by: Siva Annamalai <asiva@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
If we are generating a SubtypeTestCache for a known type that only
needs instantiator type arguments to be fully instantiated, then
don't store and check against the unneeded function type arguments.
TEST=vm/cc/STC, vm/cc/TTS, ci
Change-Id: I370adf820168079322b8a87811057670a47ee6b3
Cq-Include-Trybots: luci.dart.try:vm-tsan-linux-release-x64-try,vm-reload-rollback-linux-release-x64-try,vm-reload-linux-release-x64-try,vm-linux-release-x64-try,vm-aot-tsan-linux-release-x64-try,vm-aot-linux-release-x64-try,vm-aot-linux-product-x64-try,vm-aot-linux-debug-x64-try,vm-linux-debug-x64-try,vm-mac-debug-arm64-try,vm-aot-mac-release-arm64-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-linux-debug-ia32-try,vm-linux-release-ia32-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/311382
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Currently the generational and incremental barriers cannot both apply for the same store because the former only applies for old->new stores and the latter only applies for old->old stores. Marking through new-space will cause the incremental barrier to apply to all stores.
TEST=ci
Change-Id: I40d492cc8b5b2fff8d5d4bc8625d45f2e6a2d488
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/309830
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
This also lowers the threshold for converting from a linear cache
to a hash-based cache from 100 to 30 on non-IA32 architectures.
(IA32 still goes to runtime if there's a hash-based cache.)
For SubtypeTestCache benchmarks above this threshold, we see an
improvement varying from ~50-100% to ~700-800% on all architectures,
from lowest number of checks (50) to highest (1000).
For SubtypeTestCache benchmarks below this threshold, no major
changes are seen: generally <5% improvement or <5% regression per
check at most.
TEST=vm/cc/TTS
Change-Id: I83aa7c085ab5a411e944ec660d6b8eba7a788ee0
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-simriscv64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-release-x64-try,vm-aot-linux-product-x64-try,vm-aot-linux-release-simarm64-try,vm-aot-linux-release-simarm_x64-try,vm-aot-linux-debug-x64c-try,vm-aot-tsan-linux-release-x64-try,vm-aot-mac-release-arm64-try,vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-aot-dwarf-linux-product-x64-try,vm-linux-release-ia32-try,vm-linux-debug-x64c-try,vm-linux-debug-x64-try,vm-linux-debug-simriscv64-try,vm-linux-release-simarm64-try,vm-linux-release-simarm-try,vm-mac-release-arm64-try,vm-mac-release-x64-try,vm-tsan-linux-release-x64-try,vm-reload-rollback-linux-release-x64-try,vm-reload-linux-release-x64-try,vm-ffi-qemu-linux-release-arm-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/308941
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
In particular, avoid storing/comparing the destination type unless the
SubtypeTestCache was created for a dynamic AssertAssignable instruction.
TEST=vm/cc/STC_{Linear,Hash}Lookup, vm/cc/TTS
Bug: https://github.com/dart-lang/sdk/issues/48345
Change-Id: I279c69a668ce19785785cf71707acf951f2b2133
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-simriscv64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-release-x64-try,vm-aot-linux-product-x64-try,vm-aot-linux-release-simarm64-try,vm-aot-linux-release-simarm_x64-try,vm-aot-linux-debug-x64c-try,vm-aot-tsan-linux-release-x64-try,vm-aot-mac-release-arm64-try,vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-aot-dwarf-linux-product-x64-try,vm-linux-release-ia32-try,vm-linux-debug-x64c-try,vm-linux-debug-x64-try,vm-linux-debug-simriscv64-try,vm-linux-release-simarm64-try,vm-linux-release-simarm-try,vm-mac-release-arm64-try,vm-mac-release-x64-try,vm-tsan-linux-release-x64-try,vm-reload-rollback-linux-release-x64-try,vm-reload-linux-release-x64-try,vm-ffi-qemu-linux-release-arm-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/310340
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Also eliminates the need to store the delayed type arguments on the
stack during the search loop on ARM.
TEST=vm/cc/TTS
Change-Id: Ife01cc8d255724c272bc277eec397486cedc07a7
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-simriscv64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-release-x64-try,vm-aot-linux-product-x64-try,vm-aot-linux-release-simarm64-try,vm-aot-linux-release-simarm_x64-try,vm-aot-linux-debug-x64c-try,vm-aot-tsan-linux-release-x64-try,vm-aot-mac-release-arm64-try,vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-aot-dwarf-linux-product-x64-try,vm-linux-release-ia32-try,vm-linux-debug-x64c-try,vm-linux-debug-x64-try,vm-linux-debug-simriscv64-try,vm-linux-release-simarm64-try,vm-linux-release-simarm-try,vm-mac-release-arm64-try,vm-mac-release-x64-try,vm-tsan-linux-release-x64-try,vm-reload-rollback-linux-release-x64-try,vm-reload-linux-release-x64-try,vm-ffi-qemu-linux-release-arm-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/307681
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
We use specialized type testing stubs in `as` checks to quickly answer
whether the type of an object is assignable to a given type. However,
perfect specialization of the type testing stub is not possible in all
cases. Thus, we use SubtypeTestCaches (STCs) to cache the result of
checking assignability in the runtime to avoid repeated runtime calls
for the same check.
Previously, the STC was linear and capped at 100 entries. This cap was
for two reasons:
* STC are created per `as` check, not per checked type, so there could
be many STCs created for a given type if assignability to that type
is checked many times throughout the code.
* STC were strictly linear, and thus eventually the cost of
traversing the cache outweighs the cost of returning a false
negative that requires going to the runtime and doing the check there.
This CL provides a partial solution to the second reason by performing
similar work to that done for instantiation caches in TypeArguments
(4f925105), where the STC is linear up to a certain threshold, and then
converted to a hash-based STC. Like that CL, only the runtime currently
handles the lookup of cached results in hash-based STCs, with the
changes needed for the `SubtypeNTestCacheStub`s coming in a followup CL.
Since the stubs have not yet been adjusted, we keep the maximum size of
linear caches the same at 100, and allow hash-based STCs to grow to a
maximum size of 1000 since the STCs are still created per `as` check.
This means once the STC has grown over 100 entries, all STC checks
require a runtime call, but for checks cached in the STC, the answer is
found and provided without either doing the slower full runtime check
for assignability and without an attempt to respecialize the type
testing stub when applicable.
Our benchmarks generally see at most a ~2-5% performance regression for
cases where the STC remains linear (except for on the Raspberry Pi 4, where the hit ranges from ~25%-40%), but a performance improvement of
~500-1300% for cases where the STC is converted to a hash table.
TEST=vm/cc/TTS_STC_{Some,Many}Asserts
Change-Id: Ie8629708b5c12fa8464359a2f77c6da56ba46c0c
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-simriscv64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-release-x64-try,vm-aot-linux-product-x64-try,vm-aot-linux-release-simarm64-try,vm-aot-linux-release-simarm_x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-aot-dwarf-linux-product-x64-try,vm-linux-release-ia32-try,vm-linux-debug-x64c-try,vm-linux-debug-x64-try,vm-linux-debug-simriscv64-try,vm-linux-release-simarm64-try,vm-linux-release-simarm-try,vm-mac-release-arm64-try,vm-mac-release-x64-try,vm-tsan-linux-release-x64-try,vm-reload-rollback-linux-release-x64-try,vm-reload-linux-release-x64-try,vm-ffi-qemu-linux-release-arm-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/284682
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Alexander Aprelev