Instead of handling FFI related execution state and safepoint
in assembly handle it in runtime code.
The transition needs to be done before JumpToFrame unwinds
stack because unwinding destroys exit frame and this can't
be done at safepoint as GC might be traversing the stack.
An incorrect order of operation was manifesting as crashes in
GC when one isolate in a group was encountering a lot of
exceptions thrown from an FFI call and another isolate is
triggering GCs.
To catch this in the future added a bit of validation to
ExitSafepoint runtime call which triggers when --use-slow-path
is enabled. Though after refactoring this code does not
trigger this code path anymore because it was completely
removed - but it is better than nothing.
This CL also removes a lot of unnecessary complexity which
was associated with handling this transition in the stub
itself.
TEST=ffi/vmspecific_handle_test
Bug: b/408377905
Cq-Include-Trybots: dart/try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-asan-linux-release-x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-msan-linux-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-tsan-linux-release-x64-try,vm-aot-ubsan-linux-release-x64-try,vm-aot-win-debug-x64-try,vm-aot-win-debug-x64c-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-arm64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-mac-debug-simarm64_arm64-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-fuchsia-release-x64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-arm64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-tsan-linux-release-arm64-try,vm-tsan-linux-release-x64-try,vm-ubsan-linux-release-arm64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-x64-try,vm-win-debug-x64c-try,vm-win-release-ia32-try
Change-Id: Ia073cb6bb9e1b5a0ea8514c7e048cee6019b84d6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/420324
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
On architectures with a weak memory model, there is the possibility that the concurrent marker can see the publishing store of a new object before it sees the initializing store of that object's header. On an M1, the barrier to prevent reordering these stores is fairly cheap, so we emit this barrier on Mac/iOS ARM64. Otherwise, this barrier is very expensive (or at least expensive for some hardware within the ABI), so instead we avoid the race by deferring marking of objects inside an active TLAB.
Disable TSAN instrumentation on the marker setting the mark bit, as TSAN does not understand fences.
TEST=ooo arm64 machines
Bug: https://github.com/dart-lang/sdk/issues/56845
Change-Id: I0676661a7cf941fdc6b451e516d890c26826bb3b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/389265
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
For non-x86 architectures, these forms directly translate to a
single instruction when possible. On x86 architectures,
XImmediate(dst, src, imm) where X is in {Lsl, ArithmeticShiftRight, And}
can be translated to:
MoveRegister(dst, src);
XImmediate(dst, imm);
in the general case since MoveRegister(dst, src) is a no-op if dst and
src are the same.
Also add versions of these assembler macros that take an OperandSize
and handle 32-bit OperandSizes appropriately on 64-bit architectures.
Ensure the implementation for LslImmediate and
ArithmeticShiftRightImmediate emits no instructions if shift == 0,
dst == src, and OperandSizeInBits(sz) == kBitsPerWord on all
architectures.
-----
Other changes:
Fix ConstantExpression::EmitMoveToLocation on ARM64 to match
other architectures, which allow any word-sized or less unboxed
integer representation.
Fill out ExtendValue on RISCV for previously unimplemented
OperandSizes, using the Zba and Zbb extensions when possible.
-----
TEST=vm/cc/Assembler_AndImmediate vm/cc/Assembler_LslImmediate
vm/cc/Assembler_ArithmeticShiftRightImmediate
Separated out of https://dart-review.googlesource.com/c/sdk/+/378706
for easier debugging/reviewing.
Change-Id: I721f1334784f7011a958ea8af29f0e56c620726c
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-release-simarm_x64-try,vm-aot-linux-debug-x64-try,vm-aot-mac-release-arm64-try,vm-aot-mac-product-arm64-try,vm-aot-linux-product-x64-try,vm-aot-dwarf-linux-product-x64-try,vm-aot-linux-debug-simarm_x64-try,vm-linux-debug-x64-try,vm-mac-debug-arm64-try,vm-mac-release-arm64-try,vm-linux-debug-ia32-try,vm-aot-android-release-arm64c-try,vm-ffi-android-debug-arm64c-try,vm-aot-linux-debug-x64c-try,vm-linux-debug-x64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/386180
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Assembler::AddressCanHoldConstantIndex, which is used by
MakeLocationSummary to determine whether to keep the index constant or
to allocate a register for it, performs the check:
const int64_t disp =
index * index_scale +
(is_external ? 0 : target::Instance::DataOffsetFor(cid) - kHeapObjectTag);
return Utils::IsInt(32, disp);
However, in Assembler::ElementAddressForIntIndex, used by EmitNativeCode to
produce an appropriate Address for the constant index, the code performs
a slightly different check:
const int64_t disp = static_cast<int64_t>(index) * index_scale +
target::Instance::DataOffsetFor(cid);
ASSERT(Utils::IsInt(32, disp));
return FieldAddress(array, static_cast<int32_t>(disp));
Thus, if a constant index produces the displacement kMaxInt32 + 1,
MakeLocationSummary will appropriately keep the index as a constant, but
the ASSERT in ElementAddressForIntIndex will trigger.
Modify ElementAddressForIntIndex in the X64 and IA32 assemblers so that
the same displacement is checked in both places.
TEST=vm/cc/IRTest_Regress_56588
Fixes: https://github.com/dart-lang/sdk/issues/56588
Change-Id: I504d6f92230540d74409a99b64b444acdb9c85f7
Cq-Include-Trybots: luci.dart.try:vm-linux-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/382801
Auto-Submit: Tess Strickland <sstrickl@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Delay changing Thread::vm_tag on callback entry and restore the tag early on callback return so that the profiler doesn't see the "running Dart" tag unless it can also see the fake return address marking the entry frame.
TEST=ffi/async_void_function_callbacks, ffi/function_callbacks_subtype, ffi/function_callbacks, ffi/isolate_local_function_callbacks
Bug: https://github.com/dart-lang/sdk/issues/52814
Change-Id: I40d80ec7c44063d078db0e211565e2d127c6b81e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/367460
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
This reverts commit bc0f02e4c8.
Reason for revert: https://github.com/dart-lang/sdk/issues/55754
Original change's description:
> [vm, gc] Incremental compaction.
>
> At the beginning of a major GC cycle, select some mostly-empty pages to be evacuated. Mark the pages and the objects on these pages. Apply a write barrier for stores creating old -> evacuation candidate pointers, and discover any such pointers that already exist during marking.
>
> At the end of a major GC cycle, evacuate objects from these pages. Forward pointers of objects in the remembered set and new-space. Free the evacuated pages.
>
> This compaction is incremental in the sense that creating the remembered set is interleaved with mutator execution. The evacuation step, however, is stop-the-world.
>
> Write-barrier elimination for x.slot = x is removed. Write-barrier elimination for x.slot = constant is removed in the JIT, kept for AOT but snapshot pages are marked as never-evacuate.
>
> TEST=ci
> Bug: https://github.com/dart-lang/sdk/issues/52513
> Change-Id: Icbc29ef7cb662ef8759b8c1d7a63b7af60766281
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/357760
> Reviewed-by: Alexander Aprelev <aam@google.com>
> Commit-Queue: Ryan Macnak <rmacnak@google.com>
Bug: https://github.com/dart-lang/sdk/issues/52513
Change-Id: I565ad6c0fca283d33f605c10f181bc0a59e7d2b2
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/366965
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Alexander Aprelev <aam@google.com>
Auto-Submit: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
At the beginning of a major GC cycle, select some mostly-empty pages to be evacuated. Mark the pages and the objects on these pages. Apply a write barrier for stores creating old -> evacuation candidate pointers, and discover any such pointers that already exist during marking.
At the end of a major GC cycle, evacuate objects from these pages. Forward pointers of objects in the remembered set and new-space. Free the evacuated pages.
This compaction is incremental in the sense that creating the remembered set is interleaved with mutator execution. The evacuation step, however, is stop-the-world.
Write-barrier elimination for x.slot = x is removed. Write-barrier elimination for x.slot = constant is removed in the JIT, kept for AOT but snapshot pages are marked as never-evacuate.
TEST=ci
Bug: https://github.com/dart-lang/sdk/issues/52513
Change-Id: Icbc29ef7cb662ef8759b8c1d7a63b7af60766281
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/357760
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Add OperandSize arguments to the non-compressed Store methods so that
the compressed Store methods can be implemented as calls to the
corresponding non-compressed method using kObjectBytes. This makes it
so that all compressed Store methods are now non-virtual.
Add a scratch register argument to all Store methods that involve a
write barrier that defaults to TMP. Originally only the IA32
implementation took a scratch register, with the others using TMP or
TMP2 internally.
Since all architectures can represent a register/offset pair as an
Address without losing information, make all the offset-based load
and store methods have an implementation that creates an Address or
FieldAddress and calls the corresponding address-based store method.
This makes all the offset-based load and store methods non-virtual.
After this, most of the Store methods are non-virtual, with
only the base methods used to implement them as virtual:
* Store
* StoreRelease
* StoreObjectIntoObjectNoBarrier
* StoreBarrier
* ArrayStoreBarrier
* VerifyStoreNeedsNoWriteBarrier (for DEBUG use only)
The Load methods can't be consolidated quite so much, as the
base methods for handling non-Smi compressed pointers must be defined
per-architecture, as each compressed pointer architecture handles
adding the upper bits for the heap separately.
Have LoadFieldInstr::EmitNativeCode use Assembler::LoadToSlot when the
result location is a single integer register and
StoreFieldInstr::EmitNativeCode use Assembler::StoreToSlot and
Assembler::StoreToSlotNoBarrier when the value location is a single
integer register to avoid code duplication between those methods.
Handle both compressed and uncompressed Smi fields in LoadFromSlot.
TEST=ci (refactoring)
Cq-Include-Trybots: luci.dart.try:vm-aot-android-release-arm64c-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-x64-try,vm-win-release-ia32-try
Change-Id: I60cc03776af220ed87918664bb4b9abafff2788a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/360641
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Remove CompareWithCompressedFieldFromOffset, which has no uses.
Rename the LoadFromOffset and StoreFromOffset methods that took
Addresses to Load and Store, respectively. This makes the names
of the Assembler methods more uniform:
* Takes an address: Load, Store, LoadField, LoadCompressedField,
StoreIntoObject, StoreCompressedIntoObject, LoadSmi,
LoadCompressedSmi, etc.
* Takes a base register and an offset: LoadFromOffset, StoreToOffset,
LoadFieldFromOffset, LoadCompressedFieldFromOffset,
StoreIntoObjectOffset, StoreCompressedIntoObjectOffset,
LoadSmiFromOffset, LoadCompressedSmiFromOffset, etc.
Create AssemblerBase methods for loading and storing compressed
pointers that weren't already there, as well as the corresponding
methods for loading and storing uncompressed values.
Make non-virtual methods that load and store uncompressed fields
that call the corresponding method for loading from and storing to
memory regions, adjusting the address or offset accordingly. This
avoids needing per-architecture overrides for these.
Make non-virtual methods that load compressed fields, calling the
corresponding method for loading a compressed value from a memory
region. (Since compressed pointers are only stored in Dart objects,
and stores into a Dart object may require a barrier, there is no
method for storing a compressed value into an arbitrary memory region.)
Create pure virtual methods for loading from or storing to an Address
or any method that does not have both an Address-taking and a
base register and offset pair-taking version (e.g., LoadAcquire).
Create methods for loading from or storing to a base register
and an offset. The base implementation takes the base register and
offset and creates an Address from it, then calls the Address-taking
equivalent. These methods are non-virtual when the implementation is
the same on all architectures and virtual to allow overriding when
necessary.
Make a non-virtual method for loading uncompressed Smis, since all
architectures have the same code for this, including the DEBUG check.
If compressed pointers are not being used, all the methods for
compressed pointers are non-virtual methods that call the
corresponding method for uncompressed values.
If compressed pointers are being used:
* Install pure virtual methods for loading compressed values from
and storing compressed values to an Address or any method that does
not have both an Address-taking and a base register and offset
pair-taking version (e.g., LoadAcquireCompressed).
* Install virtual methods for loading compressed values from and
storing compressed values to a base register and offset. Like the
uncompressed implementation, the base implementation of these
create an Address and call the Address-taking equivalent, and these
implementations are overridden on ARM64.
* Install a non-virtual method for loading compressed Smis, since the
only difference is that it loads a zero-extended 32-bit value, which
AssemblerBase can do.
TEST=ci (refactoring only)
Change-Id: I934791d26a6e2cdaa6ac5f188b0fd89dbdc491d1
Cq-Include-Trybots: luci.dart.try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-arm64-try,vm-win-debug-x64-try,vm-win-release-ia32-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/359861
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Previously, CanBeImmediateIndex calculated the scaling factor for the
index using Instance::ElementSizeFor. However, the LoadIndexed and
StoreIndexed instructions which use this function have an index_scale_
field that is not necessarily the same as the element size for the
class id. This means the displacement calculated within
CanBeImmediateIndex can differ from the actual displacement calculated
within ElementAddressForIntIndex, causing the bug seen by the fuzzer.
This CL moves the check in CanBeImmediateIndex to a new static
method Assembler::AddressCanHoldConstantIndex. In addition to the
original arguments to CanBeImmediateIndex, the new static method takes
an index_scale argument, so the field value can be passed appropriately.
It also adds an is_external argument on X64 and IA32 to match the other
architectures, since assuming a non-external typed data object could
cause a register to be used in cases where it isn't necessary.
TEST=vm/dart/regress_54486
Fixes: https://github.com/dart-lang/sdk/issues/54486
Change-Id: Ia553fb7da0500113b35f8d9af91a52df55437a3c
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-x64-try,vm-linux-debug-ia32-try,vm-aot-linux-debug-simarm_x64-try,vm-aot-linux-debug-simriscv64-try,vm-aot-mac-release-arm64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/345002
Reviewed-by: Daco Harkes <dacoharkes@google.com>
When setRange is called on a TypedData receiver and the source is also
a TypedData object with the same element size and clamping is not
required, the VM implementation now calls _boundsCheckAndMemcpyN for
element size N. The generated IL for these methods performs the copy
using the MemoryCopy instruction (mostly, see the note below).
Since the two TypedData objects might have the same underlying
buffer, the CL adds a can_overlap flag to the MemoryCopy instruction
which checks for overlapping regions. If can_overlap is set, then
the copy is performed backwards instead of forwards when needed
to ensure that elements of the source region are read before
they are overwritten.
The existing uses of the MemoryCopy instruction are adjusted as
follows:
* The IL generated for copyRangeFromUint8ListToOneByteString
passes false for can_overlap, as all uses currently ensure that
the OneByteString is non-external and thus cannot overlap.
* The IL generated for _memCopy, used by the FFI library, passes
true for can_overlap, as there is no guarantee that the regions
pointed at by the Pointer objects do not overlap.
The MemoryCopy instruction has also been adjusted so that all numeric
inputs (the two start offsets and the length) are either boxed or
unboxed instead of just the length. This exposed an issue
in the inliner, where unboxed constants in the callee graph were
replaced with boxed constants when inlining into the caller graph,
since withList calls setRange with constant starting offsets of 0.
Now the representation of constants in the callee graph are preserved
when inlining the callee graph into the caller graph.
Fixes https://github.com/dart-lang/sdk/issues/51237 by using TMP
and TMP2 for the LDP/STP calls in the 16-byte element size case, so no
temporaries need to be allocated for the instruction.
On ARM when not unrolling the memory copy loop, uses TMP and a single
additional temporary for LDM/STM calls in the 8-byte and 16-byte
element cases, with the latter just using two LDM/STM calls within
the loop, a different approach than the one described in
https://github.com/dart-lang/sdk/issues/51229 .
Note: Once the number of elements being copied reaches a certain
threshold (1048576 on X86, 256 otherwise), _boundsCheckAndMemcpyN
instead calls _nativeSetRange, which is a native call that uses memmove
from the standard C library for non-clamped inputs. It does this
because the code currently emitted for MemoryCopy performs poorly
compared to the more optimized memmove implementation when copying
larger regions of memory.
Notable benchmark changes for dart-aot:
* X64
* TypedDataDuplicate.*.fromList improvement from ~13%-~250%
* Uf8Encode.*.10 improvement from ~50%-~75%
* MapCopy.Map.*.of.Map.* improvement from ~13%-~65%
* MemoryCopy.*.setRange.* improvement from ~13%-~500%
* ARM7
* Uf8Encode.*.10 improvement from ~35%-~70%
* MapCopy.Map.*.of.Map.* improvement from ~6%-~75%
* MemoryCopy.*.setRange.{8,64} improvement from ~22%-~500%
* Improvement of ~100%-~200% for MemoryCopy.512.setRange.*.Double
* Regression of ~40% for MemoryCopy.512.setRange.*.Uint8
* Regression of ~85% for MemoryCopy.4096.setRange.*.Uint8
* ARM8
* Uf8Encode.*.10 improvement from ~35%-~70%
* MapCopy.Map.*.of.Map.* improvement from ~7%-~75%
* MemoryCopy.*.setRange.{8,64} improvement from ~22%-~500%
* Improvement of ~75%-~160% for MemoryCopy.512.setRange.*.Double
* Regression of ~40% for MemoryCopy.512.setRange.*.Uint8
* Regression of ~85% for MemoryCopy.4096.setRange.*.Uint8
TEST=vm/cc/IRTest_Memory, co19{,_2}/LibTest/typed_data,
lib{,_2}/typed_data, corelib{,_2}/list_test
Issue: https://github.com/dart-lang/sdk/issues/42072
Issue: b/294114694
Issue: b/259315681
Change-Id: Ic75521c5fe10b952b5b9ce5f2020c7e3f03672a9
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-simarm_x64-try,vm-aot-linux-debug-simriscv64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-kernel-linux-debug-x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-linux-debug-ia32-try,vm-linux-debug-simriscv64-try,vm-linux-debug-x64-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-aot-linux-release-simarm64-try,vm-aot-linux-release-simarm_x64-try,vm-aot-linux-release-x64-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-ffi-qemu-linux-release-riscv64-try,vm-ffi-qemu-linux-release-arm-try,vm-aot-msan-linux-release-x64-try,vm-msan-linux-release-x64-try,vm-aot-tsan-linux-release-x64-try,vm-tsan-linux-release-x64-try,vm-linux-release-ia32-try,vm-linux-release-simarm-try,vm-linux-release-simarm64-try,vm-linux-release-x64-try,vm-mac-release-arm64-try,vm-mac-release-x64-try,vm-kernel-precomp-linux-release-x64-try,vm-aot-android-release-arm64c-try,vm-ffi-android-debug-arm64c-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/319521
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
The VM has two parts related to TSAN: compiler & runtime.
So far both parts used self-detection whether running with TSAN. This
implies that if AOT runtime runs under TSAN that the AOT compiler then
also has to run under TSAN (which is slow).
We separate the two by making the compiler use
`TARGET_USES_THREAD_SANITIZER`. It will be auto-detected by-default
except when `DART_PRECOPMILER` is set. In the precompiler case we
require the build rules to pass in whether the target runs under
TSAN or not.
Issue b/287638965
TEST=ci
Change-Id: Ib333a8a69ed2d7bdca7596014a0d25b103af25b6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/312480
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
After this change we no longer push outgoing arguments for normal
Dart calls, but instead simply move them into an area reserved for
outgoing arguments at the top of the frame. This allows us to stop
mutating SP and avoid explicit Drop(argc) after each call.
There are still other places in the generated code where we
push something onto the stack either to free a temporary
register or spill a value across a slow-path call.
This is the first step towards switching to native SP on ARM64.
# Code Size Impact
I have tested this on a large internal application and got the
following size delta
## ARM
| | Before | After | Delta | |
|--------|----------|----------|---------|-------|
| ROData | 6366632 | 6463680 | +97048 | +1.5% |
| Code | 22714192 | 22043984 | -670208 | -2.9% |
| Total | 33723701 | 33150477 | -573224 | -1.6% |
## ARM64
| | Before | After | Delta | |
|--------|----------|----------|---------|-------|
| ROData | 7122032 | 7214288 | +92256 | +1.3% |
| Code | 21607196 | 20741788 | -865408 | -4.0% |
| Total | 33375769 | 32602544 | -773225 | -2.3% |
TEST=ci
Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-nnbd-linux-release-x64-try,vm-kernel-precomp-linux-release-simarm-try,vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-linux-debug-x64c-try,vm-kernel-precomp-linux-product-x64-try,vm-precomp-ffi-qemu-linux-release-riscv64-try,vm-precomp-ffi-qemu-linux-release-arm-try,vm-kernel-precomp-nnbd-linux-release-simarm64-try,vm-kernel-nnbd-linux-release-simarm64-try
Bug: https://github.com/dart-lang/sdk/issues/39083
Change-Id: Icace0c986285b9c16e41786cded44f88277bd25f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/282029
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
AsmIntrinsifier::OneByteString_getHashCode now uses the CombineHashes
and FinalizeHash Assembler methods introduced in 49f998dc31.
To avoid having to check for 0 again after masking the result of
FinalizeHash to fit the desired bit size, we create a more general
FinalizeHashForSize that also takes the desired bit size of the result.
The intrinsified versions for ARM/ARM64/RISCV are also modified to
remove the special casing for the empty string, and instead just check
the size of the string against the loop counter at the start of the
loop, similarly to X64/IA32. This is less efficient for the empty
string case, but avoids the need to create a version of FinalizeHash
that elides the zero check.
TEST=ci
Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-dwarf-linux-product-x64-try,vm-kernel-precomp-linux-product-x64-try,vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-nnbd-mac-release-arm64-try,vm-kernel-precomp-nnbd-linux-release-simarm_x64-try,vm-kernel-precomp-linux-release-simarm-try,vm-kernel-precomp-nnbd-linux-release-x64-try,vm-kernel-precomp-nnbd-linux-release-simarm64-try,vm-kernel-precomp-nnbd-linux-debug-simriscv64-try,vm-kernel-precomp-linux-debug-x64c-try,vm-kernel-nnbd-linux-debug-simriscv64-try,vm-kernel-linux-release-simarm-try,vm-kernel-linux-release-simarm64-try,vm-kernel-linux-release-ia32-try,vm-kernel-nnbd-linux-release-simarm-try,vm-kernel-nnbd-linux-release-simarm64-try,vm-kernel-nnbd-linux-release-ia32-try,vm-kernel-nnbd-mac-release-arm64-try,vm-kernel-linux-debug-x64c-try
Change-Id: I53db731acb8bf35d16efb55fa2c5a79d8302399f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/272800
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Lower the threshold for converting from a linear to a hash-based cache
on most architectures from 500 to 10.
Due to register pressure, the InstantiateTypeArguments stub on IA32
continues to go to the runtime for hash caches, and so we do not
lower the threshold for converting to a hash-based cache there.
The following are benchmark results for those benchmark that use enough
Instantiations to trigger the use of hash-based caches. In the following
tables, "Results 1" denotes the benchmark results from only this change,
whereas "Results 2" include from comparing to the results prior to
4f925105cf, when only linear caches were used.
Dart AOT:
* InstantiateTypeArguments.Instantiate100
Arch | CPU | Results 1 | Results 2
-------|----------------|-----------------------
ARM | Odroid-C2 | 382.8% | 381.5%
ARM | Raspberry Pi 4 | 486.7% | 449.2%
ARM64 | Odroid-C2 | 328.1% | 372.8%
ARM64 | Raspberry Pi 4 | 1283% | 1281%
ARM64C | Raspberry Pi 4 | 2353% | 2811%
X64 | Intel Xeon | 568.7% | 584.9%
* InstantiateTypeArguments.Instantiate1000
Arch | CPU | Results 1 | Results 2
-------|----------------|------------------------
ARM | Odroid-C2 | 743.7% | 3821%
ARM | Raspberry Pi 4 | 486.7% | 3218%
ARM64 | Odroid-C2 | 584.7% | 3222%
ARM64 | Raspberry Pi 4 | 430.7% | 8172%
ARM64C | Raspberry Pi 4 | 491.4% | 16699%
X64 | Intel Xeon | 954.1% | 5528%
Dart JIT:
* InstantiateTypeArguments.Instantiate100
Arch | CPU | Results 1 | Results 2
-------|----------------|-----------------------
ARM | Raspberry Pi 4 | 315.7% | 295.1%
ARM64 | Raspberry Pi 4 | 1070% | 1058%
ARM64C | Raspberry Pi 4 | 1769% | 2095%
X64 | Intel Xeon | 507.4% | 496.2%
* InstantiateTypeArguments.Instantiate1000
Arch | CPU | Results 1 | Results 2
-------|----------------|-----------------------
ARM | Raspberry Pi 4 | 565.2% | 2550%
ARM64 | Raspberry Pi 4 | 406.8% | 7375%
ARM64C | Raspberry Pi 4 | 379.6% | 12996%
X64 | Intel Xeon | 807.9% | 4459%
During work on this change, an issue was found where cache lookups
in the stub on ARM64C always failed and went to runtime, even with
the old linear-only caches, hence the much larger performance gains
in those rows above.
TEST=vm/cc/TypeArguments_Cache_{Some,Many}Instantiations
Fixes: https://github.com/dart-lang/sdk/issues/48344
Change-Id: I3d29566ba0582502954c9fc59626ceb8fd40317a
Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-dwarf-linux-product-x64-try,vm-kernel-precomp-linux-product-x64-try,vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-nnbd-mac-release-arm64-try,vm-kernel-precomp-nnbd-linux-release-simarm_x64-try,vm-kernel-precomp-linux-release-simarm-try,vm-kernel-precomp-nnbd-linux-release-x64-try,vm-kernel-precomp-nnbd-linux-release-simarm64-try,vm-kernel-precomp-nnbd-linux-debug-simriscv64-try,vm-kernel-precomp-tsan-linux-release-x64-try,vm-kernel-tsan-linux-release-x64-try,vm-kernel-precomp-linux-debug-x64c-try,vm-kernel-nnbd-linux-debug-simriscv64-try,vm-kernel-linux-release-simarm-try,vm-kernel-linux-release-simarm64-try,vm-kernel-linux-release-ia32-try,vm-kernel-nnbd-linux-release-simarm-try,vm-kernel-nnbd-linux-release-simarm64-try,vm-kernel-nnbd-linux-release-ia32-try,vm-kernel-nnbd-mac-release-arm64-try,vm-kernel-linux-debug-x64c-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/270702
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Some large applications are bumping up against the 16-bit limit.
The object header changes from hash:32,cid:16,size:8,gc:8 to hash:32,cid:20,size:4,gc:8.
Because the CID field is no longer a power of two, it takes two instructions to load instead of one, increasing code size. The alternative change to cid:32,hash:24,gc:8 was considered and rejected because the reduction in hash bits very noticeably affects performance in applications with large data sets.
TEST=ci
Bug: b/255934984
Change-Id: I5fa36ba48a6852359d994393cf80c6a761c5d84c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/269120
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Enable concurrent marking for IA32.
This removes the last write barrier relying on different alignment offsets for old and new space.
TEST=ci
Change-Id: Ib1c13124002392cf1c3ec264643325ec471a6918
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/262280
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Moves `RangeCheck` from the intrinsifier to the assembler.
Also, unifies the signature and semantics and adds documentation.
TEST=refactor
Change-Id: I274e5e46fde6a6e8baf30d8b732932f4f46d5855
Cq-Include-Trybots: luci.dart.try:vm-kernel-linux-debug-simriscv64-try,vm-kernel-precomp-linux-debug-simriscv64-try,vm-kernel-linux-release-simarm64-try,vm-kernel-linux-release-simarm-try,vm-kernel-precomp-linux-release-simarm64-try,vm-kernel-precomp-linux-release-simarm-try,vm-precomp-ffi-qemu-linux-release-riscv64-try,vm-precomp-ffi-qemu-linux-release-arm-try,vm-kernel-linux-debug-ia32-try,vm-kernel-linux-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/261681
Reviewed-by: Tess Strickland <sstrickl@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
* Merge ClassTable and SharedClassTable back together;
* Simplify handling of multiple arrays growing in sync;
* Refactor how reload deals with ClassTable.
The last change is the most important because it makes it
much easier to reason about the code. We move away from
copying bits and pieces of the class table and shared
class table into reload contexts.
Having two class table fields in the isolate group makes
it easier to reason about. One field contains program
class table (one modified by kernel loader and accessed
by various program structure cid lookups) and heap
walk class table (used by GC visitors). Normally these
two fields point to the same class table, but during
hot reload we temporary split them apart: original
class table is kept as a heap walk class table, while
program class table is replaced by a clone and updated
by reload.
If reload succeeds we drop original class table and
set program class table as heap walk one.
If reload fails we drop the program class table and
restore original one from heap walk table.
TEST=ci
Cq-Include-Trybots: luci.dart.try:vm-kernel-reload-linux-release-x64-try,vm-kernel-reload-linux-debug-x64-try,vm-kernel-reload-rollback-linux-debug-x64-try,vm-kernel-reload-rollback-linux-release-x64-try,vm-kernel-linux-debug-x64-try,vm-kernel-precomp-tsan-linux-release-x64-try,vm-kernel-tsan-linux-release-x64-try,vm-kernel-precomp-asan-linux-release-x64-try,vm-kernel-asan-linux-release-x64-try
Change-Id: I8b66259fcc474dea7dd2af063e4772df99be06c4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/258361
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
The CCallInstr takes unboxed word-sized integers and does a call to C
without any VM transitions.
This can be used for the following.
1) Leaf runtime entries. This CL removes the EnterHandleScope,
ExitHandleScope, and AllocateHandle instructions.
2) Compilation of isolate independent code where all definitions in IL
are unboxed integers.
Currently, the CCallInstr expects all it's arguments and the return type
to be word-sized.
This CL also:
- Adds loadImmediate and CCall with a register for each assembler.
- Adds a CSE pass to force-optimized so that LoadThreadInstr gets de-
duplicated.
TEST=tests/ffi(_2)/*
Change-Id: I67c6aaa1b7d6aa7d60e274477686a54e5a331216
Cq-Include-Trybots: luci.dart.try:vm-kernel-linux-debug-simriscv64-try,vm-kernel-precomp-linux-debug-simriscv64-try,vm-kernel-nnbd-mac-debug-arm64-try,vm-kernel-nnbd-mac-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-kernel-win-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/246240
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
(Macro-)assembler helper PushRegistersInOrder generates series of
register pushes in an architecture-specific way:
* Using STMDB on ARM (if possible).
* Using STP on ARM64.
* Updating SP only once on RISC-V.
TEST=ci
Change-Id: I6cbd1dd5f8acf8ce8087ed899a311264a873ad37
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/243100
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
The new implementation moves away from desugaring of async
functions on kernel AST, state machine generated in the flow graph and
capturing all local variables in the context.
Instead, async/await is implemented using a few stubs
(InitSuspendableFunction, Suspend, Resume, Return and
AsyncExceptionHandler). The stubs are implemented in a
platform-independent way using (macro-)assembler helpers.
When suspending a function, its frame is copied into a SuspendState
object, and when resuming a function it is copied back onto the stack.
No extra code is generated for accessing local variables.
Callback closures are created lazily on the first await.
Design doc: go/compact-async-await.
Part 1 (kernel): https://dart-review.googlesource.com/c/sdk/+/241842
TEST=ci
Issue: https://github.com/dart-lang/sdk/issues/48378
Change-Id: Ibad757035b7cc438ebdff80b460728b1d3eff1f5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/242000
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
Adds an additional `contains_call` mode specifying that the
register allocator should block all non-callee-save ABI registers.
As opposed to `kCall`, this frees the use of callee-save registers
across FfiCallInstr, meaning that live values do not always have
to be spilled.
For architectures with sufficient free callee-save registers
this should result in less overhead for each FfiCall.
This change also removes RequiresStackSlot as it was only being
used for FfiCall's TypedData, and wasn't implemented in the
Register Allocator.
Since FfiCall no longer blocks all registers TypedData can now
live in a register, so the previous assumptions about it always
being on stack have to be relaxed.
This change also works around an issue where the register allocator
might allocate an FPU register that does not have an associated S
register. This causes problems when FFI tries to pass 32 bit floats
in said register.
As a work around we block all FPU regs. above the 8 that have S regs.
for FFI calls in VFPv3-D32 mode.
This change also adds an additional temp registers which was previous
used but not being reserved in FfiCall IL on Arm64.
Some additional prereq. fixes were branched out into:
https://dart-review.googlesource.com/c/sdk/+/237690
TEST=Adds IRTest_FfiCallInstrLeafDoesntSpill.
Bug: https://github.com/dart-lang/sdk/issues/45468
Change-Id: Icdeccb0de77e46f5bc34dd5bd4e308d27bc1ef99
Cq-Do-Not-Cancel-Tryjobs: true
Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-win-release-x64-try, vm-precomp-ffi-qemu-linux-release-arm-try, vm-kernel-mac-release-arm64-try, vm-kernel-nnbd-mac-debug-arm64-try, vm-kernel-nnbd-mac-release-arm64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/221624
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Clement Skau <cskau@google.com>
- Fixes SIMD registers clobbered by write barrier on ARM64.
- Moves code generation out of non-compiler directory.
- Removes unnecessary building of Dart frames on leaf runtime calls.
- Removes unnecessary Threads slots for write barrier Code objects.
- Removes duplicate saves of SP in leaf runtime calls on ARM64 and RISC-V.
- Avoids some redundant SP updates on RISC-V.
TEST=ci
Change-Id: Idb92127658edc90b320923ef3d882a7219a450ae
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/236842
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
When ffi code calls Dart code that requests the isolate to exit(via Isolate.exit, for example), we ensure that isolate indeed exits on return from ffi call.
This is implemented by introduction of new safepoint bit which, once set, forces ExitSafepoint to propagate unwind error.
TEST=isolate_exit_sandwich_test
Change-Id: I2e8f5ecec7f4e59ae5f99b9525cc566f20d4b6a8
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/219846
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Alexander Aprelev <aam@google.com>
In dependent CL, const maps and sets can be shared between isolates in
the same isolate group before the index is lazily populated. To prevent
one isolate from seeing a non-null index pointer, while not all writes
to the index itself are visible to the isolate (the mutator thread of
that isolate), we use a store release barrier on arm architectures.
This CL adds a store release barrier option to the StoreInstanceField
instruction to be used in the dependent CL.
Even though there is no barrier on x64 and ia32, we still also use the
option on these architectures because on x64 we signal the barrier to
TSAN.
The stlr instruction on arm64 does not feature an address operand.
Therefore we need to branch in the assembler on whether we're using
a store-release before we construct an Address.
Design doc: go/dart-vm-const-maps
TEST=Dependent CL, split off for ease of reviewing.
Bug: https://github.com/dart-lang/sdk/issues/45908
Change-Id: I6563aa3c813319a23e815b862ca5caabb805dc9e
Cq-Include-Trybots: luci.dart.try:analyzer-nnbd-linux-release-try,app-kernel-linux-debug-x64-try,dart-sdk-linux-try,front-end-nnbd-linux-release-x64-try,pkg-linux-debug-try,vm-canary-linux-debug-try,vm-kernel-asan-linux-release-x64-try,vm-kernel-checked-linux-release-x64-try,vm-kernel-linux-debug-x64c-try,vm-kernel-linux-debug-x64-try,vm-kernel-linux-debug-simarm64c-try,vm-kernel-nnbd-linux-release-simarm-try,vm-kernel-optcounter-threshold-linux-release-x64-try,vm-kernel-precomp-android-release-arm_x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-linux-debug-x64c-try,vm-kernel-reload-linux-debug-x64-try,vm-kernel-reload-rollback-linux-debug-x64-try,vm-precomp-ffi-qemu-linux-release-arm-try,vm-kernel-precomp-linux-debug-simarm_x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-tsan-linux-release-x64-try,vm-kernel-tsan-linux-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/210726
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Tess Strickland <sstrickl@google.com>
Vyacheslav Egorov