The previous attempt was reverted because it broke unoptimized JIT
on ARM 32. This reland force-optimizes the two getters.
Stacks on top of the int.{trailingZeroBitCount,oneBitCount} API CL
(commit 754239b077). Both getters route through OTHER_RECOGNIZED_LIST
when a hardware fast path is available; otherwise the newly added
Dart bodies inline at call sites via vm:prefer-inline. The C++
natives are removed.
Backend codegen
---------------
ARM64: NEON CNT + UADDLV (popcount); RBIT + CLZ (ctz).
ARM: NEON CNT + VPADDL chain (popcount); RBIT + CLZ on the
register pair (ctz).
x64: popcntq when TargetCPUFeatures::popcnt_supported();
LoadImmediate(64) + rep_bsfq for ctz (decodes as tzcnt
on BMI1+, preserves dest on zero otherwise).
RISC-V 64: cpop / ctz when RV_baseline includes Zbb.
Per-arch availability is encapsulated in
UnaryInt64OpInstr::IsSupported(Token::Kind).
Apple M-series ARM64, AOT (us/iter, lower is better):
cardinality.swar 371
cardinality.accelerated 154 (2.4x)
forEachSetBit.swar 19031
forEachSetBit.accelerated 4988 (3.8x)
select.swar 199
select.accelerated 77 (2.6x)
complementCardinality.swar 399
complementCardinality.accel 152 (2.6x)
Work towards https://github.com/dart-lang/sdk/issues/6486 (popcount
and ctz intrinsification).
Work towards https://github.com/dart-lang/sdk/issues/1053 (efficient
BitSet implementation).
Fixes https://github.com/dart-lang/sdk/issues/52673
Fixes https://github.com/dart-lang/sdk/issues/38346
Fixes https://github.com/dart-lang/sdk/issues/63436
Issue https://github.com/dart-lang/sdk/issues/10212
Issue https://github.com/dart-lang/sdk/issues/5798
TEST=tests/corelib/int_bit_count_test
Cq-Include-Trybots: luci.dart.try:vm-linux-release-simarm-try,vm-ffi-qemu-linux-release-arm-try,vm-aot-linux-release-simarm_x64-try,vm-aot-linux-debug-simarm_x64-try,dart-sdk-linux-riscv64-try
Change-Id: Ib812cbaec6e371b9720df7a543411f78e524cac1
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/506060
Reviewed-by: Martin Kustermann <kustermann@google.com>
Auto-Submit: Modestas Valauskas <valauskasmodestas@gmail.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
Stacks on top of the int.{trailingZeroBitCount,oneBitCount} API CL
(commit 754239b077). Both getters route through OTHER_RECOGNIZED_LIST
when a hardware fast path is available; otherwise the newly added
Dart bodies inline at call sites via vm:prefer-inline. The C++
natives are removed.
Backend codegen
---------------
ARM64: NEON CNT + UADDLV (popcount); RBIT + CLZ (ctz).
ARM: NEON CNT + VPADDL chain (popcount); RBIT + CLZ on the
register pair (ctz).
x64: popcntq when TargetCPUFeatures::popcnt_supported();
LoadImmediate(64) + rep_bsfq for ctz (decodes as tzcnt
on BMI1+, preserves dest on zero otherwise).
RISC-V 64: cpop / ctz when RV_baseline includes Zbb.
Per-arch availability is encapsulated in
UnaryInt64OpInstr::IsSupported(Token::Kind).
Apple M-series ARM64, AOT (us/iter, lower is better):
cardinality.swar 371
cardinality.accelerated 154 (2.4x)
forEachSetBit.swar 19031
forEachSetBit.accelerated 4988 (3.8x)
select.swar 199
select.accelerated 77 (2.6x)
complementCardinality.swar 399
complementCardinality.accel 152 (2.6x)
Work towards https://github.com/dart-lang/sdk/issues/6486 (popcount
and ctz intrinsification).
Work towards https://github.com/dart-lang/sdk/issues/1053 (efficient
BitSet implementation).
Fixes https://github.com/dart-lang/sdk/issues/52673
Fixes https://github.com/dart-lang/sdk/issues/38346
Issue https://github.com/dart-lang/sdk/issues/10212
Issue https://github.com/dart-lang/sdk/issues/5798
TEST=tests/corelib/int_bit_count_test
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-release-simarm_x64-try,vm-aot-linux-debug-simarm_x64-try,dart-sdk-linux-riscv64-try
Change-Id: I9c2c4225fd63d54e190398a0fb6649366317dd17
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/504221
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Auto-Submit: Modestas Valauskas <valauskasmodestas@gmail.com>
Call __tsan_func_entry/__tsan_func_exit in functions that use __tsan_read/__tsan_write or call other functions. Call __tsan_func_exit once per frame when unwinding for exceptions. Do so only in AOT, since TSAN won't be able to symbolize JIT functions anyway.
TEST=ci
Bug: https://github.com/dart-lang/sdk/issues/61352
Cq-Include-Trybots: luci.dart.try:vm-tsan-linux-release-arm64-try,vm-tsan-linux-release-x64-try
Change-Id: Ie52c978c25664d78b834e9b72ecf7eb2a12cc2ba
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/444181
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
The #ifdef block that would stop ARM cross-compilation on Windows
has been outdated for a few years now, as getting the target CPU features via TargetCPUFeatures is now handled appropriately on Windows.
TEST=ci, actual test depends on upload so will be in a followup CL
Issue: https://github.com/dart-lang/sdk/issues/28617
Change-Id: I9907145ec8fb97007769276cb3c6c2442ac9e17c
Cq-Include-Trybots: luci.dart.try:pkg-linux-release-arm64-try,pkg-linux-release-try,pkg-mac-release-arm64-try,pkg-mac-release-try,pkg-win-release-arm64-try,pkg-win-release-try,dart-sdk-linux-riscv64-try,dart-sdk-linux-arm64-try,dart-sdk-linux-try,dart-sdk-mac-try,dart-sdk-win-arm64-try,dart-sdk-win-try,dart-sdk-mac-arm64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/440224
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Ivan Inozemtsev <iinozemtsev@google.com>
Instead of handling FFI related execution state and safepoint
in assembly handle it in runtime code.
The transition needs to be done before JumpToFrame unwinds
stack because unwinding destroys exit frame and this can't
be done at safepoint as GC might be traversing the stack.
An incorrect order of operation was manifesting as crashes in
GC when one isolate in a group was encountering a lot of
exceptions thrown from an FFI call and another isolate is
triggering GCs.
To catch this in the future added a bit of validation to
ExitSafepoint runtime call which triggers when --use-slow-path
is enabled. Though after refactoring this code does not
trigger this code path anymore because it was completely
removed - but it is better than nothing.
This CL also removes a lot of unnecessary complexity which
was associated with handling this transition in the stub
itself.
TEST=ffi/vmspecific_handle_test
Bug: b/408377905
Cq-Include-Trybots: dart/try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-asan-linux-release-x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-msan-linux-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-tsan-linux-release-x64-try,vm-aot-ubsan-linux-release-x64-try,vm-aot-win-debug-x64-try,vm-aot-win-debug-x64c-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-arm64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-mac-debug-simarm64_arm64-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-fuchsia-release-x64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-arm64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-tsan-linux-release-arm64-try,vm-tsan-linux-release-x64-try,vm-ubsan-linux-release-arm64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-x64-try,vm-win-debug-x64c-try,vm-win-release-ia32-try
Change-Id: Ia073cb6bb9e1b5a0ea8514c7e048cee6019b84d6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/420324
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
The transitions in C++ have asserts which run in debug mode to validate
that the transitions are happening in the right order.
The transitions in the generated code had no equivalent. This CL
introduces the equivalent for the generated code.
The transitions to generated code are misnamed currently. They come
from either VM or Native, not only Native. For example the FFI
callbacks and call return sequence can already transition from native
into the VM when entering the safepoint.
These checks don't catch the invalid FFI callbacks.
Bug: https://github.com/dart-lang/sdk/issues/60021
These are caught when looking up the FFI metadata:
https://dart-review.googlesource.com/c/sdk/+/407023
This CL omits to implement the check in ia32, since ia32 is scheduled
for removal.
The check is only enabled in debug mode. The asserts for the C++ are
also only run in debug mode. And enabling this check in release for
AOT regresses the parent CL benchmark by ~30% on Mac Arm64.
TEST=test/ffi
Change-Id: Ieaf1e43533baae294af37b2e171b68bd314fdbe0
Cq-Include-Trybots: dart/try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-asan-linux-release-x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-msan-linux-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-tsan-linux-release-x64-try,vm-aot-ubsan-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-aot-win-debug-x64-try,vm-aot-win-debug-x64c-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-arm64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-fuchsia-release-arm64-try,vm-fuchsia-release-x64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-arm64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-tsan-linux-release-arm64-try,vm-tsan-linux-release-x64-try,vm-ubsan-linux-release-arm64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-arm64-try,vm-win-debug-x64-try,vm-win-debug-x64c-try,vm-win-release-ia32-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/407021
Reviewed-by: Liam Appelbe <liama@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
On architectures with a weak memory model, there is the possibility that the concurrent marker can see the publishing store of a new object before it sees the initializing store of that object's header. On an M1, the barrier to prevent reordering these stores is fairly cheap, so we emit this barrier on Mac/iOS ARM64. Otherwise, this barrier is very expensive (or at least expensive for some hardware within the ABI), so instead we avoid the race by deferring marking of objects inside an active TLAB.
Disable TSAN instrumentation on the marker setting the mark bit, as TSAN does not understand fences.
TEST=ooo arm64 machines
Bug: https://github.com/dart-lang/sdk/issues/56845
Change-Id: I0676661a7cf941fdc6b451e516d890c26826bb3b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/389265
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
For non-x86 architectures, these forms directly translate to a
single instruction when possible. On x86 architectures,
XImmediate(dst, src, imm) where X is in {Lsl, ArithmeticShiftRight, And}
can be translated to:
MoveRegister(dst, src);
XImmediate(dst, imm);
in the general case since MoveRegister(dst, src) is a no-op if dst and
src are the same.
Also add versions of these assembler macros that take an OperandSize
and handle 32-bit OperandSizes appropriately on 64-bit architectures.
Ensure the implementation for LslImmediate and
ArithmeticShiftRightImmediate emits no instructions if shift == 0,
dst == src, and OperandSizeInBits(sz) == kBitsPerWord on all
architectures.
-----
Other changes:
Fix ConstantExpression::EmitMoveToLocation on ARM64 to match
other architectures, which allow any word-sized or less unboxed
integer representation.
Fill out ExtendValue on RISCV for previously unimplemented
OperandSizes, using the Zba and Zbb extensions when possible.
-----
TEST=vm/cc/Assembler_AndImmediate vm/cc/Assembler_LslImmediate
vm/cc/Assembler_ArithmeticShiftRightImmediate
Separated out of https://dart-review.googlesource.com/c/sdk/+/378706
for easier debugging/reviewing.
Change-Id: I721f1334784f7011a958ea8af29f0e56c620726c
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-release-simarm_x64-try,vm-aot-linux-debug-x64-try,vm-aot-mac-release-arm64-try,vm-aot-mac-product-arm64-try,vm-aot-linux-product-x64-try,vm-aot-dwarf-linux-product-x64-try,vm-aot-linux-debug-simarm_x64-try,vm-linux-debug-x64-try,vm-mac-debug-arm64-try,vm-mac-release-arm64-try,vm-linux-debug-ia32-try,vm-aot-android-release-arm64c-try,vm-ffi-android-debug-arm64c-try,vm-aot-linux-debug-x64c-try,vm-linux-debug-x64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/386180
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Decorating a field with 'vm:shared' pragma makes values in this field accessible to all isolates in an isolate group.
Introduce `channel` to the `Version` class so that the pragma can only be enabled on main and dev channels.
TEST=shared_test, shared_fail_without_flag_test
BUG=https://github.com/dart-lang/sdk/issues/55991
Change-Id: I843c9f0d2ffc9f2ced7ddc4006bb6f9ca4e2ddf4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/370064
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Alexander Aprelev <aam@google.com>
Add an additional base argument to Assembler::AddScaled so it now
computes:
dest <- base + (index << scale) + offset
If base is kNoRegister (or ZR when available), then it emits
instructions optimized for computing:
dest <- (index << scale) + offset
(i.e., its previous implementation)
Add AddScaled to AssemblerBase to ensure the same interface across all
architectures.
Rework the backend of CalculateElementAddress to use AddScaled
appropriately, which unifies it across architectures.
TEST=ci (refactoring)
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-simarm_x64-try,vm-aot-linux-debug-simriscv64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-linux-debug-ia32-try,vm-linux-debug-x64-try,vm-linux-debug-x64c-try,vm-linux-debug-simriscv64-try,vm-mac-debug-arm64-try,vm-aot-mac-release-arm64-try
Change-Id: I33c8f99604b68360f10b79050bd66ceb9d65ac9b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/370504
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
The ASSERT checks that there will be no overflow in computing the byte
offset from the index. However, any instruction that uses this method is
preceded by a bounds check of the index, and an index that would cause
the byte offset calculation to overflow is guaranteed to fail the bounds
check. Thus, this is dead code that will never be executed at runtime
and it is okay to emit code to perform the erroneous offset calculation.
TEST=vm/dart/regress_55864
Fixes: https://github.com/dart-lang/sdk/issues/55864
Change-Id: If4126e412368b8307be3c787ccb6bd451764c73f
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-simarm_x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/368563
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Delay changing Thread::vm_tag on callback entry and restore the tag early on callback return so that the profiler doesn't see the "running Dart" tag unless it can also see the fake return address marking the entry frame.
TEST=ffi/async_void_function_callbacks, ffi/function_callbacks_subtype, ffi/function_callbacks, ffi/isolate_local_function_callbacks
Bug: https://github.com/dart-lang/sdk/issues/52814
Change-Id: I40d80ec7c44063d078db0e211565e2d127c6b81e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/367460
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
This reverts commit bc0f02e4c8.
Reason for revert: https://github.com/dart-lang/sdk/issues/55754
Original change's description:
> [vm, gc] Incremental compaction.
>
> At the beginning of a major GC cycle, select some mostly-empty pages to be evacuated. Mark the pages and the objects on these pages. Apply a write barrier for stores creating old -> evacuation candidate pointers, and discover any such pointers that already exist during marking.
>
> At the end of a major GC cycle, evacuate objects from these pages. Forward pointers of objects in the remembered set and new-space. Free the evacuated pages.
>
> This compaction is incremental in the sense that creating the remembered set is interleaved with mutator execution. The evacuation step, however, is stop-the-world.
>
> Write-barrier elimination for x.slot = x is removed. Write-barrier elimination for x.slot = constant is removed in the JIT, kept for AOT but snapshot pages are marked as never-evacuate.
>
> TEST=ci
> Bug: https://github.com/dart-lang/sdk/issues/52513
> Change-Id: Icbc29ef7cb662ef8759b8c1d7a63b7af60766281
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/357760
> Reviewed-by: Alexander Aprelev <aam@google.com>
> Commit-Queue: Ryan Macnak <rmacnak@google.com>
Bug: https://github.com/dart-lang/sdk/issues/52513
Change-Id: I565ad6c0fca283d33f605c10f181bc0a59e7d2b2
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/366965
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Alexander Aprelev <aam@google.com>
Auto-Submit: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
At the beginning of a major GC cycle, select some mostly-empty pages to be evacuated. Mark the pages and the objects on these pages. Apply a write barrier for stores creating old -> evacuation candidate pointers, and discover any such pointers that already exist during marking.
At the end of a major GC cycle, evacuate objects from these pages. Forward pointers of objects in the remembered set and new-space. Free the evacuated pages.
This compaction is incremental in the sense that creating the remembered set is interleaved with mutator execution. The evacuation step, however, is stop-the-world.
Write-barrier elimination for x.slot = x is removed. Write-barrier elimination for x.slot = constant is removed in the JIT, kept for AOT but snapshot pages are marked as never-evacuate.
TEST=ci
Bug: https://github.com/dart-lang/sdk/issues/52513
Change-Id: Icbc29ef7cb662ef8759b8c1d7a63b7af60766281
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/357760
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Add OperandSize arguments to the non-compressed Store methods so that
the compressed Store methods can be implemented as calls to the
corresponding non-compressed method using kObjectBytes. This makes it
so that all compressed Store methods are now non-virtual.
Add a scratch register argument to all Store methods that involve a
write barrier that defaults to TMP. Originally only the IA32
implementation took a scratch register, with the others using TMP or
TMP2 internally.
Since all architectures can represent a register/offset pair as an
Address without losing information, make all the offset-based load
and store methods have an implementation that creates an Address or
FieldAddress and calls the corresponding address-based store method.
This makes all the offset-based load and store methods non-virtual.
After this, most of the Store methods are non-virtual, with
only the base methods used to implement them as virtual:
* Store
* StoreRelease
* StoreObjectIntoObjectNoBarrier
* StoreBarrier
* ArrayStoreBarrier
* VerifyStoreNeedsNoWriteBarrier (for DEBUG use only)
The Load methods can't be consolidated quite so much, as the
base methods for handling non-Smi compressed pointers must be defined
per-architecture, as each compressed pointer architecture handles
adding the upper bits for the heap separately.
Have LoadFieldInstr::EmitNativeCode use Assembler::LoadToSlot when the
result location is a single integer register and
StoreFieldInstr::EmitNativeCode use Assembler::StoreToSlot and
Assembler::StoreToSlotNoBarrier when the value location is a single
integer register to avoid code duplication between those methods.
Handle both compressed and uncompressed Smi fields in LoadFromSlot.
TEST=ci (refactoring)
Cq-Include-Trybots: luci.dart.try:vm-aot-android-release-arm64c-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-x64-try,vm-win-release-ia32-try
Change-Id: I60cc03776af220ed87918664bb4b9abafff2788a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/360641
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Remove CompareWithCompressedFieldFromOffset, which has no uses.
Rename the LoadFromOffset and StoreFromOffset methods that took
Addresses to Load and Store, respectively. This makes the names
of the Assembler methods more uniform:
* Takes an address: Load, Store, LoadField, LoadCompressedField,
StoreIntoObject, StoreCompressedIntoObject, LoadSmi,
LoadCompressedSmi, etc.
* Takes a base register and an offset: LoadFromOffset, StoreToOffset,
LoadFieldFromOffset, LoadCompressedFieldFromOffset,
StoreIntoObjectOffset, StoreCompressedIntoObjectOffset,
LoadSmiFromOffset, LoadCompressedSmiFromOffset, etc.
Create AssemblerBase methods for loading and storing compressed
pointers that weren't already there, as well as the corresponding
methods for loading and storing uncompressed values.
Make non-virtual methods that load and store uncompressed fields
that call the corresponding method for loading from and storing to
memory regions, adjusting the address or offset accordingly. This
avoids needing per-architecture overrides for these.
Make non-virtual methods that load compressed fields, calling the
corresponding method for loading a compressed value from a memory
region. (Since compressed pointers are only stored in Dart objects,
and stores into a Dart object may require a barrier, there is no
method for storing a compressed value into an arbitrary memory region.)
Create pure virtual methods for loading from or storing to an Address
or any method that does not have both an Address-taking and a
base register and offset pair-taking version (e.g., LoadAcquire).
Create methods for loading from or storing to a base register
and an offset. The base implementation takes the base register and
offset and creates an Address from it, then calls the Address-taking
equivalent. These methods are non-virtual when the implementation is
the same on all architectures and virtual to allow overriding when
necessary.
Make a non-virtual method for loading uncompressed Smis, since all
architectures have the same code for this, including the DEBUG check.
If compressed pointers are not being used, all the methods for
compressed pointers are non-virtual methods that call the
corresponding method for uncompressed values.
If compressed pointers are being used:
* Install pure virtual methods for loading compressed values from
and storing compressed values to an Address or any method that does
not have both an Address-taking and a base register and offset
pair-taking version (e.g., LoadAcquireCompressed).
* Install virtual methods for loading compressed values from and
storing compressed values to a base register and offset. Like the
uncompressed implementation, the base implementation of these
create an Address and call the Address-taking equivalent, and these
implementations are overridden on ARM64.
* Install a non-virtual method for loading compressed Smis, since the
only difference is that it loads a zero-extended 32-bit value, which
AssemblerBase can do.
TEST=ci (refactoring only)
Change-Id: I934791d26a6e2cdaa6ac5f188b0fd89dbdc491d1
Cq-Include-Trybots: luci.dart.try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-arm64-try,vm-win-debug-x64-try,vm-win-release-ia32-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/359861
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
After https://dart-review.googlesource.com/c/sdk/+/330600, there were
more chances for the optimizing compiler to introduce or move
GC-triggering instructions like allocations or boxings between the
retrieval of an untagged pointer to GC-moveable memory and its use.
To limit the chance of this happening, this CL removes the explicit
loading of the untagged payload address when building the initial
flow graph in most cases when the array is not known to be an external
array (an external string, an external typed data object, or an FFI
Pointer).
The remaining case is during view allocation, which extracts the
payload address of the base typed data object underlying the view
(which may be GC-movable) to calculate the payload address that should
be stored in the data field of the view object. See
https://github.com/dart-lang/sdk/issues/54884.
During canonicalization of LoadIndexed, StoreIndexed, and MemoryCopy
instructions, if the cid of an array input is an external array
(external string, external typed data object, or Pointer), then a
LoadField instruction that extracts the untagged payload address
is inserted before the instruction and the corresponding input is
rebound to that LoadField instruction.
Once all compiler passes that involve code motion have been performed,
a new pass looks for LoadIndexed, StoreIndexed, or MemoryCopy where
the cid stored in the instruction for the array is a typed data cid.
In these cases, if the array is not an internal typed data object,
then the payload address is extracted. Waiting until this point ensures
that no GC-triggering instructions are inserted between the extraction
of the payload address and the use. (Internal typed data objects are
left as-is because the payload address is inside the object itself
and doesn't require indirection through the data field of the object).
This CL also replaces code conditional on the array cid with code
that is instead conditional on the array element representation in
cases where it makes sense to do so, since this is a less brittle
check than checking the array cid (e.g., checking for kUnboxedInt8
to load, store, or copy an signed byte from an array instead of
listing all possible array cids that store signed bytes).
This CL also fixes an issue with the ARM64 assembler where calling
LoadFromOffset with an Address that has a non-Offset type would
silently generate bad code instead of triggering the ASSERT in
PrepareLargeOffset.
TEST=vm/dart/typed_list_index_checkbound_il_test
Issue: https://github.com/dart-lang/sdk/issues/54710
Cq-Include-Trybots: luci.dart.try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-fuchsia-release-x64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-arm64-try,vm-win-debug-x64-try,vm-win-release-ia32-try
Change-Id: I25b5f314943e9254d3d28986d720a5d47f12feeb
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/352363
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Moves RepresentationOfArrayElement out of LoadIndexed and StoreIndexed
into RepresentationUtils. Adjusts RepresentationOfArrayElement
to work for all concrete typed data classes. Changes
RepresentationOfArrayElement to return the precise representation of
the array element, which requires adding two new representations,
kUnboxedInt8 and kUnboxedInt16.
Remove Address::OperandSizeFor and its only (ignored) use in the
RISCV assembler and change Address::OperandSizeFor in the ARM and
ARM64 assemblers to use RepresentationUtils::OperandSize and RepresentationUtils::RepresentationOfArrayElement appropriately.
Add Boxing::NativeRepresentation, which converts representations for
unboxed integers smaller than 4 bytes to kUnboxedIntPtr and otherwise
returns the original representation. This mimics the current semantics
for LoadIndexed and StoreIndexed where returned values represented by
unboxed integer values smaller than 4 bytes are zero or sign extended
to word size appropriately, and received values represented by unboxed
integer values smaller than 4 bytes are appropriately truncated from a
word sized signed integer.
Create LoadIndexedInstr::ReturnRepresentation and
StoreIndexedInstr::ValueRepresentation, which use
Boxing::NativeRepresentation to convert the precise representation
returned by RepresentationUtils::RepresentationOfArrayElement to
the actual returned/expected representation for those instructions.
Also use Boxing::NativeRepresentation to appropriately adjust
the representations used for SIMD operations returning int8_t.
Add checks that unboxing and integer conversion never get
representations which result in a different representation when
passed to Boxing::NativeRepresentation, as those instructions do
not currently work with unboxed integer representations smaller than
4 bytes. (Adding such support can be done in a future CL.)
Inside the optimizing compiler, use the precise representation of the
array element where appropriate. That is:
* Range analysis
* Type propagation
* Redundancy elimination
This CL also cleans up how Representations are printed to C, making
that part of RepresentationUtils. It also removes the representation
printing and parsing utils from Location since they were only used by
the original IL serializer and deserializer, which has been removed.
It also renames RepresentationUtils::IsUnsigned to IsUnsignedInteger,
to make it clear that it only returns true for unboxed integer
representations.
TEST=standalone/bytedata_test
Fixes: https://github.com/dart-lang/sdk/issues/54622
Cq-Include-Trybots: luci.dart.try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-aot-win-debug-x64-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-fuchsia-release-x64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-arm64-try,vm-win-debug-x64-try,vm-win-release-ia32-try
Change-Id: I8c0dab911463ee4d0c55cd59ffea9d01e0f0bb7c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/354885
Reviewed-by: Slava Egorov <vegorov@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
The refactoring in https://dart-review.googlesource.com/c/sdk/+/345002
erroneously assumed that if the caller to
Assembler::AddressCanHoldConstantIndex didn't pass an out parameter to
detect whether a base register was needed, that that case shouldn't be
considered.
While it's true that LoadIndexedInstr::MakeLocationSummary originally
ignored the value stored in the out parameter, it did that because it
can use TMP to store the array base, unlike the code for
StoreIndexedInstr which must have a temporary register allocated during
register allocation.
Thus, fix Assembler::AddressCanHoldConstantIndex to always require
the out parameter, and add a comment in LoadIndexedInstr as to why
we ignore the out parameter in this case instead of allocating an
additional temporary register.
TEST=vm/cc/Assembler_Regress54621
Fixes: https://github.com/dart-lang/sdk/issues/54621
Change-Id: I0af557565faf657a87641457884334446e9b7cc5
Cq-Include-Trybots: luci.dart.try:vm-ffi-qemu-linux-release-arm-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/346201
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Previously, CanBeImmediateIndex calculated the scaling factor for the
index using Instance::ElementSizeFor. However, the LoadIndexed and
StoreIndexed instructions which use this function have an index_scale_
field that is not necessarily the same as the element size for the
class id. This means the displacement calculated within
CanBeImmediateIndex can differ from the actual displacement calculated
within ElementAddressForIntIndex, causing the bug seen by the fuzzer.
This CL moves the check in CanBeImmediateIndex to a new static
method Assembler::AddressCanHoldConstantIndex. In addition to the
original arguments to CanBeImmediateIndex, the new static method takes
an index_scale argument, so the field value can be passed appropriately.
It also adds an is_external argument on X64 and IA32 to match the other
architectures, since assuming a non-external typed data object could
cause a register to be used in cases where it isn't necessary.
TEST=vm/dart/regress_54486
Fixes: https://github.com/dart-lang/sdk/issues/54486
Change-Id: Ia553fb7da0500113b35f8d9af91a52df55437a3c
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-x64-try,vm-linux-debug-ia32-try,vm-aot-linux-debug-simarm_x64-try,vm-aot-linux-debug-simriscv64-try,vm-aot-mac-release-arm64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/345002
Reviewed-by: Daco Harkes <dacoharkes@google.com>
When generating code for deferred units compiler
can't emit PC relative call to the shared Mint allocation
stub. This causes compiler to emit an indirect call
through a Code object. Such calls clobber CODE_REG
which is actually an allocatable register.
Fix this by using non-allocatable register instead of
CODE_REG when generating indirect calls through
Code object in AOT mode. Callees don't expect
anything useful in CODE_REG anyway because AOT
calling convetion does not use it.
TEST=vm/cc/{BranchLinkPreservesRegisters,JumpAndLinkPreservesRegisters,CallCodePreservesRegisters}
Bug: b/242559057
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-release-arm64-try,vm-aot-linux-release-simarm_x64-try,vm-aot-linux-release-x64-try,vm-aot-linux-product-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-linux-debug-simriscv64-try,vm-ffi-qemu-linux-release-riscv64-try
Change-Id: Ib1fdc1c104d0269d41bb1ab9cbe292ad28c7cd49
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/338127
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
Adds a `CachableIdempotentCallInstr` that can be invoked via
`@pragma('vm:cachable-idempotent')` if the call-sites is force
optimized.
The object pool is not visited by the scavenger. So, we store the
results as unboxed integers. Consequently, only dart functions that
return integers can be cached.
Cachable idempotent calls should never be inlined. After the first
call the function not be called again.
The call itself is on a slow path to avoid register spilling on the
fast path.
TEST=vm/cc/IRTest_CachableIdempotentCall
TEST=runtime/tests/vm/dart/cachable_idempotent_test.dart
Bug: https://github.com/dart-lang/sdk/issues/51618
Change-Id: I612e896f27add76f57796c060157e14cc687a0fd
Cq-Include-Trybots: luci.dart.try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-asan-linux-release-x64-try,vm-aot-linux-debug-simarm_x64-try,vm-aot-linux-debug-simriscv64-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-msan-linux-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-tsan-linux-release-x64-try,vm-aot-ubsan-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-aot-win-debug-x64c-try,vm-aot-win-release-x64-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-kernel-linux-debug-x64-try,vm-kernel-precomp-linux-release-x64-try,vm-linux-debug-ia32-try,vm-linux-debug-simriscv64-try,vm-linux-debug-x64-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-tsan-linux-release-x64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-arm64-try,vm-win-debug-x64-try,vm-win-debug-x64c-try,vm-win-release-ia32-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/301601
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Follow up of:
* https://dart-review.googlesource.com/c/sdk/+/328861
* https://dart-review.googlesource.com/c/sdk/+/331087
* https://dart-review.googlesource.com/c/sdk/+/331120
The pool entries for switchable calls have their pool entries
transformed during snapshot loading in AOT. This behavior is now folded
into `SnapshotBehavior`.
Because snapshot behavior is set on the call sites, and the call sites
are already AOT specific, no conditionals inside app_snapshot.cc are
needed anymore.
The Arm64 implementation does not use the assembler for implicitly
adding pool entries, it does so explictly and loads two pool entries
in one load.
We could consider doing these explicit loads in the other architectures
as well. This would avoid adding the snapshot_behavior argument to
`LoadUnqiueObject` for the other architectures.
IA32 has no AOT support.
TEST=precomp bots
Change-Id: I75f70e00ca44f6545694d9b59e3ad08ad8bb3f8e
Cq-Include-Trybots: luci.dart.try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-linux-debug-simarm_x64-try,vm-aot-linux-debug-simriscv64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-release-arm64-try,vm-aot-linux-release-simarm_x64-try,vm-ffi-qemu-linux-release-riscv64-try,vm-linux-debug-simriscv64-try,vm-aot-mac-product-arm64-try,vm-aot-win-debug-arm64-try,vm-ffi-android-debug-arm64c-try,vm-ffi-android-debug-arm-try,vm-ffi-qemu-linux-release-arm-try,vm-aot-win-debug-x64c-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/331101
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
`NativeCallInstr::EmitNativeCode` `link_lazily` code path uses the
new snapshotting behavior, and the non-`link_lazily` checks that it's
never snapshotted.
The snapshot behavior argument is threaded from the the EmitNativeCode
to the places where the pool entry is constructed.
Does not yet migrate `kSwitchableCallMissEntryPoint` and
`kMegamorphicCallEntryPoint`.
TEST=CI
Change-Id: I8dd1dc5aa525f888b493856c60ae385087463d02
Cq-Include-Trybots: luci.dart.try:vm-ffi-qemu-linux-release-riscv64-try,vm-linux-debug-simriscv64-try,vm-aot-linux-release-arm64-try,vm-aot-android-release-arm64c-try,vm-aot-win-debug-arm64-try,vm-aot-mac-product-arm64-try,vm-mac-debug-arm64-try,vm-ffi-android-debug-arm-try,vm-ffi-qemu-linux-release-arm-try,vm-linux-debug-ia32-try,vm-linux-debug-x64-try,vm-aot-linux-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/328861
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
When copying elements that are smaller than words on non-X86
architectures, only copy element by element until the remaining elements
can be copied in word-sized chunks, and then do so.
TEST=vm/cc/IRTest_Memory, co19{,_2}/LibTest/typed_data,
lib{,_2}/typed_data, corelib{,_2}/list_test
Issue: https://github.com/dart-lang/sdk/issues/42072
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-simarm_x64-try,vm-aot-linux-debug-simriscv64-try,vm-linux-debug-simriscv64-try,vm-mac-debug-arm64-try,vm-aot-linux-release-simarm64-try,vm-aot-linux-release-simarm_x64-try,vm-aot-mac-release-arm64-try,vm-ffi-qemu-linux-release-riscv64-try,vm-ffi-qemu-linux-release-arm-try,vm-linux-release-simarm-try,vm-linux-release-simarm64-try,vm-mac-release-arm64-try,vm-aot-android-release-arm64c-try,vm-ffi-android-debug-arm64c-try
Change-Id: I61eab310b92a6bc5ebd88fa63d562103d887cb74
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/324280
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
AsmIntrinsifier::OneByteString_getHashCode now uses the CombineHashes
and FinalizeHash Assembler methods introduced in 49f998dc31.
To avoid having to check for 0 again after masking the result of
FinalizeHash to fit the desired bit size, we create a more general
FinalizeHashForSize that also takes the desired bit size of the result.
The intrinsified versions for ARM/ARM64/RISCV are also modified to
remove the special casing for the empty string, and instead just check
the size of the string against the loop counter at the start of the
loop, similarly to X64/IA32. This is less efficient for the empty
string case, but avoids the need to create a version of FinalizeHash
that elides the zero check.
TEST=ci
Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-dwarf-linux-product-x64-try,vm-kernel-precomp-linux-product-x64-try,vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-nnbd-mac-release-arm64-try,vm-kernel-precomp-nnbd-linux-release-simarm_x64-try,vm-kernel-precomp-linux-release-simarm-try,vm-kernel-precomp-nnbd-linux-release-x64-try,vm-kernel-precomp-nnbd-linux-release-simarm64-try,vm-kernel-precomp-nnbd-linux-debug-simriscv64-try,vm-kernel-precomp-linux-debug-x64c-try,vm-kernel-nnbd-linux-debug-simriscv64-try,vm-kernel-linux-release-simarm-try,vm-kernel-linux-release-simarm64-try,vm-kernel-linux-release-ia32-try,vm-kernel-nnbd-linux-release-simarm-try,vm-kernel-nnbd-linux-release-simarm64-try,vm-kernel-nnbd-linux-release-ia32-try,vm-kernel-nnbd-mac-release-arm64-try,vm-kernel-linux-debug-x64c-try
Change-Id: I53db731acb8bf35d16efb55fa2c5a79d8302399f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/272800
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Lower the threshold for converting from a linear to a hash-based cache
on most architectures from 500 to 10.
Due to register pressure, the InstantiateTypeArguments stub on IA32
continues to go to the runtime for hash caches, and so we do not
lower the threshold for converting to a hash-based cache there.
The following are benchmark results for those benchmark that use enough
Instantiations to trigger the use of hash-based caches. In the following
tables, "Results 1" denotes the benchmark results from only this change,
whereas "Results 2" include from comparing to the results prior to
4f925105cf, when only linear caches were used.
Dart AOT:
* InstantiateTypeArguments.Instantiate100
Arch | CPU | Results 1 | Results 2
-------|----------------|-----------------------
ARM | Odroid-C2 | 382.8% | 381.5%
ARM | Raspberry Pi 4 | 486.7% | 449.2%
ARM64 | Odroid-C2 | 328.1% | 372.8%
ARM64 | Raspberry Pi 4 | 1283% | 1281%
ARM64C | Raspberry Pi 4 | 2353% | 2811%
X64 | Intel Xeon | 568.7% | 584.9%
* InstantiateTypeArguments.Instantiate1000
Arch | CPU | Results 1 | Results 2
-------|----------------|------------------------
ARM | Odroid-C2 | 743.7% | 3821%
ARM | Raspberry Pi 4 | 486.7% | 3218%
ARM64 | Odroid-C2 | 584.7% | 3222%
ARM64 | Raspberry Pi 4 | 430.7% | 8172%
ARM64C | Raspberry Pi 4 | 491.4% | 16699%
X64 | Intel Xeon | 954.1% | 5528%
Dart JIT:
* InstantiateTypeArguments.Instantiate100
Arch | CPU | Results 1 | Results 2
-------|----------------|-----------------------
ARM | Raspberry Pi 4 | 315.7% | 295.1%
ARM64 | Raspberry Pi 4 | 1070% | 1058%
ARM64C | Raspberry Pi 4 | 1769% | 2095%
X64 | Intel Xeon | 507.4% | 496.2%
* InstantiateTypeArguments.Instantiate1000
Arch | CPU | Results 1 | Results 2
-------|----------------|-----------------------
ARM | Raspberry Pi 4 | 565.2% | 2550%
ARM64 | Raspberry Pi 4 | 406.8% | 7375%
ARM64C | Raspberry Pi 4 | 379.6% | 12996%
X64 | Intel Xeon | 807.9% | 4459%
During work on this change, an issue was found where cache lookups
in the stub on ARM64C always failed and went to runtime, even with
the old linear-only caches, hence the much larger performance gains
in those rows above.
TEST=vm/cc/TypeArguments_Cache_{Some,Many}Instantiations
Fixes: https://github.com/dart-lang/sdk/issues/48344
Change-Id: I3d29566ba0582502954c9fc59626ceb8fd40317a
Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-dwarf-linux-product-x64-try,vm-kernel-precomp-linux-product-x64-try,vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-nnbd-mac-release-arm64-try,vm-kernel-precomp-nnbd-linux-release-simarm_x64-try,vm-kernel-precomp-linux-release-simarm-try,vm-kernel-precomp-nnbd-linux-release-x64-try,vm-kernel-precomp-nnbd-linux-release-simarm64-try,vm-kernel-precomp-nnbd-linux-debug-simriscv64-try,vm-kernel-precomp-tsan-linux-release-x64-try,vm-kernel-tsan-linux-release-x64-try,vm-kernel-precomp-linux-debug-x64c-try,vm-kernel-nnbd-linux-debug-simriscv64-try,vm-kernel-linux-release-simarm-try,vm-kernel-linux-release-simarm64-try,vm-kernel-linux-release-ia32-try,vm-kernel-nnbd-linux-release-simarm-try,vm-kernel-nnbd-linux-release-simarm64-try,vm-kernel-nnbd-linux-release-ia32-try,vm-kernel-nnbd-mac-release-arm64-try,vm-kernel-linux-debug-x64c-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/270702
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Some large applications are bumping up against the 16-bit limit.
The object header changes from hash:32,cid:16,size:8,gc:8 to hash:32,cid:20,size:4,gc:8.
Because the CID field is no longer a power of two, it takes two instructions to load instead of one, increasing code size. The alternative change to cid:32,hash:24,gc:8 was considered and rejected because the reduction in hash bits very noticeably affects performance in applications with large data sets.
TEST=ci
Bug: b/255934984
Change-Id: I5fa36ba48a6852359d994393cf80c6a761c5d84c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/269120
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Alexander Aprelev