For non-x86 architectures, these forms directly translate to a
single instruction when possible. On x86 architectures,
XImmediate(dst, src, imm) where X is in {Lsl, ArithmeticShiftRight, And}
can be translated to:
MoveRegister(dst, src);
XImmediate(dst, imm);
in the general case since MoveRegister(dst, src) is a no-op if dst and
src are the same.
Also add versions of these assembler macros that take an OperandSize
and handle 32-bit OperandSizes appropriately on 64-bit architectures.
Ensure the implementation for LslImmediate and
ArithmeticShiftRightImmediate emits no instructions if shift == 0,
dst == src, and OperandSizeInBits(sz) == kBitsPerWord on all
architectures.
-----
Other changes:
Fix ConstantExpression::EmitMoveToLocation on ARM64 to match
other architectures, which allow any word-sized or less unboxed
integer representation.
Fill out ExtendValue on RISCV for previously unimplemented
OperandSizes, using the Zba and Zbb extensions when possible.
-----
TEST=vm/cc/Assembler_AndImmediate vm/cc/Assembler_LslImmediate
vm/cc/Assembler_ArithmeticShiftRightImmediate
Separated out of https://dart-review.googlesource.com/c/sdk/+/378706
for easier debugging/reviewing.
Change-Id: I721f1334784f7011a958ea8af29f0e56c620726c
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-release-simarm_x64-try,vm-aot-linux-debug-x64-try,vm-aot-mac-release-arm64-try,vm-aot-mac-product-arm64-try,vm-aot-linux-product-x64-try,vm-aot-dwarf-linux-product-x64-try,vm-aot-linux-debug-simarm_x64-try,vm-linux-debug-x64-try,vm-mac-debug-arm64-try,vm-mac-release-arm64-try,vm-linux-debug-ia32-try,vm-aot-android-release-arm64c-try,vm-ffi-android-debug-arm64c-try,vm-aot-linux-debug-x64c-try,vm-linux-debug-x64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/386180
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
* Use AllocateClosure instruction for closure instantiations in order
to initialize closure entry point.
* Fix null handling in operator== to be before argument type checks.
* Add _InvocationMirror._withType to dynamic interface as it is
implicitly used by noSuchMethod forwarders.
* Fix AssertAssignable for null objects.
TEST=language tests in vm-aot-dyn-linux-debug-x64 configuration
Change-Id: I7b1a037d4fde4d22ed32969e0f099b31ea4432ab
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/382500
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Assembler::AddressCanHoldConstantIndex, which is used by
MakeLocationSummary to determine whether to keep the index constant or
to allocate a register for it, performs the check:
const int64_t disp =
index * index_scale +
(is_external ? 0 : target::Instance::DataOffsetFor(cid) - kHeapObjectTag);
return Utils::IsInt(32, disp);
However, in Assembler::ElementAddressForIntIndex, used by EmitNativeCode to
produce an appropriate Address for the constant index, the code performs
a slightly different check:
const int64_t disp = static_cast<int64_t>(index) * index_scale +
target::Instance::DataOffsetFor(cid);
ASSERT(Utils::IsInt(32, disp));
return FieldAddress(array, static_cast<int32_t>(disp));
Thus, if a constant index produces the displacement kMaxInt32 + 1,
MakeLocationSummary will appropriately keep the index as a constant, but
the ASSERT in ElementAddressForIntIndex will trigger.
Modify ElementAddressForIntIndex in the X64 and IA32 assemblers so that
the same displacement is checked in both places.
TEST=vm/cc/IRTest_Regress_56588
Fixes: https://github.com/dart-lang/sdk/issues/56588
Change-Id: I504d6f92230540d74409a99b64b444acdb9c85f7
Cq-Include-Trybots: luci.dart.try:vm-linux-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/382801
Auto-Submit: Tess Strickland <sstrickl@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Decorating a field with 'vm:shared' pragma makes values in this field accessible to all isolates in an isolate group.
Introduce `channel` to the `Version` class so that the pragma can only be enabled on main and dev channels.
TEST=shared_test, shared_fail_without_flag_test
BUG=https://github.com/dart-lang/sdk/issues/55991
Change-Id: I843c9f0d2ffc9f2ced7ddc4006bb6f9ca4e2ddf4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/370064
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Alexander Aprelev <aam@google.com>
Add an additional base argument to Assembler::AddScaled so it now
computes:
dest <- base + (index << scale) + offset
If base is kNoRegister (or ZR when available), then it emits
instructions optimized for computing:
dest <- (index << scale) + offset
(i.e., its previous implementation)
Add AddScaled to AssemblerBase to ensure the same interface across all
architectures.
Rework the backend of CalculateElementAddress to use AddScaled
appropriately, which unifies it across architectures.
TEST=ci (refactoring)
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-simarm_x64-try,vm-aot-linux-debug-simriscv64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-linux-debug-ia32-try,vm-linux-debug-x64-try,vm-linux-debug-x64c-try,vm-linux-debug-simriscv64-try,vm-mac-debug-arm64-try,vm-aot-mac-release-arm64-try
Change-Id: I33c8f99604b68360f10b79050bd66ceb9d65ac9b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/370504
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
The ASSERT checks that there will be no overflow in computing the byte
offset from the index. However, any instruction that uses this method is
preceded by a bounds check of the index, and an index that would cause
the byte offset calculation to overflow is guaranteed to fail the bounds
check. Thus, this is dead code that will never be executed at runtime
and it is okay to emit code to perform the erroneous offset calculation.
TEST=vm/dart/regress_55864
Fixes: https://github.com/dart-lang/sdk/issues/55864
Change-Id: If4126e412368b8307be3c787ccb6bd451764c73f
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-simarm_x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/368563
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Delay changing Thread::vm_tag on callback entry and restore the tag early on callback return so that the profiler doesn't see the "running Dart" tag unless it can also see the fake return address marking the entry frame.
TEST=ffi/async_void_function_callbacks, ffi/function_callbacks_subtype, ffi/function_callbacks, ffi/isolate_local_function_callbacks
Bug: https://github.com/dart-lang/sdk/issues/52814
Change-Id: I40d80ec7c44063d078db0e211565e2d127c6b81e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/367460
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
This reverts commit bc0f02e4c8.
Reason for revert: https://github.com/dart-lang/sdk/issues/55754
Original change's description:
> [vm, gc] Incremental compaction.
>
> At the beginning of a major GC cycle, select some mostly-empty pages to be evacuated. Mark the pages and the objects on these pages. Apply a write barrier for stores creating old -> evacuation candidate pointers, and discover any such pointers that already exist during marking.
>
> At the end of a major GC cycle, evacuate objects from these pages. Forward pointers of objects in the remembered set and new-space. Free the evacuated pages.
>
> This compaction is incremental in the sense that creating the remembered set is interleaved with mutator execution. The evacuation step, however, is stop-the-world.
>
> Write-barrier elimination for x.slot = x is removed. Write-barrier elimination for x.slot = constant is removed in the JIT, kept for AOT but snapshot pages are marked as never-evacuate.
>
> TEST=ci
> Bug: https://github.com/dart-lang/sdk/issues/52513
> Change-Id: Icbc29ef7cb662ef8759b8c1d7a63b7af60766281
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/357760
> Reviewed-by: Alexander Aprelev <aam@google.com>
> Commit-Queue: Ryan Macnak <rmacnak@google.com>
Bug: https://github.com/dart-lang/sdk/issues/52513
Change-Id: I565ad6c0fca283d33f605c10f181bc0a59e7d2b2
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/366965
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Alexander Aprelev <aam@google.com>
Auto-Submit: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
At the beginning of a major GC cycle, select some mostly-empty pages to be evacuated. Mark the pages and the objects on these pages. Apply a write barrier for stores creating old -> evacuation candidate pointers, and discover any such pointers that already exist during marking.
At the end of a major GC cycle, evacuate objects from these pages. Forward pointers of objects in the remembered set and new-space. Free the evacuated pages.
This compaction is incremental in the sense that creating the remembered set is interleaved with mutator execution. The evacuation step, however, is stop-the-world.
Write-barrier elimination for x.slot = x is removed. Write-barrier elimination for x.slot = constant is removed in the JIT, kept for AOT but snapshot pages are marked as never-evacuate.
TEST=ci
Bug: https://github.com/dart-lang/sdk/issues/52513
Change-Id: Icbc29ef7cb662ef8759b8c1d7a63b7af60766281
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/357760
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
This pragma forces compiler to align loop headers within the
function by architecture specific boundary: 32 bytes on X64
and ARM64 (with the exception of Apple Silicon, which explicitly
discourages aligning branch targets in the optimization manual).
Current implementation is rather naive and does not do any
attempt to decide whether aligning is actually profitable
based on loop body itself.
I have found this pragma to be helpful both to stabilize
benchmark results and achieve better performance
for tight loops on Intel hardware.
Issue https://github.com/dart-lang/sdk/issues/55522
TEST=vm/dart/align_loops_test
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-product-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-linux-debug-x64-try,vm-aot-linux-release-arm64-try,vm-aot-linux-release-simarm_x64-try,vm-aot-linux-release-x64-try,vm-aot-mac-product-arm64-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-release-x64-try,vm-aot-win-debug-arm64-try,vm-aot-win-debug-x64-try,vm-aot-win-debug-x64c-try,vm-aot-win-product-x64-try,vm-aot-win-release-arm64-try,vm-aot-linux-debug-simarm_x64-try,vm-aot-linux-debug-simriscv64-try,vm-aot-dwarf-linux-product-x64-try,vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try
Change-Id: Ic22fb90d85e7fdebeeaa3908a43328c59436ab58
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/364121
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
When uploading CLs, the presubmit checks verify that the lines in the
diff are formatted correctly according to `git cl format runtime`.
However, when `buildtools/<os>-<arch>/clang/bin/clang-format` is
updated, it does not force reformatting of files that would be
reformatted.
This leads to two issues:
* Inconsistent style within the code base and within a single file.
* Spurious reformatting in CLs when (1) clang-format is used on the
whole file, or (2) the diff lines overlap.
`clang-format` doesn't change that frequently, so in general this is
not a large issue, but I've seen a bit too many "spurious formatting,
please revert" comments on CLs recently.
This CL formats the runtime to be in line with the current pinned
`clang-format`:
```
$ find runtime/ -iname *.h -o -iname *.cc | xargs buildtools/mac-arm64/clang/bin/clang-format -i
```
`git cl format` (which only formats changed lines, and does so with
`clang-format`) seems to not agree with itself, or clang-format, or
cpplint in a handful of places. This CL adds `// clang-format off`
for these. (See previous patchsets for the specific instances.)
TEST=A variety of bots including GCC, MacOS and Windows.
Change-Id: I470892e898971899fda14bb3b8f2c8efefd67686
Cq-Include-Trybots: luci.dart.try:vm-gcc-linux-try,vm-ffi-qemu-linux-release-riscv64-try,vm-ffi-qemu-linux-release-arm-try,vm-aot-win-debug-x64-try,vm-win-debug-x64c-try,vm-mac-debug-x64-try,vm-mac-debug-arm64-try,vm-aot-linux-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/362780
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
The MemoryCopyInstr on x64 in MSAN mode when forward copying was
unpoisoning at the wrong pointer. The `rep mov` changes the
destination register.
Now we unpoison _before_ the `rep mov` if it's a forward copy, and
_after_ the `rep mov` for backwards copies. This way the destination
register contains the start of the memory in both cases.
TEST=tests/ffi/msan_test.dart
b/266213262
Change-Id: I37688802e63797f650a8242b6ba8b884813ebbd0
Cq-Include-Trybots: luci.dart.try:vm-msan-linux-release-x64-try,vm-aot-msan-linux-release-x64-try,vm-linux-debug-ia32-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/361420
Reviewed-by: Tess Strickland <sstrickl@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Now Slots have an extra flag that describes whether or not the
instance object for that Slot is tagged (Dart objects) or untagged
(non-Dart objects like Thread).
Add new Slot categories for Slots with untagged instances, based on
the return value of the Slot:
* a tagged value (e.g., the finalizers_ field of Isolate),
* an untagged value (e.g., the isolate_ field of Thread), or
* an unboxed value (e.g., the enabled_ field of StreamInfo).
The primary difference in the specification of slots for untagged
instances vs tagged instances is that the slots for untagged instances
do not include an underlying type (as only Dart objects have an
Untagged<X> for each <X> class that specifies the actual object layout).
Also do a little more refactoring of Slot categories to simplify
some switch statements that now only need to be updated if the
categories the switch statement deals with change. For example,
methods for range analysis are only interested in Slots that are
guaranteed to contain integers, so now those methods are written in
a way that they only need to be updated if a new tagged integer
Slot declaration is added.
Adjust LoadFromSlot, StoreToSlot, and StoreToSlotNoBarrier to
appropriately handle Slots on untagged instances, and change
LoadField and StoreField's RequiredInputRepresentation methods
to appropriately returned kTagged or kUntagged based on the slot.
TEST=ci
Cq-Include-Trybots: luci.dart.try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-arm64-try,vm-win-debug-x64-try,vm-win-release-ia32-try
Change-Id: Ic27de4f092c9e087486066459f68a82b4e1685b0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/357341
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Add OperandSize arguments to the non-compressed Store methods so that
the compressed Store methods can be implemented as calls to the
corresponding non-compressed method using kObjectBytes. This makes it
so that all compressed Store methods are now non-virtual.
Add a scratch register argument to all Store methods that involve a
write barrier that defaults to TMP. Originally only the IA32
implementation took a scratch register, with the others using TMP or
TMP2 internally.
Since all architectures can represent a register/offset pair as an
Address without losing information, make all the offset-based load
and store methods have an implementation that creates an Address or
FieldAddress and calls the corresponding address-based store method.
This makes all the offset-based load and store methods non-virtual.
After this, most of the Store methods are non-virtual, with
only the base methods used to implement them as virtual:
* Store
* StoreRelease
* StoreObjectIntoObjectNoBarrier
* StoreBarrier
* ArrayStoreBarrier
* VerifyStoreNeedsNoWriteBarrier (for DEBUG use only)
The Load methods can't be consolidated quite so much, as the
base methods for handling non-Smi compressed pointers must be defined
per-architecture, as each compressed pointer architecture handles
adding the upper bits for the heap separately.
Have LoadFieldInstr::EmitNativeCode use Assembler::LoadToSlot when the
result location is a single integer register and
StoreFieldInstr::EmitNativeCode use Assembler::StoreToSlot and
Assembler::StoreToSlotNoBarrier when the value location is a single
integer register to avoid code duplication between those methods.
Handle both compressed and uncompressed Smi fields in LoadFromSlot.
TEST=ci (refactoring)
Cq-Include-Trybots: luci.dart.try:vm-aot-android-release-arm64c-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-x64-try,vm-win-release-ia32-try
Change-Id: I60cc03776af220ed87918664bb4b9abafff2788a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/360641
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Due to a missing return, two stores would be done for a non-tagged field
in StoreToSlot: the correct store without a write barrier and then a
second store that also applies the write barriers inappropriately.
Rewrite LoadFromSlot/StoreToSlot/StoreToSlotNoBarrier to just use
chained ifs instead of returning in some if statement bodies and not
in others.
TEST=ci
Change-Id: Ia0a204217433056856e4dc355758f34beceb79b6
Cq-Include-Trybots: luci.dart.try:vm-aot-android-release-arm64c-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-ffi-android-debug-arm64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-arm64-try,vm-win-debug-x64-try,vm-win-release-ia32-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/360862
Auto-Submit: Tess Strickland <sstrickl@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Remove CompareWithCompressedFieldFromOffset, which has no uses.
Rename the LoadFromOffset and StoreFromOffset methods that took
Addresses to Load and Store, respectively. This makes the names
of the Assembler methods more uniform:
* Takes an address: Load, Store, LoadField, LoadCompressedField,
StoreIntoObject, StoreCompressedIntoObject, LoadSmi,
LoadCompressedSmi, etc.
* Takes a base register and an offset: LoadFromOffset, StoreToOffset,
LoadFieldFromOffset, LoadCompressedFieldFromOffset,
StoreIntoObjectOffset, StoreCompressedIntoObjectOffset,
LoadSmiFromOffset, LoadCompressedSmiFromOffset, etc.
Create AssemblerBase methods for loading and storing compressed
pointers that weren't already there, as well as the corresponding
methods for loading and storing uncompressed values.
Make non-virtual methods that load and store uncompressed fields
that call the corresponding method for loading from and storing to
memory regions, adjusting the address or offset accordingly. This
avoids needing per-architecture overrides for these.
Make non-virtual methods that load compressed fields, calling the
corresponding method for loading a compressed value from a memory
region. (Since compressed pointers are only stored in Dart objects,
and stores into a Dart object may require a barrier, there is no
method for storing a compressed value into an arbitrary memory region.)
Create pure virtual methods for loading from or storing to an Address
or any method that does not have both an Address-taking and a
base register and offset pair-taking version (e.g., LoadAcquire).
Create methods for loading from or storing to a base register
and an offset. The base implementation takes the base register and
offset and creates an Address from it, then calls the Address-taking
equivalent. These methods are non-virtual when the implementation is
the same on all architectures and virtual to allow overriding when
necessary.
Make a non-virtual method for loading uncompressed Smis, since all
architectures have the same code for this, including the DEBUG check.
If compressed pointers are not being used, all the methods for
compressed pointers are non-virtual methods that call the
corresponding method for uncompressed values.
If compressed pointers are being used:
* Install pure virtual methods for loading compressed values from
and storing compressed values to an Address or any method that does
not have both an Address-taking and a base register and offset
pair-taking version (e.g., LoadAcquireCompressed).
* Install virtual methods for loading compressed values from and
storing compressed values to a base register and offset. Like the
uncompressed implementation, the base implementation of these
create an Address and call the Address-taking equivalent, and these
implementations are overridden on ARM64.
* Install a non-virtual method for loading compressed Smis, since the
only difference is that it loads a zero-extended 32-bit value, which
AssemblerBase can do.
TEST=ci (refactoring only)
Change-Id: I934791d26a6e2cdaa6ac5f188b0fd89dbdc491d1
Cq-Include-Trybots: luci.dart.try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-arm64-try,vm-win-debug-x64-try,vm-win-release-ia32-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/359861
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
After https://dart-review.googlesource.com/c/sdk/+/330600, there were
more chances for the optimizing compiler to introduce or move
GC-triggering instructions like allocations or boxings between the
retrieval of an untagged pointer to GC-moveable memory and its use.
To limit the chance of this happening, this CL removes the explicit
loading of the untagged payload address when building the initial
flow graph in most cases when the array is not known to be an external
array (an external string, an external typed data object, or an FFI
Pointer).
The remaining case is during view allocation, which extracts the
payload address of the base typed data object underlying the view
(which may be GC-movable) to calculate the payload address that should
be stored in the data field of the view object. See
https://github.com/dart-lang/sdk/issues/54884.
During canonicalization of LoadIndexed, StoreIndexed, and MemoryCopy
instructions, if the cid of an array input is an external array
(external string, external typed data object, or Pointer), then a
LoadField instruction that extracts the untagged payload address
is inserted before the instruction and the corresponding input is
rebound to that LoadField instruction.
Once all compiler passes that involve code motion have been performed,
a new pass looks for LoadIndexed, StoreIndexed, or MemoryCopy where
the cid stored in the instruction for the array is a typed data cid.
In these cases, if the array is not an internal typed data object,
then the payload address is extracted. Waiting until this point ensures
that no GC-triggering instructions are inserted between the extraction
of the payload address and the use. (Internal typed data objects are
left as-is because the payload address is inside the object itself
and doesn't require indirection through the data field of the object).
This CL also replaces code conditional on the array cid with code
that is instead conditional on the array element representation in
cases where it makes sense to do so, since this is a less brittle
check than checking the array cid (e.g., checking for kUnboxedInt8
to load, store, or copy an signed byte from an array instead of
listing all possible array cids that store signed bytes).
This CL also fixes an issue with the ARM64 assembler where calling
LoadFromOffset with an Address that has a non-Offset type would
silently generate bad code instead of triggering the ASSERT in
PrepareLargeOffset.
TEST=vm/dart/typed_list_index_checkbound_il_test
Issue: https://github.com/dart-lang/sdk/issues/54710
Cq-Include-Trybots: luci.dart.try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-fuchsia-release-x64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-arm64-try,vm-win-debug-x64-try,vm-win-release-ia32-try
Change-Id: I25b5f314943e9254d3d28986d720a5d47f12feeb
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/352363
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Moves RepresentationOfArrayElement out of LoadIndexed and StoreIndexed
into RepresentationUtils. Adjusts RepresentationOfArrayElement
to work for all concrete typed data classes. Changes
RepresentationOfArrayElement to return the precise representation of
the array element, which requires adding two new representations,
kUnboxedInt8 and kUnboxedInt16.
Remove Address::OperandSizeFor and its only (ignored) use in the
RISCV assembler and change Address::OperandSizeFor in the ARM and
ARM64 assemblers to use RepresentationUtils::OperandSize and RepresentationUtils::RepresentationOfArrayElement appropriately.
Add Boxing::NativeRepresentation, which converts representations for
unboxed integers smaller than 4 bytes to kUnboxedIntPtr and otherwise
returns the original representation. This mimics the current semantics
for LoadIndexed and StoreIndexed where returned values represented by
unboxed integer values smaller than 4 bytes are zero or sign extended
to word size appropriately, and received values represented by unboxed
integer values smaller than 4 bytes are appropriately truncated from a
word sized signed integer.
Create LoadIndexedInstr::ReturnRepresentation and
StoreIndexedInstr::ValueRepresentation, which use
Boxing::NativeRepresentation to convert the precise representation
returned by RepresentationUtils::RepresentationOfArrayElement to
the actual returned/expected representation for those instructions.
Also use Boxing::NativeRepresentation to appropriately adjust
the representations used for SIMD operations returning int8_t.
Add checks that unboxing and integer conversion never get
representations which result in a different representation when
passed to Boxing::NativeRepresentation, as those instructions do
not currently work with unboxed integer representations smaller than
4 bytes. (Adding such support can be done in a future CL.)
Inside the optimizing compiler, use the precise representation of the
array element where appropriate. That is:
* Range analysis
* Type propagation
* Redundancy elimination
This CL also cleans up how Representations are printed to C, making
that part of RepresentationUtils. It also removes the representation
printing and parsing utils from Location since they were only used by
the original IL serializer and deserializer, which has been removed.
It also renames RepresentationUtils::IsUnsigned to IsUnsignedInteger,
to make it clear that it only returns true for unboxed integer
representations.
TEST=standalone/bytedata_test
Fixes: https://github.com/dart-lang/sdk/issues/54622
Cq-Include-Trybots: luci.dart.try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-aot-win-debug-x64-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-fuchsia-release-x64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-arm64-try,vm-win-debug-x64-try,vm-win-release-ia32-try
Change-Id: I8c0dab911463ee4d0c55cd59ffea9d01e0f0bb7c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/354885
Reviewed-by: Slava Egorov <vegorov@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
The refactoring in https://dart-review.googlesource.com/c/sdk/+/345002
erroneously assumed that if the caller to
Assembler::AddressCanHoldConstantIndex didn't pass an out parameter to
detect whether a base register was needed, that that case shouldn't be
considered.
While it's true that LoadIndexedInstr::MakeLocationSummary originally
ignored the value stored in the out parameter, it did that because it
can use TMP to store the array base, unlike the code for
StoreIndexedInstr which must have a temporary register allocated during
register allocation.
Thus, fix Assembler::AddressCanHoldConstantIndex to always require
the out parameter, and add a comment in LoadIndexedInstr as to why
we ignore the out parameter in this case instead of allocating an
additional temporary register.
TEST=vm/cc/Assembler_Regress54621
Fixes: https://github.com/dart-lang/sdk/issues/54621
Change-Id: I0af557565faf657a87641457884334446e9b7cc5
Cq-Include-Trybots: luci.dart.try:vm-ffi-qemu-linux-release-arm-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/346201
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Previously, CanBeImmediateIndex calculated the scaling factor for the
index using Instance::ElementSizeFor. However, the LoadIndexed and
StoreIndexed instructions which use this function have an index_scale_
field that is not necessarily the same as the element size for the
class id. This means the displacement calculated within
CanBeImmediateIndex can differ from the actual displacement calculated
within ElementAddressForIntIndex, causing the bug seen by the fuzzer.
This CL moves the check in CanBeImmediateIndex to a new static
method Assembler::AddressCanHoldConstantIndex. In addition to the
original arguments to CanBeImmediateIndex, the new static method takes
an index_scale argument, so the field value can be passed appropriately.
It also adds an is_external argument on X64 and IA32 to match the other
architectures, since assuming a non-external typed data object could
cause a register to be used in cases where it isn't necessary.
TEST=vm/dart/regress_54486
Fixes: https://github.com/dart-lang/sdk/issues/54486
Change-Id: Ia553fb7da0500113b35f8d9af91a52df55437a3c
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-x64-try,vm-linux-debug-ia32-try,vm-aot-linux-debug-simarm_x64-try,vm-aot-linux-debug-simriscv64-try,vm-aot-mac-release-arm64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/345002
Reviewed-by: Daco Harkes <dacoharkes@google.com>
When generating code for deferred units compiler
can't emit PC relative call to the shared Mint allocation
stub. This causes compiler to emit an indirect call
through a Code object. Such calls clobber CODE_REG
which is actually an allocatable register.
Fix this by using non-allocatable register instead of
CODE_REG when generating indirect calls through
Code object in AOT mode. Callees don't expect
anything useful in CODE_REG anyway because AOT
calling convetion does not use it.
TEST=vm/cc/{BranchLinkPreservesRegisters,JumpAndLinkPreservesRegisters,CallCodePreservesRegisters}
Bug: b/242559057
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-release-arm64-try,vm-aot-linux-release-simarm_x64-try,vm-aot-linux-release-x64-try,vm-aot-linux-product-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-linux-debug-simriscv64-try,vm-ffi-qemu-linux-release-riscv64-try
Change-Id: Ib1fdc1c104d0269d41bb1ab9cbe292ad28c7cd49
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/338127
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
When disassembling functions, respect whether absolute addresses should
be used for PC offsets when printing stack map, PC descriptor and
exception handler information.
Also don't print out results for empty auxiliary information.
TEST=ci
Change-Id: Ieda990c1eabcce8be968981a81123a78746c6b6b
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/336663
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Adds a `CachableIdempotentCallInstr` that can be invoked via
`@pragma('vm:cachable-idempotent')` if the call-sites is force
optimized.
The object pool is not visited by the scavenger. So, we store the
results as unboxed integers. Consequently, only dart functions that
return integers can be cached.
Cachable idempotent calls should never be inlined. After the first
call the function not be called again.
The call itself is on a slow path to avoid register spilling on the
fast path.
TEST=vm/cc/IRTest_CachableIdempotentCall
TEST=runtime/tests/vm/dart/cachable_idempotent_test.dart
Bug: https://github.com/dart-lang/sdk/issues/51618
Change-Id: I612e896f27add76f57796c060157e14cc687a0fd
Cq-Include-Trybots: luci.dart.try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-asan-linux-release-x64-try,vm-aot-linux-debug-simarm_x64-try,vm-aot-linux-debug-simriscv64-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-msan-linux-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-tsan-linux-release-x64-try,vm-aot-ubsan-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-aot-win-debug-x64c-try,vm-aot-win-release-x64-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-kernel-linux-debug-x64-try,vm-kernel-precomp-linux-release-x64-try,vm-linux-debug-ia32-try,vm-linux-debug-simriscv64-try,vm-linux-debug-x64-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-tsan-linux-release-x64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-arm64-try,vm-win-debug-x64-try,vm-win-debug-x64c-try,vm-win-release-ia32-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/301601
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Follow up of:
* https://dart-review.googlesource.com/c/sdk/+/328861
* https://dart-review.googlesource.com/c/sdk/+/331087
* https://dart-review.googlesource.com/c/sdk/+/331120
The pool entries for switchable calls have their pool entries
transformed during snapshot loading in AOT. This behavior is now folded
into `SnapshotBehavior`.
Because snapshot behavior is set on the call sites, and the call sites
are already AOT specific, no conditionals inside app_snapshot.cc are
needed anymore.
The Arm64 implementation does not use the assembler for implicitly
adding pool entries, it does so explictly and loads two pool entries
in one load.
We could consider doing these explicit loads in the other architectures
as well. This would avoid adding the snapshot_behavior argument to
`LoadUnqiueObject` for the other architectures.
IA32 has no AOT support.
TEST=precomp bots
Change-Id: I75f70e00ca44f6545694d9b59e3ad08ad8bb3f8e
Cq-Include-Trybots: luci.dart.try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-linux-debug-simarm_x64-try,vm-aot-linux-debug-simriscv64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-release-arm64-try,vm-aot-linux-release-simarm_x64-try,vm-ffi-qemu-linux-release-riscv64-try,vm-linux-debug-simriscv64-try,vm-aot-mac-product-arm64-try,vm-aot-win-debug-arm64-try,vm-ffi-android-debug-arm64c-try,vm-ffi-android-debug-arm-try,vm-ffi-qemu-linux-release-arm-try,vm-aot-win-debug-x64c-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/331101
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
AOT no longer emit call sites that immediately use the megamorphic dispatch stub. Such call sites are now generated as unlinked switchable call sites that may eventually transition to megamorphic.
TEST=ci
Change-Id: Iec342a2b3bc8fe27612bfa14b516e3c0512c1134
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/331087
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
`NativeCallInstr::EmitNativeCode` `link_lazily` code path uses the
new snapshotting behavior, and the non-`link_lazily` checks that it's
never snapshotted.
The snapshot behavior argument is threaded from the the EmitNativeCode
to the places where the pool entry is constructed.
Does not yet migrate `kSwitchableCallMissEntryPoint` and
`kMegamorphicCallEntryPoint`.
TEST=CI
Change-Id: I8dd1dc5aa525f888b493856c60ae385087463d02
Cq-Include-Trybots: luci.dart.try:vm-ffi-qemu-linux-release-riscv64-try,vm-linux-debug-simriscv64-try,vm-aot-linux-release-arm64-try,vm-aot-android-release-arm64c-try,vm-aot-win-debug-arm64-try,vm-aot-mac-product-arm64-try,vm-mac-debug-arm64-try,vm-ffi-android-debug-arm-try,vm-ffi-qemu-linux-release-arm-try,vm-linux-debug-ia32-try,vm-linux-debug-x64-try,vm-aot-linux-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/328861
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Tess Strickland