All other architectures have a LoadImmediate that works with any uword, but x64's truncated to 32-bit, causing surprising x64-only failures when LoadImmediate is used from architecture-independent parts of the compiler.
TEST=ci
Change-Id: Ia2b2cfd1df7396833e4508ed21b4995fc9af464b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/268040
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Enable concurrent marking for IA32.
This removes the last write barrier relying on different alignment offsets for old and new space.
TEST=ci
Change-Id: Ib1c13124002392cf1c3ec264643325ec471a6918
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/262280
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Fix: Check handle contents for Smi.
Closes: https://github.com/flutter/flutter/issues/112726
Orignal CL description:
Makes `Dart_Handle` FFI returns behave as the following snippet:
```
Dart_Handle ExampleSnippet() {
Dart_Handle result = ...;
if (Dart_IsError(result)) {
Dart_PropagateError(result);
}
return result;
}
```
Also makes FFI consistent with Dart_NativeFunctions, which will
automatically throw upon return if Dart_SetReturnValue set the result
to an error.
`UnhandledExceptions` cannot flow out into Dart generated code. So,
the implementation needs to be in `FfiCallInstr::EmitNativeCode`.
Using `Dart_IsError` is slow compared to a machine code class id
check. So, we should do the handle unwrapping and class id check in
machine code.
Unwrapping Handles in machine code is only safe when the GC is
guaranteed to not run: Either (1) in `kThreadInGenerated`, or (2) in
`kThreadInNative`, but only when transitioned into safepoint. So, the
handle cannot be unwrapped immediately after the FFI call in machine code. We first need to transition back to generated.
This means we need to transition again to native to do the actual
`Dart_PropagateError` call. We can do so without the stub in JIT
because we never return with normal control flow.
Performance impact of this change is within benchmark noise in both
JIT and AOT.
Size impact is 42 bytes on x64, which is 10% in AOT and 12% in JIT.
For more numbers see: go/dart-ffi-handle-error
TEST=runtime/bin/ffi_test/ffi_test_functions_vmspecific.cc
TEST=tests/ffi/vmspecific_handle_test.dart
Closes: https://github.com/dart-lang/sdk/issues/49936
Change-Id: Id8edfd841a7d6246438386007d83747868a0a151
Cq-Include-Trybots: luci.dart.try:vm-canary-linux-debug-try,vm-ffi-android-debug-arm64c-try,vm-ffi-android-debug-arm-try,vm-kernel-gcc-linux-try,vm-kernel-linux-debug-x64-try,vm-kernel-linux-debug-x64c-try,vm-kernel-msvc-windows-try,vm-kernel-optcounter-threshold-linux-release-x64-try,vm-kernel-precomp-asan-linux-release-x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-kernel-reload-linux-debug-x64-try,vm-kernel-reload-rollback-linux-debug-x64-try,vm-kernel-tsan-linux-release-x64-try,vm-kernel-win-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-precomp-ffi-qemu-linux-release-arm-try,vm-precomp-ffi-qemu-linux-release-riscv64-try,vm-kernel-linux-debug-ia32-try,vm-kernel-mac-release-arm64-try,vm-kernel-precomp-win-debug-x64c-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/262342
Reviewed-by: Martin Kustermann <kustermann@google.com>
Auto-Submit: Daco Harkes <dacoharkes@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
This reverts commit d9c442bce8.
Reason for revert: https://github.com/flutter/flutter/issues/112726
Original change's description:
> [vm/ffi] Throw on returning `Error` in `Handle`
>
> Makes `Dart_Handle` FFI returns behave as the following snippet:
>
> ```
> Dart_Handle ExampleSnippet() {
> Dart_Handle result;
> if (Dart_IsError(result)) {
> Dart_PropagateError(result);
> }
> return result;
> }
> ```
>
> Also makes FFI consistent with Dart_NativeFunctions, which will
> automatically throw upon return if Dart_SetReturnValue set the result
> to an error.
>
> `UnhandledExceptions` cannot flow out into Dart generated code. So,
> the implementation needs to be in `FfiCallInstr::EmitNativeCode`.
>
> Using `Dart_IsError` is slow compared to a machine code class id
> check. So, we should do the handle unwrapping and class id check in
> machine code.
>
> Unwrapping Handles in machine code is only safe when the GC is
> guaranteed to not run: Either (1) in `kThreadInGenerated`, or (2) in
> `kThreadInNative`, but only when transitioned into safepoint. So, the
> handle cannot be unwrapped immediately after the FFI call in machine code. We first need to transition back to generated.
>
> This means we need to transition again to native to do the actual
> `Dart_PropagateError` call. We can do so without the stub in JIT
> because we never return with normal control flow.
>
> Performance impact of this change is within benchmark noise in both
> JIT and AOT.
> Size impact is 42 bytes on x64, which is 10% in AOT and 12% in JIT.
>
> For more numbers see: go/dart-ffi-handle-error
>
> TEST=runtime/bin/ffi_test/ffi_test_functions_vmspecific.cc
> TEST=tests/ffi/vmspecific_handle_test.dart
>
> Closes: https://github.com/dart-lang/sdk/issues/49936
> Change-Id: Ie8fabeb6d53bc80689541bc4470cb37ee2200581
> Cq-Include-Trybots: luci.dart.try:vm-canary-linux-debug-try,vm-ffi-android-debug-arm64c-try,vm-ffi-android-debug-arm-try,vm-kernel-gcc-linux-try,vm-kernel-linux-debug-x64-try,vm-kernel-linux-debug-x64c-try,vm-kernel-msvc-windows-try,vm-kernel-optcounter-threshold-linux-release-x64-try,vm-kernel-precomp-asan-linux-release-x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-kernel-reload-linux-debug-x64-try,vm-kernel-reload-rollback-linux-debug-x64-try,vm-kernel-tsan-linux-release-x64-try,vm-kernel-win-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-precomp-ffi-qemu-linux-release-arm-try,vm-precomp-ffi-qemu-linux-release-riscv64-try,vm-kernel-linux-debug-ia32-try,vm-kernel-mac-release-arm64-try,vm-kernel-precomp-win-debug-x64c-try
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/261603
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
> Reviewed-by: Martin Kustermann <kustermann@google.com>
> Commit-Queue: Daco Harkes <dacoharkes@google.com>
TBR=kustermann@google.com,rmacnak@google.com,dacoharkes@google.com,dart-scoped@luci-project-accounts.iam.gserviceaccount.com
Change-Id: I94cc63de16b54db2b0a4f92759c39a1e569b8e63
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Cq-Include-Trybots: luci.dart.try:vm-canary-linux-debug-try,vm-ffi-android-debug-arm64c-try,vm-ffi-android-debug-arm-try,vm-kernel-gcc-linux-try,vm-kernel-linux-debug-x64-try,vm-kernel-linux-debug-x64c-try,vm-kernel-msvc-windows-try,vm-kernel-optcounter-threshold-linux-release-x64-try,vm-kernel-precomp-asan-linux-release-x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-kernel-reload-linux-debug-x64-try,vm-kernel-reload-rollback-linux-debug-x64-try,vm-kernel-tsan-linux-release-x64-try,vm-kernel-win-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-precomp-ffi-qemu-linux-release-arm-try,vm-precomp-ffi-qemu-linux-release-riscv64-try,vm-kernel-linux-debug-ia32-try,vm-kernel-mac-release-arm64-try,vm-kernel-precomp-win-debug-x64c-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/262270
Reviewed-by: Zach Anderson <zra@google.com>
Reviewed-by: Liam Appelbe <liama@google.com>
Commit-Queue: Zach Anderson <zra@google.com>
Makes `Dart_Handle` FFI returns behave as the following snippet:
```
Dart_Handle ExampleSnippet() {
Dart_Handle result;
if (Dart_IsError(result)) {
Dart_PropagateError(result);
}
return result;
}
```
Also makes FFI consistent with Dart_NativeFunctions, which will
automatically throw upon return if Dart_SetReturnValue set the result
to an error.
`UnhandledExceptions` cannot flow out into Dart generated code. So,
the implementation needs to be in `FfiCallInstr::EmitNativeCode`.
Using `Dart_IsError` is slow compared to a machine code class id
check. So, we should do the handle unwrapping and class id check in
machine code.
Unwrapping Handles in machine code is only safe when the GC is
guaranteed to not run: Either (1) in `kThreadInGenerated`, or (2) in
`kThreadInNative`, but only when transitioned into safepoint. So, the
handle cannot be unwrapped immediately after the FFI call in machine code. We first need to transition back to generated.
This means we need to transition again to native to do the actual
`Dart_PropagateError` call. We can do so without the stub in JIT
because we never return with normal control flow.
Performance impact of this change is within benchmark noise in both
JIT and AOT.
Size impact is 42 bytes on x64, which is 10% in AOT and 12% in JIT.
For more numbers see: go/dart-ffi-handle-error
TEST=runtime/bin/ffi_test/ffi_test_functions_vmspecific.cc
TEST=tests/ffi/vmspecific_handle_test.dart
Closes: https://github.com/dart-lang/sdk/issues/49936
Change-Id: Ie8fabeb6d53bc80689541bc4470cb37ee2200581
Cq-Include-Trybots: luci.dart.try:vm-canary-linux-debug-try,vm-ffi-android-debug-arm64c-try,vm-ffi-android-debug-arm-try,vm-kernel-gcc-linux-try,vm-kernel-linux-debug-x64-try,vm-kernel-linux-debug-x64c-try,vm-kernel-msvc-windows-try,vm-kernel-optcounter-threshold-linux-release-x64-try,vm-kernel-precomp-asan-linux-release-x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-kernel-reload-linux-debug-x64-try,vm-kernel-reload-rollback-linux-debug-x64-try,vm-kernel-tsan-linux-release-x64-try,vm-kernel-win-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-precomp-ffi-qemu-linux-release-arm-try,vm-precomp-ffi-qemu-linux-release-riscv64-try,vm-kernel-linux-debug-ia32-try,vm-kernel-mac-release-arm64-try,vm-kernel-precomp-win-debug-x64c-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/261603
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Moves `RangeCheck` from the intrinsifier to the assembler.
Also, unifies the signature and semantics and adds documentation.
TEST=refactor
Change-Id: I274e5e46fde6a6e8baf30d8b732932f4f46d5855
Cq-Include-Trybots: luci.dart.try:vm-kernel-linux-debug-simriscv64-try,vm-kernel-precomp-linux-debug-simriscv64-try,vm-kernel-linux-release-simarm64-try,vm-kernel-linux-release-simarm-try,vm-kernel-precomp-linux-release-simarm64-try,vm-kernel-precomp-linux-release-simarm-try,vm-precomp-ffi-qemu-linux-release-riscv64-try,vm-precomp-ffi-qemu-linux-release-arm-try,vm-kernel-linux-debug-ia32-try,vm-kernel-linux-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/261681
Reviewed-by: Tess Strickland <sstrickl@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
This change fixes the following build errors when building Dart VM with
gcc:
../../runtime/vm/compiler/assembler/assembler_arm64.h: In member function ‘uint32_t dart::compiler::Address::encoding(dart::compiler::OperandSize) const’:
../../runtime/vm/compiler/assembler/assembler_arm64.h:355:3: error: control reaches end of non-void function [-Werror=return-type]
355 | }
../../runtime/vm/compiler/assembler/assembler_arm64.cc: In member function ‘void dart::compiler::Assembler::PopNativeCalleeSavedRegisters()’:
../../runtime/vm/compiler/assembler/assembler_arm64.cc:2380:52: error: bitwise comparison always evaluates to false [-Werror=tautological-compare]
2380 | bool pop_single = (kAbiPreservedFpuRegCount & 1) == 1;
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^~ ~
../../runtime/vm/compiler/assembler/assembler_arm64.cc:2400:47: error: bitwise comparison always evaluates to false [-Werror=tautological-compare]
2400 | pop_single = (kAbiPreservedCpuRegCount & 1) == 1;
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^~ ~
TEST=ci
Change-Id: Ib44790dbe99288f569a26e4df28f61ef2aa40ea3
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/260640
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
The operand size is dropped from Address/FieldAddress constructors on
arm64. This makes code using Address/FieldAddress less fragile, avoids
repeating size both in the Address/FieldAddress and load/store
instructions, and better aligns arm64 Address/FieldAddress with other
architectures.
TEST=ci
Change-Id: I92d7c5c8f0239333f022deebc0472136018bb0fa
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/260072
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Instead apply the same approach as we do in AOT: unbox based on the
static type information. There are no TFA results available in JIT,
but we could still unbox fields when running in sound null-safety.
TEST=ci
Cq-Include-Trybots: luci.dart.try:vm-kernel-reload-linux-release-x64-try,vm-kernel-reload-linux-debug-x64-try,vm-kernel-reload-rollback-linux-debug-x64-try,vm-kernel-reload-rollback-linux-release-x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-linux-product-x64-try,vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-nnbd-linux-release-simarm64-try,vm-kernel-linux-debug-simriscv64-try,vm-kernel-precomp-linux-debug-simriscv64-try,vm-kernel-nnbd-linux-release-ia32-try,vm-kernel-nnbd-linux-debug-x64-try,vm-kernel-nnbd-linux-debug-ia32-try,vm-kernel-nnbd-linux-release-simarm-try,vm-kernel-nnbd-linux-release-simarm64-try
Change-Id: Ide2e78c6659261ef8d245a4586cf699ea0fbb459
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/256211
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
* Merge ClassTable and SharedClassTable back together;
* Simplify handling of multiple arrays growing in sync;
* Refactor how reload deals with ClassTable.
The last change is the most important because it makes it
much easier to reason about the code. We move away from
copying bits and pieces of the class table and shared
class table into reload contexts.
Having two class table fields in the isolate group makes
it easier to reason about. One field contains program
class table (one modified by kernel loader and accessed
by various program structure cid lookups) and heap
walk class table (used by GC visitors). Normally these
two fields point to the same class table, but during
hot reload we temporary split them apart: original
class table is kept as a heap walk class table, while
program class table is replaced by a clone and updated
by reload.
If reload succeeds we drop original class table and
set program class table as heap walk one.
If reload fails we drop the program class table and
restore original one from heap walk table.
TEST=ci
Cq-Include-Trybots: luci.dart.try:vm-kernel-reload-linux-release-x64-try,vm-kernel-reload-linux-debug-x64-try,vm-kernel-reload-rollback-linux-debug-x64-try,vm-kernel-reload-rollback-linux-release-x64-try,vm-kernel-linux-debug-x64-try,vm-kernel-precomp-tsan-linux-release-x64-try,vm-kernel-tsan-linux-release-x64-try,vm-kernel-precomp-asan-linux-release-x64-try,vm-kernel-asan-linux-release-x64-try
Change-Id: I8b66259fcc474dea7dd2af063e4772df99be06c4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/258361
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
In tsan mode, the load-acquire/store-release assembler implementation
will make a runtime call to let TSAN know about the operation.
In order to perform the runtime call the THR register needs to be
populated, which it wasn't.
TEST=Fixes crashing vm/cc/StoreReleaseLoadAcquire tests in tsan mode
Change-Id: I1f45fee9c8e40b8069bf32b13caa97a02f13a6a1
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/256261
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
The new implementation is based on suspend/resume stubs and doesn't
use desugaring of async functions on kernel AST.
Previously, new implementation of async/async* was only supported in
AOT mode. This change adds all necessary bits for the JIT mode:
* Suspending variable-length frames (for unoptimized code).
* Handling of Code and pool pointers in Dart stack frames.
* OSR.
* Deoptimization.
* Hot reload.
* Debugger.
The new implementation is not enabled in JIT mode yet.
Design doc: go/compact-async-await.
TEST=ci
Issue: https://github.com/dart-lang/sdk/issues/48378
Change-Id: I477d6684bdce7cbc1edb179ae2271ff598b7dcc5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/246081
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
The CCallInstr takes unboxed word-sized integers and does a call to C
without any VM transitions.
This can be used for the following.
1) Leaf runtime entries. This CL removes the EnterHandleScope,
ExitHandleScope, and AllocateHandle instructions.
2) Compilation of isolate independent code where all definitions in IL
are unboxed integers.
Currently, the CCallInstr expects all it's arguments and the return type
to be word-sized.
This CL also:
- Adds loadImmediate and CCall with a register for each assembler.
- Adds a CSE pass to force-optimized so that LoadThreadInstr gets de-
duplicated.
TEST=tests/ffi(_2)/*
Change-Id: I67c6aaa1b7d6aa7d60e274477686a54e5a331216
Cq-Include-Trybots: luci.dart.try:vm-kernel-linux-debug-simriscv64-try,vm-kernel-precomp-linux-debug-simriscv64-try,vm-kernel-nnbd-mac-debug-arm64-try,vm-kernel-nnbd-mac-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-kernel-win-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/246240
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
(Macro-)assembler helper PushRegistersInOrder generates series of
register pushes in an architecture-specific way:
* Using STMDB on ARM (if possible).
* Using STP on ARM64.
* Updating SP only once on RISC-V.
TEST=ci
Change-Id: I6cbd1dd5f8acf8ce8087ed899a311264a873ad37
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/243100
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
The new implementation moves away from desugaring of async
functions on kernel AST, state machine generated in the flow graph and
capturing all local variables in the context.
Instead, async/await is implemented using a few stubs
(InitSuspendableFunction, Suspend, Resume, Return and
AsyncExceptionHandler). The stubs are implemented in a
platform-independent way using (macro-)assembler helpers.
When suspending a function, its frame is copied into a SuspendState
object, and when resuming a function it is copied back onto the stack.
No extra code is generated for accessing local variables.
Callback closures are created lazily on the first await.
Design doc: go/compact-async-await.
Part 1 (kernel): https://dart-review.googlesource.com/c/sdk/+/241842
TEST=ci
Issue: https://github.com/dart-lang/sdk/issues/48378
Change-Id: Ibad757035b7cc438ebdff80b460728b1d3eff1f5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/242000
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
1/2 or 1/4 branches executed per object initialization for uncompressed and compressed pointers respectively.
Note ARM32 already has double-wide initialization.
TEST=ci
Change-Id: Iacc61cac39f74191bb2e5e06cd48b90c006b8585
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/241622
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Ryan Macnak