Instead apply the same approach as we do in AOT: unbox based on the
static type information. There are no TFA results available in JIT,
but we could still unbox fields when running in sound null-safety.
TEST=ci
Cq-Include-Trybots: luci.dart.try:vm-kernel-reload-linux-release-x64-try,vm-kernel-reload-linux-debug-x64-try,vm-kernel-reload-rollback-linux-debug-x64-try,vm-kernel-reload-rollback-linux-release-x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-linux-product-x64-try,vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-nnbd-linux-release-simarm64-try,vm-kernel-linux-debug-simriscv64-try,vm-kernel-precomp-linux-debug-simriscv64-try,vm-kernel-nnbd-linux-release-ia32-try,vm-kernel-nnbd-linux-debug-x64-try,vm-kernel-nnbd-linux-debug-ia32-try,vm-kernel-nnbd-linux-release-simarm-try,vm-kernel-nnbd-linux-release-simarm64-try
Change-Id: Ide2e78c6659261ef8d245a4586cf699ea0fbb459
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/256211
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
This change introduces separate stubs for suspending sync* functions
at start and at yield/yield*. Suspend stub for yield/yield*
no longer calls Dart callback (in order to make it faster).
Also, ReturnSyncStar stub is removed - sync* functions now directly
return false instead of going through the stub.
TEST=ci
Issue: https://github.com/dart-lang/sdk/issues/48378
Change-Id: Iee9a1f48cab2812cf0f9f0e4e6d8e847547e49f7
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/250420
Reviewed-by: Slava Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
The new implementation is based on suspend/resume stubs and doesn't
use desugaring of async functions on kernel AST.
Previously, new implementation of async/async* was only supported in
AOT mode. This change adds all necessary bits for the JIT mode:
* Suspending variable-length frames (for unoptimized code).
* Handling of Code and pool pointers in Dart stack frames.
* OSR.
* Deoptimization.
* Hot reload.
* Debugger.
The new implementation is not enabled in JIT mode yet.
Design doc: go/compact-async-await.
TEST=ci
Issue: https://github.com/dart-lang/sdk/issues/48378
Change-Id: I477d6684bdce7cbc1edb179ae2271ff598b7dcc5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/246081
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
The new implementation moves away from desugaring of async
functions on kernel AST, state machine generated in the flow graph and
capturing all local variables in the context.
Instead, async/await is implemented using a few stubs
(InitSuspendableFunction, Suspend, Resume, Return and
AsyncExceptionHandler). The stubs are implemented in a
platform-independent way using (macro-)assembler helpers.
When suspending a function, its frame is copied into a SuspendState
object, and when resuming a function it is copied back onto the stack.
No extra code is generated for accessing local variables.
Callback closures are created lazily on the first await.
Design doc: go/compact-async-await.
Part 1 (kernel): https://dart-review.googlesource.com/c/sdk/+/241842
TEST=ci
Issue: https://github.com/dart-lang/sdk/issues/48378
Change-Id: Ibad757035b7cc438ebdff80b460728b1d3eff1f5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/242000
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
Right now all type literal usages will perform a runtime call which is
rather slow.
Flutter happens to use type literals such as `return T;` in hot code
which causes this to show up in the profile.
This CL adds fast paths for type parameter type literals if
the type parameter value (i.e. entry of TAV corresponding to T):
* is `null`: return `dynamic`
* is a non-FutureOr [Type] with compatible nullability: return value
* is [FunctionType] with compatible nullability: return value
otherwise fall back to runtime call.
It makes simple type literal uses 10x+ faster - the kinds that Flutter
is using.
Issue https://github.com/dart-lang/sdk/issues/48757
TEST=vm/dart{,_2}/instantiate_type_literal_test
Change-Id: I1139d6689aedbc68321f47ee6c9946a3323fbf6e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/241968
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
- Fixes SIMD registers clobbered by write barrier on ARM64.
- Moves code generation out of non-compiler directory.
- Removes unnecessary building of Dart frames on leaf runtime calls.
- Removes unnecessary Threads slots for write barrier Code objects.
- Removes duplicate saves of SP in leaf runtime calls on ARM64 and RISC-V.
- Avoids some redundant SP updates on RISC-V.
TEST=ci
Change-Id: Idb92127658edc90b320923ef3d882a7219a450ae
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/236842
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
- refactor designated initializers as they are c++20
- unwrap #if/#endif in macro arguments
- remove taking pointer on purely intrinsic function (setjmp_)
- use DART_WARN_UNUSED_RESULT which is msvc/clang friendly
- provide StringRAII copying constructor
Bug: https://github.com/dart-lang/sdk/issues/48544
TEST=ci,msvc build
Change-Id: I1049589c96dc0cdf49e1d31d320b6804bf6e4558
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/237929
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Alexander Aprelev <aam@google.com>
Implements a backend targeting RV32GC and RV64GC, based on Linux standardizing around GC. The assembler is written to make it easy to disable usage of C, but because the sizes of some instruction sequences are compile-time constants, an additional build configuration would need to be defined to make use of it.
The assembler and disassembler cover every RV32/64GC instruction. The simulator covers all instructions except accessing CSRs and the floating point state accessible through such, include accrued exceptions and dynamic rounding mode.
Quirks:
- RISC-V is a compare-and-branch architecture, but some existing "architecture-independent" parts of the Dart compiler assume a condition code architecture. To avoid rewriting these parts, we use a peephole in the assembler to map to compare-and-branch. See Assembler::BranchIf. Luckily nothing depended on taking multiple branches on the same condition code set.
- There are no hardware overflow checks, so we must use Hacker's Delight style software checks. Often these are very cheap: if the sign of one operand is known, a single branch is needed.
- The ranges of RISC-V branches and jumps are such that we use 3 levels of generation for forward branches, instead of the 2 levels of near and far branches used on ARM[64]. Nearly all code is handled by the first two levels with 20-bits of range, with enormous regex matchers triggering the third level that uses aupic+jalr to get 32-bits of range.
- For PC-relative calls in AOT, we always generate auipc+jalr pairs with 32-bits of range, so we never generate trampolines.
- Only a subset of registers are available in some compressed instructions, so we assign the most popular uses to these registers. In particular, THR, TMP[2], CODE and PP. This has the effect of assigning CODE and PP to volatile registers in the C calling convention, whereas they are assigned preserved registers on the other architectures. As on ARM64, PP is untagged; this is so short indices can be accessed with a compressed instruction.
- There are no push or pop instructions, so combining pushes and pops is preferred so we can update SP once.
- The C calling convention has a strongly aligned stack, but unlike on ARM64 we don't need to use an alternate stack pointer. The author ensured language was added to the RISC-V psABI making the OS responsible for realigning the stack pointer for signal handlers, allowing Dart to leave the stack pointer misaligned from the C calling convention's point of view until a foreign call.
- We don't bother with the link register tracking done on ARM[64]. Instead we make use of an alternate link register to avoid inline spilling in the write barrier.
Unimplemented:
- non-trivial FFI cases
- Compressed pointers - No intention to implement.
- Unboxed SIMD - We might make use of the V extension registers when the V extension is ratified.
- BigInt intrinsics
TEST=existing tests for IL level, new tests for assembler/disassembler/simulator
Bug: https://github.com/dart-lang/sdk/issues/38587
Bug: https://github.com/dart-lang/sdk/issues/48164
Change-Id: I991d1df4be5bf55efec5371b767b332d37dfa3e0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/217289
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
When ffi code calls Dart code that requests the isolate to exit(via Isolate.exit, for example), we ensure that isolate indeed exits on return from ffi call.
This is implemented by introduction of new safepoint bit which, once set, forces ExitSafepoint to propagate unwind error.
TEST=isolate_exit_sandwich_test
Change-Id: I2e8f5ecec7f4e59ae5f99b9525cc566f20d4b6a8
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/219846
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Alexander Aprelev <aam@google.com>
TSAN instruments C++ code by adding prologue/epilogue code which
maintains a shadow stack. Using setjmp()/longjmp() is intercepted by
TSAN and correspondingly unwinds the shadow stack.
When Dart VM throws exceptions we call the JumpToFrame stub
from C++ which will directly reset the stack to the exception handler
catch entry. This leaves the TSAN shadow stack unchanged.
This means whenever an exception is thrown we leak frames in TSAN's
shadow stack. Due to using a fixed-size shadow stack, it will cause a
buffer-overflow in TSAN when too many such frame leaks happen. This can
cause arbitrary memory to be overriden, leading to awkward crashes.
This is especially an issue on the "iso-stres" builder because it
launches - in the same process - *many* small tests, more easily hitting
that limit.
This CL will workaround the TSAN issue by making runtime call save it's
state via setjmp() and make exception throughing process go via
longjmp() (which TSAN will intercept) before actually calling the
JumpToFrame stub.
=> This will ensure the TSAN shadow stack is correctly maintained.
The [jmp_buf]'s encoding of register state is non-trivial (e.g. it uses
XOR'ing of the actual saved state under certain glibc versions). So we
store any state we need to pass to the target of the `longjmp()` on the
[Thread] instead of overriding the [jmp_buf]s register state with the
arguments.
Issue https://github.com/dart-lang/sdk/issues/47472#issuecomment-948235479
TEST=vm/dart{,_2}/regress47472_test.dart
Change-Id: Ifbf6580aa15bcce54d0584cdc3cd18cc19be0a9c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/222300
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
TSAN instruments C++ code by adding prologue/epilogue code which
maintains a shadow stack. Using setjmp()/longjmp() is intercepted by
TSAN and correspondingly unwinds the shadow stack.
When Dart VM throws exceptions we call the JumpToFrame stub
from C++ which will directly reset the stack to the exception handler
catch entry. This leaves the TSAN shadow stack unchanged.
This means whenever an exception is thrown we leak frames in TSAN's
shadow stack. Due to using a fixed-size shadow stack, it will cause a
buffer-overflow in TSAN when too many such frame leaks happen. This can
cause arbitrary memory to be overriden, leading to awkward crashes.
This is especially an issue on the "iso-stres" builder because it
launches - in the same process - *many* small tests, more easily hitting
that limit.
This CL will workaround the TSAN issue by making runtime call save it's
state via setjmp() and make exception throughing process go via
longjmp() (which TSAN will intercept) before actually calling the
JumpToFrame stub.
=> This will ensure the TSAN shadow stack is correctly maintained.
The [jmp_buf]'s encoding of register state is non-trivial (e.g. it uses
XOR'ing of the actual saved state under certain glibc versions). So we
store any state we need to pass to the target of the `longjmp()` on the
[Thread] instead of overriding the [jmp_buf]s register state with the
arguments.
Issue https://github.com/dart-lang/sdk/issues/47472#issuecomment-948235479
TEST=vm/dart{,_2}/regress47472_test.dart
Change-Id: I4b6f0d4eacef85487c55999021d72a6d932facad
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/221083
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
This changes Profiler::DumpStackTrace to include information available
in the CompilerState attached to the thread into the crash report.
This includes the name of the function being compiled, compilation mode,
current phase, flow graph, instruction being compiled.
TEST=manually by causing compiler to crash
Change-Id: I310629f032a86781e30f0a5e218d21df7d27528e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/220001
Auto-Submit: Slava Egorov <vegorov@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Tess Strickland <sstrickl@google.com>
1) double.truncate() now simply calls toInt(), omitting
truncateToDouble() call.
2) double.floor() and ceil() are now intrinsified on arm64 and AOT/x64
and generated using DoubleToInteger instruction.
3) DoubleToInteger instruction is extended to support floor and ceil.
On arm64 DoubleToInteger is implemented using fcvtms and fcvtps
instructions. On x64 DoubleToInteger is implemented using roundsd
under a check if it is supported (with a fallback to a stub and a
runtime call).
AOT/x64:
Before: BenchFloor(RunTime): 318.82148549569655 us.
After: BenchFloor(RunTime): 133.29430189936687 us.
TEST=ci
Closes https://github.com/dart-lang/sdk/issues/46876
Closes https://github.com/dart-lang/sdk/issues/46650
Change-Id: I16ca18faf97954f8e8e25f0b72a2bbfac5bdc672
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/212381
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
In the unoptimized code unboxed double inputs are now unboxed in the
instruction prologue, and unboxed double output is boxed in the
instruction epilogue. Double values are still always boxed on the
expression stack in the unoptimized mode.
Support for unboxed doubles in unoptimized code allows us to have
one implementation of double and dart:math built-in functions,
shared across optimized and unoptimized modes.
TEST=ci
Issue: https://github.com/dart-lang/sdk/issues/46650
Change-Id: Ic1e42b72ee80cb2c12019b7abab8111240b05efd
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/211302
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
This relands https://dart-review.googlesource.com/c/sdk/+/205633
but without renaming TARGET_OS_IPHONE to DART_TARGET_OS_IPHONE.
It also changes uses of TARGET_OS_IOS to
DART_TARGET_OS_MACOS_IOS to be consistent with the rest of the
VM.
TargetConditionals.h for XCode 13 defines several
TARGET_OS_* preprocessor symbols that confuse the
Dart build. There is probably a more targeted fix
for this, but renaming the symbols that Dart uses
will also prevent this problem if more symbols
are added to the platform headers in the future.
See: https://github.com/dart-lang/sdk/issues/46499
TEST=It builds.
Change-Id: Ie775c19dd23cfdf5f65e5ebc6ee4ec3a561676fa
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/205860
Commit-Queue: Zach Anderson <zra@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
TargetConditionals.h for XCode 13 defines several
TARGET_OS_* preprocessor symbols that confuse the
Dart build. There is probably a more targeted fix
for this, but renaming the symbols that Dart uses
will also prevent this problem if more symbols
are added to the platform headers in the future.
See: https://github.com/dart-lang/sdk/issues/46499
TEST=It builds.
Change-Id: I3b33a03b4a9a14b76d55fe12f8cdefec4b3c3664
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/205633
Commit-Queue: Zach Anderson <zra@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
This extends the existing safepoint operation mechanism by allowing to
perform two different operations:
* "gc safepoint operations": All mutators are stopped at places where
it's safe to GC. It therefore requires stackmaps to be available for
all optimized mutator frames.
* "deopt safepoint operations": All mutators are stopped at places
where it's safe to GC, but also safe to lazy-deopt mutator frames.
It therefore requires deopt-id/deopt-info to be available for all
optimized mutator frames.
Mutators can be asked to block for any of those two safepoint operations.
If a mutator is at a place where its safe to GC it will respond to "gc
safepoint operations" requests, if a mutator is additionally at a place
where it's also safe to lazy-deopt it will respond to "deopt safepoint
operation" requests.
Depending on how the runtime was entered (which is tracked via the
[Thread::runtime_call_deopt_ability_] value) - the mutator might
participate in both or only in gc safepoint operations.
During the start of a "deopt safepoint operation", the safepoint handler
will request all threads to stop at a "deopt safepoint". Some threads
might first want to initiate their own "gc safepoint operation"
(e.g. due to allocation failure) before they reach a "deopt safepoint".
We do allow this by letting the safepoint handler own a "deopt safepoint
operation" but still participate in other thread's "gc safepoint
operation" requests until all mutators are checked into places where
it's safe to lazy-deopt at which point the "deopt safepoint operation"
also owns a "gc safepoint operation".
In order to facilitate this, the Thread's safepoint_state will be
extended to consist of the following bits:
* AtSafepoint
* SafepointRequested
* AtDeoptSafepoint
* DeoptSafepointRequested
* BlockedForSafepoint
Issue https://github.com/dart-lang/sdk/issues/45213
TEST=vm/cc/SafepointOperation_*
Change-Id: Icdc2827718f6780818f99b829a5e806d6bb5b130
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/196927
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
As a pre-requisite of partitioning our safepoints into "gc only
safepoints" and "lazy-deopt'able safepoints" we'll let mutators keep
track of whether a runtime call into the VM is deoptable.
A mutator is lazy-deopt'able except for one particular case: When the
mutator calls into the VM from a place where no deopt-id/deopt-env
gets recorded (e.g. StoreInstanceField's box allocation goes to
runtime).
If a mutator is in the VM in a non-lazy-deopt'able state, it cannot call
out to the embedder, transition from "vm" to "native" state, ...
A future change will make new non-lazy-deopt'able runtime entries taking
advantage of this CL.
Issue https://github.com/dart-lang/sdk/issues/45213
TEST=Existing test suite & tests in future CLs.
Change-Id: I30dbee02dc839b2ba9632b20ffa233aff4d58f64
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/196922
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Attempts to acquire safepoint lock while in SafepointOperationScope could result
in deadlocks if somebody else was holding that lock when they were forced to a safepoint.
Clean up few places where locks were acquired in SafepointOperationScope.
Introduce StoppedMutatorsScope and GroupDebugger::RunUnderReadLockIfNeeded that acquires
a lock only if it runs outside of StoppedMutatorsScope - to prevent such deadlocks.
Also fix tsan warning about data race around message_notify_callback by making it atomic.
TEST=tsan runs of debugger CI tests
Fixes https://github.com/dart-lang/sdk/issues/45527
Fixes https://github.com/dart-lang/sdk/issues/45549
Issue https://github.com/dart-lang/sdk/issues/36097
Change-Id: Ibb53d9ce760d869e044e17075aeebf20fc0016a5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/193582
Commit-Queue: Alexander Aprelev <aam@google.com>
Reviewed-by: Ben Konyi <bkonyi@google.com>
Vyacheslav Egorov