The former contents of the VM isolate are now included into each isolate group. This makes each isolate group's heap independent, and in particular allows each heap to be allocated to a separate pointer cage (not done in this CL).
The duplicated stubs that allowed PC relative calls are removed, since the originals can now be the target of PC relative calls.
The bootstrapping needing to load an AppJIT or AppAOT snapshot is reduced to allocating the oddballs. The code is entirely dropped in the AOT runtime, but the JIT runtime still has it to allow for flags to affect the compilation of the stub code. Further refactoring might be able to remove this for the JIT runtime too, with only gen_snapshot knowing how to bootstrap.
Class serialization no longer distinguishes predefined classes.
The page containing null is marked as never-evacuate. null, false and true must not move because the compiler relies on their low bits having certain patterns for some optimizations. (Previously, the entire VM isolate heap never moved.)
Compaction is disabled for IA32. Due to register pressure, some stub calls must not use a scratch register and embed the address of Code.
The page containing the call-through-safepoint stub is frozen when running with --write-protect-code and the stub is created at runtime (instead of loaded from an AppJIT or AppAOT snapshot). This stub must remain executable even during a safepoint, as a foreign call might during return during a safepoint and only block after the stub directs it to the runtime.
The snapshot symbols are renamed to kDartSnapshotData and kDartSnapshotText. There is no need to distinguish the VM isolate's snapshot, and snaphots are per isolate group not per isolate. Aliases with the old names are added to ease migration.
Some global flags that were automatically set based on the VM isolate's snapshot are now isolate group flags and automatically set by the isolate group's snapshot.
TEST=ci
Change-Id: Iee82016057d609112e9b021d178fc3d4d18b5044
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/500621
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Tess Strickland <sstrickl@google.com>
SLSA-Policy-Verified: SLSA Policy Verification Service <devtools-gerritcodereview-exitgate@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Freelist implementation was trying to mark pages executable (RX)
which does not respect possibility that it is working with RW mapping
which can't be directly marked as executable. It needed to use RO
permission instead if dual mapping is enabled.
This CL consolidates all code mentioning RX permission in one place:
VirtualMemory::WriteProtectCode so that we avoid this mistake in the future.
R=kustermann@google.com
TEST=tested by doing large reloads which cause more churn in code space
Change-Id: If99692cac3ce3ff54b907e4c43d7f26bae7439ff
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/435860
Commit-Queue: Martin Kustermann <kustermann@google.com>
Auto-Submit: Slava Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Heap references always point into R/RW mapping (this simplifies the
marker) and entry points are retargeted into RX.
To simplify implementation we assume that there are no image pages
created - which means we can always go from Instructions to the start of
the page and check where it is dual mapped to.
TEST=vm/dart/macos_dual_mapping_smoke_test and manually on physical
device
Change-Id: Idbe02b7b695b3c048072cf92f7d062f5ab6e1beb
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/433940
Reviewed-by: Siva Annamalai <asiva@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
This pragma forces compiler to align loop headers within the
function by architecture specific boundary: 32 bytes on X64
and ARM64 (with the exception of Apple Silicon, which explicitly
discourages aligning branch targets in the optimization manual).
Current implementation is rather naive and does not do any
attempt to decide whether aligning is actually profitable
based on loop body itself.
I have found this pragma to be helpful both to stabilize
benchmark results and achieve better performance
for tight loops on Intel hardware.
Issue https://github.com/dart-lang/sdk/issues/55522
TEST=vm/dart/align_loops_test
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-product-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-linux-debug-x64-try,vm-aot-linux-release-arm64-try,vm-aot-linux-release-simarm_x64-try,vm-aot-linux-release-x64-try,vm-aot-mac-product-arm64-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-release-x64-try,vm-aot-win-debug-arm64-try,vm-aot-win-debug-x64-try,vm-aot-win-debug-x64c-try,vm-aot-win-product-x64-try,vm-aot-win-release-arm64-try,vm-aot-linux-debug-simarm_x64-try,vm-aot-linux-debug-simriscv64-try,vm-aot-dwarf-linux-product-x64-try,vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try
Change-Id: Ic22fb90d85e7fdebeeaa3908a43328c59436ab58
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/364121
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
When uploading CLs, the presubmit checks verify that the lines in the
diff are formatted correctly according to `git cl format runtime`.
However, when `buildtools/<os>-<arch>/clang/bin/clang-format` is
updated, it does not force reformatting of files that would be
reformatted.
This leads to two issues:
* Inconsistent style within the code base and within a single file.
* Spurious reformatting in CLs when (1) clang-format is used on the
whole file, or (2) the diff lines overlap.
`clang-format` doesn't change that frequently, so in general this is
not a large issue, but I've seen a bit too many "spurious formatting,
please revert" comments on CLs recently.
This CL formats the runtime to be in line with the current pinned
`clang-format`:
```
$ find runtime/ -iname *.h -o -iname *.cc | xargs buildtools/mac-arm64/clang/bin/clang-format -i
```
`git cl format` (which only formats changed lines, and does so with
`clang-format`) seems to not agree with itself, or clang-format, or
cpplint in a handful of places. This CL adds `// clang-format off`
for these. (See previous patchsets for the specific instances.)
TEST=A variety of bots including GCC, MacOS and Windows.
Change-Id: I470892e898971899fda14bb3b8f2c8efefd67686
Cq-Include-Trybots: luci.dart.try:vm-gcc-linux-try,vm-ffi-qemu-linux-release-riscv64-try,vm-ffi-qemu-linux-release-arm-try,vm-aot-win-debug-x64-try,vm-win-debug-x64c-try,vm-mac-debug-x64-try,vm-mac-debug-arm64-try,vm-aot-linux-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/362780
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
We are no longer using Dart VM in a setting where this
matters as a security measure and it just complicates
portability for no benefit.
There is an indication that it is causing problems when
running Linux build of Dart VM under Docker on Mac OS X.
Fixes https://github.com/dart-lang/sdk/issues/54446
TEST=ci
Cq-Include-Trybots: luci.dart.try:vm-fuchsia-release-arm64-try,vm-fuchsia-release-x64-try
Change-Id: I11bdaa8faebaca1df6fd59097049bdaea9cb8e12
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/344581
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
There is no need to adjust instruction address by Page::AliasOffset()
when copying them to the image, as instructions are still readable
even if dual-mapped. The adjustment can be harmfull if Code is not
properly finalized and instructions in the Code are not replaced
with executable version.
TEST=ci
Fixes https://github.com/dart-lang/sdk/issues/51525
Cq-Include-Trybots: luci.dart.try:vm-linux-debug-x64c-try,vm-win-debug-x64c-try
Change-Id: I4af0ded149f27f6a68d2962805748d1eef6625e3
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/286022
Auto-Submit: Alexander Markov <alexmarkov@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
- Prep work for non-moving promotion.
- The generations still have different alignment offsets.
- Old-space pages now participate in the page cache.
- Fix boolean/null bit tricks to assert the right requirements on the alignment of a page's first object.
TEST=ci
Change-Id: I4369d8c6af73228e162c226d411914868bafed33
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/260401
Reviewed-by: Siva Annamalai <asiva@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Implements a backend targeting RV32GC and RV64GC, based on Linux standardizing around GC. The assembler is written to make it easy to disable usage of C, but because the sizes of some instruction sequences are compile-time constants, an additional build configuration would need to be defined to make use of it.
The assembler and disassembler cover every RV32/64GC instruction. The simulator covers all instructions except accessing CSRs and the floating point state accessible through such, include accrued exceptions and dynamic rounding mode.
Quirks:
- RISC-V is a compare-and-branch architecture, but some existing "architecture-independent" parts of the Dart compiler assume a condition code architecture. To avoid rewriting these parts, we use a peephole in the assembler to map to compare-and-branch. See Assembler::BranchIf. Luckily nothing depended on taking multiple branches on the same condition code set.
- There are no hardware overflow checks, so we must use Hacker's Delight style software checks. Often these are very cheap: if the sign of one operand is known, a single branch is needed.
- The ranges of RISC-V branches and jumps are such that we use 3 levels of generation for forward branches, instead of the 2 levels of near and far branches used on ARM[64]. Nearly all code is handled by the first two levels with 20-bits of range, with enormous regex matchers triggering the third level that uses aupic+jalr to get 32-bits of range.
- For PC-relative calls in AOT, we always generate auipc+jalr pairs with 32-bits of range, so we never generate trampolines.
- Only a subset of registers are available in some compressed instructions, so we assign the most popular uses to these registers. In particular, THR, TMP[2], CODE and PP. This has the effect of assigning CODE and PP to volatile registers in the C calling convention, whereas they are assigned preserved registers on the other architectures. As on ARM64, PP is untagged; this is so short indices can be accessed with a compressed instruction.
- There are no push or pop instructions, so combining pushes and pops is preferred so we can update SP once.
- The C calling convention has a strongly aligned stack, but unlike on ARM64 we don't need to use an alternate stack pointer. The author ensured language was added to the RISC-V psABI making the OS responsible for realigning the stack pointer for signal handlers, allowing Dart to leave the stack pointer misaligned from the C calling convention's point of view until a foreign call.
- We don't bother with the link register tracking done on ARM[64]. Instead we make use of an alternate link register to avoid inline spilling in the write barrier.
Unimplemented:
- non-trivial FFI cases
- Compressed pointers - No intention to implement.
- Unboxed SIMD - We might make use of the V extension registers when the V extension is ratified.
- BigInt intrinsics
TEST=existing tests for IL level, new tests for assembler/disassembler/simulator
Bug: https://github.com/dart-lang/sdk/issues/38587
Bug: https://github.com/dart-lang/sdk/issues/48164
Change-Id: I991d1df4be5bf55efec5371b767b332d37dfa3e0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/217289
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Every AOT user out there has been using bare instructions mode
and continuing to maintaining non-bare instructions mode simply
adds costs (both in terms of time spent making changes to work
in a mode that is not used and CI resources spent on testing it).
This change removes FLAG_use_bare_instructions and changes the code
to assume that FLAG_precompiled_mode implies bare instructions.
TEST=ci
Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-linux-debug-x64c-try,vm-kernel-precomp-linux-product-x64-try,vm-kernel-precomp-dwarf-linux-product-x64-try,vm-kernel-precomp-obfuscate-linux-release-x64-try,app-kernel-linux-release-x64-try,app-kernel-linux-debug-x64-try
Change-Id: I5032b13bfcb613f79865f2cfa139cca8d1b42556
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/220964
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
This extends the existing safepoint operation mechanism by allowing to
perform two different operations:
* "gc safepoint operations": All mutators are stopped at places where
it's safe to GC. It therefore requires stackmaps to be available for
all optimized mutator frames.
* "deopt safepoint operations": All mutators are stopped at places
where it's safe to GC, but also safe to lazy-deopt mutator frames.
It therefore requires deopt-id/deopt-info to be available for all
optimized mutator frames.
Mutators can be asked to block for any of those two safepoint operations.
If a mutator is at a place where its safe to GC it will respond to "gc
safepoint operations" requests, if a mutator is additionally at a place
where it's also safe to lazy-deopt it will respond to "deopt safepoint
operation" requests.
Depending on how the runtime was entered (which is tracked via the
[Thread::runtime_call_deopt_ability_] value) - the mutator might
participate in both or only in gc safepoint operations.
During the start of a "deopt safepoint operation", the safepoint handler
will request all threads to stop at a "deopt safepoint". Some threads
might first want to initiate their own "gc safepoint operation"
(e.g. due to allocation failure) before they reach a "deopt safepoint".
We do allow this by letting the safepoint handler own a "deopt safepoint
operation" but still participate in other thread's "gc safepoint
operation" requests until all mutators are checked into places where
it's safe to lazy-deopt at which point the "deopt safepoint operation"
also owns a "gc safepoint operation".
In order to facilitate this, the Thread's safepoint_state will be
extended to consist of the following bits:
* AtSafepoint
* SafepointRequested
* AtDeoptSafepoint
* DeoptSafepointRequested
* BlockedForSafepoint
Issue https://github.com/dart-lang/sdk/issues/45213
TEST=vm/cc/SafepointOperation_*
Change-Id: Icdc2827718f6780818f99b829a5e806d6bb5b130
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/196927
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Right now the vm/cc/CodeRelocator_* tests are written in a strange way
to work around the conservative assumptions the relocator makes.
A future CL will make the relocator insert tramppolines precisely when
they are needed (not conservatively), which will allow writing precise
tests.
TEST=Fixes ARM64 vm/cc/CodeRelocator_* tests
Change-Id: I68f9bcce29f42a2264608f867f177e8f557804fb
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/196666
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
The tests generate & run relocated instructions. On ARM64 our macro
assembler uses X15 as stack pointer instead of the normal sp.
-> There was a missing RestoreCSP() in the generation of the call
instructions.
-> It's not an issue for running the test in simulator if CSP is wrong
when returning, though it is for HW.
Adding RestoreCSP() would require the code to be larger, so instead we
use lower level assembly instructions that operate on the normal ARM64
stack.
Closes https://github.com/dart-lang/sdk/issues/45807
TEST=Fixes vm/cc/RelocatinoTest_* on arm64 hw
Change-Id: I0c65e3d22897f536e39f0cf7e9af2be871051b0d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/196498
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
There was a missing check for whether a backwards call is out-of-range.
The CL also removes leftover code related to [max_offset_into_target_] -
which wasn't used. The intended use is already over approximated via
[max_instructions_size_].
The CL makes the pc-relative call/tail-call distances pluggable so a
newly added test can modify them for testing in-range/out-of-range
forward/backwards calls.
TEST=vm/cc/CodeRelocator_*
Fixes https://github.com/flutter/flutter/issues/80043
Change-Id: Id4bdb7176108b61235dafb7ffc125da4a2bf07fa
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/195682
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Ryan Macnak