It incorrectly assumed that all stores in Dart code write to Instances.
There is actually one exception, Contexts, which do not inherit from Instance.
I've added asserts to ensure this kind of bug cannot resurface.
The original change is in patchset 4.
Change-Id: Ic2d8d05e70a4de738eb9fb5980487b4f27111b8c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/136221
Commit-Queue: Samir Jindel <sjindel@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
This CL:
* Moves [Heap]/[SharedClassTable] from [Isolate] to [IsolateGroup], which
will make all isolates in the group use the same heap. The GC will use
the shared class table for object size information.
* Adds support for entering/leaving an isolate group as a helper thread
(e.g. via [Thread::EnterIsolateGroupAsHelper]). The current active
isolate group can be accessed via TLS `IsolateGroup::Current()` or
`Thread::isolate_group_`. When entering as a helper thread there will be
no current isolate.
* Changes the GC to use the above mechanism and ensures GC works without
a currently active isolate. The GC will use information purely available via
[IsolateGroup]. The GC will iterate all isolates within an isolate
group e.g. for scanning roots.
* Makes spawning of new isolates start in their own isolate group.
Once the isolate is fully functional it's heap will be merged into
the original isolate group
* Moves ApiState, containing persistent and weak persistent handles,
from [Isolate] to [IsolateGroup], plus adds appropriate locking.
Issue https://github.com/dart-lang/sdk/issues/36097
Change-Id: Ia8e1d8aa78750e8400864200f4825395a182c004
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/126646
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
This scope can sneak objects larger than kAllocatablePageSize directly into non-large heap pages.
This scope also holds a heap lock outside of the GC, which is likely to create issues when multiple isolates run on the same heap.
Change-Id: Ic56277c83ffccfe2e1396ed773dd30b8d172bbba
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/134040
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
This is a Dart-tailored implementation of the "row-displacement
dispatch table" technique for closed-world instance calls:
All interface targets in the program are grouped into selectors such
that all targets that could potentially be called from the same call
site have the same selector (currently just grouped by name).
Each selector is assigned a selector offset such that offset + classid
is unique for all selector/classid combinations where the class
implements the selector.
At every instance call site that has an interface target (i.e. where
the static type of the receiver is not dynamic), the selector offset +
receiver classid is computed and used as index into a global table of
entry points.
If the receiver can be null (as determined by the front-end TFA and the
VM type propagation), a null check is inserted before the call.
An arguments descriptor is provided (only) for selectors that need it
(those which have type parameters or optional/named parameters).
The dispatch table calls don't need the monomorphic entry code, so for
functions that are only called via dispatch table calls (i.e. never
called dynamically), the monomorphic entry code is left out.
Some future improvements to the table dispatch implementation are
mentioned in https://github.com/dart-lang/sdk/issues/40188
The table dispatch flag is disabled by default in this commit. A
separate commit enables the flag.
Change-Id: Ic2911742b4a2c9a8d3bc7df60605454cbe4c0714
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/126648
Commit-Queue: Aske Simon Christensen <askesc@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
This CL is responsible for inlining Float32x4 and Float64x2 arithmetic
operations in order to reduce their run-time cost.
Null-checks were required in order to allow the unbox instructions to
be non-speculative.
CheckNullInstr was modified to throw ArgumentError when checking
arguments for those operations.
Change-Id: I18b36d96a36011647480f301eb6dc6bb478f59a1
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/128589
Commit-Queue: Victor Agnez Lima <victoragnez@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Samir Jindel <sjindel@google.com>
Fuchsia is about to turn on ShadowCallStack for ARM64. Once this is enabled, we need to treat R18 like a preserved register. Generated Dart has not accessed this register because it is reserved on iOS, and in the absence of Dart exceptions this would be sufficient for us to be ShadowCallStack compatible. However, our exception handling mechanism jumps past all the C++ frames between the Dart exit frame and Exceptions::JumpToFrame, skipping code that would pop from R18.
Add save/restore of R18 in the invocation stubs, and restore of R18 in the jump stub. The latter prevents the ShadowCallStack from overflowing for code that has lots of exceptions without a native call.
Bug: https://bugs.fuchsia.dev/p/fuchsia/issues/detail?id=37449
Change-Id: I2ce6e46624c8d72507e7afa7a44839b1f0def556
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/119481
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Fixes the changes to assembly snapshot writer in SIMARM_X64 mode. This unfortunately can't be tested until the ELF loader is available.
Take 2 is in patchset 1.
Change-Id: Ib8b067dd4f09dcba3b142705e9cb4dfbf4e3eb53
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/117726
Commit-Queue: Samir Jindel <sjindel@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
The original change is in Patchset 1.
The dependency contains the fix for the non-bare-instructions bot.
Change-Id: Iab02ab0736a2ee9144b667d2777ca526341b2a36
Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-linux-product-x64-try,vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-android-release-arm-try,vm-dartkb-linux-release-x64-abi-try,vm-kernel-precomp-bare-linux-release-x64-try,vm-kernel-precomp-mac-debug-simarm_x64-try,vm-kernel-precomp-win-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/115241
Commit-Queue: Samir Jindel <sjindel@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
This reverts commit ecbea5a58e.
Reason for revert: broken with bare instructions and ABI bot
Original change's description:
> [vm/ffi] Implement FFI callbacks on AOT for ELF and Asm snapshots (excl. blobs).
>
> To do this, we add writable data sections (currently uninitialzed) to ELF and Asm snapshots
> and allow Instructions to have patchable relocations against (the start of) these sections.
>
> Issue https://github.com/dart-lang/sdk/issues/37295 (see also for design & discussion).
>
> Change-Id: If20bfa55776f4044aaa6bb8ea2101d2ada41842c
> Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-linux-product-x64-try,vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-android-release-arm-try
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/110221
> Commit-Queue: Samir Jindel <sjindel@google.com>
> Reviewed-by: Martin Kustermann <kustermann@google.com>
TBR=kustermann@google.com,rmacnak@google.com,alexmarkov@google.com,sjindel@google.com
Change-Id: I9787da6d42575ca4f5ae0a698052a19ac4275afd
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-linux-debug-x64-try, vm-kernel-precomp-linux-product-x64-try, vm-kernel-precomp-linux-release-x64-try, vm-kernel-precomp-android-release-arm-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/115240
Reviewed-by: Samir Jindel <sjindel@google.com>
Commit-Queue: Samir Jindel <sjindel@google.com>
To do this, we add writable data sections (currently uninitialzed) to ELF and Asm snapshots
and allow Instructions to have patchable relocations against (the start of) these sections.
Issue https://github.com/dart-lang/sdk/issues/37295 (see also for design & discussion).
Change-Id: If20bfa55776f4044aaa6bb8ea2101d2ada41842c
Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-linux-product-x64-try,vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-android-release-arm-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/110221
Commit-Queue: Samir Jindel <sjindel@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
The original revision is in patchset 1.
Three bugs are fixed:
1. Fix SIMARM_X64 build: no need to generate trampolines for AOT or simulated JIT.
2. Hot-reload: Fix hot-reload: don't invalidate Code for force-optimized functions.
3. Windows: Provide shadow space to runtime routines.
Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-mac-debug-simarm_x64-try,vm-kernel-precomp-mac-release-simarm_x64-try,vm-kernel-reload-linux-debug-x64-try,vm-kernel-reload-rollback-linux-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-kernel-win-debug-x64-try,vm-kernel-win-release-ia32-try,vm-kernel-win-release-x64-try
Change-Id: I326009cfacb51a84e9de4ddf9ff2d6d415460f91
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/113829
Commit-Queue: Samir Jindel <sjindel@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
This CL moves the thread registry and the safepoint handler to the
[IsolateGroup]. This will cause all threads belonging to the isolate
group to safepoint together.
=> We will therefore start to get an idea of the impact this will have on
pause times.
So far it was only possible to enter a particular isolate (e.g. as mutator
thread or auxiliary thread). This CL adds support for entering an isolate
group (instead of a particular isolate) as an auxiliarly thread. The
current isolate group is available via `IsolateGroup::Current()`.
=> This is a preparation step to let GC threads enter the isolate
group as auxiliary threads, not associated with a particular isolate but
to an isolate group as a whole.
Issue https://github.com/dart-lang/sdk/issues/36097
Change-Id: I7069d07130938d370869f02060570143bfeb1b48
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/108801
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
The thread registry is creating [Thread] objects when threads enter an
isolate (as mutator or helper). Once threads exit an isolate, the
[Thread] structure is returned and the thread registry will put it
into a cache for later re-use.
The mutator [Thread] object is an exception: Exiting and entering the
isolate as mutator will always use the same cached [Thread] object.
We want to eventually use one thread registry for an entire isolate
group. There will therefore be multiple mutator threads per thread registry.
We therefore move the cached mutator thread from the thread registry to the
[Isolate] object.
Issue https://github.com/dart-lang/sdk/issues/36097
Change-Id: Id27dff886d79ca76f6e05320151aeb72c8ba5140
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/108720
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
dart-bytecode, arm64: +4.742% geomean
dart-bytecode-jit-unopt, arm64: +12.73% geomean
dart2js-compile, x64: +3.635% geomean
In the polymorphic and unlinked cases, call to a stub the does a linear scan against an ICData.
In the monomorphic case, call to a prologue of the expected target function that checks the expected receiver class. There is additional indirection in the JIT version compared to the AOT version to also tick a usage counter so the inliner can make good decisions.
In the megamorphic case, call to a stub that does a hash table lookup against a MegamorphicCache.
Megamorphic call sites face a loss of precision in usage counts. The call site count is not recorded and the usage counter of the target function is used as an approximation.
Monomorphic and megamorphic calls sites are reset to the polymorphic/unlinked state on hot reload.
Monomorphic and megamorphic calls sites do not check the stepping state, so they are reset to the polymorphic/unlinked state when stepping begins and disabled.
Back-edges now increment the usage counter in addition to checking it. This ensures function with loops containing monomorphic calls will eventually cross the optimization threshold.
Fixed backwards use of kMonomorphicEntryOffset and kPolymorphicEntryOffset.
Fixed C stack overflow when bouncing between the KBC interpreter and a simulator.
Bug: https://github.com/dart-lang/sdk/issues/26780
Bug: https://github.com/dart-lang/sdk/issues/36409
Bug: https://github.com/dart-lang/sdk/issues/36731
Change-Id: I78a49cccd962703a459288e71ce246ed845df474
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/102820
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
In many cases, the Mutexes and Monitors have to be marked "mutable"
because they're used to synchronize const accessor methods.
Small text segment improvement for Product builds:
$ size dart.{arm,x64}.{before,after}
text data bss dec hex filename
19726069 409960 392332 20528361 1393ce9 dart.arm.before
19725525 409960 392332 20527817 1393ac9 dart.arm.after
22576021 600376 1782824 24959221 17cd8f5 dart.x64.before
22574821 600376 1782824 24958021 17cd445 dart.x64.after
Change-Id: I68f5cd5ad452044df8bfebd160910496036a3e6b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/101745
Commit-Queue: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
When the compactor moves the global object pool the pointer in Thread
was not updated because it was in a macro list which can only hold
objects inside the read-only VM isolate.
All calls into dart need to populate PP as well as all calls returning
from C++ need to restore PP. There was a missing case where we return
from natives and don't restore PP (i.e. not normal runtime calls).
This started to be an issue after we started moving old space objects.
Cq-Include-Trybots: luci.dart.try:vm-canary-linux-debug-try, vm-dartkb-linux-debug-x64-try, vm-dartkb-linux-release-x64-try, vm-kernel-asan-linux-release-x64-try, vm-kernel-checked-linux-release-x64-try, vm-kernel-linux-debug-ia32-try, vm-kernel-linux-debug-simdbc64-try, vm-kernel-linux-debug-x64-try, vm-kernel-linux-product-x64-try, vm-kernel-linux-release-ia32-try, vm-kernel-linux-release-simarm-try, vm-kernel-linux-release-simarm64-try, vm-kernel-linux-release-simdbc64-try, vm-kernel-linux-release-x64-try, vm-kernel-optcounter-threshold-linux-release-ia32-try, vm-kernel-optcounter-threshold-linux-release-x64-try, vm-kernel-precomp-android-release-arm-try, vm-kernel-precomp-bare-linux-release-simarm-try, vm-kernel-precomp-bare-linux-release-simarm64-try, vm-kernel-precomp-bare-linux-release-x64-try, vm-kernel-precomp-linux-debug-x64-try, vm-kernel-precomp-linux-product-x64-try, vm-kernel-precomp-linux-release-simarm-try, vm-kernel-precomp-linux-release-simarm64-try, vm-kernel-precomp-linux-release-x64-try, vm-kernel-precomp-obfuscate-linux-release-x64-try, vm-kernel-precomp-win-release-simarm64-try, vm-kernel-precomp-win-release-x64-try, vm-kernel-reload-linux-debug-x64-try, vm-kernel-reload-linux-release-x64-try, vm-kernel-reload-rollback-linux-debug-x64-try, vm-kernel-reload-rollback-linux-release-x64-try, vm-kernel-win-debug-ia32-try, vm-kernel-win-debug-x64-try, vm-kernel-win-product-x64-try, vm-kernel-win-release-ia32-try, vm-kernel-win-release-x64-try
Change-Id: I14776be226431e2c53ae888b3c98a1b8c540c4ec
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/98343
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
This reverts commit 324718fadd as it fixes performance issue found on Flutter benchmark.
The fix is to make TLAB smaller than new heap semi-space. Original implementation had TLAB occupy whole semi-space.
Change-Id: I4b5b5a3027b2d352ccb80538079a042406b5cf54
Reviewed-on: https://dart-review.googlesource.com/c/91171
Commit-Queue: Alexander Aprelev <aam@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
This is the next step towards preventing compiler from directly peeking
into runtime and instead interact with runtime through a well defined
surface. The goal of the refactoring to locate all places where compiler
accesses some runtime information and partion those accesses into two
categories:
- creating objects in the host runtime (e.g. allocating strings, numbers, etc)
during compilation;
- accessing properties of the target runtime (e.g. offsets of fields) to
embed those into the generated code;
This change introduces dart::compiler and dart::compiler::target namespaces.
All code in the compiler will gradually be moved into dart::compiler namespace.
One of the motivations for this change is to be able to prevent access to
globally defined host constants like kWordSize by shadowing them in the
dart::compiler namespace.
The nested namespace dart::compiler::target hosts all information about
target runtime that compiler could access, e.g. compiler::target::kWordSize
defines word size of the target which will eventually be made different
from the host kWordSize (defined by dart::kWordSize).
The API for compiler to runtime interaction is placed into compiler_api.h.
Note that we still permit runtime to access compiler internals directly -
this is not going to be decoupled as part of this work.
Issue https://github.com/dart-lang/sdk/issues/31709
Change-Id: If4396d295879391becfa6c38d4802bbff81f5b20
Reviewed-on: https://dart-review.googlesource.com/c/90242
Commit-Queue: Vyacheslav Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
ARM write barrier goes from
ldrne lr, [thr, #+508]
blxne lr
to
blne <offset>
ARM64 write barrier goes from (similarly X64)
beq +8
ldrx lr, [thr, #1056]
blr lr
to
beq +4
bl <offset>
It reduces RX on arm/arm64 by around 0.9%
Though the write barrier wrappers stub has multiple entrypoints (one for
each available register). Because of this, we modify the relocation
logic to support per-call offsets into the target.
To avoid making the assembler code depend on StubCode/FlowGraphCompiler,
we set a closure, which the assembler can call.
Issue https://github.com/dart-lang/sdk/issues/33274
Change-Id: I9e3d68260cab7ef19ea88f1235c78d6031819d6d
Reviewed-on: https://dart-review.googlesource.com/c/90063
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
This reverts https://github.com/dart-lang/sdk/commit/e9d358921dddbee84c39c951220d5a844b097bb5 and on top of original change it:
- switches from FreeList used as a filler to ForwardingCorpse because it is not as strict as to be expected only in old space;
- gets Verifier to accept ForwardingCorpse(as well as FreeList) in new space;
- asserts that thread is at safepoint(or is busy with one of GC tasks);
- has slow-path TryAllocateNewTLAB in scavenger.cc, rather than in .h;
- uses "inline" space_lock Mutex instead of allocating one on the heap;
- reverts back to less precise used space calculation to avoid requiring threads to be at a safepoint for calculation;
- adds HeapIterationScope in two places(heap_test.cc and service.cc) to accommodate newly introduced ASSERTs.
Change-Id: I42df716cff6da9651ce7737fa66d7ec2a55905b0
Reviewed-on: https://dart-review.googlesource.com/c/89041
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Alexander Aprelev <aam@google.com>
- Introduce a slimmed down version of thread.h, which just depends on the
Zone and StackResource.
- Introduce a layering check that would prevent the coupling in the future.
This is the first step towards decoupling compiler from runtime.
There are multiple reasons to introduce the decoupling but the main
reason currently is to introduce a controlled surface through which
compiler reaches into runtime to catch any places where runtime word size
might influence the compiler and then enable building compiler that
targets 32-bit runtime but is embedded into a 64-bit runtime.
Issue https://github.com/dart-lang/sdk/issues/31709
Change-Id: Id63ebbaddca55dd097298e51c90d957a73fa476e
Reviewed-on: https://dart-review.googlesource.com/c/87182
Commit-Queue: Vyacheslav Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Samir Jindel