This fixes a huge performance regression (introduced in [0]) of a
synthetic microbenchmark - MicroClosureCreateTearoffClassSecondTime -
that happens to have a gigantic function with a large number of call
sites in them.
All of those call sites stay monomorphic but all of them will have to
go to runtime for miss handler because the target code was disabled (the
unoptimized code got disabled, since optimized target code was
installed). The particular target function in question is
`_GrowableArray.add()`.
The cause for the regression is that the miss handler will lookup the
ICData based on linearly walking PcDescriptors to find deopt-id which it
uses to get ICData from function's ic-data array.
Interestingly enough it only regresses ia32 but not x64 (the only two
architectures this benchmark is run on). The reason for that is that
x64 uses exactness tracking and therefore doesn't switch to
monomorphic calls.
This CL:
If a monomorphic call site had a miss because the cached old target
was disabled, we go to runtime to find the real target. Another thread
may have transitioned the call site already from monomorphic to
polymorphic.
When resolving the target function we take advantage of a back reference
from the `ICData::entries()` back to the `ICData` to avoid the O(N)
lookups.
[0] https://dart-review.googlesource.com/c/sdk/+/206373
TEST=Existing test suite.
Change-Id: Ia066a846f5f50e5a575f0734c9453d2368f80780
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/207133
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Previously, the code in the TypeCheck runtime entry assumed that if a
lazily specialized type testing stub (TTS) returned a false negative in
JIT mode, that it should always be regenerated and that regeneration
would always result in different code. In AOT mode, false negatives
instead always cause the stub to go to runtime, even if that false
negative had been seen before, because the assumption is that false
negatives shouldn't happen when the whole class hierarchy is known at
compile time.
However, even in the current implementation of optimized TTSes, there
are cases where this assumption is false. For example, the code
generated by BuildOptimizedSubclassRangeCheckWithTypeArguments allows
for provided type arguments to be type parameters. When this happens,
the type parameter is instantiated at runtime using the instantiator or
function type arguments, and the instantiated type parameter must be
identical to the result retrieved from the type arguments of the
instance. That means that if the instantiated type parameter is not the
same type, but a supertype, of the result, then a false negative is
generated.
This CL changes TypeCheck's handling of false negatives from lazily
specialized TTSes as follows: in JIT, if the regenerated stub is the
same as the old stub, or in AOT, a false negative causes the same fall
back to SubtypeTestCaches as unoptimized stubs. This way, further
checks with the same false negative will be caught via the STC before
going to runtime, assuming the STC hasn't already filled up with false
negatives.
Currently, we only generate false negatives for reasons that will not
change when respecialization occurs due to additions to the hierarchy,
so we do not need to clear affected STCs during respecialization.
However, the previous approach to resetting STCs on reload (in
CallSiteResetter::ResetCaches) is insufficient, since there may be
caches containing reloaded types in non-reloaded code (like the TTS
invoker stub created by the TTS testing framework). Instead, clear
all caches on reload using the same ObjectVisitor as deoptimizing type
testing stubs.
Since we now have to check instruction equality to determine whether
to add to the STC, we now only replace the existing stub if the
instructions are different. This makes it easier to test whether a
TTS invocation on a false positive caused respecialization or not.
This CL also reworks the testing framework for type testing stubs,
1) creating a test case object that stores the particulars of a given
invocation, including expectations, and 2) moving most checks and
access to appropriate data structures into a state object that handles
setup and performing invocations given test cases.
TEST=vm/cc/TTS_Partial
Cq-Include-Trybots: luci.dart.try:vm-kernel-linux-release-x64-try,vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-nnbd-linux-release-x64-try,vm-kernel-nnbd-linux-release-x64-try,vm-kernel-tsan-linux-release-x64-try,vm-kernel-linux-product-x64-try,vm-kernel-precomp-linux-product-x64-try,vm-kernel-linux-release-simarm-try,vm-kernel-linux-release-simarm64-try
Change-Id: I139608c5a0f2442a85a1cf39d1c04104db7a5593
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/208653
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Adds GetNativeField(o,i) to access NativeFieldWrapperClass{1-4}
native fields directly in Dart instead of through VM API.
This will allow us to pass `this` as a raw pointer instead of as
a Dart handle which needs to be unwrapped.
This will in turn allow ffi native calls for instance methods.
TEST=Adds vm/cc/DartAPI_NativeFieldAccess
Bug: https://github.com/dart-lang/sdk/issues/43889
Change-Id: I8006c735569fcbe5e49915a4c7e72c4ca85b356e
Cq-Do-Not-Cancel-Tryjobs: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/205482
Commit-Queue: Clement Skau <cskau@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
For members of program structure that we cache in the ObjectStore we try
to provide only read-only access in the public API. The lazy population
is encapculated in the ObjectStore class.
Use the same mechanism we use for core/isolate/... library members also
for the `dart:_internal` members.
Furthermore cache the Symbol._name field in the ObjectStore to avoid
expensive lookup as well as avoid an extra TLS lookup.
TEST=Existing test suite.
Change-Id: I299faceb39f20576e23244c71c58d698f80f80c7
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/208180
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Previously VM had a hardcoded constant values for context indices of
_future variables in _Future.timeout and Future.wait (used while
collecting stack traces). As those variables are allocated in
the context after captured parameters, their indices depend on
whether the parameters are captured.
I'd like to improve tree shaking so that it will remove
part of Future.wait method and make one of the parameters
not captured, which changes the context index (and it may vary
between modes and for different programs).
So, instead of hardcoding those indices, they are now calculated
at compile time and stored in ObjectStore. This is more flexible
and accommodates for future changes in tree shaker.
TEST=existing tests
Change-Id: I47629b41e497843fe3e998e813e793cf77c18bdd
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/207202
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Clement Skau <cskau@google.com>
Revert the changes in 0920ac88 to use Closure::signature() and
remove the previously existing uses.
Refactor the majority of Object_haveSameRuntimeType out into a helper
function, which lifts the various Bool::Get()s out of the return
statements for easier reading. Also delay retrieving the class out of
the instance until after checking for closure, since we can check the
retrieved cids against kClosureCid instead.
Also move ASSERTs from debug printing of SubtypeTestCache entries
to UpdateTypeTestCache, to ensure they are checked when
--trace-type-checks is not enabled.
TEST=Refactoring, so existing tests. Checked manually on a separate
CL which triggered a case where Closure::signature() was
erroneously returning null due to the direct trip through ptrs.
Change-Id: Iff9b5c8e3f6426cc7deb5ce73c02606597d99d5b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/207124
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Closures created via Closure::New() set the delayed type arguments to
the null type argument vector for non-generic closure functions and to
the empty type argument vector for generic closure functions. However,
closures created via the method extractor stub would always have the
empty type argument vector for delayed type arguments. Make the two
consistent by creating two method extractor stubs, one for generic
extracted methods and one for non-generic ones.
With this change, Closure::IsGeneric simply compares the delayed type
arguments against the empty type argument vector. This change also makes
method extraction for non-generic functions slightly faster, since the
delayed type arguments field is not changed post-allocation.
The old version of Closure::IsGeneric was the only use of
Closure::NumTypeParameters, so remove the latter method.
Related cleanups:
* Use similar checks for Class::IsGeneric and FunctionType::IsGeneric,
since they only have non-null type_parameters fields when generic.
For Class::IsGeneric, this avoids having to fetch the current thread
if the class declaration has been loaded.
* Refactor methods that read packed fields in FunctionType into two
methods, a static method that takes a FunctionTypePtr and an instance
method that calls the static method. Replace the methods in Function
that also directly read FunctionType packed fields with delegations to
the static methods.
* Rework Closure_equals to avoid unneeded handle allocation in certain
cases for non-equal closures.
* Replace cases that allocates a handle for the closure function only
for retrieving the signature with calls to Closure::signature().
TEST=language_2/closure/tearoff_dynamic_test
language_2/regress/regress45890_test
Cq-Include-Trybots: luci.dart.try:vm-kernel-linux-debug-x64-try,vm-kernel-linux-debug-x64c-try,vm-kernel-nnbd-linux-debug-x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-linux-debug-x64c-try,vm-kernel-reload-rollback-linux-debug-x64-try,vm-kernel-reload-linux-debug-x64-try,vm-kernel-linux-release-simarm-try,vm-kernel-linux-release-simarm64-try,vm-kernel-nnbd-linux-release-simarm-try,vm-kernel-nnbd-linux-release-simarm64-try,vm-kernel-precomp-linux-release-simarm-try,vm-kernel-precomp-linux-release-simarm64-try,vm-kernel-precomp-nnbd-linux-release-simarm64-try,vm-kernel-linux-debug-simarm64c-try,vm-kernel-precomp-linux-debug-simarm_x64-try,vm-kernel-precomp-nnbd-linux-debug-simarm_x64-try
Change-Id: I29c8859c3350ed7b3f1a8f71d82a4393496b7c2b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/206820
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
We use message passing as comunication mechanism between isolates.
The transitive closure of an object to be sent is currently serialized
into a snapshot form and deserialized on the receiver side. Furthermore
the receiver side will re-hash any linked hashmaps in that graph.
If isolate gropus are enabled we have all isolates in a group work on
the same heap. That removes the need to use an intermediate
serialization format. It also removes the need for an O(n) step on the
receiver side.
This CL implements a fast transitive object copy implementation and
makes use of it a message that is to be passed to another isolate stays
within the same isolate group.
In the common case the object graph will fit into new space. So the
copy algorithm will try to take advantage of it by having a fast path
and a fallback path. Both of them effectively copy the graph in BFS
order.
The algorithm works effectively like a scavenge operation, but instead
of first copying the from-object to the to-space and then re-writing the
object in to-space to forward the pointers (which requires us writing to
the to-space memory twice), we only reserve space for to-objects and
then initialize the to-objects to it's final contents, including
forwarded pointers (i.e. write the to-space object only once).
Compared with a scavenge operation (which stores forwarding pointers in
the objects themselves), we use a [WeakTable] to store them. This is the
only remaining expensive part of the algorithm and could be further
optimized. To avoid relying on iterating the to-space, we'll remember
[from, to] addresses.
=> All of this works inside a [NoSafepointOperationScope] and avoids
usages of handles as well as write barriers.
While doing the transitive object copy, we'll share any object we can
safely share (canonical objects, strings, sendports, ...) instead of
copying it.
If the fast path fails (due to allocation failure or hitting) we'll
handlify any raw pointers and continue almost the same algorithm in a
safe way, where GC is possible at every object allocation site and
normal barriers are used for any stores of object pointers.
The copy algorithm uses templates to share the copy logic between the
fast and slow case (same copy routines can work on raw pointers as well
as handles).
There's a few special things to take into consideration:
* If we copy a view on external typed data we need to know the
external typed data address to compute the inner pointer of the
view, so we'll eagerly initialize external typed data.
* All external typed data needs to get a finalizer attached
(irrespective if the object copy suceeds or not) to ensure the
`malloc()`ed data is freed again.
* Transferables will only be transferred on successful transitive
copies. Also they need to attach finalizers to objects (which
requires all objects be in handles).
* We copy linked hashmaps as they are - instead of compressing the
data by removing deleted entries. We may need to re-hash those
hashmaps on the receiver side (similar to the snapshot-based copy
approach) since new object graph will have no identity hash codes
assigned to them. Though if the hashmaps only has sharable objects
as keys (very common, e.g. json) there is no need for re-hashing.
It changes the SendPort.* benchmarks as follows:
```
Benchmark | default | IG | IG + FOC
----------------------------------------------------------------------------------------------------------------------------
SendPort.Send.Nop(RunTimeRaw): | 0.25 us (1 x) | 0.26 us (0.96 x) | 0.25 us (1.00 x)
SendPort.Send.Json.400B(RunTimeRaw): | 4.15 us (1 x) | 1.45 us (2.86 x) | 1.05 us (3.95 x)
SendPort.Send.Json.5KB(RunTimeRaw): | 82.16 us (1 x) | 27.17 us (3.02 x) | 18.32 us (4.48 x)
SendPort.Send.Json.50KB(RunTimeRaw): | 784.70 us (1 x) | 242.10 us (3.24 x) | 165.50 us (4.74 x)
SendPort.Send.Json.500KB(RunTimeRaw): | 8510.4 us (1 x) | 3083.80 us (2.76 x) | 2311.29 us (3.68 x)
SendPort.Send.Json.5MB(RunTimeRaw): | 122381.33 us (1 x) | 62959.40 us (1.94 x) | 55492.10 us (2.21 x)
SendPort.Send.BinaryTree.2(RunTimeRaw): | 1.91 us (1 x) | 0.92 us (2.08 x) | 0.72 us (2.65 x)
SendPort.Send.BinaryTree.4(RunTimeRaw): | 6.32 us (1 x) | 2.70 us (2.34 x) | 2.10 us (3.01 x)
SendPort.Send.BinaryTree.6(RunTimeRaw): | 25.24 us (1 x) | 10.47 us (2.41 x) | 8.61 us (2.93 x)
SendPort.Send.BinaryTree.8(RunTimeRaw): | 104.08 us (1 x) | 41.08 us (2.53 x) | 33.51 us (3.11 x)
SendPort.Send.BinaryTree.10(RunTimeRaw): | 373.39 us (1 x) | 174.11 us (2.14 x) | 134.75 us (2.77 x)
SendPort.Send.BinaryTree.12(RunTimeRaw): | 1588.64 us (1 x) | 893.18 us (1.78 x) | 532.05 us (2.99 x)
SendPort.Send.BinaryTree.14(RunTimeRaw): | 6849.55 us (1 x) | 3705.19 us (1.85 x) | 2507.90 us (2.73 x)
SendPort.Receive.Nop(RunTimeRaw): | 0.67 us (1 x) | 0.69 us (0.97 x) | 0.68 us (0.99 x)
SendPort.Receive.Json.400B(RunTimeRaw): | 4.37 us (1 x) | 0.78 us (5.60 x) | 0.77 us (5.68 x)
SendPort.Receive.Json.5KB(RunTimeRaw): | 45.67 us (1 x) | 0.90 us (50.74 x) | 0.87 us (52.49 x)
SendPort.Receive.Json.50KB(RunTimeRaw): | 498.81 us (1 x) | 1.24 us (402.27 x) | 1.06 us (470.58 x)
SendPort.Receive.Json.500KB(RunTimeRaw): | 5366.02 us (1 x) | 4.22 us (1271.57 x) | 4.65 us (1153.98 x)
SendPort.Receive.Json.5MB(RunTimeRaw): | 101050.88 us (1 x) | 20.81 us (4855.88 x) | 21.0 us (4811.95 x)
SendPort.Receive.BinaryTree.2(RunTimeRaw): | 3.91 us (1 x) | 0.76 us (5.14 x) | 0.74 us (5.28 x)
SendPort.Receive.BinaryTree.4(RunTimeRaw): | 9.90 us (1 x) | 0.79 us (12.53 x) | 0.76 us (13.03 x)
SendPort.Receive.BinaryTree.6(RunTimeRaw): | 33.09 us (1 x) | 0.87 us (38.03 x) | 0.84 us (39.39 x)
SendPort.Receive.BinaryTree.8(RunTimeRaw): | 126.77 us (1 x) | 0.92 us (137.79 x) | 0.88 us (144.06 x)
SendPort.Receive.BinaryTree.10(RunTimeRaw): | 533.09 us (1 x) | 0.94 us (567.12 x) | 0.92 us (579.45 x)
SendPort.Receive.BinaryTree.12(RunTimeRaw): | 2223.23 us (1 x) | 3.03 us (733.74 x) | 3.04 us (731.33 x)
SendPort.Receive.BinaryTree.14(RunTimeRaw): | 8945.66 us (1 x) | 4.03 us (2219.77 x) | 4.30 us (2080.39 x)
```
Issue https://github.com/dart-lang/sdk/issues/36097
TEST=vm/dart{,_2}/isolates/fast_object_copy{,2}_test
Change-Id: I835c59dab573d365b8a4b9d7c5359a6ea8d8b0a7
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/203776
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
`_CompactLinkedHashSet` now extends `_HashVMBase`. The class hierarchy
is organized as mixins similar to LinkedHashMap to accomodate for the
other Sets not extending `_HashVMBase`.
Also, rearranges some code so that introducing ImmutableHashMap and
ImmutableHashSet is easier.
1) snapshot.h and snapshot.cc now have a MapReadFrom, MapWriteTo,
SetReadFrom, and SetWriteTo to facilitate code sharing between
mutable and immutable implementations similar to ArrayReadFrom and
ArrayWriteTo.
2) Macros for CLASS_LIST_MAPS and CLASS_LIST_SETS to facilitate
treating mutable and immutable implementations with the same handle.
Also similar to Array.
Clustered snapshots for HashMaps is currently dead code. This CL makes
it explicit by marking these as unreachable. Immutable maps and sets
will end up in the clustered snapshot in follow up CLs.
Bug: https://github.com/dart-lang/sdk/issues/36077
Bug: https://github.com/dart-lang/sdk/issues/45908
TEST=runtime/vm/object_test.cc
TEST=tests/**_test.dart on many bots
Change-Id: If3cc5ebb3138535aeb0d5e06d9da3d1c9fb2deb2
Cq-Include-Trybots: luci.dart.try:analyzer-nnbd-linux-release-try,app-kernel-linux-debug-x64-try,dart-sdk-linux-try,front-end-nnbd-linux-release-x64-try,pkg-linux-debug-try,vm-canary-linux-debug-try,vm-kernel-asan-linux-release-x64-try,vm-kernel-checked-linux-release-x64-try,vm-kernel-linux-debug-x64c-try,vm-kernel-linux-debug-x64-try,vm-kernel-linux-debug-simarm64c-try,vm-kernel-nnbd-linux-release-simarm-try,vm-kernel-optcounter-threshold-linux-release-x64-try,vm-kernel-precomp-android-release-arm_x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-linux-debug-x64c-try,vm-kernel-reload-linux-debug-x64-try,vm-kernel-reload-rollback-linux-debug-x64-try,vm-precomp-ffi-qemu-linux-release-arm-try,vm-kernel-precomp-linux-release-simarm_x64-try,vm-kernel-precomp-linux-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/206222
Reviewed-by: Tess Strickland <sstrickl@google.com>
Reviewed-by: Ben Konyi <bkonyi@google.com>
Previously there were several pieces of information shared between
both FunctionType and Function, mostly in the packed fields, but
named argument names were also kept in both places.
Now the FunctionType is the primary source for this information, with
the Function only keeping the names of positional arguments, which are
discarded in AOT snapshots.
This does mean extra work to access this information via the function
object, but for the most part, this information is only accessed in the
compiler or during dynamic lookups or checks in the runtime.
After adding the count of type parameters to the packed information
in FunctionType, the packed information has been split into two pieces:
one for parameter counts, another for type parameter counts. This
split does not increase the size of UntaggedFunctionType, as there
were 2 bytes available in the existing padding.
Changes on flutter gallery in release mode:
* ARM7 code size: total -0.91%, readonly -0.22%, isolate -4.32%
* ARM7 heap size: total -2.00%
* ARM8 code size: total -0.93%, readonly -0.22%, isolate -4.32%
* ARM8 heap size: total -2.12%
Changes on flutter gallery in release-sizeopt mode:
* ARM7 code size: total -0.24%, readonly -0.08%, isolate -1.49%
* ARM7 heap size: total -0.88%
* ARM8 code size: total -0.26%, readonly -0.11%, isolate -1.49%
* ARM8 heap size: total -1.01%
TEST=Refactoring, so existing tests.
Cq-Include-Trybots: luci.dart.try:vm-kernel-linux-debug-x64-try,vm-kernel-nnbd-linux-debug-x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-nnbd-linux-debug-x64-try,vm-kernel-reload-linux-debug-x64-try,vm-kernel-reload-rollback-linux-debug-x64-try,vm-kernel-precomp-linux-debug-simarm_x64-try,vm-kernel-precomp-nnbd-linux-debug-simarm_x64-try,vm-kernel-linux-debug-ia32-try,vm-kernel-nnbd-linux-debug-ia32-try,vm-kernel-linux-release-simarm-try,vm-kernel-linux-release-simarm64-try,vm-kernel-precomp-linux-debug-simarm_x64-try,vm-kernel-precomp-linux-release-simarm-try,vm-kernel-precomp-linux-release-simarm64-try,vm-kernel-precomp-nnbd-linux-debug-simarm_x64-try,vm-kernel-precomp-nnbd-linux-release-simarm64-try
Change-Id: Ic4d59a7b4acca039a5647f9163e716f6019163f5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/203241
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Régis Crelier <regis@google.com>
When intalling a new TTS stub two fields have to be updated:
- AbstractType::type_test_stub
- AbstractType::type_test_stub_entry_point_
Right now the code takes the program lock when updating those two
fields, which is very heavy weight, since this happens very often and
therefore the lock will be contended.
Instead we'll solve the write-write races by ensuring they are applied
using atomics in (any) order - though ensuring that they will be in
sync.
It is fine for them to be out-of sync for a short time, as long as we're
guaranteed that whenever we hit a gc safepoint they are in sync (because GC
might collect old TTS stubs that are not referenced anymore - i.e. we
need to avoid situations where the cached entrypoint is old but the code
pointer is new)
There can be multiple mutators computing and installing a TTS at the same
time, they will try to install in sequence.
=> This reduces the average waiting time on write access to
program lock by 3x
Fixes https://github.com/dart-lang/sdk/issues/46124
Issue https://github.com/dart-lang/sdk/issues/46252
TEST=Existing test coverage.
Change-Id: Id1ed29adeb26fa93494e8710206fd680d427bd34
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/205520
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
state_bits_ field in ICData can be accessed concurrently, so this CL makes it an atomic field. Concurrent access comes for example from monomorphic miss handlers that read tracking_exactness for the field, while deopt code writes deopt_reason for the same field.
Fixes https://github.com/dart-lang/sdk/issues/46508
TEST=IsolateSpawn on tsan
Change-Id: I42375e52590f3e7a6506fcd40c0053f86e6aa6e6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/205100
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Alexander Aprelev <aam@google.com>
The new macros mean that the field type is only generalized to ObjectPtr
in the precompiler, and otherwise stays the same. Since the serializer
is the only part of the system that needs to know about
WeakSerializationReferences and it accesses the fields of the untagged
object directly, have the macro create a getter that automatically
unwraps if needed and casts the wrapped value to the normally expected
type so no other code in the VM needs to change.
This change also changes WeakSerializationReference::New to return an
ObjectPtr, not a WeakSerializationReferencePtr, so we can return the
target instead of wrapping it if it's an object that's guaranteed to be
serialized (e.g., null).
WeakSerializationReference::New also checks to see if the target is
already a WeakSerializationReference, and returns it if the replacement
is the same. If the replacement is different, it uses the target of the
WSR as its target. This way, WSRs are never nested, which fits the
assumptions from the original implementation.
TEST=Primarily a refactoring, so existing tests.
Cq-Include-Trybots: luci.dart.try:vm-kernel-linux-debug-x64-try,vm-kernel-nnbd-linux-debug-x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-linux-debug-x64c-try,vm-kernel-precomp-nnbd-linux-debug-x64-try,vm-kernel-linux-debug-x64c-try
Change-Id: Ia6790e4e5bd5cf302b732180dee42f796d28d59e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/204141
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
events on Developer stream.
Also removes intrinsic implementations for UserTag_makeCurrent to allow
for service events to be sent on UserTag change.
TEST=DartAPI_UserTags,pkg/vm_service/test/user_tag_changed_test.dart
Change-Id: I5dc9ee77c0048590d3c6e33a652eee5bc3bf522a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/204440
Commit-Queue: Ben Konyi <bkonyi@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
This is a reland of 09d2025685
Fixes https://github.com/dart-lang/sdk/issues/23746
Intrinsics for sameRuntimeType wrongly assumed that types with unequal type arguments are unequal, which is not true:
1) nullability of individual type arguments may be different
2) one vector may be a prefix of the other vector
Note that the intrinsic for type equality did not make this assumption.
Case 2 above was not handled properly in the runtime.
TEST=added regression test
Original change's description:
> [VM/runtime] Handle generic types in intrinsics for type equality and runtimeType comparison.
>
> Generic types with equal class ids and equal type arguments are now considered equal by the intrinsics and a runtime call is avoided.
>
> Fixes https://github.com/dart-lang/sdk/issues/23746
>
> TEST=existing ones
>
> Change-Id: I668db119ac6d2525eac3a4f17a44f36c53b9dbf5
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/203143
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
> Commit-Queue: Régis Crelier <regis@google.com>
Change-Id: I53fc00d856ecd9d9b8d66b8da95285e6e0bd508e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/204363
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Régis Crelier <regis@google.com>
Adds support for marking external functions as @FfiNative's,
which will be called using fast FFI calls.
Resolution happens by calling out to a new embedder provided
Dart_FfiNativeResolver which the embedder can specify via
Dart_SetFfiNativeResolver.
TEST=vm/cc/DartAPI_FfiNativeResolver
Bug: https://github.com/dart-lang/sdk/issues/43889
Change-Id: I3cfff360b05314499a81444b90f4ea0a1b937b0b
Cq-Do-Not-Cancel-Tryjobs: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/170092
Commit-Queue: Clement Skau <cskau@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Usually the identity hashcode is implemented by returning if present and
otherwise generating a random number, setting the hash code and
returning it.
This works fine in single-mutator environments, though when multiple
mutators might race to install identity hash codes we should ensure only
one of them wins and all others will obtain whatever hash code has been
set by the winner.
This will matter once we start sharing more objects across isolates.
Issue https://github.com/dart-lang/sdk/issues/36097
TEST=vm/cc/AsmIntrinsifier_SetHashIfNotSetYet
Change-Id: Ie760ca9658e6ec0640255361544d6822b07574e2
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/201827
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
Avoid calling SetTypeTestingStub from PostLoad because it
uses write lock.
Saves ~10-15% on ReadProgramSnapshot on a large Flutter app.
Issue https://github.com/dart-lang/sdk/issues/46116
TEST=ci
Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-linux-product-x64-try,vm-kernel-precomp-linux-debug-x64-try
Change-Id: If843828661e68f18df19824af204df326bf016a0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/201180
Auto-Submit: Vyacheslav Egorov <vegorov@google.com>
Commit-Queue: Vyacheslav Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Currently external data is counted twice on the receiver size: once for
received transferable, second for newly created external typed data.
Ensure new space is collected when allocating external at the limit.
TEST=SendReceiveBytesTransferable with verbose-gc
Change-Id: I129d913f89e098b5d3066ce249ac0c4702e1394d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/199842
Commit-Queue: Alexander Aprelev <aam@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
This CL adds FFI leaf calls by adding `lookupFunction(.., isLeaf)`
and `_asFunctionInternal(.., isLeaf)`, which generate FFI leaf calls.
These calls skip a lot of the usual frame building and generated <->
native transition overhead.
`benchmark/FfiCall/` shows a 1.1x - 4.3x speed-up between the regular
FFI calls and their leaf call counterparts (JIT, x64, release).
TEST=Adds `tests/ffi{,_2}/vmspecific_leaf_call_test.dart`. Tested FFI tests.
Closes: https://github.com/dart-lang/sdk/issues/36707
Cq-Include-Trybots: luci.dart.try:vm-precomp-ffi-qemu-linux-release-arm-try,vm-ffi-android-release-arm64-try,vm-ffi-android-release-arm-try,vm-ffi-android-product-arm64-try,vm-ffi-android-product-arm-try,vm-ffi-android-debug-arm64-try,vm-ffi-android-debug-arm-try,vm-kernel-linux-debug-ia32-try,vm-kernel-win-debug-ia32-try,vm-kernel-win-debug-x64-try,vm-kernel-win-release-x64-try,vm-kernel-mac-debug-x64-try,vm-kernel-precomp-nnbd-mac-release-simarm64-try,vm-kernel-precomp-android-release-arm64-try,vm-kernel-precomp-asan-linux-release-x64-try,vm-kernel-precomp-linux-release-simarm_x64-try,vm-kernel-precomp-obfuscate-linux-release-x64-try,vm-kernel-precomp-ubsan-linux-release-x64-try,vm-kernel-precomp-tsan-linux-release-x64-try,vm-kernel-precomp-win-release-x64-try,vm-precomp-ffi-qemu-linux-release-arm-try,vm-kernel-reload-rollback-linux-debug-x64-try,vm-kernel-reload-linux-debug-x64-try
Bug: https://github.com/dart-lang/sdk/issues/36707
Change-Id: Id8824f36b0006bf09951207bd004356fe6e9f46e
Cq-Do-Not-Cancel-Tryjobs: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/179768
Commit-Queue: Clement Skau <cskau@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
The same signature (canonical function type) may be shared between different closure functions.
Clean up asserts verifying that members are canonical when updating type test cache.
TEST=existing ones
Change-Id: Ic206409e56d34dfd02d6705af93065fd3b83b8fa
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/199561
Commit-Queue: Régis Crelier <regis@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
This avoids the extra redirection through the closure function, which
does not need to be loaded otherwise during closure calls in this mode,
and thus removes another runtime dependency on the closure function in
bare instructions mode.
In non-bare mode, CODE_REG is populated with the code object for the
function, so caching wouldn't change the number of loads there.
This does not increase the size of closure objects, as there was
already a free word in them due to object alignment.
TEST=Existing tests.
Cq-Include-Trybots: luci.dart.try:vm-kernel-precomp-linux-debug-simarm64c-try,vm-kernel-precomp-linux-debug-simarm_x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-linux-debug-x64c-try,vm-kernel-precomp-linux-product-x64-try,vm-kernel-precomp-linux-release-x64-try,vm-kernel-precomp-linux-release-simarm64-try,vm-kernel-precomp-linux-release-simarm-try,vm-kernel-linux-debug-ia32-try,vm-kernel-linux-debug-x64-try,vm-kernel-linux-product-x64-try,vm-kernel-linux-release-simarm64-try,vm-kernel-linux-release-simarm-try,vm-kernel-linux-release-x64-try,vm-kernel-linux-debug-x64c-try
Change-Id: Ida6e0d277919259a8c0e8dcbfaa101379fd22ff1
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/195920
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
-----
Adds a few type traits for operating on (Compressed)ObjectPtr types:
* is_compressed_ptr<T>::value is true for compressed pointer types and
false for uncompressed pointer types.
* is_uncompressed_ptr<T>::value is false for compressed pointer types
and true for uncompressed pointer types.
* base_ptr_type<T>::type is ObjectPtr for uncompressed pointer types and
CompressedObjectPtr for compressed pointer types.
Note: If DART_COMPRESSED_POINTERS is not enabled, all pointers are
uncompressed and the above traits function accordingly. That means
that is_compressed_ptr<T>::value is always false,
is_uncompressed_ptr<T>::value is true for all object pointers, and
base_ptr_type<T>::type is ObjectPtr for all object pointers, even if
they contain Compressed in the name.
-----
The following changes have been made to the VISIT_* macros:
* VISIT_NOTHING: no change.
* VISIT_FROM: takes only a field name now, and retrieves the type of
the field to determine which base pointer type to use for from().
Note that VISIT_FROM must now come _after_ the declaration of the
field, since it retrieves the type from the declaration.
* VISIT_TO: takes only a field name now, and retrieves the type of the
field to determine which base pointer type to use for to().
* (removed) VISIT_TO_LENGTH: Instead, VISIT_TO creates a to() method
that takes an optional length argument (defaults to 0).
* (new) VISIT_FROM_PAYLOAD_START: takes the object pointer type of the
payload elements and creates a from() that returns the beginning
element of the payload. Note that a payload element must be a single
object ptr to use this macro.
* (new) VISIT_TO_PAYLOAD_END: takes the object pointer type of the
payload elements and creates a to() which takes a non-optional length
argument and returns the last element of the payload. Note that a
payload element must be a single object ptr to use this macro.
If one of the {COMPRESSED_,}VARIABLE_POINTER_FIELDS macros are used,
then VISIT_TO_PAYLOAD_END should not be, as those macros already
include a use of it.
-----
In addition, each Untagged<Class> class now has a static constexpr bool
field kContainsCompressedPointers. This field is false for
UntaggedObject, and for other Untagged<Class> classes, it is either
inherited from the parent class or set during VISIT_FROM and the new
VISIT_FROM_PAYLOAD_START macro.
VISIT_TO and the new VISIT_TO_PAYLOAD_END macro double-check at compile
time that the type retrieved or provided is a compressed type if
kContainsCompressedPointers is true, and uncompressed if false.
If the elements of a payload are not object pointers, but rather
a composite object that contains object pointers, then
VISIT_FROM_PAYLOAD_START/VISIT_TO_PAYLOAD_END cannot be used. Instead,
use DEFINE_COMPRESSED_POINTERS(type), where type is the type of one of
the object pointers in the composite object, to appropriately define
kContainsCompressedPointers.
Note that this field will always be false when DART_COMPRESSED_POINTERS
is not defined.
-----
For classes, a static constexpr ContainsCompressedPointers() predicate
is created, based on the associated untagged class's field.
For instances of Class, there is an instance predicate
Class::HasCompressedPointers() that returns whether instances of a given
class object contain compressed pointers.
Change all calls to InitializeObject and Object::Allocate to pass in
the result of the appropriate predicate.
TEST=Refactoring, so current tests on CI, especially on x64c trybots.
Cq-Include-Trybots: luci.dart.try:vm-kernel-linux-debug-x64c-try,vm-kernel-precomp-linux-debug-x64c-try,vm-kernel-precomp-linux-debug-x64-try,vm-kernel-linux-debug-x64-try
Change-Id: Ifb61f72885bd8b951167becbccf8ec3337a922b1
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/196931
Reviewed-by: Liam Appelbe <liama@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
We once relied on making Symbols etc unreachable in the compiler's heap to remove unused ones from the snapshot, but we have since moved to using reachablity during snapshotting.
TEST=ci
Change-Id: I6a95bd71582deedc271441e4b6732201b2293c5b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/197442
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Tess Strickland <sstrickl@google.com>
Martin Kustermann