Commit Graph

80 Commits

Author SHA1 Message Date
Daco Harkes 4e2343c290 [vm/ffi] Introduce Allocator API
Introduces the Allocator API in `dart:ffi`.

This CL does not yet roll `package:ffi` to use `Allocator`, because that
breaks the checked in Dart in Fluter in g3. Instead, this coppies
`_MallocAllocator` from `package:ffi` into the ffi tests for testing.

This CL does not yet migrate off `allocate` and `free` in the SDK. That
is done in a dependent CL.

Issue: https://github.com/dart-lang/sdk/issues/44621
Issue: https://github.com/dart-lang/sdk/issues/38721

TEST=tests/ffi/allocator_test.dart
TEST=tests/ffi/calloc_test.dart
TEST=tests/ffi/vmspecific_static_checks_test.dart

Change-Id: I173e213a750b8b3f594bb8d4fc72575f2b6b91f7
Cq-Include-Trybots: luci.dart.try:vm-precomp-ffi-qemu-linux-release-arm-try,analyzer-analysis-server-linux-try,analyzer-linux-release-try,analyzer-nnbd-linux-release-try,front-end-linux-release-x64-try,front-end-nnbd-linux-release-x64-try,benchmark-linux-try,dart-sdk-linux-try,pkg-linux-release-try,vm-ffi-android-release-arm-try,vm-ffi-android-release-arm64-try,vm-kernel-nnbd-win-debug-x64-try,vm-kernel-win-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/177705
Reviewed-by: Clement Skau <cskau@google.com>
2021-01-13 17:04:08 +00:00
Daco Harkes 28a228b64b [analyzer/ffi] Add tests for #44584
Add tests for analyzer and CFE covering `Pointer<void>` and
`Pointer<NativeFunction>` when used in `lookupFunction`.
Both were missing in test suites.

Added TODOs for the things that need to be addressed.

Issue: https://github.com/dart-lang/sdk/issues/44584
Issue: https://github.com/dart-lang/sdk/issues/44593
Issue: https://github.com/dart-lang/sdk/issues/44594

Change-Id: I24ce7caa1a20af51e14e75a7fa83578e00a2b440
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/177700
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Dmitry Stefantsov <dmitryas@google.com>
2021-01-06 19:26:30 +00:00
Daco Harkes b6b82dd3ac [vm/ffi] Support nested structs
This CL adds support for nested structs in FFI calls, callbacks, and
memory loads and stores through the Struct classes itself.

Nesting empty structs and nesting a structs in themselves (directly or
indirectly) is reported as error.

This feature is almost fully implemented in the CFE transformation.

Because structs depend on the sizes of their nested structs, the structs
now need to be processed in topological order.

Field access to nested structs branches at runtime on making a derived
Pointer if the backing memory of the outer struct was a Pointer or
making a TypedDataView if the backing memory of the outer struct was
a TypedData.

Assigning to a nested struct is a byte for byte copy from the source.

The only changes in the VM are contained in the native calling
convention calculation which now recursively needs to reason about
fundamental types instead of just 1 struct deep.

Because of the amount of corner cases in the calling conventions that
need to be covered, the tests are generated, rather than hand-written.

ABIs tested on CQ: x64 (Linux, MacOS, Windows), ia32 (Linux, Windows),
arm (Android softFP, Linux hardFP), arm64 Android.
ABIs tested locally through Flutter: arm64 iOS.
ABIs not tested: ia32 Android (emulator), x64 iOS (simulator), arm iOS.
TEST=runtime/bin/ffi_test/ffi_test_functions_generated.cc
TEST=runtime/bin/ffi_test/ffi_test_functions.cc
TEST=tests/{ffi,ffi_2}/function_structs_by_value_generated_test.dart
TEST=tests/{ffi,ffi_2}/function_callbacks_structs_by_value_generated_tes
TEST=tests/{ffi,ffi_2}/function_callbacks_structs_by_value_test.dart
TEST=tests/{ffi,ffi_2}/vmspecific_static_checks_test.dart

Closes https://github.com/dart-lang/sdk/issues/37271.

Contains a temporary workaround for
https://github.com/dart-lang/sdk/issues/44454.

Change-Id: I5e5d10e09e5c3fc209f5f7e997efe17bd362214d
Cq-Include-Trybots: luci.dart.try:dart-sdk-linux-try,dart-sdk-mac-try,dart-sdk-win-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64-try,vm-kernel-asan-linux-release-x64-try,vm-kernel-mac-debug-x64-try,vm-kernel-linux-debug-ia32-try,vm-kernel-linux-debug-x64-try,vm-kernel-nnbd-linux-debug-x64-try,vm-kernel-nnbd-linux-debug-ia32-try,vm-kernel-nnbd-mac-release-x64-try,vm-kernel-nnbd-win-debug-x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-linux-debug-simarm_x64-try,vm-kernel-precomp-nnbd-linux-debug-x64-try,vm-kernel-precomp-win-release-x64-try,vm-kernel-reload-linux-debug-x64-try,vm-kernel-reload-rollback-linux-debug-x64-try,vm-kernel-win-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-precomp-ffi-qemu-linux-release-arm-try,vm-kernel-precomp-obfuscate-linux-release-x64-try,vm-kernel-msan-linux-release-x64-try,vm-kernel-precomp-msan-linux-release-x64-try,vm-kernel-precomp-android-release-arm_x64-try,analyzer-analysis-server-linux-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/169221
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Dmitry Stefantsov <dmitryas@google.com>
2020-12-18 09:34:35 +00:00
Daco Harkes 3e7cda8a4e [vm/ffi] Support passing structs by value
This CL adds passing structs by value in FFI trampolines.
Nested structs and inline arrays are future work.
C defines passing empty structs as undefined behavior, so that is not
supported in this CL.

Suggested review order:
1) commit message
2) ffi/marshaller (decisions for what is done in IL and what in MC)
3) frontend/kernel_to_il (IL construction)
4) backend/il (MC generation from IL)
5) rest in VM

Overall architecture is that structs are split up into word-size chunks
in IL when this is possible: 1 definition in IL per chunk, 1 Location in
IL per chunk, and 1 NativeLocation for the backend per chunk.
In some cases it is not possible or less convenient to split into
chunks. In these cases TypedDataBase objects are stored into and loaded
from directly in machine code.
The various cases:
- FFI call arguments which are not passed as pointers: pass individual
  chunks to FFI call which already have the right location.
- FFI call arguments which are passed as pointers: Pass in TypedDataBase
  to FFI call, allocate space on the stack, and make a copy on the stack
  and pass the copies' address to the callee.
- FFI call return value: pass in TypedData to FFI call, and copy result
  in machine code.
- FFI callback arguments which are not passed as pointers: IL definition
  for each chunk, and populate a new TypedData with those chunks.
- FFI callback arguments which are passed as pointer: IL definition for
  the pointer, and copying of contents in IL.
- FFI return value when location is pointer: Copy data to callee result
  location in IL.
- FFI return value when location is not a pointer: Copy data in machine
  code to the right registers.

Some other notes about the implementation:
- Due to Store/LoadIndexed loading doubles from float arrays, we use
  a int32 instead and use the BitCastInstr.
- Linux ia32 uses `ret 4` when returning structs by value. This requires
  special casing in the FFI callback trampolines to either use `ret` or
  `ret 4` when returning.
- The 1 IL definition, 1 Location, and 1 NativeLocation approach does
  not remove the need for special casing PairLocations in the machine
  code generation because they are 1 Location belonging to 1 definition.

Because of the amount of corner cases in the calling conventions that
need to be covered, the tests are generated, rather than hand-written.

ABIs tested on CQ: x64 (Linux, MacOS, Windows), ia32 (Linux, Windows),
arm (Android softFP, Linux hardFP), arm64 Android.
ABIs tested locally through Flutter: ia32 Android (emulator), x64 iOS
(simulator), arm64 iOS.
ABIs not tested: arm iOS.
TEST=runtime/bin/ffi_test/ffi_test_functions_generated.cc
TEST=runtime/bin/ffi_test/ffi_test_functions.cc
TEST=tests/{ffi,ffi_2}/function_structs_by_value_generated_test.dart
TEST=tests/{ffi,ffi_2}/function_callbacks_structs_by_value_generated_tes
TEST=tests/{ffi,ffi_2}/function_callbacks_structs_by_value_test.dart
TEST=tests/{ffi,ffi_2}/vmspecific_static_checks_test.dart

Closes https://github.com/dart-lang/sdk/issues/36730.

Change-Id: I474d3a4ee1faadbe767ddadd1b696e24d8dc364c
Cq-Include-Trybots: luci.dart.try:dart-sdk-linux-try,dart-sdk-mac-try,dart-sdk-win-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64-try,vm-kernel-asan-linux-release-x64-try,vm-kernel-mac-debug-x64-try,vm-kernel-linux-debug-ia32-try,vm-kernel-linux-debug-x64-try,vm-kernel-nnbd-linux-debug-x64-try,vm-kernel-nnbd-linux-debug-ia32-try,vm-kernel-nnbd-mac-release-x64-try,vm-kernel-nnbd-win-debug-x64-try,vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-linux-debug-simarm_x64-try,vm-kernel-precomp-nnbd-linux-debug-x64-try,vm-kernel-precomp-win-release-x64-try,vm-kernel-reload-linux-debug-x64-try,vm-kernel-reload-rollback-linux-debug-x64-try,vm-kernel-win-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-precomp-ffi-qemu-linux-release-arm-try,vm-kernel-precomp-obfuscate-linux-release-x64-try,vm-kernel-msan-linux-release-x64-try,vm-kernel-precomp-msan-linux-release-x64-try,vm-kernel-precomp-android-release-arm_x64-try,analyzer-analysis-server-linux-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/140290
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2020-12-14 16:22:48 +00:00
Daco Harkes 550d580d5e [vm/ffi] Fuchsia FFI testing package
This does not include enabling the FFI on Fuchsia. So these tests will
fail without the dependent CL.

This CL adds an `all_positive.dart` test file to be packaged in the
.far file, such that it is easier to run all FFI tests in one go.

We don't run tests on the bots, but we do build this test package on
the bots.

To build the .far files locally:
```
tools/build.py --os=fuchsia -m debug fuchsia_ffi_test_package
tools/build.py --os=fuchsia -m debug -a arm64 fuchsia_ffi_test_package
```

Issue: https://github.com/dart-lang/sdk/issues/44434

Change-Id: Iec418e0c10d77afbf811fb83151664bcc2710b99
Cq-Include-Trybots: luci.dart.try:vm-fuchsia-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/175484
Reviewed-by: Liam Appelbe <liama@google.com>
2020-12-10 12:15:17 +00:00
Tess Strickland 52930fe2b9 [vm/compiler] Do not perform non-covariant checks in FFI trampolines.
Since they are accessed via closures, dynamic calls are handled via
dynamic closure call dispatchers and so no non-covariant argument
checking remains at this point.

TEST=Existing FFI tests on trybots, also added a few dynamic cast
tests just in case.

Bug: https://github.com/dart-lang/sdk/issues/44343

Cq-Include-Trybots: luci.dart.try:vm-precomp-ffi-qemu-linux-release-arm-try,vm-ffi-android-product-arm64-try,vm-ffi-android-product-arm-try
Change-Id: If8daa2d71de4e9391dd35b31b62b67a8e7e69ffe
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/174463
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
2020-12-01 16:01:15 +00:00
Vyacheslav Egorov a2acd59d73 [vm/ffi] Fix LR clobber in JumpToFrame stub on ARM
Both ARM and ARM64 variants did not account for the
fact that TransitionNativeToGenerated will clobber
LR on slow path.

TEST=tests/ffi/regress_jump_to_frame_test.dart

Cq-Include-Trybots: luci.dart.try:vm-ffi-android-release-arm-try,vm-ffi-android-release-arm64-try,vm-kernel-linux-release-simarm64-try,vm-kernel-linux-release-simarm-try,vm-kernel-precomp-linux-release-simarm-try,vm-kernel-precomp-linux-release-simarm64-try
Change-Id: I40d48b720b2a7f31ff20903a887295ab2034662c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/173962
Commit-Queue: Vyacheslav Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2020-11-27 09:11:49 +00:00
Daco Harkes 6101e4653b [test/ffi] Add struct stack alignment test
A smoke test to exercise alignment of structs with various field types
to see how they are aligned on the stack. Only contains small structs
because most ABIs only pass structs less than 16 bytes on the stack.

Especially tests iOS64, where `AlignOf(struct{float32;float32;}) == 4`
but `AlignOf(struct{int32;int32;}) == 8` due to the special treatment
of homogenous floats in the arm64 ABI combined with iOS decision to not
align everything on the stack to at least word size.

Test tested on iOS64 through Flutter.

structs_by_value_tests_configuration.dart is the only non-generated
file.

The tests are tested in the dependent CL. Tests are ignored in status
file and are un-ignored in dependent CL.

Tests for https://github.com/dart-lang/sdk/issues/36730.

Change-Id: I6ec4523208db740b8ea3f8a4ab1e5717f1088467
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/170691
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Auto-Submit: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2020-11-06 13:54:23 +00:00
Daco Harkes 5278383736 [vm] Native API: Make Dart_NewWeakPersistentHandle not auto delete
Changes Dart_NewWeakPersistentHandle to no longer auto delete the
weak persistent handle.

Changes the signatures of WeakPersistentHandleFinalizers to no longer
have access to the handle.

Flutter PR: https://github.com/flutter/engine/pull/19843
g3 presubmit: cl/318028238

Issue: https://github.com/dart-lang/sdk/issues/42312

TEST=runtime/vm/dart_api_impl_test.cc

Change-Id: I3f77db9954d9486759f903b78c03a494f73c68ba
Cq-Include-Trybots:dart/try:vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64-try,app-kernel-linux-debug-x64-try,vm-kernel-linux-debug-ia32-try,vm-kernel-win-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-android-release-arm64-try,vm-kernel-asan-linux-release-x64-try,vm-kernel-msan-linux-release-x64-try,vm-kernel-precomp-msan-linux-release-x64-try,vm-kernel-linux-release-simarm-try,vm-kernel-linux-release-simarm64-try,vm-kernel-precomp-android-release-arm_x64-try,vm-kernel-precomp-obfuscate-linux-release-x64-try,dart-sdk-linux-try,analyzer-analysis-server-linux-try,analyzer-linux-release-try,front-end-linux-release-x64-try,vm-kernel-precomp-win-release-x64-try,vm-kernel-mac-debug-x64-try,vm-precomp-ffi-qemu-linux-release-arm-try,vm-kernel-nnbd-linux-debug-x64-try,analyzer-nnbd-linux-release-try,front-end-nnbd-linux-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/151525
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2020-11-03 10:27:44 +00:00
Alexander Markov 061848e97e Cleanup misc remaining references to bytecode
Change-Id: I37c2dfb29a5bc4faf1478fb31623749cb8f67d67
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/169502
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Régis Crelier <regis@google.com>
2020-10-28 21:51:20 +00:00
Daco Harkes 312a572ade [vm/ffi] Fix prepare_flutter_bundle.dart
It now includes /ffi_test_functions_generated.cc and prints
instructions on how to use the generated bundle.

This CL adds some TODOs and deletes the non-migrated copy that was in
the NNBD folder.

Finally, this CL disable a check that fails because of static linking
on iOS.

Issue https://github.com/dart-lang/sdk/issues/39570.

Change-Id: Ie85d92568bb95eaafd2991725a1a31cdecd246be
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/169402
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2020-10-28 21:33:35 +00:00
Daco Harkes 957487b5da [test/ffi] Structs by value 1-byte aligned
The 3 and 7 byte tests in
https://dart-review.googlesource.com/c/sdk/+/168829
had their sizes rounded up to the next power of two due to alignment of
fields inside the structs.

In order to properly exercise behavior on win32, win64 and iOS, this CL
changes those structs to only have uint8 fields to prevent alignment
influencing the sizes.

Old tests are renamed to exercise the size rounding due to alignment.

This also swaps around the two 9byte structs to have consistent naming
with the new 3 and 7-byte tests.

structs_by_value_tests_confguration.dart is the only non-generated file.

The tests are tested in the dependent CL. Tests are ignored in status
file and are un-ignored in dependent CL.

Tests for https://github.com/dart-lang/sdk/issues/36730.

Change-Id: If47cfcaebe56f43d50265f0d6e2749c56fe7b195
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/169101
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Tess Strickland <sstrickl@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2020-10-26 17:26:56 +00:00
Daco Harkes b62e715073 [test/ffi] Passing structs by value test generator
Split off from https://dart-review.googlesource.com/c/sdk/+/140290/33
to make that CL smaller and review the test generator separately.

Because of the amount of corner cases in the calling conventions that
need to be covered, the tests are generated, rather than hand-written.

The tests are tested in the dependent CL. Tests are ignored in status
file and are un-ignored in dependent CL.

Change-Id: I5507f52229793fdf07da3066b49045fdb703a69d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/168829
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2020-10-26 12:01:16 +00:00
Daco Harkes d650ea90c4 [vm/ffi] Fix alignment of Uint64 fields in structs on some ABIs
Affected ABIs: ia32 Linux, ia32 Android, ia32 MacOS, and arm32 iOS.

Fixes https://github.com/dart-lang/sdk/issues/43693

Change-Id: I35f719abb69bb46d99f647d66847e3d5874d9371
Cq-Include-Trybots: luci.dart.try:vm-kernel-linux-debug-ia32-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/166843
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Auto-Submit: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2020-10-09 16:32:08 +00:00
Daco Harkes a7cb764aa4 [tests/ffi] Reinstate deleted test
Test was added in https://dart-review.googlesource.com/c/sdk/+/132843,
duplicated in https://dart-review.googlesource.com/c/sdk/+/132604,
and then got deleted in the non-NNBD folder, but that does not show up
in https://github.com/dart-lang/sdk/commits/master/tests/ffi_2.

Change-Id: Id3dbe2c065d3b74144f1007d35c8d4c8d962f528
Cq-Include-Trybots:dart/try:vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/161487
Auto-Submit: Daco Harkes <dacoharkes@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2020-09-02 12:12:52 +00:00
Daco Harkes ba18050bad [vm/ffi] Add missing SharedObject for regression test
Fix test for https://github.com/dart-lang/sdk/issues/43016 on Android.

Change-Id: I0dde428ff7b9618714e3ba3db3099a7d8fd1f0a5
Cq-Include-Trybots:dart/try: vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/158560
Reviewed-by: Tess Strickland <sstrickl@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
2020-08-14 17:21:16 +00:00
Daco Harkes 44fdd8f69d [vm/ffi] fix callbacks in --no-use-bare-instructions
Closes: https://github.com/dart-lang/sdk/issues/43035

Change-Id: I3a4e972f00ef7c7436d8ce5176bdc43c7636c25d
Cq-Include-Trybots:dart/try:vm-kernel-precomp-linux-debug-x64-try,vm-kernel-precomp-linux-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/158399
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Tess Strickland <sstrickl@google.com>
2020-08-14 12:49:37 +00:00
Daco Harkes 5781b4ccb3 [vm/ffi] Regression test for dartbug.com/43016
Regression test for https://github.com/dart-lang/sdk/issues/43016

Change-Id: I5787bd7abd8a7b5159a4535da59fe3918489ac8c
Cq-Include-Trybots:dart/try:vm-kernel-linux-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/158545
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Auto-Submit: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Aske Simon Christensen <askesc@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
2020-08-14 11:18:57 +00:00
Daco Harkes 9871331450 [vm/ffi] Clarify DynamicLibrary.open behavior
Both dlopen and LoadLibrary have caching behavior and only load the
dynamic library once into the DartVM, even with multiple isolates.

Change-Id: If16791baf28706c61acb1ecdae668485b264a373
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/156682
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Aske Simon Christensen <askesc@google.com>
2020-07-31 15:15:20 +00:00
Daco Harkes 4ea6d6e130 [tests/ffi] Re-enable commented out tests
Change-Id: I34eb75416a141c329cc1f92cccb958fd71e66c4e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/155605
Reviewed-by: Tess Strickland <sstrickl@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
2020-07-23 16:08:18 +00:00
Daco Harkes 202e32afeb [vm] Native API: Introduce Dart_FinalizableHandle
Introduces Dart_NewFinalizableHandle which does auto delete itself,
but does not allow accessing the weak referenced object.

Issue: https://github.com/dart-lang/sdk/issues/42312

Change-Id: I24ea732925122c453213c4fa3f629761c352f838
Cq-Include-Trybots:dart/try:vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64-try,app-kernel-linux-debug-x64-try,vm-kernel-linux-debug-ia32-try,vm-kernel-win-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-kernel-precomp-linux-debug-x64-try,vm-dartkb-linux-release-x64-abi-try,vm-kernel-precomp-android-release-arm64-try,vm-kernel-asan-linux-release-x64-try,vm-kernel-msan-linux-release-x64-try,vm-kernel-precomp-msan-linux-release-x64-try,vm-kernel-linux-release-simarm-try,vm-kernel-linux-release-simarm64-try,vm-kernel-precomp-android-release-arm_x64-try,vm-kernel-precomp-obfuscate-linux-release-x64-try,dart-sdk-linux-try,analyzer-analysis-server-linux-try,analyzer-linux-release-try,front-end-linux-release-x64-try,vm-kernel-precomp-win-release-x64-try,vm-kernel-mac-debug-x64-try,vm-precomp-ffi-qemu-linux-release-arm-try,vm-kernel-nnbd-linux-debug-x64-try,analyzer-nnbd-linux-release-try,front-end-nnbd-linux-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/154695
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2020-07-23 11:30:39 +00:00
Daco Harkes da39a4abff [vm/ffi] Remove try-catch from ffi trampoline if no handle scope
https://dart-review.googlesource.com/c/sdk/+/145591 introduced a try
catch into FFI calls to call ExitHandleScope on the exception path.
However, we only need this try-catch if we actually need to exit the
handle scope on the exception path, which is not the case if we have
no handles in the signature. So this CL makes the try catch optional.

This speeds up ffi calls without handles (tested on JIT x64):
FfiCall.Uint8x01(RunTime): 206.4801280066068 us.
->
FfiCall.Uint8x01(RunTime): 203.7240782236708 us.

Also adds a test that checks that an exception can still be propagated
with Dart_PropagateError from native code when the FFI trampoline has
no try catch.

Change-Id: I9fac7078381c60fb8055b64fff29ea364fbc948f
Cq-Include-Trybots: luci.dart.try:vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64-try,app-kernel-linux-debug-x64-try,vm-kernel-linux-debug-ia32-try,vm-kernel-win-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-kernel-precomp-linux-debug-x64-try,vm-dartkb-linux-release-x64-abi-try,vm-kernel-precomp-android-release-arm64-try,vm-kernel-asan-linux-release-x64-try,vm-kernel-msan-linux-release-x64-try,vm-kernel-precomp-msan-linux-release-x64-try,vm-kernel-linux-release-simarm-try,vm-kernel-linux-release-simarm64-try,vm-kernel-precomp-android-release-arm_x64-try,vm-kernel-precomp-obfuscate-linux-release-x64-try,dart-sdk-linux-try,analyzer-analysis-server-linux-try,analyzer-linux-release-try,front-end-linux-release-x64-try,vm-kernel-precomp-win-release-x64-try,vm-kernel-mac-debug-x64-try,vm-precomp-ffi-qemu-linux-release-arm-try,vm-kernel-nnbd-linux-debug-x64-try,analyzer-nnbd-linux-release-try,front-end-nnbd-linux-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/151239
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
2020-06-26 12:03:02 +00:00
Daco Harkes 7eac9f355e [vm/ffi] Expose a subset of dart_(native_)api.h for dynamic linking.
This CL introduces dart_api_dl.h which exposes a subset of dart_api.h
and dart_native_api.h for dynamic linking at runtime through the FFI.
Dynamic linking is done through including dart_api_dl.cc in a shared
library and passing NativeApi.initializeApiDLData to the init function.

This CL also includes Native API versioning to deal with possible
version skew between native api version against which native libraries
are compiled and the version in the DartVM the code is run on.

The subset of symbols in the CL includes handle related symbols, error
related symbols, handle scope symbols, and native port sumbols.

Design: http://go/dart-ffi-expose-dart-api

Closes: https://github.com/dart-lang/sdk/issues/40607
Closes: https://github.com/dart-lang/sdk/issues/36858
Closes: https://github.com/dart-lang/sdk/issues/41319
Closes: https://github.com/flutter/flutter/issues/46887
Closes: https://github.com/flutter/flutter/issues/47061

Misc:
Closes: https://github.com/dart-lang/sdk/issues/42260

Change-Id: I9e557808dbc99b341f23964cbddbb05f26d7a6c5
Cq-Include-Trybots: luci.dart.try:vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64-try,app-kernel-linux-debug-x64-try,vm-kernel-linux-debug-ia32-try,vm-kernel-win-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-kernel-precomp-linux-debug-x64-try,vm-dartkb-linux-release-x64-abi-try,vm-kernel-precomp-android-release-arm64-try,vm-kernel-asan-linux-release-x64-try,vm-kernel-msan-linux-release-x64-try,vm-kernel-precomp-msan-linux-release-x64-try,vm-kernel-linux-release-simarm-try,vm-kernel-linux-release-simarm64-try,vm-kernel-precomp-android-release-arm_x64-try,vm-kernel-precomp-obfuscate-linux-release-x64-try,dart-sdk-linux-try,analyzer-analysis-server-linux-try,analyzer-linux-release-try,front-end-linux-release-x64-try,vm-kernel-precomp-win-release-x64-try,vm-kernel-mac-debug-x64-try,vm-precomp-ffi-qemu-linux-release-arm-try,vm-kernel-nnbd-linux-debug-x64-try,analyzer-nnbd-linux-release-try,front-end-nnbd-linux-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/145592
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
2020-06-18 07:30:43 +00:00
Daco Harkes 8e2fa30e6c [test/ffi] Disable handles test on msan
Properly running msan on this test would require us to emit
msan_unpoison and msan_poison calls in the ffi trampolines to mark
explicitly what we are doing with the stack during ffi calls.

Issue: https://github.com/dart-lang/sdk/issues/42314

Also, this CL unifies the two status files a bit more.

Change-Id: I3a234b9c6dc430883a2d224b9ed3d7d48f4f678a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/151399
Reviewed-by: Tess Strickland <sstrickl@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
2020-06-17 08:26:39 +00:00
Daco Harkes 6544c69e23 [vm/ffi] Convert Objects to Dart_Handles in FFI calls
This includes support for calling Dart_PropagateError in native code
when doing FFI calls, and catching uncaught exceptions with Dart_IsError
when doing FFI callbacks.

The support for Dart_PropagateError adds a catch entry to the FFI
trampoline, which prevents inlining these trampolines in AOT. This
regresses the FfiCall benchmarks by 1-2% in AOT.

In addition, Dart_PropagateError requires maintaining a bit whether we
entered native/VM code from generated code through FFI or not. That way
we can do the proper transition on the exception path. When entering
generated code, we store this bit on the stack, right after the entry
frame.

Design: http://go/dart-ffi-handles

Issue: https://github.com/dart-lang/sdk/issues/36858
Issue: https://github.com/dart-lang/sdk/issues/41319

Change-Id: Idfd7ff69132fb29cc730931a4113d914d4437396
Cq-Include-Trybots: luci.dart.try:vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64-try,app-kernel-linux-debug-x64-try,vm-kernel-linux-debug-ia32-try,vm-kernel-win-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-kernel-precomp-linux-debug-x64-try,vm-dartkb-linux-release-x64-abi-try,vm-kernel-precomp-android-release-arm64-try,vm-kernel-asan-linux-release-x64-try,vm-kernel-linux-release-simarm-try,vm-kernel-linux-release-simarm64-try,vm-kernel-precomp-android-release-arm_x64-try,vm-kernel-precomp-obfuscate-linux-release-x64-try,dart-sdk-linux-try,analyzer-analysis-server-linux-try,analyzer-linux-release-try,front-end-linux-release-x64-try,vm-kernel-precomp-win-release-x64-try,vm-kernel-mac-debug-x64-try,vm-precomp-ffi-qemu-linux-release-arm-try,vm-kernel-nnbd-linux-debug-x64-try,analyzer-nnbd-linux-release-try,front-end-nnbd-linux-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/145591
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
2020-06-12 11:14:22 +00:00
Ryan Macnak 13ec6ef5ff [vm] Address TSAN errors in FFI tests.
Bug: https://github.com/dart-lang/sdk/issues/39611
Change-Id: I34556155d1d20ab48af952ddc4fdb6959db960d5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/140250
Reviewed-by: Daco Harkes <dacoharkes@google.com>
2020-03-23 21:19:36 +00:00
Daco Harkes c1467ab5d3 [vm/ffi] Change asFunction and lookFunction to extension methods
This prevents them from being called dynamically.
Moreover, it prevents asFunction from being called on a non-NativeFunction type argument, simplifying the amount of manual checks.

Note that this CL had to change the CFE and analzyer, and their tests (including mock_sdk) as well.

This can potentially be a breaking change, as the extension methods are only visible when `dart:ffi` is imported, while methods on objects are always visible.

Issue: https://github.com/dart-lang/sdk/issues/35903

Change-Id: I1e291f154228d5d9a34b21a022088bf493f6557d
Cq-Include-Trybots: luci.dart.try:vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64-try,app-kernel-linux-debug-x64-try,vm-kernel-linux-debug-ia32-try,vm-kernel-win-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-kernel-precomp-linux-debug-x64-try,vm-dartkb-linux-release-x64-abi-try,vm-kernel-precomp-android-release-arm64-try,vm-kernel-asan-linux-release-x64-try,vm-kernel-linux-release-simarm-try,vm-kernel-linux-release-simarm64-try,vm-kernel-precomp-android-release-arm_x64-try,vm-kernel-precomp-obfuscate-linux-release-x64-try,dart-sdk-linux-try,analyzer-analysis-server-linux-try,analyzer-linux-release-try,front-end-linux-release-x64-try,vm-kernel-precomp-win-release-x64-try,vm-kernel-mac-debug-x64-try,analyzer-nnbd-linux-release-try,dart2js-nnbd-linux-x64-chrome-try,ddc-nnbd-linux-release-chrome-try,front-end-nnbd-linux-release-x64-try,vm-kernel-nnbd-linux-debug-x64-try,vm-kernel-nnbd-linux-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/135463
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2020-02-14 15:46:37 +00:00
Daco Harkes 160614929a [vm/ffi] Fix sign extension for small int arguments on x64 Linux/MacOS
Sign extension of small integers is not symmetric between register arguments and register return values.
On Linux/MacOS x64, the caller is responsible for both the arguments and return value.

Previous tests only tested equality, which tests the equality of the lowest byte. However, when printing or using the small ints arithmetic all 4 bytes are used. The new tests pass back the number in a different format.
The callback tests test the number in Dart, so they did not suffer from implicit conversions. (TestTakeMaxUint8x10 in runtime/bin/ffi_test/ffi_test_functions.cc and tests/ffi_2/function_callbacks_test.dart.)

Fixes: https://github.com/dart-lang/sdk/issues/40537

Change-Id: I6a57cb7cb43206926eb101a66e3b2abfacec72a0
Cq-Include-Trybots: luci.dart.try:vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64-try,app-kernel-linux-debug-x64-try,vm-kernel-linux-debug-ia32-try,vm-kernel-win-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-kernel-precomp-linux-debug-x64-try,vm-dartkb-linux-release-x64-abi-try,vm-kernel-precomp-android-release-arm64-try,vm-kernel-asan-linux-release-x64-try,vm-kernel-linux-release-simarm-try,vm-kernel-linux-release-simarm64-try,vm-kernel-precomp-android-release-arm_x64-try,vm-kernel-precomp-obfuscate-linux-release-x64-try,dart-sdk-linux-try,analyzer-analysis-server-linux-try,analyzer-linux-release-try,front-end-linux-release-x64-try,vm-kernel-precomp-win-release-x64-try,vm-kernel-mac-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/134825
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2020-02-10 17:05:14 +00:00
Daco Harkes a71d60c1cf [vm/ffi] Support Linux on Arm 32-bit (hardfp)
Implement the hardfp calling convention.

Tested QEMU locally as the bots are not online yet.

tools/test.py -m debug -a arm -c dartkp --use-qemu ffi_2 ; tools/test.py -m debug -a arm -c dartk --use-qemu ffi_2
Test configuration:
    custom configuration(architecture: arm, compiler: dartkp, mode: debug, runtime: dart_precompiled, system: linux, use-qemu)
Suites tested: ffi_2
[00:45 | 100% | +  189 | -    0]
Test configuration:
    custom configuration(architecture: arm, compiler: dartk, mode: debug, runtime: vm, system: linux, use-qemu)
Suites tested: ffi_2
[09:52 | 100% | +  189 | -    0]

Fixes: https://github.com/dart-lang/sdk/issues/36309

Change-Id: Ic34175bdf9b9f359f1126aeb157362e76443921a
Cq-Include-Trybots: luci.dart.try:vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64-try,app-kernel-linux-debug-x64-try,vm-kernel-linux-debug-ia32-try,vm-kernel-win-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-kernel-precomp-linux-debug-x64-try,vm-dartkb-linux-release-x64-abi-try,vm-kernel-precomp-android-release-arm64-try,vm-kernel-asan-linux-release-x64-try,vm-kernel-linux-release-simarm-try,vm-kernel-linux-release-simarm64-try,vm-kernel-precomp-android-release-arm_x64-try,vm-kernel-precomp-obfuscate-linux-release-x64-try,dart-sdk-linux-try,analyzer-analysis-server-linux-try,analyzer-linux-release-try,front-end-linux-release-x64-try,vm-kernel-precomp-win-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/124136
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2020-02-05 14:22:59 +00:00
Daco Harkes 9920801424 [vm/ffi] Split FFI tests in NNBD and legacy
Closes: https://github.com/dart-lang/sdk/issues/40233

This CL creates nnbd versions of the tests and runs them on the nnbd sdk.

This CL does not (1) migrate sdk_nnbd/lib/ffi fully yet, and does not (2) fix all the tests/ffi (which is NNBD tests) yet.

Uncovered new issues:
Issue: https://github.com/dart-lang/sdk/issues/40234 nullptr should have type Pointer<Never>.
Issue: https://github.com/dart-lang/sdk/issues/40247 Structs need external fields.
Issue: https://github.com/dart-lang/sdk/issues/40271 Callbacks hit assert in debug.

Change-Id: Icb1b83577e03ed283165eb17703fc8dfc7fa5960
Cq-Include-Trybots: luci.dart.try:vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64-try,app-kernel-linux-debug-x64-try,vm-kernel-linux-debug-ia32-try,vm-kernel-win-debug-x64-try,vm-kernel-win-debug-ia32-try,vm-kernel-precomp-linux-debug-x64-try,vm-dartkb-linux-release-x64-abi-try,vm-kernel-precomp-android-release-arm64-try,vm-kernel-asan-linux-release-x64-try,vm-kernel-linux-release-simarm-try,vm-kernel-linux-release-simarm64-try,vm-kernel-precomp-android-release-arm_x64-try,vm-kernel-precomp-obfuscate-linux-release-x64-try,dart-sdk-linux-try,analyzer-analysis-server-linux-try,analyzer-linux-release-try,front-end-linux-release-x64-try,vm-kernel-precomp-win-release-x64-try,analyzer-nnbd-linux-release-try,dart2js-nnbd-linux-x64-chrome-try,ddc-nnbd-linux-release-chrome-try,front-end-nnbd-linux-release-x64-try,vm-kernel-nnbd-linux-debug-x64-try,vm-kernel-nnbd-linux-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/132604
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
2020-01-29 14:04:41 +00:00