This reland commit 5a32d8bc7c with a fix
for thread leak (Issue #56717): when `ThreadPool` is shutting down
asynchronously the last worker should detach itself to prevent
leaking associated low-level data structures, because no thread will
join it.
A hang in service isolate shutdown (caused by an existing bug) was fixed by commit 157a0dc7f9.
This CL turns native ports into a thin abstraction over underlying
thread pool instead of building them as full fledged MessageHandler.
This allows to easily implement a variation of native ports which can
handle messages concurrently with the given degree of concurrency.
This type of port can be used to greatly simplify implementation of
IOService - which previously had to do its own concurrency management
on top of "single threaded" native ports. This capability is exposed
as `Dart_NewConcurrentNativePort` API.
The new implementation is in general much cleaner then the old one
with one exception: `Dart_CloseNativePort` API has unfortunate design
where underlying message handler is destroyed asynchronously and
`Dart_CloseNativePort` returns immediately without waiting for pending
tasks to complete. Implementing this on top of `ThreadPool` requires
some changes to thread pool implementation.
Issue https://github.com/dart-lang/sdk/issues/55844
Closes https://github.com/dart-lang/sdk/issues/56717
TEST=ci
Change-Id: Ic68bfb60757685afd75c80a70cdec66cc13c149b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/385000
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
This reverts commit 5a32d8bc7c.
Reason for revert: DartIsolateTest.CanCreateServiceIsolate flutter engine unit test hangs on Windows.
Original change's description:
> [vm] Simplify implementation of native ports
>
> This CL turns native ports into a thin abstraction over underlying
> thread pool instead of building them as full fledged MessageHandler.
>
> This allows to easily implement a variation of native ports which can
> handle messages concurrently with the given degree of concurrency.
> This type of port can be used to greatly simplify implementation of
> IOService - which previously had to do its own concurrency management
> on top of "single threaded" native ports. This capability is exposed
> as `Dart_NewConcurrentNativePort` API.
>
> The new implementation is in general much cleaner then the old one
> with one exception: `Dart_CloseNativePort` API has unfortunate design
> where underlying message handler is destroyed asynchronously and
> `Dart_CloseNativePort` returns immediately without waiting for pending
> tasks to complete. Implementing this on top of `ThreadPool` requires
> some changes to thread pool implementation.
>
> Issue https://github.com/dart-lang/sdk/issues/55844
>
> TEST=ci
>
> Change-Id: I062040ff233e93962ae93684e9b044d8facdaffc
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/382163
> Commit-Queue: Slava Egorov <vegorov@google.com>
> Reviewed-by: Martin Kustermann <kustermann@google.com>
Change-Id: I7de27793a54072e974bf1a9f17a07c12159a202d
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/384481
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
This CL turns native ports into a thin abstraction over underlying
thread pool instead of building them as full fledged MessageHandler.
This allows to easily implement a variation of native ports which can
handle messages concurrently with the given degree of concurrency.
This type of port can be used to greatly simplify implementation of
IOService - which previously had to do its own concurrency management
on top of "single threaded" native ports. This capability is exposed
as `Dart_NewConcurrentNativePort` API.
The new implementation is in general much cleaner then the old one
with one exception: `Dart_CloseNativePort` API has unfortunate design
where underlying message handler is destroyed asynchronously and
`Dart_CloseNativePort` returns immediately without waiting for pending
tasks to complete. Implementing this on top of `ThreadPool` requires
some changes to thread pool implementation.
Issue https://github.com/dart-lang/sdk/issues/55844
TEST=ci
Change-Id: I062040ff233e93962ae93684e9b044d8facdaffc
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/382163
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
- `build/mac/find_sdk.py` can search for iPhone and Watch SDKs and their simulators
- `tools/build.py` supports `--os=ios` and `-os=ios_simulator` now. Treating simulator as a separate os to minimize changes and avoid an additional dimension for configs.
- `vm-mac-(release|debug)-arm64-try` tryjobs make sure a shared library builds successfully for ios.
TEST=ci
Change-Id: I76358ec8fd33752260bf0b8462da22a13cd7562e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/381623
Auto-Submit: Ivan Inozemtsev <iinozemtsev@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Throw error when a native service port cannot be created instead of
returning null. Fix the dispatch code to account for potential error
when creating the native service port and have it handle the error
in the catch block.
Should fix https://github.com/dart-lang/sdk/issues/55606
TEST=ci (a new test for this specific condition was hard to add)
Change-Id: I6ba6a66d36448ed3c323ee3350a2d4984288f359
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/378544
Commit-Queue: Siva Annamalai <asiva@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
This CL also removes ability to assign names to mutexes which was
added in c25ebfff96 but did not yield
any interesting data.
TEST=ci
CoreLibraryReviewExempt: Changes to dart:concurrent only.
Cq-Include-Trybots: luci.dart.try:vm-fuchsia-release-x64-try,vm-win-debug-x64-try,vm-win-release-x64-try,vm-aot-linux-product-x64-try
Change-Id: Id41e1d29832f6008e02f0a571ee67564e1a84224
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/375300
Reviewed-by: Martin Kustermann <kustermann@google.com>
This is just a cleanup without functional changes.
TEST=standalone/io
Cq-Include-Trybots: luci.dart.try:vm-win-debug-x64-try,pkg-win-release-try,analyzer-win-release-try,vm-win-release-x64-try
Change-Id: If50b55a1b8596b70899a08b91f48c97c861bf672
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/378921
Reviewed-by: Martin Kustermann <kustermann@google.com>
Move repetitive pattern into a helper function, no
changes in behavior.
TEST=standalone/io
Cq-Include-Trybots: luci.dart.try:vm-win-debug-x64-try,pkg-win-release-try,analyzer-win-release-try,vm-win-release-x64-try
Change-Id: I87ecc7f598d3c62f33b75c1e41c3e9cca2c3be98
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/378710
Reviewed-by: Martin Kustermann <kustermann@google.com>
Also use `OverlappedBuffer` destructor to close `client_` socket in
case of failures instead of closing it manually.
TEST=standalone/io
Cq-Include-Trybots: luci.dart.try:vm-win-debug-x64-try,pkg-win-release-try,analyzer-win-release-try,vm-win-release-x64-try
Change-Id: I546e450e0e9cce30e63d543633488ff7811be864
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/378709
Reviewed-by: Martin Kustermann <kustermann@google.com>
`Handle` objects which used by Windows specific implementation of
`EventHandler` have complicated life cycle and haphazard approach
to locking. This CL is the first in a series of CLs which attempts
to clean this up a bit - with a final goal to avoid recursively
locking `Handle::monitor_`.
In this CL we remove `Handle::EnsureInitialized` method which
was primarily responsible for associating `HANDLE` with
`EventHandler`'s completion port. Instead we move this logic
into `Handle` constructor.
We also cleanup reference counting logic applied to handles which
participate in asynchronous operations: instead of eagerly
incrementing reference count when `HANDLE` is associated with
the completion port we increment it whenever asynchronous operation
is in progress and decrement it when it completes. This makes the
code much simpler to reason about and avoids necessity to write
code like:
handle->Release();
handle->Release();
accompanied with comments explaining why we need to drop
two references due to eager increment.
Finally we cleanup the handling of provider-specific extensions
accessed through `WSAIoctl`'s `SIO_GET_EXTENSION_FUNCTION_POINTER`.
We were fetching them for every individual socket instead of
fetching them once and caching them globally.
TEST=ci
Cq-Include-Trybots: luci.dart.try:vm-win-debug-x64-try,pkg-win-release-try,analyzer-win-release-try,vm-win-release-x64-try
Change-Id: Ia8b391b713b1a1afb9df9e931eada6abf97f7854
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/378140
Reviewed-by: Martin Kustermann <kustermann@google.com>
Adds a new `@Array.variable()` to specify that the last element of
structs is a variable length inline array.
This CL does not add any checks for passing structs with variable
length inline arrays by value or directly calling them with
`AllocatorAlloc.call`. Instead, the implementation defaults to what
C does, allocate as if there are 0 elements in the variable length
inline array.
TEST=tests/ffi/*
CoreLibraryReviewExempt: VM only
Closes: https://github.com/dart-lang/sdk/issues/55964
Change-Id: I524d8a1d710b1a744b392e05fa884908c3ff1f12
Cq-Include-Trybots: dart/try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-asan-linux-release-x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-msan-linux-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-tsan-linux-release-x64-try,vm-aot-ubsan-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-aot-win-debug-x64-try,vm-aot-win-debug-x64c-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-arm64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-fuchsia-release-arm64-try,vm-fuchsia-release-x64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-arm64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-tsan-linux-release-arm64-try,vm-tsan-linux-release-x64-try,vm-ubsan-linux-release-arm64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-arm64-try,vm-win-debug-x64-try,vm-win-debug-x64c-try,vm-win-release-ia32-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/371960
Reviewed-by: Brian Wilkerson <brianwilkerson@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Lasse Nielsen <lrn@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
This CL extends the test generator to support struct pointers. For
callbacks, only synchronous use of pointers is supported in tests.
(Ownership of memory is _not_ passed to Dart.) We have plenty of other
tests passing ownership. These tests are meant to check the ABIs, for
which sync calls and callbacks suffice.
Split off https://dart-review.googlesource.com/c/sdk/+/371960 to make
the changes in that CL only relate to variable length arrays. That CL
will pass pointers to structs with variable length arrays.
TEST=tests/ffi/*
Change-Id: Ib8ff7b4e1d0f2451892ea693555803682bbf3bc0
tools/find_builders.dart ffi/function_structs_by_value_generated_args_test
Cq-Include-Trybots: dart/try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-asan-linux-release-x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-msan-linux-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-tsan-linux-release-x64-try,vm-aot-ubsan-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-aot-win-debug-x64-try,vm-aot-win-debug-x64c-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-arm64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-fuchsia-release-arm64-try,vm-fuchsia-release-x64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-arm64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-tsan-linux-release-arm64-try,vm-tsan-linux-release-x64-try,vm-ubsan-linux-release-arm64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-arm64-try,vm-win-debug-x64-try,vm-win-debug-x64c-try,vm-win-release-ia32-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/378704
Reviewed-by: Tess Strickland <sstrickl@google.com>
Reviewed-by: Liam Appelbe <liama@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
This is a reland of commit 1a614b1d86
Bug: b/348315159
This CL removes the native assets fallback from the VM.
TEST=pkg/dartdev/test/native_assets/build_test.dart
TEST=tests/ffi/native_assets/asset_relative_test.dart
Closes: https://github.com/dart-lang/sdk/issues/55523
Closes: https://github.com/dart-lang/sdk/issues/55925
Change-Id: Ib40d3a8c7d4b7c6226b6e4db4b3016cc32591239
Cq-Include-Trybots: dart/try:vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-aot-win-debug-x64-try,vm-aot-win-debug-x64c-try,pkg-linux-debug-try,pkg-linux-release-arm64-try,pkg-mac-release-try,pkg-mac-release-arm64-try,pkg-win-release-try,pkg-win-release-arm64-try,vm-aot-asan-linux-release-x64-try,vm-asan-linux-release-x64-try,vm-aot-msan-linux-release-x64-try,vm-msan-linux-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/372500
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
* Fix `ToWinAPIFilePath` handling of UNC paths. This function did
not handle UNC paths correctly: it was prefixing them with
`\\?\` instead of `\\?\UNC\`;
* Fix issue in `File::Copy` forcing the wrong path to long form:
temporary files are created next to the destination, so it
is destination (`new_path`) that should be forced into long form and
not the source;
* Fix issue in `GetDirectoryPath` not preserving the path separator.
The newly added test should cover all these corner cases to prevent
future regressions.
TEST=standalone/io/regress_56049
Fixes https://github.com/dart-lang/sdk/issues/56049
Fixed: 56049
Change-Id: I4970efd0674989b8cc291778cf87806b0ea318ff
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/372400
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
This CL fixes symlink resolution by properly decoding a URI to a file
path before doing symlink resolution (on the file path).
File paths have spaces, but URIs can be encoded with percent encoding.
Before resolving the relative paths, the symlink-resolved script path
is encoded as URI again. This encoding is trivial (percentage
encoding is optional), so it only needs to prepend a scheme.
Also, this CL improves the error message for the relative path
resolution if the dylib cannot be found to include the script uri.
TEST=tests/ffi/native_assets/asset_relative_test.dart
Closes: https://github.com/dart-lang/sdk/issues/56053
Change-Id: I3ac9748e971e6eacbe14f3485bf3a3943d587d7d
Cq-Include-Trybots: dart/try:vm-aot-asan-linux-release-x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-msan-linux-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-tsan-linux-release-x64-try,vm-aot-ubsan-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-aot-win-debug-x64-try,vm-aot-win-debug-x64c-try,vm-asan-linux-release-arm64-try,vm-msan-linux-release-arm64-try,vm-tsan-linux-release-arm64-try,vm-ubsan-linux-release-arm64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/372421
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Move optimized handling of Double.truncateToDouble, floorToDouble,
ceilToDouble, floor and ceil from flow graph construction to
the canonicalization pass.
This is needed in order to maintain stable flow graphs between
app-jit snapshot training (which is performed with --target_unknown_cpu)
and running from app-jit snapshot without --target_unknown_cpu.
TEST=ci (vm-appjit-*)
Fixes https://github.com/dart-lang/sdk/issues/56018
Change-Id: I7e3e4ce917984f52db239506ea3de0c55af00567
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/372140
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Before this CL, becase of optimizations, coverage of some of these goes
away, making coverage unreliable.
I believe this fixes the issues for "regular" runs (at least it seems to
be stable on the CFE coverage tests).
If setting `--optimization-counter-threshold=-1` there'll still be
trouble though and we would have to also insert these calls in the start
of FunctionBody and the start of FieldInitializer for it to produce the
same results.
TEST=pkg/vm_service/test/coverage_instance_call_after_optimization_test.dart,pkg/vm_service/test/coverage_static_call_after_optimization_test.dart
Bug: https://github.com/dart-lang/sdk/issues/42061
Bug: https://github.com/dart-lang/sdk/issues/55959
Bug: https://github.com/dart-lang/sdk/issues/56018
Change-Id: I34947f0d4b123e52ce67b71a195782d31e4bda16
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/370501
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
This CL moves native assets resolution to the embedder.
The Dart VM looks up the asset path (for example
`['relative', 'foo.so']`) with the asset id. The embedder defines
callbacks for asset loading, and returns a handle to the dylib.
Then the VM calls the embedder again with `dlsym` to lookup the symbol.
The Dart VM & kernel compiler are responsible for the asset id to
asset path mapping. The kernel compiler compiles it into the snapshot
and the VM looks it up in the snapshot.
Relative paths are resolved relative to the isolate script uri (kernel
snapshot, jit snapshot, aot snapshot, or `dart compile exe` result).
The embedder is responsible for remembering the script uri it set when
spawning the isolate group.
This CL does not add `dlclose` or `dladdr` embedder callbacks yet.
Bug: https://github.com/dart-lang/sdk/issues/55521
Bug: https://github.com/dart-lang/sdk/issues/55966
TEST=pkg/dartdev/test/native_assets/build_test.dart
TEST=tests/ffi/native_assets/asset_relative_test.dart
Bug: https://github.com/dart-lang/sdk/issues/55410
Bug: https://github.com/dart-lang/sdk/issues/55523
Bug: https://github.com/dart-lang/sdk/issues/55207
Change-Id: I75ec4a368c5fb3d2f76b03771e796ff56bcac941
Cq-Include-Trybots: dart/try:vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-aot-win-debug-x64-try,vm-aot-win-debug-x64c-try,pkg-linux-debug-try,pkg-linux-release-arm64-try,pkg-mac-release-try,pkg-mac-release-arm64-try,pkg-win-release-try,pkg-win-release-arm64-try,vm-aot-asan-linux-release-x64-try,vm-asan-linux-release-x64-try,vm-aot-msan-linux-release-x64-try,vm-msan-linux-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/361881
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
This will enable the standalone embedder to resolve uris in a follow
up CL:
https://dart-review.googlesource.com/c/sdk/+/361881
The implementation was using zones for allocation, this CL switches
that to using `malloc`. The caller is responsible for freeing the
memory if resolving succeeds.
This CL removes `Dart_DefaultCanonicalizeUrl`.
We don't have a run_platform_tests, so this CL keeps the tests in
run_vm_tests.
TEST=vm/cc/ParseUri
TEST=vm/cc/ResolveUri
TEST=tests/ffi/native_assets/asset_relative_test.dart
Bug: https://github.com/dart-lang/sdk/issues/55523
Change-Id: Ifb300d8164eb50506f22ce619fad0811f74ef34c
Cq-Include-Trybots: luci.dart.try:vm-aot-asan-linux-release-x64-try,vm-aot-msan-linux-release-x64-try,vm-asan-linux-release-arm64-try,vm-asan-linux-release-x64-try,vm-msan-linux-release-arm64-try,vm-msan-linux-release-x64-try,pkg-linux-debug-try,pkg-linux-release-arm64-try,pkg-linux-release-try,pkg-mac-release-try,pkg-mac-release-arm64-try,pkg-win-release-arm64-try,pkg-win-release-try,vm-aot-linux-debug-x64-try,vm-linux-debug-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-aot-win-debug-x64-try,vm-win-debug-arm64-try,vm-mac-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/368423
Reviewed-by: Martin Kustermann <kustermann@google.com>
Ryan Macnak