Factor out the .eh_frame section content generation from ElfWriter,
putting in as a static method in Dwarf instead.
Create a __eh_frame section in the __TEXT section of non-MacOS Mach-O
snapshots with the same information as .eh_frame in ELF snapshots.
TEST=ci
Issue: https://github.com/dart-lang/sdk/issues/60307
Change-Id: I4e123f586e59e1263385c838b755c1575a6330eb
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-x64-try,vm-mac-release-arm64-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-dwarf-linux-product-x64-try,vm-linux-debug-x64-try,vm-mac-debug-arm64-try,vm-fuchsia-release-x64-try,vm-fuchsia-release-arm64-try,vm-linux-debug-ia32-try,vm-aot-linux-debug-simarm_x64-try,vm-aot-linux-debug-simriscv32-try,vm-aot-linux-debug-simriscv64-try,vm-aot-linux-release-simarm_x64-try,vm-gcc-linux-try,vm-ubsan-linux-release-arm64-try,vm-aot-win-release-arm64-try,vm-aot-win-release-x64-try,vm-win-release-x64-try,vm-win-release-arm64-try,vm-aot-win-debug-arm64-try,vm-win-debug-arm64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/431600
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Adds a Mach-O loader, similar to the existing ELF loader, that
can be used on platforms that do not support loading Mach-O dynamic
libraries via dlopen().
Other changes:
* Renames the --force-load-elf-from-memory command line argument
to --force-load-from-memory.
* Use CPU_TYPE_ANY and CPU_SUBTYPE_ANY for architectures that do
not have specific constants in <mach/machine.h> (e.g., RISCV),
as the snapshot header check after loading also catches architecture
mismatches.
* Emit the unwinding information at the end of the text segment
for Windows Mach-O snapshots as is done for ELF ones.
TEST=vm/dart/unobfuscated_static_symbols_test
vm/dart/use_dwarf_stack_traces_flag_test
Issue: https://github.com/dart-lang/sdk/issues/60307
Change-Id: I34a2a334f47d18d5c4f4a712956e71fd0ac94024
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-x64-try,vm-mac-release-arm64-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-dwarf-linux-product-x64-try,vm-linux-debug-x64-try,vm-mac-debug-arm64-try,vm-fuchsia-release-x64-try,vm-fuchsia-release-arm64-try,vm-linux-debug-ia32-try,vm-aot-linux-debug-simarm_x64-try,vm-aot-linux-debug-simriscv32-try,vm-aot-linux-debug-simriscv64-try,vm-aot-linux-release-simarm_x64-try,vm-gcc-linux-try,vm-ubsan-linux-release-arm64-try,vm-aot-win-release-arm64-try,vm-aot-win-release-x64-try,vm-win-release-x64-try,vm-win-release-arm64-try,vm-aot-win-debug-arm64-try,vm-win-debug-arm64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/430100
Reviewed-by: Slava Egorov <vegorov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
This is a reland of commit 38ef28a058
Fixes:
* Fix comparisons in ASSERT_EQUAL statements on 32-bit architectures.
* Take simulated architectures into account when deciding whether
to use dlopen() for native shared object formats.
* Fix struct/field name collision for GCC.
* Use CPU_TYPE_ANY/CPU_SUBTYPE_ANY for architectures that do not
have more specific cpu_type_t/cpu_subtype_t constants defined.
Original change's description:
> [vm, gen_snapshot] Add app-aot-macho-dylib option for AOT snapshots.
>
> This is the initial framework for creating snapshots as Mach-O dynamic
> libraries. Note that this framework is not 100% feature complete
> compared to generating Mach-O snapshots via assembly. In particular,
> the directly-compiled Mach-O dylib does not yet contain compact
> unwinding information.
>
> Other changes:
>
> * Adds UuidCommand to the native_stack_traces package's Mach-O reader,
> which now appropriately returns the UUID as the build ID for Mach-O
> shared objects.
>
> * Adds Utils::Basename(path) for portably retrieving the basename
> from a path. (Returns nullptr for all arguments where it is not
> currently implemented on Fuchsia or Windows.)
>
> * Adjusts vm/timeline.h to avoid pulling in <mach_o/loader.h> on MacOS,
> as that interferes with uses of the namespaced Mach-O definitions
> in platform/mach_o.h.
>
> * Only attempt to dlopen() a snapshot if ELF is the native format
> for the host platform or the snapshot is not an ELF shared object.
> If dlopen() is used, report the error message if it fails rather
> than attempting to manually load the snapshot as an ELF shared object.
>
> * Fix the magic number stored in DylibAppSnapshot for loaded non-ELF
> dynamic libraries.
>
> * Remove the detection of reverse-endian Mach-O magic numbers in
> DartUtils::SniffForMagicNumber(), since all our Mach-O related code
> assumes host-endian Mach-O files and so there's no point other than
> to give a slightly better error message when failing.
>
> TEST=vm/dart/exported_symbols_test
> vm/dart/unobfuscated_static_symbols_test
> vm/dart/use_dwarf_stack_traces_flag_test
> vm/cc/CanDetectMachOFiles
>
> Issue: https://github.com/dart-lang/sdk/issues/60307
> Change-Id: Idf5b49d6c6d035ab033509613212b95520d65965
> Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-x64-try,vm-mac-release-arm64-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-dwarf-linux-product-x64-try,vm-linux-debug-x64-try,vm-mac-debug-arm64-try,vm-fuchsia-release-x64-try,vm-fuchsia-release-arm64-try
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/415020
> Reviewed-by: Slava Egorov <vegorov@google.com>
> Commit-Queue: Tess Strickland <sstrickl@google.com>
TEST=vm/dart/exported_symbols_test
vm/dart/unobfuscated_static_symbols_test
vm/dart/use_dwarf_stack_traces_flag_test
vm/cc/CanDetectMachOFiles
ci on trybots that failed on the original CL
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-x64-try,vm-mac-release-arm64-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-dwarf-linux-product-x64-try,vm-linux-debug-x64-try,vm-mac-debug-arm64-try,vm-fuchsia-release-x64-try,vm-fuchsia-release-arm64-try,vm-linux-debug-ia32-try,vm-aot-linux-debug-simarm_x64-try,vm-aot-linux-debug-simriscv32-try,vm-aot-linux-debug-simriscv64-try,vm-aot-linux-release-simarm_x64-try,vm-gcc-linux-try,vm-ubsan-linux-release-arm64-try
Change-Id: Iaffea0ddc6173100c8b5b2a9fe46c45f4f611a2e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/431240
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
This reverts commit 38ef28a058.
Reason for revert: Breaks build on debug mode 32-bit arches,
release mode simarm_x64
Issue: https://github.com/dart-lang/sdk/issues/60307
Original change's description:
> [vm, gen_snapshot] Add app-aot-macho-dylib option for AOT snapshots.
>
> This is the initial framework for creating snapshots as Mach-O dynamic
> libraries. Note that this framework is not 100% feature complete
> compared to generating Mach-O snapshots via assembly. In particular,
> the directly-compiled Mach-O dylib does not yet contain compact
> unwinding information.
>
> Other changes:
>
> * Adds UuidCommand to the native_stack_traces package's Mach-O reader,
> which now appropriately returns the UUID as the build ID for Mach-O
> shared objects.
>
> * Adds Utils::Basename(path) for portably retrieving the basename
> from a path. (Returns nullptr for all arguments where it is not
> currently implemented on Fuchsia or Windows.)
>
> * Adjusts vm/timeline.h to avoid pulling in <mach_o/loader.h> on MacOS,
> as that interferes with uses of the namespaced Mach-O definitions
> in platform/mach_o.h.
>
> * Only attempt to dlopen() a snapshot if ELF is the native format
> for the host platform or the snapshot is not an ELF shared object.
> If dlopen() is used, report the error message if it fails rather
> than attempting to manually load the snapshot as an ELF shared object.
>
> * Fix the magic number stored in DylibAppSnapshot for loaded non-ELF
> dynamic libraries.
>
> * Remove the detection of reverse-endian Mach-O magic numbers in
> DartUtils::SniffForMagicNumber(), since all our Mach-O related code
> assumes host-endian Mach-O files and so there's no point other than
> to give a slightly better error message when failing.
>
> TEST=vm/dart/exported_symbols_test
> vm/dart/unobfuscated_static_symbols_test
> vm/dart/use_dwarf_stack_traces_flag_test
> vm/cc/CanDetectMachOFiles
>
> Issue: https://github.com/dart-lang/sdk/issues/60307
> Change-Id: Idf5b49d6c6d035ab033509613212b95520d65965
> Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-x64-try,vm-mac-release-arm64-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-dwarf-linux-product-x64-try,vm-linux-debug-x64-try,vm-mac-debug-arm64-try,vm-fuchsia-release-x64-try,vm-fuchsia-release-arm64-try
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/415020
> Reviewed-by: Slava Egorov <vegorov@google.com>
> Commit-Queue: Tess Strickland <sstrickl@google.com>
Issue: https://github.com/dart-lang/sdk/issues/60307
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-x64-try,vm-mac-release-arm64-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-dwarf-linux-product-x64-try,vm-linux-debug-x64-try,vm-mac-debug-arm64-try,vm-fuchsia-release-x64-try,vm-fuchsia-release-arm64-try
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Change-Id: Iff2ab4c84a513a184784129f456bd357ae8e3a67
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/431220
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Slava Egorov <vegorov@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: Tess Strickland <sstrickl@google.com>
This is the initial framework for creating snapshots as Mach-O dynamic
libraries. Note that this framework is not 100% feature complete
compared to generating Mach-O snapshots via assembly. In particular,
the directly-compiled Mach-O dylib does not yet contain compact
unwinding information.
Other changes:
* Adds UuidCommand to the native_stack_traces package's Mach-O reader,
which now appropriately returns the UUID as the build ID for Mach-O
shared objects.
* Adds Utils::Basename(path) for portably retrieving the basename
from a path. (Returns nullptr for all arguments where it is not
currently implemented on Fuchsia or Windows.)
* Adjusts vm/timeline.h to avoid pulling in <mach_o/loader.h> on MacOS,
as that interferes with uses of the namespaced Mach-O definitions
in platform/mach_o.h.
* Only attempt to dlopen() a snapshot if ELF is the native format
for the host platform or the snapshot is not an ELF shared object.
If dlopen() is used, report the error message if it fails rather
than attempting to manually load the snapshot as an ELF shared object.
* Fix the magic number stored in DylibAppSnapshot for loaded non-ELF
dynamic libraries.
* Remove the detection of reverse-endian Mach-O magic numbers in
DartUtils::SniffForMagicNumber(), since all our Mach-O related code
assumes host-endian Mach-O files and so there's no point other than
to give a slightly better error message when failing.
TEST=vm/dart/exported_symbols_test
vm/dart/unobfuscated_static_symbols_test
vm/dart/use_dwarf_stack_traces_flag_test
vm/cc/CanDetectMachOFiles
Issue: https://github.com/dart-lang/sdk/issues/60307
Change-Id: Idf5b49d6c6d035ab033509613212b95520d65965
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-x64-try,vm-mac-release-arm64-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-dwarf-linux-product-x64-try,vm-linux-debug-x64-try,vm-mac-debug-arm64-try,vm-fuchsia-release-x64-try,vm-fuchsia-release-arm64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/415020
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Attempting to use the native definition when available causes problems
when cross-compiling, and the original attempts to avoid this broke in
subtle ways. Instead, just make a non-native definition that mimics
the native definition and use that with reinterpret_casts when sending
uses of the non-native definition to Windows API calls that expect
a value of the native definition.
TEST=windows ci
Fixes: https://github.com/dart-lang/sdk/issues/60771
Change-Id: I347fcaf8cccd809a3d8e6f041cc7e360b0b8226e
Cq-Include-Trybots: luci.dart.try:vm-aot-win-release-arm64-try,vm-aot-win-release-x64-try,vm-win-release-arm64-try,vm-win-release-x64-try,vm-aot-win-debug-arm64-try,vm-aot-win-debug-x64-try,pkg-win-release-try,pkg-win-release-arm64-try,dart-sdk-win-try,dart-sdk-win-arm64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/430400
Reviewed-by: Slava Egorov <vegorov@google.com>
This CL pulls out the refactorings used to support the new MachOWriter
in a followup CL to allow them to be reviewed separately.
Rename Elf -> ElfWriter. Also rename model classes used by ElfWriter
for concepts that exist both in ELF and Mach-O to ElfX. For example,
the old ELF-specific SymbolTable is renamed to ElfSymbolTable.
Adds SharedObjectWriter to serve as a base class for both ElfWriter
and the upcoming MachOWriter.
Adds a new AbstractWriteStream that serves as a common superclass
of both BaseWriteStream and SharedObjectWriter::WriteStream and
allows the creation of fully delegating WriteStreams that do not
maintain a local buffer.
Abstract the old Elf::SymbolData class into
SharedObjectWriter::SymbolData, which stores an enum value as the type
of the symbol instead of storing the ELF encoding of the type.
Rename the DwarfElfStream (which actually wasn't ELF specific, as
all the ELF-specific DWARF information is handled by ElfWriter) to
DwarfSharedObjectStream and put it in a separate header file.
Rename Image::compiled_to_elf() to Image::compiled_to_shared_object()
and add a separate Image::compiled_to_elf() that checks for the ELF
magic value at the DSO base. Also add Image::shared_object_start()
and Image::build_id_start() to return pointers to the DSO base and
the build ID note, respectively.
Refactor Image::build_id() and Image::build_id_length() to check
compiled_for_elf() prior to decoding the data pointed to by
build_id_start() as an ELF note section.
Create an AOTSnapshotType enum to specific the snapshot writer to use
in CreateAppAOTSnapshot instead of using an as_elf boolean.
TEST=refactorings, so existing tests on ci
Change-Id: Ia3ab37a4dff93b6e00390b123753be5a51fbdaaa
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-x64-try,vm-aot-mac-release-arm64-try,vm-aot-dwarf-linux-product-x64-try,vm-linux-debug-x64-try,vm-mac-debug-arm64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/421301
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
This is a reland of commit 7ae1d75869
TEST=manual, no watchOS SDK on bots yet.
Original change's description:
> watchOS target os
>
> - add support for --os=watchos | --os=watchos_simulator
> - do not depend on perfetto if dart_support_perfetto is false
> - change the default value for dart_support_perfetto to !is_watchos
>
> To minimize changes and to avoid modifying third-party zlib BUILD file, is_watchos implies is_ios for now.
>
> TEST=manual, no watchOS SDK on bots yet.
>
> Cq-Include-Trybots: luci.dart.try:vm-mac-debug-arm64-try,vm-mac-release-arm64-try
> Change-Id: If2130068ef546162a07a9ba53f94b11ff25fb565
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/415021
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
> Commit-Queue: Ivan Inozemtsev <iinozemtsev@google.com>
Cq-Include-Trybots: luci.dart.try:vm-mac-debug-arm64-try,vm-mac-release-arm64-try
Change-Id: I2b50f7eb4da173627aeb4833b88a08712b278d9f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/425523
Commit-Queue: Ivan Inozemtsev <iinozemtsev@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
This reverts commit 7ae1d75869.
Reason for revert: breaks flutter build
Original change's description:
> watchOS target os
>
> - add support for --os=watchos | --os=watchos_simulator
> - do not depend on perfetto if dart_support_perfetto is false
> - change the default value for dart_support_perfetto to !is_watchos
>
> To minimize changes and to avoid modifying third-party zlib BUILD file, is_watchos implies is_ios for now.
>
> TEST=manual, no watchOS SDK on bots yet.
>
> Cq-Include-Trybots: luci.dart.try:vm-mac-debug-arm64-try,vm-mac-release-arm64-try
> Change-Id: If2130068ef546162a07a9ba53f94b11ff25fb565
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/415021
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
> Commit-Queue: Ivan Inozemtsev <iinozemtsev@google.com>
Cq-Include-Trybots: luci.dart.try:vm-mac-debug-arm64-try,vm-mac-release-arm64-try
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Change-Id: I4bede9ede04ac3c533b8ff7ae2f1adbd84eb2397
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/425262
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Alexander Thomas <athom@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
Auto-Submit: Ivan Inozemtsev <iinozemtsev@google.com>
- add support for --os=watchos | --os=watchos_simulator
- do not depend on perfetto if dart_support_perfetto is false
- change the default value for dart_support_perfetto to !is_watchos
To minimize changes and to avoid modifying third-party zlib BUILD file, is_watchos implies is_ios for now.
TEST=manual, no watchOS SDK on bots yet.
Cq-Include-Trybots: luci.dart.try:vm-mac-debug-arm64-try,vm-mac-release-arm64-try
Change-Id: If2130068ef546162a07a9ba53f94b11ff25fb565
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/415021
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Ivan Inozemtsev <iinozemtsev@google.com>
HWASAN tags stack regions (and stack pointers) with 0,
but also selectively tags some stack allocations with
non-zero tags. These tags need to be cleared when
allocation goes out of scope so that stack can be
later safely reused - otherwise stale tags confuse
HWASAN instrumentation later.
This CL teaches Exceptions::JumpToFrame to do that.
CL also adds basic build changes to support building
with HWASAN, though I don't wire it into CI for now
as it can only really run on Android ARM64
TEST=manually on Android phone and on internal tests
Bug: b/374433249
Change-Id: I678443dc1de693c999a226bee0d71a6f3582d4f9
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/421180
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Auto-Submit: Slava Egorov <vegorov@google.com>
This will play better with native tooling compared to Dart_LoadELF.
Cq-Include-Trybots: luci.dart.try:vm-fuchsia-release-arm64-try,vm-fuchsia-release-x64-try
Change-Id: If5265d079200d7be754b82a9775d977aa66c15f9
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/419684
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Zijie He <zijiehe@google.com>
Intern the following fields:
* Category names
* Event labels
* Debug annotation keys and values
This significantly reduces the size of the timeline
(e.g. a timeline containing 60k slices goes from 15Mb
to 5Mb timeline)
This relands commit f2614d24f8
with a fix for Android build.
TEST=ci and manually
Cq-Include-Trybots: luci.dart.try:vm-ffi-android-release-arm64c-try
Change-Id: I88d4c5e1142ff66b270a22b82bacd1e9313fa953
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/418220
Reviewed-by: Derek Xu <derekx@google.com>
Intern the following fields:
* Category names
* Event labels
* Debug annotation keys and values
This significantly reduces the size of the timeline
(e.g. a timeline containing 60k slices goes from 15Mb
to 5Mb timeline)
TEST=ci and manually
Change-Id: I59e850279b6714b8b75c5b8e79e6a8b0a981a261
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/417202
Reviewed-by: Derek Xu <derekx@google.com>
This is a reland of commit e6d55c6c2f
Reworks guards for 64-bit Windows unwinding support code so that
it is not included in any 64-bit Windows gen_snapshot executable that
targets a non-64-bit Windows platform.
TEST=ci, build only
Original change's description:
> Add gen_snapshot binaries, producing Linux ARM64/x64 snapshots
>
> Inspired by `//runtime/bin:gen_snapshot_fuchsia`, required for go/dart-cross-compile.
>
> TEST=ci, build only
>
> Change-Id: Ie521c984a8f44d25f3f78d946af9416582a572dc
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/410402
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
> Commit-Queue: Ivan Inozemtsev <iinozemtsev@google.com>
Change-Id: I4f1323ac8f7ab215c2dffeef188f70c466fb62ea
Cq-Include-Trybots: luci.dart.try:vm-win-release-arm64-try,vm-win-release-ia32-try,vm-win-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/413402
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ivan Inozemtsev <iinozemtsev@google.com>
Commit-Queue: Ivan Inozemtsev <iinozemtsev@google.com>
We have to place these classes into a platform namespace to avoid
conflicts with vm/lockers.h.
To avoid changes to files in bin we simply change bin/lockers.h
to introduce bin::{Monitor,Mutex}Locker aliases for
platform::{Monitor,Mutex}Locker.
TEST=ci
R=kustermann@google.com
Change-Id: I975f24a8499ea85d0ba887180a81287fcf4101bb
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/405460
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
The same bundling that is used for `dart build` is now also used for
`dart test` and `dart run`, except that the output directory is
`.dart_tool/native_assets`. This way all native code assets are placed
next to each other in the `lib` directory, and loaded from there
instead of loading them in place from where the build/link hooks
placed them. By standardizing on this layout the different modes of
running dart code that support native assets can use the same
mechanisms to support dynamic linking between libraries.
On macOS, install names of dylibs are rewritten to support dynamic
linking, similar to the changes in
https://github.com/flutter/flutter/pull/153054.
On Windows, loading of DLLs is altered so that the directory of the DLL
that is being loaded is considered when loading dependent DLLs.
Tests are added to verify that dynamic linking works as expected.
TEST=pkg/dartdev/test/native_assets/{build,run,test}_test.dart
R=mosum@google.com
Related: https://github.com/dart-lang/native/issues/190
Fixes: https://github.com/dart-lang/sdk/issues/56459
Change-Id: Ie4a41e5b7382ab1cea39e93d29d085bf9986828b
Cq-Include-Trybots: luci.dart.try:pkg-linux-debug-try,pkg-linux-release-arm64-try,pkg-linux-release-try,pkg-mac-release-arm64-try,pkg-mac-release-try,pkg-win-release-arm64-try,pkg-win-release-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/381580
Reviewed-by: Moritz Sümmermann <mosum@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
This reland commit 5a32d8bc7c with a fix
for thread leak (Issue #56717): when `ThreadPool` is shutting down
asynchronously the last worker should detach itself to prevent
leaking associated low-level data structures, because no thread will
join it.
A hang in service isolate shutdown (caused by an existing bug) was fixed by commit 157a0dc7f9.
This CL turns native ports into a thin abstraction over underlying
thread pool instead of building them as full fledged MessageHandler.
This allows to easily implement a variation of native ports which can
handle messages concurrently with the given degree of concurrency.
This type of port can be used to greatly simplify implementation of
IOService - which previously had to do its own concurrency management
on top of "single threaded" native ports. This capability is exposed
as `Dart_NewConcurrentNativePort` API.
The new implementation is in general much cleaner then the old one
with one exception: `Dart_CloseNativePort` API has unfortunate design
where underlying message handler is destroyed asynchronously and
`Dart_CloseNativePort` returns immediately without waiting for pending
tasks to complete. Implementing this on top of `ThreadPool` requires
some changes to thread pool implementation.
Issue https://github.com/dart-lang/sdk/issues/55844
Closes https://github.com/dart-lang/sdk/issues/56717
TEST=ci
Change-Id: Ic68bfb60757685afd75c80a70cdec66cc13c149b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/385000
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
This reverts commit 5a32d8bc7c.
Reason for revert: DartIsolateTest.CanCreateServiceIsolate flutter engine unit test hangs on Windows.
Original change's description:
> [vm] Simplify implementation of native ports
>
> This CL turns native ports into a thin abstraction over underlying
> thread pool instead of building them as full fledged MessageHandler.
>
> This allows to easily implement a variation of native ports which can
> handle messages concurrently with the given degree of concurrency.
> This type of port can be used to greatly simplify implementation of
> IOService - which previously had to do its own concurrency management
> on top of "single threaded" native ports. This capability is exposed
> as `Dart_NewConcurrentNativePort` API.
>
> The new implementation is in general much cleaner then the old one
> with one exception: `Dart_CloseNativePort` API has unfortunate design
> where underlying message handler is destroyed asynchronously and
> `Dart_CloseNativePort` returns immediately without waiting for pending
> tasks to complete. Implementing this on top of `ThreadPool` requires
> some changes to thread pool implementation.
>
> Issue https://github.com/dart-lang/sdk/issues/55844
>
> TEST=ci
>
> Change-Id: I062040ff233e93962ae93684e9b044d8facdaffc
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/382163
> Commit-Queue: Slava Egorov <vegorov@google.com>
> Reviewed-by: Martin Kustermann <kustermann@google.com>
Change-Id: I7de27793a54072e974bf1a9f17a07c12159a202d
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/384481
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
This CL turns native ports into a thin abstraction over underlying
thread pool instead of building them as full fledged MessageHandler.
This allows to easily implement a variation of native ports which can
handle messages concurrently with the given degree of concurrency.
This type of port can be used to greatly simplify implementation of
IOService - which previously had to do its own concurrency management
on top of "single threaded" native ports. This capability is exposed
as `Dart_NewConcurrentNativePort` API.
The new implementation is in general much cleaner then the old one
with one exception: `Dart_CloseNativePort` API has unfortunate design
where underlying message handler is destroyed asynchronously and
`Dart_CloseNativePort` returns immediately without waiting for pending
tasks to complete. Implementing this on top of `ThreadPool` requires
some changes to thread pool implementation.
Issue https://github.com/dart-lang/sdk/issues/55844
TEST=ci
Change-Id: I062040ff233e93962ae93684e9b044d8facdaffc
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/382163
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
This CL also removes ability to assign names to mutexes which was
added in c25ebfff96 but did not yield
any interesting data.
TEST=ci
CoreLibraryReviewExempt: Changes to dart:concurrent only.
Cq-Include-Trybots: luci.dart.try:vm-fuchsia-release-x64-try,vm-win-debug-x64-try,vm-win-release-x64-try,vm-aot-linux-product-x64-try
Change-Id: Id41e1d29832f6008e02f0a571ee67564e1a84224
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/375300
Reviewed-by: Martin Kustermann <kustermann@google.com>
This is a reland of commit 1a614b1d86
Bug: b/348315159
This CL removes the native assets fallback from the VM.
TEST=pkg/dartdev/test/native_assets/build_test.dart
TEST=tests/ffi/native_assets/asset_relative_test.dart
Closes: https://github.com/dart-lang/sdk/issues/55523
Closes: https://github.com/dart-lang/sdk/issues/55925
Change-Id: Ib40d3a8c7d4b7c6226b6e4db4b3016cc32591239
Cq-Include-Trybots: dart/try:vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-aot-win-debug-x64-try,vm-aot-win-debug-x64c-try,pkg-linux-debug-try,pkg-linux-release-arm64-try,pkg-mac-release-try,pkg-mac-release-arm64-try,pkg-win-release-try,pkg-win-release-arm64-try,vm-aot-asan-linux-release-x64-try,vm-asan-linux-release-x64-try,vm-aot-msan-linux-release-x64-try,vm-msan-linux-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/372500
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
This will enable the standalone embedder to resolve uris in a follow
up CL:
https://dart-review.googlesource.com/c/sdk/+/361881
The implementation was using zones for allocation, this CL switches
that to using `malloc`. The caller is responsible for freeing the
memory if resolving succeeds.
This CL removes `Dart_DefaultCanonicalizeUrl`.
We don't have a run_platform_tests, so this CL keeps the tests in
run_vm_tests.
TEST=vm/cc/ParseUri
TEST=vm/cc/ResolveUri
TEST=tests/ffi/native_assets/asset_relative_test.dart
Bug: https://github.com/dart-lang/sdk/issues/55523
Change-Id: Ifb300d8164eb50506f22ce619fad0811f74ef34c
Cq-Include-Trybots: luci.dart.try:vm-aot-asan-linux-release-x64-try,vm-aot-msan-linux-release-x64-try,vm-asan-linux-release-arm64-try,vm-asan-linux-release-x64-try,vm-msan-linux-release-arm64-try,vm-msan-linux-release-x64-try,pkg-linux-debug-try,pkg-linux-release-arm64-try,pkg-linux-release-try,pkg-mac-release-try,pkg-mac-release-arm64-try,pkg-win-release-arm64-try,pkg-win-release-try,vm-aot-linux-debug-x64-try,vm-linux-debug-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-aot-win-debug-x64-try,vm-win-debug-arm64-try,vm-mac-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/368423
Reviewed-by: Martin Kustermann <kustermann@google.com>
Hoist `CStringUniquePtr` out of the `Utils` class as there
is no reason it has to be nested inside a class - it just makes
code more verbose.
This simplifies code of the form
std::unique_ptr<T, decltype(std::free)> a = { nullptr, std::free };
to
CAllocUniquePtr<T> a;
TEST=ci
Change-Id: Ice42c1b16dfa5b20b321c13fbe5b28b3918581cb
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/368425
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
This reverts commit bc0f02e4c8.
Reason for revert: https://github.com/dart-lang/sdk/issues/55754
Original change's description:
> [vm, gc] Incremental compaction.
>
> At the beginning of a major GC cycle, select some mostly-empty pages to be evacuated. Mark the pages and the objects on these pages. Apply a write barrier for stores creating old -> evacuation candidate pointers, and discover any such pointers that already exist during marking.
>
> At the end of a major GC cycle, evacuate objects from these pages. Forward pointers of objects in the remembered set and new-space. Free the evacuated pages.
>
> This compaction is incremental in the sense that creating the remembered set is interleaved with mutator execution. The evacuation step, however, is stop-the-world.
>
> Write-barrier elimination for x.slot = x is removed. Write-barrier elimination for x.slot = constant is removed in the JIT, kept for AOT but snapshot pages are marked as never-evacuate.
>
> TEST=ci
> Bug: https://github.com/dart-lang/sdk/issues/52513
> Change-Id: Icbc29ef7cb662ef8759b8c1d7a63b7af60766281
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/357760
> Reviewed-by: Alexander Aprelev <aam@google.com>
> Commit-Queue: Ryan Macnak <rmacnak@google.com>
Bug: https://github.com/dart-lang/sdk/issues/52513
Change-Id: I565ad6c0fca283d33f605c10f181bc0a59e7d2b2
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/366965
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Alexander Aprelev <aam@google.com>
Auto-Submit: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
At the beginning of a major GC cycle, select some mostly-empty pages to be evacuated. Mark the pages and the objects on these pages. Apply a write barrier for stores creating old -> evacuation candidate pointers, and discover any such pointers that already exist during marking.
At the end of a major GC cycle, evacuate objects from these pages. Forward pointers of objects in the remembered set and new-space. Free the evacuated pages.
This compaction is incremental in the sense that creating the remembered set is interleaved with mutator execution. The evacuation step, however, is stop-the-world.
Write-barrier elimination for x.slot = x is removed. Write-barrier elimination for x.slot = constant is removed in the JIT, kept for AOT but snapshot pages are marked as never-evacuate.
TEST=ci
Bug: https://github.com/dart-lang/sdk/issues/52513
Change-Id: Icbc29ef7cb662ef8759b8c1d7a63b7af60766281
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/357760
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
When uploading CLs, the presubmit checks verify that the lines in the
diff are formatted correctly according to `git cl format runtime`.
However, when `buildtools/<os>-<arch>/clang/bin/clang-format` is
updated, it does not force reformatting of files that would be
reformatted.
This leads to two issues:
* Inconsistent style within the code base and within a single file.
* Spurious reformatting in CLs when (1) clang-format is used on the
whole file, or (2) the diff lines overlap.
`clang-format` doesn't change that frequently, so in general this is
not a large issue, but I've seen a bit too many "spurious formatting,
please revert" comments on CLs recently.
This CL formats the runtime to be in line with the current pinned
`clang-format`:
```
$ find runtime/ -iname *.h -o -iname *.cc | xargs buildtools/mac-arm64/clang/bin/clang-format -i
```
`git cl format` (which only formats changed lines, and does so with
`clang-format`) seems to not agree with itself, or clang-format, or
cpplint in a handful of places. This CL adds `// clang-format off`
for these. (See previous patchsets for the specific instances.)
TEST=A variety of bots including GCC, MacOS and Windows.
Change-Id: I470892e898971899fda14bb3b8f2c8efefd67686
Cq-Include-Trybots: luci.dart.try:vm-gcc-linux-try,vm-ffi-qemu-linux-release-riscv64-try,vm-ffi-qemu-linux-release-arm-try,vm-aot-win-debug-x64-try,vm-win-debug-x64c-try,vm-mac-debug-x64-try,vm-mac-debug-arm64-try,vm-aot-linux-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/362780
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
After https://dart-review.googlesource.com/c/sdk/+/330600, there were
more chances for the optimizing compiler to introduce or move
GC-triggering instructions like allocations or boxings between the
retrieval of an untagged pointer to GC-moveable memory and its use.
To limit the chance of this happening, this CL removes the explicit
loading of the untagged payload address when building the initial
flow graph in most cases when the array is not known to be an external
array (an external string, an external typed data object, or an FFI
Pointer).
The remaining case is during view allocation, which extracts the
payload address of the base typed data object underlying the view
(which may be GC-movable) to calculate the payload address that should
be stored in the data field of the view object. See
https://github.com/dart-lang/sdk/issues/54884.
During canonicalization of LoadIndexed, StoreIndexed, and MemoryCopy
instructions, if the cid of an array input is an external array
(external string, external typed data object, or Pointer), then a
LoadField instruction that extracts the untagged payload address
is inserted before the instruction and the corresponding input is
rebound to that LoadField instruction.
Once all compiler passes that involve code motion have been performed,
a new pass looks for LoadIndexed, StoreIndexed, or MemoryCopy where
the cid stored in the instruction for the array is a typed data cid.
In these cases, if the array is not an internal typed data object,
then the payload address is extracted. Waiting until this point ensures
that no GC-triggering instructions are inserted between the extraction
of the payload address and the use. (Internal typed data objects are
left as-is because the payload address is inside the object itself
and doesn't require indirection through the data field of the object).
This CL also replaces code conditional on the array cid with code
that is instead conditional on the array element representation in
cases where it makes sense to do so, since this is a less brittle
check than checking the array cid (e.g., checking for kUnboxedInt8
to load, store, or copy an signed byte from an array instead of
listing all possible array cids that store signed bytes).
This CL also fixes an issue with the ARM64 assembler where calling
LoadFromOffset with an Address that has a non-Offset type would
silently generate bad code instead of triggering the ASSERT in
PrepareLargeOffset.
TEST=vm/dart/typed_list_index_checkbound_il_test
Issue: https://github.com/dart-lang/sdk/issues/54710
Cq-Include-Trybots: luci.dart.try:vm-aot-android-release-arm64c-try,vm-aot-android-release-arm_x64-try,vm-aot-linux-debug-x64-try,vm-aot-linux-debug-x64c-try,vm-aot-mac-release-arm64-try,vm-aot-mac-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-win-debug-arm64-try,vm-appjit-linux-debug-x64-try,vm-asan-linux-release-x64-try,vm-checked-mac-release-arm64-try,vm-eager-optimization-linux-release-ia32-try,vm-eager-optimization-linux-release-x64-try,vm-ffi-android-debug-arm-try,vm-ffi-android-debug-arm64c-try,vm-ffi-qemu-linux-release-arm-try,vm-ffi-qemu-linux-release-riscv64-try,vm-fuchsia-release-x64-try,vm-linux-debug-ia32-try,vm-linux-debug-x64-try,vm-linux-debug-x64c-try,vm-mac-debug-arm64-try,vm-mac-debug-x64-try,vm-msan-linux-release-x64-try,vm-reload-linux-debug-x64-try,vm-reload-rollback-linux-debug-x64-try,vm-ubsan-linux-release-x64-try,vm-win-debug-arm64-try,vm-win-debug-x64-try,vm-win-release-ia32-try
Change-Id: I25b5f314943e9254d3d28986d720a5d47f12feeb
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/352363
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
This change may require a codereview from flutter side before moving
forward. See https://github.com/flutter/engine/pull/51072.
This reverts commit 283051acba.
Reason for revert: Redo the change with the similar one in flutter.
After this change, the next dart -> flutter roll needs extra cares
in https://github.com/flutter/engine/pull/51072. It also contains
more details about the reason why it could not be two-way
compatible.
Original change's description:
> Revert "[Fuchsia] Remove the legacy gn build rules for fuchsia"
>
> This reverts commit 094202bb91.
>
> Reason for revert: Break flutter, this change should happen after
> flutter being migrated to the same gn-sdk.
>
> Original change's description:
> > [Fuchsia] Remove the legacy gn build rules for fuchsia
> >
> > The updated test-scripts contains https://crrev.com/c/5341620 which
> > allows to generate fidl apis with an env var controlled location
> > rather than a hard-coded one. So the existing fuchsia gn build rules
> > in dart/sdk become obsolete and can be fully removed in favor of the
> > one in //third_party/fuchsia/gn-sdk/.
> >
> > Meanwhile the gn-sdk has been updated with
> > https://crrev.com/c/5325282 to use api-level version'ed idk but not
> > the ones in arch/.
> >
> > Cq-Include-Trybots: luci.dart.try:vm-fuchsia-release-arm64-try,vm-fuchsia-release-x64-try
> > Tested: ^^^^^
> > Bug: 40935282
> > Change-Id: I2ce958e6db1ff8221beef7b7ff953c32bb4525ba
> > Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/355283
> > Reviewed-by: Ryan Macnak <rmacnak@google.com>
> > Reviewed-by: Alexander Thomas <athom@google.com>
> > Commit-Queue: Zijie He <zijiehe@google.com>
>
> Bug: 40935282
> Change-Id: I7c455d1d362210523671c97d99ef018ede1743f4
> Cq-Include-Trybots: luci.dart.try:vm-fuchsia-release-arm64-try,vm-fuchsia-release-x64-try
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/356307
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Auto-Submit: Zijie He <zijiehe@google.com>
> Reviewed-by: Derek Xu <derekx@google.com>
> Commit-Queue: Zijie He <zijiehe@google.com>
Bug: 40935282
Change-Id: Id5a9a98013350359037b19f8506f24158c9a3120
Cq-Include-Trybots: luci.dart.try:vm-fuchsia-release-arm64-try,vm-fuchsia-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/356924
Commit-Queue: Zijie He <zijiehe@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
* Switch to `std::unique_ptr<wchar_t[]>` to represent a
dynamically strings instead of wrappers like
`StringRAII` and `Utf8ToWideScope`;
* Avoid back and forth conversion between UTF8 and UTF16:
convert to UTF16 first then work on that. This also simplifies
code - previously it tried to work with strings allocated in
different ways uniformly, which is actually unnecessary if
resulting string needs to be converted to UTF16 (and allocated
with `malloc`) anyway;
* Fix a bug in `File::CreateLink`: it was handling relative
links with long paths incorrectly. Change file_long_path_test
to cover this case and make it run on non-Windows systems as
well to ensure that all behavior that can match actually
matches.
TEST=ci
Change-Id: I1279aff1d2cdace5e2ce8633c2f7ea69a34fe41a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/356680
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
This reverts commit 094202bb91.
Reason for revert: Break flutter, this change should happen after
flutter being migrated to the same gn-sdk.
Original change's description:
> [Fuchsia] Remove the legacy gn build rules for fuchsia
>
> The updated test-scripts contains https://crrev.com/c/5341620 which
> allows to generate fidl apis with an env var controlled location
> rather than a hard-coded one. So the existing fuchsia gn build rules
> in dart/sdk become obsolete and can be fully removed in favor of the
> one in //third_party/fuchsia/gn-sdk/.
>
> Meanwhile the gn-sdk has been updated with
> https://crrev.com/c/5325282 to use api-level version'ed idk but not
> the ones in arch/.
>
> Cq-Include-Trybots: luci.dart.try:vm-fuchsia-release-arm64-try,vm-fuchsia-release-x64-try
> Tested: ^^^^^
> Bug: 40935282
> Change-Id: I2ce958e6db1ff8221beef7b7ff953c32bb4525ba
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/355283
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
> Reviewed-by: Alexander Thomas <athom@google.com>
> Commit-Queue: Zijie He <zijiehe@google.com>
Bug: 40935282
Change-Id: I7c455d1d362210523671c97d99ef018ede1743f4
Cq-Include-Trybots: luci.dart.try:vm-fuchsia-release-arm64-try,vm-fuchsia-release-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/356307
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: Zijie He <zijiehe@google.com>
Reviewed-by: Derek Xu <derekx@google.com>
Commit-Queue: Zijie He <zijiehe@google.com>
Ryan Macnak