- Introduced Dart_SetObfuscationMap to restore obfuscation maps before AOT precompilation.
- Added Dart_AotPatchInstallOptions structure for AOT patch installation options.
- Implemented Dart_AotPatchingEnabled to check if compact AOT patching is supported.
- Created Dart_SetAotPatchKeyCallback for AES key resolution during AOT patch installation.
- Developed Dart_InstallAotPatch for validating and installing encrypted AOT patches.
- Added Dart_FreeAotPatchPayload to free memory allocated for patch payloads.
- Updated runtime_args.gni to include dart_enable_aot_patching flag.
- Added tests for AOT patching functionality and ensured exported symbols include new APIs.
- Refactored existing code to accommodate new AOT patching features and improve error handling.
Put local actions in a common pool, especially
- Assembler (many run for boringssl)
- Linker (high memory use)
- Dart (high memory use)
This should limit the number of concurrent local actions to the number of harts available, avoiding the racing actions triggering out-of-memory failures.
Change-Id: Ifa168012d73472a3db349acc36545dd176385bbc
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/510190
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
Several C++-only warning suppressions were incorrectly placed in the
shared 'default_warning_flags' list instead of
'default_warning_flags_cc'. This caused these flags (such as
-Wno-microsoft-unqualified-friend, -Wno-microsoft-cast)
to be applied to pure C targets (like zlib and BoringSSL assembly).
This CL moves these C++-only warnings to 'default_warning_flags_cc',
ensuring they are only applied to C++ compilation units. This improves
compile database accuracy, prevents IDE/clangd from surfacing phantom
diagnostics on C files, and makes the build configuration explicit and
robust.
Change-Id: I117de03496164d7e2e39b5deab7754fd85452923
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/510181
Commit-Queue: Kevin Moore <kevmoo@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Auto-Submit: Kevin Moore <kevmoo@google.com>
- Break false dependency of non-VM platforms on VM platform
- Break false dependency of bootstrap gen_kernel on VM platform
- Extend precompile_tools to gen_kernel steps
- Rename intermediate kernel files so JIT versus AOT is visible in ninjatracing
TEST=ci
Change-Id: I07011abe8303597af61d2b8c73e788b661482cc0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/510060
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
This will make GN notice conflicting copy_tree destinations.
It may also fix a race when two copy_tree's with a common parent directory create that parent directory.
Change-Id: I1783ea170f0390d4026123ed43c00cd15eb0b1c5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/508241
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Samuel Rawlins <srawlins@google.com>
Extend --check-clean (enabled on bots) to verify build commands don't include absolute paths.
The non-RBE GCC and MSVC builds still have absolute paths. GCC doesn't have -ffile-compilation-dir so it uses -fdebug-prefix-map, which is not fixable. The MSVC build has absolute path in the toolchain wrappers, which might be fixable.
TEST=ci
Change-Id: I3b984aaab7aefa7ff527f0a039ca42281224a09d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/506505
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed in https://dart-review.googlesource.com/c/sdk/+/497583
[dartpad] Part 1: Protocol and Scaffolding
This sets up folders for `package:dartpad` and `package:dartpad_worker`,
R=athom@google.com, sigurdm@google.com
including:
* `OWNERS` files,
* Protocol definition,
* Exceptions, and,
* Virtual file system utilities.
[dartpad] Part 2: HotReloadCompiler
Adds the hot reload compiler logic to dartpad_worker.
This is models on dartdevc with `--reload-last-accepted-kernel` and
`--reload-delta-kernel` options. I don't think this does an incremental
compilation, instead it does a modular compilation. Meaning, we can have
a huge collection of precompiled DDC modules and the compiler just gets
a DDC outline/summary dill files for these modules. But on the other hand
the actual code that is compiled will be recompiled on every compile()
call.
This is very important for flutter_web.js / flutter_web.dill where the
precompiled DDC modules is around 60 MB in total. The fact that
package:flutter is precompiled into flutter_web.js is critical for
compilation performance for small dartpad-style example apps.
The downside is that unlike incremental compilation with _frontendserver_,
if the user were to install package:http and use it in the app they are
compiling then every `compile()` call will recompile `package:http`.
Where as with _frontendserver_ we'd be able to invalidate specific files,
and only those would be recompiled.
This leaves future work to either:
* (A) Add support for modular compilation to _frontendserver_, such that
we can do incremental compilation without having to first compile all
of `package:flutter`.
* (B) Refactor compilation strategy implemented here, to leverage
modular further. We could for example compile all dependencies from
`PUB_CACHE` into a `pub_cache_outline.dill` inside the dartpad worker.
Then recompilation would only touch the user-code. Granted this is
still less than ideal.
At the moment the lack of incremental compilation is not a blocker,
current dartpad doesn't have incremental compilation, but granted it also
only supported a single file. This dartpad environment will support
multiple files, but adding many files will result in poor performance
until we refactor and figure out how to do incremental compilation.
Hence, why we shouldn't block progress waiting for incremental compilation
to work.
R=nshahan@google.com
[dartpad] Part 3: Pub execution wrapper
Adds the internal wrapper for executing pub commands inside the
worker's virtual file system.
R=sigurdm@google.com
[dartpad] Part 4: Language Server wrapper
Adds the internal wrapper for spawning and communicating with the
Dart LSP inside the DartPad worker.
R=scheglov@google.com
[dartpad] Part 5: DartPad Worker
Implements the main worker entrypoint, RPC message handling. The
`bin/worker.dart` is what will be compiled to WASM and will run as a
_Web Worker_.
R=sigurdm@google.com
[dartpad] Part 6: DartPad Client library
The public-facing `dartpad` library that developers use to embed
the compilation environment in their web applications.
This finishes `package:dartpad`featuring:
* `DartPad.create()`, which creates a _Web Worker_ running the compiled
`pkg/dartpad_worker/bin/worker.dart`, and returns a `DartPad` instance
wrapping RPC calls into the _Web Worker_.
* `Sandbox.createIFrame()` which creates a _sandboxed iframe_ containing
precompiled DDC modules from the SDK assets and `sandbox.js` for
wrapping `ddc_module_loader.js` with an RPC interface. This returns a
`Sandbox` instance wrapping RPC calls into the _sandboxed iframe_,
making it easy to load modules, run library entrypoints (main()),
initiate hot-reload, or launch a flutter app.
This is an initial API design that works. There are minor inconsistencies
and improvements that we should do. But I propose that we do so in
follow-up PRs.
R=sigurdm@google.com
[dartpad] Part 7: Build Targets
Adds a `dartpad` target to the Dart SDK, which produces:
```
out/ReleaseX64/dartpad/
├── dart
│ ├── dart_sdk.js.map
│ ├── sdk.js
│ └── sdk.tar
├── ddc_module_loader.js
├── sandbox.js
├── worker.loader.js
├── worker.mjs
├── worker.support.js
├── worker.wasm
└── worker.wasm.map
```
The `dartpad/` output folder is intended to be used as `assetBaseUrl` in
client libraries provided in `package:dartpad`. And the `dartpad/dart/`
folder is intended to be used as `sdkLocation`. These files are not
intended to be distributed along side the normal Dart SDK, instead we'll
publish them on a CDN and let people use `package:dartpad` to point at
these files. If they want to self-host everything, they can copy from our
CDN or rebuild the files using a local Dart SDK checkout.
These files are also necessary for testing, to ensure that we have proper
integration tests for `pkg/dartpad_worker/`. As we will want to publish
these files on a CDN the actual compiled size in release-mode matters,
thus, we have enabled wasm optimizations steps in this mode.
R=rmacnak@google.com
[dartpad] Part 8: Tests for `package:dartpad` and `pkg/dartpad_worker/`.
We have 3 kinds of tests:
* Unit tests (vm, browser or both),
* Worker tests (vm and browser),
* Integration tests (browser-only).
As the Dart SDK test runner simply executes `*_test.dart` files with
`dart` and doesn't support compiling tests to the browser, launching and
running tests in the browser, we employ `dart test` to run tests. We do
this by having `pkg/dartpad_worker/dart_test.yaml` specify that
`dart test` should look for files names `test_*.dart`, and we then create
`pkg/dartpad_worker/test/ci_test.dart´ to be a single test that simply
runs `dart test` as a subprocess. Thus, the only test file triggered by
Dart SDK test runner is `ci_test.dart`, and if you locally run `dart test`
this will run the individual `test_*.dart` files.
While this hack to use `dart test` isn't ideal it gives everyone a decent
work flow. And saves us from having to invent complex test harness for
`pkg/dartpad_worker/`.
**Worker tests** defined in `test/dart/worker/` and `test/flutter/worker/`
are all imported into `test_dart_worker.dart` and
`test_flutter_worker.dart` to reduce test compilation time. These have a
non-trivial compilation time because the worker is running in the test
process. This allows testing on VM, which can be easier to debug.
**Integration tests** unlike _worker tests_ launch the worker compiled by
the `dartpad` build target in a _Web Worker_. This has less compilation
overhead for the individual tests, but involves more setup, and can be
a bit harder to debug.
Both worker and integration tests employ `test/asset_server/` which serves
assets built by the `dartpad` build target, and serves as a pub-server for
tests. It also serves assets built by `tool/setup_local_flutter.dart`
which creates flutter assets, though this is not intended to run as part
of CI, since we do not have a Flutter checkout available. Thus, for the
time being flutter testing is local only. Maybe, later we'll move this
script to another repository.
R=sigurdm@google.com
Cq-Include-Trybots: luci.dart.try:pkg-linux-release-try,pkg-win-release-try
Change-Id: I3a99939ec5217b9f3a855fc5b9ad9699047d02cc
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/507321
Commit-Queue: Jonas Jensen <jonasfj@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
On Mac and Linux, the makes the output of non-RBE, RBE local and RBE remote builds identical, and also independent of the build directories path. On Windows, non-RBE and RBE builds still disagree because paths are rewritten from \ to / to run on the Linux workers.
Bug: b/316893839
Change-Id: I5935785489c73445e4c6ca275020e6cf7464433d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/506281
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
The former contents of the VM isolate are now included into each isolate group. This makes each isolate group's heap independent, and in particular allows each heap to be allocated to a separate pointer cage (not done in this CL).
The duplicated stubs that allowed PC relative calls are removed, since the originals can now be the target of PC relative calls.
The bootstrapping needing to load an AppJIT or AppAOT snapshot is reduced to allocating the oddballs. The code is entirely dropped in the AOT runtime, but the JIT runtime still has it to allow for flags to affect the compilation of the stub code. Further refactoring might be able to remove this for the JIT runtime too, with only gen_snapshot knowing how to bootstrap.
Class serialization no longer distinguishes predefined classes.
The page containing null is marked as never-evacuate. null, false and true must not move because the compiler relies on their low bits having certain patterns for some optimizations. (Previously, the entire VM isolate heap never moved.)
Compaction is disabled for IA32. Due to register pressure, some stub calls must not use a scratch register and embed the address of Code.
The page containing the call-through-safepoint stub is frozen when running with --write-protect-code and the stub is created at runtime (instead of loaded from an AppJIT or AppAOT snapshot). This stub must remain executable even during a safepoint, as a foreign call might during return during a safepoint and only block after the stub directs it to the runtime.
The snapshot symbols are renamed to kDartSnapshotData and kDartSnapshotText. There is no need to distinguish the VM isolate's snapshot, and snaphots are per isolate group not per isolate. Aliases with the old names are added to ease migration.
Some global flags that were automatically set based on the VM isolate's snapshot are now isolate group flags and automatically set by the isolate group's snapshot.
TEST=ci
Change-Id: Iee82016057d609112e9b021d178fc3d4d18b5044
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/500621
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Tess Strickland <sstrickl@google.com>
SLSA-Policy-Verified: SLSA Policy Verification Service <devtools-gerritcodereview-exitgate@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
This change adds initial support for working with isolates in the Dart
Runtime Service and its backends.
The new `IsolateManager` base class tracks the set of active isolates
and their lifecycle events. The `VmIsolateManager` extends this class,
adding support specific to interacting with isolates within the Dart VM.
TEST=vm/cc/DartAPI_InvokeVMServiceMethod*_Exp
Change-Id: I3dfa298722c40dbdfdd58105cc78f31d058dd7a2
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/486560
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Ben Konyi <bkonyi@google.com>
Reviewed-by: Jessy Yameogo <yjessy@google.com>
This allows bloaty to attribute functions (but not variables) written in assembly to their source file. Which in turn allows the binary_size visualization to group, e.g., all the boringssl crypto kernels under boringssl instead of them all appearing at the top level.
Change-Id: I6f601f44931d6c3d2fe2018a43804b070643fba5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/486520
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Drop libprotobuf dependency and update compile_perfetto_protos
to rely on externally provided protoc and protozero_plugin.
This is done because protoc does not build with our current
toolchain and fixing this build requires rolling new protobuf,
protobuf_gn and perfetto. Which also pulls more external dependencies
like abseil.
Regenerate Perfetto *.pbzero{.cc,.h}, *.dart files
TEST=ci
Change-Id: I11a32aa07e810b48ac12502e526bbaee6a6a6964
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/482781
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
- copy sanitizer runtime when using ubsan;
- avoid use of winnt.h CONTAINING_RECORD since it triggers ubsan "member access within null-pointer of type" error, use our copy which uses `offsetof`;
- have default virtual destructor in `ValueObject` to avoid ubsan complains about "insufficient space for an object of type 'dart:ValueObject'" at NoTemporaryAllocator use/declaration site;
- have virtual destructor in ZoneAllocated to avoid ubsan complains about "not having enough space to allocate object" at new RegExpEmpty() instantiation site;
- avoid using crashpad with ubsan as it causes dartvm to exit with error code 3;
- switch to windows, mac-friendly `[[gnu::no_sanitize(check)]]` from `__GNUC__` and `__has_feature` checks.
TEST=ci
Bug: https://github.com/dart-lang/sdk/issues/62267
Change-Id: I8b922a8da329af276d4cefaa88fb841cc0457124
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/469840
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Alexander Aprelev <aam@google.com>
We bumped NTDDI_VERSION to Windows 10 but have not updated WINVER in
the same way making them diverge for no good reason.
FWIW NTDDI_VERSION should probably be set by winver config, but I am
not going to fix that now.
Change-Id: I53cdf1cde40aa50911aa1311ca15705d6a6a6964
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/471181
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
This CL tries to address version skew issues which occur in internal
monorepo which we created by checking in *.pbzero.h files into
runtime/vm/protos/*. These files refer to protozero internal headers,
which in the internal monorepo come from a shared Perfetto dependency
which does not necessarily match the version we pinned in our DEPS and
used to generated these files.
This version skew caused problems for the Perfetto team, see
https://github.com/google/perfetto/pull/3195
To resolve this change layout of our repo to allow us to depend on the
shared Perfetto entirely - including its version of *.pbzero.h headers.
We change include paths in the VM source to
#include "third_party/perfetto/protos/..."
which match location of these headers in the internal monorepo.
To make the code compile in the normal SDK checkout we shift location of
Perfetto and checked in code:
* Perfetto source moves from third_party/perfetto to
third_party/perfetto/src.
* Generated code moves from runtime/vm/protos into
third_party/perfetto/protos.
Note: experiments show that *.pbzero.h files tree-shake very well and
you pay only for things that you use. That means depending on *.pbzero.h
files generated from full message definitions rather than our manually
tree-shaken versions does not actually increase the size of the VM
binaries. We could switch our external build to do the same but this
would make protoc a build time dependency, which seems excessive.
TEST=ci
Change-Id: I724b9d9304be83302a5d83c2337154b36a6a6964
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/454260
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Ben Konyi <bkonyi@google.com>
Sanitizers can pickup configuration and suppressions via
special symbols in the binary (e.g. __*_default_suppressions and
__*_default_options). We have a bunch of stale files in the
buildroot which showed how to do it, but none of these were
actually used by the build process.
Update our BUILDCONFIG.gn to actually link this code into
the binaries and clean it up leaving behind only one relevant
suppression for TSAN.
Additionally fix libplatform targets - libdart_platform_no_tsan was
accidentally disabling TSAN for the whole libplatform by adding
no_tsan_config into public_configs instead of extra_configs.
This is needed to unblock landing https://dart-review.googlesource.com/c/sdk/+/444983
TEST=manually with the referenced CL to verify that suppressions work
Cq-Include-Trybots: luci.dart.try:vm-tsan-linux-release-x64-try,vm-tsan-linux-release-arm64-try
Change-Id: Id4e8a5b89c665cf5d89b18c4f5881ef31c3c9396
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/445941
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Tony