The front-end, dart2js, dart2wasm, ddc and analyzer bots do not need the cross-compilation or sanitizer tools, nor the VM test binaries.
Change-Id: I2f5a1041b1b29c562bace7210830e9904a31c397
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/510040
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
dartfuzz has been taking ~35 min to finish the build and upload steps even with 100% RBE cache hit rate, causing it to fail to finish before the first wave of nightly builders trigger.
Change-Id: I932facd77bc08f4d7236ae8c760013406a69fb0e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/507267
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
Reviewed in https://dart-review.googlesource.com/c/sdk/+/497583
[dartpad] Part 1: Protocol and Scaffolding
This sets up folders for `package:dartpad` and `package:dartpad_worker`,
R=athom@google.com, sigurdm@google.com
including:
* `OWNERS` files,
* Protocol definition,
* Exceptions, and,
* Virtual file system utilities.
[dartpad] Part 2: HotReloadCompiler
Adds the hot reload compiler logic to dartpad_worker.
This is models on dartdevc with `--reload-last-accepted-kernel` and
`--reload-delta-kernel` options. I don't think this does an incremental
compilation, instead it does a modular compilation. Meaning, we can have
a huge collection of precompiled DDC modules and the compiler just gets
a DDC outline/summary dill files for these modules. But on the other hand
the actual code that is compiled will be recompiled on every compile()
call.
This is very important for flutter_web.js / flutter_web.dill where the
precompiled DDC modules is around 60 MB in total. The fact that
package:flutter is precompiled into flutter_web.js is critical for
compilation performance for small dartpad-style example apps.
The downside is that unlike incremental compilation with _frontendserver_,
if the user were to install package:http and use it in the app they are
compiling then every `compile()` call will recompile `package:http`.
Where as with _frontendserver_ we'd be able to invalidate specific files,
and only those would be recompiled.
This leaves future work to either:
* (A) Add support for modular compilation to _frontendserver_, such that
we can do incremental compilation without having to first compile all
of `package:flutter`.
* (B) Refactor compilation strategy implemented here, to leverage
modular further. We could for example compile all dependencies from
`PUB_CACHE` into a `pub_cache_outline.dill` inside the dartpad worker.
Then recompilation would only touch the user-code. Granted this is
still less than ideal.
At the moment the lack of incremental compilation is not a blocker,
current dartpad doesn't have incremental compilation, but granted it also
only supported a single file. This dartpad environment will support
multiple files, but adding many files will result in poor performance
until we refactor and figure out how to do incremental compilation.
Hence, why we shouldn't block progress waiting for incremental compilation
to work.
R=nshahan@google.com
[dartpad] Part 3: Pub execution wrapper
Adds the internal wrapper for executing pub commands inside the
worker's virtual file system.
R=sigurdm@google.com
[dartpad] Part 4: Language Server wrapper
Adds the internal wrapper for spawning and communicating with the
Dart LSP inside the DartPad worker.
R=scheglov@google.com
[dartpad] Part 5: DartPad Worker
Implements the main worker entrypoint, RPC message handling. The
`bin/worker.dart` is what will be compiled to WASM and will run as a
_Web Worker_.
R=sigurdm@google.com
[dartpad] Part 6: DartPad Client library
The public-facing `dartpad` library that developers use to embed
the compilation environment in their web applications.
This finishes `package:dartpad`featuring:
* `DartPad.create()`, which creates a _Web Worker_ running the compiled
`pkg/dartpad_worker/bin/worker.dart`, and returns a `DartPad` instance
wrapping RPC calls into the _Web Worker_.
* `Sandbox.createIFrame()` which creates a _sandboxed iframe_ containing
precompiled DDC modules from the SDK assets and `sandbox.js` for
wrapping `ddc_module_loader.js` with an RPC interface. This returns a
`Sandbox` instance wrapping RPC calls into the _sandboxed iframe_,
making it easy to load modules, run library entrypoints (main()),
initiate hot-reload, or launch a flutter app.
This is an initial API design that works. There are minor inconsistencies
and improvements that we should do. But I propose that we do so in
follow-up PRs.
R=sigurdm@google.com
[dartpad] Part 7: Build Targets
Adds a `dartpad` target to the Dart SDK, which produces:
```
out/ReleaseX64/dartpad/
├── dart
│ ├── dart_sdk.js.map
│ ├── sdk.js
│ └── sdk.tar
├── ddc_module_loader.js
├── sandbox.js
├── worker.loader.js
├── worker.mjs
├── worker.support.js
├── worker.wasm
└── worker.wasm.map
```
The `dartpad/` output folder is intended to be used as `assetBaseUrl` in
client libraries provided in `package:dartpad`. And the `dartpad/dart/`
folder is intended to be used as `sdkLocation`. These files are not
intended to be distributed along side the normal Dart SDK, instead we'll
publish them on a CDN and let people use `package:dartpad` to point at
these files. If they want to self-host everything, they can copy from our
CDN or rebuild the files using a local Dart SDK checkout.
These files are also necessary for testing, to ensure that we have proper
integration tests for `pkg/dartpad_worker/`. As we will want to publish
these files on a CDN the actual compiled size in release-mode matters,
thus, we have enabled wasm optimizations steps in this mode.
R=rmacnak@google.com
[dartpad] Part 8: Tests for `package:dartpad` and `pkg/dartpad_worker/`.
We have 3 kinds of tests:
* Unit tests (vm, browser or both),
* Worker tests (vm and browser),
* Integration tests (browser-only).
As the Dart SDK test runner simply executes `*_test.dart` files with
`dart` and doesn't support compiling tests to the browser, launching and
running tests in the browser, we employ `dart test` to run tests. We do
this by having `pkg/dartpad_worker/dart_test.yaml` specify that
`dart test` should look for files names `test_*.dart`, and we then create
`pkg/dartpad_worker/test/ci_test.dart´ to be a single test that simply
runs `dart test` as a subprocess. Thus, the only test file triggered by
Dart SDK test runner is `ci_test.dart`, and if you locally run `dart test`
this will run the individual `test_*.dart` files.
While this hack to use `dart test` isn't ideal it gives everyone a decent
work flow. And saves us from having to invent complex test harness for
`pkg/dartpad_worker/`.
**Worker tests** defined in `test/dart/worker/` and `test/flutter/worker/`
are all imported into `test_dart_worker.dart` and
`test_flutter_worker.dart` to reduce test compilation time. These have a
non-trivial compilation time because the worker is running in the test
process. This allows testing on VM, which can be easier to debug.
**Integration tests** unlike _worker tests_ launch the worker compiled by
the `dartpad` build target in a _Web Worker_. This has less compilation
overhead for the individual tests, but involves more setup, and can be
a bit harder to debug.
Both worker and integration tests employ `test/asset_server/` which serves
assets built by the `dartpad` build target, and serves as a pub-server for
tests. It also serves assets built by `tool/setup_local_flutter.dart`
which creates flutter assets, though this is not intended to run as part
of CI, since we do not have a Flutter checkout available. Thus, for the
time being flutter testing is local only. Maybe, later we'll move this
script to another repository.
R=sigurdm@google.com
Cq-Include-Trybots: luci.dart.try:pkg-linux-release-try,pkg-win-release-try
Change-Id: I3a99939ec5217b9f3a855fc5b9ad9699047d02cc
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/507321
Commit-Queue: Jonas Jensen <jonasfj@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
The former contents of the VM isolate are now included into each isolate group. This makes each isolate group's heap independent, and in particular allows each heap to be allocated to a separate pointer cage (not done in this CL).
The duplicated stubs that allowed PC relative calls are removed, since the originals can now be the target of PC relative calls.
The bootstrapping needing to load an AppJIT or AppAOT snapshot is reduced to allocating the oddballs. The code is entirely dropped in the AOT runtime, but the JIT runtime still has it to allow for flags to affect the compilation of the stub code. Further refactoring might be able to remove this for the JIT runtime too, with only gen_snapshot knowing how to bootstrap.
Class serialization no longer distinguishes predefined classes.
The page containing null is marked as never-evacuate. null, false and true must not move because the compiler relies on their low bits having certain patterns for some optimizations. (Previously, the entire VM isolate heap never moved.)
Compaction is disabled for IA32. Due to register pressure, some stub calls must not use a scratch register and embed the address of Code.
The page containing the call-through-safepoint stub is frozen when running with --write-protect-code and the stub is created at runtime (instead of loaded from an AppJIT or AppAOT snapshot). This stub must remain executable even during a safepoint, as a foreign call might during return during a safepoint and only block after the stub directs it to the runtime.
The snapshot symbols are renamed to kDartSnapshotData and kDartSnapshotText. There is no need to distinguish the VM isolate's snapshot, and snaphots are per isolate group not per isolate. Aliases with the old names are added to ease migration.
Some global flags that were automatically set based on the VM isolate's snapshot are now isolate group flags and automatically set by the isolate group's snapshot.
TEST=ci
Change-Id: Iee82016057d609112e9b021d178fc3d4d18b5044
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/500621
Reviewed-by: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Tess Strickland <sstrickl@google.com>
SLSA-Policy-Verified: SLSA Policy Verification Service <devtools-gerritcodereview-exitgate@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
When compiled to WebAssembly, the Dart SDK needs access to external
host functions to implement regular expressions, stack traces, timers
and more. Currently, `dart2wasm` relies on `js_interop` definition to
implement these functions in JavaScript.
As discussed in https://github.com/dart-lang/sdk/issues/53884, an
alternative is to use `wasm:import` annotations to let an arbitrary
embedder that doesn't necessarily run in a JavaScript context inject
implementations for these host functions.
This would allow running `dart2wasm` apps by e.g.
- using a runtime like wasmtime and defining host functions in Rust.
- defining a wrapper module implementing required functions by
delegating to WASI definitions, and then using say `wasm-merge` to
run the app in any WASI-compatible runtime.
This prepares the `--standalone` flag on `dart2wasm` to do just that.
When enabled, the compiler uses a different SDK platform to use imports
instead of JS interop. For now, these platforms are almost identical:
I've ported the timer logic to use wasm imports as a demo, but the rest
is still based on existing patch files. We can revisit in subsequent
CLs to incrementally reduce `js_interop` dependencies before removing
that library from the `dart2wasm_standalone` target entirely.
Change-Id: I3f406afbf2dab65506094de5c3f4067f4db66f3e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/486380
Reviewed-by: Slava Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
This change includes an initial implementation of the new VM service
implementation based on `package:dart_runtime_service`, along with the
necessary plumbing to start it in place of the legacy VM service
implementation.
The entrypoint for the new VM service implementation is located in
dart_runtime_service_vm/bin/vm_service_entrypoint.dart, which is
compiled into AppJIT and AOT snapshots when the
`--include-experimental-vm-service` flag is provided to `build.py`. To run
the VM with the new VM service implementation, the
`--experimental-vm-service` flag must be provided.
Currently, the experimental VM service implementation supports:
- User specified ports
- Authentication code flags
- Enabling the HTTP server via SIGQUIT
- Some service protocol RPCs that don't require an isolate ID (e.g.,
`getVM`)
See go/dart-runtime-services-unification for more details.
TEST=Manual
CoreLibraryReviewExempt: dart:_vmservice is private
Change-Id: I4a58cd1fa0a386313baa3d5c5345720231279123
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/484820
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Ben Konyi <bkonyi@google.com>
This change creates `pkg/dart_runtime_services/` with the initial
implementation of `package:dart_runtime_service`.
`package:dart_runtime_service` provides a simple interface to create
Dart services with common platform-agnostic features and behaviors,
while also allowing for configuration of custom service backends.
This initial implementation focuses on providing the initial shared
server functionality, including:
- Handlers for web socket and SSE connections
- Authentication code generation + validation
- Client management
- Initial RPC routing with two sample RPCs (getClientName, setClientName)
Change-Id: I1458269a5de7edd97120103a2c5fc2d0348eff31
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/478760
Reviewed-by: Alexander Thomas <athom@google.com>
Commit-Queue: Ben Konyi <bkonyi@google.com>
This roll moves `package:record_use` to the dart-lang/native repo.
Change-Id: I31183dc8b72272d7e94ed3031ca0b8bfca583e0d
Cq-Include-Trybots: luci.dart.try:pkg-linux-debug-try,pkg-linux-release-arm64-try,pkg-linux-release-try,pkg-mac-release-arm64-try,pkg-mac-release-try,pkg-win-release-arm64-try,pkg-win-release-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/463662
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Michael Goderbauer <goderbauer@google.com>
Shards on vm-aot-android-debug-arm64c and vm-aot-android-debug-arm_x64
bots have been timing out recently, exceeding 1.5h testing time.
It seems like they are making progress, but number of tests is quite
large.
Increase number of shards from 10 to 16 in order to reduce running
time of each shard and avoid timeouts.
Change-Id: Id53d2cdf1eaa2606874118153c0f9c822c212daf
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/461840
Auto-Submit: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Previously these were being run with the standard "pkg" unit test suite.
All the other backends have their own unit test suite so this introduces
one for dart2wasm.
This ensures the tests run with the correct fileset (including d8).
Change-Id: I29cd0048a4d6731da3a20110f6e496fe7da0c3b6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/454760
Commit-Queue: Nate Biggs <natebiggs@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
This adds a wasm binary reader that produces an `ir.Module`.
We also make a few changes to existing code
* Represent the import section with an `ir.Imports` object (similar to
`ir.Exports`, `ir.Functions`, ...)
* We make a bunch of data structures allocatable in uninitialized state
(the fields being usually uninitialized `late final` fields) where the
deserializer can create those objects and then fill in details later.
=> This comes partly due to the way wasm binaries are structured
themselves: The "data count" section comes first so a reader knows
how many data sections there will be, then the "code section" can
refer to those data sections. Then afterwards the actual "data
segment" comes that fills in the data of the section.
* We make names consistently optional: Wasm objects don't have to have
names, so the names should be optional, so we make them `String?`. We
also make them non-final as that's consistent with other names.
* We make the `ir.Types`, `ir.Functions`, ... objects have `operator[]`
and the index used is the same index used e.g. in wasm instructions.
* We make static constants for section ids and custom section names.
Issue https://github.com/dart-lang/sdk/issues/60928
Change-Id: I5394d6b82cf4dc68d24cea1dee66c5b33eb2f60f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/452144
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ömer Ağacan <omersa@google.com>
When the shard count is greater than the number of machines in the pool, shards are guarenteed to wait, increase latency relative to a lower shard count.
Change-Id: I1fd61a59837b767748c13120dec30ca6eb352f28
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/450946
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Ryan Macnak