dartfuzz has been taking ~35 min to finish the build and upload steps even with 100% RBE cache hit rate, causing it to fail to finish before the first wave of nightly builders trigger.
Change-Id: I932facd77bc08f4d7236ae8c760013406a69fb0e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/507267
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
Reviewed in https://dart-review.googlesource.com/c/sdk/+/497583
[dartpad] Part 1: Protocol and Scaffolding
This sets up folders for `package:dartpad` and `package:dartpad_worker`,
R=athom@google.com, sigurdm@google.com
including:
* `OWNERS` files,
* Protocol definition,
* Exceptions, and,
* Virtual file system utilities.
[dartpad] Part 2: HotReloadCompiler
Adds the hot reload compiler logic to dartpad_worker.
This is models on dartdevc with `--reload-last-accepted-kernel` and
`--reload-delta-kernel` options. I don't think this does an incremental
compilation, instead it does a modular compilation. Meaning, we can have
a huge collection of precompiled DDC modules and the compiler just gets
a DDC outline/summary dill files for these modules. But on the other hand
the actual code that is compiled will be recompiled on every compile()
call.
This is very important for flutter_web.js / flutter_web.dill where the
precompiled DDC modules is around 60 MB in total. The fact that
package:flutter is precompiled into flutter_web.js is critical for
compilation performance for small dartpad-style example apps.
The downside is that unlike incremental compilation with _frontendserver_,
if the user were to install package:http and use it in the app they are
compiling then every `compile()` call will recompile `package:http`.
Where as with _frontendserver_ we'd be able to invalidate specific files,
and only those would be recompiled.
This leaves future work to either:
* (A) Add support for modular compilation to _frontendserver_, such that
we can do incremental compilation without having to first compile all
of `package:flutter`.
* (B) Refactor compilation strategy implemented here, to leverage
modular further. We could for example compile all dependencies from
`PUB_CACHE` into a `pub_cache_outline.dill` inside the dartpad worker.
Then recompilation would only touch the user-code. Granted this is
still less than ideal.
At the moment the lack of incremental compilation is not a blocker,
current dartpad doesn't have incremental compilation, but granted it also
only supported a single file. This dartpad environment will support
multiple files, but adding many files will result in poor performance
until we refactor and figure out how to do incremental compilation.
Hence, why we shouldn't block progress waiting for incremental compilation
to work.
R=nshahan@google.com
[dartpad] Part 3: Pub execution wrapper
Adds the internal wrapper for executing pub commands inside the
worker's virtual file system.
R=sigurdm@google.com
[dartpad] Part 4: Language Server wrapper
Adds the internal wrapper for spawning and communicating with the
Dart LSP inside the DartPad worker.
R=scheglov@google.com
[dartpad] Part 5: DartPad Worker
Implements the main worker entrypoint, RPC message handling. The
`bin/worker.dart` is what will be compiled to WASM and will run as a
_Web Worker_.
R=sigurdm@google.com
[dartpad] Part 6: DartPad Client library
The public-facing `dartpad` library that developers use to embed
the compilation environment in their web applications.
This finishes `package:dartpad`featuring:
* `DartPad.create()`, which creates a _Web Worker_ running the compiled
`pkg/dartpad_worker/bin/worker.dart`, and returns a `DartPad` instance
wrapping RPC calls into the _Web Worker_.
* `Sandbox.createIFrame()` which creates a _sandboxed iframe_ containing
precompiled DDC modules from the SDK assets and `sandbox.js` for
wrapping `ddc_module_loader.js` with an RPC interface. This returns a
`Sandbox` instance wrapping RPC calls into the _sandboxed iframe_,
making it easy to load modules, run library entrypoints (main()),
initiate hot-reload, or launch a flutter app.
This is an initial API design that works. There are minor inconsistencies
and improvements that we should do. But I propose that we do so in
follow-up PRs.
R=sigurdm@google.com
[dartpad] Part 7: Build Targets
Adds a `dartpad` target to the Dart SDK, which produces:
```
out/ReleaseX64/dartpad/
├── dart
│ ├── dart_sdk.js.map
│ ├── sdk.js
│ └── sdk.tar
├── ddc_module_loader.js
├── sandbox.js
├── worker.loader.js
├── worker.mjs
├── worker.support.js
├── worker.wasm
└── worker.wasm.map
```
The `dartpad/` output folder is intended to be used as `assetBaseUrl` in
client libraries provided in `package:dartpad`. And the `dartpad/dart/`
folder is intended to be used as `sdkLocation`. These files are not
intended to be distributed along side the normal Dart SDK, instead we'll
publish them on a CDN and let people use `package:dartpad` to point at
these files. If they want to self-host everything, they can copy from our
CDN or rebuild the files using a local Dart SDK checkout.
These files are also necessary for testing, to ensure that we have proper
integration tests for `pkg/dartpad_worker/`. As we will want to publish
these files on a CDN the actual compiled size in release-mode matters,
thus, we have enabled wasm optimizations steps in this mode.
R=rmacnak@google.com
[dartpad] Part 8: Tests for `package:dartpad` and `pkg/dartpad_worker/`.
We have 3 kinds of tests:
* Unit tests (vm, browser or both),
* Worker tests (vm and browser),
* Integration tests (browser-only).
As the Dart SDK test runner simply executes `*_test.dart` files with
`dart` and doesn't support compiling tests to the browser, launching and
running tests in the browser, we employ `dart test` to run tests. We do
this by having `pkg/dartpad_worker/dart_test.yaml` specify that
`dart test` should look for files names `test_*.dart`, and we then create
`pkg/dartpad_worker/test/ci_test.dart´ to be a single test that simply
runs `dart test` as a subprocess. Thus, the only test file triggered by
Dart SDK test runner is `ci_test.dart`, and if you locally run `dart test`
this will run the individual `test_*.dart` files.
While this hack to use `dart test` isn't ideal it gives everyone a decent
work flow. And saves us from having to invent complex test harness for
`pkg/dartpad_worker/`.
**Worker tests** defined in `test/dart/worker/` and `test/flutter/worker/`
are all imported into `test_dart_worker.dart` and
`test_flutter_worker.dart` to reduce test compilation time. These have a
non-trivial compilation time because the worker is running in the test
process. This allows testing on VM, which can be easier to debug.
**Integration tests** unlike _worker tests_ launch the worker compiled by
the `dartpad` build target in a _Web Worker_. This has less compilation
overhead for the individual tests, but involves more setup, and can be
a bit harder to debug.
Both worker and integration tests employ `test/asset_server/` which serves
assets built by the `dartpad` build target, and serves as a pub-server for
tests. It also serves assets built by `tool/setup_local_flutter.dart`
which creates flutter assets, though this is not intended to run as part
of CI, since we do not have a Flutter checkout available. Thus, for the
time being flutter testing is local only. Maybe, later we'll move this
script to another repository.
R=sigurdm@google.com
Cq-Include-Trybots: luci.dart.try:pkg-linux-release-try,pkg-win-release-try
Change-Id: I3a99939ec5217b9f3a855fc5b9ad9699047d02cc
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/507321
Commit-Queue: Jonas Jensen <jonasfj@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
When compiled to WebAssembly, the Dart SDK needs access to external
host functions to implement regular expressions, stack traces, timers
and more. Currently, `dart2wasm` relies on `js_interop` definition to
implement these functions in JavaScript.
As discussed in https://github.com/dart-lang/sdk/issues/53884, an
alternative is to use `wasm:import` annotations to let an arbitrary
embedder that doesn't necessarily run in a JavaScript context inject
implementations for these host functions.
This would allow running `dart2wasm` apps by e.g.
- using a runtime like wasmtime and defining host functions in Rust.
- defining a wrapper module implementing required functions by
delegating to WASI definitions, and then using say `wasm-merge` to
run the app in any WASI-compatible runtime.
This prepares the `--standalone` flag on `dart2wasm` to do just that.
When enabled, the compiler uses a different SDK platform to use imports
instead of JS interop. For now, these platforms are almost identical:
I've ported the timer logic to use wasm imports as a demo, but the rest
is still based on existing patch files. We can revisit in subsequent
CLs to incrementally reduce `js_interop` dependencies before removing
that library from the `dart2wasm_standalone` target entirely.
Change-Id: I3f406afbf2dab65506094de5c3f4067f4db66f3e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/486380
Reviewed-by: Slava Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
This change includes an initial implementation of the new VM service
implementation based on `package:dart_runtime_service`, along with the
necessary plumbing to start it in place of the legacy VM service
implementation.
The entrypoint for the new VM service implementation is located in
dart_runtime_service_vm/bin/vm_service_entrypoint.dart, which is
compiled into AppJIT and AOT snapshots when the
`--include-experimental-vm-service` flag is provided to `build.py`. To run
the VM with the new VM service implementation, the
`--experimental-vm-service` flag must be provided.
Currently, the experimental VM service implementation supports:
- User specified ports
- Authentication code flags
- Enabling the HTTP server via SIGQUIT
- Some service protocol RPCs that don't require an isolate ID (e.g.,
`getVM`)
See go/dart-runtime-services-unification for more details.
TEST=Manual
CoreLibraryReviewExempt: dart:_vmservice is private
Change-Id: I4a58cd1fa0a386313baa3d5c5345720231279123
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/484820
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Ben Konyi <bkonyi@google.com>
This change creates `pkg/dart_runtime_services/` with the initial
implementation of `package:dart_runtime_service`.
`package:dart_runtime_service` provides a simple interface to create
Dart services with common platform-agnostic features and behaviors,
while also allowing for configuration of custom service backends.
This initial implementation focuses on providing the initial shared
server functionality, including:
- Handlers for web socket and SSE connections
- Authentication code generation + validation
- Client management
- Initial RPC routing with two sample RPCs (getClientName, setClientName)
Change-Id: I1458269a5de7edd97120103a2c5fc2d0348eff31
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/478760
Reviewed-by: Alexander Thomas <athom@google.com>
Commit-Queue: Ben Konyi <bkonyi@google.com>
This roll moves `package:record_use` to the dart-lang/native repo.
Change-Id: I31183dc8b72272d7e94ed3031ca0b8bfca583e0d
Cq-Include-Trybots: luci.dart.try:pkg-linux-debug-try,pkg-linux-release-arm64-try,pkg-linux-release-try,pkg-mac-release-arm64-try,pkg-mac-release-try,pkg-win-release-arm64-try,pkg-win-release-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/463662
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Michael Goderbauer <goderbauer@google.com>
Shards on vm-aot-android-debug-arm64c and vm-aot-android-debug-arm_x64
bots have been timing out recently, exceeding 1.5h testing time.
It seems like they are making progress, but number of tests is quite
large.
Increase number of shards from 10 to 16 in order to reduce running
time of each shard and avoid timeouts.
Change-Id: Id53d2cdf1eaa2606874118153c0f9c822c212daf
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/461840
Auto-Submit: Alexander Markov <alexmarkov@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Previously these were being run with the standard "pkg" unit test suite.
All the other backends have their own unit test suite so this introduces
one for dart2wasm.
This ensures the tests run with the correct fileset (including d8).
Change-Id: I29cd0048a4d6731da3a20110f6e496fe7da0c3b6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/454760
Commit-Queue: Nate Biggs <natebiggs@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
This adds a wasm binary reader that produces an `ir.Module`.
We also make a few changes to existing code
* Represent the import section with an `ir.Imports` object (similar to
`ir.Exports`, `ir.Functions`, ...)
* We make a bunch of data structures allocatable in uninitialized state
(the fields being usually uninitialized `late final` fields) where the
deserializer can create those objects and then fill in details later.
=> This comes partly due to the way wasm binaries are structured
themselves: The "data count" section comes first so a reader knows
how many data sections there will be, then the "code section" can
refer to those data sections. Then afterwards the actual "data
segment" comes that fills in the data of the section.
* We make names consistently optional: Wasm objects don't have to have
names, so the names should be optional, so we make them `String?`. We
also make them non-final as that's consistent with other names.
* We make the `ir.Types`, `ir.Functions`, ... objects have `operator[]`
and the index used is the same index used e.g. in wasm instructions.
* We make static constants for section ids and custom section names.
Issue https://github.com/dart-lang/sdk/issues/60928
Change-Id: I5394d6b82cf4dc68d24cea1dee66c5b33eb2f60f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/452144
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ömer Ağacan <omersa@google.com>
When the shard count is greater than the number of machines in the pool, shards are guarenteed to wait, increase latency relative to a lower shard count.
Change-Id: I1fd61a59837b767748c13120dec30ca6eb352f28
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/450946
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
* Shard dart2js modular tests (and decrease the shards of the
unit tests)
* Remove folders we look for dart files in --- the remove folders
doesn't contain any anyway and would have to be copied if keeping
these lines.
* Fix sharding, previously trying to shard in 2 shards would only allow
you to run ~50% of the tests:
```
$ out/ReleaseX64/dart-sdk/bin/dart pkg/compiler/tool/modular_test_suite.dart -nweb-unittest-asserts-linux --verbose --use-sdk --shards=2 --shard=0
Error: shard should be between 0 and 1, but got 0
$ out/ReleaseX64/dart-sdk/bin/dart pkg/compiler/tool/modular_test_suite.dart -nweb-unittest-asserts-linux --verbose --use-sdk --shards=2 --shard=2
Error: shard should be between 0 and 1, but got 2
```
This has been corrected to allow from `1..n` for `n` shards to fit
with what the testing system sends when specifying `shards` in
`tools/bots/test_matrix.json`.
For previous try runs I extracted this:
Build 60615:
Shard #1: --- Total time: 04:27 ---
Shard #2: --- Total time: 05:19 ---
Shard #3: --- Total time: 04:23 ---
Shard #4: --- Total time: 10:29 ---
=> A total of 24:35 --- combined finish of 10:29
Modular tests: 32:19
Total bot runtime: 35:48
Build 60614:
Shard #1: --- Total time: 03:57 ---
Shard #2: --- Total time: 05:18 ---
Shard #3: --- Total time: 05:21 ---
Shard #4: --- Total time: 05:34 ---
=> A total of 20:10 --- combined finish of 5:34
Modular tests: 29:41 secs
Total bot runtime: 35:55
Build: 60613
Shard #1: --- Total time: 03:56 ---
Shard #2: --- Total time: 05:15 ---
Shard #3: --- Total time: 04:32 ---
Shard #4: --- Total time: 05:34 ---
=> A total of 19:17 --- combined finish of 5:34
Modular tests: 33:39
Total bot runtime: 38:51
With the new sharding I'd estimate that the unit tests and modular tests
would have finished in less than 13 minutes, making the bots finish in
~17 minutes, ~20 minutes and ~19 minutes instead.
The try-run with this ran in 17:06
Possibly a follow-up could do more stuff on the "main bot".
Change-Id: Ie5c96206deb9c0c6db3385bbca04ae6f4eab4c3a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/448381
Reviewed-by: Nate Biggs <natebiggs@google.com>
Reviewed-by: Ivan Inozemtsev <iinozemtsev@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
Call __tsan_func_entry/__tsan_func_exit in functions that use __tsan_read/__tsan_write or call other functions. Call __tsan_func_exit once per frame when unwinding for exceptions. Do so only in AOT, since TSAN won't be able to symbolize JIT functions anyway.
TEST=ci
Bug: https://github.com/dart-lang/sdk/issues/61352
Cq-Include-Trybots: luci.dart.try:vm-tsan-linux-release-arm64-try,vm-tsan-linux-release-x64-try
Change-Id: Ie52c978c25664d78b834e9b72ecf7eb2a12cc2ba
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/444181
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
The directory location depends on the current OS:
- on Windows:
- `%LOCALAPPDATA%\Dart\<tool>`
- on Mac OS:
- `$HOME/Library/Application Support/Dart/<tool>`
- on Linux:
- `$XDG_STATE_HOME/Dart/<tool>` if `$XDG_STATE_HOME` is defined,
and
- `$HOME/.local/state/Dart/<tool>` otherwise.
The Dart data home can be overridden with the `DART_DATA_HOME`
environment variable.
This CL does not start using the new location yet.
Bug: https://github.com/dart-lang/sdk/issues/60922
Bug: https://github.com/dart-lang/sdk/issues/41560
Bug: https://github.com/flutter/flutter/issues/59430
Change-Id: I55e0ba610f8665ea3c9f053b36c943b097313046
Cq-Include-Trybots: luci.dart.try:pkg-linux-debug-try,pkg-linux-release-arm64-try,pkg-linux-release-try,pkg-mac-release-arm64-try,pkg-mac-release-try,pkg-win-release-arm64-try,pkg-win-release-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/440462
Reviewed-by: Alexander Thomas <athom@google.com>
Auto-Submit: Daco Harkes <dacoharkes@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Introduce Dart_LoadLibraryFromBytecode to support loading unit test
libraries from bytecode.
TEST=ci
Change-Id: I7706fdb2acaf906f01f27f671e7193b9755efe0f
Cq-Include-Trybots: luci.dart.try:vm-aot-dyn-linux-debug-x64-try,vm-aot-dyn-linux-product-x64-try,vm-dyn-linux-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/439080
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
This new flag will run the CFE to create a kernel and then run a series of checks over the resulting kernel to look for errors that could block a wasm migration.
The compiler will then exit before actually starting the wasm compilation process. This means no output file will be emitted so callers must be aware of this.
This first CL is not meant to cover every check we could add here. It adds some initial checks and we can expand on this to include more in follow-up changes.
One of the checks implemented here is also provided by a lint. While ideally we would share code between lints and these checks, the delta in the CFE vs analyzer model makes that infeasible today.
Sample output:
```
Found incompatibilities with WebAssembly.
package:dryrun/test.dart 5:15 - Cannot test a JS value against String (3)
package:dryrun/test.dart 6:7 - JS interop class 'B' cannot extend Dart class 'A'. (2)
```
Bug: https://github.com/dart-lang/sdk/issues/60050
Change-Id: Ib2c8e3501cc42d57b86ebaa749359ce6c5dba974
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/437960
Commit-Queue: Nate Biggs <natebiggs@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
kevmoo