The usual deflaking logic is to simply rerun the test 5 times, but if the failure was a timeout this can be very expensive. Instead, only rerun timeout failures twice, and use the last successful run time to set a tighter timeout.
Repeat counts and timeouts are now per-test. The name, repeat count, and timeout make a `DeflakeInfo`. This object is constructed in compare_results.dart, and passed to test.py's `--tests` flag. The recipe was already passing the output of compare_results directly to that flag, so no change is needed to the recipies.
Backwards compatibility:
If you want the old behavior of compare_results.dart, use the `--name-only` flag.
test.py only uses this new logic if it detects that the `--tests` flag is JSON (if it starts with a `{`).
Change-Id: I4113b68c54bfb7fd9a5e8fc9dab7a265807f3e77
Bug: https://github.com/dart-lang/sdk/issues/55044
Fixes: https://github.com/dart-lang/sdk/issues/55044
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/435760
Commit-Queue: Liam Appelbe <liama@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
The name has always been annoying because it did not add strong typing. And now there aren't variants of the VM platform to distinguish.
Leave a copy at the old name to not immediately break illegal uses.
TEST=ci
Cq-Include-Trybots: luci.dart.try:flutter-analyze-try,flutter-frontend-try,flutter-linux-try
Change-Id: Ie76fa7f16940aa1ba8d582eb5197f0ae55dc8938
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/429828
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
dart2bytecode can optionally add source position information
(including line starts) to the generated bytecode
(when '--bytecode-options=source-positions' flag is specified).
If bytecode has source positions, they are now shown in stack traces
involving interpreter frames.
TEST=ci
Change-Id: I1ae3326bac21201040be32c712514e71e96f51e2
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/433760
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Use "out.aotsnapshot" as the name for all non-assembled snapshots and
names with the appropriate shared object extension for the host OS for
assembled snapshots.
This ensures the test checks for .dll extensions added in
https://dart-review.googlesource.com/c/sdk/+/428200 only trigger
for assembled snapshots as expected, and not for other formats
(e.g., Windows trybots using sanitizers, which still use direct
to ELF snapshots).
Also fixes incorrect line numbers in dwarf_stack_trace_test.dart.
TEST=vm/dart/build_id_test
standalone/dwarf_stack_trace_test
Cq-Include-Trybots: luci.dart.try:vm-asan-win-release-x64-try,vm-linux-release-x64-try,vm-mac-release-arm64-try
Bug: https://github.com/dart-lang/sdk/issues/60846
Change-Id: Ife06613d86874d76adeb99dcbc50fc3723aae584
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/432660
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
`tools/test.py --help` output before: (only relevant parts)
```
-c, --compiler How the Dart code should be compiled or statically processed.
dart2js: Compile to JavaScript using dart2js.
dart2analyzer: Perform static analysis on Dart code using the analyzer.
compare_analyzer_cfe: Compare analyzer and common front end representations.
ddc: Compile to JavaScript using dartdevc.
app_jitk: Compile the Dart code into Kernel and then into an app
snapshot.
dartk: Compile the Dart code into Kernel before running test.
dartkp: Compile the Dart code into Kernel and then Kernel into
AOT snapshot before running the test.
spec_parser: Parse Dart code using the specification parser.
fasta: Compile using CFE for errors, but do not run.
[dart2js, dart2analyzer, dart2wasm, ddc, app_jitk, dartk, dartkp, spec_parser, fasta, dart2bytecode]
-r, --runtime Where the tests should be run.
vm: Run Dart code on the standalone Dart VM.
dart_precompiled: Run a precompiled snapshot on the VM without a JIT.
d8: Run JavaScript from the command line using Chrome's v8.
jsc: Run JavaScript from the command line using Safari/WebKit's jsc.
jsshell: Run JavaScript from the command line using Firefox's js-shell.
firefox:
chrome:
safari:
chromeOnAndroid: Run JavaScript in the specified browser.
none: No runtime, compile only.
[vm, flutter, dart_precompiled, d8, jsc, jsshell, firefox, chrome, safari, edge, chromeOnAndroid, none]
...
-p, --progress Progress indication mode.
Allowed values are:
compact, color, line, verbose, silent, status
[compact (default), color, line, verbose, silent, status]
...
--nnbd Which set of non-nullable type features to use.
Allowed values are: legacy, weak, strong
[legacy, weak, strong (default)]
```
After:
```
-c, --compiler How the Dart code should be compiled or statically processed.
[dart2js] Compile to JavaScript using dart2js.
[dart2analyzer] Perform static analysis on Dart code using the analyzer.
[compare_analyzer_cfe] Compare analyzer and common front end representations.
[ddc] Compile to JavaScript using dartdevc.
[app_jitk] Compile the Dart code into Kernel and then into an app snapshot.
[dartk] Compile the Dart code into Kernel before running test.
[dartkp] Compile the Dart code into Kernel and then Kernel into AOT snapshot before running the test.
[spec_parser] Parse Dart code using the specification parser.
[fasta] Compile using CFE for errors, but do not run.
-r, --runtime Where the tests should be run.
[vm] Run Dart code on the standalone Dart VM.
[dart_precompiled] Run a precompiled snapshot on the VM without a JIT.
[d8] Run JavaScript from the command line using Chrome's v8.
[jsc] Run JavaScript from the command line using Safari/WebKit's jsc.
[jsshell] Run JavaScript from the command line using Firefox's js-shell.
[firefox] Run JavaScript in Firefox.
[chrome] Run JavaScript in Chrome.
[safari] Run JavaScript in Safari.
[chromeOnAndroid] Run JavaScript in Chrome on Android.
[none] No runtime, compile only.
...
-p, --progress Progress indication mode.
[compact (default), color, line, verbose, silent, status]
--nnbd Which set of non-nullable type features to use.
[legacy, weak, strong (default)]
```
Changes:
- Duplicate "allowed values are ..." phrases removed.
- With `package:args`'s `allowedHelp` messages, allowed values documentations are now consistent with the rest of the ecosystem.
Change-Id: I25d5e310e7dd38e0e94622815d9773ec84ab8fb3
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/431860
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Ömer Ağacan <omersa@google.com>
When the test runner output is piped to a file, 'compact' progress
indicator prints carriage returns to the file.
To avoid adding carriage returns to text files, only use 'compact'
indicator when stdout is a terminal. Otherwise use 'line' indicator.
Change-Id: Ic8ca111059799876caf401d3b040a5d3b5a7c731
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/431861
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Ömer Ağacan <omersa@google.com>
Currently, the configuration code only has a boolean option, 'use-elf'
for specifying the output format for gen_snapshot. If false, then the
output format is assumed to be assembly.
Add a new GenSnapshotFormat enum and replace the old 'use-elf' option
with a new 'gen-snapshot-format' option. The new enum has two getters:
* snapshotType returns the appropriate string to pass to gen_snapshot
via the '--snapshot-type' option.
* fileOption returns the name of the option used to specify the
output path for the given format.
In addition, make it so the Configuration only has a non-null
genSnapshotFormat field if the compiler is Compiler.dartkp to avoid
spurious differences in Configurations that are created with different
GenSnapshotFormat values when the configurations in question don't
actually call gen_snapshot.
TEST=pkg/smith/test/configuration_test.dart (and the CI in general)
Change-Id: I7f17dc8d3a1bb6d4bf57750bc5ef4a16b8a78c11
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/429980
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
`package:native_assets_cli` has been split up and
`package:native_assets_builder` has been renamed.
Change-Id: Ic9d5c21bb3c7ecf7924fdee6b4281153ad37fac5
Cq-Include-Trybots: luci.dart.try:pkg-linux-debug-try,pkg-linux-release-arm64-try,pkg-linux-release-try,pkg-mac-release-arm64-try,pkg-win-release-arm64-try,pkg-mac-release-try,pkg-win-release-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/425504
Reviewed-by: Ivan Inozemtsev <iinozemtsev@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
This is a reland of commit b9b77058a9
Original change's description:
> Migrate to use pub workspace
>
> Use `pub get` to generate `.dart_tool/package_config.json` on gclient sync.
>
> All pkg/ (and a few third_party) packages that are developed inside the sdk repo are included in the workspace from the root `pubspec.yaml`.
>
> All dependencies that are pulled in via DEPS are added as path dependencies via `dependency_overrides` in the root `pubspec.yaml`.
>
> Bug: https://github.com/dart-lang/sdk/issues/56220
> Change-Id: I38c12b608c68da54c57821116cf9aa6696936746
> Tested: relies on CQ of existing tests. Should have no effect on functionality
> CoreLibraryReviewExempt: only core library change is adding a `// ignore:` comment. Should have no influence on functionality
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/397164
> Commit-Queue: Sigurd Meldgaard <sigurdm@google.com>
> Reviewed-by: Alexander Thomas <athom@google.com>
Bug: https://github.com/dart-lang/sdk/issues/56220
Change-Id: I29afabade2d2447dea05121cb87ff50bb21a4b76
Cq-Include-Trybots: luci.dart.try:flutter-linux-try,flutter-web-try
Tested: relies on CQ of existing tests. Should have no effect on functionality
CoreLibraryReviewExempt: only core library change is adding a `//
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/415561
Commit-Queue: Sigurd Meldgaard <sigurdm@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
This ensures we exercise the `<app>.support.js` code in our tests and
also catch it if a browser doesn't support what's needed (e.g. running
tests on Safari with `js-string` builtin)
Change-Id: Ib8874231d3aa82b598e0e206a2e27cb66775bbec
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/415882
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ömer Ağacan <omersa@google.com>
When parsing `--define` or `-D` arguments don't split the the value by
commas.
This is consistent with how dart2js handles `-D`, but inconsistent with
how VM handles it.
Example:
void main() {
print(const String.fromEnvironment("FOO"));
}
When compiled with `dart compile js -DFOO="a, b"` and run, dart2js
prints
a, b
VM prints (when compiled to exe)
a
Between these two, I think dart2js' behavior is more common, so we
follow dart2js.
Also update compile_benchmark to avoid splitting a single argument "a b"
into "a" and "b" when parsing the arguments and then splicing them back
before calling `dart2wasm`.
Also update the test runner and ddc batch mode argument parser to handle
splitting quoted arguments in `// dart2jsOption = ...` and the same
options for ddc and dart2wasm, by moving dart2js's `splitLine` to a new
library and reusing it in the test runner and ddc.
Fixes https://github.com/flutter/flutter/issues/164873.
See also https://github.com/dart-lang/sdk/issues/60341 for relevant
future work.
Change-Id: Idbdf69072fa212c8e4a390990577eb5a57b49e8a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/415280
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Ömer Ağacan <omersa@google.com>
This reverts commit b9b77058a9.
Revert "Add missing sample pubspec to workspace"
This reverts commit 892ea15ac7.
These seem to break the engine.
Change-Id: Ieee26deb7928c3869a1b6265326c3ce568ffe731
Tested: this is a revert.
CoreLibraryReviewExempt: this is a revert
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/415582
Commit-Queue: Sigurd Meldgaard <sigurdm@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
Use `pub get` to generate `.dart_tool/package_config.json` on gclient sync.
All pkg/ (and a few third_party) packages that are developed inside the sdk repo are included in the workspace from the root `pubspec.yaml`.
All dependencies that are pulled in via DEPS are added as path dependencies via `dependency_overrides` in the root `pubspec.yaml`.
Bug: https://github.com/dart-lang/sdk/issues/56220
Change-Id: I38c12b608c68da54c57821116cf9aa6696936746
Tested: relies on CQ of existing tests. Should have no effect on functionality
CoreLibraryReviewExempt: only core library change is adding a `// ignore:` comment. Should have no influence on functionality
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/397164
Commit-Queue: Sigurd Meldgaard <sigurdm@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
The static error test format allows an expected error message to contain newlines like:
```dart
some bad code;
// [cfe] The error message
// is two lines long.
```
Here, the expected error is "The error message\nistwo lines long."
This has an unfortunate side effect in that if you have a comment
immediately after a static error expectation, it will be treated as part
of the expected error:
```dart
some bad code;
// [cfe] Error.
// Unrelated comment.
```
This will expect the CFE to report "Error.\nUnrelated comment."
I looked at every static error test and there are no intentional uses
of multi-line expectations. The only two I found were both bugs.
Since this functionality is unused and error-prone, this CL removes it.
Error expectations can only be one line.
Fix#60137.
Change-Id: I7edb1f60a0c87d4160b93810806de49bc8f55b15
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/412185
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Commit-Queue: Johnni Winther <johnniwinther@google.com>
* Simplify runCommand using async/await.
* Remove outdated memory leak hack for DDC.
* Gather batch runner functionality in BatchRunnerProcess.
* Fix process leak that likely caused a hang on test.py shutdown.
* Add a test.
Change-Id: I87584e4c951ad0fe1650c67db090910acf0f37e4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/408700
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
Running tests currently result in a warning by D8:
Warning: unknown flag --turboshaft-wasm.
Turboshaft was enabled by default in V8 and is no longer exposed as a
flag.
Change-Id: I487f145ba60b20d7c61397c4ad4c5af1d0263b60
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/404320
Reviewed-by: Ömer Ağacan <omersa@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
The static error test updater needs to ask analyzer, CFE, and dart2js
for the errors for each updated test file. Previously, it would invoke
each of those as a separate process, one at a time, for each test file.
This was comically slow.
I recently updated it to invoke analyzer as a library and analyze all
the files at once, which made that part >100x faster.
This CL does essentially the same thing for CFE. It's still invoking CFE
as a process, but it does so with a batch of files. It's not as fast as
analyzer is, but it's much better.
It's still calling dart2js once per file but, strangely, that isn't too
slow. Also, web static error tests aren't very common, so this isn't as
important.
Change-Id: I6756901bb761579dc90f8af4d9774014b04bb009
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/399885
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Paul Berry <paulberry@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Prior to this change, the static error test updater would spawn a new
analyzer CLI for each updated test file. This was incredibly slow, like
5-10 seconds for each file.
This change instead uses the analyzer as a library. It creates a single
analysis context collection for all of the files to be updated and then
gets the errors from that.
On my machine, it's about 100x faster. It's fast enough to routinely
update *all* of the test expectations in a batch. (Or, at least, it will
be if we can make a similar change for CFE.)
Change-Id: I39059489e966abf040e9f995505c7013b18d4930
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/396574
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Reviewed-by: Paul Berry <paulberry@google.com>
Commit-Queue: Konstantin Shcheglov <scheglov@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
I'm slowly going through and reformatting the tests. That also means
regenerating the static error expectations because of #57042. But I
don't need to generate static error markers for files that don't already
have them.
So this adds a flag to the test updater to let it skip over any test
file that isn't already known to be a static error test.
Change-Id: I61f10d29924f1f9d2dabf61bb604e53a6622017f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/394940
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Ryan Macnak