The usual deflaking logic is to simply rerun the test 5 times, but if the failure was a timeout this can be very expensive. Instead, only rerun timeout failures twice, and use the last successful run time to set a tighter timeout.
Repeat counts and timeouts are now per-test. The name, repeat count, and timeout make a `DeflakeInfo`. This object is constructed in compare_results.dart, and passed to test.py's `--tests` flag. The recipe was already passing the output of compare_results directly to that flag, so no change is needed to the recipies.
Backwards compatibility:
If you want the old behavior of compare_results.dart, use the `--name-only` flag.
test.py only uses this new logic if it detects that the `--tests` flag is JSON (if it starts with a `{`).
Change-Id: I4113b68c54bfb7fd9a5e8fc9dab7a265807f3e77
Bug: https://github.com/dart-lang/sdk/issues/55044
Fixes: https://github.com/dart-lang/sdk/issues/55044
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/435760
Commit-Queue: Liam Appelbe <liama@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
The name has always been annoying because it did not add strong typing. And now there aren't variants of the VM platform to distinguish.
Leave a copy at the old name to not immediately break illegal uses.
TEST=ci
Cq-Include-Trybots: luci.dart.try:flutter-analyze-try,flutter-frontend-try,flutter-linux-try
Change-Id: Ie76fa7f16940aa1ba8d582eb5197f0ae55dc8938
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/429828
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
dart2bytecode can optionally add source position information
(including line starts) to the generated bytecode
(when '--bytecode-options=source-positions' flag is specified).
If bytecode has source positions, they are now shown in stack traces
involving interpreter frames.
TEST=ci
Change-Id: I1ae3326bac21201040be32c712514e71e96f51e2
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/433760
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
Use "out.aotsnapshot" as the name for all non-assembled snapshots and
names with the appropriate shared object extension for the host OS for
assembled snapshots.
This ensures the test checks for .dll extensions added in
https://dart-review.googlesource.com/c/sdk/+/428200 only trigger
for assembled snapshots as expected, and not for other formats
(e.g., Windows trybots using sanitizers, which still use direct
to ELF snapshots).
Also fixes incorrect line numbers in dwarf_stack_trace_test.dart.
TEST=vm/dart/build_id_test
standalone/dwarf_stack_trace_test
Cq-Include-Trybots: luci.dart.try:vm-asan-win-release-x64-try,vm-linux-release-x64-try,vm-mac-release-arm64-try
Bug: https://github.com/dart-lang/sdk/issues/60846
Change-Id: Ife06613d86874d76adeb99dcbc50fc3723aae584
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/432660
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
`tools/test.py --help` output before: (only relevant parts)
```
-c, --compiler How the Dart code should be compiled or statically processed.
dart2js: Compile to JavaScript using dart2js.
dart2analyzer: Perform static analysis on Dart code using the analyzer.
compare_analyzer_cfe: Compare analyzer and common front end representations.
ddc: Compile to JavaScript using dartdevc.
app_jitk: Compile the Dart code into Kernel and then into an app
snapshot.
dartk: Compile the Dart code into Kernel before running test.
dartkp: Compile the Dart code into Kernel and then Kernel into
AOT snapshot before running the test.
spec_parser: Parse Dart code using the specification parser.
fasta: Compile using CFE for errors, but do not run.
[dart2js, dart2analyzer, dart2wasm, ddc, app_jitk, dartk, dartkp, spec_parser, fasta, dart2bytecode]
-r, --runtime Where the tests should be run.
vm: Run Dart code on the standalone Dart VM.
dart_precompiled: Run a precompiled snapshot on the VM without a JIT.
d8: Run JavaScript from the command line using Chrome's v8.
jsc: Run JavaScript from the command line using Safari/WebKit's jsc.
jsshell: Run JavaScript from the command line using Firefox's js-shell.
firefox:
chrome:
safari:
chromeOnAndroid: Run JavaScript in the specified browser.
none: No runtime, compile only.
[vm, flutter, dart_precompiled, d8, jsc, jsshell, firefox, chrome, safari, edge, chromeOnAndroid, none]
...
-p, --progress Progress indication mode.
Allowed values are:
compact, color, line, verbose, silent, status
[compact (default), color, line, verbose, silent, status]
...
--nnbd Which set of non-nullable type features to use.
Allowed values are: legacy, weak, strong
[legacy, weak, strong (default)]
```
After:
```
-c, --compiler How the Dart code should be compiled or statically processed.
[dart2js] Compile to JavaScript using dart2js.
[dart2analyzer] Perform static analysis on Dart code using the analyzer.
[compare_analyzer_cfe] Compare analyzer and common front end representations.
[ddc] Compile to JavaScript using dartdevc.
[app_jitk] Compile the Dart code into Kernel and then into an app snapshot.
[dartk] Compile the Dart code into Kernel before running test.
[dartkp] Compile the Dart code into Kernel and then Kernel into AOT snapshot before running the test.
[spec_parser] Parse Dart code using the specification parser.
[fasta] Compile using CFE for errors, but do not run.
-r, --runtime Where the tests should be run.
[vm] Run Dart code on the standalone Dart VM.
[dart_precompiled] Run a precompiled snapshot on the VM without a JIT.
[d8] Run JavaScript from the command line using Chrome's v8.
[jsc] Run JavaScript from the command line using Safari/WebKit's jsc.
[jsshell] Run JavaScript from the command line using Firefox's js-shell.
[firefox] Run JavaScript in Firefox.
[chrome] Run JavaScript in Chrome.
[safari] Run JavaScript in Safari.
[chromeOnAndroid] Run JavaScript in Chrome on Android.
[none] No runtime, compile only.
...
-p, --progress Progress indication mode.
[compact (default), color, line, verbose, silent, status]
--nnbd Which set of non-nullable type features to use.
[legacy, weak, strong (default)]
```
Changes:
- Duplicate "allowed values are ..." phrases removed.
- With `package:args`'s `allowedHelp` messages, allowed values documentations are now consistent with the rest of the ecosystem.
Change-Id: I25d5e310e7dd38e0e94622815d9773ec84ab8fb3
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/431860
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Ömer Ağacan <omersa@google.com>
When the test runner output is piped to a file, 'compact' progress
indicator prints carriage returns to the file.
To avoid adding carriage returns to text files, only use 'compact'
indicator when stdout is a terminal. Otherwise use 'line' indicator.
Change-Id: Ic8ca111059799876caf401d3b040a5d3b5a7c731
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/431861
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Ömer Ağacan <omersa@google.com>
Currently, the configuration code only has a boolean option, 'use-elf'
for specifying the output format for gen_snapshot. If false, then the
output format is assumed to be assembly.
Add a new GenSnapshotFormat enum and replace the old 'use-elf' option
with a new 'gen-snapshot-format' option. The new enum has two getters:
* snapshotType returns the appropriate string to pass to gen_snapshot
via the '--snapshot-type' option.
* fileOption returns the name of the option used to specify the
output path for the given format.
In addition, make it so the Configuration only has a non-null
genSnapshotFormat field if the compiler is Compiler.dartkp to avoid
spurious differences in Configurations that are created with different
GenSnapshotFormat values when the configurations in question don't
actually call gen_snapshot.
TEST=pkg/smith/test/configuration_test.dart (and the CI in general)
Change-Id: I7f17dc8d3a1bb6d4bf57750bc5ef4a16b8a78c11
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/429980
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
`package:native_assets_cli` has been split up and
`package:native_assets_builder` has been renamed.
Change-Id: Ic9d5c21bb3c7ecf7924fdee6b4281153ad37fac5
Cq-Include-Trybots: luci.dart.try:pkg-linux-debug-try,pkg-linux-release-arm64-try,pkg-linux-release-try,pkg-mac-release-arm64-try,pkg-win-release-arm64-try,pkg-mac-release-try,pkg-win-release-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/425504
Reviewed-by: Ivan Inozemtsev <iinozemtsev@google.com>
Commit-Queue: Daco Harkes <dacoharkes@google.com>
This ensures we exercise the `<app>.support.js` code in our tests and
also catch it if a browser doesn't support what's needed (e.g. running
tests on Safari with `js-string` builtin)
Change-Id: Ib8874231d3aa82b598e0e206a2e27cb66775bbec
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/415882
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ömer Ağacan <omersa@google.com>
When parsing `--define` or `-D` arguments don't split the the value by
commas.
This is consistent with how dart2js handles `-D`, but inconsistent with
how VM handles it.
Example:
void main() {
print(const String.fromEnvironment("FOO"));
}
When compiled with `dart compile js -DFOO="a, b"` and run, dart2js
prints
a, b
VM prints (when compiled to exe)
a
Between these two, I think dart2js' behavior is more common, so we
follow dart2js.
Also update compile_benchmark to avoid splitting a single argument "a b"
into "a" and "b" when parsing the arguments and then splicing them back
before calling `dart2wasm`.
Also update the test runner and ddc batch mode argument parser to handle
splitting quoted arguments in `// dart2jsOption = ...` and the same
options for ddc and dart2wasm, by moving dart2js's `splitLine` to a new
library and reusing it in the test runner and ddc.
Fixes https://github.com/flutter/flutter/issues/164873.
See also https://github.com/dart-lang/sdk/issues/60341 for relevant
future work.
Change-Id: Idbdf69072fa212c8e4a390990577eb5a57b49e8a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/415280
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Ömer Ağacan <omersa@google.com>
The static error test format allows an expected error message to contain newlines like:
```dart
some bad code;
// [cfe] The error message
// is two lines long.
```
Here, the expected error is "The error message\nistwo lines long."
This has an unfortunate side effect in that if you have a comment
immediately after a static error expectation, it will be treated as part
of the expected error:
```dart
some bad code;
// [cfe] Error.
// Unrelated comment.
```
This will expect the CFE to report "Error.\nUnrelated comment."
I looked at every static error test and there are no intentional uses
of multi-line expectations. The only two I found were both bugs.
Since this functionality is unused and error-prone, this CL removes it.
Error expectations can only be one line.
Fix#60137.
Change-Id: I7edb1f60a0c87d4160b93810806de49bc8f55b15
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/412185
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Commit-Queue: Johnni Winther <johnniwinther@google.com>
* Simplify runCommand using async/await.
* Remove outdated memory leak hack for DDC.
* Gather batch runner functionality in BatchRunnerProcess.
* Fix process leak that likely caused a hang on test.py shutdown.
* Add a test.
Change-Id: I87584e4c951ad0fe1650c67db090910acf0f37e4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/408700
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
Running tests currently result in a warning by D8:
Warning: unknown flag --turboshaft-wasm.
Turboshaft was enabled by default in V8 and is no longer exposed as a
flag.
Change-Id: I487f145ba60b20d7c61397c4ad4c5af1d0263b60
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/404320
Reviewed-by: Ömer Ağacan <omersa@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
Prior to this change, the static error test updater would spawn a new
analyzer CLI for each updated test file. This was incredibly slow, like
5-10 seconds for each file.
This change instead uses the analyzer as a library. It creates a single
analysis context collection for all of the files to be updated and then
gets the errors from that.
On my machine, it's about 100x faster. It's fast enough to routinely
update *all* of the test expectations in a batch. (Or, at least, it will
be if we can make a similar change for CFE.)
Change-Id: I39059489e966abf040e9f995505c7013b18d4930
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/396574
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Reviewed-by: Paul Berry <paulberry@google.com>
Commit-Queue: Konstantin Shcheglov <scheglov@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
The old formatter had a special rule that if a line comment as at the
left edge of the page, it would stay there regardless of the surrounding
indentation. So if you had:
```
class C {
m() {
// comment
}
}
```
After formatting, the comment would still be there instead of being
indented. The intent of that was to not shift over code that had been
commented out.
But all of the IDEs I tested don't actually work that way. When they
comment out code, they tend to put the `//` at the indentation of the
surrounding code. So the new formatter doesn't have this special rule
and always indents line comments following the surrounding indentation.
This is good because it also means that code generators that don't write
any leading whitespace will still get nicely formatted comments.
However, the static error test updater took advantage of this rule and
would write static error marker comments at column zero if needed to
get the carets to align with the code on the previous line and assume
that the formatter wouldn't move the comment.
This fixes the static error test updater. It always writes comments
using indentation from the previous line of code and if the caret
doesn't fit that way, it uses an explicit column marker.
Fix#57042.
Change-Id: I40fd7cd19d08dc228b6a6797e6a26965d1343d32
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/394363
Reviewed-by: Nate Bosch <nbosch@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Nate Bosch <nbosch@google.com>
This reverts commit 74c5aa3a7a.
Reason for revert: Fix golem breakage by not changing the script dart_precompiled_runtime2
TEST=ci
Original change's description:
> Revert ""[SDK/VM] - Rename dart_precompiled_runtime to dartaotruntime, ensures we have a uniform name for the executable between the build directories and the SDK directory""
>
> This reverts commit f81a402aa1.
>
> Reason for revert: golem benchmarks are failing to run
>
> TEST=ci
>
> Original change's description:
> > "[SDK/VM] - Rename dart_precompiled_runtime to dartaotruntime, ensures we have a uniform name for the executable between the build directories and the SDK directory"
> >
> > Fixed golem breakage by temporarily copying dartaotruntime to dart_precompiled_runtime
> >
> > This reverts commit 75e6a748f7.
> >
> > TEST=ci
> >
Change-Id: I9efe40643c59bc617f6fb484b89b038deaffbb93
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/393941
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Siva Annamalai <asiva@google.com>
Adds a stackTrace helper that returns a stringified StackTrace for
an arbitrary error. Removes importLibrary references in test_runner
now that we have that and removes some calls that didn't do anything
(addAsyncCallback, removeAsyncCallback, and startRootIsolate).
addAsyncCallback and removeAsyncCallback are also removed from the
runtime. Calling these were removed in https://github.com/dart-lang/sdk/commit/8bd3690ab06c3eda2f4996ca60b94fdcc9f76105
so this is just finishing up the clean-up.
Change-Id: I9dacc2b3fd3e27468061d6e59a3207eecdb7cac8
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/393800
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Srujan Gaddam <srujzs@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
This reverts commit f81a402aa1.
Reason for revert: golem benchmarks are failing to run
TEST=ci
Original change's description:
> "[SDK/VM] - Rename dart_precompiled_runtime to dartaotruntime, ensures we have a uniform name for the executable between the build directories and the SDK directory"
>
> Fixed golem breakage by temporarily copying dartaotruntime to dart_precompiled_runtime
>
> This reverts commit 75e6a748f7.
>
> TEST=ci
>
> Original change's description:
> > Revert "[SDK/VM] - Rename dart_precompiled_runtime to dartaotruntime, ensures we have a uniform name for the executable between the build directories and the SDK directory"
> >
> > This reverts commit 1b331d05c2.
> >
> > Reason for revert: golem builds are failing
> >
> > Original change's description:
> > > [SDK/VM] - Rename dart_precompiled_runtime to dartaotruntime, ensures we have a uniform name for the executable between the build directories and the SDK directory
> > >
> > > TEST=ci
> > >
>
> Change-Id: Id0f383eabb496c06c0acebc639c8e3b056ba82d0
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/393781
> Commit-Queue: Siva Annamalai <asiva@google.com>
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
Change-Id: Iec494940412aa31dbefdc5280e35ae99e8cecb26
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/393764
Reviewed-by: Siva Annamalai <asiva@google.com>
Reviewed-by: Liam Appelbe <liama@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Siva Annamalai <asiva@google.com>
Fixed golem breakage by temporarily copying dartaotruntime to dart_precompiled_runtime
This reverts commit 75e6a748f7.
TEST=ci
Original change's description:
> Revert "[SDK/VM] - Rename dart_precompiled_runtime to dartaotruntime, ensures we have a uniform name for the executable between the build directories and the SDK directory"
>
> This reverts commit 1b331d05c2.
>
> Reason for revert: golem builds are failing
>
> Original change's description:
> > [SDK/VM] - Rename dart_precompiled_runtime to dartaotruntime, ensures we have a uniform name for the executable between the build directories and the SDK directory
> >
> > TEST=ci
> >
Change-Id: Id0f383eabb496c06c0acebc639c8e3b056ba82d0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/393781
Commit-Queue: Siva Annamalai <asiva@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Converts kernel_worker.dart.snapshot and dartdevc.dart.snapshot to
AOT snapshots. Fixes tests and paths in the code accordingly.
TESTS=ci
Change-Id: Ib99b2a3343e23252c3a6b5295b0d20f0f486aede
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/381388
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Siva Annamalai <asiva@google.com>
Collects files from `package:async_helper` and `tests/language`
that are generally useful, so that all test-related helpers are
in `package:expect`.
Moves the two libraries from `package:async_helper` into `package:expect`,
and the `tests/language/static_type_helper.dart` file too.
Deprecates `async_minitest.dart`, to follow `minitest.dart`,
expecting the Flutter use of it to have been fixed to not break
on deprecation (I believe Flutter no longer breaks builds on deprecations at all).
Patch 1 is the actual change.
Patch 2+4+8 is changing all existing references to the files.
Patch 6 ignores deprecation in files still using `async_minitest.dart`.
3+5+7+9 are updating this text to make the numbers match.
Then it's just test-expectations and small tweaks from there.
Change-Id: I1b665135b5fef9b9a0c3b340ffe9daf874d0174c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/373120
Reviewed-by: Nate Bosch <nbosch@google.com>
Reviewed-by: Devon Carew <devoncarew@google.com>
Commit-Queue: Lasse Nielsen <lrn@google.com>
Ryan Macnak