This reverts commit f95a84da5c.
Reason for revert: Testing if CL caused increased meta-flakiness of non-timeout failures.
Bug: https://github.com/dart-lang/sdk/issues/55044
Original change's description:
> [test_runner] Improve timeout deflaking
>
> The usual deflaking logic is to simply rerun the test 5 times, but if the failure was a timeout this can be very expensive. Instead, only rerun timeout failures twice, and use the last successful run time to set a tighter timeout.
>
> Repeat counts and timeouts are now per-test. The name, repeat count, and timeout make a `DeflakeInfo`. This object is constructed in compare_results.dart, and passed to test.py's `--tests` flag. The recipe was already passing the output of compare_results directly to that flag, so no change is needed to the recipies.
>
> Backwards compatibility:
>
> If you want the old behavior of compare_results.dart, use the `--name-only` flag.
>
> test.py only uses this new logic if it detects that the `--tests` flag is JSON (if it starts with a `{`).
>
> Change-Id: I4113b68c54bfb7fd9a5e8fc9dab7a265807f3e77
> Bug: https://github.com/dart-lang/sdk/issues/55044
> Fixes: https://github.com/dart-lang/sdk/issues/55044
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/435760
> Commit-Queue: Liam Appelbe <liama@google.com>
> Reviewed-by: Alexander Thomas <athom@google.com>
Bug: https://github.com/dart-lang/sdk/issues/55044
Change-Id: I68268dcaffbe857ccec20e211c34325c03643b37
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/439180
Commit-Queue: Liam Appelbe <liama@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
The usual deflaking logic is to simply rerun the test 5 times, but if the failure was a timeout this can be very expensive. Instead, only rerun timeout failures twice, and use the last successful run time to set a tighter timeout.
Repeat counts and timeouts are now per-test. The name, repeat count, and timeout make a `DeflakeInfo`. This object is constructed in compare_results.dart, and passed to test.py's `--tests` flag. The recipe was already passing the output of compare_results directly to that flag, so no change is needed to the recipies.
Backwards compatibility:
If you want the old behavior of compare_results.dart, use the `--name-only` flag.
test.py only uses this new logic if it detects that the `--tests` flag is JSON (if it starts with a `{`).
Change-Id: I4113b68c54bfb7fd9a5e8fc9dab7a265807f3e77
Bug: https://github.com/dart-lang/sdk/issues/55044
Fixes: https://github.com/dart-lang/sdk/issues/55044
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/435760
Commit-Queue: Liam Appelbe <liama@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
`testConfigurations` expects at least one configuration, so avoid
calling it when there are no configurations.
This fixes the unhandled exception with `test_runner.dart --help`:
```
No build targets found.
Unhandled exception:
RangeError (index): Invalid value: Valid value range is empty: 0
<asynchronous suspension>
```
With this commit, `--help` just prints "No build targets found" and
exits with 0.
This crash was previously fixed in
02a0396f74.
Change-Id: Id82c83f66871c4e811a7502f1e831a0592db2065
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/257580
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Commit-Queue: Ömer Ağacan <omersa@google.com>
The vendored package was just used in some tooling that should be easy
to update to any breaking changes in args. Language versioning ensures
we can handle changes in the language as well.
Bug: b/232495224
Change-Id: I15223187f19b6fe2229dd4ac05e51fd794cbfbd4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/244627
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Auto-Submit: Alexander Thomas <athom@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
For some reason, we launched Safari from a file, which since macOS 10.14
requires the user to confirm the operation in a modal dialog which
caused timeouts.
* Removed unused http test driver port option.
* Added some basic tests for test.py's browser interactions.
Bug: b/208186791
Change-Id: I070529148d37bf312f219c576abca3da972f73e6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/204202
Commit-Queue: Alexander Thomas <athom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
Currently, running `test.py --build` will attempt to build
the necessary files, before running the tests specified.
However, the tests are run even if this build fails, which
usually means old build artefacts are used for running the tests.
This can lead to some very confusing results if you do not yourself
catch that the build failed half-way through the run.
This change makes `test.py` exit with an error code and a message,
and does not proceed with the tests, if the build fails.
TEST=Manual testing locally.
Change-Id: I4e0f1d3f8d695219f2e4c0d7b474542af136aca9
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/241740
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Clement Skau <cskau@google.com>
* Move compare_results.dart to pkg/test_runner.
* Clean up analyzer issues in compare_results.dart.
* Add a test for compare_results.dart.
* Fix compare and extend results to detect expectation changes.
Change-Id: I451ddc2d0b01a7e5fd7c9d791b000c786c2f2c93
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/214805
Commit-Queue: Alexander Thomas <athom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
This flag causes test.py and test.dart to (re)build the necessary
build targets locally to run the selected configurations.
Change-Id: I526de702ac0eb8269c91d0be3673af096c94c1f1
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/170822
Reviewed-by: Alexander Thomas <athom@google.com>
- Better name than "optedIn".
- Actually hook up the command-line option to the configuration.
- Add tests, which would have caught the previous mistake.
- Don't allow comma-separated values for "--progress" and "--nnbd".
- Remove unused dead "--strong" option.
Change-Id: I57d7cb0d81af50d662dcf3f7f4c9ca1f2b102f2f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/116544
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Auto-Submit: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Mayank Patke <fishythefish@google.com>
- Run dartfmt --fix. This converts JavaDoc comments to "///", removes
"new" and extraneous "const", and a couple of other things.
- Fix SCREAMING_CAPS constants to lowerCamelCase.
- Use collection literals where possible.
- Use UI-as-code in a couple of places where it seemed obvious.
- Use "var" for more local variables.
- Use "const" instead of "final" when possible.
- Make members private when possible. Deleted a few that then became
obviously unused.
- ".length > 0" -> ".isNotEmpty".
There are no meaningful changes.
Change-Id: Ic6c5a74b2af9b3ebcbe881dbed69f65488bdef09
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/105880
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
This makes it an actual Pub package like most other code inside the SDK
repo. The main goal is to make it easier to write tests for the test
runner itself.
This change:
- Moves all of the code from tools/testing/dart/ over to
pkg/test_runner. Most of it ends up under test_runner/lib/src.
- Move tools/testing/dart/main.dart to
pkg/test_runner/bin/test_runner.dart.
- Move standalone_2/io/test_runner_test.dart to
pkg/test_runner/test/test_runner_test.dart. I don't think it currently
works, but it wasn't being run in its old location either.
- Add test_runner to the analysis-server bot. This ensures the
test_runner package is static error clean.
- Remove standalone_2/io/test_runner_analyze_test.dart which used to
attempt to do the above and is no longer needed.
- Update test.py to look for the test runner at its new location.
- Add test_runner to the repo .packages file and remove the weird
test_dart pseudo-package. (I think this fixes #35279.)
- Remove status file entries for the removed standalone_2 tests.
There are no code changes to the test runner itself aside from fixing
up import paths.
Change-Id: I3d05d50d222b291848fa5a30de2846e803bc81e6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/105821
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Jonas Termansen <sortie@google.com>
Konstantin Shcheglov