This adds a "--deflake" flag to test.dart. Deflaking will only run if
that flag is set.
Deflaking is time consuming and delays the information that users want
to see when most of the time they know that the result is not caused by
flakiness.
A typical session may look like this:
* Run test.dart with a broad test selector without deflaking.
* Re-run test.dart with a narrow test selector for suspected flakes with
"--deflake". This second step is optional.
Additionally, a new "--report-flakes" flag is added that can be
used to report test failures for tests known to be flaky.
Change-Id: I543d0b40c32065eb0a50338c55e7050b7887abce
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/102381
Reviewed-by: Jonas Termansen <sortie@google.com>
This change makes test.dart handle if the baseline commit hasn't been built
on the requested builder and instead attempts to search for whether there's
results for up to 25 preceding commits, which it then uses as an inexact
baseline.
Closes https://github.com/dart-lang/sdk/issues/36211
Change-Id: Ie44042c67c9460826736f4a2cf159d8d7b455f0b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/96944
Reviewed-by: William Hesse <whesse@google.com>
This option makes the available named configurations much more discoverable.
Additionally this change expands the available named configurations by
generalizing the operating system and processor architecture patterns
in the named configurations. This change should ensure that nobody is doing
any local testing that isn't covered by a named configuration.
Change-Id: I776105955a86e9f0403ce07a3cdf971e4213646f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/96320
Commit-Queue: Jonas Termansen <sortie@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
This change lets one download the results for one named configuration but do
the local testing with another named configuration. This change addresses
the issue of developers needing to do local testing where there is no
builder matching their local environment. This allows e.g. a dart2js
developer on Windows to use the results for the Linux variant of the same
named configuration, which should be identical or close enough for the
comparison to be meaningful.
Fixes https://github.com/dart-lang/sdk/issues/36151
Change-Id: I1a387a9767cdf5ba99964535478201c0142f15e1
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/96102
Reviewed-by: William Hesse <whesse@google.com>
This change makes named configurations central in test.dart. Previously
test.dart would attempt to emulate the steps taken by a builder. Instead it
now runs a single named configuration as a single step and invokes test.py
directly without any other arguments from the test steps. This better
matches the behavior of test.py and what the developers expect.
Either a named configuration or a builder must be specified. If both are
provided, then the results for that builder is downloaded and used to
compare with when running the named configuration. Otherwise if the only the
named configuration is provided, then it downloads results from all the
builders using the named configuration. Finally if only the builder is
provided, then the named configuration defaults to the one tested by the
builder. If the builder has multiple named configuration, the user is
asked to clarify which one should be tested.
This change adds support for branches to the test matrix. test.dart needs to
know which builders are on which branch, so it can download the relevant
results, and not look on builders that won't be building the right commit.
Fixes https://github.com/dart-lang/sdk/issues/35873
Change-Id: Ie7b75445b954250493528299a0b45eca4e0bb2e5
Reviewed-on: https://dart-review.googlesource.com/c/92780
Reviewed-by: William Hesse <whesse@google.com>
This change takes advantage of the new --clean-exit and --silent-failures
options to test.py, which means the test.py native progress bar can be used.
test.py exiting 0 if results were written out, no matter the results, lets
us properly handle if the program failed.
The output is now sectioned to make it more readable, and if no tests
failed, then the program says so.
This change also fixes logs which were broken because the logs for the steps
weren't merged, and the main test.py run wasn't even being passed the
--write-logs option.
This change also fixes directly starting dart scripts as an executable
rather than using the platform resolved executable.
TBR=whesse@google.com
Bug: https://github.com/dart-lang/sdk/issues/35359
Change-Id: I7bed009475df3ffea01c9e805513d0e04e77427c
Reviewed-on: https://dart-review.googlesource.com/c/86568
Reviewed-by: Jonas Termansen <sortie@google.com>
test.dart locates where the current branch branched off master and compares
the local testing results with the appropriate mainline builder results,
letting you know how the current change compares without the need for status
files.
Bug: https://github.com/dart-lang/sdk/issues/35086
Change-Id: Ib79479b867c5ac131302fea1bdf7effd0422a83a
Reviewed-on: https://dart-review.googlesource.com/c/83281
Reviewed-by: Alexander Thomas <athom@google.com>
This is mainly so that all of the code relating to test.dart is in one
directory tree so things like "Find All Usages" work a little better.
It felt weird to me to have a .dart file two directories up importing a
bunch of stuff within "testing/dart/".
Also cleaned up the affected code since it could use a little love. I'm
working on getting test.dart running DDC tests, but from poking around,
it seems like it could use some housekeeping as well.
R=vsm@google.com, whesse@google.com
Review-Url: https://codereview.chromium.org/2848103002 .
Steps to make this work:
a) Make sure you have an android sdk checkout:
=> See https://github.com/dart-lang/sdk/wiki/Building-Dart-SDK-for-Android
b) Building using android-toolchain crosscompiler:
$ ./tools/build.py -mrelease -aarm --os=android dart_precompiled_runtime
=> Notice --os=android
c) Testing using attached android phones:
$ export PATH=$PATH:$PWD/third_party/android_tools/sdk/platform-tools
$ ./tools/test.py -mrelease -aarm --system=android -cprecompiler -rdart_precompiled --use-blobs
=> Notice --system=android
Failing tests on android can be marked in status files via:
[ $compiler == precompiler && $runtime == dart_precompiled && $system == android ]
R=whesse@google.com
Review URL: https://codereview.chromium.org/1922163002 .
This will be used on the bots to be able to archive dumps from unexpectec crashes.
The tools/archive_crash.py tool will be used to actually archive the dumps to gcs and delete the local copies. The reason for the split is to enable developers to actually utilize this locally as well (run tools/test.py in a loop but don't archive the expected crashes). Additionally, this puts the bot specific code in the small python script.
R=whesse@google.com
Review URL: https://codereview.chromium.org//145273024
git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@32111 260f80e4-7a28-3924-810f-c04153c831b5
This CL adds
- support for fetching dependencies via 'pub get' of samples/packages
- support for building applications/samples via 'pub build' (or a custom
build.dart file)
- pkg/pkgbuild.status: Status files for all package checkouts/builds
- the '--use-public-packages' option to test.dart
- the CleanDirectoryCopyCommand, PubCommand, ModifyPubspecYamlCommand commands
The test suite can be executed with:
$ ./tools/test.py -mrelease --use-sdk pkgbuild
$ ./tools/test.py -mrelease --use-sdk --use-public-packages pkgbuild
R=ricow@google.com
Review URL: https://codereview.chromium.org//122443003
git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@31535 260f80e4-7a28-3924-810f-c04153c831b5
This CL will:
- fix a bug in the legacy AnalyzerCommandOutputImpl class (in certain cases
we did not report that the analyzer was missing a warning).
- make the outcome of running the analyzer on a test be one of:
* CompileTimeError
* MissingCompileTimeError
* StaticWarning
* MissingStaticWarning
- Make status file updates using the new Expectation markers
- Remove support for '@static-clean':
Previously running the analyer on a test could either result in
'Pass' or in 'Fail'. The '@static-clean' annotation has been used
as a (poor) mechanism to distinguish between errors and warnings.
Having the 4 markers mentioned above means we can remove '@static-clean'
- Remove 100 lines of rotted legacy code in our testing scripts.
- Make a status file entry for every test that has analyzer errors or warnings
R=ricow@google.com
Review URL: https://codereview.chromium.org//46163002
git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@29350 260f80e4-7a28-3924-810f-c04153c831b5
Alexander Thomas