Commit Graph

53 Commits

Author SHA1 Message Date
William Hesse 30596c2f03 Output a results.json file containing the expected and actual results of each test
Also outputs a run.json file with information about this run of test.py,
such as the commit hash and the named configuration.

Bug: https://github.com/dart-lang/sdk/issues/34348
Change-Id: Idb4d2a6b892d13314ed81e6163933d23c1b8b609
Reviewed-on: https://dart-review.googlesource.com/72682
Commit-Queue: William Hesse <whesse@google.com>
Reviewed-by: Jonas Termansen <sortie@google.com>
2018-09-07 15:34:46 +00:00
William Hesse 18047b2757 Refactor test.dart by changing class Configuration to TestConfiguration
Change-Id: I03624c2cefc6bf5c293ecf016c52de6740e5893f
Reviewed-on: https://dart-review.googlesource.com/67462
Reviewed-by: Alexander Thomas <athom@google.com>
2018-07-31 11:11:16 +00:00
Zach Anderson d0510501d2 [build] Add -a kbc for interpreter
This CL adds support to build.py and test.py for building/testing
a VM with the interpreter on x64, e.g.:

$ ./tools/gn.py -m release -a x64 --bytecode
$ ./tools/build.py -m release -a x64 --bytecode runtime
$ ./tools/test.py -m release -a x64 -r vm -c dartkb language_2

Change-Id: I956d23790636609d4a2e71129481fcbd7afef9a0
Reviewed-on: https://dart-review.googlesource.com/65206
Commit-Queue: Zach Anderson <zra@google.com>
Reviewed-by: Régis Crelier <regis@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
2018-07-17 21:51:41 +00:00
Devon Carew 5b5e36f0be Remove use of deprecated methods and constants.
Change-Id: Ic6867233dd7d432eaa0973dbfca13d53f15cc80a
Reviewed-on: https://dart-review.googlesource.com/63663
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Devon Carew <devoncarew@google.com>
2018-07-04 17:18:06 +00:00
Alexander Thomas 583a81ade3 [infra] Make test.py's --strong option a no-op to prepare for removing it
Remove --preview-dart-2 option.


Change-Id: I4ca07e00391ee9a9f7f454d0b56e82898c38250c
Reviewed-on: https://dart-review.googlesource.com/62601
Commit-Queue: Alexander Thomas <athom@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
2018-06-28 09:39:03 +00:00
Lasse R.H. Nielsen 973a1a0219 Remove uses of upper-case constants in remaining SDK code.
This includes Fasta, tools and observatory, so the checked-in SDK must
have the lower-case constants.

Change-Id: I8380ad041ad058f7d02ae19caccfecd434d13d75
Reviewed-on: https://dart-review.googlesource.com/50201
Commit-Queue: Lasse R.H. Nielsen <lrn@google.com>
Reviewed-by: Leaf Petersen <leafp@google.com>
2018-04-16 16:11:57 +00:00
Jens Johansen 501cca4741 [test.py] Fix race condition when copying files
When testing multitests needed files are copied, but some files are
copied multiple times, creating race conditions that might cause tests
to fail because incomplete files are read.

For instance running
"python tools/test.py -m release -c dartk --strong language_2"
is fine (i.e. everything passes), but running
"python tools/test.py -m release -c dartk --strong language_2/generic_constructor_mixin3_test"
can get 0, 1 or 2 errors depending on the computers mood...

This fixes the problem by introducing a map from copied-to-path to Future,
so the same Future is returned when asked to copy a file subsequent times,
thus avoiding the race condition.

Fixes #31368.

Bug:
Change-Id: Id7d4a12424085c75f3543bd06a617121ad5eff35
Reviewed-on: https://dart-review.googlesource.com/32282
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: William Hesse <whesse@google.com>
2018-01-05 07:50:54 +00:00
Alexander Thomas a8e0c63e55 [Infra] Delete generated compilations for dartk/dartkp compilers as well
This should prevent the bots from running out of disk space.

Change-Id: I3ca7794f8006f616473cf02c62f000a4c44d5b36
Reviewed-on: https://dart-review.googlesource.com/31120
Commit-Queue: Alexander Thomas <athom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
2017-12-21 16:18:21 +00:00
Bob Nystrom 746336ba10 Improve test output:
- Clean up browser test failure output. Instead of barfing a giant blob
  of JSON, show just the runtime error and its stack trace. In verbose
  and on the buildbots, it still shows all the events, but even there
  they are formatted nicely.

- Don't redundantly show the failure summary when most of the output is
  already failure text.

- Color the failure summary end message.

Change-Id: I49724b204be55a297fc5e723b80b32e0e00748f0
Reviewed-on: https://dart-review.googlesource.com/15767
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
2017-10-23 19:03:48 +00:00
Bob Nystrom c341575a53 Colorize section headers and slightly clean up test output.
Of course, if you're not using the color formatter, it stays uncolored.
But this gets rid of some unnecessary empty lines and section headers
that have no content.

Change-Id: I5b83345c583b79468af84f4e67a12ee757a3105f
Reviewed-on: https://dart-review.googlesource.com/14720
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
2017-10-19 21:45:57 +00:00
Bob Nystrom ba2e4aff24 Some minor clean-up in test.dart.
- Fix fuzzy arrow errors.
- Default the Dart repo URI so it doesn't always need to be explicitly
  set.
- Run dartfmt on everything.
- Move some members out of the garbage bin "TestUtils" class:
  - Move the stuff around the Dart repo directory to a new Repository
    class.
  - Move absolute() into Path where it belongs.
  - Move workingDirectory in Path since it is a Path.
  - Delete the random number stuff since it was apparently unused.

Change-Id: I3dab3a4f1713b7a749e64b6776149d05a0ce1b69
Reviewed-on: https://dart-review.googlesource.com/14502
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
2017-10-19 19:05:58 +00:00
Morten Krogh-Jespersen 70e3e8aac1 Added json result of test output to output debug directory.
R=whesse@google.com

Review-Url: https://codereview.chromium.org/3005013002 .
2017-09-04 10:13:27 +02:00
Bob Nystrom 0b2c9b048d Dynamically load packages for dartdevc tests in test.dart.
This involves a few pieces:

- Add support to DDC for specifying the module name associated with a
  given summary. This lets test.dart invoke DDC using summaries in the
  build directory outside of the directory containing the test itself.

- Add support to the build scripts for building the packages. This adds
  a new GN target that builds everything needed to run test.dart with
  dartdevc. In particular, it invokes build_pkgs.dart to compile the
  relevant packages to JS+summary so that the tests can use them.

  This requires some changes to build_pkgs.dart so it can output to a
  given directory.

- In test.dart, when tests are compiled with dartdevc, pass in the
  summaries for the packages so they don't get compiled in. Then, when
  the test is run, configure require.js with the right paths to their
  JS files so they can be loaded.

  I also removed a bunch of unneeded buildDir parameters being passed
  around the various CompilerConfiguration class methods now that they
  have direct access to the configuration.

Fix #29923.

R=vsm@google.com, whesse@google.com, zra@google.com

Review-Url: https://codereview.chromium.org/2955513002 .
2017-06-29 13:45:57 -07:00
Bob Nystrom dfb4f52b43 Simplify enqueueBrowserTest().
It's still big and monolithic, but it's not as big as it was before.

- Remove old unused Polymer stuff.
- Move some of the logic out into a helper function.
- Other small tweaks.

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2949823003 .
2017-06-23 15:02:36 -07:00
Bob Nystrom 19cb0cafb7 Simplify test_suite.dart.
- Get rid of baseCommand, which is only ever an empty list.
- Move some utility classes out of the giant test_suite.dart.

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2915843003 .
2017-06-09 16:46:49 -07:00
Bob Nystrom 7e2d9ac85d Revert "Revert "Replace the configuration map with a typed object.""
This reverts commit 8bada4873a.

R=nweiz@google.com

Review-Url: https://codereview.chromium.org/2919573003 .
2017-05-31 15:39:23 -07:00
Bob Nystrom 8bada4873a Revert "Replace the configuration map with a typed object."
This reverts commit 06f75fe5cd.

Review-Url: https://codereview.chromium.org/2914893003 .
2017-05-31 15:08:21 -07:00
Bob Nystrom 06f75fe5cd Replace the configuration map with a typed object.
This is a pretty massive change but my hope is it will make
test.dart easier to maintain going forward. Instead of
passing around a stringly-typed Map<String, dynamic> all
throughout the program, we parse the options and then create
a Configuration object that has typed getters for all of the
various bits of configuration data.

This is a little tedious because it means declaring a new
option requires also declaring a corresponding field in the
Configuration class and passing it through the constructor.
I think it's worth it.

Also, enum-like configuration properties like architecture
and runtime now have their own classes as well. Moved a
bunch of stuff from TestUtil into those classes now that
there is an object to hang those methods off of.

In the process, I found a few typos in string literals where
the code wasn't correctly looking up a configuration
property.

Added more sanity checking and validation to status file
parsing. You will get an error at parse time if you try to
refer to a variable that isn't in the whitelist of known
variables. Also, you'll get an error if you try to compare
a variable to a value that it isn't expected to have.

Many other small-scale cleanups.

Aside from the status file validation, this should behave
mostly the same as current test.dart except that tests
may be enqueued in a slightly different order. The
rewritten code for expanding configurations iterates
through the architecture, runtime, etc. options in a
slightly different order.

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2901923003 .
2017-05-31 14:47:31 -07:00
Bob Nystrom 7ce978bf91 Tighten types in test.dart even more.
This removes all implicit casts and many implicit uses of dynamic. It
adds a gratuitous number of explicit "as" casts and arguably makes the
code worse.

This is an interim step towards replacing the big configuration map
with an actual typed object. These "as" casts should help catch places
where the configuration object is being used and where the code will
need to be changed to use a new object.

R=sigmund@google.com

Review-Url: https://codereview.chromium.org/2903703002 .
2017-05-30 14:02:24 -07:00
Bob Nystrom e0180925f6 Tighten up a bunch of types in test.dart.
This doesn't get rid of all implicit dynamic and casts, but it covers
many of them.

Also did some minor modernization when I noticed it:

- Using "var" for local variables where inference does the right thing.
- camelCase for variable names.
- More collection literals.

There are (or should be!) zero behavioral changes.

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2863253002 .
2017-05-15 15:26:40 -07:00
Jacob Richman 119b2d58ce Format all files under tools and utils directory.
BUG=
R=terry@google.com

Review-Url: https://codereview.chromium.org/2827793002 .
2017-04-20 09:08:31 -07:00
Asger Feldthaus 902d9ee7e8 Do some housekeeping in tools/testing/dart
- Fix dartanalyzer warnings about unused variables and missing returns
- Remove trailing whitespace
- Add a dummy pubspec.yaml so editors recognize it as a Dart project

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2633213003 .
2017-01-17 16:02:45 +01:00
William Hesse c63b10b611 Format the test scripts in /tools/testing/dart using dartfmt 1.16.0-dev.3.0.
R=ahe@google.com

Review URL: https://codereview.chromium.org/1859973002 .
2016-04-06 14:49:18 +02:00
ricow@google.com d6e6b50a64 Allow uri in deepJsonCompare
R=sgjesse@google.com, whesse@google.com
BUG=

Review URL: https://codereview.chromium.org//957253002

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@44043 260f80e4-7a28-3924-810f-c04153c831b5
2015-02-26 09:29:30 +00:00
kevmoo@google.com 47de553025 tools/testing: move code into individual libraries
R=ricow@google.com

Review URL: https://codereview.chromium.org//748773004

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@42108 260f80e4-7a28-3924-810f-c04153c831b5
2014-12-04 15:32:57 +00:00
whesse@google.com 6650aeb3d6 Clean up test_runner Command subclass hash/equality.
BUG=
R=ricow@google.com

Review URL: https://codereview.chromium.org//645533002

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@41037 260f80e4-7a28-3924-810f-c04153c831b5
2014-10-10 12:05:52 +00:00
ricow@google.com 6be37386e0 Add mobile safari simulator support in the testing scripts
R=whesse@google.com

Review URL: https://codereview.chromium.org//262763005

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@35626 260f80e4-7a28-3924-810f-c04153c831b5
2014-05-01 11:32:08 +00:00
ricow@google.com ffb5397a8b Add default location for ie11
R=whesse@google.com

Review URL: https://codereview.chromium.org//208653011

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@34350 260f80e4-7a28-3924-810f-c04153c831b5
2014-03-25 09:14:32 +00:00
kustermann@google.com 338fabdb6c Cancel stdout/stderr subscriptions once subprocesses are dead, add more debugging information in case the debug timer expires
R=whesse@google.com

Review URL: https://codereview.chromium.org//128703002

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@31649 260f80e4-7a28-3924-810f-c04153c831b5
2014-01-09 09:04:46 +00:00
kustermann@google.com dc24212755 Added PkgBuildTestSuite (aka 'pkgbuild')
This CL adds
 - support for fetching dependencies via 'pub get' of samples/packages
 - support for building applications/samples via 'pub build' (or a custom
   build.dart file)
 - pkg/pkgbuild.status: Status files for all package checkouts/builds
 - the '--use-public-packages' option to test.dart
 - the CleanDirectoryCopyCommand, PubCommand, ModifyPubspecYamlCommand commands

The test suite can be executed with:
 $ ./tools/test.py -mrelease --use-sdk pkgbuild
 $ ./tools/test.py -mrelease --use-sdk --use-public-packages pkgbuild

R=ricow@google.com

Review URL: https://codereview.chromium.org//122443003

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@31535 260f80e4-7a28-3924-810f-c04153c831b5
2014-01-07 09:36:27 +00:00
kustermann@google.com 18a8f25608 Command class hieracy refactoring, environmentOverride extraction
R=ricow@google.com, whesse@google.com

Review URL: https://codereview.chromium.org//118783003

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@31269 260f80e4-7a28-3924-810f-c04153c831b5
2013-12-19 13:01:46 +00:00
kustermann@google.com ca307461d0 test.py: Removed JUnitTestSuite, since it's no longer used
R=whesse@google.com

Review URL: https://codereview.chromium.org//113713006

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@31214 260f80e4-7a28-3924-810f-c04153c831b5
2013-12-18 08:48:00 +00:00
kustermann@google.com bb54b776ba Checked-in binary update to v1.0.0.3
R=ricow@google.com

Review URL: https://codereview.chromium.org//91473002

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@30717 260f80e4-7a28-3924-810f-c04153c831b5
2013-11-27 14:17:44 +00:00
kustermann@google.com e3d8608ca2 test.py: Sending JSON between test_controller.js <-> browser_controller
Currently test.py will look for FAIL/PASS messages in the DOM to determine if a
test has failed or not. test_controller.js will add these PASS/FAIL messages to
the DOM (in addition to other messages).

This approach has several issues:
- The test itself could create nodes in the DOM containing FAIL/PASS. This means
  that the outcome of a test cannot be reliably determined.
- The test ifself can remove nodes from the DOM (e.g. setting
  document.body.innerHTML). This could result in a situation where the
  stacktraces/... test_controller.js prints to the DOM are lost.
- It is unclear which parts of the DOM we see in test failure messages
  (reported by test.py) come from the test itself and which parts come from
  test_controller.js.

This CL tries to fix this by recording all events that happend with global
variables in test_controller.js and send back a JSON string with information
about
  - debug messages
  - posted messages
  - print() calls of the test
  - DOM
  - errors (from window.onerror, script.onerror ...)

Furthermore this CL fixes unittest to report
unittest-suite-success/unittest-suite-fail instead of unittest-suite-done (which
is broken).

R=ricow@google.com

Review URL: https://codereview.chromium.org//36913002

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@30407 260f80e4-7a28-3924-810f-c04153c831b5
2013-11-19 12:54:41 +00:00
whesse@google.com 0f2f482388 Add browser path options to testing scripts.
BUG=
R=kustermann@google.com

Review URL: https://codereview.chromium.org//60503002

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@29981 260f80e4-7a28-3924-810f-c04153c831b5
2013-11-06 16:24:33 +00:00
floitsch@google.com 7006de6f23 Remove deprecated parts of dart:async.
Attempt to fix most uses of the deprecated features.

R=floitsch@google.com, nweiz@google.com, rnystrom@google.com

Review URL: https://codereview.chromium.org//48483002

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@29628 260f80e4-7a28-3924-810f-c04153c831b5
2013-10-31 02:24:29 +00:00
lrn@google.com 41ff075e31 Remove deprecated dart:utf library.
BUG= http://dartbug.com/12843
R=fschneider@google.com, sgjesse@google.com

Review URL: https://codereview.chromium.org//48133002

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@29537 260f80e4-7a28-3924-810f-c04153c831b5
2013-10-30 12:25:02 +00:00
kustermann@google.com c9ba1fcbe4 test.py: Propagate the global --dartium option to the browser controller
R=kasperl@google.com

Review URL: https://codereview.chromium.org//25876002

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@28222 260f80e4-7a28-3924-810f-c04153c831b5
2013-10-03 14:55:09 +00:00
kustermann@google.com 45c42e08f2 Update checked-in binary to 0.6.21.3_r26639 (M6)
R=ricow@google.com

Review URL: https://codereview.chromium.org//23568002

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@26704 260f80e4-7a28-3924-810f-c04153c831b5
2013-08-27 11:16:37 +00:00
kustermann@google.com 92303dffda test.py: First step towards support of caching dart2js compilations across runtimes
This CL changes the way we run tests completely: When enqueueing TestCases we
build up a dependency graph of Commands. This has the advantage that if more
TestCases share common commands, we only execute it once.

R=ricow@google.com

Review URL: https://codereview.chromium.org//21001003

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@25760 260f80e4-7a28-3924-810f-c04153c831b5
2013-08-05 07:54:17 +00:00
kustermann@google.com 8984694c97 Print out experimental test.py reproduction command
R=ricow@google.com

Review URL: https://codereview.chromium.org//17361003

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@24231 260f80e4-7a28-3924-810f-c04153c831b5
2013-06-20 14:19:11 +00:00
kustermann@google.com 4efd9609a5 Moving timing info to utils.dart:DebugLogger
R=ricow@google.com

Review URL: https://codereview.chromium.org//16958003

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@23967 260f80e4-7a28-3924-810f-c04153c831b5
2013-06-13 11:20:21 +00:00
floitsch@google.com d8930c4ae6 Revert changes in tools/testing while the Dart executable can't handle it.
Review URL: https://codereview.chromium.org//14113018

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@21511 260f80e4-7a28-3924-810f-c04153c831b5
2013-04-15 20:26:36 +00:00
floitsch@google.com 5431a5af54 Remove AsyncError with Expando.
Review URL: https://codereview.chromium.org//14251006

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@21498 260f80e4-7a28-3924-810f-c04153c831b5
2013-04-15 18:58:32 +00:00
sgjesse@google.com d6955d176c Reapply "Update the test runner to use the new dart:io API"
This reverts r20031.

R=ager@google.com

BUG=

Review URL: https://codereview.chromium.org//12751005

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@20070 260f80e4-7a28-3924-810f-c04153c831b5
2013-03-15 11:13:39 +00:00
sgjesse@google.com f5270bf202 Revert "Update the test runner to use the new dart:io API" again
This reverts r20027 and r20029

TBR=ager@google.com

BUG=

Review URL: https://codereview.chromium.org//12475010

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@20031 260f80e4-7a28-3924-810f-c04153c831b5
2013-03-14 15:38:18 +00:00
sgjesse@google.com fdaca2fe83 Revert "Update the test runner to use the new dart:io API" again
This reapplies r20004 and r20008 and fixes a HTTP server issue.

TBR=ager@google.com

BUG=

Review URL: https://codereview.chromium.org//12431020

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@20027 260f80e4-7a28-3924-810f-c04153c831b5
2013-03-14 15:24:36 +00:00
sgjesse@google.com 3e2ffb4394 Revert "Update the test runner to use the new dart:io API" again
This reverts r20004 and r20008.

TBR=ager@google.com

BUG=

Review URL: https://codereview.chromium.org//12559013

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@20009 260f80e4-7a28-3924-810f-c04153c831b5
2013-03-14 12:13:37 +00:00
sgjesse@google.com db7821cdf3 Reapply "Update the test runner to use the new dart:io API"
The Linux binary will work on the bots. Fixed a number of issues.

The new binaries are from r19850.

R=ager@google.com

BUG=

Review URL: https://codereview.chromium.org//12608007

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@20004 260f80e4-7a28-3924-810f-c04153c831b5
2013-03-14 11:39:20 +00:00
sgjesse@google.com 78403ce605 Revert "Update the test runner to use the new dart:io API"
This reverts commit r19987.

The included binaries did not work on the bots due to a glibc version issue.

TBR=ager@google.com

BUG=

Review URL: https://codereview.chromium.org//12864003

git-svn-id: https://dart.googlecode.com/svn/branches/bleeding_edge/dart@19988 260f80e4-7a28-3924-810f-c04153c831b5
2013-03-14 08:08:56 +00:00