Commit Graph

1689 Commits

Author SHA1 Message Date
Bob Nystrom 19cb0cafb7 Simplify test_suite.dart.
- Get rid of baseCommand, which is only ever an empty list.
- Move some utility classes out of the giant test_suite.dart.

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2915843003 .
2017-06-09 16:46:49 -07:00
Bob Nystrom 8bc4e89219 Remove unused record and replay functionality.
R=whesse@google.com

Review-Url: https://codereview.chromium.org/2920633002 .
2017-06-08 09:18:03 -07:00
Ryan Macnak b69b3e58f0 Update test.py's --arch=all to match that of build.py.
R=zra@google.com

Review-Url: https://codereview.chromium.org/2918403002 .
2017-06-05 13:07:30 -07:00
Alexander Aprelev 6573c74301 Reenable vm tests with -cdartk DFE parser.
This was accidentally disabled in 7e2d9ac85d.

BUG=#28264
R=asiva@google.com

Review-Url: https://codereview.chromium.org/2920733004 .
2017-06-02 09:34:11 -07:00
Lasse R.H. Nielsen 5720e3556c Add negation to single-identifier tests in status files.
Optimize some regegps too.

Fixes #29756

BUG= http://dartbug.com/29756
R=rnystrom@google.com

Review-Url: https://codereview.chromium.org/2913963002 .
2017-06-02 10:38:14 +02:00
Bob Nystrom db06324975 Remove temp code to ignore "--failure-summary".
Now that the BuildBots aren't passing it, we don't need to accept it.

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2916123002 .
2017-06-01 15:45:55 -07:00
William Hesse e161ad12ac Allow test.py to run dartdoc tests. Run dartdoc tests on pub bots.
BUG=
R=jcollins@google.com

Review-Url: https://codereview.chromium.org/2914233002 .
2017-06-01 18:53:04 +02:00
Bob Nystrom a9d4afc077 Handle "ff" as a runtime name in status files.
R=efortuna@google.com

Review-Url: https://codereview.chromium.org/2917843002 .
2017-05-31 17:49:22 -07:00
Bob Nystrom 7e2d9ac85d Revert "Revert "Replace the configuration map with a typed object.""
This reverts commit 8bada4873a.

R=nweiz@google.com

Review-Url: https://codereview.chromium.org/2919573003 .
2017-05-31 15:39:23 -07:00
Bob Nystrom 8bada4873a Revert "Replace the configuration map with a typed object."
This reverts commit 06f75fe5cd.

Review-Url: https://codereview.chromium.org/2914893003 .
2017-05-31 15:08:21 -07:00
Bob Nystrom 06f75fe5cd Replace the configuration map with a typed object.
This is a pretty massive change but my hope is it will make
test.dart easier to maintain going forward. Instead of
passing around a stringly-typed Map<String, dynamic> all
throughout the program, we parse the options and then create
a Configuration object that has typed getters for all of the
various bits of configuration data.

This is a little tedious because it means declaring a new
option requires also declaring a corresponding field in the
Configuration class and passing it through the constructor.
I think it's worth it.

Also, enum-like configuration properties like architecture
and runtime now have their own classes as well. Moved a
bunch of stuff from TestUtil into those classes now that
there is an object to hang those methods off of.

In the process, I found a few typos in string literals where
the code wasn't correctly looking up a configuration
property.

Added more sanity checking and validation to status file
parsing. You will get an error at parse time if you try to
refer to a variable that isn't in the whitelist of known
variables. Also, you'll get an error if you try to compare
a variable to a value that it isn't expected to have.

Many other small-scale cleanups.

Aside from the status file validation, this should behave
mostly the same as current test.dart except that tests
may be enqueued in a slightly different order. The
rewritten code for expanding configurations iterates
through the architecture, runtime, etc. options in a
slightly different order.

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2901923003 .
2017-05-31 14:47:31 -07:00
Bob Nystrom 7ce978bf91 Tighten types in test.dart even more.
This removes all implicit casts and many implicit uses of dynamic. It
adds a gratuitous number of explicit "as" casts and arguably makes the
code worse.

This is an interim step towards replacing the big configuration map
with an actual typed object. These "as" casts should help catch places
where the configuration object is being used and where the code will
need to be changed to use a new object.

R=sigmund@google.com

Review-Url: https://codereview.chromium.org/2903703002 .
2017-05-30 14:02:24 -07:00
Bob Nystrom c296f75d10 Revert "Revert "Revert "Revert "Refactor test option parsing code.""""
This reverts commit 02547656cc.

R=efortuna@google.com

Review-Url: https://codereview.chromium.org/2904313002 .
2017-05-26 15:04:55 -07:00
Bob Nystrom 02547656cc Revert "Revert "Revert "Refactor test option parsing code."""
This reverts commit 79af418430.

Review-Url: https://codereview.chromium.org/2912563002 .
2017-05-26 14:46:04 -07:00
Bob Nystrom 79af418430 Revert "Revert "Refactor test option parsing code.""
This reverts commit d03835783c.

R=efortuna@google.com

Review-Url: https://codereview.chromium.org/2908833002 .
2017-05-26 14:18:13 -07:00
Bob Nystrom d03835783c Revert "Refactor test option parsing code."
This reverts commit 9c2dff8ebc.

R=nweiz@google.com

Review-Url: https://codereview.chromium.org/2909723002 .
2017-05-26 13:51:26 -07:00
Bob Nystrom 9c2dff8ebc Refactor test option parsing code.
This doesn’t touch the resulting configuration map, which is what this
is leading up to, but it cleans up the specification and parsing of the
options and removes a lot of redundancy.

Behavior should be the same as it was before, except that all options
now allow both their underscore-separated and hyphen-separated names.
So this is now valid:

test.py --hot_reload --append-logs

Where before, you use to have to do --hot-reload (OK) and
--append_logs (bad, since almost all other options expect hyphens).

R=sigmund@google.com

Review-Url: https://codereview.chromium.org/2902023002 .
2017-05-26 13:36:44 -07:00
Alexander Aprelev 16c0db1022 Revert "Revert "With this change 'tools/test.py -cdartk vm' will run vm tests with Dart Frontend parser.""
This reverts commit e0cdea811a.

Update status of vm tests, that don't work with DFE yet.

BUG=https://github.com/dart-lang/sdk/issues/28264
R=asiva@google.com

Review-Url: https://codereview.chromium.org/2895153005 .
2017-05-24 20:16:48 -07:00
Alexander Aprelev e0cdea811a Revert "With this change 'tools/test.py -cdartk vm' will run vm tests with Dart Frontend parser."
This reverts commit f62a2a9562.

Commit above [expectedly] broke runtime/vm tests since it uses DFE
instead of vm parser, but I didn't realize we actually test -cdartk
configuration. This has to be resubmitted with status file update to
mark 235 vm tests failing with DFE.

TBR=siva
BUG=

Review-Url: https://codereview.chromium.org/2901733002 .
2017-05-22 19:42:53 -07:00
Alexander Aprelev f62a2a9562 With this change 'tools/test.py -cdartk vm' will run vm tests with Dart Frontend parser.
BUG=https://github.com/dart-lang/sdk/issues/28264
R=asiva@google.com

Review-Url: https://codereview.chromium.org/2881833003 .
2017-05-22 17:24:38 -07:00
Martin Kustermann 22c7536d36 Reduce temporarily flakiness rate of dart2js-drt by re-running until they have been fixed
We should either remove dart2js-drt or use a stable version of
content_shell from upstream chromium (instead of our dartium-based one)
for dart2js-drt.

Issue #29655

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2891343002 .
2017-05-19 13:13:40 +02:00
Bob Nystrom 92267afc66 Refactor and clean up the status file parsing code. - Make the parser less error tolerant. The expression parser used to ignore any unrecognized tokens, which means a status like "RuntimeError CompileError" (not the missing comma) was parsed as simply "RuntimeError", which seems bad. Now it reports an error. Fixed a couple of status files that thought they were setting statuses that they weren't (!).
- Separate out parsing a status file from applying the environment to
  determine which sections are active. This makes it possible to, for
  example, generate expectation sets for multiple environments without
  having to reparse each time.

- Simplify expression parsing. Remove set expressions since they weren't
  used for anything useful. A test's expectations are a simple
  comma-separated list and don't need anything beyond that. Merge
  Scanner and Tokenizer since the latter was a glorified function.

- Make more names private so that it's clearer what's used outside of
  various libraries.

- Generally modernize the style.

- Add *lots* of documentation.

Again, there should be no behavioral changes. I ran:

  ./tools/test.py -m release,debug -c none,dart2js,dart2analyzer -r none,vm,d8 corelib

Before and after the change and verified that the output was the same
(aside from timing).

R=sigmund@google.com

Review-Url: https://codereview.chromium.org/2891753003 .
2017-05-18 12:42:52 -07:00
Bob Nystrom c269ef53a5 Revert "Re-apply status file parser changes from 0b7728da1bef08c1c1e092005d9fd8c8bff5fa6c."
This reverts commit a881aed354.

TBR.

R=paulberry@google.com

Review-Url: https://codereview.chromium.org/2888213002 .
2017-05-17 13:45:40 -07:00
Bob Nystrom a881aed354 Re-apply status file parser changes from 0b7728da1b.
R=whesse@google.com

Review-Url: https://codereview.chromium.org/2880203004 .
2017-05-17 13:04:35 -07:00
Bob Nystrom 01e5b1b40c Revert "Refactor and clean up the status file parsing code."
This reverts commit 0b7728da1b.

R=nbosch@google.com

Review-Url: https://codereview.chromium.org/2885623002 .
2017-05-15 15:58:05 -07:00
Bob Nystrom 0b7728da1b Refactor and clean up the status file parsing code.
- Make the parser less error tolerant. The expression parser used to
  ignore any unrecognized tokens, which means a status like
  "RuntimeError CompileError" (not the missing comma) was parsed as
  simply "RuntimeError", which seems bad. Now it reports an error.
  Fixed a couple of status files that thought they were setting statuses
  that they weren't (!).

- Separate out parsing a status file from applying the environment to
  determine which sections are active. This makes it possible to, for
  example, generate expectation sets for multiple environments without
  having to reparse each time.

- Simplify expression parsing. Remove set expressions since they weren't
  used for anything useful. A test's expectations are a simple
  comma-separated list and don't need anything beyond that. Merge
  Scanner and Tokenizer since the latter was a glorified function.

- Make more names private so that it's clearer what's used outside of
  various libraries.

- Generally modernize the style.

- Add *lots* of documentation.

Again, there should be no behavioral changes. I ran:

  ./tools/test.py -m release,debug -c none,dart2js,dart2analyzer -r none,vm,d8 corelib

Before and after the change and verified that the output was the same
(aside from timing).

BUG=
R=whesse@google.com

Review-Url: https://codereview.chromium.org/2875203005 .
2017-05-15 15:42:15 -07:00
Bob Nystrom e0180925f6 Tighten up a bunch of types in test.dart.
This doesn't get rid of all implicit dynamic and casts, but it covers
many of them.

Also did some minor modernization when I noticed it:

- Using "var" for local variables where inference does the right thing.
- camelCase for variable names.
- More collection literals.

There are (or should be!) zero behavioral changes.

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2863253002 .
2017-05-15 15:26:40 -07:00
William Hesse 6f3d45f5a3 Check for infrastructure failure before reporting timeouts and crashes.
BUG=http://dartbug.com/28955

Review-Url: https://codereview.chromium.org/2881073002 .
2017-05-15 18:27:46 +02:00
William Hesse c235977ddb Increase startup time allowed for browsers
On a cold, overloaded system, IE sometimes takes more than 60 seconds to start.

BUG=https://github.com/dart-lang/sdk/issues/28955
R=kustermann@google.com

Review-Url: https://codereview.chromium.org/2878423002 .
2017-05-15 13:36:44 +02:00
William Hesse bff560292c Fix bug in test_runner.dart
This error was introduced in https://codereview.chromium.org/2875683002/
committed as d6ca1a5def.

BUG=
TBR=karlklose@google.com

Review-Url: https://codereview.chromium.org/2874193002 .
2017-05-11 13:39:46 +02:00
William Hesse d6ca1a5def Report IE11 timeouts in debug log, not as failing tests.
BUG=https://github.com/dart-lang/sdk/issues/28955
R=sigmund@google.com

Review-Url: https://codereview.chromium.org/2875683002 .
2017-05-11 11:18:42 +02:00
Terry Lucas f27144d7b5 Roll 50: Updated for push to origin/master.
TBR=jacobr@google.com

Review-Url: https://codereview.chromium.org/2875773003 .
2017-05-10 18:16:51 -07:00
Bob Nystrom 0592404cc3 Re-add missing space in test failed message.
R=sigmund@google.com

Review-Url: https://codereview.chromium.org/2866273002 .
2017-05-09 11:16:42 -07:00
Bob Nystrom 0635d4d559 Rename analysis options file for test.dart.
R=nbosch@google.com

Review-Url: https://codereview.chromium.org/2858323002 .
2017-05-04 14:05:34 -07:00
Bob Nystrom f5471fb948 Make test.dart strong mode clean!
R=whesse@google.com

Review-Url: https://codereview.chromium.org/2855883008 .
2017-05-04 10:48:39 -07:00
Bob Nystrom 9cd2f83ef7 Small-scale clean ups in test.dart, mainly around test_progress.
Stuff like:

- Use core lib methods like padLeft() now that they exist.
- Use collection literals (!).
- Use getters where appropriate.
- Make Formatter enum-like since it has no interesting instance state.

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2855073002 .
2017-05-03 16:50:45 -07:00
Vyacheslav Egorov 7b2e162775 Gardening: update path to capture_screenshot.ps1 script
R=kustermann@google.com
BUG=

Review-Url: https://codereview.chromium.org/2857923002 .
2017-05-03 16:04:20 +02:00
Bob Nystrom 53b2ef34ca Move test.dart into testing/dart.
This is mainly so that all of the code relating to test.dart is in one
directory tree so things like "Find All Usages" work a little better.
It felt weird to me to have a .dart file two directories up importing a
bunch of stuff within "testing/dart/".

Also cleaned up the affected code since it could use a little love. I'm
working on getting test.dart running DDC tests, but from poking around,
it seems like it could use some housekeeping as well.

R=vsm@google.com, whesse@google.com

Review-Url: https://codereview.chromium.org/2848103002 .
2017-05-02 16:48:28 -07:00
Erik Corry aa6353b6da Dart SDK Spelling b, c, and d.
R=kmillikin@google.com
BUG=

Review-Url: https://codereview.chromium.org/2850783002 .
2017-05-01 08:28:10 +02:00
Kevin Millikin 385f8fb054 Read platform.dill in the VM.
1. A --platform flag is added to dart to give a path to a Kernel
   binary for the platform libraries (as produced by building the
   runtime_kernel target).

2. This binary is used for bootstrapping.  Since it contains libraries
   other then the VM's bootstrap libraries, they are also loaded.

3. The frontend does not send any library with a dart: import URI
   scheme.  Note that it does not (yet) prune the canonical name
   table, which will contain a lot of unnecessary names used for
   internal linkages in the platform libraries.

4. There is a single dependency in the platform libraries on the
   script: _getMainClosure in dart:_builtin.  This is patched after
   the script is loaded.

BUG=
R=ahe@google.com, kustermann@google.com, vegorov@google.com

Review-Url: https://codereview.chromium.org/2786083002 .
2017-04-25 20:04:25 +02:00
Bob Nystrom 5a03535766 Remove the "pkgbuild" step on the bots.
Those tests are flaky since they fail when AppEngine flakes out. Even
when they work, they don't provide much value.

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2801143003 .
2017-04-24 13:28:16 -07:00
Erik Corry 2fb5d08102 Spelling a
R=ahe@google.com
BUG=

Review-Url: https://codereview.chromium.org/2841543002 .
2017-04-24 14:53:51 +02:00
Jacob Richman 119b2d58ce Format all files under tools and utils directory.
BUG=
R=terry@google.com

Review-Url: https://codereview.chromium.org/2827793002 .
2017-04-20 09:08:31 -07:00
Martin Kustermann 0de8c02b6c Try work around issues executing powershell scripts on windows on the bots
R=ricow@google.com

Review-Url: https://codereview.chromium.org/2817553005 .
2017-04-12 12:28:20 +02:00
Martin Kustermann a3e9cd8ff6 Attempt at capturing screenshot on IE if tests time out
The hypothesis is that a modal dialog from Internet Explorer causes
the currently running test to hang until test.dart kills the browser.

Capturing a screenshot might give an insight into there is a dialog showing up.

BUG=https://github.com/dart-lang/sdk/issues/28955
R=johnniwinther@google.com

Review-Url: https://codereview.chromium.org/2811093003 .
2017-04-12 10:57:58 +02:00
Erik Corry 782fbee690 Improve test script help output for --compiler
R=kustermann@google.com
BUG=

Review-Url: https://codereview.chromium.org/2811973002 .
2017-04-11 14:54:51 +02:00
William Hesse ac23816bb7 Fix an error where html_test (which is usually null) is used without checking for null, when reporting status back to the test controller.
BUG=
R=ahe@google.com

Review-Url: https://codereview.chromium.org/2804543004 .
2017-04-06 17:16:38 +02:00
William Hesse 87f38760e6 Add retry to test scripts' attempt to delete temporary Chrome config dirs.
BUG=https://github.com/dart-lang/sdk/issues/29282
R=sortie@google.com

Review-Url: https://codereview.chromium.org/2803613005 .
2017-04-06 16:05:59 +02:00
William Hesse c3d4674a8c Remove unused call to discoverPackagesInRepository from test scripts.
Rewrite StandardTestSuite.forEachTest using async/await.

BUG=
R=kustermann@google.com

Review-Url: https://codereview.chromium.org/2793333002 .
2017-04-04 17:31:42 +02:00
Ryan Macnak 21e9a4fc65 Use -no_compact_unwind when creating dylibs on Mac.
Instructs the linker not to attempt generating _eh_frame from the DWARF info, avoiding warnings like

R=vegorov@google.com

ld: warning: could not create compact unwind for _kDartVmSnapshotInstructions: register saved more than once (might be shrink wrap)
Review-Url: https://codereview.chromium.org/2784253003 .
2017-03-31 13:37:09 -07:00