Commit Graph

1498 Commits

Author SHA1 Message Date
William Hesse d8ec6450d0 Fix compiler==none case in browser testing (Dartium case)
Error was introduced in https://codereview.chromium.org/2947473002/

BUG=
R=kustermann@google.com

Review-Url: https://codereview.chromium.org/2950173004 .
2017-06-22 11:30:23 +02:00
Bob Nystrom b452b39962 Basic support for dev_compiler in test.dart.
It can compile and run tests on Chrome. There are a lot of failing tests
that I (or the team) will need to triage, but I think at least basic
tests are working as expected.

There is code that could be cleaned up to more neatly factor how dart2js
and dartdevc are handled now that there are two separate compilers to
JS. There's also some redundant code between the path for testing
compile errors (enqueueStandardTest()) and the path for running a test
in the browser.

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2947473002 .
2017-06-21 13:19:28 -07:00
Siva Chandra bac83e0973 Replace the --platform vm_option with --kernel-binaries option.
This new flag value specifies the directory in which the VM should look
up the platform.dill file. A future change will require the VM to load
another kernel binary from disk. This binary will also live in the same
directory that --kernel-binaries specifies. This way, we avoid adding a
different flag for each of the different binaries.

R=asiva@google.com

Review-Url: https://codereview.chromium.org/2933203004 .
2017-06-16 12:22:31 -07:00
William Hesse f99509ad42 Fix bad merge in refactoring
The land of https://codereview.chromium.org/2933973002 merged badly
with https://codereview.chromium.org/2934243002/

BUG=https://github.com/dart-lang/sdk/issues/28955
R=sortie@google.com

Review-Url: https://codereview.chromium.org/2938383002 .
2017-06-16 10:10:54 +02:00
Bob Nystrom a6ca718e98 Simplify Command classes.
- Get rid of separate CommandBuilder class and singleton pattern. It
  was being passed around explicitly even though half of the places
  that received a CommandBuilder as a parameter still directly called
  CommandBuilder.instance instead of using it.
- Get rid of Command caching. As far as I can tell, it makes no
  measurable difference in runtime performance or memory usage. Even
  with a large invocation of a lot of configurations and tests, the
  Command classes don't seem to be a significant use of memory.
- Shorten the factory names. "get" adds no value, and we know it
  returns a "Command" since it's on Command.

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2933973002 .
2017-06-15 14:19:03 -07:00
Bob Nystrom 979026abef Don't allow "none" as a runtime for dartk.
Also, add a little sanity checking in TestCase.

Fix #29842.

R=ahe@google.com

Review-Url: https://codereview.chromium.org/2941603002 .
2017-06-14 10:31:09 -07:00
William Hesse e64ee95959 Move ie11 timeout handling code to BrowserControllerTestOutcome
Only content_shell testing uses the BrowserCommandOutputImpl class.
Rename the class to ContentShellCommandOutputImpl.

BUG=https://github.com/dart-lang/sdk/issues/28955
BUG=https://github.com/dart-lang/sdk/issues/29869
R=kustermann@google.com

Review-Url: https://codereview.chromium.org/2934243002 .
2017-06-14 15:50:41 +02:00
William Hesse d98d32b63c Report ie11 timeouts in debug log, not as errors (attempt 2)
BUG=https://github.com/dart-lang/sdk/issues/28955
R=kustermann@google.com

Review-Url: https://codereview.chromium.org/2938813002 .
2017-06-14 14:17:27 +02:00
Bob Nystrom 571839ed90 Remove negated variables now that we support "!".
R=lrn@google.com

Review-Url: https://codereview.chromium.org/2924183002 .
2017-06-12 15:50:59 -07:00
Bob Nystrom 19cb0cafb7 Simplify test_suite.dart.
- Get rid of baseCommand, which is only ever an empty list.
- Move some utility classes out of the giant test_suite.dart.

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2915843003 .
2017-06-09 16:46:49 -07:00
Bob Nystrom 8bc4e89219 Remove unused record and replay functionality.
R=whesse@google.com

Review-Url: https://codereview.chromium.org/2920633002 .
2017-06-08 09:18:03 -07:00
Ryan Macnak b69b3e58f0 Update test.py's --arch=all to match that of build.py.
R=zra@google.com

Review-Url: https://codereview.chromium.org/2918403002 .
2017-06-05 13:07:30 -07:00
Alexander Aprelev 6573c74301 Reenable vm tests with -cdartk DFE parser.
This was accidentally disabled in 7e2d9ac85d.

BUG=#28264
R=asiva@google.com

Review-Url: https://codereview.chromium.org/2920733004 .
2017-06-02 09:34:11 -07:00
Lasse R.H. Nielsen 5720e3556c Add negation to single-identifier tests in status files.
Optimize some regegps too.

Fixes #29756

BUG= http://dartbug.com/29756
R=rnystrom@google.com

Review-Url: https://codereview.chromium.org/2913963002 .
2017-06-02 10:38:14 +02:00
Bob Nystrom db06324975 Remove temp code to ignore "--failure-summary".
Now that the BuildBots aren't passing it, we don't need to accept it.

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2916123002 .
2017-06-01 15:45:55 -07:00
William Hesse e161ad12ac Allow test.py to run dartdoc tests. Run dartdoc tests on pub bots.
BUG=
R=jcollins@google.com

Review-Url: https://codereview.chromium.org/2914233002 .
2017-06-01 18:53:04 +02:00
Bob Nystrom a9d4afc077 Handle "ff" as a runtime name in status files.
R=efortuna@google.com

Review-Url: https://codereview.chromium.org/2917843002 .
2017-05-31 17:49:22 -07:00
Bob Nystrom 7e2d9ac85d Revert "Revert "Replace the configuration map with a typed object.""
This reverts commit 8bada4873a.

R=nweiz@google.com

Review-Url: https://codereview.chromium.org/2919573003 .
2017-05-31 15:39:23 -07:00
Bob Nystrom 8bada4873a Revert "Replace the configuration map with a typed object."
This reverts commit 06f75fe5cd.

Review-Url: https://codereview.chromium.org/2914893003 .
2017-05-31 15:08:21 -07:00
Bob Nystrom 06f75fe5cd Replace the configuration map with a typed object.
This is a pretty massive change but my hope is it will make
test.dart easier to maintain going forward. Instead of
passing around a stringly-typed Map<String, dynamic> all
throughout the program, we parse the options and then create
a Configuration object that has typed getters for all of the
various bits of configuration data.

This is a little tedious because it means declaring a new
option requires also declaring a corresponding field in the
Configuration class and passing it through the constructor.
I think it's worth it.

Also, enum-like configuration properties like architecture
and runtime now have their own classes as well. Moved a
bunch of stuff from TestUtil into those classes now that
there is an object to hang those methods off of.

In the process, I found a few typos in string literals where
the code wasn't correctly looking up a configuration
property.

Added more sanity checking and validation to status file
parsing. You will get an error at parse time if you try to
refer to a variable that isn't in the whitelist of known
variables. Also, you'll get an error if you try to compare
a variable to a value that it isn't expected to have.

Many other small-scale cleanups.

Aside from the status file validation, this should behave
mostly the same as current test.dart except that tests
may be enqueued in a slightly different order. The
rewritten code for expanding configurations iterates
through the architecture, runtime, etc. options in a
slightly different order.

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2901923003 .
2017-05-31 14:47:31 -07:00
Bob Nystrom 7ce978bf91 Tighten types in test.dart even more.
This removes all implicit casts and many implicit uses of dynamic. It
adds a gratuitous number of explicit "as" casts and arguably makes the
code worse.

This is an interim step towards replacing the big configuration map
with an actual typed object. These "as" casts should help catch places
where the configuration object is being used and where the code will
need to be changed to use a new object.

R=sigmund@google.com

Review-Url: https://codereview.chromium.org/2903703002 .
2017-05-30 14:02:24 -07:00
Bob Nystrom c296f75d10 Revert "Revert "Revert "Revert "Refactor test option parsing code.""""
This reverts commit 02547656cc.

R=efortuna@google.com

Review-Url: https://codereview.chromium.org/2904313002 .
2017-05-26 15:04:55 -07:00
Bob Nystrom 02547656cc Revert "Revert "Revert "Refactor test option parsing code."""
This reverts commit 79af418430.

Review-Url: https://codereview.chromium.org/2912563002 .
2017-05-26 14:46:04 -07:00
Bob Nystrom 79af418430 Revert "Revert "Refactor test option parsing code.""
This reverts commit d03835783c.

R=efortuna@google.com

Review-Url: https://codereview.chromium.org/2908833002 .
2017-05-26 14:18:13 -07:00
Bob Nystrom d03835783c Revert "Refactor test option parsing code."
This reverts commit 9c2dff8ebc.

R=nweiz@google.com

Review-Url: https://codereview.chromium.org/2909723002 .
2017-05-26 13:51:26 -07:00
Bob Nystrom 9c2dff8ebc Refactor test option parsing code.
This doesn’t touch the resulting configuration map, which is what this
is leading up to, but it cleans up the specification and parsing of the
options and removes a lot of redundancy.

Behavior should be the same as it was before, except that all options
now allow both their underscore-separated and hyphen-separated names.
So this is now valid:

test.py --hot_reload --append-logs

Where before, you use to have to do --hot-reload (OK) and
--append_logs (bad, since almost all other options expect hyphens).

R=sigmund@google.com

Review-Url: https://codereview.chromium.org/2902023002 .
2017-05-26 13:36:44 -07:00
Alexander Aprelev 16c0db1022 Revert "Revert "With this change 'tools/test.py -cdartk vm' will run vm tests with Dart Frontend parser.""
This reverts commit e0cdea811a.

Update status of vm tests, that don't work with DFE yet.

BUG=https://github.com/dart-lang/sdk/issues/28264
R=asiva@google.com

Review-Url: https://codereview.chromium.org/2895153005 .
2017-05-24 20:16:48 -07:00
Alexander Aprelev e0cdea811a Revert "With this change 'tools/test.py -cdartk vm' will run vm tests with Dart Frontend parser."
This reverts commit f62a2a9562.

Commit above [expectedly] broke runtime/vm tests since it uses DFE
instead of vm parser, but I didn't realize we actually test -cdartk
configuration. This has to be resubmitted with status file update to
mark 235 vm tests failing with DFE.

TBR=siva
BUG=

Review-Url: https://codereview.chromium.org/2901733002 .
2017-05-22 19:42:53 -07:00
Alexander Aprelev f62a2a9562 With this change 'tools/test.py -cdartk vm' will run vm tests with Dart Frontend parser.
BUG=https://github.com/dart-lang/sdk/issues/28264
R=asiva@google.com

Review-Url: https://codereview.chromium.org/2881833003 .
2017-05-22 17:24:38 -07:00
Martin Kustermann 22c7536d36 Reduce temporarily flakiness rate of dart2js-drt by re-running until they have been fixed
We should either remove dart2js-drt or use a stable version of
content_shell from upstream chromium (instead of our dartium-based one)
for dart2js-drt.

Issue #29655

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2891343002 .
2017-05-19 13:13:40 +02:00
Bob Nystrom 92267afc66 Refactor and clean up the status file parsing code. - Make the parser less error tolerant. The expression parser used to ignore any unrecognized tokens, which means a status like "RuntimeError CompileError" (not the missing comma) was parsed as simply "RuntimeError", which seems bad. Now it reports an error. Fixed a couple of status files that thought they were setting statuses that they weren't (!).
- Separate out parsing a status file from applying the environment to
  determine which sections are active. This makes it possible to, for
  example, generate expectation sets for multiple environments without
  having to reparse each time.

- Simplify expression parsing. Remove set expressions since they weren't
  used for anything useful. A test's expectations are a simple
  comma-separated list and don't need anything beyond that. Merge
  Scanner and Tokenizer since the latter was a glorified function.

- Make more names private so that it's clearer what's used outside of
  various libraries.

- Generally modernize the style.

- Add *lots* of documentation.

Again, there should be no behavioral changes. I ran:

  ./tools/test.py -m release,debug -c none,dart2js,dart2analyzer -r none,vm,d8 corelib

Before and after the change and verified that the output was the same
(aside from timing).

R=sigmund@google.com

Review-Url: https://codereview.chromium.org/2891753003 .
2017-05-18 12:42:52 -07:00
Bob Nystrom c269ef53a5 Revert "Re-apply status file parser changes from 0b7728da1bef08c1c1e092005d9fd8c8bff5fa6c."
This reverts commit a881aed354.

TBR.

R=paulberry@google.com

Review-Url: https://codereview.chromium.org/2888213002 .
2017-05-17 13:45:40 -07:00
Bob Nystrom a881aed354 Re-apply status file parser changes from 0b7728da1b.
R=whesse@google.com

Review-Url: https://codereview.chromium.org/2880203004 .
2017-05-17 13:04:35 -07:00
Bob Nystrom 01e5b1b40c Revert "Refactor and clean up the status file parsing code."
This reverts commit 0b7728da1b.

R=nbosch@google.com

Review-Url: https://codereview.chromium.org/2885623002 .
2017-05-15 15:58:05 -07:00
Bob Nystrom 0b7728da1b Refactor and clean up the status file parsing code.
- Make the parser less error tolerant. The expression parser used to
  ignore any unrecognized tokens, which means a status like
  "RuntimeError CompileError" (not the missing comma) was parsed as
  simply "RuntimeError", which seems bad. Now it reports an error.
  Fixed a couple of status files that thought they were setting statuses
  that they weren't (!).

- Separate out parsing a status file from applying the environment to
  determine which sections are active. This makes it possible to, for
  example, generate expectation sets for multiple environments without
  having to reparse each time.

- Simplify expression parsing. Remove set expressions since they weren't
  used for anything useful. A test's expectations are a simple
  comma-separated list and don't need anything beyond that. Merge
  Scanner and Tokenizer since the latter was a glorified function.

- Make more names private so that it's clearer what's used outside of
  various libraries.

- Generally modernize the style.

- Add *lots* of documentation.

Again, there should be no behavioral changes. I ran:

  ./tools/test.py -m release,debug -c none,dart2js,dart2analyzer -r none,vm,d8 corelib

Before and after the change and verified that the output was the same
(aside from timing).

BUG=
R=whesse@google.com

Review-Url: https://codereview.chromium.org/2875203005 .
2017-05-15 15:42:15 -07:00
Bob Nystrom e0180925f6 Tighten up a bunch of types in test.dart.
This doesn't get rid of all implicit dynamic and casts, but it covers
many of them.

Also did some minor modernization when I noticed it:

- Using "var" for local variables where inference does the right thing.
- camelCase for variable names.
- More collection literals.

There are (or should be!) zero behavioral changes.

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2863253002 .
2017-05-15 15:26:40 -07:00
William Hesse 6f3d45f5a3 Check for infrastructure failure before reporting timeouts and crashes.
BUG=http://dartbug.com/28955

Review-Url: https://codereview.chromium.org/2881073002 .
2017-05-15 18:27:46 +02:00
William Hesse c235977ddb Increase startup time allowed for browsers
On a cold, overloaded system, IE sometimes takes more than 60 seconds to start.

BUG=https://github.com/dart-lang/sdk/issues/28955
R=kustermann@google.com

Review-Url: https://codereview.chromium.org/2878423002 .
2017-05-15 13:36:44 +02:00
William Hesse bff560292c Fix bug in test_runner.dart
This error was introduced in https://codereview.chromium.org/2875683002/
committed as d6ca1a5def.

BUG=
TBR=karlklose@google.com

Review-Url: https://codereview.chromium.org/2874193002 .
2017-05-11 13:39:46 +02:00
William Hesse d6ca1a5def Report IE11 timeouts in debug log, not as failing tests.
BUG=https://github.com/dart-lang/sdk/issues/28955
R=sigmund@google.com

Review-Url: https://codereview.chromium.org/2875683002 .
2017-05-11 11:18:42 +02:00
Terry Lucas f27144d7b5 Roll 50: Updated for push to origin/master.
TBR=jacobr@google.com

Review-Url: https://codereview.chromium.org/2875773003 .
2017-05-10 18:16:51 -07:00
Bob Nystrom 0592404cc3 Re-add missing space in test failed message.
R=sigmund@google.com

Review-Url: https://codereview.chromium.org/2866273002 .
2017-05-09 11:16:42 -07:00
Bob Nystrom 0635d4d559 Rename analysis options file for test.dart.
R=nbosch@google.com

Review-Url: https://codereview.chromium.org/2858323002 .
2017-05-04 14:05:34 -07:00
Bob Nystrom f5471fb948 Make test.dart strong mode clean!
R=whesse@google.com

Review-Url: https://codereview.chromium.org/2855883008 .
2017-05-04 10:48:39 -07:00
Bob Nystrom 9cd2f83ef7 Small-scale clean ups in test.dart, mainly around test_progress.
Stuff like:

- Use core lib methods like padLeft() now that they exist.
- Use collection literals (!).
- Use getters where appropriate.
- Make Formatter enum-like since it has no interesting instance state.

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2855073002 .
2017-05-03 16:50:45 -07:00
Vyacheslav Egorov 7b2e162775 Gardening: update path to capture_screenshot.ps1 script
R=kustermann@google.com
BUG=

Review-Url: https://codereview.chromium.org/2857923002 .
2017-05-03 16:04:20 +02:00
Bob Nystrom 53b2ef34ca Move test.dart into testing/dart.
This is mainly so that all of the code relating to test.dart is in one
directory tree so things like "Find All Usages" work a little better.
It felt weird to me to have a .dart file two directories up importing a
bunch of stuff within "testing/dart/".

Also cleaned up the affected code since it could use a little love. I'm
working on getting test.dart running DDC tests, but from poking around,
it seems like it could use some housekeeping as well.

R=vsm@google.com, whesse@google.com

Review-Url: https://codereview.chromium.org/2848103002 .
2017-05-02 16:48:28 -07:00
Erik Corry aa6353b6da Dart SDK Spelling b, c, and d.
R=kmillikin@google.com
BUG=

Review-Url: https://codereview.chromium.org/2850783002 .
2017-05-01 08:28:10 +02:00
Kevin Millikin 385f8fb054 Read platform.dill in the VM.
1. A --platform flag is added to dart to give a path to a Kernel
   binary for the platform libraries (as produced by building the
   runtime_kernel target).

2. This binary is used for bootstrapping.  Since it contains libraries
   other then the VM's bootstrap libraries, they are also loaded.

3. The frontend does not send any library with a dart: import URI
   scheme.  Note that it does not (yet) prune the canonical name
   table, which will contain a lot of unnecessary names used for
   internal linkages in the platform libraries.

4. There is a single dependency in the platform libraries on the
   script: _getMainClosure in dart:_builtin.  This is patched after
   the script is loaded.

BUG=
R=ahe@google.com, kustermann@google.com, vegorov@google.com

Review-Url: https://codereview.chromium.org/2786083002 .
2017-04-25 20:04:25 +02:00
Bob Nystrom 5a03535766 Remove the "pkgbuild" step on the bots.
Those tests are flaky since they fail when AppEngine flakes out. Even
when they work, they don't provide much value.

R=whesse@google.com

Review-Url: https://codereview.chromium.org/2801143003 .
2017-04-24 13:28:16 -07:00