The placeholder executable is replaced by a shell script that invokes
pkg/analyzer_fe_comparison (in a similar way to how
sdk/bin/dartanalyzer invokes pkg/analyzer_cli). This required some
rewriting of pkg/analyzer_fe_comparison to support analysis of a
single test case (as opposed to analysis of a package, which is what
it did before).
It's now possible to run the comparison tool over the test suite in
language_2. There are many failures, which I'll begin addressing in
follow-up CLs.
Change-Id: I52d8f3b53064e8e29028571d416f2573c6cba35d
Reviewed-on: https://dart-review.googlesource.com/74820
Reviewed-by: Brian Wilkerson <brianwilkerson@google.com>
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Commit-Queue: Paul Berry <paulberry@google.com>
This CL just adds the compiler=compareAnalyzerCfe option to the test
framework, and updates the status files so that no language_2 failures
are expected when this "compiler" is used. A placeholder executable
has been added to sdk/bin, but it is not wired up yet--that will be
done in a future CL.
In a future CL I will wire up the placeholder executable to the code
in pkg/analyzer_fe_comparison, and fill in the methods in
CompareAnalyzerCfeCommandOutput to process the output accordingly.
Change-Id: I27439de4ac609cb8b003f157a0ff88d181bfbcae
Reviewed-on: https://dart-review.googlesource.com/73561
Commit-Queue: Paul Berry <paulberry@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
These will be the values reported to the test result database.
There was previously no value with type Expectation that reflected
the desired semantics of realExpected. The result was processed
using flags like "hasCompileTimeError" and "isNegative" to produce
a processed result. The new realResult field avoids that processing.
Change-Id: Ic59a9bf2c94cc0ccdaa31db3c7f5fde004918004
Reviewed-on: https://dart-review.googlesource.com/72105
Reviewed-by: Jonas Termansen <sortie@google.com>
This includes Fasta, tools and observatory, so the checked-in SDK must
have the lower-case constants.
Change-Id: I8380ad041ad058f7d02ae19caccfecd434d13d75
Reviewed-on: https://dart-review.googlesource.com/50201
Commit-Queue: Lasse R.H. Nielsen <lrn@google.com>
Reviewed-by: Leaf Petersen <leafp@google.com>
test.py looks at the status code to find out if a test crashed or not.
On linux everything < 0 is a crash, but on Windows a bit pattern is
in play.
It is unclear why this is the case, but some test currently 'Fail'
(instead of the expected 'Crash') on Windows because of exit code
-1073740791 (aka 3221226505 aka 0xC0000409 aka
STATUS_STACK_BUFFER_OVERRUN
(https://msdn.microsoft.com/en-us/library/cc704588.aspx)) indicating
that something isn't quite right with the crash-sniffing on Windows.
This CL changes it to indicate a crash if the exit-code is "0xC0...",
"0x80..." or "0x40..." as listed by
https://msdn.microsoft.com/en-us/library/cc704588.aspx.
Change-Id: I6c692173c3c3b5fb6c857a55a739d8f136d662bf
Reviewed-on: https://dart-review.googlesource.com/36261
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
This reverts commit 87a009422e.
Reason for revert: Doesn't cover 0x80 and 0x40. Upcomming CL will do that.
Original change's description:
> [test.py] Refactor and change crash-sniffing
>
> test.py looks at the status code to find out if a test crashed or not.
> On linux everything < 0 is a crash, but on Windows a bit pattern is
> in play.
> It is unclear why this is the case, but some test currently 'Fail'
> (instead of the expected 'Crash') on Windows because of exit code
> -1073740791 (aka 3221226505 aka 0xC0000409 aka
> STATUS_STACK_BUFFER_OVERRUN
> (https://msdn.microsoft.com/en-us/library/cc704588.aspx)) indicating
> that something isn't quite right with the crash-sniffing on Windows.
>
> This CL changes it to indicate a crash if the exit-code is "0xC00..." as
> this is a "NTSTATUS code" (which must mean crash) according to
> https://blogs.msdn.microsoft.com/joshpoley/2011/06/13/common-process-termination-values/.
>
> Change-Id: Ie7e1a00b3b870ea176dbdc4db41dc458e61e9a28
> Reviewed-on: https://dart-review.googlesource.com/36240
> Reviewed-by: Johnni Winther <johnniwinther@google.com>
> Commit-Queue: Jens Johansen <jensj@google.com>
TBR=jensj@google.com,johnniwinther@google.com
Change-Id: Iba1d3e1aa1c7525e374e2d0763926b13a67169ad
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/36260
Reviewed-by: Jens Johansen <jensj@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
test.py looks at the status code to find out if a test crashed or not.
On linux everything < 0 is a crash, but on Windows a bit pattern is
in play.
It is unclear why this is the case, but some test currently 'Fail'
(instead of the expected 'Crash') on Windows because of exit code
-1073740791 (aka 3221226505 aka 0xC0000409 aka
STATUS_STACK_BUFFER_OVERRUN
(https://msdn.microsoft.com/en-us/library/cc704588.aspx)) indicating
that something isn't quite right with the crash-sniffing on Windows.
This CL changes it to indicate a crash if the exit-code is "0xC00..." as
this is a "NTSTATUS code" (which must mean crash) according to
https://blogs.msdn.microsoft.com/joshpoley/2011/06/13/common-process-termination-values/.
Change-Id: Ie7e1a00b3b870ea176dbdc4db41dc458e61e9a28
Reviewed-on: https://dart-review.googlesource.com/36240
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
The batch runner uses an exitCode of 1 to signal compile-time errors,
which we need to support in the VMKernelCompilationCommandOutput.
Furthermore allow running the output of a dart -> kernel compilation on
the VM even though we expect an compile-time error (sometimes the
frontend has bugs and doesn't emit the compile-time errors and we need
to run the VM to get them)
Issue https://github.com/dart-lang/sdk/issues/31585
Change-Id: I8b4c34557dbf3de487247d75b02777bacbd452c1
Reviewed-on: https://dart-review.googlesource.com/28642
Reviewed-by: Martin Kustermann <kustermann@google.com>
This makes isolate tests fail, since we no longer run from "source" (or
rather use the kernel-isolate to to "source -> dill" for us).
The special vm/cc suite will continue to be run via the kerne-isolate, so we
have the coverage for these (which probably include reload tests).
Issue https://github.com/dart-lang/sdk/issues/31585
Change-Id: I51bd2f9345d650b4ff2a98aa1c8365c765e0d013
Reviewed-on: https://dart-review.googlesource.com/28722
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Preliminary testing shows, this increases performance by around 20%. The
main benefit is by re-using a warmed-up VM and not start one from
scratch for every compilation.
Going forward we can do more optimizations, e.g. reading the platform
dill file only once into memory (instead of repeatedly) ...
=> This requires us using the new state-full IKG compiler.
Issue https://github.com/dart-lang/sdk/issues/31585
Change-Id: I2d3448783fc118611baf4671187a897227a65a6c
Reviewed-on: https://dart-review.googlesource.com/28400
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Preliminary testing shows, this increases performance by around 20%. The
main benefit is by re-using a warmed-up VM and not start one from
scratch for every compilation.
Going forward we can do more optimizations, e.g. reading the platform
dill file only once into memory (instead of repeatedly) ...
=> This requires us using the new state-full IKG compiler.
Change-Id: I74db7dbb1aa79289d1045ef41f960215cf5b3b35
Reviewed-on: https://dart-review.googlesource.com/28240
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
This CL modifies the Dart source used from test.py such that it takes
`syntax error` into account as an expected outcome in test files (so
that we can have `//# 01: syntax error` with a similar meaning as
`//# 01: compile-time error`).
For all tools except the spec_parser, `syntax error` is the same
outcome as `compile-time error`; that is, nobody else will see the
difference.
For the spec_parser, `syntax error` is the outcome where parsing has
failed; `compile-time error` is taken to mean some other compile-time
error, i.e., the spec_parser is expected to _succeed_ when the
expected outcome is `compile-time error`.
Test files in language and language_2 have been adjusted to use the
outcome `syntax error` where appropriate.
The status files in language and language_2 for the spec_parser have
been adjusted such that they fit all the new `syntax error` outcomes
in test files.
Other status files have been adjusted in a few cases where tests were
corrected (because a compile-time error which was clearly not intended
to be a syntax error turned out to be caused by a typo, which means
that the actual compile-time error has never been tested).
The spec grammar Dart.g was adjusted in a few cases, when some bugs
were discovered. In particular, the treatment of Function has been
changed: It is now known by the parser that Function does not take
any type arguments. This makes no difference for developers, because
they cannot declare a type named Function anyway, but it means that
a number of tricky parsing issues were resolved.
Dart.g was also adjusted to allow `qualified` to contain three
identifiers, which is an old bug (preventing things like metadata on
the form `@p.C.myConst`).
Change-Id: Ie420887d45c882ef97c84143365219f8aa0d2933
Reviewed-on: https://dart-review.googlesource.com/18262
Commit-Queue: Erik Ernst <eernst@google.com>
Reviewed-by: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Lasse R.H. Nielsen <lrn@google.com>
- Clean up browser test failure output. Instead of barfing a giant blob
of JSON, show just the runtime error and its stack trace. In verbose
and on the buildbots, it still shows all the events, but even there
they are formatted nicely.
- Don't redundantly show the failure summary when most of the output is
already failure text.
- Color the failure summary end message.
Change-Id: I49724b204be55a297fc5e723b80b32e0e00748f0
Reviewed-on: https://dart-review.googlesource.com/15767
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: William Hesse <whesse@google.com>
It's added as a new compiler, so pass "-c dartdevk" to use it.
It doesn't support any test packages yet, so tests that, say, import
package expect won't compile. I'll work on that next, but it will
require adding some stuff to the build scripts to build .dill files for
those packages.
This does get test.dart invoking the compiler, running the resulting
test, and correctly reporting the result:
- A test that doesn't throw an exception and stays within the bounds of
what is currently implemented in dartdevk passes.
- A test that compiles correctly but fails at runtime fails with a
RuntimeError.
- A test that contains a compile error fails with a non-zero exit code
and is reported as a CompileTimeError.
Change-Id: Icacbf1ff54dfe7aa4d245382d3b0aeb375cf105b
Reviewed-on: https://dart-review.googlesource.com/15420
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Vijay Menon <vsm@google.com>
Of course, if you're not using the color formatter, it stays uncolored.
But this gets rid of some unnecessary empty lines and section headers
that have no content.
Change-Id: I5b83345c583b79468af84f4e67a12ee757a3105f
Reviewed-on: https://dart-review.googlesource.com/14720
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Commit-Queue: Bob Nystrom <rnystrom@google.com>
This still doesn't *display* then very nicely (they are buried in the
giant blob of "events" JSON in the output), but it captures them and
plumbs them out of the browser into test.dart.
Change-Id: I54ab024ed55f18af89da7742967117f0a212673d
Reviewed-on: https://dart-review.googlesource.com/15005
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Vijay Menon <vsm@google.com>
Interesting changes:
- A static getter colliding with an inherited non-static setter is a
compile error, not a type warning.
- Trying to call a setter on what is only a getter is a compile error
with no runtime behavior.
- Add support to test.dart for negative tests in DDC.
BUG=
R=jcollins@google.com
Review-Url: https://codereview.chromium.org/3005643002 .
This required munging a bit of test.dart code too to tease out some
dependencies, but the changes are minor. I considered moving all of
test.dart out into a package and making the status file library a
public one that other packages in the repo could import but this seemed
like the less intrusive change.
R=bkonyi@google.com
Review-Url: https://codereview.chromium.org/2984203002 .
- Merge CommandOutputImpl and CommandOutput. There were no classes that
implemented CommandOutput that didn't extend the Impl.
- Remove "Impl" from the other class names.
- Make stuff private when possible. Likewise final.
- Other tiny style changes.
R=whesse@google.com
Review-Url: https://codereview.chromium.org/2946783002 .
- Get rid of separate CommandBuilder class and singleton pattern. It
was being passed around explicitly even though half of the places
that received a CommandBuilder as a parameter still directly called
CommandBuilder.instance instead of using it.
- Get rid of Command caching. As far as I can tell, it makes no
measurable difference in runtime performance or memory usage. Even
with a large invocation of a lot of configurations and tests, the
Command classes don't seem to be a significant use of memory.
- Shorten the factory names. "get" adds no value, and we know it
returns a "Command" since it's on Command.
R=whesse@google.com
Review-Url: https://codereview.chromium.org/2933973002 .
Paul Berry