Only run these tests with sound null safety. Follow up changes will remove more tests configurations, and null safety mode plumbing for
the DDC infra.
Change-Id: I0b511d778e54da2a8f5140c4a00b36eefb8000ac
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/365540
Reviewed-by: Mark Zhou <markzipan@google.com>
Commit-Queue: Nicholas Shahan <nshahan@google.com>
The first in a series of changes that deletes all DDC tests running
with unsound null safety. Follow up changes will remove more tests
configurations, and null safety mode plumbing for the DDC infra.
* Delete the unsound version of some suites.
* Renames the sound suites to be more descriptive of the code in the
test instead of the mode it is run in.
Change-Id: If2edfecc811bb78a77a1e93bd6d0b75f22c3e80c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/365520
Reviewed-by: Mark Zhou <markzipan@google.com>
Reviewed-by: Nate Bosch <nbosch@google.com>
* Extends the frontend server controller to validate/reject compile errors in compiles/recompiles.
* Extends the config to permit an 'expectedError' entry.
* Adds a handful of tests to the suite (adapted from the VM's hot reload tests).
Change-Id: I47d814e375c4c72d0406ebf5bdfee3f1975c64f0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/363800
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Mark Zhou <markzipan@google.com>
VM hot reload are run via:
1) We first emit a dill for every generation ahead of time (full dill on gen 0, incremental deltas subsequently).
2) We start a VM process at generation 0.
3) The VM process runs until it hits a `hotReload` command. It then uses the VM service protocol to connect to itself and reload the next generation.
4) The VM exits when the next generation isn't found.
* Adds config files to reload tests that allow runtime filtering.
* Implements VM-side hot reloading
* Adds several VM-specific hot reload tests
Change-Id: I1c6ad5c4eed426a0189c1b4af31297c9c1dba717
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/359200
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Mark Zhou <markzipan@google.com>
Reviewed-by: Sigmund Cherem <sigmund@google.com>
In the refactor to speed up this test, we removed most retry logic
(except for the "unreliable tests" that now apply retries in a
predictable manner).
The "noisy" tests have a similar behavior but for different reasons.
These put a lot of timing presure by serving hundreds of very large
files. The default response timeout of 5s trips this sometimes.
We could try to make this more predictable by providing a larger timeout
upfront, but I believe part of the intent with these tests was also to
incorporate retries for reasons like response timeouts. So instead, I've
added a retry, which will use by default a larger timeout on the second
attempt. This will hopefully be sufficient to remove this source of
flakiness.
Change-Id: I8eb5e9dceed5e7af36f6db45147c2d961247f6e2
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/360503
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Sigmund Cherem <sigmund@google.com>
This test was flaky about 3% of the time in linux bots due to a race
condition. Details are shared in
https://github.com/dart-lang/sdk/issues/55299. TLDR, when one test case
completes and it's execution is resumed (program 1), the test driver
sets up a breakpoint for a new test case concurrently and schedules to
run main again (program 2). Depending on when the breakpoint is set, it
could be hit by either program 1 or program 2. If both programs are in
the same state when they hit the breakpoint, then it doesn't matter
which program we use to validate the test expectations (a benign race
condition). However, this is problematic when the state is different,
which can happen when test 1 and test 2 both are using the same
breakpoint in a loop. This is what happened for the `forLoopTestBP` in
this CL.
To workaround this data race between two test cases, I decided to simply
combine them into a single test case. A different alternative is to copy
the test function and use a different breakpoint, but that didn't seem
worthwhile in this case.
Change-Id: I561c2a25d33d56a8543fe061342cab9cceafd4e4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/359721
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Sigmund Cherem <sigmund@google.com>
Currently our end-2-end tests create a temporary folder and delete
it after the suite has completed. Only one folder is created for the
entire test run, so the deletion only gets scheduled when the entire
suite has completed.
On windows bots, the cleanup very often fails causing up to 40%
flakiness on some tests.
This CL keeps the cleanup logic as is, but will now deem tests as
passing even when we can't do the proper cleanup.
Change-Id: I8e2e2d39d1bbdaf48c138c0a05a9a2540345fc28
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/359760
Commit-Queue: Sigmund Cherem <sigmund@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
* Adds tests for the memory filesystem
* Enables hot restart/reload tests
* Adds flags to the hot reload suite
* Makes path resolution logic windows-friendly
* Adds test-reporting logic for trybots to hot reload tests
Change-Id: Ic51a0b8a3c6f8b6de20b58b2ac185dacf444cf47
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/353788
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Reviewed-by: Nate Bosch <nbosch@google.com>
Commit-Queue: Mark Zhou <markzipan@google.com>
* Delete unused stuff
* rename 'path' to 'root'
* accept 'includeEndsWith' as a plain text string so we often can avoid
using regexp
* accept 'subRoots' to filter to directories faster and more precisly
than when using regexps in 'pattern'
* make 'list' async instead of async* (no more yield stuff which we
promptly turn into a list when actually using it)
Note that some changes in testing.json is not 100% semantic-preserving,
e.g. the "parser_all" suite previously had a pattern "/tests/.*\\.dart$"
which was probably meant to include all dart files in the "tests" folder
in the root, but in fact included all dart files in a "tests" folder
anywhere. The updated version does not.
Change-Id: Idd014274f86bf6214dee0753a7738ec80bc6a49e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/358442
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
While working on the e2e tests, I noticed this test was taking a
long time. Turns out this was because it fully exercises an http
client retry logic using exponential backoff and up to 4 retries.
To reduce the costs, I changed the test to use controlled
environment instead. This still hits the same code-paths but
provides a faster turnaround by using at most 1 retry, and issuing
retries immediately.
With these changes, the test completes in 3s (instead of 28s).
I noticed that some of the tests to check an "unreliable" server
where non-deterministic (multiple parallel requests that sometimes
fail or succeed). The changes here also make it deterministic. While
this test seems to have some flakes due to other reasons, my hope is
that these changes will reduce the chances of this test being flaky
on the bots.
Change-Id: I8eb2a00c7a489ab4e5a46e2b5cdbee40d1ffb8d5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/356623
Commit-Queue: Sigmund Cherem <sigmund@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Ideally we would write each source next to each test to make tests more
readable, but clustering all tests to use a common test file really
helps boost the runtime of the end2end tests now that we reuse the same
page between tests that share the same code.
This CL refactos the tests to mostly use one shared test file. Doing so
speeds up tests between 2x-4x depending on the test.
Change-Id: I9afd00c6a42226899969e3f605ac18e0d2b1a906
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/356285
Reviewed-by: Mark Zhou <markzipan@google.com>
Commit-Queue: Sigmund Cherem <sigmund@google.com>
Since we changed how the test driver sets up breakpoints, we now we
have the ability to reuse a page for multiple tests.
Before, the test harness would follow this sequence:
* on each group of tests, it would call `_initSource`, which sets up
a boostraper and code to execute on the browser. This code was
reused by all tests in the group.
* on each test, it would indirectly call `_loadScript`, which
navigated to brand new page for each test. If there were multiple
tests in a group, this would load the boostraper and code once per
test.
This CL keeps the sequence of operations the same, except, that
`_loadScript` will reuse the existing page if it already loaded the
boostraper and DDC code needed by the next test.
This greatly reduces the total running time of some of the
expresison compiler shards. On my local machine one such test took
24s before both this and the parent CL, but now takes about 5s.
Change-Id: I7774df9e83142b42efc4ad523f589758a2a4660c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/356301
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Reviewed-by: Mark Zhou <markzipan@google.com>
Instead of pausing the application while loading, this
changes the bootstrap logic to not invoke main immediately.
Later the test harness uses an expression evaluation to
trigger the start of the application when it is ready for
it.
This change will enable the ability to reuse the same page
and script across multiple tests, which will save a lot of
test execution time (see related child CL).
Change-Id: Ifa1050ac777aabb3e21b69953e85c6578b3e0cc0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/356281
Reviewed-by: Mark Zhou <markzipan@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
The data shows a lot of variation by machine. The slowest test samples
bring the p90 above 4s, which explains that we often hit the limit.
I expect that we'll continue seeing flakes, but less often as a result
of this change. That said, some of the e2e test are taking close to 4
minutes, so we may need additional sharding or switch our approach if
that ocurrs more often.
Change-Id: I47c0dbce6f766edf7c2845d26f11ae7c6372a7bc
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/355980
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Sigmund Cherem <sigmund@google.com>
The hot reload runner currently only supports d8, but I plan to add support for Chrome and VM execution.
Notable changes:
* Creates `package:reload_test` with helpers for running this suite.
* Updates the module loader with D8-specific branches and hooks for hot reload/restart.
* Exposes DDC runtime variables via a `HotReloadTestRuntime` API.
* Ports constant equality hot restart tests from webdev/dwds (validated to fail if either cache-clearing mechanism fails).
* Partially rolls DDC's d8 preamble forward (towards dart2js's).
* Wraps D8's timer implementation with custom timeout logic to better match Chrome's timing semantics when executing with native JS async.
Tests for the framework and matrix updates will be added in an upcoming change.
Change-Id: I2773b29f464cfd0330e4c653c05e117ae150b4a6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/350021
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Mark Zhou <markzipan@google.com>
Multiple end-to-end tests are failing in the bots because some steps
during the test execution are timing out. We currently have a
timeout of 5 seconds for these steps, but it's unclear if the
timeout is because of conditions in the bot machines (e.g. they are
slow), one off exceptions, or something else.
To address the flakiness, I expect we will want to increase the timeout,
but before we do it may be valuable to start measuring what is the
current distribution of timing in the bots to figure out the best value
to use.
This CL adds logging for timing data for that purpose. I've added
tracking for some interesting steps in the expresson-compiler test
framework, including the top two steps that usually hit a timeout in
practice (find-script and pause-event-for-line).
Example output on my local machine:
```
init-source: 233ms (avg), 121ms (p50), 696ms (p90), 696ms (max), 7 (total)
find-script: 18ms (avg), 18ms (p50), 21ms (p90), 25ms (max), 52 (total)
load-script: 22ms (avg), 22ms (p50), 27ms (p90), 30ms (max), 52 (total)
set-breakpoint: 0ms (avg), 1ms (p50), 1ms (p90), 2ms (max), 52 (total)
pause-event-for-line: 422ms (avg), 410ms (p50), 440ms (p90), 658ms (max), 52 (total)
check-in-frame: 459ms (avg), 446ms (p50), 480ms (p90), 695ms (max), 51 (total)
```
Change-Id: I04734b9905da8e1b4c9d06451be65a2dee32ba1c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/352988
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Sigmund Cherem <sigmund@google.com>
When a macro declaration is supplied as dill, the macro api library
is not directly needed by the source libraries. Therefore, in order
to determine whether macro declarations are available, we need to also
look up the macro api library in the dill loader.
Closes#54560
Change-Id: I131d3881f557566871d9e8c9f38c19029637c534
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/346000
Reviewed-by: Morgan :) <davidmorgan@google.com>
Commit-Queue: Johnni Winther <johnniwinther@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
This CL:
* adds empty dart:_dart2js_only and dart:_ddc_only libraries for use in
conditional imports,
* updates pkg/dart2js_runtime_metrics to use the new libraries rather
than dart:_dart2js_runtime_metrics and dart:js,
* and removes some unnecessary libraries, including VM-specific
implementations in pkg/dart2js_runtime_metrics and the DDC
implementation of dart:_dart2js_runtime_metrics.
Change-Id: I9500aa303fa5ad8aba0e1d413f69957c268f3f11
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/350681
Reviewed-by: Sigmund Cherem <sigmund@google.com>
When landing https://dart-review.googlesource.com/c/sdk/+/342400
the ddc-mac builder (but not Linux and Windows) started failing.
I could reproduce on Linux though so I'm not sure why those builders
didn't fail (nor why it wasn't caught by the try bot), either way this
fixes the issues:
* Test update: E.g. a breakpoint at the end of the scope doesn't work
because it will not be inside the (wanted) scope.
* Test update: Evalating "this" now actually works.
* Scope finding update: DDC adds sourcemapping entries for the *end* of
things so e.g. the getter "c" will have offset and offset+1 added as
source mappings. When translating from javascript position to dart
position we might pick that and thus ask for the scope of offset+1,
but nothing will be found because no node has that offset.
Here I add a fall-back saying that if we get no results we ask again
for the nearest lower offset.
Change-Id: I7e4430d9954466494b514cf51d999358483c9f8c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/345501
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Anna Gringauze <annagrin@google.com>
This allows using the new scope finder and facilitates expression
compilation with and in extension types.
For now the script uri is optional (and only passable in the new json
input via package:frontend_server), and only if the script uri is passed
we'll use the new scope finder.
Flutter etc should be updated to pass the new data.
Change-Id: I36eed1ea76a825e63e4c5b9ea60daf18aee39f3d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/342400
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
Reviewed-by: Anna Gringauze <annagrin@google.com>
Stop running expression evaluation and module symbols tests on legacy
code. These still run with unsound null safety, just on migrated code.
Change-Id: Ifb602c058ebcc3fbd8254e8c2f431f8b18146a64
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/337615
Commit-Queue: Nicholas Shahan <nshahan@google.com>
Reviewed-by: Anna Gringauze <annagrin@google.com>
Update source code in tests to be migrated to null safety. Tests are
still running in unsound null safety.
In most cases types are now inferred to be non-nullable so the code that
makes expressions to be seen as nullable now require `as dynamic` to
allow them to compile.
Update nullable inference logic to recognize `.toString()` on a String
as non-nullable in code that has been migrated to null safety.
Change-Id: Id0771cc5317e3dafbf10e766b80d2994752bacc8
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/337700
Commit-Queue: Nicholas Shahan <nshahan@google.com>
Reviewed-by: Mark Zhou <markzipan@google.com>
This adds support using experimental features in id testing in DDC and
uses it to add tests for scopes in extensions, extension types and
mixins.
The tests show that the DartScopeBuilder and DartScopeBuilder2 differ
on the scope on instance members in extensions and extension types for
the synthetic #this variable.
Change-Id: Iec0f3b938da567578b1245ada885370fb2a8b33f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/335824
Reviewed-by: Jens Johansen <jensj@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Johnni Winther <johnniwinther@google.com>
This adds a CFE test for DartScope that shares the data with the
corresponding DDC test. The CFE test uses the DartScopeBuilder2
which is currently in development to replace the DartScopeBuilder.
Change-Id: I0d6c76957e7a5d28babab14110d430935e686b22
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/335821
Reviewed-by: Jens Johansen <jensj@google.com>
Reviewed-by: Nicholas Shahan <nshahan@google.com>
Commit-Queue: Johnni Winther <johnniwinther@google.com>
Nicholas Shahan