Jens Johansen
69f6378ed5
In for instance https://dart-review.googlesource.com/c/sdk/+/193961/5 we had an instance of something good happening (a crashing-ish test that started to pass) but where the author was told no (red bot) and not given a path to make the bot happy (bad messaging). Specifically tests failed and the best log one got was ``` To re-run this test, run: dart pkg/front_end/test/unit_test_suites.dart -p pkg/front_end/test/fasta/textual_outline/general/constants/js_semantics/issue45376 ``` which isn't very helpful. If running it locally a little more information is given via stdout, but having to do that isn't great (and the little extra information still wasn't great). This CL updates the messaging to - hopefully - give a path to make the bot happy. The messaging will now - directly from the log on the bot - be ``` FAILED: pkg/front_end/test/fasta/textual_outline/general/constants/js_semantics/issue45376: Pass To re-run this test, run: dart pkg/front_end/test/unit_test_suites.dart -p pkg/front_end/test/fasta/textual_outline/general/constants/js_semantics/issue45376 The test passed, but wasn't expected to. You should update the status file for this test. There's a status entry looking something like general/constants/js_semantics/issue45376: FormatterCrash which should be removed. The status file is file:///[wherever]/sdk/pkg/front_end/testcases/textual_outline.status. ``` which a) gives the extra information previously only available via stdout when running locally (the first line). b) Gives an explanation as to why it failed ("The test passed, but wasn't expected to."). c) A path to make the bot green ("You should update the status file for this test. [...] should be removed"). Hopefully this will make it easier to deal with in the future when something good happens. Change-Id: I838699abce580334c956d8b4753884fd600802ae Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/194242 Commit-Queue: Jens Johansen <jensj@google.com> Reviewed-by: Johnni Winther <johnniwinther@google.com>
Package validation
The packages in pkg/ are automatically validated on the LUCI CI bots. The
validation is largely done by the tools/package_deps package; it can be tested
locally via:
dart tools/package_deps/bin/package_deps.dart
Packages which are published
There are several packages developed in pkg/ which are published to pub.
Validation of these packages is particularly important because the pub tools are
not used for these packages during development; we get our dependency versions
from the DEPS file. Its very easy for the dependencies specified in a package's
pubspec file to get out of date wrt the packages and versions actually used.
In order to better ensure we're publishing correct packages, we validate some properties of the pubspec files on our CI system. These validations include:
- that the dependencies listed in the pubspec are used in the package
- that all the packages used by the source are listed in the pubspec
- that we don't use relative path deps to pkg/ or third_party/ packages
Packages which are not published
For packages in pkg/ which we do not intend to be published, we put the following comment in the pubspec.yaml file:
# This package is not intended for consumption on pub.dev. DO NOT publish.
publish_to: none
These pubspecs are still validated by the package validation tool. The contents are more informational as the pubspecs for these packages are not consumed by the pub tool or ecosystem.
We validate:
- that the dependencies listed in the pubspec are used in the package
- that all the packages used by the source are listed in the pubspec
- that a reference to a pkg/ package is done via a relative path dependency