Jonas Termansen
69cd7cb43f
tools/bots/compare_results.dart compares the previous and current test
results in the results.json format and lists the differences, taking the
flakiness data into account.
tools/bots/update_flakiness.dart reads new result.json files and updates the
flakiness data in the flaky.json format file. The updated flakiness data
contains the list of tests that were already known to be flaky, plus any
new tests with multiple different outcomes in the provided results.json
files.
For instance, after running the tests, to find the list of tests that
changed result and needs to be deflaked, excluding tests that are already
known to be flaky:
compare_results.dart \
--flakiness-data flaky.json
--changed \
--passing \
--failing \
previous.json results.json
After the tests needing deflakinghas been run again, the flakiness data can
be updated:
update_flakiness.dart -i flaky.json -o flaky.json results.json more.json
Finally a human readable report can explain what happened, exiting 1 if any
tests started failing in a non-flaky manner:
compare_results.dart \
--flakiness-data flaky.json \
--judgement \
--human \
--verbose \
--changed \
--failing \
--flaky \
previous.json results.json
Bug: https://github.com/dart-lang/sdk/issues/34517
Bug: https://github.com/dart-lang/sdk/issues/34518
Change-Id: I156a8a49b8df09c0aebcb77376b69d365d0aa2ac
Reviewed-on: https://dart-review.googlesource.com/75540
Reviewed-by: William Hesse <whesse@google.com>
31 lines
1.0 KiB
Dart
31 lines
1.0 KiB
Dart
// Copyright (c) 2018, the Dart project authors. Please see the AUTHORS file
|
|
// for details. All rights reserved. Use of this source code is governed by a
|
|
// BSD-style license that can be found in the LICENSE file.
|
|
|
|
// results.json and flaky.json parses.
|
|
|
|
import 'dart:async';
|
|
import 'dart:convert';
|
|
import 'dart:io';
|
|
|
|
Future<List<Map<String, dynamic>>> loadResults(String path) async {
|
|
final results = <Map<String, dynamic>>[];
|
|
final lines = new File(path)
|
|
.openRead()
|
|
.transform(utf8.decoder)
|
|
.transform(new LineSplitter());
|
|
await for (final line in lines) {
|
|
final Map<String, dynamic> map = jsonDecode(line);
|
|
results.add(map);
|
|
}
|
|
return results;
|
|
}
|
|
|
|
Map<String, Map<String, dynamic>> createResultsMap(
|
|
List<Map<String, dynamic>> results) =>
|
|
new Map<String, Map<String, dynamic>>.fromIterable(results,
|
|
key: (dynamic result) => (result as Map<String, dynamic>)['name']);
|
|
|
|
Future<Map<String, Map<String, dynamic>>> loadResultsMap(String path) async =>
|
|
createResultsMap(await loadResults(path));
|