Also fix --check-clean. `ninja target -t commands` reports the commands for the default target, fix to use `ninja -t commands target` instead.
Change-Id: Ice179290e75b083a96abadb7540ca960eafcc982
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/510440
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Put local actions in a common pool, especially
- Assembler (many run for boringssl)
- Linker (high memory use)
- Dart (high memory use)
This should limit the number of concurrent local actions to the number of harts available, avoiding the racing actions triggering out-of-memory failures.
Change-Id: Ifa168012d73472a3db349acc36545dd176385bbc
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/510190
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
Extend --check-clean (enabled on bots) to verify build commands don't include absolute paths.
The non-RBE GCC and MSVC builds still have absolute paths. GCC doesn't have -ffile-compilation-dir so it uses -fdebug-prefix-map, which is not fixable. The MSVC build has absolute path in the toolchain wrappers, which might be fixable.
TEST=ci
Change-Id: I3b984aaab7aefa7ff527f0a039ca42281224a09d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/506505
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Switch build to use Clang. This somewhat degrades FFI testing, but reduces build time by roughly half or one hour.
Cq-Include-Trybots: luci.dart.try:pkg-win-release-arm64-try,vm-aot-win-debug-arm64-try,vm-aot-win-release-arm64-try,vm-win-debug-arm64-try,vm-win-release-arm64-try
Change-Id: Ie267942e99934312b228cfadcd6d5b8477f855c8
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/429421
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
This is a reland of commit b3253d9c8b
Fix the gn rules so the RBE offloading only happens in the Dart build
and the conflicting RBE configuration is not included otherwise. This
ensures this change can roll into Flutter while we design a better
solution to compose the projects' RBE usages.
Original change's description:
> [infra] RBE all the things.
>
> This change offloads all Dart actions during the build to RBE using
> the new rewrapper_dart.py script that understands all the different
> command line invocations of Dart programs and translates it into the
> appropriate rewrapper invocation, with a full list of input and output
> files and no absolute paths.
>
> The dart actions are all considered to be expensive RBE compilation
> steps that may be much slower on RBE than locally due to the low
> parallelism of the build during these stages and the comperatively
> slower bots in the RBE pools. The build uses the racing strategy by
> default for these compilations, such that cache hits are still used
> if available, and otherwise a local build is transparently used if it
> happens to be faster. The bots will use the remote strategy by default
> unlike developers, such that the rewrapper_dart.py script does not fail.
>
> rewrapper_dart.py contains a tiny dart import proprocessor written
> in python and a big argument parser that understands every command
> invoked during the build and the semantic meaning of every option.
>
> The absolute paths used during the Dart SDK build is not solved in
> this changelist, which just works around them initially, but these
> will be fixed in follow up changes with the appropriate teams now
> that this change proves they are not needed.
>
> Bug: b/333595242
> Change-Id: I36603ec1bf16f4ac87d56635cc1c98e8686a4028
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/335827
> Reviewed-by: William Hesse <whesse@google.com>
> Commit-Queue: Jonas Termansen <sortie@google.com>
Bug: b/333595242
Change-Id: I43bec2e7fcad375e972ca447aa13ae65b0ce09ae
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/362661
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Jonas Termansen <sortie@google.com>
This reverts commit b3253d9c8b.
Reason for revert: Flutter build is broken, it also has a GN variable rbi_dir, and dart_action.gni now imports two conflicting definitions.
Original change's description:
> [infra] RBE all the things.
>
> This change offloads all Dart actions during the build to RBE using
> the new rewrapper_dart.py script that understands all the different
> command line invocations of Dart programs and translates it into the
> appropriate rewrapper invocation, with a full list of input and output
> files and no absolute paths.
>
> The dart actions are all considered to be expensive RBE compilation
> steps that may be much slower on RBE than locally due to the low
> parallelism of the build during these stages and the comperatively
> slower bots in the RBE pools. The build uses the racing strategy by
> default for these compilations, such that cache hits are still used
> if available, and otherwise a local build is transparently used if it
> happens to be faster. The bots will use the remote strategy by default
> unlike developers, such that the rewrapper_dart.py script does not fail.
>
> rewrapper_dart.py contains a tiny dart import proprocessor written
> in python and a big argument parser that understands every command
> invoked during the build and the semantic meaning of every option.
>
> The absolute paths used during the Dart SDK build is not solved in
> this changelist, which just works around them initially, but these
> will be fixed in follow up changes with the appropriate teams now
> that this change proves they are not needed.
>
> Bug: b/333595242
> Change-Id: I36603ec1bf16f4ac87d56635cc1c98e8686a4028
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/335827
> Reviewed-by: William Hesse <whesse@google.com>
> Commit-Queue: Jonas Termansen <sortie@google.com>
Bug: b/333595242
Change-Id: I56c9df11e672e7bd096f2a6e7d0243dccd0396c9
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/362385
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: William Hesse <whesse@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
This change offloads all Dart actions during the build to RBE using
the new rewrapper_dart.py script that understands all the different
command line invocations of Dart programs and translates it into the
appropriate rewrapper invocation, with a full list of input and output
files and no absolute paths.
The dart actions are all considered to be expensive RBE compilation
steps that may be much slower on RBE than locally due to the low
parallelism of the build during these stages and the comperatively
slower bots in the RBE pools. The build uses the racing strategy by
default for these compilations, such that cache hits are still used
if available, and otherwise a local build is transparently used if it
happens to be faster. The bots will use the remote strategy by default
unlike developers, such that the rewrapper_dart.py script does not fail.
rewrapper_dart.py contains a tiny dart import proprocessor written
in python and a big argument parser that understands every command
invoked during the build and the semantic meaning of every option.
The absolute paths used during the Dart SDK build is not solved in
this changelist, which just works around them initially, but these
will be fixed in follow up changes with the appropriate teams now
that this change proves they are not needed.
Bug: b/333595242
Change-Id: I36603ec1bf16f4ac87d56635cc1c98e8686a4028
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/335827
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Jonas Termansen <sortie@google.com>
The build isn't going to go through anyway and the slow 3 minute default
timeout is just confusing and annoying when this happens. There isn't
any additional useful context being output like the comment suggested.
Bug: b/331346866
Change-Id: I0475002469c372fefe3a54d104b54692c89ca9f5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/359800
Commit-Queue: Jonas Termansen <sortie@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Automatically detect the appropriate configuration file if RBE=1 is
set to request RBE on all projects, or if DART_RBE=1 is set to request
RBE for Dart only, or otherwise respect the explicit configuration file.
Automatically set the server_address location to the build directory
if it has not already been set. This is unfortunately not supported on
Windows due to the environment variable being set during build but it
needed to be set during gn.
Retain the older configuration files during the transitory period.
Bug: b/296994239
Fixes: b/320876546
Change-Id: I62d1fbfed35248477731cceda3f7267c605c4969
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/355400
Commit-Queue: Jonas Termansen <sortie@google.com>
Reviewed-by: William Hesse <whesse@google.com>
Place the windows toolchain in the build root so it can be sent to RBE.
Use relative paths in debug symbols when building with Windows clang,
which is needed for remotely building with RBE.
Set the executable bit when invoking clang-cl remotely as it has not
been set on Windows RBE clients.
Stop including .rst files during build commands as @options-from-file
is not supported by rewrapper at this time.
Forward RBE_cfg into the msvc build environment and rewrite the
INCLUDE, LIB, and LIBPATH environment variables to use relative paths
instead of absolute paths.
Don't download reclient on windows-arm64 where there is no cipd
package and the windows-arm64 python lies and says it's x64 because
they actually packaged a x64 python.
Tune the Windows -j performance to match the historic bot performance
as most casual Windows machines won't handle -j 1000 well.
Bug: b/296994239
Change-Id: I2bd1b4478fca796f0a326c062a5f478f16c2c6ef
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/343440
Reviewed-by: William Hesse <whesse@google.com>
Commit-Queue: Jonas Termansen <sortie@google.com>
* Migrate to python3; drop python support.
* Update Windows toolchain support.
* Remove some unused methods.
* Python 2.7 is still needed on Windows.
* Update gsutil to a version that supports python3.
Fixes: https://github.com/dart-lang/sdk/issues/28793
TEST=Manually tested common user journeys.
Change-Id: I663a22b237a548bb82dc2e601e399e3bc3649211
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/192182
Reviewed-by: William Hesse <whesse@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
It is very tricky to make scripts using the python multiprocessing
package terminate cleanly on SIGINT (ctrl-c).
Use subprocesses run in parallel instead for the simple uses
in these scripts.
This is necessary when running build.py from within another script,
so that it terminates cleanly when the process group receives
a broadcast SIGINT.
Change-Id: Ifc858225a49f369c4bd9cee62639ac6fc8ff737e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/171120
Commit-Queue: William Hesse <whesse@google.com>
Reviewed-by: Jonas Termansen <sortie@google.com>
The workflow of using tools/gn.py followed-by tools/build.py is no
longer working as it used to (the latter will override whatever the
former configured).
Instead one should pick between:
tools/gn.py & ninja
tools/build.py
=> Both tools/gn.py and tools/build.py suport the same options.
This CL also makes tools/build.py stop using tools/generate_buildfiles.py (which
would get just overriden by the re-run of GN).
Change-Id: Ie698d7395e8c5862ae04a479c7c820c76ac5565d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/154323
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: William Hesse <whesse@google.com>
This reverts commit 4ce18ab931.
Reason for revert: bad merge, caused build breakages in flutter/engine
Original change's description:
> Add an --os=fuchsia option to build.py:
>
> tools/build.py --os=fuchsia runtime create_sdk
>
> This is analogous to --os=android. It cross compiles from Linux x64 to
> Fuchsia.
>
> A lot of the build rules are just slightly different between the existing
> Fuchsia build rules used by Flutter, and the ones added by GN SDK. For
> example "$fuchsia_sdk_root/pkg:fdio" is now "$fuchsia_sdk_root/pkg/fdio".
> So to support this I had to add a new variable, using_fuchsia_gn_sdk,
> analogous to using_fuchsia_sdk. Flutter will need to set this to false.
>
> Change-Id: Ief275d65f30a42a801607de93cf2d27a1fe825dd
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/150689
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
> Commit-Queue: Liam Appelbe <liama@google.com>
TBR=rmacnak@google.com,asiva@google.com,liama@google.com,kaushikiska@google.com,matthewcarroll@google.com
Change-Id: If01ee34eba906c55f2d56ba650748a86e81e701c
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/151321
Auto-Submit: George Wright <wrightgeorge@google.com>
Commit-Queue: Liam Appelbe <liama@google.com>
Reviewed-by: Liam Appelbe <liama@google.com>
tools/build.py --os=fuchsia runtime create_sdk
This is analogous to --os=android. It cross compiles from Linux x64 to
Fuchsia.
A lot of the build rules are just slightly different between the existing
Fuchsia build rules used by Flutter, and the ones added by GN SDK. For
example "$fuchsia_sdk_root/pkg:fdio" is now "$fuchsia_sdk_root/pkg/fdio".
So to support this I had to add a new variable, using_fuchsia_gn_sdk,
analogous to using_fuchsia_sdk. Flutter will need to set this to false.
Change-Id: Ief275d65f30a42a801607de93cf2d27a1fe825dd
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/150689
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Liam Appelbe <liama@google.com>
- Flips the flag from --nnbd to --no-nnbd so that by default it builds
the NNBD version
- using the --no-nnbd flag results in the SDK being built in a
directory which has the 'Legacy' suffix added to it
(e.g: out/DebugX64Legacy)
- the '--enable-experiment=non-nullable' flag still needs to be passed
in during execution so that CFE runs in that mode. This is different
from the 'null_safety' flag
Change-Id: I729630a7bd36ea7ae38134f9285b44e93c283c68
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/138902
Reviewed-by: Paul Berry <paulberry@google.com>
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Reviewed-by: Régis Crelier <regis@google.com>
This reverts commit cab69e7706.
Reason for revert: It was a temporary submit
Original change's description:
> [BUILD] - Initial CL to unfork the NNBD Dart SDK
> NOTE: This is a trial submit and will be reverted immediately.
>
> - Flips the flag from --nnbd to --no-nnbd so that by default it builds
> the NNBD version
> - using the --no-nnbd flag results in the SDK being built in a
> directory which has the 'Legacy' suffix added to it
> (e.g: out/DebugX64Legacy)
> - the '--enable-experiment=non-nullable' flag still needs to be passed
> in during execution so that CFE runs in that mode. This is different
> from the 'null_safety' flag
>
> Change-Id: I7d25d9710818af5919c0bb83abe51153172f5886
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/144880
> Reviewed-by: Siva Annamalai <asiva@google.com>
TBR=asiva@google.com
Change-Id: Ib0f99fc1a42c9c7a8b21f8542f4a35dba9105d5c
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/144900
Reviewed-by: Siva Annamalai <asiva@google.com>
Commit-Queue: Siva Annamalai <asiva@google.com>
NOTE: This is a trial submit and will be reverted immediately.
- Flips the flag from --nnbd to --no-nnbd so that by default it builds
the NNBD version
- using the --no-nnbd flag results in the SDK being built in a
directory which has the 'Legacy' suffix added to it
(e.g: out/DebugX64Legacy)
- the '--enable-experiment=non-nullable' flag still needs to be passed
in during execution so that CFE runs in that mode. This is different
from the 'null_safety' flag
Change-Id: I7d25d9710818af5919c0bb83abe51153172f5886
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/144880
Reviewed-by: Siva Annamalai <asiva@google.com>
This reverts commit 393468dc89.
Reason for revert: This was a trial submit and so reverting it.
Original change's description:
> This is a trial submit of the unfork CL, it will be reverted immediately.
> This commit is being done so we can roll it into Flutter and run some device lab testing to see the effect this CL will have on Flutter.
>
> [BUILD] - Initial CL to unfork the NNBD Dart SDK
>
> - Flips the flag from --nnbd to --no-nnbd so that by default it builds
> the NNBD version
> - using the --no-nnbd flag results in the SDK being built in a
> directory which has the 'Legacy' suffix added to it
> (e.g: out/DebugX64Legacy)
> - the '--enable-experiment=non-nullable' flag still needs to be passed
> in during execution so that CFE runs in that mode. This is different
> from the 'null_safety' flag
>
> Change-Id: I83c813aee2a5b61ad876d8a7b0988705bb24c839
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/144082
> Reviewed-by: Siva Annamalai <asiva@google.com>
TBR=asiva@google.com,liama@google.com
Change-Id: I7b50fae71764eceb17893338d1981e40ea2de520
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/144083
Reviewed-by: Siva Annamalai <asiva@google.com>
Commit-Queue: Siva Annamalai <asiva@google.com>
This commit is being done so we can roll it into Flutter and run some device lab testing to see the effect this CL will have on Flutter.
[BUILD] - Initial CL to unfork the NNBD Dart SDK
- Flips the flag from --nnbd to --no-nnbd so that by default it builds
the NNBD version
- using the --no-nnbd flag results in the SDK being built in a
directory which has the 'Legacy' suffix added to it
(e.g: out/DebugX64Legacy)
- the '--enable-experiment=non-nullable' flag still needs to be passed
in during execution so that CFE runs in that mode. This is different
from the 'null_safety' flag
Change-Id: I83c813aee2a5b61ad876d8a7b0988705bb24c839
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/144082
Reviewed-by: Siva Annamalai <asiva@google.com>
Running tools/test.py locally behaves differently than on the bots
because the bots will have special environment variables set.
=> We want also local runs of tools/test.py to report ASAN errors, so
this CL moves the setting of environment variables to tools/test.py
Furthermore we enable "additional check_initialization_order=true" asan
option to detect ordering issues when initializing global state.
Make the IsolateGroup::isolate_group_random_ heap allocated on VM
startup to avoid initialization ordering issue (turns out our `Random`
can depend on embedder entropy source callback being available)
Also add locking around assigning of isolate group ids -- without it we
could have two threads racing and possibly getting the same id.
See b/149978682
Change-Id: I41b9a34c66934dcc0d3804233f5acf09004ba3bd
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/136630
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
This treats whether or not to use the forked NNBD SDK as a build flag
similar to mode or architecture. When this flag is passed, it appends
"NNBD" to the main output directory, like xcodebuild/ReleaseX64NNBD.
It also defines a "use_nnbd" flag that is available inside BUILD.gn
files to determine whether or not to use the forked SDK. This flag
currently isn't used. Implementation teams will need to edit their
build steps to take that into account.
In order to use this mode on the bots, we'll need to tweak the bot
scripts to figure out whether the mode is enabled or not. There are a
couple of TODOs in the Python scripts for that. I'm not sure how that
should be specified in dart_sdk.py.
This CL does not remove my previous change to allow building the NNBD
fork next to dart-sdk in the same root output directory. I'll revert
those changes in a different CL once this one lands.
Change-Id: I3bd28f273106ee90caf9474b2fadad120b2c2d0b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/119602
Commit-Queue: Bob Nystrom <rnystrom@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
- Add `.style.yapf` with configuration to use Google style.
- Run `yapf` on all `.py` files in this repo.
- Manually fix one trailing space in a doc string.
- Run `git cl format runtime` to satisfy presubmit.
Change-Id: I7e6bd11e91f07926b9188362599af398551eed79
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/111600
Commit-Queue: Nate Bosch <nbosch@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
This is variant of SIMARM build (ARM target and non-ARM host) but with a
word size mismatch (32-bit target and 64-bit host). We only expect to
build gen_snapshot binary in this mode.
This is the first step towards enabling AOT compilation targeting ARM
using 64-bit gen_snapshot binary.
This change also introduces --gen-snapshot flag for test.py which allows
to specify which gen_snapshot binary should be used for running tests.
Expected workflow with SIMARM_X64 build:
$ tools/build.py -a simarm_x64 -m release gen_snapshot
$ tools/build.py -a simarm -m release dart_precompiled_runtime
vm_platform
$ tools/test.py -a simarm -m release -c dartkp --gen-snapshot
out/ReleaseSIMARM_X64/gen_snapshot
Note that our ARM simulator can't be built as a 64-binary so we are
going to be using SIMARM runtime to test AOT compiled code produced by
SIMARM_X64 binary.
Issue https://github.com/dart-lang/sdk/issues/36839
Issue https://github.com/flutter/flutter/issues/22598
Change-Id: Id003900e8b46fc9d57975ab82b0c21852a176079
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/100968
Commit-Queue: Vyacheslav Egorov <vegorov@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Right now most of the dart SDK's python is compatible with python2
or python3. This change fixes a few of the build scripts to make
that completely true (at least when building the standard build on
Linux). There are only four types of changes:
- Bare `print` statements now use the `print ()` function
- `commands.getoutput` becomes `subprocess.check_output` with `shell=True`
- `xrange` becomes `range`
- `print >> sys.stderr` becomes `sys.stderr.write`
Starts work on addressing (but does not completely fix):
https://github.com/dart-lang/sdk/issues/28793
See related issue:
https://fuchsia-review.googlesource.com/c/fuchsia/+/272925
This change applys to both the `dev` and `master` branches.
Change-Id: Ibd3eb9b1f57520d2d745f05c2ac430b1d20943da
Closes#36662https://github.com/dart-lang/sdk/pull/36662
GitOrigin-RevId: beab165294982a7e369daf6d61aea63efcab1b9b
Change-Id: I6d240749a9ba0889b5a45a08f3c4c2c20291f484
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/99707
Reviewed-by: Alexander Thomas <athom@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
Ryan Macnak