Put local actions in a common pool, especially
- Assembler (many run for boringssl)
- Linker (high memory use)
- Dart (high memory use)
This should limit the number of concurrent local actions to the number of harts available, avoiding the racing actions triggering out-of-memory failures.
Change-Id: Ifa168012d73472a3db349acc36545dd176385bbc
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/510190
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
Extend --check-clean (enabled on bots) to verify build commands don't include absolute paths.
The non-RBE GCC and MSVC builds still have absolute paths. GCC doesn't have -ffile-compilation-dir so it uses -fdebug-prefix-map, which is not fixable. The MSVC build has absolute path in the toolchain wrappers, which might be fixable.
TEST=ci
Change-Id: I3b984aaab7aefa7ff527f0a039ca42281224a09d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/506505
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
On Mac and Linux, the makes the output of non-RBE, RBE local and RBE remote builds identical, and also independent of the build directories path. On Windows, non-RBE and RBE builds still disagree because paths are rewritten from \ to / to run on the Linux workers.
Bug: b/316893839
Change-Id: I5935785489c73445e4c6ca275020e6cf7464433d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/506281
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
This is a reland of commit 7ae1d75869
TEST=manual, no watchOS SDK on bots yet.
Original change's description:
> watchOS target os
>
> - add support for --os=watchos | --os=watchos_simulator
> - do not depend on perfetto if dart_support_perfetto is false
> - change the default value for dart_support_perfetto to !is_watchos
>
> To minimize changes and to avoid modifying third-party zlib BUILD file, is_watchos implies is_ios for now.
>
> TEST=manual, no watchOS SDK on bots yet.
>
> Cq-Include-Trybots: luci.dart.try:vm-mac-debug-arm64-try,vm-mac-release-arm64-try
> Change-Id: If2130068ef546162a07a9ba53f94b11ff25fb565
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/415021
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
> Commit-Queue: Ivan Inozemtsev <iinozemtsev@google.com>
Cq-Include-Trybots: luci.dart.try:vm-mac-debug-arm64-try,vm-mac-release-arm64-try
Change-Id: I2b50f7eb4da173627aeb4833b88a08712b278d9f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/425523
Commit-Queue: Ivan Inozemtsev <iinozemtsev@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
This reverts commit 7ae1d75869.
Reason for revert: breaks flutter build
Original change's description:
> watchOS target os
>
> - add support for --os=watchos | --os=watchos_simulator
> - do not depend on perfetto if dart_support_perfetto is false
> - change the default value for dart_support_perfetto to !is_watchos
>
> To minimize changes and to avoid modifying third-party zlib BUILD file, is_watchos implies is_ios for now.
>
> TEST=manual, no watchOS SDK on bots yet.
>
> Cq-Include-Trybots: luci.dart.try:vm-mac-debug-arm64-try,vm-mac-release-arm64-try
> Change-Id: If2130068ef546162a07a9ba53f94b11ff25fb565
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/415021
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
> Commit-Queue: Ivan Inozemtsev <iinozemtsev@google.com>
Cq-Include-Trybots: luci.dart.try:vm-mac-debug-arm64-try,vm-mac-release-arm64-try
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Change-Id: I4bede9ede04ac3c533b8ff7ae2f1adbd84eb2397
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/425262
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Alexander Thomas <athom@google.com>
Commit-Queue: Alexander Thomas <athom@google.com>
Auto-Submit: Ivan Inozemtsev <iinozemtsev@google.com>
- add support for --os=watchos | --os=watchos_simulator
- do not depend on perfetto if dart_support_perfetto is false
- change the default value for dart_support_perfetto to !is_watchos
To minimize changes and to avoid modifying third-party zlib BUILD file, is_watchos implies is_ios for now.
TEST=manual, no watchOS SDK on bots yet.
Cq-Include-Trybots: luci.dart.try:vm-mac-debug-arm64-try,vm-mac-release-arm64-try
Change-Id: If2130068ef546162a07a9ba53f94b11ff25fb565
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/415021
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Ivan Inozemtsev <iinozemtsev@google.com>
1. Added `toolchain_suite` template and switched toolchain definitions
to it, so now each toolchain definition defines a regular and
`_shared` toolchain, with different values for `is_shared_library`
argument.
2. Changed the logic of choosing between `fPIC` and `fPIE` based on
`is_shared_library` arg.
3. Changed `shared_library` template to define a group, depending on
the same target in a shared toolchain, and a copy of an output
artifact.
4. Made a copy of `runtime:dart_shared_lib` config and add it
automatically to all shared libraries.
5. Removed `runtime:dart_shared_lib` config and `DART_SHARED_LIB`
define from existing `shared_library` targets.
This CL should be a no-op refactoring to enable
https://dart-review.googlesource.com/c/sdk/+/394102.
TEST=ci
Change-Id: I6c044254d8e74b6b3ddadbf784e58d5f641fbcf3
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/392661
Commit-Queue: Ivan Inozemtsev <iinozemtsev@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
- `build/mac/find_sdk.py` can search for iPhone and Watch SDKs and their simulators
- `tools/build.py` supports `--os=ios` and `-os=ios_simulator` now. Treating simulator as a separate os to minimize changes and avoid an additional dimension for configs.
- `vm-mac-(release|debug)-arm64-try` tryjobs make sure a shared library builds successfully for ios.
TEST=ci
Change-Id: I76358ec8fd33752260bf0b8462da22a13cd7562e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/381623
Auto-Submit: Ivan Inozemtsev <iinozemtsev@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
This can be controlled via codesigning_identity GN arg.
For example, setting codesigning_identity="-"
would produce ad-hoc signed binaries.
This CL also includes changes in vm/cc tests which are needed
for tests to be green when running with hardened runtime.
Issue https://github.com/dart-lang/sdk/issues/53928
Tested: enabled ad-hoc signing and tested on bots.
Cq-Include-Trybots: luci.dart.try:vm-mac-debug-arm64-try,vm-mac-release-arm64-try,vm-mac-release-x64-try,vm-mac-debug-x64-try
Change-Id: I3c3a6265c62b2904e43a326b7d8223bcfd393577
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/333401
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
We were exporting all symbols to the dynamic table so that they could be looked up using `dladdr` for the profiler and backtracer. The symbols include our statically-linked libcxx, which can create trouble when another DSO has a different version of libcxx. Now we export only the VM embedding API functions (`Dart_*`) and use a specially produced table to do the symbolization.
TEST=runtime/tests/vm/dart/exported_symbols_test.dart
TEST=runtime/tests/vm/dart/symbolized_crash_test.dart
Bug: https://github.com/dart-lang/sdk/issues/53267
Change-Id: I2ee494fba86f67127ba0f6f402622f01a4662207
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/323702
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
To avoid these flags being applied to unexpected build rules, we
separate them out into a separate config that is then pulled in
explicitly in the `dart_precompiled_runtime{,_product}` executable
rules. Only those two executables need the additional empty section:
the product version because it becomes `dartaotruntime` in the SDK
bundle, and the non-product version because some of our tests build
standalone executables using it.
The linker flags were originally removed due to
https://github.com/flutter/flutter/issues/112687.
With the recent removal of bitcode support from Flutter (see
https://github.com/flutter/flutter/issues/107883), I can build
the `ios_release` target locally without any issue.
This also reverts the clang DEPS changes from CL 256208 (again).
TEST=pkg/dartdev/test/commands/compile_test
Bug: https://github.com/dart-lang/sdk/issues/49783
Cq-Include-Trybots: luci.dart.try:analyzer-mac-release-try,dart-sdk-mac-arm64-try,dart-sdk-mac-try,pkg-mac-release-arm64-try,pkg-mac-release-try,vm-kernel-precomp-mac-product-x64-try,vm-kernel-precomp-nnbd-mac-release-arm64-try
Change-Id: Ie46402ec2eeda23109247eb9d7a64935ec2052cb
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/262429
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
These linker flags are causing issues with the Dart->Flutter roll,
and so I'm removing them for now until we can figure out how to
reintroduce them later.
To avoid overwriting section contents, even at a much reduced
amount, I also reland the clang DEPS revert in CL 256208.
TEST=pkg/dartdev/test/commands/compile_test
Bug: https://github.com/flutter/flutter/issues/112687
Change-Id: I35d3ff6a2a0a46e2f589469fb47bca5f9ca6df1d
Cq-Include-Trybots: luci.dart.try:analyzer-mac-release-try,dart-sdk-mac-arm64-try,dart-sdk-mac-try,pkg-mac-release-arm64-try,pkg-mac-release-try,vm-kernel-precomp-mac-product-x64-try,vm-kernel-precomp-nnbd-mac-release-arm64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/262161
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
To create a Dart standalone executable on MacOS, we modify the
dartaotruntime executable to add the snapshot contents, and the
VM looks into the executable on disk to find the snapshot to load.
Previously, we did this by adding a new 64-bit segment load command
with a single section, where the section's file offset and size
describes the inserted snapshot. This meant the Mach-O header size increased by 152 bytes.
Originally, this wasn't an issue as there was plenty of padding, but
later clang updates removed most of this padding, and so writing the
new header actually overwrote the initial contents of the first section
in the file, which happens to be the __text section. In addition, since
the first section's offset was now declared to be within the header,
utilities that strictly validated the Mach-O format, like codesign,
would report errors.
This CL changes it so that we actually reserve space in the
dartaotruntime header using the -add_empty_section flag to the linker.
In addition, we change from using a segment load command to using a
(40 byte) note load command. This is because a segment load command
specifies that the contents should be loaded in memory, but we don't
use that loaded version. Instead, the VM reloads it from the executable
on disk so it can appropriately mmap the different parts of the
snapshot. A note section instead just declares a section of the
executable as arbitrary data that the owner can read from the file
and use as desired, which is semantically closer to our current usage.
This CL also adds a test to pkg/dartdev/test/commands/compile_test to
ensure that corrupting a random part of the snapshot in the executable
causes signature verification to fail.
This CL also reverts CL 256208, thus relanding the clang changes
starting from June that originally raised awareness of the issue by
greatly reduced the amount of padding after the load commands.
TEST=pkg/dartdev/test/commands/compile_test
Bug: https://github.com/dart-lang/sdk/issues/49783
Change-Id: Iee554d87b0eabaecd7a534ca4e4facfefbce6385
Cq-Include-Trybots: luci.dart.try:analyzer-mac-release-try,dart-sdk-mac-arm64-try,dart-sdk-mac-try,pkg-mac-release-arm64-try,pkg-mac-release-try,vm-kernel-precomp-mac-product-x64-try,vm-kernel-precomp-nnbd-mac-release-arm64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/260108
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>
Newer GN doesn't allow `*.framework` elements in `libs`. It
requires using `frameworks` instead. The toolchain definitions need
to pass the new substituted variable to the link to make use of the
new required way to express these dependencies.
Bug: fxbug.dev/56683
Change-Id: I62e68a8e5bdfef095b72b38d9d41f757e510012b
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/155482
Auto-Submit: Roland McGrath <mcgrathr@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
The flag "mac_use_goma_rbe" creates a symlink to the Xcode SDK
directory, under $root_build_dir/sdk/xcode_links.
e.g. the -isysroot arg in the build command becomes:
-isysroot sdk/xcode_links/MacOSX10.15.sdk
This is based on a similar solution for Chromium iOS builders:
http://crrev.com/c/1946174http://crrev.com/c/2102649
Bug: b/151697301
Change-Id: Ie65d3ae2b98da5977ad0f5a75f865c838bbbb963
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/139782
Commit-Queue: Simon Que <sque@google.com>
Reviewed-by: Alexander Thomas <athom@google.com>
This CL switches the ia32 and arm64 builds to use clang by default.
The arm build can't be switched at least until we roll clang to
pick up the fix for https://reviews.llvm.org/D34691.
This CL also changes to use the sysroots from wheezy (and jessie
for arm64) by default so that we can be building/testing locally with
the same setup as we are shipping in the SDK.
Change-Id: I09a1907ee7f78c3f4bb8d56ffdbbd6107ed054d9
Reviewed-on: https://dart-review.googlesource.com/25160
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Commit-Queue: Zach Anderson <zra@google.com>
Build spuriously worked after initial roll due to stale files hanging
around longer than they should have. This CL relands and updates
(hopefully) all the necessary paths.
R=rmacnak@google.com
Review-Url: https://codereview.chromium.org/3010023002 .
This has a few advantages:
- We can track dependencies more precisely
- ninja can assemble things in parallel as they're ready rather than
sequentially all at once.
- It is easier to customize SDKs depending on target platform, e.g.
Fuchsia.
This CL also has a number of cleanups:
- Defining is_fuchsia and is_fuchsia host so we don't always have to check
- Piping through toolchain overrides in more places
- Fixing bugs in copy_tree.py, not using list_files.py, which is broken on Windows
related #29558R=whesse@google.com
Review-Url: https://codereview.chromium.org/2848943003 .
Flutter uses the same clang toolchain as Fuchsia. This CL puts Dart
on that toolchain, as well. This roll should entail no changes to
glibc version requirements on Linux.
The Fuchsia buildtools distribute clang-format with the toolchain
rather than separately, so this CL introduces forwarding scripts
that are copied to the right place under //buildtools.
This CL sets the default for the number of workers in //tools/gn.py
to 1, as the newer GN version fails occasionally when run
concurrently with other invocations.
As Flutter does, this CL uses the clang toolchain to build for
Android. Thus, it is now possible to build for Android on MacOS.
This change is a prerequisite for assembling the SDK in GN
rather than with a python script:
https://codereview.chromium.org/2848943003/
and will be nicer for making prebuilt SDKs for Fuchsia.
Resubmitting this with the following fixes:
1. Rolls further forward to get past a GN bug
2. Fixes the Android build.
3. Style cleanups in gn.py
Review-Url: https://codereview.chromium.org/2858873005 .
This reverts commit 6bf2c800fd.
There is still some strangeness with the new GN binary not wanting
to do an incremental build on top of a build that used an old GN
binary.
Review-Url: https://codereview.chromium.org/2861903002 .
Flutter uses the same clang toolchain as Fuchsia. This CL puts Dart
on that toolchain, as well. This roll should entail no changes to
glibc version requirements on Linux.
The Fuchsia buildtools distribute clang-format with the toolchain
rather than separately, so this CL introduces forwarding scripts
that are copied to the right place under //buildtools.
This CL sets the default for the number of workers in //tools/gn.py
to 1, as the newer GN version fails occasionally when run
concurrently with other invocations.
As Flutter does, this CL uses the clang toolchain to build for
Android. Thus, it is now possible to build for Android on MacOS.
This change is a prerequisite for assembling the SDK in GN
rather than with a python script:
https://codereview.chromium.org/2848943003/
and will be nicer for making prebuilt SDKs for Fuchsia.
Review-Url: https://codereview.chromium.org/2855283002 .
This reverts commit cc968ca45b.
This CL will have to be relanded in two pieces so that the
update to tools/gn.py lands before the unmodified version
can invoke the updated GN binary.
Review-Url: https://codereview.chromium.org/2855973006 .
Flutter uses the same clang toolchain as Fuchsia. This CL puts Dart
on that toolchain, as well. This roll should entail no changes to
glibc version requirements on Linux.
The Fuchsia buildtools distribute clang-format with the toolchain
rather than separately, so this CL introduces forwarding scripts
that are copied to the right place under //buildtools.
This CL sets the default for the number of workers in //tools/gn.py
to 1, as the newer GN version fails occasionally when run
concurrently with other invocations.
As Flutter does, this CL uses the clang toolchain to build for
Android. Thus, it is now possible to build for Android on MacOS.
This change is a prerequisite for assembling the SDK in GN
rather than with a python script:
https://codereview.chromium.org/2848943003/
and will be nicer for making prebuilt SDKs for Fuchsia.
R=whesse@google.com
Review-Url: https://codereview.chromium.org/2854583002 .
This change enables gn.py to take comma separated lists of modes,
architectures, and os's similar to build.py. It also accepts 'all'
for each of these just like build.py. GN is invoked in parallel to
save time.
This change also adds an option --gn to build.py to invoke ninja
instead of the usual build system.
Will investigate invoking tools/gn.py automagically from build.py in
a future change.
R=johnmccutchan@google.com
Review URL: https://codereview.chromium.org/2392603005 .
Ryan Macnak