Commit Graph

40 Commits

Author SHA1 Message Date
Alexander Markov a37d042e68 [vm] Initial implementation of module snapshot reader
TEST=manual
Issue: https://github.com/dart-lang/sdk/issues/61635

Change-Id: I27f712d9687d45a09173b1018bc08812dd18658e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/460580
Reviewed-by: Slava Egorov <vegorov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Alexander Markov <alexmarkov@google.com>
2025-11-21 06:42:16 -08:00
Daco Harkes 592e5c144a [native assets] Support PATHEXT invocations with appended snapshot
This CL introduces an `IsolateData::asset_base_path` which contains
the fully resolved base path used for assets which are loaded with a
relative path. (Resolving both PATHEXT and symlinks.)

The asset base path is now only resolved once, instead of on every
relative asset load.

The asset base path is now eagerly resolved.
To prevent existing error messages for files not existing changing in
`dartdev` and `Isolate.spawnUri`, files not existing errors are
ignored.
To prevent `tests/standalone/io/named_pipe_script_test.dart` from
failing, failing to canonicalize a file path is treated as no asset
base path rather than reporting a failure.

TEST=pkg/dartdev/test/native_assets/*.dart

Bug: https://github.com/dart-lang/sdk/issues/60946
Change-Id: I79a9144d220dfb3e54311725d41f8f91c88db3a1
Cq-Include-Trybots: luci.dart.try:pkg-linux-debug-try,pkg-linux-release-arm64-try,pkg-linux-release-try,pkg-mac-release-try,pkg-mac-release-arm64-try,pkg-win-release-arm64-try,pkg-win-release-try,vm-aot-dyn-linux-debug-x64-try,vm-aot-android-release-arm64c-try,vm-aot-linux-debug-x64-try,vm-aot-mac-debug-arm64-try,vm-aot-win-debug-arm64-try,vm-aot-optimization-level-linux-release-x64-try,vm-aot-obfuscate-linux-release-x64-try,vm-msan-linux-release-x64-try,vm-asan-linux-release-x64-try,vm-linux-release-x64-try,vm-linux-debug-x64-try
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/435421
Commit-Queue: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
2025-06-26 07:47:32 -07:00
Ryan Macnak b68351fbc3 [vm] Update NULL to nullptr in runtime/bin.
TEST=build
Change-Id: Ie3be570c274b0275a995a0f54b5e6ccdfc77ccd3
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/292287
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Brian Quinlan <bquinlan@google.com>
2023-04-12 01:11:05 +00:00
Ryan Macnak c5a94db091 [vm] Initial implementation of deferred loading.
(Assignment of libraries to loading units is already done in the kernel generation step.)

After compiling and before serializing, we walk the program and for each Code we assign its Instructions, CodeSourceMap and CompressedStackMap to the loading unit of that Code's defining library. Deduplication may cause Instructions, CodeSourceMaps and CompressedStackMaps to belong to more than one loading unit; in this case the objects are assigned to the root loading unit. Later they can be more precisely assigned to the dominating loading unit.

All objects except some Instructions, CodeSourceMaps and CompressedStackMaps belong to the root loading unit's snapshot. This snapshot is written like an unsplit snapshot, except that when serializing Code, we will write a reference to a stub or null when the Code's Instructions, CodeSourceMap or CompressedStackMap belongs to a non-root loading unit. The snapshots of non-root loading units contain these deferred objects and references to the corresponding Code objects to patch. The types of objects we defer (Instructions, CodeSourceMaps and CompressedStackMaps) usually represent 70+% of the snapshot size.

Bare instructions mode must be disabled when splitting because we cannot have PC-relative calls between loading units. Later we can re-enable this for calls within loading units.

Broken: Compactor probably crashes we can now have an unbounded number of image pages and the compactor assumes a fixed number.

Embedder's guide:

At compile-time, gen_snapshot should be passed --loading_unit_manifest with a path, which will enable splitting and output a mapping from loading unit ids to snapshot output paths.

At runtime, sometime during isolate startup, an embedder should call Dart_SetDeferredLoadHandler, probably near an existing call to Dart_SetLibraryTagHandler. The callback is given a loading unit id, and should eventually call Dart_DeferredLoadComplete[Error].

Bug: https://github.com/dart-lang/sdk/issues/41974
Change-Id: Ib597eb87c8cd634416d5ee1f00629c5550aebb00
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/152427
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Markov <alexmarkov@google.com>
2020-07-17 19:29:47 +00:00
Alexander Aprelev d33e51df3c [build] Support --clang when building dart sdk on Windows.
It relies on flutter copy of clang distribution, same one that is used to build flutter/engine.

It addressed several deprecated warnings from clang compiler for functions like strdup, unlink, etc.

It allows few warnings still since they are triggered in third_party sources.

Change-Id: Ieb13792c011438d46dbbc0fa030e1b5e4ea14315
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/142704
Commit-Queue: Alexander Aprelev <aam@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
2020-06-25 03:36:41 +00:00
Siva Annamalai d46a696957 Revert "Reland : [VM/Runtime] Cleanup package config initialization code"
This reverts commit 03777dba31.

Reason for revert: Failures in couple of bots

Original change's description:
> Reland : [VM/Runtime] Cleanup package config initialization code
> 
> - Use _Init function for initialization of package config
> - Remove DartUtils::SetWorkingDirectory, SingleArgDart_Invoke,
> 
>   DartUtils::SetupPackageConfig
> - Rename Loader::InitForSnapshot to Loader::Init
> - Remove _setPackagesConfig, _setWorkingDirectory from builtin library
> - Rename _setPackagesMap to _setPackagesConfig
> Change-Id: I1e4a25e79706d7764247c9fbfcc8e8ce3231343c
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/151633
> Reviewed-by: Alexander Aprelev <aam@google.com>
> Commit-Queue: Siva Annamalai <asiva@google.com>

TBR=aam@google.com,asiva@google.com

Change-Id: Ibea8c35fe310ac7708f00bb9ffd58715b6924cb5
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/151869
Reviewed-by: Siva Annamalai <asiva@google.com>
Auto-Submit: Siva Annamalai <asiva@google.com>
Commit-Queue: Siva Annamalai <asiva@google.com>
2020-06-20 08:02:53 +00:00
asiva 03777dba31 Reland : [VM/Runtime] Cleanup package config initialization code
- Use _Init function for initialization of package config
- Remove DartUtils::SetWorkingDirectory, SingleArgDart_Invoke,

  DartUtils::SetupPackageConfig
- Rename Loader::InitForSnapshot to Loader::Init
- Remove _setPackagesConfig, _setWorkingDirectory from builtin library
- Rename _setPackagesMap to _setPackagesConfig
Change-Id: I1e4a25e79706d7764247c9fbfcc8e8ce3231343c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/151633
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Siva Annamalai <asiva@google.com>
2020-06-20 05:50:49 +00:00
asiva 4b9aa2bd7e Revert "[VM/Runtime] Cleanup package config initialization code"
This reverts commit 29e93bcdbd.

Reason for revert: Windows bots are failing with package resolution
errors. Looks like we are missing a windows path sanitization step.

Change-Id: Ib56f7e926b4f385fa3fde74c9c71947f64b8be1c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/151469
Reviewed-by: Siva Annamalai <asiva@google.com>
2020-06-17 00:03:19 +00:00
asiva 29e93bcdbd [VM/Runtime] Cleanup package config initialization code
- Use _Init function for initialization of package config
- Remove DartUtils::SetWorkingDirectory, SingleArgDart_Invoke,
  DartUtils::SetupPackageConfig
- Rename Loader::InitForSnapshot to Loader::Init
- Remove _setPackagesConfig, _setWorkingDirectory from builtin library
- Rename _setPackagesMap to _setPackagesConfig
Change-Id: I0b83574593fa95e1b948134fcd4a7b4858ea19c2
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/151045
Commit-Queue: Siva Annamalai <asiva@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
2020-06-16 23:24:26 +00:00
Martin Kustermann a94b8ae45d Reland "[vm] Make VM support .dart_tool/package_config.json"
This relands the following CLs

    [vm] Make VM support .dart_tool/package_config.json
    https://dart-review.googlesource.com/c/sdk/+/145462

    [vm] Make Isolate.resolveUri() work for both `.packages` and `dart_tool/package_config.json`
    https://dart-review.googlesource.com/c/sdk/+/145803

    [vm] Skip (newly enabled) isolate tests on simarm* and hot-reload configs
    https://dart-review.googlesource.com/c/sdk/+/145861

    [vm] Remove ../0 postfix from status file entries (test framework does not recognize those multitests)
    https://dart-review.googlesource.com/c/sdk/+/145940

    [vm] - Fix hang when no package config or .packages exist
    https://dart-review.googlesource.com/c/sdk/+/145661

The only change (extra patchset) is to let VM prefer the old/legacy
".packages" format and only use the newer
".dart_tool/package_config.json" if we failed to find a ".packages".

Closes https://github.com/dart-lang/sdk/issues/41739
Closes https://github.com/dart-lang/sdk/issues/41649

Change-Id: I220f808540e6e521d985ae763968f64dbf57fe67
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/145942
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
2020-05-04 14:51:14 +00:00
asiva 170d047b20 The rolls of Dart into Flutter have been failing for more than a day now.
We have accumulated almost 69 commits. Please see issue
https://github.com/dart-lang/sdk/issues/41739
for details of why these CLs had to be reverted.

Revert "Reland "[vm] Make VM support .dart_tool/package_config.json""
This reverts commit fa6cc57a32.

Revert "[VM] - Fix hang when no package config or .packages exist"
This reverts commit 31be795457.

Revert "[vm/isolate] Make Isolate.resolveUri() work for both `.packages` and `dart_tool/package_config.json`"
This reverts commit a99f7aef6e.

Revert "[vm] Skip (newly enabled) isolate tests on simarm* and hot-reload configs"
This reverts commit 7082e99c28.

Revert "[vm] Remove ../0 postfix from status file entries (test framework does not recognize those multitests)"
This reverts commit 708468684a.

Change-Id: Id6b951bde4f379fdcfac3cc23e0057a77a7e6db0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/146041
Reviewed-by: Ben Konyi <bkonyi@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Siva Annamalai <asiva@google.com>
2020-05-01 22:11:55 +00:00
Martin Kustermann fa6cc57a32 Reland "[vm] Make VM support .dart_tool/package_config.json"
This CL makes the VM add support for .dart_tool/package_config.json via
  * the --packages=<file> command line parameter
  * the `packageConfig: <file>` parameter to `Isolate.spawnUri`

It also allows now `package:*` uris being used in `Isolate.spawnUri`

The CL also removes the unused `packageRoot` parameter from the Dart to
C++ interface as well as in various places in C++.

The CL also aligns the implementations between sdk and sdk_nnbd more.

Issue https://github.com/dart-lang/sdk/issues/41649
Issue https://github.com/dart-lang/sdk/issues/41245
Issue https://github.com/dart-lang/sdk/issues/41246

Change-Id: I0a7aa3040332abafa19bf80bdbd8a8f8594cc6fe
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/145462
Reviewed-by: Alexander Aprelev <aam@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
2020-04-29 19:05:26 +00:00
Alexander Aprelev 334284a69b Revert "Reland "[vm] Make VM support .dart_tool/package_config.json""
This reverts commit 30085612f4 as it
broke vm-kernel-precomp-nnbd-linux-release-x64 bot.

Seems to be the same failure as the last time it was reverted: https://ci.chromium.org/p/dart/builders/ci.sandbox/vm-kernel-precomp-nnbd-linux-release-x64/537

Change-Id: I35a1897367bcb65f518b58659001280c5d5ed930
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/145262
Reviewed-by: Alexander Aprelev <aam@google.com>
2020-04-28 23:23:44 +00:00
Martin Kustermann 30085612f4 Reland "[vm] Make VM support .dart_tool/package_config.json"
This CL makes the VM add support for .dart_tool/package_config.json via
  * the --packages=<file> command line parameter
  * the `packageConfig: <file>` parameter to `Isolate.spawnUri`

It also allows now `package:*` uris being used in `Isolate.spawnUri`

The CL also removes the unused `packageRoot` parameter from the Dart to
C++ interface as well as in various places in C++.

The CL also aligns the implementations between sdk and sdk_nnbd more.

Issue https://github.com/dart-lang/sdk/issues/41649
Issue https://github.com/dart-lang/sdk/issues/41245
Issue https://github.com/dart-lang/sdk/issues/41246

Change-Id: Idec0990c731f34efda9c39672067eae9174319f1
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/144826
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
2020-04-28 16:52:02 +00:00
Ryan Macnak 3cb7eaa4cb Revert "[vm] Fix test to use Uri.file() instead of Uri.parse()"
This reverts commit 87ce4a504f.

Revert "[vm] Make VM support .dart_tool/package_config.json"

This reverts commit b112aa0d8b.

Reason for revert: Failures on Windows

Change-Id: I3e4f601d7638b25df313773e525c9284737fa729
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/144985
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
2020-04-27 21:55:46 +00:00
Martin Kustermann b112aa0d8b [vm] Make VM support .dart_tool/package_config.json
This CL makes the VM add support for .dart_tool/package_config.json via
  * the --packages=<file> command line parameter
  * the `packageConfig: <file>` parameter to `Isolate.spawnUri`

It also allows now `package:*` uris being used in `Isolate.spawnUri`

The CL also removes the unused `packageRoot` parameter from the Dart to
C++ interface as well as in various places in C++.

The CL also aligns the implementations between sdk and sdk_nnbd more.

Issue https://github.com/dart-lang/sdk/issues/41649
Issue https://github.com/dart-lang/sdk/issues/41245
Issue https://github.com/dart-lang/sdk/issues/41246

Change-Id: I642dfb89f82b3f2c1613e443850bf0b4fb91f4a1
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/144766
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
2020-04-27 12:54:38 +00:00
asiva d0139e8e38 [VM] More cleanup in the loader code
- Removed ResolveDependenciesAsFilePaths functionality as the loader
  does not track any dependencies (this is now provided by the kernel
  isolate)

- Removed ResolveAsFilePath functionality from the loader class

- Removed some left over code related to deferred loading

Change-Id: Ic1b1a79b07020bb7082e9efb7544b73931290faf
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/122220
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Siva Annamalai <asiva@google.com>
2019-10-21 17:44:18 +00:00
Alexander Aprelev 0425997b31 Second attempt to reland "[vm/concurrency] Introduce concept of Isolate Groups"
This reverts commit 3d14b75f97.

An Isolate Group (IG) is a collection of isolates which were spawned from the
same source. This allows the VM to:

  * have a guarantee that all isolates within one IG can safely exchange
    structured objects (currently we rely on embedder for this
    guarantee)

  * hot-reload all isolates together (currently we only reload one
    isolate, leaving same-source isolates in inconsistent state)

  * make a shared heap for all isolates from the same IG, which paves
    the way for faster communication and sharing of immutable objects.

All isolates within one IG will share the same IsolateGroupSource.

**Embedder changes**

This change makes breaking embedder API changes to support this new
concept of Isolate Groups: The existing isolate lifecycle callbacks
given to Dart_Initialize will become Isolate Group lifecycle callbacks.
A new callback `initialize_isolate` callback will be added which can
initialize a new isolate within an existing IG.

Existing embedders can be updated by performing the following renames

  Dart_CreateIsolate -> Dart_CreateIsolateGroup
  Dart_IsolateCreateCallback -> Dart_IsolateGroupCreateCallback
  Dart_IsolateCleanupCallback -> Dart_IsolateGroupShutdownCallback
  Dart_CreateIsolateFromKernel -> Dart_CreateIsolateGroupFromKernel
  Dart_CurrentIsolateData -> Dart_CurrentIsolateGroupData
  Dart_IsolateData -> Dart_IsolateGroupData
  Dart_GetNativeIsolateData -> Dart_GetNativeIsolateGroupData
  Dart_InitializeParams.create -> Dart_InitializeParams.create_group
  Dart_InitializeParams.cleanup -> Dart_InitializeParams.shutdown_group
  Dart_InitializeParams.shutdown -> Dart_InitializeParams.shutdown_isolate

By default `Isolate.spawn` will cause the creation of a new IG.

Though an embedder can opt-into supporting multiple isolates within one IG by
providing a callback to the newly added `Dart_InitializeParams.initialize_isolate`.
The responsibility of this new callback is to initialize an existing
isolate (which was setup by re-using source code from the spawning
isolate - i.e. the one which used `Isolate.spawn`) by setting native
resolvers, initializing global state, etc.

Issue https://github.com/dart-lang/sdk/issues/36648
Issue https://github.com/dart-lang/sdk/issues/36097

Original review: https://dart-review.googlesource.com/c/sdk/+/105241

Difference to original review:

  * Give each isolate it's own [Loader] (for now)
  * Sort classes during initialization for spawned isolates if app-jit is used (to match main isolate)
  * Fix IsolateData memory leak if isolate startup fails

Difference to first reland(Patchset 2):

  * Fix typo where memory was freed twice.

Change-Id: Ib1c83fe83b629cd50ae6af90ee99fdd44da882d4
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/108367
Commit-Queue: Alexander Aprelev <aam@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
2019-07-08 23:49:05 +00:00
Vyacheslav Egorov 3d14b75f97 Revert "Reland "[vm/concurrency] Introduce concept of Isolate Groups""
This reverts commit 67ab3be10d.

Reason for revert: Causes non-deterministic failures on front-end bots on Windows with "Isolate creation failed" error. See https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket.appspot.com/8909292129328681248/+/steps/unit_tests/0/stdout

Original change's description:
> Reland "[vm/concurrency] Introduce concept of Isolate Groups"
> 
> An Isolate Group (IG) is a collection of isolates which were spawned from the
> same source. This allows the VM to:
> 
>   * have a guarantee that all isolates within one IG can safely exchange
>     structured objects (currently we rely on embedder for this
>     guarantee)
> 
>   * hot-reload all isolates together (currently we only reload one
>     isolate, leaving same-source isolates in inconsistent state)
> 
>   * make a shared heap for all isolates from the same IG, which paves
>     the way for faster communication and sharing of immutable objects.
> 
> All isolates within one IG will share the same IsolateGroupSource.
> 
> **Embedder changes**
> 
> This change makes breaking embedder API changes to support this new
> concept of Isolate Groups: The existing isolate lifecycle callbacks
> given to Dart_Initialize will become Isolate Group lifecycle callbacks.
> A new callback `initialize_isolate` callback will be added which can
> initialize a new isolate within an existing IG.
> 
> Existing embedders can be updated by performing the following renames
> 
>   Dart_CreateIsolate -> Dart_CreateIsolateGroup
>   Dart_IsolateCreateCallback -> Dart_IsolateGroupCreateCallback
>   Dart_IsolateCleanupCallback -> Dart_IsolateGroupShutdownCallback
>   Dart_CreateIsolateFromKernel -> Dart_CreateIsolateGroupFromKernel
>   Dart_CurrentIsolateData -> Dart_CurrentIsolateGroupData
>   Dart_IsolateData -> Dart_IsolateGroupData
>   Dart_GetNativeIsolateData -> Dart_GetNativeIsolateGroupData
>   Dart_InitializeParams.create -> Dart_InitializeParams.create_group
>   Dart_InitializeParams.cleanup -> Dart_InitializeParams.shutdown_group
>   Dart_InitializeParams.shutdown -> Dart_InitializeParams.shutdown_isolate
> 
> By default `Isolate.spawn` will cause the creation of a new IG.
> 
> Though an embedder can opt-into supporting multiple isolates within one IG by
> providing a callback to the newly added `Dart_InitializeParams.initialize_isolate`.
> The responsibility of this new callback is to initialize an existing
> isolate (which was setup by re-using source code from the spawning
> isolate - i.e. the one which used `Isolate.spawn`) by setting native
> resolvers, initializing global state, etc.
> 
> Issue https://github.com/dart-lang/sdk/issues/36648
> Issue https://github.com/dart-lang/sdk/issues/36097
> 
> Original review: https://dart-review.googlesource.com/c/sdk/+/105241
> 
> Difference to original review:
> 
>   * Give each isolate it's own [Loader] (for now)
>   * Sort classes during initialization for spawned isolates if app-jit is used (to match main isolate)
>   * Fix IsolateData memory leak if isolate startup fails
> 
> Change-Id: I98277d3d10fe275aa9b8a16b6bdd446bbea0b100
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/107506
> Commit-Queue: Martin Kustermann <kustermann@google.com>
> Reviewed-by: Ryan Macnak <rmacnak@google.com>

TBR=kustermann@google.com,aam@google.com,rmacnak@google.com

# Not skipping CQ checks because original CL landed > 1 day ago.

Change-Id: Ia4e0f4f9fc317499d3570a371c5bdf9aed799e77
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/108101
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Commit-Queue: Vyacheslav Egorov <vegorov@google.com>
2019-07-03 12:19:15 +00:00
Martin Kustermann 67ab3be10d Reland "[vm/concurrency] Introduce concept of Isolate Groups"
An Isolate Group (IG) is a collection of isolates which were spawned from the
same source. This allows the VM to:

  * have a guarantee that all isolates within one IG can safely exchange
    structured objects (currently we rely on embedder for this
    guarantee)

  * hot-reload all isolates together (currently we only reload one
    isolate, leaving same-source isolates in inconsistent state)

  * make a shared heap for all isolates from the same IG, which paves
    the way for faster communication and sharing of immutable objects.

All isolates within one IG will share the same IsolateGroupSource.

**Embedder changes**

This change makes breaking embedder API changes to support this new
concept of Isolate Groups: The existing isolate lifecycle callbacks
given to Dart_Initialize will become Isolate Group lifecycle callbacks.
A new callback `initialize_isolate` callback will be added which can
initialize a new isolate within an existing IG.

Existing embedders can be updated by performing the following renames

  Dart_CreateIsolate -> Dart_CreateIsolateGroup
  Dart_IsolateCreateCallback -> Dart_IsolateGroupCreateCallback
  Dart_IsolateCleanupCallback -> Dart_IsolateGroupShutdownCallback
  Dart_CreateIsolateFromKernel -> Dart_CreateIsolateGroupFromKernel
  Dart_CurrentIsolateData -> Dart_CurrentIsolateGroupData
  Dart_IsolateData -> Dart_IsolateGroupData
  Dart_GetNativeIsolateData -> Dart_GetNativeIsolateGroupData
  Dart_InitializeParams.create -> Dart_InitializeParams.create_group
  Dart_InitializeParams.cleanup -> Dart_InitializeParams.shutdown_group
  Dart_InitializeParams.shutdown -> Dart_InitializeParams.shutdown_isolate

By default `Isolate.spawn` will cause the creation of a new IG.

Though an embedder can opt-into supporting multiple isolates within one IG by
providing a callback to the newly added `Dart_InitializeParams.initialize_isolate`.
The responsibility of this new callback is to initialize an existing
isolate (which was setup by re-using source code from the spawning
isolate - i.e. the one which used `Isolate.spawn`) by setting native
resolvers, initializing global state, etc.

Issue https://github.com/dart-lang/sdk/issues/36648
Issue https://github.com/dart-lang/sdk/issues/36097

Original review: https://dart-review.googlesource.com/c/sdk/+/105241

Difference to original review:

  * Give each isolate it's own [Loader] (for now)
  * Sort classes during initialization for spawned isolates if app-jit is used (to match main isolate)
  * Fix IsolateData memory leak if isolate startup fails

Change-Id: I98277d3d10fe275aa9b8a16b6bdd446bbea0b100
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/107506
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
2019-06-29 18:45:11 +00:00
Martin Kustermann b2171f16f1 Revert "[vm/concurrency] Introduce concept of Isolate Groups"
This reverts commit b75057b687.

Reason for revert: Caused some failures on app-jit builders

Original change's description:
> [vm/concurrency] Introduce concept of Isolate Groups
> 
> An Isolate Group (IG) is a collection of isolates which were spawned from the
> same source. This allows the VM to:
> 
>   * have a guarantee that all isolates within one IG can safely exchange
>     structured objects (currently we rely on embedder for this
>     guarantee)
> 
>   * hot-reload all isolates together (currently we only reload one
>     isolate, leaving same-source isolates in inconsistent state)
> 
>   * make a shared heap for all isolates from the same IG, which paves
>     the way for faster communication and sharing of immutable objects.
> 
> All isolates within one IG will share the same IsolateGroupSource.
> 
> **Embedder changes**
> 
> This change makes breaking embedder API changes to support this new
> concept of Isolate Groups: The existing isolate lifecycle callbacks
> given to Dart_Initialize will become Isolate Group lifecycle callbacks.
> A new callback `initialize_isolate` callback will be added which can
> initialize a new isolate within an existing IG.
> 
> Existing embedders can be updated by performing the following renames
> 
>   Dart_CreateIsolate -> Dart_CreateIsolateGroup
>   Dart_IsolateCreateCallback -> Dart_IsolateGroupCreateCallback
>   Dart_IsolateCleanupCallback -> Dart_IsolateGroupShutdownCallback
>   Dart_CreateIsolateFromKernel -> Dart_CreateIsolateGroupFromKernel
>   Dart_CurrentIsolateData -> Dart_CurrentIsolateGroupData
>   Dart_IsolateData -> Dart_IsolateGroupData
>   Dart_GetNativeIsolateData -> Dart_GetNativeIsolateGroupData
>   Dart_InitializeParams.create -> Dart_InitializeParams.create_group
>   Dart_InitializeParams.cleanup -> Dart_InitializeParams.shutdown_group
>   Dart_InitializeParams.shutdown -> Dart_InitializeParams.shutdown_isolate
> 
> By default `Isolate.spawn` will cause the creation of a new IG.
> 
> Though an embedder can opt-into supporting multiple isolates within one IG by
> providing a callback to the newly added `Dart_InitializeParams.initialize_isolate`.
> The responsibility of this new callback is to initialize an existing
> isolate (which was setup by re-using source code from the spawning
> isolate - i.e. the one which used `Isolate.spawn`) by setting native
> resolvers, initializing global state, etc.
> 
> Issue https://github.com/dart-lang/sdk/issues/36648
> Issue https://github.com/dart-lang/sdk/issues/36097
> 
> Change-Id: I82437ac017ca33018d45e02f353b0672db155f6a
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/105241
> Commit-Queue: Martin Kustermann <kustermann@google.com>
> Reviewed-by: Ryan Macnak <rmacnak@google.com>
> Reviewed-by: Alexander Aprelev <aam@google.com>

TBR=kustermann@google.com,aam@google.com,rmacnak@google.com

Change-Id: Ibd90992a01d61188f27b445c21532e0c46f996ea
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/107405
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
2019-06-26 16:04:27 +00:00
Martin Kustermann b75057b687 [vm/concurrency] Introduce concept of Isolate Groups
An Isolate Group (IG) is a collection of isolates which were spawned from the
same source. This allows the VM to:

  * have a guarantee that all isolates within one IG can safely exchange
    structured objects (currently we rely on embedder for this
    guarantee)

  * hot-reload all isolates together (currently we only reload one
    isolate, leaving same-source isolates in inconsistent state)

  * make a shared heap for all isolates from the same IG, which paves
    the way for faster communication and sharing of immutable objects.

All isolates within one IG will share the same IsolateGroupSource.

**Embedder changes**

This change makes breaking embedder API changes to support this new
concept of Isolate Groups: The existing isolate lifecycle callbacks
given to Dart_Initialize will become Isolate Group lifecycle callbacks.
A new callback `initialize_isolate` callback will be added which can
initialize a new isolate within an existing IG.

Existing embedders can be updated by performing the following renames

  Dart_CreateIsolate -> Dart_CreateIsolateGroup
  Dart_IsolateCreateCallback -> Dart_IsolateGroupCreateCallback
  Dart_IsolateCleanupCallback -> Dart_IsolateGroupShutdownCallback
  Dart_CreateIsolateFromKernel -> Dart_CreateIsolateGroupFromKernel
  Dart_CurrentIsolateData -> Dart_CurrentIsolateGroupData
  Dart_IsolateData -> Dart_IsolateGroupData
  Dart_GetNativeIsolateData -> Dart_GetNativeIsolateGroupData
  Dart_InitializeParams.create -> Dart_InitializeParams.create_group
  Dart_InitializeParams.cleanup -> Dart_InitializeParams.shutdown_group
  Dart_InitializeParams.shutdown -> Dart_InitializeParams.shutdown_isolate

By default `Isolate.spawn` will cause the creation of a new IG.

Though an embedder can opt-into supporting multiple isolates within one IG by
providing a callback to the newly added `Dart_InitializeParams.initialize_isolate`.
The responsibility of this new callback is to initialize an existing
isolate (which was setup by re-using source code from the spawning
isolate - i.e. the one which used `Isolate.spawn`) by setting native
resolvers, initializing global state, etc.

Issue https://github.com/dart-lang/sdk/issues/36648
Issue https://github.com/dart-lang/sdk/issues/36097

Change-Id: I82437ac017ca33018d45e02f353b0672db155f6a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/105241
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
2019-06-26 11:34:48 +00:00
Alexander Aprelev 7d14785115 This relands "Load isolate from parent's kernel in Isolate.spawn calls.".
Original PR was reverted because it broke hot-reload bots. This CL adds steps to initialize kernel_service compiler for child isolate. Compiler is needed to support hot-reload request. Child isolate's compiler is created from full kernel file produced by main isolate's compiler. Changes since original PR are pkg/vm/bin/kernel_service.dart in pkg/vm/lib/incremental_compiler.dart.

Further this CL changes kernel fingerprint calculation for interface types so it calculates the hash of the canonical names themselves, rather than indices(that might change from one compilation to another).

This reverts commit 63fd8f63e6.

Change-Id: I6fe5b2ef99f209b32cd4087dfd1c8cac229c2d8b
Reviewed-on: https://dart-review.googlesource.com/c/87265
Commit-Queue: Alexander Aprelev <aam@google.com>
Reviewed-by: Liam Appelbe <liama@google.com>
2018-12-14 23:45:57 +00:00
Liam Appelbe 63fd8f63e6 Revert "Load isolate from parent's kernel in Isolate.spawn calls."
This reverts commit e143a52531.

Reason for revert: Broke all the reload-kernel bots.

Original change's description:
> Load isolate from parent's kernel in Isolate.spawn calls.
> 
> Bug: https://github.com/dart-lang/sdk/issues/6610
> Change-Id: Icd6e1c5d6d4b64b611fc58333c051a8a9a50679d
> Reviewed-on: https://dart-review.googlesource.com/c/85724
> Commit-Queue: Liam Appelbe <liama@google.com>
> Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
> Reviewed-by: Ryan Macnak <rmacnak@google.com>

TBR=vegorov@google.com,rmacnak@google.com,asiva@google.com,liama@google.com

Change-Id: Ie9c305256da9b6478153b99502d0c63b0f43a3e6
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: https://github.com/dart-lang/sdk/issues/6610
Reviewed-on: https://dart-review.googlesource.com/c/86082
Reviewed-by: Liam Appelbe <liama@google.com>
Commit-Queue: Liam Appelbe <liama@google.com>
2018-12-04 20:26:24 +00:00
Liam Appelbe e143a52531 Load isolate from parent's kernel in Isolate.spawn calls.
Bug: https://github.com/dart-lang/sdk/issues/6610
Change-Id: Icd6e1c5d6d4b64b611fc58333c051a8a9a50679d
Reviewed-on: https://dart-review.googlesource.com/c/85724
Commit-Queue: Liam Appelbe <liama@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
Reviewed-by: Ryan Macnak <rmacnak@google.com>
2018-12-04 18:02:55 +00:00
Martin Kustermann dc04320c64 [VM] Fix memory leak after landing embedder abstraction change
Change-Id: I6597cbd871539d06d37991c479aeab2dae0ab089
Reviewed-on: https://dart-review.googlesource.com/65324
Reviewed-by: Zach Anderson <zra@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
2018-07-17 20:36:00 +00:00
Vyacheslav Egorov c910738825 [vm/embedder] Introduce dart_standalone_embedder_api.h
This enables other emdedders to reuse parts of the standalone embedder
implementation without depending too much on the implementations details.

As part of the change we also remove a dependency from DartUtils on
IsolateData - by removing dart:_builtin library caching. Lookup
of this library does not seem to be on hot path anymore in Kernel world.

Change-Id: Ia35b28886121828fad7a96f00bcbceff75e00ae8
Reviewed-on: https://dart-review.googlesource.com/64848
Reviewed-by: Zach Anderson <zra@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
2018-07-17 09:04:05 +00:00
asiva 02e47a9731 [VM] Fix for issue 32188 (cannot use pub with --preview-dart-2)
- Fix for 'unable to read .packages file' error when generating
  kernel file (script snapshot).
- Make sync-async an isolate specific flag and turn it off for
  the kernel isolate as it runs in Dart 1 mode
- Include 'reify-generic-functions' and 'sync-async' flags to the
  list of flags to check an app JIT snapshot against.

Change-Id: I2b16270f65705213660487f9c15fe0348fc84c8b
Reviewed-on: https://dart-review.googlesource.com/55589
Commit-Queue: Siva Annamalai <asiva@google.com>
Reviewed-by: Zach Anderson <zra@google.com>
2018-05-18 18:12:08 +00:00
Ryan Macnak a5c11d7d03 [vm] Remove Dart_ReadKernelBinary, take 2.
Update status for simulators, which the test harness invokes with DIL files instead of source files.

Change-Id: I8444ad7e17a0a71a1ce3c0021487397baa1c3e65
Reviewed-on: https://dart-review.googlesource.com/54622
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>
2018-05-14 20:58:08 +00:00
Ryan Macnak 407872b9e5 Revert "[vm] Remove Dart_ReadKernelBinary."
This reverts commit 2998c32951.

Reason for revert: Failures with kernel + simulators

Original change's description:
> [vm] Remove Dart_ReadKernelBinary.
> 
>  - Fix const-ness of Dart_FileReadCallback and callers.
>  - Fix leak on read error in DartUtils::FileRead.
> 
> This is progress towards sharing kernel memory on Fuchsia.
> 
> Bug: https://github.com/dart-lang/sdk/issues/32618
> Change-Id: I47f8b224905d6a105a5ca0ab2ee4ab6a42b5e342
> Reviewed-on: https://dart-review.googlesource.com/47102
> Commit-Queue: Ryan Macnak <rmacnak@google.com>
> Reviewed-by: Siva Annamalai <asiva@google.com>

TBR=aam@google.com,rmacnak@google.com,asiva@google.com

Change-Id: Ibb636b817de1629a062fe0a6ef9462a2fef669ef
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: https://github.com/dart-lang/sdk/issues/32618
Reviewed-on: https://dart-review.googlesource.com/54560
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
2018-05-10 01:36:39 +00:00
Ryan Macnak 2998c32951 [vm] Remove Dart_ReadKernelBinary.
- Fix const-ness of Dart_FileReadCallback and callers.
 - Fix leak on read error in DartUtils::FileRead.

This is progress towards sharing kernel memory on Fuchsia.

Bug: https://github.com/dart-lang/sdk/issues/32618
Change-Id: I47f8b224905d6a105a5ca0ab2ee4ab6a42b5e342
Reviewed-on: https://dart-review.googlesource.com/47102
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
2018-05-10 00:35:37 +00:00
Siva Chandra 6a6103b9c1 Do not compile script to kernel if the isolate was not created from kernel.
This change, and the code introduced, is probably not going to be required
once the platform kernel file is linked into the executable.

Change-Id: I24a837c3a633a625145cf90ff83d3e51b3d23eb6
Reviewed-on: https://dart-review.googlesource.com/36720
Reviewed-by: Ryan Macnak <rmacnak@google.com>
Commit-Queue: Siva Chandra <sivachandra@google.com>
2018-01-25 18:18:25 +00:00
Siva Annamalai 1e15e6f2d7 Fix for flutter issue 11260 (flutter/flutter#11260)
Do not rely on the udp_receive_buffer field from IsolateData as a buffer for
reading data in the recvfrom call.

Bug:
Change-Id: I43d19f624fd2a406429cc5387bd536524472b4f7
Reviewed-on: https://dart-review.googlesource.com/10212
Reviewed-by: Zach Anderson <zra@google.com>
2017-10-03 15:09:31 +00:00
Siva Annamalai b4ddc97fd5 Revert "Fix for Flutter issue 11260 (https://github.com/flutter/flutter/issues/11260)"
This reverts commit 8817846da6.

BUG=

Review-Url: https://codereview.chromium.org/3005663002 .
2017-08-25 16:15:05 -07:00
Siva Annamalai 8817846da6 Fix for Flutter issue 11260 (https://github.com/flutter/flutter/issues/11260)
Do not rely on the udp_receive_buffer field from IsolateData as a buffer for
reading data in the recvfrom call.

BUG=
R=zra@google.com

Review-Url: https://codereview.chromium.org/3009643002 .
2017-08-25 15:26:55 -07:00
Zachary Anderson 6cd8a79078 VM: Re-format to use at most one newline between functions
R=asiva@google.com

Review-Url: https://codereview.chromium.org/2974233002 .
2017-07-13 08:08:37 -07:00
Siva Annamalai 6656404ad6 1. Restructure code to enable reloading when a kernel dill file is specified on the command line instead of sources.
2. Read the platform dill file at startup not for every isolate creation

3. Store the memory representation of kernel program in isolate data so that it is deleted when the isolate is shutdown

R=aam@google.com

Review-Url: https://codereview.chromium.org/2927493002 .
2017-06-07 13:42:20 -07:00
Vyacheslav Egorov a5e1f89583 VM: Fix an app-jit related shutdown race.
IsolateData contains AppSnapshot which might own some HeapPage-s, so we can't
destroy IsolateData in the Dart_IsolateShutdownCallback, because some thread
running in the isolate (e.g. background compiler) might still touch
thoses pages or objects they host.

We need to delay destruction of AppSnapshot until after the isolate shutdown.

Current API does not allow that, so we introduce a new isolate lifecycle
callback - Dart_IsolateCleanupCallback, which is invoked at the end of the
isolote lifecycle when things like background compiler have been stopped
and no Dart code is supposed to run.

BUG=
R=rmacnak@google.com

Review-Url: https://codereview.chromium.org/2720723005 .
2017-02-28 21:10:04 +01:00
Ryan Macnak 781fd36086 Add an option to gen_snapshot for creating a Makefile describing a snapshot's dependencies. This option may be used when generating any snapshot kind, or without generating a snapshot.
- Factor out MallocGrowableArray from vm to platform.
 - Add File::Print.

R=asiva@google.com

Review-Url: https://codereview.chromium.org/2715463003 .
2017-02-23 12:40:48 -08:00
Ryan Macnak a317b90d63 Cleanup app snapshots on isolate/vm exit.
R=zra@google.com

Review-Url: https://codereview.chromium.org/2694103004 .
2017-02-16 13:54:57 -08:00