This is a reland of commit b9e25afcb4
Original change's description:
> [dart2js] Stop emitting a dill with closed world.
>
> All transformations to the dill are now performed with the CFE linker phase (phase0b). This means we no longer need to emit a dill as part of the closed world computation. This saves both time and memory resources associated with the cost of serialization during that phase.
>
> Change-Id: I28dde8a1eecd71e823880027505f62fc804fcca6
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/332821
> Reviewed-by: Sigmund Cherem <sigmund@google.com>
> Reviewed-by: Mayank Patke <fishythefish@google.com>
> Commit-Queue: Nate Biggs <natebiggs@google.com>
Change-Id: Ia693213dcdc949a8ef4b4204cc033130274d2381
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/336982
Reviewed-by: Mayank Patke <fishythefish@google.com>
Commit-Queue: Nate Biggs <natebiggs@google.com>
This is a reland of commit bebd08746b
Original change's description:
> [dart2js] Evaluate CFE consts as part of phase 0b (CFE linker).
>
> Constants are current evaluated in a few places during closed world generation, primarily as part of the ScopeModelBuilder. The scope visitor was modifying the AST which meant we had to emit a new dill with these evaluated constants along with the closed world results.
>
> This change instead evaluates the constants directly after linking the Kernel as part of the global transformations. This means we can update the ScopeModelBuilder to not mutate the AST at all as all constants are already simplified.
>
> A potential follow up here is to simplify the ScopeModelBuilder since all nodes should already be simplified if they can be, we should be able to avoid visiting some children.
>
> After this change we only directly create a single ConstantEvaluator, the one in `load_kernel`. The const simplifier also creates one and a follow up CL moves this to to run right after this new transformation.
>
> Note: Alternate versions of this CL tried to make the global transformation simpler by either:
> 1) Running the const evaluator indiscriminately on all expressions. This didn't work because it lead to exponential computation on constants set up as a DAG (see tests/language/const/constant_dag_test).
> 2) Only evaluating ConstantExpression nodes to update UnevaluatedConstants. This does not cover all the cases where the ScopeModelBuilder is modifying the tree and lead to a different compiler output.
>
> Change-Id: I746d889b37feddc9ab6c386c6252016dec745e6e
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/332601
> Reviewed-by: Mayank Patke <fishythefish@google.com>
Change-Id: I53871a57144a3a1bd363af141d8f31c8ffa5ca6d
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/336221
Reviewed-by: Mayank Patke <fishythefish@google.com>
Commit-Queue: Nate Biggs <natebiggs@google.com>
All transformations to the dill are now performed with the CFE linker phase (phase0b). This means we no longer need to emit a dill as part of the closed world computation. This saves both time and memory resources associated with the cost of serialization during that phase.
Change-Id: I28dde8a1eecd71e823880027505f62fc804fcca6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/332821
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Reviewed-by: Mayank Patke <fishythefish@google.com>
Commit-Queue: Nate Biggs <natebiggs@google.com>
Constants are current evaluated in a few places during closed world generation, primarily as part of the ScopeModelBuilder. The scope visitor was modifying the AST which meant we had to emit a new dill with these evaluated constants along with the closed world results.
This change instead evaluates the constants directly after linking the Kernel as part of the global transformations. This means we can update the ScopeModelBuilder to not mutate the AST at all as all constants are already simplified.
A potential follow up here is to simplify the ScopeModelBuilder since all nodes should already be simplified if they can be, we should be able to avoid visiting some children.
After this change we only directly create a single ConstantEvaluator, the one in `load_kernel`. The const simplifier also creates one and a follow up CL moves this to to run right after this new transformation.
Note: Alternate versions of this CL tried to make the global transformation simpler by either:
1) Running the const evaluator indiscriminately on all expressions. This didn't work because it lead to exponential computation on constants set up as a DAG (see tests/language/const/constant_dag_test).
2) Only evaluating ConstantExpression nodes to update UnevaluatedConstants. This does not cover all the cases where the ScopeModelBuilder is modifying the tree and lead to a different compiler output.
Change-Id: I746d889b37feddc9ab6c386c6252016dec745e6e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/332601
Reviewed-by: Mayank Patke <fishythefish@google.com>
The element map indexing system adds complexity to our element model with little benefit. Serialization already has its own indexing logic that we now reuse for entities.
Some things to note:
1) Libraries are still assigned an index. We only use that index in one place in the namer to disambiguate members. I now assign these indices as a late final field when adding the libraries to the element map.
2) The sizes of serialized files get slightly bigger since serialization indexing uses "addresses" rather than incrementing numbers for the indices. This effect is very minimal (<4MB+ for the largest programs).
3) "Late" members (constructor and generator bodies) which are created during codegen must be tracked and registered separately. Before the index was used to determine which entities were "late".
I see a very small memory improvement across all phases because of this change. There is also a small regression in runtime because the entity map queries are hash map lookups instead of list index lookups now. I think this very small regression is worth the reduced complexity in the compiler.
If we care about getting that time regression back we can add an "IndexedEntityMixin" with a late final index member that the registry would set/manage internally and would allow it to go back to tracking entities in a List. The field should be private so it would only be used for this purpose. This would only be necessary if we find the added time overhead of the map lookups to be too much.
Change-Id: Idd6d22bfc6ab61943a07feb58b35287d50e7d72e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/326461
Reviewed-by: Mayank Patke <fishythefish@google.com>
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Reviewed-by: Stephen Adams <sra@google.com>
Commit-Queue: Nate Biggs <natebiggs@google.com>
Simplifies indices management for serialization by reusing the same indexing readers/writers for each phase. This way the caches are shared across all the phases and we can decouple the indices with any specific data read.
This fixes an issue that prevented the indices from being used in certain read/write patterns. None of our current usages of these indices exhibited this pattern but later changes I have planned do run into this issue.
This new pattern is also more resilient to sharding/parallel data files. Previously there could be collisions if two files were written in "parallel" (such as the codegen shards) and later their address spaces had to be merged. Now we do the address space merging at read time when we have full knowledge of the parallel files and can therefore avoid collisions.
Change-Id: Iff4c1461e734fc00f251d81f9fff1b9db83484d6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/326460
Reviewed-by: Stephen Adams <sra@google.com>
Isolates data needed from emitter phase for dump info task. The new DumpInfoProgramData captures only the necessary data and allows it to be serialized so that dump info can be run independently of the emitter task.
This change also ends up improving the combined emitter and dump info task. By processing the JS AST and extracting out the relevant info up front, the AST is able to be GCed before the dump info task runs. In local runs on a big app this resulted in ~1.4GB less ram being.
Runtime stat comparison:
Without --dump-info: 117s 5.4GB
Before with --dump-info: 205s 18.3GB
Before with --dump-info=binary: 148s 9.9GB
After with --dump-info: 205s 16.9GB
After with --dump-info=binary: 150s 7.5GB
After with --dump-info-write: 132s 6.2GB
After with --dump-info-read: 118s 14.8GB
After with --dump-info-read --dump-info=binary: 67.9s 5.5GB
By providing the new "--dump-info-write-uri=URI" a binary file will be written with the necessary dump info data. Then if Dart2JS is invoked with the same parameters and "--dump-info-read-uri=URI" then the compiler will skip generating/emitting JS and use the output file to generate the info dump. The existing invocation of just adding "--dump-info" works as before.
I also had the goal of removing DumpInfoTask dependencies on other parts of the compiler and vice versa.
Change-Id: I3f40b0136eb77a709c49737607a2ad6127127e49
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/322843
Commit-Queue: Nate Biggs <natebiggs@google.com>
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Maintains functionality of existing '--write-X-uri' and '--read-X-uri' but also introduces new '--stage' flag that indicates which phases of the compiler should run.
Also expands the behavior of the `--output` flag to support taking a file prefix and/or an output directory that files are written out to.
Valid invocations can look like:
`dart2js main.dart --stage=cfe`
`dart2js main.dart --stage=cfe --output=prefix1-`
`dart2js main.dart --stage=cfe --output=/some/path/`
Change-Id: If90fb16f549d35f16f77d660c51f8d28673e454f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/300520
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Reviewed-by: Mayank Patke <fishythefish@google.com>
Commit-Queue: Nate Biggs <natebiggs@google.com>
This change locks down the Dart2JS command line API to the few invocation flows we use today:
1) Full compile from Dart sources.
2) Full compile from pre-compiled dill sources (modular CFE).
3) Phased compilation from Dart sources.
4) Phased compilation from pre-compiled dill sources (modular CFE).
It also ensures that each phase is run individually (accounting for sharding in codegen/emitter).
There are already places in the code where we assume that Dart2JS will be invoked in one of these ways so this should codify those assumptions.
Change-Id: I4e2a85d123e0c2ad924e2dd1b230548a995af0da
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/299760
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Reviewed-by: Mayank Patke <fishythefish@google.com>
We encode line/column in a single 64-bit integer and derive offset from this using the lineStarts stored on the kernel Source object.
Saves 90-100MB of memory within the emitter phase on a large program. Most of that is from the smaller deserialized SourceLocation objects but some memory is also saved due to the smaller output size from the compact serial representation. The input to the emitter phase is ~14MB smaller (with the savings split across multiple codegen shards).
Change-Id: Ic71231fb43fc746293b6f6f77e0ee2c3b17742cf
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/291900
Commit-Queue: Nate Biggs <natebiggs@google.com>
Reviewed-by: Stephen Adams <sra@google.com>
- Remove unsound platform .dill files from the packaged SDK.
- Throw and error referencing the sound null safety requirements of
Dart 3 when compiling with `--no-sound-null-safety` and an unsound
platform .dill file is not found.
Change-Id: I703a446e3e92c76a6b48168577fc0161eea83854
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/289280
Commit-Queue: Nate Biggs <natebiggs@google.com>
Reviewed-by: Sigmund Cherem <sigmund@google.com>
This prepares dart2js to default to sound null safety in Dart 3.0.
The compiler no longer infers the default mode based on the input
program, instead uses sound null safety unless
'--no-sound-null-safety' is specified.
On a separate change, we expect to remove the option of disabling
null safety.
Note: this change should also be reflected in the CHANGELOG, but
we've currently not included it to redeuce merge conflicts, as we plan
to land multiple 3.0-alpha changes in a short window of time
Change-Id: Id87498cc5976548ec89d1f36c60674b72406950a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/270860
Reviewed-by: Nate Biggs <natebiggs@google.com>
Commit-Queue: Siva Annamalai <asiva@google.com>
Reviewed-by: Mayank Patke <fishythefish@google.com>
This test catches an issue previously uncaught by tests where offsets for indices were written incorrectly and could therefore lead to errors when deserializingelements in deferred mode.
Change-Id: I14bf80bbe4fa47c12e543679803c7be38582a0e5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/252300
Commit-Queue: Nate Biggs <natebiggs@google.com>
Reviewed-by: Sigmund Cherem <sigmund@google.com>
At this point the only imports of the pre-nnbd DataSourceReader and DataSinkWriter are files that need to instantiate the class. Other references have been migrated.
Change-Id: I79672fc5f8c681d05a183990e184535f1becc0e9
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/249742
Reviewed-by: Joshua Litt <joshualitt@google.com>
Commit-Queue: Nate Biggs <natebiggs@google.com>
- Pass options to all DataSourceReader and DataSinkWriter constructor calls.
- Add DataAndIndices to replace ClosedWorldAndIndices. Maintains the same indices as before for now.
Change-Id: I2e25d4a4c25ef12276fa808146ff8aae27da792c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/247401
Reviewed-by: Joshua Litt <joshualitt@google.com>
Commit-Queue: Nate Biggs <natebiggs@google.com>
Always import 'lib/compiler_api.dart' with prefix 'api'.
Remove import of 'lib/src/compiler.dart' with prefix 'api'.
Some of the test helper files under 'compiler/test' are a little tedious
with lots of 'api.' prefixes and I could be convinced to change them
back but I think we should stick with always using a prefix under
'compiler/lib/'.
Change-Id: I3f82d7d31f1c90a860c9811fce031a64cea04ddb
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/241867
Reviewed-by: Joshua Litt <joshualitt@google.com>
Commit-Queue: Stephen Adams <sra@google.com>
- Pulls dumpInfo out from the Linking phase.
- Forks dumpInfo into ElementInfoCollector and KernelInfoCollector.
- Introduces a temporary compiler variable that will be more cleanly integrated later.
Change-Id: I5f0970d9dbd5b1f215063ac161468b2774fb28a3
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/239005
Reviewed-by: Joshua Litt <joshualitt@google.com>
Commit-Queue: Mark Zhou <markzipan@google.com>
This cl adds a temporary flag '--no-closed-world-in-data' in order to
support serializing global inference results with and without the closed
world.
This flag only exists to allow us to roll this into google3. Afterwards,
it will be removed.
Change-Id: I6e9712739edd148edcf43007b6ece35efa688538
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/192124
Commit-Queue: Joshua Litt <joshualitt@google.com>
Reviewed-by: Sigmund Cherem <sigmund@google.com>
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Nate Biggs