[rfc] Trying to make sense of opt-in Node integration in mozilla-central

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[rfc] Trying to make sense of opt-in Node integration in mozilla-central

Nicholas Alexander
Colleagues,

I have a patch series that adds an opt-in --enable-node-environment
configure flag and, when that flag is set, uses Node (via Webpack) to
generate the Activity Stream content bundle.  This patch series does not
try to solve a few hard problems:

1) vendoring Node modules into the tree
2) installing $topobjdir/node_modules at build time efficiently.

There's a green artifact build of my prototype at

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d138f854139f2e389867b01d2f2afe59f2975783

I owe some folks (dmose, jlaster) details on what is need in order to
land this opt-in prototype and, more importantly, how to make the
prototype not opt-in.  To that end, I talked to most of the build peers
(chmanchester, gps, mshal, ted) yesterday in YVR.

The results were not what I expected of that discussion were not what I
expected.

First, some context.  The build system is investing heavily into
capturing the full dependency DAG (Directed Acyclic Graph) in order to
produce correct builds.  The current build backends, and in particular
the dominant RecursiveMake build backend, do not capture the full DAG.
Capturing the full DAG is required to use "modern" build systems like
Tup, Buck, or Bazel.  Any sub-component of the build must therefore
either correspond to edges in the DAG (these inputs, these outputs) or,
if it does its own caching and invalidation, expose its internal DAG.
In the current build system, C compiler invocations are the prototype of
the first situation and cargo is the prototype of the second situation.
What I did not know is that the build peers are contributing code to
cargo to have it expose its internal DAG, and that all of the "modern"
build systems (in particular Buck) need this functionality to integrate
against cargo.

Second, my appraisal of the situation.

Integrating Node will be very challenging.  On the one hand, |yarn
install| (or
|npm install|) is, like cargo, in the second situation -- it is its own
build system that does its own caching and invalidation.  That means
that to integrate into the build system it must expose its internal DAG.
It's possible that yarn could expose its own DAG, but Node modules can
define arbitrary pre- and post-install scripts, which are essential to
the module ecosystem.  I can't imagine us being able to capture the
"leaf DAG" of every installed module -- there are no rules out at the
leaves.

On the second hand, the most general form of integration (which I have
been pursuing) is to enable the build system to invoke arbitrary yarn
verbs (like `GENERATED_FILES[...].script = 'yarn.py';
GENERATED_FILES[...].flags = 'run arbitary_yarn_verb').  Arbitrary yarn
verbs are, well, arbitrary -- they could be simple, like C compiler
invocations, or they could be build systems in their own right, like
Webpack.  For arbitrary yarn verbs, I don't think it's feasible to
extract DAGs from the Node ecosystem tools involved.

Third, what is to be done.

The build peers most invested in the transition to a "modern" build
system (here, Tup) are chmanchester and mshal.  They conclude that it is
not possible to integrate build systems into each other without
significant work exposing internal DAGs (which we are willing to do for
cargo).  They instead propose that build systems not integrate but
instead run in serial.  That is, the "Node bits" run either first (and
provide inputs to the rest of the build system) or run second (and
consume outputs from the rest of the build system).  Of course, that
arrangement sacrifices parallelism and throughput, but at least the
final output will be correct.

This leads me to propose that we treat |yarn install| as a separate
build system that runs before the main build system.  It manages its own
caching and invalidation, and produces $topobjdir/node_modules.  |yarn
install| is intended to efficiently determine that its output is
up-to-date, so perhaps the overhead of running it every time we build
will be acceptable.  (Otherwise, we try to find ways to invoke it less
frequently.)

We then have a choice.  We can either push _all_ Node invocations into
the first build system and accept what I expect to be a big performance
penalty in practice; or we can restrict the Node integration in the main
build system to commands that we are confident are not their own build
systems.

The former is fully general but will require non-trivial effort to
implement in the build system, I expect -- perhaps a new build backend,
specialized to Node, and some glue code in |mach build| to manage
ordering the systems.  In addition, such an arrangement could never
allow Node bits to depend on regular build system bits, since the Node
bits would always happen first.  That might make some sense right now,
since all of the Node projects we're integrating stand-alone (usually on
GitHub!) but as more of the core Firefox front-end functionality
leverages Node that will look worse and worse.  Even exposing
AppConstants.jsm to Node could be fraught (if the actual contents are
required, for example to tree shake on the basis of build flags).

The latter is restrictive -- for example, we might support only Rollup
but not Webpack, since Rollup is more clearly inputs-to-outputs and
Webpack is more focused on incremental builds -- and requires labour to
audit and add support for new tools.  However, it requires less up front
build system modifications and is easier to transition to gradually.

Fourth, my conclusion.

I prefer working within the existing build system and invoking Node
commands rather than arbitrary yarn verbs.

The fast path to landing this as an opt-in therefore looks like:

- adding a new "node" build tier before "pre-export" that runs |yarn install|
- restricting to audited Node-consuming commands like |node webpack| and
  |node rollup| in the build system.

After that we can tackle vendoring Node modules into the tree, which does
not appear to have anything fundamental blocking it.

Phew!  That's a wall of text.  Please correct me if I'm misunderstanding
things, or if my explanations need clarification.  As I said, the
results of this discussion were not what I expected, so this is mostly
new to me :/

I'll wait to collect some feedback on this summary before trying to
figure out next steps.

Yours,
Nick


_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: [rfc] Trying to make sense of opt-in Node integration in mozilla-central

Gregory Szorc-3
On Fri, Apr 20, 2018 at 10:24 AM, Nicholas Alexander <[hidden email]> wrote:
Colleagues,

I have a patch series that adds an opt-in --enable-node-environment
configure flag and, when that flag is set, uses Node (via Webpack) to
generate the Activity Stream content bundle.  This patch series does not
try to solve a few hard problems:

1) vendoring Node modules into the tree
2) installing $topobjdir/node_modules at build time efficiently.

There's a green artifact build of my prototype at

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d138f854139f2e389867b01d2f2afe59f2975783

I owe some folks (dmose, jlaster) details on what is need in order to
land this opt-in prototype and, more importantly, how to make the
prototype not opt-in.  To that end, I talked to most of the build peers
(chmanchester, gps, mshal, ted) yesterday in YVR.

The results were not what I expected of that discussion were not what I
expected.

First, some context.  The build system is investing heavily into
capturing the full dependency DAG (Directed Acyclic Graph) in order to
produce correct builds.  The current build backends, and in particular
the dominant RecursiveMake build backend, do not capture the full DAG.
Capturing the full DAG is required to use "modern" build systems like
Tup, Buck, or Bazel.  Any sub-component of the build must therefore
either correspond to edges in the DAG (these inputs, these outputs) or,
if it does its own caching and invalidation, expose its internal DAG.
In the current build system, C compiler invocations are the prototype of
the first situation and cargo is the prototype of the second situation.
What I did not know is that the build peers are contributing code to
cargo to have it expose its internal DAG, and that all of the "modern"
build systems (in particular Buck) need this functionality to integrate
against cargo.

Second, my appraisal of the situation.

Integrating Node will be very challenging.  On the one hand, |yarn
install| (or
|npm install|) is, like cargo, in the second situation -- it is its own
build system that does its own caching and invalidation.  That means
that to integrate into the build system it must expose its internal DAG.
It's possible that yarn could expose its own DAG, but Node modules can
define arbitrary pre- and post-install scripts, which are essential to
the module ecosystem.  I can't imagine us being able to capture the
"leaf DAG" of every installed module -- there are no rules out at the
leaves.

On the second hand, the most general form of integration (which I have
been pursuing) is to enable the build system to invoke arbitrary yarn
verbs (like `GENERATED_FILES[...].script = 'yarn.py';
GENERATED_FILES[...].flags = 'run arbitary_yarn_verb').  Arbitrary yarn
verbs are, well, arbitrary -- they could be simple, like C compiler
invocations, or they could be build systems in their own right, like
Webpack.  For arbitrary yarn verbs, I don't think it's feasible to
extract DAGs from the Node ecosystem tools involved.

Third, what is to be done.

The build peers most invested in the transition to a "modern" build
system (here, Tup) are chmanchester and mshal.  They conclude that it is
not possible to integrate build systems into each other without
significant work exposing internal DAGs (which we are willing to do for
cargo).  They instead propose that build systems not integrate but
instead run in serial.  That is, the "Node bits" run either first (and
provide inputs to the rest of the build system) or run second (and
consume outputs from the rest of the build system).  Of course, that
arrangement sacrifices parallelism and throughput, but at least the
final output will be correct.

This leads me to propose that we treat |yarn install| as a separate
build system that runs before the main build system.  It manages its own
caching and invalidation, and produces $topobjdir/node_modules.  |yarn
install| is intended to efficiently determine that its output is
up-to-date, so perhaps the overhead of running it every time we build
will be acceptable.  (Otherwise, we try to find ways to invoke it less
frequently.)

We then have a choice.  We can either push _all_ Node invocations into
the first build system and accept what I expect to be a big performance
penalty in practice; or we can restrict the Node integration in the main
build system to commands that we are confident are not their own build
systems.

The former is fully general but will require non-trivial effort to
implement in the build system, I expect -- perhaps a new build backend,
specialized to Node, and some glue code in |mach build| to manage
ordering the systems.  In addition, such an arrangement could never
allow Node bits to depend on regular build system bits, since the Node
bits would always happen first.  That might make some sense right now,
since all of the Node projects we're integrating stand-alone (usually on
GitHub!) but as more of the core Firefox front-end functionality
leverages Node that will look worse and worse.  Even exposing
AppConstants.jsm to Node could be fraught (if the actual contents are
required, for example to tree shake on the basis of build flags).

The latter is restrictive -- for example, we might support only Rollup
but not Webpack, since Rollup is more clearly inputs-to-outputs and
Webpack is more focused on incremental builds -- and requires labour to
audit and add support for new tools.  However, it requires less up front
build system modifications and is easier to transition to gradually.

Fourth, my conclusion.

I prefer working within the existing build system and invoking Node
commands rather than arbitrary yarn verbs.

The fast path to landing this as an opt-in therefore looks like:

- adding a new "node" build tier before "pre-export" that runs |yarn install|
- restricting to audited Node-consuming commands like |node webpack| and
  |node rollup| in the build system.

After that we can tackle vendoring Node modules into the tree, which does
not appear to have anything fundamental blocking it.

Phew!  That's a wall of text.  Please correct me if I'm misunderstanding
things, or if my explanations need clarification.  As I said, the
results of this discussion were not what I expected, so this is mostly
new to me :/

I'll wait to collect some feedback on this summary before trying to
figure out next steps.

This is a good summary and captures the fears that many of us build system maintainers have with integrating 3rd party tools that behave like or are build systems.

I think it helps to divide the problem space into 2 parts:

a) Enabling the running of Node in the build system (i.e. Node package management)
b) Running Node in the build system

We kind of have a precedent for both parts with Python. For Python, we create a virtualenv in the objdir during configure. And then for each Python process we run, we either teach the build system about the dependencies and outputs explicitly (often via moz.build files or custom Python code running during moz.build evaluation time). Or we do it inline in the Python that is executed at build time (e.g. by spitting out a make dependencies file).

I think "a" should live in configure *or* should be invoked at build time by the Python code backing `mach build`. I don't think it should live in the pre-export tier because that is specific to the make backend and having it live there will require us to implement logic for invoking it in every backend. This "step" of the build is common to all backends and should not live in a single backend.

"b" is a harder problem because it isn't well-defined. We *will* need every Node invocation during the build to define its dependencies and contribution to the overall DAG. We can do things like assume all installed Node module files [managed by "a"] are dependencies for *every* Node process. But for the non-obvious inputs and for all outputs, we'll need to annotate those somehow. There's no getting around that. Well, we could ignore enumerating those inputs and outputs. But then we invoke Node processes on every build and this will make no-op and light builds slower. We don't like making builds slower by running things that shouldn't need to run. So we'll end up enumerating inputs and outputs.

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds