Better incremental builds - Tup backend ready for early adopters on Linux

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Better incremental builds - Tup backend ready for early adopters on Linux

Michael Shal

Hi everyone,


Tup is a modern build system that enables fast and correct builds. I'm pleased to announce the availability of Tup for building Firefox on Linux. Compared to Make, building with Tup will result in faster incremental build times and reduce the need for clobber builds. See the in-tree docs for details [1].


We’re looking for some early adopters to try it out and let us know if it improves your workflow, or if you hit any blockers or barriers to adoption that keep you on the Make backend for now.


Quick howto:


You’ll need to install the Tup executable, as well as the nightly rust/cargo toolchain:


cd ~/.mozbuild && mach artifact toolchain --from-build linux64-tup
rustup install nightly
rustup default nightly

Your mozconfig needs to describe how to find the executable if it’s not in your PATH, and enable the Tup backend:


export TUP=~/.mozbuild/tup/tup
ac_add_options --enable-build-backends=Tup


If you have any issues, feel free to file a bug blocking “buildtup” [2], or contact mshal or chmanchester in #build on IRC.

[1] https://firefox-source-docs.mozilla.org/build/buildsystem/tup.html

[2] https://bugzilla.mozilla.org/show_bug.cgi?id=buildtup


Thanks!

-Mike



_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Better incremental builds - Tup backend ready for early adopters on Linux

Jonathan Watt-3
This is great! A big thank you to all the folks who have worked for so long to
get us to this stage. I'm really looking forward to this work maturing and
extending to other platforms.

I've done a bit of testing on my workstation (14 core/28 thread i9, U.2
connected NVMe SSD, 64 GB DDR4 RAM) and got the following build times:

For -j28 (yes, -j28 is well into diminishing returns territory, but I wanted a
comparison of the best case scenarios for both backends):

      |      cold       |    top-level
      |  initial build  |  no-op rebuild
------------------------------------------
make  |      8m 10s     |       15s
tup   |      6m 35s     |        1.4s

And for -j8:

      |      cold       |    top-level
      |  initial build  |  no-op rebuild
------------------------------------------
make  |     11m 33s     |       15s
tup   |     10m 52s     |        1.4s

(To confirm: the rebuild numbers are the same for both -j28 and -j8.)

So a 20% reduction in build time on cold builds of a fresh tree with -j28, and
6% reduction with -j8. In addition to speeding up the build, Tup seems to
parallelize better which is great! Of course the highlight is the big reduction
in rebuild overhead which will make the edit-compile-run cycle a lot less
annoying (for me, 50s down to 37s after touching nsDocumentViewer.cpp,
rebuilding, and linking libxul, for both -j28 and -j8.)

These improvements, plus the promise of rarely having to clobber/figure out
which directories to rebuild in, put a big smile on my face. (Currently having
to turn off CARGO_INCREMENTAL, not being able to use sccache and other current
limitations, less so, but it's early days. :) )

A few notes for others trying this out (thanks cmanchester and mshal for the
help debugging some of these):

  * Make sure you use:

      --enable-build-backends=Tup

    not:

      --enable-build-backend=Tup

  * You must be using an in-tree OBJDIR.

  * MOZ_PARALLEL_BUILD isn't currently recognized, so you'll need to use
    `mach build`s -j argument to control the parallel job count for now
    ( see https://bugzilla.mozilla.org/show_bug.cgi?id=1481441 )

mshal: I think you could cross-post this to dev-platform for wider testing. I
know there are a bunch of caveats (which people should be made aware of), but
since the initial teething issues I had were figured out I've so far had no
other problems building, or rebuilding with real code changes.

Jonathan

On 02/08/2018 16:34, Michael Shal wrote:

> Hi everyone,
>
>
> Tup is a modern build system that enables fast and correct builds. I'm pleased
> to announce the availability of Tup for building Firefox on Linux. Compared to
> Make, building with Tup will result in faster incremental build times and reduce
> the need for clobber builds. See the in-tree docs for details [1].
>
>
> We’re looking for some early adopters to try it out and let us know if it
> improves your workflow, or if you hit any blockers or barriers to adoption that
> keep you on the Make backend for now.
>
>
> Quick howto:
>
>
> You’ll need to install the Tup executable, as well as the nightly rust/cargo
> toolchain:
>
>
> cd ~/.mozbuild && mach artifact toolchain --from-build linux64-tup
> rustup install nightly
> rustup default nightly
>
> Your mozconfig needs to describe how to find the executable if it’s not in your
> PATH, and enable the Tup backend:
>
>
> export TUP=~/.mozbuild/tup/tup
> ac_add_options --enable-build-backends=Tup
>
>
> If you have any issues, feel free to file a bug blocking “buildtup” [2], or
> contact mshal or chmanchester in #build on IRC.
>
> [1] https://firefox-source-docs.mozilla.org/build/buildsystem/tup.html
>
> [2] https://bugzilla.mozilla.org/show_bug.cgi?id=buildtup
>
>
> Thanks!
>
> -Mike
>
>

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Better incremental builds - Tup backend ready for early adopters on Linux

Boris Zbarsky
On 8/7/18 9:18 AM, Jonathan Watt wrote:
>    * You must be using an in-tree OBJDIR.

Do you happen to know whether this is a temporary limitation or a
permanent one?

-Boris
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Better incremental builds - Tup backend ready for early adopters on Linux

Jonathan Watt-3
On 07/08/2018 15:06, Boris Zbarsky wrote:
> On 8/7/18 9:18 AM, Jonathan Watt wrote:
>>    * You must be using an in-tree OBJDIR.
>
> Do you happen to know whether this is a temporary limitation or a
> permanent one?

When I previously discussed that with mshal he said the following (edited to
replace locations specific to my setup):

"A little background about tup: by default it keeps track of everything in the
directory where 'tup init' gets run, and all subdirectories (if you look in your
top-level source directory, you should have a ".tup" directory there). So in the
case you use an out-of-tree $OBJDIR Tup doesn't see all the Tupfiles created in
that directory, since that is outside of where `tup init` gets run. I think it
might be possible to support this in the future, but that is likely a
non-trivial effort."
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Better incremental builds - Tup backend ready for early adopters on Linux

Boris Zbarsky
On 8/7/18 11:27 AM, Jonathan Watt wrote:
> When I previously discussed that with mshal he said the following (edited to
> replace locations specific to my setup):
>
> "A little background about tup: by default it keeps track of everything in the
> directory where 'tup init' gets run, and all subdirectories

Hmm.  So if my objdirs are siblings of my srcdir, could I just run "tup
init" in the common parent?

Being able to have objdirs outside the srcdir is pretty important in my
case; it makes it a lot easier to search code without having the objdir
interfere with the searches...

-Boris
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Better incremental builds - Tup backend ready for early adopters on Linux

mozilla
On Tue, Aug 7, 2018 at 9:37 AM Boris Zbarsky <[hidden email]> wrote:
Being able to have objdirs outside the srcdir is pretty important in my
case; it makes it a lot easier to search code without having the objdir
interfere with the searches...

Depending on what tool you use, it should be possible to exclude objdir's from search anyway. I know ripgrep[1] (for example) will look at `.gitignore` in-tree and ignore any top-level directories that begin with `obj`. I guess pretty much any tool should be able to be configured to ignore object directories similarly.

-- Tom


_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Better incremental builds - Tup backend ready for early adopters on Linux

Boris Zbarsky
On 8/7/18 11:49 AM, Tom Prince wrote:
> Depending on what tool you use, it should be possible to exclude
> objdir's from search anyway.

There are multiple tools, depending on what I'm doing.  "find", "grep",
"mkid" (from idutils) are the most common ones.

"find" and "mkid" do have ways to exclude directories via command-line
options, but not a permanent configuration that I'm aware of.

-Boris
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Better incremental builds - Tup backend ready for early adopters on Linux

Nicholas Nethercote
In reply to this post by Jonathan Watt-3
>  * You must be using an in-tree OBJDIR.

Does it handle multiple in-tree objdirs? E.g. I have d64 (debug), o64 (opt), etc.

Nick


On Tue, Aug 7, 2018 at 11:18 PM, Jonathan Watt <[hidden email]> wrote:
This is great! A big thank you to all the folks who have worked for so long to
get us to this stage. I'm really looking forward to this work maturing and
extending to other platforms.

I've done a bit of testing on my workstation (14 core/28 thread i9, U.2
connected NVMe SSD, 64 GB DDR4 RAM) and got the following build times:

For -j28 (yes, -j28 is well into diminishing returns territory, but I wanted a
comparison of the best case scenarios for both backends):

      |      cold       |    top-level
      |  initial build  |  no-op rebuild
------------------------------------------
make  |      8m 10s     |       15s
tup   |      6m 35s     |        1.4s

And for -j8:

      |      cold       |    top-level
      |  initial build  |  no-op rebuild
------------------------------------------
make  |     11m 33s     |       15s
tup   |     10m 52s     |        1.4s

(To confirm: the rebuild numbers are the same for both -j28 and -j8.)

So a 20% reduction in build time on cold builds of a fresh tree with -j28, and
6% reduction with -j8. In addition to speeding up the build, Tup seems to
parallelize better which is great! Of course the highlight is the big reduction
in rebuild overhead which will make the edit-compile-run cycle a lot less
annoying (for me, 50s down to 37s after touching nsDocumentViewer.cpp,
rebuilding, and linking libxul, for both -j28 and -j8.)

These improvements, plus the promise of rarely having to clobber/figure out
which directories to rebuild in, put a big smile on my face. (Currently having
to turn off CARGO_INCREMENTAL, not being able to use sccache and other current
limitations, less so, but it's early days. :) )

A few notes for others trying this out (thanks cmanchester and mshal for the
help debugging some of these):

  * Make sure you use:

      --enable-build-backends=Tup

    not:

      --enable-build-backend=Tup

  * You must be using an in-tree OBJDIR.

  * MOZ_PARALLEL_BUILD isn't currently recognized, so you'll need to use
    `mach build`s -j argument to control the parallel job count for now
    ( see https://bugzilla.mozilla.org/show_bug.cgi?id=1481441 )

mshal: I think you could cross-post this to dev-platform for wider testing. I
know there are a bunch of caveats (which people should be made aware of), but
since the initial teething issues I had were figured out I've so far had no
other problems building, or rebuilding with real code changes.

Jonathan

On 02/08/2018 16:34, Michael Shal wrote:
> Hi everyone,
>
>
> Tup is a modern build system that enables fast and correct builds. I'm pleased
> to announce the availability of Tup for building Firefox on Linux. Compared to
> Make, building with Tup will result in faster incremental build times and reduce
> the need for clobber builds. See the in-tree docs for details [1].
>
>
> We’re looking for some early adopters to try it out and let us know if it
> improves your workflow, or if you hit any blockers or barriers to adoption that
> keep you on the Make backend for now.
>
>
> Quick howto:
>
>
> You’ll need to install the Tup executable, as well as the nightly rust/cargo
> toolchain:
>
>
> cd ~/.mozbuild && mach artifact toolchain --from-build linux64-tup
> rustup install nightly
> rustup default nightly
>
> Your mozconfig needs to describe how to find the executable if it’s not in your
> PATH, and enable the Tup backend:
>
>
> export TUP=~/.mozbuild/tup/tup
> ac_add_options --enable-build-backends=Tup
>
>
> If you have any issues, feel free to file a bug blocking “buildtup” [2], or
> contact mshal or chmanchester in #build on IRC.
>
> [1] https://firefox-source-docs.mozilla.org/build/buildsystem/tup.html
>
> [2] https://bugzilla.mozilla.org/show_bug.cgi?id=buildtup
>
>
> Thanks!
>
> -Mike
>
>

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds


_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Better incremental builds - Tup backend ready for early adopters on Linux

Michael Shal
In reply to this post by Boris Zbarsky
On Tue, Aug 7, 2018 at 11:32 AM, Boris Zbarsky <[hidden email]> wrote:
On 8/7/18 11:27 AM, Jonathan Watt wrote:
When I previously discussed that with mshal he said the following (edited to
replace locations specific to my setup):

"A little background about tup: by default it keeps track of everything in the
directory where 'tup init' gets run, and all subdirectories

Hmm.  So if my objdirs are siblings of my srcdir, could I just run "tup init" in the common parent?

Being able to have objdirs outside the srcdir is pretty important in my case; it makes it a lot easier to search code without having the objdir interfere with the searches...

 Yes, that should work, though with a caveat - you'd probably want to make sure that the common parent is only used for one srcdir and objdir. So this works fine:

mydir/mozilla-central
mydir/objdir
mydir/.tup

So you would run 'tup init' manually in "mydir", and have a mozconfig like:

mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../objdir
ac_add_options --build-backends=Tup

I've confirmed locally that this works, though you just get an innocuous warning when mozbuild tries to run 'tup init' inside the srcdir: "tup warning: database already exists in directory: .../mydir". We could improve support for this by trying to run 'tup init' automatically in the parent dir if the objdir is a sibling of the srcdir, which shouldn't be too hard to add.

For comparison, the "default" case of an objdir inside the srcdir looks like this:

mydir/mozilla-central
mydir/mozilla-central/objdir
mydir/mozilla-central/.tup

Without a file monitor, tup scans everything under where ".tup" exists, so all of "mydir" in the first example and "mydir/mozilla-central" in the default case.

The caveat applies in what I think jwatt's case is (jwatt, correct me if I'm wrong - I may have misunderstood your setup) where you have multiple trees and objdirs all as siblings. For example:

mydir/mozilla-central
mydir/mozilla-inbound
mydir/mozilla-beta
mydir/objdir-central-debug
mydir/objdir-central-opt
mydir/objdir-inbound-debug
mydir/objdir-inbound-opt
...
mydir/.tup

In this case, since we don't have the file monitor integrated yet, the initial scan would likely take way too long and you'd lose much of the benefit of switching backends in the first place. I think we could support this structure in the future, but it would require some changes in tup itself, and is likely not a quick fix.

So if you already have the srcdir/objdir pairs in an isolated parent, you should be good to go now with a manual 'tup init' in the parent, and we can support that workflow fairly easily. If it's the latter case where many trees are in a single common root, I don't have a good solution for that at the moment.

-Mike

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Better incremental builds - Tup backend ready for early adopters on Linux

Michael Shal
In reply to this post by Nicholas Nethercote
On Wed, Aug 8, 2018 at 3:03 AM, Nicholas Nethercote <[hidden email]> wrote:
>  * You must be using an in-tree OBJDIR.

Does it handle multiple in-tree objdirs? E.g. I have d64 (debug), o64 (opt), etc.

That should work with a similar caveat that tup is going to scan everything under where ".tup" exists, which would include both objdirs. So there will be some additional startup overhead until we have the file monitor integrated -- I guess at the moment it's a question of how many objdirs you have and how much of an impact it adds to the startup overhead. I'll try it out locally myself, but I'd be interested to hear your experiences as well!

-Mike

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Better incremental builds - Tup backend ready for early adopters on Linux

Boris Zbarsky
In reply to this post by Michael Shal
On 8/8/18 10:29 AM, Michael Shal wrote:
>   Yes, that should work, though with a caveat - you'd probably want to
> make sure that the common parent is only used for one srcdir and objdir.

I definitely have multiple objdirs per srcdir.  ;)  opt and debug at the
very least; in some cases a few others.

> The caveat applies in what I think jwatt's case is (jwatt, correct me if
> I'm wrong - I may have misunderstood your setup) where you have multiple
> trees and objdirs all as siblings.

Right, that's not my situation.  I have basically a bunch of dirs like
(names changed for clarity):

   dir1/srctree
   dir1/obj-debug
   dir1/obj-opt
   dir2/srctree
   dir2/obj-debug
   dir2/obj-opt
   dir2/obj-opt-asan

etc.  So all the srcdirs are siloed off from each other in my case.

Sounds like in my case tup would scan all the objdirs and the srcdir, right?

-Boris
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Better incremental builds - Tup backend ready for early adopters on Linux

Michael Shal
On Wed, Aug 8, 2018 at 12:59 PM, Boris Zbarsky <[hidden email]> wrote:
On 8/8/18 10:29 AM, Michael Shal wrote:

The caveat applies in what I think jwatt's case is (jwatt, correct me if I'm wrong - I may have misunderstood your setup) where you have multiple trees and objdirs all as siblings.

Right, that's not my situation.  I have basically a bunch of dirs like (names changed for clarity):

  dir1/srctree
  dir1/obj-debug
  dir1/obj-opt
  dir2/srctree
  dir2/obj-debug
  dir2/obj-opt
  dir2/obj-opt-asan

etc.  So all the srcdirs are siloed off from each other in my case.

Sounds like in my case tup would scan all the objdirs and the srcdir, right?

Yes, so it should work if you run 'tup init' manually in dir1 and again in dir2. I ran some tests on my machine, and it looks like each additional objdir only added about 100ms to the scan time. So I think with a handful of objdirs it's not likely you'll notice a huge difference as compared to a single objdir. I had 3 objdirs and a no-op 'mach build' was still under 2 seconds, but YMMV :)

-Mike

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Better incremental builds - Tup backend ready for early adopters on Linux

Boris Zbarsky
On 8/8/18 3:48 PM, Michael Shal wrote:
> Yes, so it should work if you run 'tup init' manually in dir1 and again
> in dir2. I ran some tests on my machine, and it looks like each
> additional objdir only added about 100ms to the scan time.

That's awesome.  ;)

-Boris
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Better incremental builds - Tup backend ready for early adopters on Linux

Myk Melez-4
In reply to this post by Michael Shal
Michael Shal wrote on 2018-08-02 08:34:

Hi everyone,


Tup is a modern build system that enables fast and correct builds. I'm pleased to announce the availability of Tup for building Firefox on Linux. Compared to Make, building with Tup will result in faster incremental build times and reduce the need for clobber builds. See the in-tree docs for details [1].

Should Btup bustage trigger immediate backout? I appear to have busted it with a recent change [1], triggering an immediate backout, and it isn't obvious to me that this is the optimal policy for an experimental build backend.

(I'll investigate the bustage regardless. My question is more generally about the policy we want to enforce for this build task.)

-myk

[1] https://phabricator.services.mozilla.com/rMOZILLACENTRAL08fa47a24e89697e4e43177860b55cc28298bbff


_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Better incremental builds - Tup backend ready for early adopters on Linux

Mike Hommey
On Wed, Aug 08, 2018 at 05:24:07PM -0700, Myk Melez wrote:

> Michael Shal wrote on 2018-08-02 08:34:
> >
> > Hi everyone,
> >
> >
> > Tup is a modern build system that enables fast and correct builds. I'm
> > pleased to announce the availability of Tup for building Firefox on
> > Linux. Compared to Make, building with Tup will result in faster
> > incremental build times and reduce the need for clobber builds. See the
> > in-tree docs for details [1].
> >
> Should Btup bustage trigger immediate backout? I appear to have busted it
> with a recent change [1], triggering an immediate backout, and it isn't
> obvious to me that this is the optimal policy for an experimental build
> backend.
>
> (I'll investigate the bustage regardless. My question is more generally
> about the policy we want to enforce for this build task.)

I think now that it has been advertized and that people may be using it,
it should be turned into tier 1.

Mike
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Better incremental builds - Tup backend ready for early adopters on Linux

Michael Shal
In reply to this post by Myk Melez-4
On Wed, Aug 8, 2018 at 8:24 PM, Myk Melez <[hidden email]> wrote:
Michael Shal wrote on 2018-08-02 08:34:

Hi everyone,


Tup is a modern build system that enables fast and correct builds. I'm pleased to announce the availability of Tup for building Firefox on Linux. Compared to Make, building with Tup will result in faster incremental build times and reduce the need for clobber builds. See the in-tree docs for details [1].

Should Btup bustage trigger immediate backout? I appear to have busted it with a recent change [1], triggering an immediate backout, and it isn't obvious to me that this is the optimal policy for an experimental build backend.

(I'll investigate the bustage regardless. My question is more generally about the policy we want to enforce for this build task.)


We created Btup as tier-2 so that any breaking it wouldn't prevent people from landing their other code, with the intention that chmanchester or myself would apply fixes to the tup backend in followup commits. I should've done a better job communicating with the sheriffs to enforce this - sorry about that! But as glandium suggests, it should probably be tier-1 now (or soon) anyway, so I filed bug 1482502 for that.

In any case, if anyone hits bustage in Btup when make-based builds are successful, feel free to grab one of us in #build to help out.

-Mike


_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds