Rethinking build defaults

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Rethinking build defaults

Mike Hommey
Hi everyone,

There's been a lot of traction recently about our builds getting slower
and what we could do about it, and what not.

Starting with bug 904979, I would like to change the way we're thinking
about default flags and options. And I am therefore opening a discussion
about it.

The main thing bug 904979 does is to make release engineering builds (as
well as linux distros, palemoon, icecat, you name it) use a special
--enable-release configure flag to use flags that we deem necessary for
a build of Firefox, the product. The flip side is that builds without
this flag, which matches builds from every developer, obviously, would
use flags that make the build faster. For the moment, on Linux systems,
this means disabling identical code folding and dead code removal (which,
while they make the binary size smaller, increase link time), and
forcing the use of the gold linker when it's available but is not system
default. With bug 905646, it will mean enabling -gsplit-dwarf when it's
available, which make link time on linux really very much faster (<4s
on my machine instead of 30s). We could and should do the same kind
of things for other platforms, with the goal of making linking
libxul.so/xul.dll/XUL faster, making edit-compile-edit cycles faster.
If that works reliably, for instance, we should for instance use
incremental linking. Please feel free to file Core::Build Config bugs
for what you think would help on your favorite build platform (and if
you do, for better tracking, make them depend on bug 904979).

That being said, this is not the discussion I want to have here, that
was merely an introduction.

The web has grown in the past few years, and so has our code base, to
support new technologies. As Nathan noted on his blog[1] disabling
webrtc calls for great build time improvements. And I think it's
something we should address by a switch in strategy.

- How many people are working on webrtc code?
- How many people are working on peripheral code that may affect webrtc?
- How many people are building webrtc code they're not working on and
  not using?

I'm fairly certain the answer to the above is that the latter population
is much bigger than the other two, by probably more than an order of
magnitude.

So here's the discussion opener: why not make things like webrtc (I'm
sure we can find many more[2]) opt-in instead of opt-out, for local,
developer builds? What do you think are good candidates for such a
switch?

Mike

1. https://blog.mozilla.org/nfroyd/2013/08/15/better-build-times-through-configury/
2. and we can already start with ICU, because it's built and not even
used. And to add injury to pain, it's currently built without
parallelism (and the patch to make it not do so was backed out).
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking build defaults

Andreas Gal-2

First of all, thanks for raising this. Its definitely a problem that
needs fixing.

I am not convinced by your approach though. In a few months from now
disabling WebRTC is like calling for the DOM or JS or CSS to be disabled
in local developer builds. It will become a natural part of the core Web
platform.

I would like to propose the opposite approach:

- Remove all conditional feature configuration from configure. WebRTC et
al are always on. Features should be disabled dynamically (prefs), if at
all.
- Reduce configure settings to choice of OS and release or developer.
- Require triple super-reviews (hand signed, in blood) for any changes
to configure.
- Make parts of the code base more modular and avoid super include files
cross modules (hello LayoutUtils.h).

Rationale:

Its not slow for you to build WebRTC. Its slow for you to have it build
over and over. Almost every time I pull from mozilla-central, someone
touched configure and I have to rebuild from scratch, which is
infuriating (argh!). Minimizing changes to configure and banning static
defines for feature management would solve that. If we make sure major
subsystems like WebRTC can stand on their own, you will take a hit
building it one time, and then occasionally as the team lands changes.
Its a pretty small team, so the amount of code they can possibly check
in is actually pretty small. You will see churn all over the three when
you pull, but you won't have to rebuild the entire Universe every time
you pull.

What do you think?

Andreas

Mike Hommey wrote:

> Hi everyone,
>
> There's been a lot of traction recently about our builds getting slower
> and what we could do about it, and what not.
>
> Starting with bug 904979, I would like to change the way we're thinking
> about default flags and options. And I am therefore opening a discussion
> about it.
>
> The main thing bug 904979 does is to make release engineering builds (as
> well as linux distros, palemoon, icecat, you name it) use a special
> --enable-release configure flag to use flags that we deem necessary for
> a build of Firefox, the product. The flip side is that builds without
> this flag, which matches builds from every developer, obviously, would
> use flags that make the build faster. For the moment, on Linux systems,
> this means disabling identical code folding and dead code removal (which,
> while they make the binary size smaller, increase link time), and
> forcing the use of the gold linker when it's available but is not system
> default. With bug 905646, it will mean enabling -gsplit-dwarf when it's
> available, which make link time on linux really very much faster (<4s
> on my machine instead of 30s). We could and should do the same kind
> of things for other platforms, with the goal of making linking
> libxul.so/xul.dll/XUL faster, making edit-compile-edit cycles faster.
> If that works reliably, for instance, we should for instance use
> incremental linking. Please feel free to file Core::Build Config bugs
> for what you think would help on your favorite build platform (and if
> you do, for better tracking, make them depend on bug 904979).
>
> That being said, this is not the discussion I want to have here, that
> was merely an introduction.
>
> The web has grown in the past few years, and so has our code base, to
> support new technologies. As Nathan noted on his blog[1] disabling
> webrtc calls for great build time improvements. And I think it's
> something we should address by a switch in strategy.
>
> - How many people are working on webrtc code?
> - How many people are working on peripheral code that may affect webrtc?
> - How many people are building webrtc code they're not working on and
>    not using?
>
> I'm fairly certain the answer to the above is that the latter population
> is much bigger than the other two, by probably more than an order of
> magnitude.
>
> So here's the discussion opener: why not make things like webrtc (I'm
> sure we can find many more[2]) opt-in instead of opt-out, for local,
> developer builds? What do you think are good candidates for such a
> switch?
>
> Mike
>
> 1. https://blog.mozilla.org/nfroyd/2013/08/15/better-build-times-through-configury/
> 2. and we can already start with ICU, because it's built and not even
> used. And to add injury to pain, it's currently built without
> parallelism (and the patch to make it not do so was backed out).
> _______________________________________________
> dev-platform mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-platform
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking build defaults

Mike Hommey
On Fri, Aug 16, 2013 at 05:43:22PM +0800, Andreas Gal wrote:
>
> First of all, thanks for raising this. Its definitely a problem that
> needs fixing.
>
> I am not convinced by your approach though. In a few months from now
> disabling WebRTC is like calling for the DOM or JS or CSS to be
> disabled in local developer builds. It will become a natural part of
> the core Web platform.

I'm not running my local builds to browse the web. I bet a vast
majority don't. I don't think what we build for local developer builds
should be driven by what defines the web, but what brings useful results
to developers in a timely manner. If disabling non central features can
allow to build significantly faster, it's worth at least talking about
it. No amount of widespread-ness on the web is going to make webrtc as
central as js, layout, or gfx.

If default is not an option, then maybe we can have
--disable-all-the-things subsequently allowing opt-ins like
--enable-webrtc. But I like to think that most people shouldn't have to
edit a .mozconfig.

> I would like to propose the opposite approach:
>
> - Remove all conditional feature configuration from configure.
> WebRTC et al are always on. Features should be disabled dynamically
> (prefs), if at all.
> - Reduce configure settings to choice of OS and release or developer.

With my Debian hat on, let me say this is both x86/x86-64/arm and
Firefox/Firefox-OS centric.

There are features that don't build on non mainstream architectures (hey
webrtc, i'm looking at you), and while I do understand the horror it can
be to some people that there could be a Firefox (^WIceweasel) build that
doesn't support all the web, it's still better to have a browser that
doesn't support everything than no browser at all (and considering i get
bug reports from people using or trying to use iceweasel on e.g ppc or
ia64, yes, there *are* people out there that like or would like to have
a working browser, even if it doesn't make coffee).

And Gecko is not used only to build web browsers (for how long?), so it
makes sense for some features to be disableable at build time.

> - Require triple super-reviews (hand signed, in blood) for any
> changes to configure.
> - Make parts of the code base more modular and avoid super include
> files cross modules (hello LayoutUtils.h).
>
> Rationale:
>
> Its not slow for you to build WebRTC. Its slow for you to have it
> build over and over. Almost every time I pull from mozilla-central,
> someone touched configure and I have to rebuild from scratch, which
> is infuriating (argh!). Minimizing changes to configure and banning
> static defines for feature management would solve that. If we make
> sure major subsystems like WebRTC can stand on their own, you will
> take a hit building it one time, and then occasionally as the team
> lands changes. Its a pretty small team, so the amount of code they
> can possibly check in is actually pretty small. You will see churn
> all over the three when you pull, but you won't have to rebuild the
> entire Universe every time you pull.

With my build-config peer hat on, let me say this may not be as much a
problem as you feel it is. At least not one that requires drastic
restrictive measures.
There are three different things that make configure changes a pain:
- Re-running configure even with no changes, changes some central files,
  and that means rebuilding a lot of things. I recently fixed bug 903341
  (guess what, running configure meant rebuilding all webrtc), and bug
  903321 (running configure meant rebuilding all js). There remains bug
  903369, which makes us rebuild FFI, which makes us relink the js
  library and everything that statically links against it, including
  many tests and libxul. Another one that is yet to file is new and
  similar: running configure runs ICU's configure which makes us rebuild
  ICU, relinking the js library, and so on. When I get 903369 to work on
  windows, it will be easy to fix ICU in a similar way.
- Adding or removing a AC_DEFINE in configure means modifying
  mozilla-config.h, which is included in every single file. Adding or
  removing an AC_DEFINE means rebuilding the entire tree. Now you know
  why I'm always reluctant when i see people adding an AC_DEFINE for
  something that is #ifdef'ed in 3 different directories. bug 902825
  is on file about this. Note we're not adding or removing AC_DEFINEs
  /that/ often.
- Adding or removing a AC_SUBST in configure means modifying
  autoconf.mk if the added variable has a value (if the variable doesn't
  have a value, it ends up in emptyvars.mk, which is treated
  differently). Autoconf.mk is essentially made a dependency of
  everything, so a change in autoconf.mk is going to rebuild everything.
  emptyvars.mk isn't.
  And now that I'm writing this, I'm thinking of an obvious solution to
  the adding/removing AC_SUBST problem: for features enabled by default,
  instead of landing AC_SUBST(ENABLE_FEATURE), land
  AC_SUBST(DISABLE_FEATURE). That would make the variable empty by
  default, not changing autoconf.mk. That's something we can enforce at
  review time, and any change to configure.in should already be going
  through build config peers.

Mike
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking build defaults

Adam Roach
In reply to this post by Mike Hommey
I think the key argument against this approach is that system components are never truly isolated. Sure, some of them can be compiled out and still produce a working system. That doesn't mean that testing without those components is going to have good test coverage.

What I'm worried about, if we start disabling various modules, is that we're going to have regressions that go unnoticed on developer systems, blow up on m-i, and then take a _long_ time to track down. We already have m-i closed for about four hours a day as it is, frequently during prime working hours for a substantial fraction of Mozilla's contributors. Further varying developers' local build environments from those of the builders will only make this problem worse.

/a

On 8/16/13 04:32, Mike Hommey wrote:
Hi everyone,

There's been a lot of traction recently about our builds getting slower
and what we could do about it, and what not.

Starting with bug 904979, I would like to change the way we're thinking
about default flags and options. And I am therefore opening a discussion
about it.

The main thing bug 904979 does is to make release engineering builds (as
well as linux distros, palemoon, icecat, you name it) use a special
--enable-release configure flag to use flags that we deem necessary for
a build of Firefox, the product. The flip side is that builds without
this flag, which matches builds from every developer, obviously, would
use flags that make the build faster. For the moment, on Linux systems,
this means disabling identical code folding and dead code removal (which,
while they make the binary size smaller, increase link time), and
forcing the use of the gold linker when it's available but is not system
default. With bug 905646, it will mean enabling -gsplit-dwarf when it's
available, which make link time on linux really very much faster (<4s
on my machine instead of 30s). We could and should do the same kind
of things for other platforms, with the goal of making linking
libxul.so/xul.dll/XUL faster, making edit-compile-edit cycles faster.
If that works reliably, for instance, we should for instance use
incremental linking. Please feel free to file Core::Build Config bugs
for what you think would help on your favorite build platform (and if
you do, for better tracking, make them depend on bug 904979).

That being said, this is not the discussion I want to have here, that
was merely an introduction.

The web has grown in the past few years, and so has our code base, to
support new technologies. As Nathan noted on his blog[1] disabling
webrtc calls for great build time improvements. And I think it's
something we should address by a switch in strategy.

- How many people are working on webrtc code?
- How many people are working on peripheral code that may affect webrtc?
- How many people are building webrtc code they're not working on and
  not using?

I'm fairly certain the answer to the above is that the latter population
is much bigger than the other two, by probably more than an order of
magnitude.

So here's the discussion opener: why not make things like webrtc (I'm
sure we can find many more[2]) opt-in instead of opt-out, for local,
developer builds? What do you think are good candidates for such a
switch?

Mike

1. https://blog.mozilla.org/nfroyd/2013/08/15/better-build-times-through-configury/
2. and we can already start with ICU, because it's built and not even
used. And to add injury to pain, it's currently built without
parallelism (and the patch to make it not do so was backed out).
_______________________________________________
dev-platform mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-platform


--
Adam Roach
Principal Platform Engineer
[hidden email]
+1 650 903 0800 x863

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking build defaults

Andrew  McCreight


----- Original Message -----

> I think the key argument against this approach is that system components
> are never truly isolated. Sure, some of them can be compiled out and
> still produce a working system. That doesn't mean that testing without
> those components is going to have good test coverage.
>
> What I'm worried about, if we start disabling various modules, is that
> we're going to have regressions that go unnoticed on developer systems,
> blow up on m-i, and then take a _long_ time to track down. We already
> have m-i closed for about four hours a day as it is, frequently during
> prime working hours for a substantial fraction of Mozilla's
> contributors. Further varying developers' local build environments from
> those of the builders will only make this problem worse.

I imagine that most people only run tests locally that are related to the code they are working on, so it doesn't seem like it would make things any worse.

Andrew

>
> /a
>
> On 8/16/13 04:32, Mike Hommey wrote:
> > Hi everyone,
> >
> > There's been a lot of traction recently about our builds getting slower
> > and what we could do about it, and what not.
> >
> > Starting with bug 904979, I would like to change the way we're thinking
> > about default flags and options. And I am therefore opening a discussion
> > about it.
> >
> > The main thing bug 904979 does is to make release engineering builds (as
> > well as linux distros, palemoon, icecat, you name it) use a special
> > --enable-release configure flag to use flags that we deem necessary for
> > a build of Firefox, the product. The flip side is that builds without
> > this flag, which matches builds from every developer, obviously, would
> > use flags that make the build faster. For the moment, on Linux systems,
> > this means disabling identical code folding and dead code removal (which,
> > while they make the binary size smaller, increase link time), and
> > forcing the use of the gold linker when it's available but is not system
> > default. With bug 905646, it will mean enabling -gsplit-dwarf when it's
> > available, which make link time on linux really very much faster (<4s
> > on my machine instead of 30s). We could and should do the same kind
> > of things for other platforms, with the goal of making linking
> > libxul.so/xul.dll/XUL faster, making edit-compile-edit cycles faster.
> > If that works reliably, for instance, we should for instance use
> > incremental linking. Please feel free to file Core::Build Config bugs
> > for what you think would help on your favorite build platform (and if
> > you do, for better tracking, make them depend on bug 904979).
> >
> > That being said, this is not the discussion I want to have here, that
> > was merely an introduction.
> >
> > The web has grown in the past few years, and so has our code base, to
> > support new technologies. As Nathan noted on his blog[1] disabling
> > webrtc calls for great build time improvements. And I think it's
> > something we should address by a switch in strategy.
> >
> > - How many people are working on webrtc code?
> > - How many people are working on peripheral code that may affect webrtc?
> > - How many people are building webrtc code they're not working on and
> >    not using?
> >
> > I'm fairly certain the answer to the above is that the latter population
> > is much bigger than the other two, by probably more than an order of
> > magnitude.
> >
> > So here's the discussion opener: why not make things like webrtc (I'm
> > sure we can find many more[2]) opt-in instead of opt-out, for local,
> > developer builds? What do you think are good candidates for such a
> > switch?
> >
> > Mike
> >
> > 1.
> > https://blog.mozilla.org/nfroyd/2013/08/15/better-build-times-through-configury/
> > 2. and we can already start with ICU, because it's built and not even
> > used. And to add injury to pain, it's currently built without
> > parallelism (and the patch to make it not do so was backed out).
> > _______________________________________________
> > dev-platform mailing list
> > [hidden email]
> > https://lists.mozilla.org/listinfo/dev-platform
>
>
> --
> Adam Roach
> Principal Platform Engineer
> [hidden email]
> +1 650 903 0800 x863
> _______________________________________________
> dev-platform mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-platform
>
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking build defaults

Thomas Zimmermann-2
On 16.08.2013 16:23, Andrew McCreight wrote:

>
> ----- Original Message -----
>> I think the key argument against this approach is that system components
>> are never truly isolated. Sure, some of them can be compiled out and
>> still produce a working system. That doesn't mean that testing without
>> those components is going to have good test coverage.
>>
>> What I'm worried about, if we start disabling various modules, is that
>> we're going to have regressions that go unnoticed on developer systems,
>> blow up on m-i, and then take a _long_ time to track down. We already
>> have m-i closed for about four hours a day as it is, frequently during
>> prime working hours for a substantial fraction of Mozilla's
>> contributors. Further varying developers' local build environments from
>> those of the builders will only make this problem worse.
> I imagine that most people only run tests locally that are related to the code they are working on, so it doesn't seem like it would make things any worse.

I would expect that the number of build failures increases if people
disable parts of the code locally. Everything works for them, but when
they push to m-i, things don't build any longer because the patches are
incomplete.

Best regards
Thomas


>
> Andrew
>
>> /a
>>
>> On 8/16/13 04:32, Mike Hommey wrote:
>>> Hi everyone,
>>>
>>> There's been a lot of traction recently about our builds getting slower
>>> and what we could do about it, and what not.
>>>
>>> Starting with bug 904979, I would like to change the way we're thinking
>>> about default flags and options. And I am therefore opening a discussion
>>> about it.
>>>
>>> The main thing bug 904979 does is to make release engineering builds (as
>>> well as linux distros, palemoon, icecat, you name it) use a special
>>> --enable-release configure flag to use flags that we deem necessary for
>>> a build of Firefox, the product. The flip side is that builds without
>>> this flag, which matches builds from every developer, obviously, would
>>> use flags that make the build faster. For the moment, on Linux systems,
>>> this means disabling identical code folding and dead code removal (which,
>>> while they make the binary size smaller, increase link time), and
>>> forcing the use of the gold linker when it's available but is not system
>>> default. With bug 905646, it will mean enabling -gsplit-dwarf when it's
>>> available, which make link time on linux really very much faster (<4s
>>> on my machine instead of 30s). We could and should do the same kind
>>> of things for other platforms, with the goal of making linking
>>> libxul.so/xul.dll/XUL faster, making edit-compile-edit cycles faster.
>>> If that works reliably, for instance, we should for instance use
>>> incremental linking. Please feel free to file Core::Build Config bugs
>>> for what you think would help on your favorite build platform (and if
>>> you do, for better tracking, make them depend on bug 904979).
>>>
>>> That being said, this is not the discussion I want to have here, that
>>> was merely an introduction.
>>>
>>> The web has grown in the past few years, and so has our code base, to
>>> support new technologies. As Nathan noted on his blog[1] disabling
>>> webrtc calls for great build time improvements. And I think it's
>>> something we should address by a switch in strategy.
>>>
>>> - How many people are working on webrtc code?
>>> - How many people are working on peripheral code that may affect webrtc?
>>> - How many people are building webrtc code they're not working on and
>>>    not using?
>>>
>>> I'm fairly certain the answer to the above is that the latter population
>>> is much bigger than the other two, by probably more than an order of
>>> magnitude.
>>>
>>> So here's the discussion opener: why not make things like webrtc (I'm
>>> sure we can find many more[2]) opt-in instead of opt-out, for local,
>>> developer builds? What do you think are good candidates for such a
>>> switch?
>>>
>>> Mike
>>>
>>> 1.
>>> https://blog.mozilla.org/nfroyd/2013/08/15/better-build-times-through-configury/
>>> 2. and we can already start with ICU, because it's built and not even
>>> used. And to add injury to pain, it's currently built without
>>> parallelism (and the patch to make it not do so was backed out).
>>> _______________________________________________
>>> dev-platform mailing list
>>> [hidden email]
>>> https://lists.mozilla.org/listinfo/dev-platform
>>
>> --
>> Adam Roach
>> Principal Platform Engineer
>> [hidden email]
>> +1 650 903 0800 x863
>> _______________________________________________
>> dev-platform mailing list
>> [hidden email]
>> https://lists.mozilla.org/listinfo/dev-platform
>>
> _______________________________________________
> dev-platform mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-platform

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking build defaults

Justin Lebar-3
In reply to this post by Andreas Gal-2
> - Remove all conditional feature configuration from configure. WebRTC et al are always
>   on. Features should be disabled dynamically (prefs), if at all.

Note that Fabrice has seen sizable memory-usage wins on B2G by
disabling non-web features such as printing and XUL.  That alone is,
in my mind, a very strong argument for keeping at least some of these
configure options around.

On Fri, Aug 16, 2013 at 2:43 AM, Andreas Gal <[hidden email]> wrote:

>
> First of all, thanks for raising this. Its definitely a problem that needs
> fixing.
>
> I am not convinced by your approach though. In a few months from now
> disabling WebRTC is like calling for the DOM or JS or CSS to be disabled in
> local developer builds. It will become a natural part of the core Web
> platform.
>
> I would like to propose the opposite approach:
>
> - Remove all conditional feature configuration from configure. WebRTC et al
> are always on. Features should be disabled dynamically (prefs), if at all.
> - Reduce configure settings to choice of OS and release or developer.
> - Require triple super-reviews (hand signed, in blood) for any changes to
> configure.
> - Make parts of the code base more modular and avoid super include files
> cross modules (hello LayoutUtils.h).
>
> Rationale:
>
> Its not slow for you to build WebRTC. Its slow for you to have it build over
> and over. Almost every time I pull from mozilla-central, someone touched
> configure and I have to rebuild from scratch, which is infuriating (argh!).
> Minimizing changes to configure and banning static defines for feature
> management would solve that. If we make sure major subsystems like WebRTC
> can stand on their own, you will take a hit building it one time, and then
> occasionally as the team lands changes. Its a pretty small team, so the
> amount of code they can possibly check in is actually pretty small. You will
> see churn all over the three when you pull, but you won't have to rebuild
> the entire Universe every time you pull.
>
> What do you think?
>
> Andreas
>
>
> Mike Hommey wrote:
>>
>> Hi everyone,
>>
>> There's been a lot of traction recently about our builds getting slower
>> and what we could do about it, and what not.
>>
>> Starting with bug 904979, I would like to change the way we're thinking
>> about default flags and options. And I am therefore opening a discussion
>> about it.
>>
>> The main thing bug 904979 does is to make release engineering builds (as
>> well as linux distros, palemoon, icecat, you name it) use a special
>> --enable-release configure flag to use flags that we deem necessary for
>> a build of Firefox, the product. The flip side is that builds without
>> this flag, which matches builds from every developer, obviously, would
>> use flags that make the build faster. For the moment, on Linux systems,
>> this means disabling identical code folding and dead code removal (which,
>> while they make the binary size smaller, increase link time), and
>> forcing the use of the gold linker when it's available but is not system
>> default. With bug 905646, it will mean enabling -gsplit-dwarf when it's
>> available, which make link time on linux really very much faster (<4s
>> on my machine instead of 30s). We could and should do the same kind
>> of things for other platforms, with the goal of making linking
>> libxul.so/xul.dll/XUL faster, making edit-compile-edit cycles faster.
>> If that works reliably, for instance, we should for instance use
>> incremental linking. Please feel free to file Core::Build Config bugs
>> for what you think would help on your favorite build platform (and if
>> you do, for better tracking, make them depend on bug 904979).
>>
>> That being said, this is not the discussion I want to have here, that
>> was merely an introduction.
>>
>> The web has grown in the past few years, and so has our code base, to
>> support new technologies. As Nathan noted on his blog[1] disabling
>> webrtc calls for great build time improvements. And I think it's
>> something we should address by a switch in strategy.
>>
>> - How many people are working on webrtc code?
>> - How many people are working on peripheral code that may affect webrtc?
>> - How many people are building webrtc code they're not working on and
>>    not using?
>>
>> I'm fairly certain the answer to the above is that the latter population
>> is much bigger than the other two, by probably more than an order of
>> magnitude.
>>
>> So here's the discussion opener: why not make things like webrtc (I'm
>> sure we can find many more[2]) opt-in instead of opt-out, for local,
>> developer builds? What do you think are good candidates for such a
>> switch?
>>
>> Mike
>>
>> 1.
>> https://blog.mozilla.org/nfroyd/2013/08/15/better-build-times-through-configury/
>> 2. and we can already start with ICU, because it's built and not even
>> used. And to add injury to pain, it's currently built without
>> parallelism (and the patch to make it not do so was backed out).
>> _______________________________________________
>> dev-platform mailing list
>> [hidden email]
>> https://lists.mozilla.org/listinfo/dev-platform
>
> _______________________________________________
> dev-platform mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-platform
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking build defaults

Trevor Saunders
In reply to this post by Adam Roach
On Fri, Aug 16, 2013 at 09:14:56AM -0500, Adam Roach wrote:
> I think the key argument against this approach is that system components are
> never truly isolated. Sure, some of them can be compiled out and still produce
> a working system. That doesn't mean that testing without those components is
> going to have good test coverage.

obviously you won't be able to test the effects your changes have on
things you've disabled, but chances are very good you wheren't going to
test those locally anyway.  There's some risk you'll break building
those things, but that's the same problem as code that we only build on
one platform.

> What I'm worried about, if we start disabling various modules, is that we're
> going to have regressions that go unnoticed on developer systems, blow up on
> m-i, and then take a _long_ time to track down. We already have m-i closed for
> about four hours a day as it is, frequently during prime working hours for a
> substantial fraction of Mozilla's contributors. Further varying developers'
> local build environments from those of the builders will only make this problem
> worse.

 I and I believe several others have been doing thins sort of thing
 ourselves for a long time with very few or no issues.  I've burned the
 tree because of platform specific stuff any number of times, but I
 don't think I've ever done it because of disabling webrtc / webm / ogg
 / crash reporter.   If this does cause bustage it will probably be by
 breaking the build which shouldn't be too slow, and if disabling
 certain things by default turns out to be a bad idea we can of course
 reenable them.

 Trev

>
> /a
>
> On 8/16/13 04:32, Mike Hommey wrote:
>
>     Hi everyone,
>
>     There's been a lot of traction recently about our builds getting slower
>     and what we could do about it, and what not.
>
>     Starting with bug 904979, I would like to change the way we're thinking
>     about default flags and options. And I am therefore opening a discussion
>     about it.
>
>     The main thing bug 904979 does is to make release engineering builds (as
>     well as linux distros, palemoon, icecat, you name it) use a special
>     --enable-release configure flag to use flags that we deem necessary for
>     a build of Firefox, the product. The flip side is that builds without
>     this flag, which matches builds from every developer, obviously, would
>     use flags that make the build faster. For the moment, on Linux systems,
>     this means disabling identical code folding and dead code removal (which,
>     while they make the binary size smaller, increase link time), and
>     forcing the use of the gold linker when it's available but is not system
>     default. With bug 905646, it will mean enabling -gsplit-dwarf when it's
>     available, which make link time on linux really very much faster (<4s
>     on my machine instead of 30s). We could and should do the same kind
>     of things for other platforms, with the goal of making linking
>     libxul.so/xul.dll/XUL faster, making edit-compile-edit cycles faster.
>     If that works reliably, for instance, we should for instance use
>     incremental linking. Please feel free to file Core::Build Config bugs
>     for what you think would help on your favorite build platform (and if
>     you do, for better tracking, make them depend on bug 904979).
>
>     That being said, this is not the discussion I want to have here, that
>     was merely an introduction.
>
>     The web has grown in the past few years, and so has our code base, to
>     support new technologies. As Nathan noted on his blog[1] disabling
>     webrtc calls for great build time improvements. And I think it's
>     something we should address by a switch in strategy.
>
>     - How many people are working on webrtc code?
>     - How many people are working on peripheral code that may affect webrtc?
>     - How many people are building webrtc code they're not working on and
>       not using?
>
>     I'm fairly certain the answer to the above is that the latter population
>     is much bigger than the other two, by probably more than an order of
>     magnitude.
>
>     So here's the discussion opener: why not make things like webrtc (I'm
>     sure we can find many more[2]) opt-in instead of opt-out, for local,
>     developer builds? What do you think are good candidates for such a
>     switch?
>
>     Mike
>
>     1. https://blog.mozilla.org/nfroyd/2013/08/15/better-build-times-through-configury/
>     2. and we can already start with ICU, because it's built and not even
>     used. And to add injury to pain, it's currently built without
>     parallelism (and the patch to make it not do so was backed out).
>     _______________________________________________
>     dev-platform mailing list
>     [hidden email]
>     https://lists.mozilla.org/listinfo/dev-platform
>
>
>
> --
> [cid] Adam Roach
> Principal Platform Engineer
> [hidden email]
> +1 650 903 0800 x863

> _______________________________________________
> dev-builds mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-builds

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking build defaults

Ed Morley-2
In reply to this post by Adam Roach
On 16/08/2013 15:14, Adam Roach wrote:
> What I'm worried about, if we start disabling various modules, is that
> we're going to have regressions that go unnoticed on developer systems,
> blow up on m-i, and then take a _long_ time to track down.

They shouldn't take a long time to track down - a simple backout + let
the dev build locally without the "disable foo, bar & baz by default"
pref set, will do the trick.

It's also worth bearing in mind that we already have many cases of devs
not testing locally (or not even building locally!) before pushing to
inbound et al. There is a good chance that less people will fall into
that group, if we speed up their local builds by default. As such, even
if this proposal caused additional instances of broken builds when
people pushed to inbound, it may still be outweighed by the reduced
breakage from more devs testing locally.

Best wishes,

Ed
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking build defaults

Brian Smith-19
In reply to this post by Andreas Gal-2
On Fri, Aug 16, 2013 at 2:43 AM, Andreas Gal <[hidden email]> wrote:
I would like to propose the opposite approach:

- Remove all conditional feature configuration from configure. WebRTC et al are always on. Features should be disabled dynamically (prefs), if at all.
- Reduce configure settings to choice of OS and release or developer.
- Require triple super-reviews (hand signed, in blood) for any changes to configure.
- Make parts of the code base more modular and avoid super include files cross modules (hello LayoutUtils.h).

I would much rather we try (a milder version of) what Andreas suggests before we start disabling shipping features by default in developer builds. If we had closer to 100% test coverage then turning things off by default might maybe make sense. But, as it is, it is important that any manual testing we do is testing something as close as possible to what we ship.

I do think we could benefit from a much simpler build system with fewer options that result in us modifying configure.in and other rebuild-the-world dependencies much less often. I don't think we should completely eliminate conditionals in our build system. Like jlebar, I have found there are cases where we can eliminate a lot of dead code on a per-platform basis. However, we should be able to do said dead-code-removal conditional compilation without modifying configure.in. Maybe that is as simple as allowing support for a mozconfig-like addendum mechanism for configure.in that lets somebody enable off-by-default features in their local build without first needing to modify configure.in in a way that would affect everybody else's builds.

If I think back to some of my recent changes to configure.in, they were often changes that only affected one or two modules, and that logic could have been done in a whatever.mk file or moz.build file that was included by the affected modules instead of modifying configure.in. The only reason I modified configure.in is because that's how we've always done it. For my next such change, I will try to localize the conditional logic to my module to avoid touching configure.in and whatever/confvars.sh. Perhaps others can try the same experiment.

Cheers,
Brian

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking build defaults

Brian Smith-19
In reply to this post by Mike Hommey
On Fri, Aug 16, 2013 at 4:27 AM, Mike Hommey <[hidden email]> wrote:
> - Remove all conditional feature configuration from configure.
> WebRTC et al are always on. Features should be disabled dynamically
> (prefs), if at all.
> - Reduce configure settings to choice of OS and release or developer.

With my Debian hat on, let me say this is both x86/x86-64/arm and
Firefox/Firefox-OS centric.

I understand that you see that as a negative, but I see that as a big positive because it maximizes the benefit:cost ratio of platform support.
 
There are features that don't build on non mainstream architectures (hey
webrtc, i'm looking at you), and while I do understand the horror it can
be to some people that there could be a Firefox (^WIceweasel) build that
doesn't support all the web, it's still better to have a browser that
doesn't support everything than no browser at all (and considering i get
bug reports from people using or trying to use iceweasel on e.g ppc or
ia64, yes, there *are* people out there that like or would like to have
a working browser, even if it doesn't make coffee).

And Gecko is not used only to build web browsers (for how long?), so it
makes sense for some features to be disableable at build time.

If it helps at all, I'd be happy for us to stop trying to support any (product, target platform, build platform, toolchain) combination in mozilla-central that isn't (planned to be) used in a normal mozilla-central TBPL run: https://tbpl.mozilla.org/.

Cheers,
Brian

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds