Resolving which build system entry points need to exist

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Resolving which build system entry points need to exist

Gregory Szorc-3
As I'm working to remove client.mk, I'm encountering quite the spiderweb of dependencies between:

* mach
* configure.in and configure
* moz.configure / configure.py
* config.status
* build backend (Makefile.in, etc)

Today, it is possible to bypass our frontends (mach and client.mk) and invoke the underlying components (configure, moz.configure, Makefile) directly. This results in hacks like configure.in being both a valid shell and m4 file. (The build system copies it to configure but you can also run `autoconf` to achieve a similar result.)

While my work to remove client.mk is centered around eliminating the redundancy between `mach` and client.mk by effectively moving logic into mach, as I'm touching the code it is obvious that all the code around configure[.in], moz.configure, config.status, etc could use a major overhaul. I'd love to say that "`mach` is the only supported interface to build Firefox." After all, from my perspective as a build system maintainer, it is much, much easier to support 1 thing instead of N>1.

This poses the question: is it important to preserve the illusion that our build system is following autotools "standards"? If not, can we declare `mach` is the only supported interface and [re]move things like configure.in, configure, configure.py, config.status, and Makefile.in at our leisure?

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Resolving which build system entry points need to exist

Nicholas Alexander


On Fri, Nov 17, 2017 at 1:06 PM, Gregory Szorc <[hidden email]> wrote:
As I'm working to remove client.mk, I'm encountering quite the spiderweb of dependencies between:

* mach
* configure.in and configure
* moz.configure / configure.py
* config.status
* build backend (Makefile.in, etc)

Today, it is possible to bypass our frontends (mach and client.mk) and invoke the underlying components (configure, moz.configure, Makefile) directly. This results in hacks like configure.in being both a valid shell and m4 file. (The build system copies it to configure but you can also run `autoconf` to achieve a similar result.)

While my work to remove client.mk is centered around eliminating the redundancy between `mach` and client.mk by effectively moving logic into mach, as I'm touching the code it is obvious that all the code around configure[.in], moz.configure, config.status, etc could use a major overhaul. I'd love to say that "`mach` is the only supported interface to build Firefox." After all, from my perspective as a build system maintainer, it is much, much easier to support 1 thing instead of N>1.

This poses the question: is it important to preserve the illusion that our build system is following autotools "standards"? If not, can we declare `mach` is the only supported interface and [re]move things like configure.in, configure, configure.py, config.status, and Makefile.in at our leisure?

Yes.  We're a million miles away from autoconf at this point, and trying to pretend we're not holds us back in a million ways.

Nick

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Resolving which build system entry points need to exist

mozilla
In reply to this post by Gregory Szorc-3
It would be nice if the entry points that comm-central's buildbot configuration continued working until after 59 branches. I am planning to not use buildbot for the 59 release, but I'd like to have buildbot still working as a fallback.

I'm not sure exactly what interfaces it is using, but I can do some work to figure it out. Off-hand, I know it at least calls `./configure` and calls `make` in the objdir. These happen via the `client.mk` in comm-central, directly in the buildbot configs and in various scripts in the build-tools repo.

On Fri, Nov 17, 2017 at 2:06 PM Gregory Szorc <[hidden email]> wrote:
As I'm working to remove client.mk, I'm encountering quite the spiderweb of dependencies between:

* mach
* configure.in and configure
* moz.configure / configure.py
* config.status
* build backend (Makefile.in, etc)

Today, it is possible to bypass our frontends (mach and client.mk) and invoke the underlying components (configure, moz.configure, Makefile) directly. This results in hacks like configure.in being both a valid shell and m4 file. (The build system copies it to configure but you can also run `autoconf` to achieve a similar result.)

While my work to remove client.mk is centered around eliminating the redundancy between `mach` and client.mk by effectively moving logic into mach, as I'm touching the code it is obvious that all the code around configure[.in], moz.configure, config.status, etc could use a major overhaul. I'd love to say that "`mach` is the only supported interface to build Firefox." After all, from my perspective as a build system maintainer, it is much, much easier to support 1 thing instead of N>1.

This poses the question: is it important to preserve the illusion that our build system is following autotools "standards"? If not, can we declare `mach` is the only supported interface and [re]move things like configure.in, configure, configure.py, config.status, and Makefile.in at our leisure?
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Resolving which build system entry points need to exist

Mike Hommey
In reply to this post by Gregory Szorc-3
On Fri, Nov 17, 2017 at 01:06:05PM -0800, Gregory Szorc wrote:

> As I'm working to remove client.mk, I'm encountering quite the spiderweb of
> dependencies between:
>
> * mach
> * client.mk
> * configure.in and configure
> * moz.configure / configure.py
> * config.status
> * build backend (Makefile.in, etc)
>
> Today, it is possible to bypass our frontends (mach and client.mk) and
> invoke the underlying components (configure, moz.configure, Makefile)
> directly. This results in hacks like configure.in being both a valid shell
> and m4 file. (The build system copies it to configure but you can also run
> `autoconf` to achieve a similar result.)
>
> While my work to remove client.mk is centered around eliminating the
> redundancy between `mach` and client.mk by effectively moving logic into
> mach, as I'm touching the code it is obvious that all the code around
> configure[.in], moz.configure, config.status, etc could use a major
> overhaul. I'd love to say that "`mach` is the only supported interface to
> build Firefox." After all, from my perspective as a build system
> maintainer, it is much, much easier to support 1 thing instead of N>1.
>
> This poses the question: is it important to preserve the illusion that our
> build system is following autotools "standards"? If not, can we declare
> `mach` is the only supported interface and [re]move things like configure.in,
> configure, configure.py, config.status, and Makefile.in at our leisure?

I'd rather preserve the configure && make way of building things, but
I'll note that the only reason we still have configure.in is only
partially because of that.

The real problem is that if you add configure to the repository, hg
update fails because of the configure file already in the working
tree.

Yes, the problem would not exist if the file had a different name.

That said, now that I think about it from a different perspective, we
could do this in a few steps to avoid disruption for most people that
update their tree frequently enough:
- Step 1: Make client.mk (or whatever replaces it) create and use a
  different file name *and* remove the configure file.
- Step 2, few days/weeks later: rename configure.in to configure.

As for the reason configure.in can still be "parsed" by autoconf?
Because until recently we still had buildbot jobs that ran autoconf
manually from out-of-tree scripts.

Mike
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Resolving which build system entry points need to exist

Gregory Szorc-3
On Fri, Nov 17, 2017 at 2:08 PM, Mike Hommey <[hidden email]> wrote:
On Fri, Nov 17, 2017 at 01:06:05PM -0800, Gregory Szorc wrote:
> As I'm working to remove client.mk, I'm encountering quite the spiderweb of
> dependencies between:
>
> * mach
> * client.mk
> * configure.in and configure
> * moz.configure / configure.py
> * config.status
> * build backend (Makefile.in, etc)
>
> Today, it is possible to bypass our frontends (mach and client.mk) and
> invoke the underlying components (configure, moz.configure, Makefile)
> directly. This results in hacks like configure.in being both a valid shell
> and m4 file. (The build system copies it to configure but you can also run
> `autoconf` to achieve a similar result.)
>
> While my work to remove client.mk is centered around eliminating the
> redundancy between `mach` and client.mk by effectively moving logic into
> mach, as I'm touching the code it is obvious that all the code around
> configure[.in], moz.configure, config.status, etc could use a major
> overhaul. I'd love to say that "`mach` is the only supported interface to
> build Firefox." After all, from my perspective as a build system
> maintainer, it is much, much easier to support 1 thing instead of N>1.
>
> This poses the question: is it important to preserve the illusion that our
> build system is following autotools "standards"? If not, can we declare
> `mach` is the only supported interface and [re]move things like configure.in,
> configure, configure.py, config.status, and Makefile.in at our leisure?

I'd rather preserve the configure && make way of building things, but
I'll note that the only reason we still have configure.in is only
partially because of that.

The real problem is that if you add configure to the repository, hg
update fails because of the configure file already in the working
tree.

Yes, the problem would not exist if the file had a different name.

That said, now that I think about it from a different perspective, we
could do this in a few steps to avoid disruption for most people that
update their tree frequently enough:
- Step 1: Make client.mk (or whatever replaces it) create and use a
  different file name *and* remove the configure file.
- Step 2, few days/weeks later: rename configure.in to configure.

I was going to suggest moving everything out of the root directory. That way there are fewer files there to confuse people. But we can obviously only do that if we stop supporting `configure && make`.
 

As for the reason configure.in can still be "parsed" by autoconf?
Because until recently we still had buildbot jobs that ran autoconf
manually from out-of-tree scripts.

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Resolving which build system entry points need to exist

Mike Hommey
On Fri, Nov 17, 2017 at 02:13:29PM -0800, Gregory Szorc wrote:

> On Fri, Nov 17, 2017 at 2:08 PM, Mike Hommey <[hidden email]> wrote:
>
> > On Fri, Nov 17, 2017 at 01:06:05PM -0800, Gregory Szorc wrote:
> > > As I'm working to remove client.mk, I'm encountering quite the
> > spiderweb of
> > > dependencies between:
> > >
> > > * mach
> > > * client.mk
> > > * configure.in and configure
> > > * moz.configure / configure.py
> > > * config.status
> > > * build backend (Makefile.in, etc)
> > >
> > > Today, it is possible to bypass our frontends (mach and client.mk) and
> > > invoke the underlying components (configure, moz.configure, Makefile)
> > > directly. This results in hacks like configure.in being both a valid
> > shell
> > > and m4 file. (The build system copies it to configure but you can also
> > run
> > > `autoconf` to achieve a similar result.)
> > >
> > > While my work to remove client.mk is centered around eliminating the
> > > redundancy between `mach` and client.mk by effectively moving logic into
> > > mach, as I'm touching the code it is obvious that all the code around
> > > configure[.in], moz.configure, config.status, etc could use a major
> > > overhaul. I'd love to say that "`mach` is the only supported interface to
> > > build Firefox." After all, from my perspective as a build system
> > > maintainer, it is much, much easier to support 1 thing instead of N>1.
> > >
> > > This poses the question: is it important to preserve the illusion that
> > our
> > > build system is following autotools "standards"? If not, can we declare
> > > `mach` is the only supported interface and [re]move things like
> > configure.in,
> > > configure, configure.py, config.status, and Makefile.in at our leisure?
> >
> > I'd rather preserve the configure && make way of building things, but
> > I'll note that the only reason we still have configure.in is only
> > partially because of that.
> >
> > The real problem is that if you add configure to the repository, hg
> > update fails because of the configure file already in the working
> > tree.
> >
>
> > Yes, the problem would not exist if the file had a different name.
> >
> > That said, now that I think about it from a different perspective, we
> > could do this in a few steps to avoid disruption for most people that
> > update their tree frequently enough:
> > - Step 1: Make client.mk (or whatever replaces it) create and use a
> >   different file name *and* remove the configure file.
> > - Step 2, few days/weeks later: rename configure.in to configure.
> >
>
> I was going to suggest moving everything out of the root directory. That
> way there are fewer files there to confuse people. But we can obviously
> only do that if we stop supporting `configure && make`.

Note, I would be less opposed to stop supporting configure && make if
mach didn't fumble its auto-objdir guessing so much (although maybe I
fixed all the problems there, but I'm not sure).

Now, we could also just put the file in VCS, remove it from
.gitignore/.hgignore, and tell people they have to remove it manually
before updating. They'll be annoyed only once...

... or every time they bisect... dammit we can't have nice things, can
we?

Mike
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Resolving which build system entry points need to exist

Nicholas Alexander


On Fri, Nov 17, 2017 at 2:19 PM, Mike Hommey <[hidden email]> wrote:
On Fri, Nov 17, 2017 at 02:13:29PM -0800, Gregory Szorc wrote:
> On Fri, Nov 17, 2017 at 2:08 PM, Mike Hommey <[hidden email]> wrote:
>
> > On Fri, Nov 17, 2017 at 01:06:05PM -0800, Gregory Szorc wrote:
> > > As I'm working to remove client.mk, I'm encountering quite the
> > spiderweb of
> > > dependencies between:
> > >
> > > * mach
> > > * client.mk
> > > * configure.in and configure
> > > * moz.configure / configure.py
> > > * config.status
> > > * build backend (Makefile.in, etc)
> > >
> > > Today, it is possible to bypass our frontends (mach and client.mk) and
> > > invoke the underlying components (configure, moz.configure, Makefile)
> > > directly. This results in hacks like configure.in being both a valid
> > shell
> > > and m4 file. (The build system copies it to configure but you can also
> > run
> > > `autoconf` to achieve a similar result.)
> > >
> > > While my work to remove client.mk is centered around eliminating the
> > > redundancy between `mach` and client.mk by effectively moving logic into
> > > mach, as I'm touching the code it is obvious that all the code around
> > > configure[.in], moz.configure, config.status, etc could use a major
> > > overhaul. I'd love to say that "`mach` is the only supported interface to
> > > build Firefox." After all, from my perspective as a build system
> > > maintainer, it is much, much easier to support 1 thing instead of N>1.
> > >
> > > This poses the question: is it important to preserve the illusion that
> > our
> > > build system is following autotools "standards"? If not, can we declare
> > > `mach` is the only supported interface and [re]move things like
> > configure.in,
> > > configure, configure.py, config.status, and Makefile.in at our leisure?
> >
> > I'd rather preserve the configure && make way of building things, but
> > I'll note that the only reason we still have configure.in is only
> > partially because of that.
> >
> > The real problem is that if you add configure to the repository, hg
> > update fails because of the configure file already in the working
> > tree.
> >
>
> > Yes, the problem would not exist if the file had a different name.
> >
> > That said, now that I think about it from a different perspective, we
> > could do this in a few steps to avoid disruption for most people that
> > update their tree frequently enough:
> > - Step 1: Make client.mk (or whatever replaces it) create and use a
> >   different file name *and* remove the configure file.
> > - Step 2, few days/weeks later: rename configure.in to configure.
> >
>
> I was going to suggest moving everything out of the root directory. That
> way there are fewer files there to confuse people. But we can obviously
> only do that if we stop supporting `configure && make`.

Note, I would be less opposed to stop supporting configure && make if
mach didn't fumble its auto-objdir guessing so much (although maybe I
fixed all the problems there, but I'm not sure).

I expect that one |mach| invocation to rule them all make it much easier to stop fumbling objdir guessing.  And many other similar situations where the plethora of parts all have to implement the same heuristics, and agree across them all

Now, we could also just put the file in VCS, remove it from
.gitignore/.hgignore, and tell people they have to remove it manually
before updating. They'll be annoyed only once...

... or every time they bisect... dammit we can't have nice things, can
we?

That I have no particular solution -- or hope of a solution -- for :(

Nick

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Resolving which build system entry points need to exist

ISHIKAWA,chiaki
In reply to this post by Gregory Szorc-3
On 2017/11/18 6:29, Tom Prince wrote:

> It would be nice if the entry points that comm-central's buildbot
> configuration continued working until after 59 branches. I am planning
> to not use buildbot for the 59 release, but I'd like to have buildbot
> still working as a fallback.
>
> I'm not sure exactly what interfaces it is using, but I can do some work
> to figure it out. Off-hand, I know it at least calls `./configure` and
> calls `make` in the objdir. These happen via the `client.mk
> <http://client.mk>` in comm-central, directly in the buildbot configs
> and in various scripts in the build-tools repo.
>

I am, like Tom, contributing C-C patches, and I have a nagging feeling
that if client.mk and configure disappears, something bad may happen.
But, right now it  is a feeling only.

I suspect C-C developers will notice the bust caused by the new change
first.


> On Fri, Nov 17, 2017 at 2:06 PM Gregory Szorc <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     As I'm working to remove client.mk <http://client.mk>, I'm
>     encountering quite the spiderweb of dependencies between:
>
>     * mach
>     * client.mk <http://client.mk>
>     * configure.in <http://configure.in> and configure
>     * moz.configure / configure.py
>     * config.status
>     * build backend (Makefile.in, etc)
>
>     Today, it is possible to bypass our frontends (mach and client.mk
>     <http://client.mk>) and invoke the underlying components (configure,
>     moz.configure, Makefile) directly. This results in hacks like
>     configure.in <http://configure.in> being both a valid shell and m4
>     file. (The build system copies it to configure but you can also run
>     `autoconf` to achieve a similar result.)
>
>     While my work to remove client.mk <http://client.mk> is centered
>     around eliminating the redundancy between `mach` and client.mk
>     <http://client.mk> by effectively moving logic into mach, as I'm
>     touching the code it is obvious that all the code around
>     configure[.in], moz.configure, config.status, etc could use a major
>     overhaul. I'd love to say that "`mach` is the only supported
>     interface to build Firefox." After all, from my perspective as a
>     build system maintainer, it is much, much easier to support 1 thing
>     instead of N>1.
>
>     This poses the question: is it important to preserve the illusion
>     that our build system is following autotools "standards"? If not,
>     can we declare `mach` is the only supported interface and [re]move
>     things like configure.in <http://configure.in>, configure,
>     configure.py, config.status, and Makefile.in at our leisure?
>     _______________________________________________
>     dev-builds mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.mozilla.org/listinfo/dev-builds
>

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds