cc-rework, final steps

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

cc-rework, final steps

Joshua Cranmer 🐧
[Cross-posted to several groups to make sure anyone whose input is desirable will see it]

cc-rework, as you may or may not recall, is the attempt of comm-central to divorce ourselves from having to maintain a partial copy of mozilla-central's build system, by placing comm-central as a subdirectory of mozilla-central instead of the parent directory. So where our tree now would be comm-central/mozilla, it would be mozilla-central/comm after these changes.

As of right now, I have enough patches queued up to be able to make this build configuration work. The patches I need for this amount to:
1. Fix makefiles to account for stricter rules in mozilla-central's build system (presence of IS_COMPONENT is the only one I'm aware of at the moment) [This can done prior to the switch].
2. Automatically rewrite paths in Makefiles and jar.mn files to account for the change in the top-source directory.
3. Fix paths that are not touched by those automatic rules.
4. Make some rules (like testsuite targets) work when COMM_BUILD is not defined [This sounds like it could be done before the switch, but it really can't].
5. Remove the remnants of the current build system in comm-central (client.mk, config/*, configure.in, top-level Makefile/moz.build, a few things from build/*). Not necessary, but clean up is good.

With all of these applied, I can not only build a working Thunderbird on Linux, I can also run through all the build steps that buildbot uses and get more or less the same results--such as a working installer and a working test package. This means that the biggest blocker to doing this is to make all of the necessary changes to the releng configs to get this to work properly. I suspect only the builders and the l10n packaging would need changes, largely due to shifts in paths that affect where things need to be checked out.

Now one change which I have not yet attempted to make is client.py. The projects that use comm-central can need up to 6 repositories: comm-central, mozilla-central, ldap/sdks, extensions/irc, extensions/inspector, and extensions/venkman (the latter three are used by SeaMonkey but not Thunderbird). The present form of getting the source code amounts to the following steps:
$ hg clone http://hg.mozilla.org/comm-central/
$ cd comm-central && python client.py checkout

Under the final form as envisioned by cc-rework, these steps are no longer necessarily sufficient. Here are several options and my thoughts on their tenability:
A. Keep client.py largely the same and expect users to set directories up correctly:
$ mkdir mozilla && cd mozilla
$ hg clone http://hg.mozilla.org/comm-central comm
$ cd comm && python client.py checkout

This requires as little changes as possible, but requires users to know to correctly name the comm-central directory, which requires checking out comm-central to a non-default repository name. It also means that client.py has slightly greater risk to screw you over if you don't follow steps properly.

B. A variant of A, where client.py does not attempt to update mozilla-central.
$ hg clone http://hg.mozilla.org/mozilla-central mozilla
$ hg clone http://hg.mozilla.org/comm-central mozilla/comm
$ cd mozilla/comm && python client.py checkout

Now client.py won't ever attempt to touch .., which gives it safer error handling (at worst, it only affects the current directory). On the downside, mozilla-central must be manually updated when updating a currently existing checkout in addition to client.py checkout.

C. Nuke client.py, and expect users to update repositories manually.

Since a regular comm-central requires effectively the tips of three build systems [1], one of which updates almost never. However, people who use all three of the other extensions need to remember to update more repositories buried far deeper in the tree, even if they update on relatively weak velocities.

D. Merge the code into fewer repositories, and combine with any of the above variants.

The big downside here is that merging risks causing a political avalanche de merde, particularly if it is the comm-central-into-mozilla-central one. The three extension repositories have some benefits to remaining separate repositories, in that it allows them to update at faster velocities than riding the trains to release; this is much harder to maintain if merged into either comm-central or mozilla-central.

It turns out that we appear to have roughly 2.5 repositories that update on a frequent basis [2]: mozilla-central and comm-central both see several checkins a day on average. The DOM Inspector addon sees the most development of the other four repositories, followed by Chatzilla, while the Venkman addon and the LDAP C SDK have seen basically no development in years. Since Venkman relies on an API that is slated to be removed (and there are no signs of developers willing to pitch in to move it to the new API) and is largely obsoleted by recent chrome debugging efforts, I would anticipate much pushback on moving it into mozilla-central. I don't think many people would object to merging the other two extensions into mozilla-central (they're too small to really make much of a difference), but I don't know if the maintainers of those add-ons are willing to see them merge into mozilla-central or not. I'm against merging the LDAP SDKs in for other reasons [3]. Merging comm-central into mozilla-central is something that needs its own, full discussion anyways, so I'm not going to suggest this as an immediate alleviation measure.

E. Merge the comm-central checkout code into mozilla-central's client.py.
$ hg clone http://hg.mozilla.org/mozilla-central mozilla
$ cd mozilla && python client.py checkout-comm

This would require some approval from mozilla-central's build peers, and the current use of mozilla-central's client.py is largely to update external repositories that are checked into mozilla-central like NSS, NSPR, etc. However, I do not think it would be difficult to get such approval for this action, and this has roughly minimal changes from the current workflow for obtaining or updating the build tree.

F. Like E, but use mach instead
$ hg clone http://hg.mozilla.org/mozilla-central mozilla
$ cd mozilla && ./mach checkout-comm

... Unless build peers would rather I use mach instead of client.py. Since releng currently uses client.py to prepare source copies for building, this would be the first use of mach in production releng that I am aware of, and I don't know how much I trust throwing that switch right now.


Personally, I think the only tenable long-term solution is to merge everything into mozilla-central. As it stands right now, we are effectively suffering from the submodules repository problem, and we currently make a wonderful case study on how annoying this problem can be. However, this is a potentially contentious decision, and earlier discussions with principals leads to the conclusion that this step, if taken at all, should be done as a separate step from the cc-rework step. Since that is out of the question for the moment, options E or F seem to me to be the best way to go.


Thoughts/questions/comments/concerns/flames?

[1] Actually, LDAP C-SDKS is checked out to a specific revision. However, that revision has effectively been tip for the existence of LDAP SDKs in a mercurial repository, so the distinction is minor.
[2] This list, believe it or not, is completely ordered by the date of the last substantive checkin--those that exclude build fixes or compatibility updates. In particular, Venkman's last change was January 2012, while LDAP's last change was December 2010.
[3] I gave a lot of plausible future paths for LDAP support in Thunderbird in a very detailed blog post a few months ago. Since the only way to have a shot at decent testing is to use OpenLDAP as a server, if we integrate and LDAP code into comm-central, I'd rather integrate OpenLDAP instead of the current C SDKs.
-- 
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: cc-rework, final steps

Ted Mielczarek-2
On 7/18/2013 1:03 AM, Joshua Cranmer � wrote:

E. Merge the comm-central checkout code into mozilla-central's client.py.
$ hg clone http://hg.mozilla.org/mozilla-central mozilla
$ cd mozilla && python client.py checkout-comm
This seems like a fairly straightforward option.

-Ted


_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: cc-rework, final steps

David Lechner
In reply to this post by Joshua Cranmer 🐧
On 7/18/2013 12:03 AM, Joshua Cranmer 🐧 wrote:
> [Cross-posted to several groups to make sure anyone whose input is
>
> Thoughts/questions/comments/concerns/flames?
>

As someone who only occasionally contributes code to Thunderbird, I
would strongly prefer option E.

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: cc-rework, final steps

Neil-4
In reply to this post by Joshua Cranmer 🐧
Joshua Cranmer wrote:

> A. Keep client.py largely the same and expect users to set directories
> up correctly:
> $ mkdir mozilla && cd mozilla
> $ hg clone http://hg.mozilla.org/comm-central comm
> $ cd comm && python client.py checkout
>
> This requires as little changes as possible, but requires users to
> know to correctly name the comm-central directory, which requires
> checking out comm-central to a non-default repository name. It also
> means that client.py has slightly greater risk to screw you over if
> you don't follow steps properly.

Can you make client.py check that a) it's in a folder named comm b) its
parent is a mozilla-like repo?

> E. Merge the comm-central checkout code into mozilla-central's client.py.
> $ hg clone http://hg.mozilla.org/mozilla-central mozilla
> $ cd mozilla && python client.py checkout-comm

python client.py checkout is available ;-)

--
Warning: May contain traces of nuts.
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: cc-rework, final steps

Gregory Szorc
In reply to this post by Joshua Cranmer 🐧
On 7/17/13 10:03 PM, Joshua Cranmer 🐧 wrote:

> E. Merge the comm-central checkout code into mozilla-central's client.py.
> $ hg clone http://hg.mozilla.org/mozilla-central mozilla
> $ cd mozilla && python client.py checkout-comm
>
> This would require some approval from mozilla-central's build peers, and
> the current use of mozilla-central's client.py is largely to update
> external repositories that are checked into mozilla-central like NSS,
> NSPR, etc. However, I do not think it would be difficult to get such
> approval for this action, and this has roughly minimal changes from the
> current workflow for obtaining or updating the build tree.

 From a purist perspective, it would be nice to have complete separation
and a generic solution that scales to one-offs. i.e. no references to
c-c in m-c.

> F. Like E, but use mach instead
> $ hg clone http://hg.mozilla.org/mozilla-central mozilla
> $ cd mozilla && ./mach checkout-comm
>
> ... Unless build peers would rather I use mach instead of client.py.
> Since releng currently uses client.py to prepare source copies for
> building, this would be the first use of mach in production releng that
> I am aware of, and I don't know how much I trust throwing that switch
> right now.

If we were writing client.py today, we'd likely write it as a mach
command. This reminds me, I want to move it out of the root directory
because IMO its presence in m-c just confused people (since only a
minority of developers there use it).

I would throw out a separate option: merge mozilla-central's Mercurial
repository into comm-central. Conceptually, you start with a Mercurial
repository with a /comm directory. Then, you transplant/graft/rebase
mozilla-central's changesets into / of that new repository. Put another
way, you start with m-c then rebase comm-central's commits into /comm. I
/think/ I've seen a specialized Mercurial extension that does this type
of repository "merging." If it doesn't exist, it should be simple enough
to hack together with existing extensions like transplant and graft.

With this approach, you maintain a 99% clone of m-c. However, you have
the flexibility to modify client.py, mach, etc as you see fit. There is
the potential for "merge conflicts," sure. But if things are done right,
those should be few and far between.
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: cc-rework, final steps

Ehsan Akhgari
On 2013-07-18 3:13 PM, Gregory Szorc wrote:

> On 7/17/13 10:03 PM, Joshua Cranmer 🐧 wrote:
>> E. Merge the comm-central checkout code into mozilla-central's client.py.
>> $ hg clone http://hg.mozilla.org/mozilla-central mozilla
>> $ cd mozilla && python client.py checkout-comm
>>
>> This would require some approval from mozilla-central's build peers, and
>> the current use of mozilla-central's client.py is largely to update
>> external repositories that are checked into mozilla-central like NSS,
>> NSPR, etc. However, I do not think it would be difficult to get such
>> approval for this action, and this has roughly minimal changes from the
>> current workflow for obtaining or updating the build tree.
>
>  From a purist perspective, it would be nice to have complete separation
> and a generic solution that scales to one-offs. i.e. no references to
> c-c in m-c.
>
>> F. Like E, but use mach instead
>> $ hg clone http://hg.mozilla.org/mozilla-central mozilla
>> $ cd mozilla && ./mach checkout-comm
>>
>> ... Unless build peers would rather I use mach instead of client.py.
>> Since releng currently uses client.py to prepare source copies for
>> building, this would be the first use of mach in production releng that
>> I am aware of, and I don't know how much I trust throwing that switch
>> right now.
>
> If we were writing client.py today, we'd likely write it as a mach
> command. This reminds me, I want to move it out of the root directory
> because IMO its presence in m-c just confused people (since only a
> minority of developers there use it).
>
> I would throw out a separate option: merge mozilla-central's Mercurial
> repository into comm-central. Conceptually, you start with a Mercurial
> repository with a /comm directory. Then, you transplant/graft/rebase
> mozilla-central's changesets into / of that new repository. Put another
> way, you start with m-c then rebase comm-central's commits into /comm. I
> /think/ I've seen a specialized Mercurial extension that does this type
> of repository "merging." If it doesn't exist, it should be simple enough
> to hack together with existing extensions like transplant and graft.

See the discussion in bug 787208, please.

Ehsan

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: cc-rework, final steps

Joshua Cranmer 🐧
In reply to this post by Gregory Szorc
On 7/18/2013 2:13 PM, Gregory Szorc wrote:

> On 7/17/13 10:03 PM, Joshua Cranmer 🐧 wrote:
>> E. Merge the comm-central checkout code into mozilla-central's
>> client.py.
>> $ hg clone http://hg.mozilla.org/mozilla-central mozilla
>> $ cd mozilla && python client.py checkout-comm
>>
>> This would require some approval from mozilla-central's build peers, and
>> the current use of mozilla-central's client.py is largely to update
>> external repositories that are checked into mozilla-central like NSS,
>> NSPR, etc. However, I do not think it would be difficult to get such
>> approval for this action, and this has roughly minimal changes from the
>> current workflow for obtaining or updating the build tree.
>
> From a purist perspective, it would be nice to have complete
> separation and a generic solution that scales to one-offs. i.e. no
> references to c-c in m-c.

Even today, there are scattered references to comm-central in
mozilla-central, from the hacks in mozbuild to make it work to some
configure checks that respond to magic variables defined by
comm-central. Comm-central is basically the part of the tree that
doesn't live in mozilla-central.

> I would throw out a separate option: merge mozilla-central's Mercurial
> repository into comm-central. Conceptually, you start with a Mercurial
> repository with a /comm directory. Then, you transplant/graft/rebase
> mozilla-central's changesets into / of that new repository. Put
> another way, you start with m-c then rebase comm-central's commits
> into /comm. I /think/ I've seen a specialized Mercurial extension that
> does this type of repository "merging." If it doesn't exist, it should
> be simple enough to hack together with existing extensions like
> transplant and graft.
>
> With this approach, you maintain a 99% clone of m-c. However, you have
> the flexibility to modify client.py, mach, etc as you see fit. There
> is the potential for "merge conflicts," sure. But if things are done
> right, those should be few and far between.

This is a situation which I rejected immediately for what I hoped were
obvious reasons. Managing patches that touch both m-c and c-c becomes
even harder in this scenario, since you need two patches unless you want
merge headaches. Updating m-c is a specialized process which is going to
be in the capability of the one or two people due to the difficult
nature of the setup. Given the fast cadence of mozilla-central, updating
would also need to be a more grueling daily process for those people.
The biggest benefit of merging c-c with m-c--that tree-wide changes
become much easier to deploy and test--ceases to exist.

--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds