comm-central future as a branch

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

comm-central future as a branch

Magnus Melin-2
I'd like feedback on the below plan for having Thunderbird in a single
repository.
Let's have follow-ups go to dev-planning

TL;DR version: planning to have current comm-central live as a branch
instead of a separate repository.

---

Background
----------------

When the Mozilla move from CVS to hg took place in 2008, the code
related to Thunderbird was not included in mozilla-central and instead
placed in the comm-central repository. Under the hood Thunderbird uses
the mozilla-central code, so to build it currently has to check out both
repositories. This is not ideal, for many many reasons.

I wrote [an email to
maildev](http://lists.thunderbird.net/pipermail/maildev_lists.thunderbird.net/2017-November/000875.html)
last year listing some of the drawbacks of this setup - but essentially
what it comes down to is that version control was not meant to be used
like this and it makes workflows error prone, in addition to having to
do various hacks in the build automation. Enabling autoland, phabricator
and other tools used by Mozilla core could also be done much more easily
for comm-central with a single repository.

The mail spurred some related discussion on git vs hg and importing of
history, but basically there was agreement among the comm-central
developers to go ahead with this change.

Thunderbird has now migrated its build automation to Taskcluster, and
also moved to building with mozilla-central as topdir, with a comm/
sub-folder containing comm-central code. (We used to instead have a
mozilla subdir checkout of mozilla-central as child the comm-central
checkout.)

In 2014 there was [a proposal to merge comm-central code into
mozilla-central](https://groups.google.com/forum/#!msg/mozilla.dev.planning/P8oHKTCWSiQ/76ECituhK-IJ)
but this was rejected since the powers that be did not want 1) the
confusion about what code belongs to what project, 2) pull/tree size,
and 3) it was seen as irrelevant to Firefox. The proposal below has none
of these obstacles, and depending on the physical repository location,
may not change anything at all for a Firefox developer.

I think it's now time to revisit the single repository idea and pull
that through in one way or the other. So, on to the proposal. For
reference, on disk comm/ is on currently 122 MB (+ comm/.hg 192 MB), so
size-wise it is comparatively small, with a mozilla-central checkout
weighing in at 2.3 GB. Including the relevant CVS history, the comm/.hg
would be 317 MB in total.

How
------

To capture the essence of what Thunderbird is, we would make
comm-central a named branch of the mozilla-central. Let's call this
branch "comm/default".

Initially, we'd pull in a hg converted version of the current
comm-central default branch, putting all the files into a comm
sub-folder in the base directory of the comm/default branch. This is the
way that the Thunderbird build system already wants the code laid out.
With the comm sub-folder it will be clear to everyone involved which
code is on mozilla-central code and which is not. It is possible to
import the CVS history for the related directories too while we're at
it, but it adds some size. Importing related CVS history only adds
around 125 MB extra so I'm inclined to include it. With the new
comm/default branch pulled in, we merge the curent mozilla-central
default branch to comm/default. The resulting code is now the same as
status quo (of the both repos pulled in), just that it's all version
controlled in the same repository.

Once this is all set up, Thunderbird development work would move on on
the comm/default branch, and there would be periodical merges with the
(mozilla-central) default branch. These merges could be automated if wanted.

Quite often it's the case that Thunderbird code needs to be adjusted to
build, due to changes in mozilla-central code. For cases of known
incoming bustage Thunderbird now would have the possibility of waiting
to do the merge until a fix is available. I would suggest never to back
out bustage causing mozilla-central changesets from the comm/default
branch, but to detect other incoming bustage by doing builds from a
comm/band-aid branch (branced from comm/default). The details on this
can be discussed later though, not to derail discussion about the main
issue here. There are many alternatives, and it's also completely
possible to do what we currently do.

Where
---------

In the earlier discussions i's become clear there is a bunch of
confusion around what is a branch and what is a repository, so please
note the difference. Aa repository is the physical location where the
history of a project is stored. Every repository does not have to carry
every branch. I.e., the mozilla-central repository could carry the
comm/default branch, or the comm/default branch could exist only in a
comm-central2 repository somewhere else. In all cases it's trivial for
developers who have mozilla-central checked out, to add the comm/default
branch to their local checkout.

So for the location there are at least three options:

A: branch in the mozilla-central repository
B: branch in the mozilla-unified repository
C: branch in a repository elsewhere

Not to disrupt operations too much I think it would be preferable to use
option B - create the branch in the mozilla-unified repository. It does
depend a bit on what plans Mozilla has for these repositories.

Creating the branch in mozilla-unified of course needs buy-in from the
Mozilla hg people, so please let me hear your opinions.

What about history?
--------------------------

For the proposed approach (going through `hg convert`) commits are
preserved, but AFAIK it's not possible to preserve actual commit hashes.
`hg convert` will graft the original commits, and this adds the original
hash as an extra field to the new commit (use `hg log --debug` to see
it). I don't think this is such a big problem. The original commits
would be linked in the pushlog, the same way there is "converted from"
for mozilla-central-cvs, like
https://hg.mozilla.org/experimental/mozilla-central-cvs/rev/ab117d946959.
I don't see them e.g. in
https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/rev/8f8b0933b611 
so perhaps there is some server side feature that needs turning on?

Can I try it?
---------------

You can check
https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/ which
carries the comm/default branch.

In your mozilla tree, pull in the branch like this:

      hg pull -u -b comm/default
https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central

To completely remove it again, use `hg strip "branch(comm/default)`

How did you do it?
------------------------

     CVS_GIT_REPO=~/tmp/mozilla-cvs-history/
     CC_HG_REPO=~/Code/tb/mozilla/comm/

     git clone https://github.com/ehsan/mozilla-cvs-history.git 
$CVS_GIT_REPO

     echo default comm/default | tee branchmap.txt
     echo rename . comm | tee filemap.txt cc0.txt

     hg --cwd=$CC_HG_REPO up 0
     (cd $CC_HG_REPO && ls -d */ | cut -f1 -d'/' | sed 's/^/include /')
 >> cc0.txt

     hg convert --filemap=cc0.txt --branchmap=branchmap.txt
$CVS_GIT_REPO single-repo/

     CVS_TIP=`hg id --cwd=single-repo -i -r tip --debug`
     CC_0=`hg id --cwd=$CC_HG_REPO -i -r 0 --debug`
     echo $CC_0 $CVS_TIP | tee splicemap.txt
     # e4f4569d451a5e0d12a6aa33ebd916f979dd8faa
93dd94bad983ce2eff5f228719c349b78a8a7f7d

     hg convert --config convert.hg.saverev=True
--splicemap=splicemap.txt --filemap=filemap.txt
--branchmap=branchmap.txt $CC_HG_REPO single-repo/

For fun, see all the history down to 1999 is there, like

     hg log comm/mailnews/mime/src/mimei.cpp

Then go on and add the default branch from mozilla-central and do the merge

     cd single-repo
     hg pull -f -b default $CC_HG_REPO/..

     hg up comm/default
     hg merge default && hg commit -m "Merge default to comm/default branch"

     hg push -f -b comm/default --new-branch thunderbird-central-push

---

Thoughts, comments, feedback appreciated!

  -Magnus

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: comm-central future as a branch

Ben Bucksch-2
I like this idea. I'm in favor.

If I understood correctly, you're proposing to take a mozilla-central
(or a mozilla-unified) repo, put comm-central checkout into the comm/
subdirectory, and push that to the branch "comm/default". I think that's
the right way to do things, yes.

We do not even need buy-in from Firefox people, because hg is a
distributed system, so we can simply have our own repository that
contains a copy of mozilla-central plus comm/ (from former comm-central).

I'm also in favor of re-adding CVS history. It's important for us, more
important than for Firefox, due to the longevity of our project.

As for "which combination of mozilla and thunderbird is building and
working?", I would suggest to make a branch comm/bleeding where we merge
mozilla-central immediately and completely without tests. So, if there
are API changes in Mozilla and Thunderbird hasn't been updated yet, it
will be broken. (Code-wise, this is what we currently get, by checking
out mozilla-central and comm-central independently.) So, you have the
option of easily continuing to check out the latest and bleeding edge.
Then, once we tested the combination and we confirmed that it's working,
we merge comm/bleeding into comm/default, and it becomes the new
official Thunderbird development branch. This way, fixing an API
breakage becomes a hg commit that fixes the API and hg merge operation.

The main advantage of this setup is that it's clear which version of
Mozilla works with which version of the Thunderbid code, down to the
commit level. And we capture this information using the version control
system itself, but with some build scripts.

That should dramatically simplify development and build scripts.

Ben


Am 10/9/2018 um 10:06 AM schrieb Magnus Melin:

> I'd like feedback on the below plan for having Thunderbird in a single
> repository.
> Let's have follow-ups go to dev-planning
>
> TL;DR version: planning to have current comm-central live as a branch
> instead of a separate repository.
>
> ---
>
> Background
> ----------------
>
> When the Mozilla move from CVS to hg took place in 2008, the code
> related to Thunderbird was not included in mozilla-central and instead
> placed in the comm-central repository. Under the hood Thunderbird uses
> the mozilla-central code, so to build it currently has to check out
> both repositories. This is not ideal, for many many reasons.
>
> I wrote [an email to
> maildev](http://lists.thunderbird.net/pipermail/maildev_lists.thunderbird.net/2017-November/000875.html)
> last year listing some of the drawbacks of this setup - but
> essentially what it comes down to is that version control was not
> meant to be used like this and it makes workflows error prone, in
> addition to having to do various hacks in the build automation.
> Enabling autoland, phabricator and other tools used by Mozilla core
> could also be done much more easily for comm-central with a single
> repository.
>
> The mail spurred some related discussion on git vs hg and importing of
> history, but basically there was agreement among the comm-central
> developers to go ahead with this change.
>
> Thunderbird has now migrated its build automation to Taskcluster, and
> also moved to building with mozilla-central as topdir, with a comm/
> sub-folder containing comm-central code. (We used to instead have a
> mozilla subdir checkout of mozilla-central as child the comm-central
> checkout.)
>
> In 2014 there was [a proposal to merge comm-central code into
> mozilla-central](https://groups.google.com/forum/#!msg/mozilla.dev.planning/P8oHKTCWSiQ/76ECituhK-IJ)
> but this was rejected since the powers that be did not want 1) the
> confusion about what code belongs to what project, 2) pull/tree size,
> and 3) it was seen as irrelevant to Firefox. The proposal below has
> none of these obstacles, and depending on the physical repository
> location, may not change anything at all for a Firefox developer.
>
> I think it's now time to revisit the single repository idea and pull
> that through in one way or the other. So, on to the proposal. For
> reference, on disk comm/ is on currently 122 MB (+ comm/.hg 192 MB),
> so size-wise it is comparatively small, with a mozilla-central
> checkout weighing in at 2.3 GB. Including the relevant CVS history,
> the comm/.hg would be 317 MB in total.
>
> How
> ------
>
> To capture the essence of what Thunderbird is, we would make
> comm-central a named branch of the mozilla-central. Let's call this
> branch "comm/default".
>
> Initially, we'd pull in a hg converted version of the current
> comm-central default branch, putting all the files into a comm
> sub-folder in the base directory of the comm/default branch. This is
> the way that the Thunderbird build system already wants the code laid
> out. With the comm sub-folder it will be clear to everyone involved
> which code is on mozilla-central code and which is not. It is possible
> to import the CVS history for the related directories too while we're
> at it, but it adds some size. Importing related CVS history only adds
> around 125 MB extra so I'm inclined to include it. With the new
> comm/default branch pulled in, we merge the curent mozilla-central
> default branch to comm/default. The resulting code is now the same as
> status quo (of the both repos pulled in), just that it's all version
> controlled in the same repository.
>
> Once this is all set up, Thunderbird development work would move on on
> the comm/default branch, and there would be periodical merges with the
> (mozilla-central) default branch. These merges could be automated if
> wanted.
>
> Quite often it's the case that Thunderbird code needs to be adjusted
> to build, due to changes in mozilla-central code. For cases of known
> incoming bustage Thunderbird now would have the possibility of waiting
> to do the merge until a fix is available. I would suggest never to
> back out bustage causing mozilla-central changesets from the
> comm/default branch, but to detect other incoming bustage by doing
> builds from a comm/band-aid branch (branced from comm/default). The
> details on this can be discussed later though, not to derail
> discussion about the main issue here. There are many alternatives, and
> it's also completely possible to do what we currently do.
>
> Where
> ---------
>
> In the earlier discussions i's become clear there is a bunch of
> confusion around what is a branch and what is a repository, so please
> note the difference. Aa repository is the physical location where the
> history of a project is stored. Every repository does not have to
> carry every branch. I.e., the mozilla-central repository could carry
> the comm/default branch, or the comm/default branch could exist only
> in a comm-central2 repository somewhere else. In all cases it's
> trivial for developers who have mozilla-central checked out, to add
> the comm/default branch to their local checkout.
>
> So for the location there are at least three options:
>
> A: branch in the mozilla-central repository
> B: branch in the mozilla-unified repository
> C: branch in a repository elsewhere
>
> Not to disrupt operations too much I think it would be preferable to
> use option B - create the branch in the mozilla-unified repository. It
> does depend a bit on what plans Mozilla has for these repositories.
>
> Creating the branch in mozilla-unified of course needs buy-in from the
> Mozilla hg people, so please let me hear your opinions.
>
> What about history?
> --------------------------
>
> For the proposed approach (going through `hg convert`) commits are
> preserved, but AFAIK it's not possible to preserve actual commit
> hashes. `hg convert` will graft the original commits, and this adds
> the original hash as an extra field to the new commit (use `hg log
> --debug` to see it). I don't think this is such a big problem. The
> original commits would be linked in the pushlog, the same way there is
> "converted from" for mozilla-central-cvs, like
> https://hg.mozilla.org/experimental/mozilla-central-cvs/rev/ab117d946959.
> I don't see them e.g. in
> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/rev/8f8b0933b611 
> so perhaps there is some server side feature that needs turning on?
>
> Can I try it?
> ---------------
>
> You can check
> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/ which
> carries the comm/default branch.
>
> In your mozilla tree, pull in the branch like this:
>
>      hg pull -u -b comm/default
> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central
>
> To completely remove it again, use `hg strip "branch(comm/default)`
>
> How did you do it?
> ------------------------
>
>     CVS_GIT_REPO=~/tmp/mozilla-cvs-history/
>     CC_HG_REPO=~/Code/tb/mozilla/comm/
>
>     git clone https://github.com/ehsan/mozilla-cvs-history.git 
> $CVS_GIT_REPO
>
>     echo default comm/default | tee branchmap.txt
>     echo rename . comm | tee filemap.txt cc0.txt
>
>     hg --cwd=$CC_HG_REPO up 0
>     (cd $CC_HG_REPO && ls -d */ | cut -f1 -d'/' | sed 's/^/include /')
> >> cc0.txt
>
>     hg convert --filemap=cc0.txt --branchmap=branchmap.txt
> $CVS_GIT_REPO single-repo/
>
>     CVS_TIP=`hg id --cwd=single-repo -i -r tip --debug`
>     CC_0=`hg id --cwd=$CC_HG_REPO -i -r 0 --debug`
>     echo $CC_0 $CVS_TIP | tee splicemap.txt
>     # e4f4569d451a5e0d12a6aa33ebd916f979dd8faa
> 93dd94bad983ce2eff5f228719c349b78a8a7f7d
>
>     hg convert --config convert.hg.saverev=True
> --splicemap=splicemap.txt --filemap=filemap.txt
> --branchmap=branchmap.txt $CC_HG_REPO single-repo/
>
> For fun, see all the history down to 1999 is there, like
>
>     hg log comm/mailnews/mime/src/mimei.cpp
>
> Then go on and add the default branch from mozilla-central and do the
> merge
>
>     cd single-repo
>     hg pull -f -b default $CC_HG_REPO/..
>
>     hg up comm/default
>     hg merge default && hg commit -m "Merge default to comm/default
> branch"
>
>     hg push -f -b comm/default --new-branch thunderbird-central-push
>
> ---
>
> Thoughts, comments, feedback appreciated!
>
>  -Magnus
>
> _______________________________________________
> tb-planning mailing list
> [hidden email]
> https://mail.mozilla.org/listinfo/tb-planning

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: comm-central future as a branch

Kartikaya Gupta-6
Have you given any thought as to what would happen to the git mirror
of the comm-central repo that currently lives at
https://github.com/mozilla/releases-comm-central ? I ask because
searchfox uses this git repo to do the indexing of comm-central and
that might break with these changes. If you care about still having
searchfox indexing available for comm-central after this change it
would be good to continue having some sort of git mirror. If it's a
brand new git mirror with different hashes we'll have to rebuild the
blame repo and any comm-central permalinks (in bugs, etc.) will likely
break. Or we can try harder to keep permalinks working by mapping the
old hashes to the new ones but that's more work.

Cheers,
kats
On Tue, Oct 9, 2018 at 11:48 AM Ben Bucksch <[hidden email]> wrote:

>
> I like this idea. I'm in favor.
>
> If I understood correctly, you're proposing to take a mozilla-central
> (or a mozilla-unified) repo, put comm-central checkout into the comm/
> subdirectory, and push that to the branch "comm/default". I think that's
> the right way to do things, yes.
>
> We do not even need buy-in from Firefox people, because hg is a
> distributed system, so we can simply have our own repository that
> contains a copy of mozilla-central plus comm/ (from former comm-central).
>
> I'm also in favor of re-adding CVS history. It's important for us, more
> important than for Firefox, due to the longevity of our project.
>
> As for "which combination of mozilla and thunderbird is building and
> working?", I would suggest to make a branch comm/bleeding where we merge
> mozilla-central immediately and completely without tests. So, if there
> are API changes in Mozilla and Thunderbird hasn't been updated yet, it
> will be broken. (Code-wise, this is what we currently get, by checking
> out mozilla-central and comm-central independently.) So, you have the
> option of easily continuing to check out the latest and bleeding edge.
> Then, once we tested the combination and we confirmed that it's working,
> we merge comm/bleeding into comm/default, and it becomes the new
> official Thunderbird development branch. This way, fixing an API
> breakage becomes a hg commit that fixes the API and hg merge operation.
>
> The main advantage of this setup is that it's clear which version of
> Mozilla works with which version of the Thunderbid code, down to the
> commit level. And we capture this information using the version control
> system itself, but with some build scripts.
>
> That should dramatically simplify development and build scripts.
>
> Ben
>
>
> Am 10/9/2018 um 10:06 AM schrieb Magnus Melin:
>
> > I'd like feedback on the below plan for having Thunderbird in a single
> > repository.
> > Let's have follow-ups go to dev-planning
> >
> > TL;DR version: planning to have current comm-central live as a branch
> > instead of a separate repository.
> >
> > ---
> >
> > Background
> > ----------------
> >
> > When the Mozilla move from CVS to hg took place in 2008, the code
> > related to Thunderbird was not included in mozilla-central and instead
> > placed in the comm-central repository. Under the hood Thunderbird uses
> > the mozilla-central code, so to build it currently has to check out
> > both repositories. This is not ideal, for many many reasons.
> >
> > I wrote [an email to
> > maildev](http://lists.thunderbird.net/pipermail/maildev_lists.thunderbird.net/2017-November/000875.html)
> > last year listing some of the drawbacks of this setup - but
> > essentially what it comes down to is that version control was not
> > meant to be used like this and it makes workflows error prone, in
> > addition to having to do various hacks in the build automation.
> > Enabling autoland, phabricator and other tools used by Mozilla core
> > could also be done much more easily for comm-central with a single
> > repository.
> >
> > The mail spurred some related discussion on git vs hg and importing of
> > history, but basically there was agreement among the comm-central
> > developers to go ahead with this change.
> >
> > Thunderbird has now migrated its build automation to Taskcluster, and
> > also moved to building with mozilla-central as topdir, with a comm/
> > sub-folder containing comm-central code. (We used to instead have a
> > mozilla subdir checkout of mozilla-central as child the comm-central
> > checkout.)
> >
> > In 2014 there was [a proposal to merge comm-central code into
> > mozilla-central](https://groups.google.com/forum/#!msg/mozilla.dev.planning/P8oHKTCWSiQ/76ECituhK-IJ)
> > but this was rejected since the powers that be did not want 1) the
> > confusion about what code belongs to what project, 2) pull/tree size,
> > and 3) it was seen as irrelevant to Firefox. The proposal below has
> > none of these obstacles, and depending on the physical repository
> > location, may not change anything at all for a Firefox developer.
> >
> > I think it's now time to revisit the single repository idea and pull
> > that through in one way or the other. So, on to the proposal. For
> > reference, on disk comm/ is on currently 122 MB (+ comm/.hg 192 MB),
> > so size-wise it is comparatively small, with a mozilla-central
> > checkout weighing in at 2.3 GB. Including the relevant CVS history,
> > the comm/.hg would be 317 MB in total.
> >
> > How
> > ------
> >
> > To capture the essence of what Thunderbird is, we would make
> > comm-central a named branch of the mozilla-central. Let's call this
> > branch "comm/default".
> >
> > Initially, we'd pull in a hg converted version of the current
> > comm-central default branch, putting all the files into a comm
> > sub-folder in the base directory of the comm/default branch. This is
> > the way that the Thunderbird build system already wants the code laid
> > out. With the comm sub-folder it will be clear to everyone involved
> > which code is on mozilla-central code and which is not. It is possible
> > to import the CVS history for the related directories too while we're
> > at it, but it adds some size. Importing related CVS history only adds
> > around 125 MB extra so I'm inclined to include it. With the new
> > comm/default branch pulled in, we merge the curent mozilla-central
> > default branch to comm/default. The resulting code is now the same as
> > status quo (of the both repos pulled in), just that it's all version
> > controlled in the same repository.
> >
> > Once this is all set up, Thunderbird development work would move on on
> > the comm/default branch, and there would be periodical merges with the
> > (mozilla-central) default branch. These merges could be automated if
> > wanted.
> >
> > Quite often it's the case that Thunderbird code needs to be adjusted
> > to build, due to changes in mozilla-central code. For cases of known
> > incoming bustage Thunderbird now would have the possibility of waiting
> > to do the merge until a fix is available. I would suggest never to
> > back out bustage causing mozilla-central changesets from the
> > comm/default branch, but to detect other incoming bustage by doing
> > builds from a comm/band-aid branch (branced from comm/default). The
> > details on this can be discussed later though, not to derail
> > discussion about the main issue here. There are many alternatives, and
> > it's also completely possible to do what we currently do.
> >
> > Where
> > ---------
> >
> > In the earlier discussions i's become clear there is a bunch of
> > confusion around what is a branch and what is a repository, so please
> > note the difference. Aa repository is the physical location where the
> > history of a project is stored. Every repository does not have to
> > carry every branch. I.e., the mozilla-central repository could carry
> > the comm/default branch, or the comm/default branch could exist only
> > in a comm-central2 repository somewhere else. In all cases it's
> > trivial for developers who have mozilla-central checked out, to add
> > the comm/default branch to their local checkout.
> >
> > So for the location there are at least three options:
> >
> > A: branch in the mozilla-central repository
> > B: branch in the mozilla-unified repository
> > C: branch in a repository elsewhere
> >
> > Not to disrupt operations too much I think it would be preferable to
> > use option B - create the branch in the mozilla-unified repository. It
> > does depend a bit on what plans Mozilla has for these repositories.
> >
> > Creating the branch in mozilla-unified of course needs buy-in from the
> > Mozilla hg people, so please let me hear your opinions.
> >
> > What about history?
> > --------------------------
> >
> > For the proposed approach (going through `hg convert`) commits are
> > preserved, but AFAIK it's not possible to preserve actual commit
> > hashes. `hg convert` will graft the original commits, and this adds
> > the original hash as an extra field to the new commit (use `hg log
> > --debug` to see it). I don't think this is such a big problem. The
> > original commits would be linked in the pushlog, the same way there is
> > "converted from" for mozilla-central-cvs, like
> > https://hg.mozilla.org/experimental/mozilla-central-cvs/rev/ab117d946959.
> > I don't see them e.g. in
> > https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/rev/8f8b0933b611
> > so perhaps there is some server side feature that needs turning on?
> >
> > Can I try it?
> > ---------------
> >
> > You can check
> > https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/ which
> > carries the comm/default branch.
> >
> > In your mozilla tree, pull in the branch like this:
> >
> >      hg pull -u -b comm/default
> > https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central
> >
> > To completely remove it again, use `hg strip "branch(comm/default)`
> >
> > How did you do it?
> > ------------------------
> >
> >     CVS_GIT_REPO=~/tmp/mozilla-cvs-history/
> >     CC_HG_REPO=~/Code/tb/mozilla/comm/
> >
> >     git clone https://github.com/ehsan/mozilla-cvs-history.git
> > $CVS_GIT_REPO
> >
> >     echo default comm/default | tee branchmap.txt
> >     echo rename . comm | tee filemap.txt cc0.txt
> >
> >     hg --cwd=$CC_HG_REPO up 0
> >     (cd $CC_HG_REPO && ls -d */ | cut -f1 -d'/' | sed 's/^/include /')
> > >> cc0.txt
> >
> >     hg convert --filemap=cc0.txt --branchmap=branchmap.txt
> > $CVS_GIT_REPO single-repo/
> >
> >     CVS_TIP=`hg id --cwd=single-repo -i -r tip --debug`
> >     CC_0=`hg id --cwd=$CC_HG_REPO -i -r 0 --debug`
> >     echo $CC_0 $CVS_TIP | tee splicemap.txt
> >     # e4f4569d451a5e0d12a6aa33ebd916f979dd8faa
> > 93dd94bad983ce2eff5f228719c349b78a8a7f7d
> >
> >     hg convert --config convert.hg.saverev=True
> > --splicemap=splicemap.txt --filemap=filemap.txt
> > --branchmap=branchmap.txt $CC_HG_REPO single-repo/
> >
> > For fun, see all the history down to 1999 is there, like
> >
> >     hg log comm/mailnews/mime/src/mimei.cpp
> >
> > Then go on and add the default branch from mozilla-central and do the
> > merge
> >
> >     cd single-repo
> >     hg pull -f -b default $CC_HG_REPO/..
> >
> >     hg up comm/default
> >     hg merge default && hg commit -m "Merge default to comm/default
> > branch"
> >
> >     hg push -f -b comm/default --new-branch thunderbird-central-push
> >
> > ---
> >
> > Thoughts, comments, feedback appreciated!
> >
> >  -Magnus
> >
> > _______________________________________________
> > tb-planning mailing list
> > [hidden email]
> > https://mail.mozilla.org/listinfo/tb-planning
>
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: comm-central future as a branch

Frank-Rainer Grahl-3
In reply to this post by Ben Bucksch-2
Well not knowing if SeaMonkey will see another working version from
comm-central but it is still compiling so lets assume the best.

Cons:
- I would still need separate comm-repos for at least development and esr
release. Working with branches and mercurial queues is cumbersome. Switching
branches on the fly and compiling a different release will usually just result
in a clobber build. Same for working with a stable and unstable central.
Doesn't matter much.

If unified:
- Working with a current mozilla repo is dead slow under Windows and somewhat
slow under Linux. These are too big already. Adding the comm history on top
and you probably end up with super dead slow :)

- Needs synchronisation with the mozilla repos. So might need many more merges
which will probably won't make it faster.

Pros:
- Easier for new contributors to just check out one unified repo.
- Merges between releases might be easier (or not).
- Makes a possible future m-c fork easier.
- Tool integration might be easier but most of them are a complete overkill
for smaller projects such as TB and SM anyway.

Overall indifferent. Current system works for me but there is always room for
improvement.

FRG


Ben Bucksch wrote:

> I like this idea. I'm in favor.
>
> If I understood correctly, you're proposing to take a mozilla-central (or a
> mozilla-unified) repo, put comm-central checkout into the comm/ subdirectory,
> and push that to the branch "comm/default". I think that's the right way to do
> things, yes.
>
> We do not even need buy-in from Firefox people, because hg is a distributed
> system, so we can simply have our own repository that contains a copy of
> mozilla-central plus comm/ (from former comm-central).
>
> I'm also in favor of re-adding CVS history. It's important for us, more
> important than for Firefox, due to the longevity of our project.
>
> As for "which combination of mozilla and thunderbird is building and
> working?", I would suggest to make a branch comm/bleeding where we merge
> mozilla-central immediately and completely without tests. So, if there are API
> changes in Mozilla and Thunderbird hasn't been updated yet, it will be broken.
> (Code-wise, this is what we currently get, by checking out mozilla-central and
> comm-central independently.) So, you have the option of easily continuing to
> check out the latest and bleeding edge.
> Then, once we tested the combination and we confirmed that it's working, we
> merge comm/bleeding into comm/default, and it becomes the new official
> Thunderbird development branch. This way, fixing an API breakage becomes a hg
> commit that fixes the API and hg merge operation.
>
> The main advantage of this setup is that it's clear which version of Mozilla
> works with which version of the Thunderbid code, down to the commit level. And
> we capture this information using the version control system itself, but with
> some build scripts.
>
> That should dramatically simplify development and build scripts.
>
> Ben
>
>
> Am 10/9/2018 um 10:06 AM schrieb Magnus Melin:
>
>> I'd like feedback on the below plan for having Thunderbird in a single
>> repository.
>> Let's have follow-ups go to dev-planning
>>
>> TL;DR version: planning to have current comm-central live as a branch
>> instead of a separate repository.
>>
>> ---
>>
>> Background
>> ----------------
>>
>> When the Mozilla move from CVS to hg took place in 2008, the code related to
>> Thunderbird was not included in mozilla-central and instead placed in the
>> comm-central repository. Under the hood Thunderbird uses the mozilla-central
>> code, so to build it currently has to check out both repositories. This is
>> not ideal, for many many reasons.
>>
>> I wrote [an email to
>> maildev](http://lists.thunderbird.net/pipermail/maildev_lists.thunderbird.net/2017-November/000875.html)
>> last year listing some of the drawbacks of this setup - but essentially what
>> it comes down to is that version control was not meant to be used like this
>> and it makes workflows error prone, in addition to having to do various
>> hacks in the build automation. Enabling autoland, phabricator and other
>> tools used by Mozilla core could also be done much more easily for
>> comm-central with a single repository.
>>
>> The mail spurred some related discussion on git vs hg and importing of
>> history, but basically there was agreement among the comm-central developers
>> to go ahead with this change.
>>
>> Thunderbird has now migrated its build automation to Taskcluster, and also
>> moved to building with mozilla-central as topdir, with a comm/ sub-folder
>> containing comm-central code. (We used to instead have a mozilla subdir
>> checkout of mozilla-central as child the comm-central checkout.)
>>
>> In 2014 there was [a proposal to merge comm-central code into
>> mozilla-central](https://groups.google.com/forum/#!msg/mozilla.dev.planning/P8oHKTCWSiQ/76ECituhK-IJ)
>> but this was rejected since the powers that be did not want 1) the confusion
>> about what code belongs to what project, 2) pull/tree size, and 3) it was
>> seen as irrelevant to Firefox. The proposal below has none of these
>> obstacles, and depending on the physical repository location, may not change
>> anything at all for a Firefox developer.
>>
>> I think it's now time to revisit the single repository idea and pull that
>> through in one way or the other. So, on to the proposal. For reference, on
>> disk comm/ is on currently 122 MB (+ comm/.hg 192 MB), so size-wise it is
>> comparatively small, with a mozilla-central checkout weighing in at 2.3 GB.
>> Including the relevant CVS history, the comm/.hg would be 317 MB in total.
>>
>> How
>> ------
>>
>> To capture the essence of what Thunderbird is, we would make comm-central a
>> named branch of the mozilla-central. Let's call this branch "comm/default".
>>
>> Initially, we'd pull in a hg converted version of the current comm-central
>> default branch, putting all the files into a comm sub-folder in the base
>> directory of the comm/default branch. This is the way that the Thunderbird
>> build system already wants the code laid out. With the comm sub-folder it
>> will be clear to everyone involved which code is on mozilla-central code and
>> which is not. It is possible to import the CVS history for the related
>> directories too while we're at it, but it adds some size. Importing related
>> CVS history only adds around 125 MB extra so I'm inclined to include it.
>> With the new comm/default branch pulled in, we merge the curent
>> mozilla-central default branch to comm/default. The resulting code is now
>> the same as status quo (of the both repos pulled in), just that it's all
>> version controlled in the same repository.
>>
>> Once this is all set up, Thunderbird development work would move on on the
>> comm/default branch, and there would be periodical merges with the
>> (mozilla-central) default branch. These merges could be automated if wanted.
>>
>> Quite often it's the case that Thunderbird code needs to be adjusted to
>> build, due to changes in mozilla-central code. For cases of known incoming
>> bustage Thunderbird now would have the possibility of waiting to do the
>> merge until a fix is available. I would suggest never to back out bustage
>> causing mozilla-central changesets from the comm/default branch, but to
>> detect other incoming bustage by doing builds from a comm/band-aid branch
>> (branced from comm/default). The details on this can be discussed later
>> though, not to derail discussion about the main issue here. There are many
>> alternatives, and it's also completely possible to do what we currently do.
>>
>> Where
>> ---------
>>
>> In the earlier discussions i's become clear there is a bunch of confusion
>> around what is a branch and what is a repository, so please note the
>> difference. Aa repository is the physical location where the history of a
>> project is stored. Every repository does not have to carry every branch.
>> I.e., the mozilla-central repository could carry the comm/default branch, or
>> the comm/default branch could exist only in a comm-central2 repository
>> somewhere else. In all cases it's trivial for developers who have
>> mozilla-central checked out, to add the comm/default branch to their local
>> checkout.
>>
>> So for the location there are at least three options:
>>
>> A: branch in the mozilla-central repository
>> B: branch in the mozilla-unified repository
>> C: branch in a repository elsewhere
>>
>> Not to disrupt operations too much I think it would be preferable to use
>> option B - create the branch in the mozilla-unified repository. It does
>> depend a bit on what plans Mozilla has for these repositories.
>>
>> Creating the branch in mozilla-unified of course needs buy-in from the
>> Mozilla hg people, so please let me hear your opinions.
>>
>> What about history?
>> --------------------------
>>
>> For the proposed approach (going through `hg convert`) commits are
>> preserved, but AFAIK it's not possible to preserve actual commit hashes. `hg
>> convert` will graft the original commits, and this adds the original hash as
>> an extra field to the new commit (use `hg log --debug` to see it). I don't
>> think this is such a big problem. The original commits would be linked in
>> the pushlog, the same way there is "converted from" for mozilla-central-cvs,
>> like
>> https://hg.mozilla.org/experimental/mozilla-central-cvs/rev/ab117d946959. I
>> don't see them e.g. in
>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/rev/8f8b0933b611 
>> so perhaps there is some server side feature that needs turning on?
>>
>> Can I try it?
>> ---------------
>>
>> You can check
>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/ which
>> carries the comm/default branch.
>>
>> In your mozilla tree, pull in the branch like this:
>>
>>      hg pull -u -b comm/default
>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central
>>
>> To completely remove it again, use `hg strip "branch(comm/default)`
>>
>> How did you do it?
>> ------------------------
>>
>>     CVS_GIT_REPO=~/tmp/mozilla-cvs-history/
>>     CC_HG_REPO=~/Code/tb/mozilla/comm/
>>
>>     git clone https://github.com/ehsan/mozilla-cvs-history.git $CVS_GIT_REPO
>>
>>     echo default comm/default | tee branchmap.txt
>>     echo rename . comm | tee filemap.txt cc0.txt
>>
>>     hg --cwd=$CC_HG_REPO up 0
>>     (cd $CC_HG_REPO && ls -d */ | cut -f1 -d'/' | sed 's/^/include /') >>
>> cc0.txt
>>
>>     hg convert --filemap=cc0.txt --branchmap=branchmap.txt $CVS_GIT_REPO
>> single-repo/
>>
>>     CVS_TIP=`hg id --cwd=single-repo -i -r tip --debug`
>>     CC_0=`hg id --cwd=$CC_HG_REPO -i -r 0 --debug`
>>     echo $CC_0 $CVS_TIP | tee splicemap.txt
>>     # e4f4569d451a5e0d12a6aa33ebd916f979dd8faa
>> 93dd94bad983ce2eff5f228719c349b78a8a7f7d
>>
>>     hg convert --config convert.hg.saverev=True --splicemap=splicemap.txt
>> --filemap=filemap.txt --branchmap=branchmap.txt $CC_HG_REPO single-repo/
>>
>> For fun, see all the history down to 1999 is there, like
>>
>>     hg log comm/mailnews/mime/src/mimei.cpp
>>
>> Then go on and add the default branch from mozilla-central and do the merge
>>
>>     cd single-repo
>>     hg pull -f -b default $CC_HG_REPO/..
>>
>>     hg up comm/default
>>     hg merge default && hg commit -m "Merge default to comm/default branch"
>>
>>     hg push -f -b comm/default --new-branch thunderbird-central-push
>>
>> ---
>>
>> Thoughts, comments, feedback appreciated!
>>
>>  -Magnus
>>
>> _______________________________________________
>> tb-planning mailing list
>> [hidden email]
>> https://mail.mozilla.org/listinfo/tb-planning
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: comm-central future as a branch

Jörg Knobloch
In reply to this post by Magnus Melin-2
On 09/10/2018 10:06, Magnus Melin wrote:

> Quite often it's the case that Thunderbird code needs to be adjusted
> to build, due to changes in mozilla-central code. For cases of known
> incoming bustage Thunderbird now would have the possibility of waiting
> to do the merge until a fix is available. I would suggest never to
> back out bustage causing mozilla-central changesets from the
> comm/default branch, but to detect other incoming bustage by doing
> builds from a comm/band-aid branch (branced from comm/default). The
> details on this can be discussed later though, not to derail
> discussion about the main issue here. There are many alternatives, and
> it's also completely possible to do what we currently do.

So what is the "main issue" if not handling bustage less stressfully?

As the main firefighter I can tell you that waiting can be fatal, since
the next bustage can be in the next merge you're not doing since you're
waiting. Singular backouts could be attractive although their management
would become messy real soon.

So if we're delaying the merge, what would be the process to detect
bustage? Do the merge onto some other branch with which we run a try build?

Jörg.

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: comm-central future as a branch

Kartikaya Gupta-6
In reply to this post by Kartikaya Gupta-6
I believe that script is maintained by the Mozilla releng team, so
you'd have to talk to them to get a new git repo set up.

It's not clear to me if you'd have two git repos (one for the
mozilla-central stuff, and one pulls in just the comm/ folder) or a
single repo. In the latter case it sounds like it would be a brand new
conversion of mozilla-central from hg to git, which sounds kind of
undesirable given that we already have
https://github.com/mozilla/gecko-dev and
https://github.com/mozilla/gecko which are two different conversions
of m-c to git. I don't really know what the best way forward here, I
just wanted to mention this because if you want searchfox to keep
working we'll need some sort of git repo with the code.

Also, my messages are getting stuck in the dev-planning moderation
queue so I'm cc'ing Magnus directly here.

Cheers,
kats

On Tue, Oct 9, 2018 at 12:51 PM Ben Bucksch <[hidden email]> wrote:

>
> Hey kats,
>
> just a quick thank-you note for providing searchfox :)
>
> I think we'd need a new git repo, but that it would work like the old,
> just that it already contains mozilla code, so it should be simpler,
> actually.
>
> The main trick is that whatever script pushes to github needs to be adapted.
>
> Ben
>
> Am 10/9/2018 um 6:32 PM schrieb Kartikaya Gupta:
> > Have you given any thought as to what would happen to the git mirror
> > of the comm-central repo that currently lives at
> > https://github.com/mozilla/releases-comm-central ? I ask because
> > searchfox uses this git repo to do the indexing of comm-central and
> > that might break with these changes. If you care about still having
> > searchfox indexing available for comm-central after this change it
> > would be good to continue having some sort of git mirror. If it's a
> > brand new git mirror with different hashes we'll have to rebuild the
> > blame repo and any comm-central permalinks (in bugs, etc.) will likely
> > break. Or we can try harder to keep permalinks working by mapping the
> > old hashes to the new ones but that's more work.
> >
> > Cheers,
> > kats
> > On Tue, Oct 9, 2018 at 11:48 AM Ben Bucksch <[hidden email]> wrote:
> >> I like this idea. I'm in favor.
> >>
> >> If I understood correctly, you're proposing to take a mozilla-central
> >> (or a mozilla-unified) repo, put comm-central checkout into the comm/
> >> subdirectory, and push that to the branch "comm/default". I think that's
> >> the right way to do things, yes.
> >>
> >> We do not even need buy-in from Firefox people, because hg is a
> >> distributed system, so we can simply have our own repository that
> >> contains a copy of mozilla-central plus comm/ (from former comm-central).
> >>
> >> I'm also in favor of re-adding CVS history. It's important for us, more
> >> important than for Firefox, due to the longevity of our project.
> >>
> >> As for "which combination of mozilla and thunderbird is building and
> >> working?", I would suggest to make a branch comm/bleeding where we merge
> >> mozilla-central immediately and completely without tests. So, if there
> >> are API changes in Mozilla and Thunderbird hasn't been updated yet, it
> >> will be broken. (Code-wise, this is what we currently get, by checking
> >> out mozilla-central and comm-central independently.) So, you have the
> >> option of easily continuing to check out the latest and bleeding edge.
> >> Then, once we tested the combination and we confirmed that it's working,
> >> we merge comm/bleeding into comm/default, and it becomes the new
> >> official Thunderbird development branch. This way, fixing an API
> >> breakage becomes a hg commit that fixes the API and hg merge operation.
> >>
> >> The main advantage of this setup is that it's clear which version of
> >> Mozilla works with which version of the Thunderbid code, down to the
> >> commit level. And we capture this information using the version control
> >> system itself, but with some build scripts.
> >>
> >> That should dramatically simplify development and build scripts.
> >>
> >> Ben
> >>
> >>
> >> Am 10/9/2018 um 10:06 AM schrieb Magnus Melin:
> >>
> >>> I'd like feedback on the below plan for having Thunderbird in a single
> >>> repository.
> >>> Let's have follow-ups go to dev-planning
> >>>
> >>> TL;DR version: planning to have current comm-central live as a branch
> >>> instead of a separate repository.
> >>>
> >>> ---
> >>>
> >>> Background
> >>> ----------------
> >>>
> >>> When the Mozilla move from CVS to hg took place in 2008, the code
> >>> related to Thunderbird was not included in mozilla-central and instead
> >>> placed in the comm-central repository. Under the hood Thunderbird uses
> >>> the mozilla-central code, so to build it currently has to check out
> >>> both repositories. This is not ideal, for many many reasons.
> >>>
> >>> I wrote [an email to
> >>> maildev](http://lists.thunderbird.net/pipermail/maildev_lists.thunderbird.net/2017-November/000875.html)
> >>> last year listing some of the drawbacks of this setup - but
> >>> essentially what it comes down to is that version control was not
> >>> meant to be used like this and it makes workflows error prone, in
> >>> addition to having to do various hacks in the build automation.
> >>> Enabling autoland, phabricator and other tools used by Mozilla core
> >>> could also be done much more easily for comm-central with a single
> >>> repository.
> >>>
> >>> The mail spurred some related discussion on git vs hg and importing of
> >>> history, but basically there was agreement among the comm-central
> >>> developers to go ahead with this change.
> >>>
> >>> Thunderbird has now migrated its build automation to Taskcluster, and
> >>> also moved to building with mozilla-central as topdir, with a comm/
> >>> sub-folder containing comm-central code. (We used to instead have a
> >>> mozilla subdir checkout of mozilla-central as child the comm-central
> >>> checkout.)
> >>>
> >>> In 2014 there was [a proposal to merge comm-central code into
> >>> mozilla-central](https://groups.google.com/forum/#!msg/mozilla.dev.planning/P8oHKTCWSiQ/76ECituhK-IJ)
> >>> but this was rejected since the powers that be did not want 1) the
> >>> confusion about what code belongs to what project, 2) pull/tree size,
> >>> and 3) it was seen as irrelevant to Firefox. The proposal below has
> >>> none of these obstacles, and depending on the physical repository
> >>> location, may not change anything at all for a Firefox developer.
> >>>
> >>> I think it's now time to revisit the single repository idea and pull
> >>> that through in one way or the other. So, on to the proposal. For
> >>> reference, on disk comm/ is on currently 122 MB (+ comm/.hg 192 MB),
> >>> so size-wise it is comparatively small, with a mozilla-central
> >>> checkout weighing in at 2.3 GB. Including the relevant CVS history,
> >>> the comm/.hg would be 317 MB in total.
> >>>
> >>> How
> >>> ------
> >>>
> >>> To capture the essence of what Thunderbird is, we would make
> >>> comm-central a named branch of the mozilla-central. Let's call this
> >>> branch "comm/default".
> >>>
> >>> Initially, we'd pull in a hg converted version of the current
> >>> comm-central default branch, putting all the files into a comm
> >>> sub-folder in the base directory of the comm/default branch. This is
> >>> the way that the Thunderbird build system already wants the code laid
> >>> out. With the comm sub-folder it will be clear to everyone involved
> >>> which code is on mozilla-central code and which is not. It is possible
> >>> to import the CVS history for the related directories too while we're
> >>> at it, but it adds some size. Importing related CVS history only adds
> >>> around 125 MB extra so I'm inclined to include it. With the new
> >>> comm/default branch pulled in, we merge the curent mozilla-central
> >>> default branch to comm/default. The resulting code is now the same as
> >>> status quo (of the both repos pulled in), just that it's all version
> >>> controlled in the same repository.
> >>>
> >>> Once this is all set up, Thunderbird development work would move on on
> >>> the comm/default branch, and there would be periodical merges with the
> >>> (mozilla-central) default branch. These merges could be automated if
> >>> wanted.
> >>>
> >>> Quite often it's the case that Thunderbird code needs to be adjusted
> >>> to build, due to changes in mozilla-central code. For cases of known
> >>> incoming bustage Thunderbird now would have the possibility of waiting
> >>> to do the merge until a fix is available. I would suggest never to
> >>> back out bustage causing mozilla-central changesets from the
> >>> comm/default branch, but to detect other incoming bustage by doing
> >>> builds from a comm/band-aid branch (branced from comm/default). The
> >>> details on this can be discussed later though, not to derail
> >>> discussion about the main issue here. There are many alternatives, and
> >>> it's also completely possible to do what we currently do.
> >>>
> >>> Where
> >>> ---------
> >>>
> >>> In the earlier discussions i's become clear there is a bunch of
> >>> confusion around what is a branch and what is a repository, so please
> >>> note the difference. Aa repository is the physical location where the
> >>> history of a project is stored. Every repository does not have to
> >>> carry every branch. I.e., the mozilla-central repository could carry
> >>> the comm/default branch, or the comm/default branch could exist only
> >>> in a comm-central2 repository somewhere else. In all cases it's
> >>> trivial for developers who have mozilla-central checked out, to add
> >>> the comm/default branch to their local checkout.
> >>>
> >>> So for the location there are at least three options:
> >>>
> >>> A: branch in the mozilla-central repository
> >>> B: branch in the mozilla-unified repository
> >>> C: branch in a repository elsewhere
> >>>
> >>> Not to disrupt operations too much I think it would be preferable to
> >>> use option B - create the branch in the mozilla-unified repository. It
> >>> does depend a bit on what plans Mozilla has for these repositories.
> >>>
> >>> Creating the branch in mozilla-unified of course needs buy-in from the
> >>> Mozilla hg people, so please let me hear your opinions.
> >>>
> >>> What about history?
> >>> --------------------------
> >>>
> >>> For the proposed approach (going through `hg convert`) commits are
> >>> preserved, but AFAIK it's not possible to preserve actual commit
> >>> hashes. `hg convert` will graft the original commits, and this adds
> >>> the original hash as an extra field to the new commit (use `hg log
> >>> --debug` to see it). I don't think this is such a big problem. The
> >>> original commits would be linked in the pushlog, the same way there is
> >>> "converted from" for mozilla-central-cvs, like
> >>> https://hg.mozilla.org/experimental/mozilla-central-cvs/rev/ab117d946959.
> >>> I don't see them e.g. in
> >>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/rev/8f8b0933b611
> >>> so perhaps there is some server side feature that needs turning on?
> >>>
> >>> Can I try it?
> >>> ---------------
> >>>
> >>> You can check
> >>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/ which
> >>> carries the comm/default branch.
> >>>
> >>> In your mozilla tree, pull in the branch like this:
> >>>
> >>>       hg pull -u -b comm/default
> >>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central
> >>>
> >>> To completely remove it again, use `hg strip "branch(comm/default)`
> >>>
> >>> How did you do it?
> >>> ------------------------
> >>>
> >>>      CVS_GIT_REPO=~/tmp/mozilla-cvs-history/
> >>>      CC_HG_REPO=~/Code/tb/mozilla/comm/
> >>>
> >>>      git clone https://github.com/ehsan/mozilla-cvs-history.git
> >>> $CVS_GIT_REPO
> >>>
> >>>      echo default comm/default | tee branchmap.txt
> >>>      echo rename . comm | tee filemap.txt cc0.txt
> >>>
> >>>      hg --cwd=$CC_HG_REPO up 0
> >>>      (cd $CC_HG_REPO && ls -d */ | cut -f1 -d'/' | sed 's/^/include /')
> >>>>> cc0.txt
> >>>      hg convert --filemap=cc0.txt --branchmap=branchmap.txt
> >>> $CVS_GIT_REPO single-repo/
> >>>
> >>>      CVS_TIP=`hg id --cwd=single-repo -i -r tip --debug`
> >>>      CC_0=`hg id --cwd=$CC_HG_REPO -i -r 0 --debug`
> >>>      echo $CC_0 $CVS_TIP | tee splicemap.txt
> >>>      # e4f4569d451a5e0d12a6aa33ebd916f979dd8faa
> >>> 93dd94bad983ce2eff5f228719c349b78a8a7f7d
> >>>
> >>>      hg convert --config convert.hg.saverev=True
> >>> --splicemap=splicemap.txt --filemap=filemap.txt
> >>> --branchmap=branchmap.txt $CC_HG_REPO single-repo/
> >>>
> >>> For fun, see all the history down to 1999 is there, like
> >>>
> >>>      hg log comm/mailnews/mime/src/mimei.cpp
> >>>
> >>> Then go on and add the default branch from mozilla-central and do the
> >>> merge
> >>>
> >>>      cd single-repo
> >>>      hg pull -f -b default $CC_HG_REPO/..
> >>>
> >>>      hg up comm/default
> >>>      hg merge default && hg commit -m "Merge default to comm/default
> >>> branch"
> >>>
> >>>      hg push -f -b comm/default --new-branch thunderbird-central-push
> >>>
> >>> ---
> >>>
> >>> Thoughts, comments, feedback appreciated!
> >>>
> >>>   -Magnus
> >>>
> >>> _______________________________________________
> >>> tb-planning mailing list
> >>> [hidden email]
> >>> https://mail.mozilla.org/listinfo/tb-planning
> >> _______________________________________________
> >> dev-planning mailing list
> >> [hidden email]
> >> https://lists.mozilla.org/listinfo/dev-planning
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: comm-central future as a branch

Ben Bucksch-2
Yes, you're right. The git version of the repo should preferably re-use
the existing gecko-dev git repo and simply merge it in. That would
preserve commit IDs and make manual merges from other branches much much
easier. It would complicate the script code somewhat, but it may the
runtime of the script faster, because the mozilla commits don't need to
be processed again, just merged in. The script would simply need a way
to match the parents of the hg merge commit with the corresponding git
commit in gecko-dev. It wouldn't even need to match all commits, only
the merge parents.

Am 10/10/2018 um 2:16 PM schrieb Kartikaya Gupta:

> I believe that script is maintained by the Mozilla releng team, so
> you'd have to talk to them to get a new git repo set up.
>
> It's not clear to me if you'd have two git repos (one for the
> mozilla-central stuff, and one pulls in just the comm/ folder) or a
> single repo. In the latter case it sounds like it would be a brand new
> conversion of mozilla-central from hg to git, which sounds kind of
> undesirable given that we already have
> https://github.com/mozilla/gecko-dev and
> https://github.com/mozilla/gecko which are two different conversions
> of m-c to git. I don't really know what the best way forward here, I
> just wanted to mention this because if you want searchfox to keep
> working we'll need some sort of git repo with the code.
>
> Also, my messages are getting stuck in the dev-planning moderation
> queue so I'm cc'ing Magnus directly here.
>
> Cheers,
> kats
>
> On Tue, Oct 9, 2018 at 12:51 PM Ben Bucksch <[hidden email]> wrote:
>> Hey kats,
>>
>> just a quick thank-you note for providing searchfox :)
>>
>> I think we'd need a new git repo, but that it would work like the old,
>> just that it already contains mozilla code, so it should be simpler,
>> actually.
>>
>> The main trick is that whatever script pushes to github needs to be adapted.
>>
>> Ben
>>
>> Am 10/9/2018 um 6:32 PM schrieb Kartikaya Gupta:
>>> Have you given any thought as to what would happen to the git mirror
>>> of the comm-central repo that currently lives at
>>> https://github.com/mozilla/releases-comm-central ? I ask because
>>> searchfox uses this git repo to do the indexing of comm-central and
>>> that might break with these changes. If you care about still having
>>> searchfox indexing available for comm-central after this change it
>>> would be good to continue having some sort of git mirror. If it's a
>>> brand new git mirror with different hashes we'll have to rebuild the
>>> blame repo and any comm-central permalinks (in bugs, etc.) will likely
>>> break. Or we can try harder to keep permalinks working by mapping the
>>> old hashes to the new ones but that's more work.
>>>
>>> Cheers,
>>> kats
>>> On Tue, Oct 9, 2018 at 11:48 AM Ben Bucksch <[hidden email]> wrote:
>>>> I like this idea. I'm in favor.
>>>>
>>>> If I understood correctly, you're proposing to take a mozilla-central
>>>> (or a mozilla-unified) repo, put comm-central checkout into the comm/
>>>> subdirectory, and push that to the branch "comm/default". I think that's
>>>> the right way to do things, yes.
>>>>
>>>> We do not even need buy-in from Firefox people, because hg is a
>>>> distributed system, so we can simply have our own repository that
>>>> contains a copy of mozilla-central plus comm/ (from former comm-central).
>>>>
>>>> I'm also in favor of re-adding CVS history. It's important for us, more
>>>> important than for Firefox, due to the longevity of our project.
>>>>
>>>> As for "which combination of mozilla and thunderbird is building and
>>>> working?", I would suggest to make a branch comm/bleeding where we merge
>>>> mozilla-central immediately and completely without tests. So, if there
>>>> are API changes in Mozilla and Thunderbird hasn't been updated yet, it
>>>> will be broken. (Code-wise, this is what we currently get, by checking
>>>> out mozilla-central and comm-central independently.) So, you have the
>>>> option of easily continuing to check out the latest and bleeding edge.
>>>> Then, once we tested the combination and we confirmed that it's working,
>>>> we merge comm/bleeding into comm/default, and it becomes the new
>>>> official Thunderbird development branch. This way, fixing an API
>>>> breakage becomes a hg commit that fixes the API and hg merge operation.
>>>>
>>>> The main advantage of this setup is that it's clear which version of
>>>> Mozilla works with which version of the Thunderbid code, down to the
>>>> commit level. And we capture this information using the version control
>>>> system itself, but with some build scripts.
>>>>
>>>> That should dramatically simplify development and build scripts.
>>>>
>>>> Ben
>>>>
>>>>
>>>> Am 10/9/2018 um 10:06 AM schrieb Magnus Melin:
>>>>
>>>>> I'd like feedback on the below plan for having Thunderbird in a single
>>>>> repository.
>>>>> Let's have follow-ups go to dev-planning
>>>>>
>>>>> TL;DR version: planning to have current comm-central live as a branch
>>>>> instead of a separate repository.
>>>>>
>>>>> ---
>>>>>
>>>>> Background
>>>>> ----------------
>>>>>
>>>>> When the Mozilla move from CVS to hg took place in 2008, the code
>>>>> related to Thunderbird was not included in mozilla-central and instead
>>>>> placed in the comm-central repository. Under the hood Thunderbird uses
>>>>> the mozilla-central code, so to build it currently has to check out
>>>>> both repositories. This is not ideal, for many many reasons.
>>>>>
>>>>> I wrote [an email to
>>>>> maildev](http://lists.thunderbird.net/pipermail/maildev_lists.thunderbird.net/2017-November/000875.html)
>>>>> last year listing some of the drawbacks of this setup - but
>>>>> essentially what it comes down to is that version control was not
>>>>> meant to be used like this and it makes workflows error prone, in
>>>>> addition to having to do various hacks in the build automation.
>>>>> Enabling autoland, phabricator and other tools used by Mozilla core
>>>>> could also be done much more easily for comm-central with a single
>>>>> repository.
>>>>>
>>>>> The mail spurred some related discussion on git vs hg and importing of
>>>>> history, but basically there was agreement among the comm-central
>>>>> developers to go ahead with this change.
>>>>>
>>>>> Thunderbird has now migrated its build automation to Taskcluster, and
>>>>> also moved to building with mozilla-central as topdir, with a comm/
>>>>> sub-folder containing comm-central code. (We used to instead have a
>>>>> mozilla subdir checkout of mozilla-central as child the comm-central
>>>>> checkout.)
>>>>>
>>>>> In 2014 there was [a proposal to merge comm-central code into
>>>>> mozilla-central](https://groups.google.com/forum/#!msg/mozilla.dev.planning/P8oHKTCWSiQ/76ECituhK-IJ)
>>>>> but this was rejected since the powers that be did not want 1) the
>>>>> confusion about what code belongs to what project, 2) pull/tree size,
>>>>> and 3) it was seen as irrelevant to Firefox. The proposal below has
>>>>> none of these obstacles, and depending on the physical repository
>>>>> location, may not change anything at all for a Firefox developer.
>>>>>
>>>>> I think it's now time to revisit the single repository idea and pull
>>>>> that through in one way or the other. So, on to the proposal. For
>>>>> reference, on disk comm/ is on currently 122 MB (+ comm/.hg 192 MB),
>>>>> so size-wise it is comparatively small, with a mozilla-central
>>>>> checkout weighing in at 2.3 GB. Including the relevant CVS history,
>>>>> the comm/.hg would be 317 MB in total.
>>>>>
>>>>> How
>>>>> ------
>>>>>
>>>>> To capture the essence of what Thunderbird is, we would make
>>>>> comm-central a named branch of the mozilla-central. Let's call this
>>>>> branch "comm/default".
>>>>>
>>>>> Initially, we'd pull in a hg converted version of the current
>>>>> comm-central default branch, putting all the files into a comm
>>>>> sub-folder in the base directory of the comm/default branch. This is
>>>>> the way that the Thunderbird build system already wants the code laid
>>>>> out. With the comm sub-folder it will be clear to everyone involved
>>>>> which code is on mozilla-central code and which is not. It is possible
>>>>> to import the CVS history for the related directories too while we're
>>>>> at it, but it adds some size. Importing related CVS history only adds
>>>>> around 125 MB extra so I'm inclined to include it. With the new
>>>>> comm/default branch pulled in, we merge the curent mozilla-central
>>>>> default branch to comm/default. The resulting code is now the same as
>>>>> status quo (of the both repos pulled in), just that it's all version
>>>>> controlled in the same repository.
>>>>>
>>>>> Once this is all set up, Thunderbird development work would move on on
>>>>> the comm/default branch, and there would be periodical merges with the
>>>>> (mozilla-central) default branch. These merges could be automated if
>>>>> wanted.
>>>>>
>>>>> Quite often it's the case that Thunderbird code needs to be adjusted
>>>>> to build, due to changes in mozilla-central code. For cases of known
>>>>> incoming bustage Thunderbird now would have the possibility of waiting
>>>>> to do the merge until a fix is available. I would suggest never to
>>>>> back out bustage causing mozilla-central changesets from the
>>>>> comm/default branch, but to detect other incoming bustage by doing
>>>>> builds from a comm/band-aid branch (branced from comm/default). The
>>>>> details on this can be discussed later though, not to derail
>>>>> discussion about the main issue here. There are many alternatives, and
>>>>> it's also completely possible to do what we currently do.
>>>>>
>>>>> Where
>>>>> ---------
>>>>>
>>>>> In the earlier discussions i's become clear there is a bunch of
>>>>> confusion around what is a branch and what is a repository, so please
>>>>> note the difference. Aa repository is the physical location where the
>>>>> history of a project is stored. Every repository does not have to
>>>>> carry every branch. I.e., the mozilla-central repository could carry
>>>>> the comm/default branch, or the comm/default branch could exist only
>>>>> in a comm-central2 repository somewhere else. In all cases it's
>>>>> trivial for developers who have mozilla-central checked out, to add
>>>>> the comm/default branch to their local checkout.
>>>>>
>>>>> So for the location there are at least three options:
>>>>>
>>>>> A: branch in the mozilla-central repository
>>>>> B: branch in the mozilla-unified repository
>>>>> C: branch in a repository elsewhere
>>>>>
>>>>> Not to disrupt operations too much I think it would be preferable to
>>>>> use option B - create the branch in the mozilla-unified repository. It
>>>>> does depend a bit on what plans Mozilla has for these repositories.
>>>>>
>>>>> Creating the branch in mozilla-unified of course needs buy-in from the
>>>>> Mozilla hg people, so please let me hear your opinions.
>>>>>
>>>>> What about history?
>>>>> --------------------------
>>>>>
>>>>> For the proposed approach (going through `hg convert`) commits are
>>>>> preserved, but AFAIK it's not possible to preserve actual commit
>>>>> hashes. `hg convert` will graft the original commits, and this adds
>>>>> the original hash as an extra field to the new commit (use `hg log
>>>>> --debug` to see it). I don't think this is such a big problem. The
>>>>> original commits would be linked in the pushlog, the same way there is
>>>>> "converted from" for mozilla-central-cvs, like
>>>>> https://hg.mozilla.org/experimental/mozilla-central-cvs/rev/ab117d946959.
>>>>> I don't see them e.g. in
>>>>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/rev/8f8b0933b611
>>>>> so perhaps there is some server side feature that needs turning on?
>>>>>
>>>>> Can I try it?
>>>>> ---------------
>>>>>
>>>>> You can check
>>>>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/ which
>>>>> carries the comm/default branch.
>>>>>
>>>>> In your mozilla tree, pull in the branch like this:
>>>>>
>>>>>        hg pull -u -b comm/default
>>>>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central
>>>>>
>>>>> To completely remove it again, use `hg strip "branch(comm/default)`
>>>>>
>>>>> How did you do it?
>>>>> ------------------------
>>>>>
>>>>>       CVS_GIT_REPO=~/tmp/mozilla-cvs-history/
>>>>>       CC_HG_REPO=~/Code/tb/mozilla/comm/
>>>>>
>>>>>       git clone https://github.com/ehsan/mozilla-cvs-history.git
>>>>> $CVS_GIT_REPO
>>>>>
>>>>>       echo default comm/default | tee branchmap.txt
>>>>>       echo rename . comm | tee filemap.txt cc0.txt
>>>>>
>>>>>       hg --cwd=$CC_HG_REPO up 0
>>>>>       (cd $CC_HG_REPO && ls -d */ | cut -f1 -d'/' | sed 's/^/include /')
>>>>>>> cc0.txt
>>>>>       hg convert --filemap=cc0.txt --branchmap=branchmap.txt
>>>>> $CVS_GIT_REPO single-repo/
>>>>>
>>>>>       CVS_TIP=`hg id --cwd=single-repo -i -r tip --debug`
>>>>>       CC_0=`hg id --cwd=$CC_HG_REPO -i -r 0 --debug`
>>>>>       echo $CC_0 $CVS_TIP | tee splicemap.txt
>>>>>       # e4f4569d451a5e0d12a6aa33ebd916f979dd8faa
>>>>> 93dd94bad983ce2eff5f228719c349b78a8a7f7d
>>>>>
>>>>>       hg convert --config convert.hg.saverev=True
>>>>> --splicemap=splicemap.txt --filemap=filemap.txt
>>>>> --branchmap=branchmap.txt $CC_HG_REPO single-repo/
>>>>>
>>>>> For fun, see all the history down to 1999 is there, like
>>>>>
>>>>>       hg log comm/mailnews/mime/src/mimei.cpp
>>>>>
>>>>> Then go on and add the default branch from mozilla-central and do the
>>>>> merge
>>>>>
>>>>>       cd single-repo
>>>>>       hg pull -f -b default $CC_HG_REPO/..
>>>>>
>>>>>       hg up comm/default
>>>>>       hg merge default && hg commit -m "Merge default to comm/default
>>>>> branch"
>>>>>
>>>>>       hg push -f -b comm/default --new-branch thunderbird-central-push
>>>>>
>>>>> ---
>>>>>
>>>>> Thoughts, comments, feedback appreciated!
>>>>>
>>>>>    -Magnus
>>>>>
>>>>> _______________________________________________
>>>>> tb-planning mailing list
>>>>> [hidden email]
>>>>> https://mail.mozilla.org/listinfo/tb-planning
>>>> _______________________________________________
>>>> dev-planning mailing list
>>>> [hidden email]
>>>> https://lists.mozilla.org/listinfo/dev-planning
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: comm-central future as a branch

Magnus Melin-2
In reply to this post by Frank-Rainer Grahl-3
On 09-10-2018 22:17, Frank-Rainer Grahl wrote:
>
> Pros:
> - Easier for new contributors to just check out one unified repo.
> - Merges between releases might be easier (or not).
> - Makes a possible future m-c fork easier.

That's a non-goal, but it is possible there could be cases where a
temporary forking of code is required.

> - Tool integration might be easier but most of them are a complete
> overkill for smaller projects such as TB and SM anyway.

There are *very* basic needs that are not met at the moment: you can't*
push a try build for what you have locally, links in crash reports are
not properly linked, there are many cases of broken permalinks
(searchfox and dxr), our build engineer is spending a bunch of time
trying to sort out bugs related to version mismatching and build
automation in general has to jump through hoops to be taught about
multiple repos and multiple checkouts.

I'm not saying these and other issues can't be resolved independently.
Of course it they can. It's just that it's wasted effort, compared to
just making it work the way it is supposed to work. Some of the systems
involved needs to be adjusted to understand branches better, but that
should be more of a one-time thing to fix.

  -Magnus

*) well you can, but you need to look up the m-c revision you want, and
set that up manually. if you know how

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: comm-central future as a branch

Magnus Melin-2
In reply to this post by Kartikaya Gupta-6
We definitely want to keep Searchfox working, I use it all the time.

I think at this stage it's hard to say anything about the details. If
Searchfox permalinks aren't based on the hg revisions, I imagine
supporting old permalinks can get somewhat difficult. OT but why isn't
it based on the mercurial version?

  -Magnus

On 09-10-2018 19:32, Kartikaya Gupta wrote:

> Have you given any thought as to what would happen to the git mirror
> of the comm-central repo that currently lives at
> https://github.com/mozilla/releases-comm-central ? I ask because
> searchfox uses this git repo to do the indexing of comm-central and
> that might break with these changes. If you care about still having
> searchfox indexing available for comm-central after this change it
> would be good to continue having some sort of git mirror. If it's a
> brand new git mirror with different hashes we'll have to rebuild the
> blame repo and any comm-central permalinks (in bugs, etc.) will likely
> break. Or we can try harder to keep permalinks working by mapping the
> old hashes to the new ones but that's more work.
>
> Cheers,
> kats
> On Tue, Oct 9, 2018 at 11:48 AM Ben Bucksch <[hidden email]> wrote:
>> I like this idea. I'm in favor.
>>
>> If I understood correctly, you're proposing to take a mozilla-central
>> (or a mozilla-unified) repo, put comm-central checkout into the comm/
>> subdirectory, and push that to the branch "comm/default". I think that's
>> the right way to do things, yes.
>>
>> We do not even need buy-in from Firefox people, because hg is a
>> distributed system, so we can simply have our own repository that
>> contains a copy of mozilla-central plus comm/ (from former comm-central).
>>
>> I'm also in favor of re-adding CVS history. It's important for us, more
>> important than for Firefox, due to the longevity of our project.
>>
>> As for "which combination of mozilla and thunderbird is building and
>> working?", I would suggest to make a branch comm/bleeding where we merge
>> mozilla-central immediately and completely without tests. So, if there
>> are API changes in Mozilla and Thunderbird hasn't been updated yet, it
>> will be broken. (Code-wise, this is what we currently get, by checking
>> out mozilla-central and comm-central independently.) So, you have the
>> option of easily continuing to check out the latest and bleeding edge.
>> Then, once we tested the combination and we confirmed that it's working,
>> we merge comm/bleeding into comm/default, and it becomes the new
>> official Thunderbird development branch. This way, fixing an API
>> breakage becomes a hg commit that fixes the API and hg merge operation.
>>
>> The main advantage of this setup is that it's clear which version of
>> Mozilla works with which version of the Thunderbid code, down to the
>> commit level. And we capture this information using the version control
>> system itself, but with some build scripts.
>>
>> That should dramatically simplify development and build scripts.
>>
>> Ben
>>
>>
>> Am 10/9/2018 um 10:06 AM schrieb Magnus Melin:
>>
>>> I'd like feedback on the below plan for having Thunderbird in a single
>>> repository.
>>> Let's have follow-ups go to dev-planning
>>>
>>> TL;DR version: planning to have current comm-central live as a branch
>>> instead of a separate repository.
>>>
>>> ---
>>>
>>> Background
>>> ----------------
>>>
>>> When the Mozilla move from CVS to hg took place in 2008, the code
>>> related to Thunderbird was not included in mozilla-central and instead
>>> placed in the comm-central repository. Under the hood Thunderbird uses
>>> the mozilla-central code, so to build it currently has to check out
>>> both repositories. This is not ideal, for many many reasons.
>>>
>>> I wrote [an email to
>>> maildev](http://lists.thunderbird.net/pipermail/maildev_lists.thunderbird.net/2017-November/000875.html)
>>> last year listing some of the drawbacks of this setup - but
>>> essentially what it comes down to is that version control was not
>>> meant to be used like this and it makes workflows error prone, in
>>> addition to having to do various hacks in the build automation.
>>> Enabling autoland, phabricator and other tools used by Mozilla core
>>> could also be done much more easily for comm-central with a single
>>> repository.
>>>
>>> The mail spurred some related discussion on git vs hg and importing of
>>> history, but basically there was agreement among the comm-central
>>> developers to go ahead with this change.
>>>
>>> Thunderbird has now migrated its build automation to Taskcluster, and
>>> also moved to building with mozilla-central as topdir, with a comm/
>>> sub-folder containing comm-central code. (We used to instead have a
>>> mozilla subdir checkout of mozilla-central as child the comm-central
>>> checkout.)
>>>
>>> In 2014 there was [a proposal to merge comm-central code into
>>> mozilla-central](https://groups.google.com/forum/#!msg/mozilla.dev.planning/P8oHKTCWSiQ/76ECituhK-IJ)
>>> but this was rejected since the powers that be did not want 1) the
>>> confusion about what code belongs to what project, 2) pull/tree size,
>>> and 3) it was seen as irrelevant to Firefox. The proposal below has
>>> none of these obstacles, and depending on the physical repository
>>> location, may not change anything at all for a Firefox developer.
>>>
>>> I think it's now time to revisit the single repository idea and pull
>>> that through in one way or the other. So, on to the proposal. For
>>> reference, on disk comm/ is on currently 122 MB (+ comm/.hg 192 MB),
>>> so size-wise it is comparatively small, with a mozilla-central
>>> checkout weighing in at 2.3 GB. Including the relevant CVS history,
>>> the comm/.hg would be 317 MB in total.
>>>
>>> How
>>> ------
>>>
>>> To capture the essence of what Thunderbird is, we would make
>>> comm-central a named branch of the mozilla-central. Let's call this
>>> branch "comm/default".
>>>
>>> Initially, we'd pull in a hg converted version of the current
>>> comm-central default branch, putting all the files into a comm
>>> sub-folder in the base directory of the comm/default branch. This is
>>> the way that the Thunderbird build system already wants the code laid
>>> out. With the comm sub-folder it will be clear to everyone involved
>>> which code is on mozilla-central code and which is not. It is possible
>>> to import the CVS history for the related directories too while we're
>>> at it, but it adds some size. Importing related CVS history only adds
>>> around 125 MB extra so I'm inclined to include it. With the new
>>> comm/default branch pulled in, we merge the curent mozilla-central
>>> default branch to comm/default. The resulting code is now the same as
>>> status quo (of the both repos pulled in), just that it's all version
>>> controlled in the same repository.
>>>
>>> Once this is all set up, Thunderbird development work would move on on
>>> the comm/default branch, and there would be periodical merges with the
>>> (mozilla-central) default branch. These merges could be automated if
>>> wanted.
>>>
>>> Quite often it's the case that Thunderbird code needs to be adjusted
>>> to build, due to changes in mozilla-central code. For cases of known
>>> incoming bustage Thunderbird now would have the possibility of waiting
>>> to do the merge until a fix is available. I would suggest never to
>>> back out bustage causing mozilla-central changesets from the
>>> comm/default branch, but to detect other incoming bustage by doing
>>> builds from a comm/band-aid branch (branced from comm/default). The
>>> details on this can be discussed later though, not to derail
>>> discussion about the main issue here. There are many alternatives, and
>>> it's also completely possible to do what we currently do.
>>>
>>> Where
>>> ---------
>>>
>>> In the earlier discussions i's become clear there is a bunch of
>>> confusion around what is a branch and what is a repository, so please
>>> note the difference. Aa repository is the physical location where the
>>> history of a project is stored. Every repository does not have to
>>> carry every branch. I.e., the mozilla-central repository could carry
>>> the comm/default branch, or the comm/default branch could exist only
>>> in a comm-central2 repository somewhere else. In all cases it's
>>> trivial for developers who have mozilla-central checked out, to add
>>> the comm/default branch to their local checkout.
>>>
>>> So for the location there are at least three options:
>>>
>>> A: branch in the mozilla-central repository
>>> B: branch in the mozilla-unified repository
>>> C: branch in a repository elsewhere
>>>
>>> Not to disrupt operations too much I think it would be preferable to
>>> use option B - create the branch in the mozilla-unified repository. It
>>> does depend a bit on what plans Mozilla has for these repositories.
>>>
>>> Creating the branch in mozilla-unified of course needs buy-in from the
>>> Mozilla hg people, so please let me hear your opinions.
>>>
>>> What about history?
>>> --------------------------
>>>
>>> For the proposed approach (going through `hg convert`) commits are
>>> preserved, but AFAIK it's not possible to preserve actual commit
>>> hashes. `hg convert` will graft the original commits, and this adds
>>> the original hash as an extra field to the new commit (use `hg log
>>> --debug` to see it). I don't think this is such a big problem. The
>>> original commits would be linked in the pushlog, the same way there is
>>> "converted from" for mozilla-central-cvs, like
>>> https://hg.mozilla.org/experimental/mozilla-central-cvs/rev/ab117d946959.
>>> I don't see them e.g. in
>>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/rev/8f8b0933b611
>>> so perhaps there is some server side feature that needs turning on?
>>>
>>> Can I try it?
>>> ---------------
>>>
>>> You can check
>>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/ which
>>> carries the comm/default branch.
>>>
>>> In your mozilla tree, pull in the branch like this:
>>>
>>>       hg pull -u -b comm/default
>>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central
>>>
>>> To completely remove it again, use `hg strip "branch(comm/default)`
>>>
>>> How did you do it?
>>> ------------------------
>>>
>>>      CVS_GIT_REPO=~/tmp/mozilla-cvs-history/
>>>      CC_HG_REPO=~/Code/tb/mozilla/comm/
>>>
>>>      git clone https://github.com/ehsan/mozilla-cvs-history.git
>>> $CVS_GIT_REPO
>>>
>>>      echo default comm/default | tee branchmap.txt
>>>      echo rename . comm | tee filemap.txt cc0.txt
>>>
>>>      hg --cwd=$CC_HG_REPO up 0
>>>      (cd $CC_HG_REPO && ls -d */ | cut -f1 -d'/' | sed 's/^/include /')
>>>>> cc0.txt
>>>      hg convert --filemap=cc0.txt --branchmap=branchmap.txt
>>> $CVS_GIT_REPO single-repo/
>>>
>>>      CVS_TIP=`hg id --cwd=single-repo -i -r tip --debug`
>>>      CC_0=`hg id --cwd=$CC_HG_REPO -i -r 0 --debug`
>>>      echo $CC_0 $CVS_TIP | tee splicemap.txt
>>>      # e4f4569d451a5e0d12a6aa33ebd916f979dd8faa
>>> 93dd94bad983ce2eff5f228719c349b78a8a7f7d
>>>
>>>      hg convert --config convert.hg.saverev=True
>>> --splicemap=splicemap.txt --filemap=filemap.txt
>>> --branchmap=branchmap.txt $CC_HG_REPO single-repo/
>>>
>>> For fun, see all the history down to 1999 is there, like
>>>
>>>      hg log comm/mailnews/mime/src/mimei.cpp
>>>
>>> Then go on and add the default branch from mozilla-central and do the
>>> merge
>>>
>>>      cd single-repo
>>>      hg pull -f -b default $CC_HG_REPO/..
>>>
>>>      hg up comm/default
>>>      hg merge default && hg commit -m "Merge default to comm/default
>>> branch"
>>>
>>>      hg push -f -b comm/default --new-branch thunderbird-central-push
>>>
>>> ---
>>>
>>> Thoughts, comments, feedback appreciated!
>>>
>>>   -Magnus
>>>
>>> _______________________________________________
>>> tb-planning mailing list
>>> [hidden email]
>>> https://mail.mozilla.org/listinfo/tb-planning
>> _______________________________________________
>> dev-planning mailing list
>> [hidden email]
>> https://lists.mozilla.org/listinfo/dev-planning
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: comm-central future as a branch

Magnus Melin-2
In reply to this post by Magnus Melin-2
On 10-10-2018 00:25, Axel Hecht wrote:

> Am 09.10.18 um 10:06 schrieb Magnus Melin:
>> I'd like feedback on the below plan for having Thunderbird in a single
>> repository.
>> Let's have follow-ups go to dev-planning
>>
>> TL;DR version: planning to have current comm-central live as a branch
>> instead of a separate repository.
>
> I like the general idea.
>
> But. I'll need the Thunderbird team to provide the patches necessary for
> localization. That involves cross-channel/gecko-strings code generation,
> l10n.mozilla.org, the automation behind it, and pontoon.

Yes, there are many systems that need adjustment (and/or verification),
but it shouldn't be an enormous effort. As it's conceptually clean, the
adjustments should not need to be hacks.

> Regarding comm-central revisions, cross-channel references them like in
> https://hg.mozilla.org/l10n/gecko-strings/rev/290a9db4634e. No idea what
> other parts of the web link to comm-central revisions. I guess bugzilla
> commit comments?

Bugzilla comments appears to be the main place yes. The old links would
continue to work as long as the comm-central repository is there. Third
party permalinks not based on the original hg commits (like Searchfox)
are more difficult.

  -Magnus


_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: comm-central future as a branch

Magnus Melin-2
In reply to this post by Jörg Knobloch
On 10-10-2018 02:57, Jörg Knobloch wrote:

> On 09/10/2018 10:06, Magnus Melin wrote:
>> Quite often it's the case that Thunderbird code needs to be adjusted
>> to build, due to changes in mozilla-central code. For cases of known
>> incoming bustage Thunderbird now would have the possibility of
>> waiting to do the merge until a fix is available. I would suggest
>> never to back out bustage causing mozilla-central changesets from the
>> comm/default branch, but to detect other incoming bustage by doing
>> builds from a comm/band-aid branch (branced from comm/default). The
>> details on this can be discussed later though, not to derail
>> discussion about the main issue here. There are many alternatives,
>> and it's also completely possible to do what we currently do.
>
> So what is the "main issue" if not handling bustage less stressfully?

The current setup is very difficult to understand for anybody new trying
to contribute. Many tools do not work properly, especially re getting
permalinks wrong, build automation having to deal with a very special
case... Something as basic as trying to create a build from a given date
should just work (but doesn't atm).

And handling bustage less stressfully is one of the main issues too. I
just said the details of how to do that should be discussed later.

>
> As the main firefighter I can tell you that waiting can be fatal,
> since the next bustage can be in the next merge you're not doing since
> you're waiting.

Agreed. The details for improvements in the flow can be sorted out
later. The proposal gives you options which you currently do not
reasonably have.

  -Magnus



_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: comm-central future as a branch

Gregory Szorc-3
In reply to this post by Magnus Melin-2
On Tue, Oct 9, 2018 at 5:49 PM Magnus Melin <[hidden email]> wrote:

> I'd like feedback on the below plan for having Thunderbird in a single
> repository.
> Let's have follow-ups go to dev-planning
>
> TL;DR version: planning to have current comm-central live as a branch
> instead of a separate repository.
>
> ---
>
> Background
> ----------------
>
> When the Mozilla move from CVS to hg took place in 2008, the code
> related to Thunderbird was not included in mozilla-central and instead
> placed in the comm-central repository. Under the hood Thunderbird uses
> the mozilla-central code, so to build it currently has to check out both
> repositories. This is not ideal, for many many reasons.
>
> I wrote [an email to
> maildev](
> http://lists.thunderbird.net/pipermail/maildev_lists.thunderbird.net/2017-November/000875.html)
>
> last year listing some of the drawbacks of this setup - but essentially
> what it comes down to is that version control was not meant to be used
> like this and it makes workflows error prone, in addition to having to
> do various hacks in the build automation. Enabling autoland, phabricator
> and other tools used by Mozilla core could also be done much more easily
> for comm-central with a single repository.
>
> The mail spurred some related discussion on git vs hg and importing of
> history, but basically there was agreement among the comm-central
> developers to go ahead with this change.
>
> Thunderbird has now migrated its build automation to Taskcluster, and
> also moved to building with mozilla-central as topdir, with a comm/
> sub-folder containing comm-central code. (We used to instead have a
> mozilla subdir checkout of mozilla-central as child the comm-central
> checkout.)
>
> In 2014 there was [a proposal to merge comm-central code into
> mozilla-central](
> https://groups.google.com/forum/#!msg/mozilla.dev.planning/P8oHKTCWSiQ/76ECituhK-IJ)
>
> but this was rejected since the powers that be did not want 1) the
> confusion about what code belongs to what project, 2) pull/tree size,
> and 3) it was seen as irrelevant to Firefox. The proposal below has none
> of these obstacles, and depending on the physical repository location,
> may not change anything at all for a Firefox developer.
>
> I think it's now time to revisit the single repository idea and pull
> that through in one way or the other. So, on to the proposal. For
> reference, on disk comm/ is on currently 122 MB (+ comm/.hg 192 MB), so
> size-wise it is comparatively small, with a mozilla-central checkout
> weighing in at 2.3 GB. Including the relevant CVS history, the comm/.hg
> would be 317 MB in total.
>
> How
> ------
>
> To capture the essence of what Thunderbird is, we would make
> comm-central a named branch of the mozilla-central. Let's call this
> branch "comm/default".
>
> Initially, we'd pull in a hg converted version of the current
> comm-central default branch, putting all the files into a comm
> sub-folder in the base directory of the comm/default branch. This is the
> way that the Thunderbird build system already wants the code laid out.
> With the comm sub-folder it will be clear to everyone involved which
> code is on mozilla-central code and which is not. It is possible to
> import the CVS history for the related directories too while we're at
> it, but it adds some size. Importing related CVS history only adds
> around 125 MB extra so I'm inclined to include it. With the new
> comm/default branch pulled in, we merge the curent mozilla-central
> default branch to comm/default. The resulting code is now the same as
> status quo (of the both repos pulled in), just that it's all version
> controlled in the same repository.
>
> Once this is all set up, Thunderbird development work would move on on
> the comm/default branch, and there would be periodical merges with the
> (mozilla-central) default branch. These merges could be automated if
> wanted.
>
> Quite often it's the case that Thunderbird code needs to be adjusted to
> build, due to changes in mozilla-central code. For cases of known
> incoming bustage Thunderbird now would have the possibility of waiting
> to do the merge until a fix is available. I would suggest never to back
> out bustage causing mozilla-central changesets from the comm/default
> branch, but to detect other incoming bustage by doing builds from a
> comm/band-aid branch (branced from comm/default). The details on this
> can be discussed later though, not to derail discussion about the main
> issue here. There are many alternatives, and it's also completely
> possible to do what we currently do.
>
> Where
> ---------
>
> In the earlier discussions i's become clear there is a bunch of
> confusion around what is a branch and what is a repository, so please
> note the difference. Aa repository is the physical location where the
> history of a project is stored. Every repository does not have to carry
> every branch. I.e., the mozilla-central repository could carry the
> comm/default branch, or the comm/default branch could exist only in a
> comm-central2 repository somewhere else. In all cases it's trivial for
> developers who have mozilla-central checked out, to add the comm/default
> branch to their local checkout.
>
> So for the location there are at least three options:
>
> A: branch in the mozilla-central repository
> B: branch in the mozilla-unified repository
> C: branch in a repository elsewhere
>
> Not to disrupt operations too much I think it would be preferable to use
> option B - create the branch in the mozilla-unified repository. It does
> depend a bit on what plans Mozilla has for these repositories.
>
> Creating the branch in mozilla-unified of course needs buy-in from the
> Mozilla hg people, so please let me hear your opinions.
>

So we're all on the same page with regards to terminology and how things
work...

"branch" is overloaded to mean different things.

From a Mercurial VCS perpsective, a "branch" is a name attached to
changesets within a repository. Mercurial branches are permanent and pretty
heavyweight. They are typically useful for tracking long-lived lines of
work, such as releases.

A Mercurial "branch" is different from a Git "branch." (We won't talk about
Git branches here because I don't think it is relevant.)

From a Firefox/Thunderbird Mercurial repository perspective, "branch" also
means "a specific Firefox repository." e.g. we refer to
hg.mozilla.org/{mozilla-central, integration/mozilla-inbound,
projects/larch, releases/mozilla-release} as "branches." I find this
terminology confusing because "branch" is ambiguous. But it's what a lot of
us use. (I would prefer the name "repository" or even "fork" instead.)

The Firefox Mercurial repositories typically have a single Mercurial
"branch:" "default." Exceptions are the release repositories, which have
various branches named *_RELBRANCH.

For historical reasons, the different Firefox Mercurial repositories mostly
all have a single DAG head. And that head is on the "default" Mercurial
branch.

The mozilla-unified repository is a read-only aggregation of various
Firefox repositories. Because the source repositories all use the "default"
branch, putting all these changesets in one repository means it isn't clear
which changesets "belong" to which repositories because all changesets
exist on the "default" branch. The aggregation process invents "bookmarks"
so names are associated with each DAG head and you can easily get a pointer
to the latest changeset from each source repository.

There are various things we don't like about the current Firefox repository
management.
https://docs.google.com/document/d/1mddoxa2i9ZgPLL_lGOZD0yNyS_ck3v1mzvp4GTEhC-0/edit?usp=sharing
tracks changes we'd like to make. There is general buy-in to make those
changes. But when we last talked about it seriously (at 1 of the prior 2
all hands), we agreed it wasn't high priority. So the work is on hold.

Regarding this proposal, I'm going to rule out adding Thunderbird code to
mozilla-central (A) at this point in time. People (and automated tools)
expect mozilla-central to have a single DAG head / branch and I think
adding a 2nd head/branch will cause problems. Plus there's a practical
concern of not wanting Thunderbird's code bloating the size of the
repository.

Because Thunderbird has separate project governance and support
requirements from Firefox (AFAIK), I think we should also rule out adding
Thunderbird code to mozilla-unified (B). I'm not super opposed to adding
Thunderbird code into mozilla-unified eventually if it makes sense. But I
don't think we should do it initially.

I think this new Thunderbird repository should live on its own, independent
from the Firefox repositories. Whether changesets should be in the
"default" branch or in some other Mercurial branch (so when changesets are
pulled into a Firefox repository you know they belong to Thunderbird), I
don't have a strong opinion. Having them named differently is more
compatible with a world where you are pulling Firefox and Thunderbird
repositories into the same repo. But having different branches/names also
introduces additional workflow requirements (people need to know to switch
between branches - and since we don't use Mercurial branches heavily in
Firefox land, people may not know how to do that).


>
> What about history?
> --------------------------
>
> For the proposed approach (going through `hg convert`) commits are
> preserved, but AFAIK it's not possible to preserve actual commit hashes.
> `hg convert` will graft the original commits, and this adds the original
> hash as an extra field to the new commit (use `hg log --debug` to see
> it). I don't think this is such a big problem. The original commits
> would be linked in the pushlog, the same way there is "converted from"
> for mozilla-central-cvs, like
> https://hg.mozilla.org/experimental/mozilla-central-cvs/rev/ab117d946959.
> I don't see them e.g. in
>
> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/rev/8f8b0933b611
> so perhaps there is some server side feature that needs turning on?
>

Correct. We have a very minimal server extension to render the
"convert_revision" changeset metadata (run `hg log --debug` to see it) as a
link to another repository. Since the conversion only stores the hash of
the source revision, we need to add a config option to the .hg/hgrc of the
repository to define which source repository to point to.

It appears modern versions of Mercurial support recording "convert_source"
changeset metadata (https://www.mercurial-scm.org/repo/hg/rev/c9093d4d1ff6).
We should change our server extension to always render that, if available.
Please file a Developer Services :: hg.mozilla.org bug to request that
feature if you will use it.

Anyway, I added the server config setting so the "converted from" links now
render on your user repository.


>
> Can I try it?
> ---------------
>
> You can check
> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/ which
> carries the comm/default branch.
>
> In your mozilla tree, pull in the branch like this:
>
>       hg pull -u -b comm/default
> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central
>
> To completely remove it again, use `hg strip "branch(comm/default)`
>
> How did you do it?
> ------------------------
>
>      CVS_GIT_REPO=~/tmp/mozilla-cvs-history/
>      CC_HG_REPO=~/Code/tb/mozilla/comm/
>
>      git clone https://github.com/ehsan/mozilla-cvs-history.git
> $CVS_GIT_REPO
>
>      echo default comm/default | tee branchmap.txt
>      echo rename . comm | tee filemap.txt cc0.txt
>
>      hg --cwd=$CC_HG_REPO up 0
>      (cd $CC_HG_REPO && ls -d */ | cut -f1 -d'/' | sed 's/^/include /')
>  >> cc0.txt
>
>      hg convert --filemap=cc0.txt --branchmap=branchmap.txt
> $CVS_GIT_REPO single-repo/
>
>      CVS_TIP=`hg id --cwd=single-repo -i -r tip --debug`
>      CC_0=`hg id --cwd=$CC_HG_REPO -i -r 0 --debug`
>      echo $CC_0 $CVS_TIP | tee splicemap.txt
>      # e4f4569d451a5e0d12a6aa33ebd916f979dd8faa
> 93dd94bad983ce2eff5f228719c349b78a8a7f7d
>
>      hg convert --config convert.hg.saverev=True
> --splicemap=splicemap.txt --filemap=filemap.txt
> --branchmap=branchmap.txt $CC_HG_REPO single-repo/
>

You may also want to add the config options from
https://hg.mozilla.org/hgcustom/version-control-tools/file/d54b06ec9d16/vcssync/mozvcssync/git2hg.py#l71
to ensure file copies and renames are more robustly detected. The
conversion will take longer. But it's a one-time thing and accuracy is
probably worth burning a few trillion CPU cycles on :)


>
> For fun, see all the history down to 1999 is there, like
>
>      hg log comm/mailnews/mime/src/mimei.cpp
>
> Then go on and add the default branch from mozilla-central and do the merge
>
>      cd single-repo
>      hg pull -f -b default $CC_HG_REPO/..
>
>      hg up comm/default
>      hg merge default && hg commit -m "Merge default to comm/default
> branch"
>
>      hg push -f -b comm/default --new-branch thunderbird-central-push
>
> ---
>

As for converting CVS history, I'm +1 on that.

I wouldn't plan on the hashes aligning between any future "official"
conversion of the Firefox CVS history to Mercurial and what you did or what
any other conversion did. *If* we produce an official Mercurial repository
with CVS history, we will likely do so using a different technique than
what existing conversions have done with conventional tools. (If we do an
official conversion or any other kind of flag day conversion, we'll likely
switch to Mercurial's "tree manifests" feature and that will rewrite hashes
by definition. And we will also likely wait for a non-SHA-1 hash to be
supported by Mercurial.)


>
> Thoughts, comments, feedback appreciated!
>

Overall this all seems pretty reasonable! Speaking as an hg.mo service
operator and someone with domain expertise about converting repositories, I
would like to perform a final review on everything before this goes into
production though. Please flag me for needinfo on a bug or reach out
privately. I'm on PTO a bit in the next ~2 weeks though. Please reach out
to my manager - Kim Moir - if I'm not responsive. And feel free to drop
into #vcs to ask questions.

Thank you for taking the time to write all this up and for reaching out for
feedback.


>
>   -Magnus
>
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: comm-central future as a branch

Kartikaya Gupta-6
In reply to this post by Magnus Melin-2
I wasn't working on searchfox during the early days when the
permalinks were made based on the git revision, so I don't have the
full context on that. But searchfox generally has git-centric
architecture (e.g. the tool to generate the blame repo works off a git
repo). In general more codebases have git repos than hg repos, so it
makes sense from that perspective.

On Thu, Oct 11, 2018 at 9:58 AM Magnus Melin <[hidden email]> wrote:

>
> We definitely want to keep Searchfox working, I use it all the time.
>
> I think at this stage it's hard to say anything about the details. If
> Searchfox permalinks aren't based on the hg revisions, I imagine
> supporting old permalinks can get somewhat difficult. OT but why isn't
> it based on the mercurial version?
>
>   -Magnus
>
> On 09-10-2018 19:32, Kartikaya Gupta wrote:
> > Have you given any thought as to what would happen to the git mirror
> > of the comm-central repo that currently lives at
> > https://github.com/mozilla/releases-comm-central ? I ask because
> > searchfox uses this git repo to do the indexing of comm-central and
> > that might break with these changes. If you care about still having
> > searchfox indexing available for comm-central after this change it
> > would be good to continue having some sort of git mirror. If it's a
> > brand new git mirror with different hashes we'll have to rebuild the
> > blame repo and any comm-central permalinks (in bugs, etc.) will likely
> > break. Or we can try harder to keep permalinks working by mapping the
> > old hashes to the new ones but that's more work.
> >
> > Cheers,
> > kats
> > On Tue, Oct 9, 2018 at 11:48 AM Ben Bucksch <[hidden email]> wrote:
> >> I like this idea. I'm in favor.
> >>
> >> If I understood correctly, you're proposing to take a mozilla-central
> >> (or a mozilla-unified) repo, put comm-central checkout into the comm/
> >> subdirectory, and push that to the branch "comm/default". I think that's
> >> the right way to do things, yes.
> >>
> >> We do not even need buy-in from Firefox people, because hg is a
> >> distributed system, so we can simply have our own repository that
> >> contains a copy of mozilla-central plus comm/ (from former comm-central).
> >>
> >> I'm also in favor of re-adding CVS history. It's important for us, more
> >> important than for Firefox, due to the longevity of our project.
> >>
> >> As for "which combination of mozilla and thunderbird is building and
> >> working?", I would suggest to make a branch comm/bleeding where we merge
> >> mozilla-central immediately and completely without tests. So, if there
> >> are API changes in Mozilla and Thunderbird hasn't been updated yet, it
> >> will be broken. (Code-wise, this is what we currently get, by checking
> >> out mozilla-central and comm-central independently.) So, you have the
> >> option of easily continuing to check out the latest and bleeding edge.
> >> Then, once we tested the combination and we confirmed that it's working,
> >> we merge comm/bleeding into comm/default, and it becomes the new
> >> official Thunderbird development branch. This way, fixing an API
> >> breakage becomes a hg commit that fixes the API and hg merge operation.
> >>
> >> The main advantage of this setup is that it's clear which version of
> >> Mozilla works with which version of the Thunderbid code, down to the
> >> commit level. And we capture this information using the version control
> >> system itself, but with some build scripts.
> >>
> >> That should dramatically simplify development and build scripts.
> >>
> >> Ben
> >>
> >>
> >> Am 10/9/2018 um 10:06 AM schrieb Magnus Melin:
> >>
> >>> I'd like feedback on the below plan for having Thunderbird in a single
> >>> repository.
> >>> Let's have follow-ups go to dev-planning
> >>>
> >>> TL;DR version: planning to have current comm-central live as a branch
> >>> instead of a separate repository.
> >>>
> >>> ---
> >>>
> >>> Background
> >>> ----------------
> >>>
> >>> When the Mozilla move from CVS to hg took place in 2008, the code
> >>> related to Thunderbird was not included in mozilla-central and instead
> >>> placed in the comm-central repository. Under the hood Thunderbird uses
> >>> the mozilla-central code, so to build it currently has to check out
> >>> both repositories. This is not ideal, for many many reasons.
> >>>
> >>> I wrote [an email to
> >>> maildev](http://lists.thunderbird.net/pipermail/maildev_lists.thunderbird.net/2017-November/000875.html)
> >>> last year listing some of the drawbacks of this setup - but
> >>> essentially what it comes down to is that version control was not
> >>> meant to be used like this and it makes workflows error prone, in
> >>> addition to having to do various hacks in the build automation.
> >>> Enabling autoland, phabricator and other tools used by Mozilla core
> >>> could also be done much more easily for comm-central with a single
> >>> repository.
> >>>
> >>> The mail spurred some related discussion on git vs hg and importing of
> >>> history, but basically there was agreement among the comm-central
> >>> developers to go ahead with this change.
> >>>
> >>> Thunderbird has now migrated its build automation to Taskcluster, and
> >>> also moved to building with mozilla-central as topdir, with a comm/
> >>> sub-folder containing comm-central code. (We used to instead have a
> >>> mozilla subdir checkout of mozilla-central as child the comm-central
> >>> checkout.)
> >>>
> >>> In 2014 there was [a proposal to merge comm-central code into
> >>> mozilla-central](https://groups.google.com/forum/#!msg/mozilla.dev.planning/P8oHKTCWSiQ/76ECituhK-IJ)
> >>> but this was rejected since the powers that be did not want 1) the
> >>> confusion about what code belongs to what project, 2) pull/tree size,
> >>> and 3) it was seen as irrelevant to Firefox. The proposal below has
> >>> none of these obstacles, and depending on the physical repository
> >>> location, may not change anything at all for a Firefox developer.
> >>>
> >>> I think it's now time to revisit the single repository idea and pull
> >>> that through in one way or the other. So, on to the proposal. For
> >>> reference, on disk comm/ is on currently 122 MB (+ comm/.hg 192 MB),
> >>> so size-wise it is comparatively small, with a mozilla-central
> >>> checkout weighing in at 2.3 GB. Including the relevant CVS history,
> >>> the comm/.hg would be 317 MB in total.
> >>>
> >>> How
> >>> ------
> >>>
> >>> To capture the essence of what Thunderbird is, we would make
> >>> comm-central a named branch of the mozilla-central. Let's call this
> >>> branch "comm/default".
> >>>
> >>> Initially, we'd pull in a hg converted version of the current
> >>> comm-central default branch, putting all the files into a comm
> >>> sub-folder in the base directory of the comm/default branch. This is
> >>> the way that the Thunderbird build system already wants the code laid
> >>> out. With the comm sub-folder it will be clear to everyone involved
> >>> which code is on mozilla-central code and which is not. It is possible
> >>> to import the CVS history for the related directories too while we're
> >>> at it, but it adds some size. Importing related CVS history only adds
> >>> around 125 MB extra so I'm inclined to include it. With the new
> >>> comm/default branch pulled in, we merge the curent mozilla-central
> >>> default branch to comm/default. The resulting code is now the same as
> >>> status quo (of the both repos pulled in), just that it's all version
> >>> controlled in the same repository.
> >>>
> >>> Once this is all set up, Thunderbird development work would move on on
> >>> the comm/default branch, and there would be periodical merges with the
> >>> (mozilla-central) default branch. These merges could be automated if
> >>> wanted.
> >>>
> >>> Quite often it's the case that Thunderbird code needs to be adjusted
> >>> to build, due to changes in mozilla-central code. For cases of known
> >>> incoming bustage Thunderbird now would have the possibility of waiting
> >>> to do the merge until a fix is available. I would suggest never to
> >>> back out bustage causing mozilla-central changesets from the
> >>> comm/default branch, but to detect other incoming bustage by doing
> >>> builds from a comm/band-aid branch (branced from comm/default). The
> >>> details on this can be discussed later though, not to derail
> >>> discussion about the main issue here. There are many alternatives, and
> >>> it's also completely possible to do what we currently do.
> >>>
> >>> Where
> >>> ---------
> >>>
> >>> In the earlier discussions i's become clear there is a bunch of
> >>> confusion around what is a branch and what is a repository, so please
> >>> note the difference. Aa repository is the physical location where the
> >>> history of a project is stored. Every repository does not have to
> >>> carry every branch. I.e., the mozilla-central repository could carry
> >>> the comm/default branch, or the comm/default branch could exist only
> >>> in a comm-central2 repository somewhere else. In all cases it's
> >>> trivial for developers who have mozilla-central checked out, to add
> >>> the comm/default branch to their local checkout.
> >>>
> >>> So for the location there are at least three options:
> >>>
> >>> A: branch in the mozilla-central repository
> >>> B: branch in the mozilla-unified repository
> >>> C: branch in a repository elsewhere
> >>>
> >>> Not to disrupt operations too much I think it would be preferable to
> >>> use option B - create the branch in the mozilla-unified repository. It
> >>> does depend a bit on what plans Mozilla has for these repositories.
> >>>
> >>> Creating the branch in mozilla-unified of course needs buy-in from the
> >>> Mozilla hg people, so please let me hear your opinions.
> >>>
> >>> What about history?
> >>> --------------------------
> >>>
> >>> For the proposed approach (going through `hg convert`) commits are
> >>> preserved, but AFAIK it's not possible to preserve actual commit
> >>> hashes. `hg convert` will graft the original commits, and this adds
> >>> the original hash as an extra field to the new commit (use `hg log
> >>> --debug` to see it). I don't think this is such a big problem. The
> >>> original commits would be linked in the pushlog, the same way there is
> >>> "converted from" for mozilla-central-cvs, like
> >>> https://hg.mozilla.org/experimental/mozilla-central-cvs/rev/ab117d946959.
> >>> I don't see them e.g. in
> >>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/rev/8f8b0933b611
> >>> so perhaps there is some server side feature that needs turning on?
> >>>
> >>> Can I try it?
> >>> ---------------
> >>>
> >>> You can check
> >>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/ which
> >>> carries the comm/default branch.
> >>>
> >>> In your mozilla tree, pull in the branch like this:
> >>>
> >>>       hg pull -u -b comm/default
> >>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central
> >>>
> >>> To completely remove it again, use `hg strip "branch(comm/default)`
> >>>
> >>> How did you do it?
> >>> ------------------------
> >>>
> >>>      CVS_GIT_REPO=~/tmp/mozilla-cvs-history/
> >>>      CC_HG_REPO=~/Code/tb/mozilla/comm/
> >>>
> >>>      git clone https://github.com/ehsan/mozilla-cvs-history.git
> >>> $CVS_GIT_REPO
> >>>
> >>>      echo default comm/default | tee branchmap.txt
> >>>      echo rename . comm | tee filemap.txt cc0.txt
> >>>
> >>>      hg --cwd=$CC_HG_REPO up 0
> >>>      (cd $CC_HG_REPO && ls -d */ | cut -f1 -d'/' | sed 's/^/include /')
> >>>>> cc0.txt
> >>>      hg convert --filemap=cc0.txt --branchmap=branchmap.txt
> >>> $CVS_GIT_REPO single-repo/
> >>>
> >>>      CVS_TIP=`hg id --cwd=single-repo -i -r tip --debug`
> >>>      CC_0=`hg id --cwd=$CC_HG_REPO -i -r 0 --debug`
> >>>      echo $CC_0 $CVS_TIP | tee splicemap.txt
> >>>      # e4f4569d451a5e0d12a6aa33ebd916f979dd8faa
> >>> 93dd94bad983ce2eff5f228719c349b78a8a7f7d
> >>>
> >>>      hg convert --config convert.hg.saverev=True
> >>> --splicemap=splicemap.txt --filemap=filemap.txt
> >>> --branchmap=branchmap.txt $CC_HG_REPO single-repo/
> >>>
> >>> For fun, see all the history down to 1999 is there, like
> >>>
> >>>      hg log comm/mailnews/mime/src/mimei.cpp
> >>>
> >>> Then go on and add the default branch from mozilla-central and do the
> >>> merge
> >>>
> >>>      cd single-repo
> >>>      hg pull -f -b default $CC_HG_REPO/..
> >>>
> >>>      hg up comm/default
> >>>      hg merge default && hg commit -m "Merge default to comm/default
> >>> branch"
> >>>
> >>>      hg push -f -b comm/default --new-branch thunderbird-central-push
> >>>
> >>> ---
> >>>
> >>> Thoughts, comments, feedback appreciated!
> >>>
> >>>   -Magnus
> >>>
> >>> _______________________________________________
> >>> tb-planning mailing list
> >>> [hidden email]
> >>> https://mail.mozilla.org/listinfo/tb-planning
> >> _______________________________________________
> >> dev-planning mailing list
> >> [hidden email]
> >> https://lists.mozilla.org/listinfo/dev-planning
> > _______________________________________________
> > dev-planning mailing list
> > [hidden email]
> > https://lists.mozilla.org/listinfo/dev-planning
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: comm-central future as a branch

Andrew  McCreight
In reply to this post by Magnus Melin-2
On Thu, Oct 11, 2018 at 6:58 AM Magnus Melin <[hidden email]> wrote:

> We definitely want to keep Searchfox working, I use it all the time.
>
> I think at this stage it's hard to say anything about the details. If
> Searchfox permalinks aren't based on the hg revisions, I imagine
> supporting old permalinks can get somewhat difficult. OT but why isn't
> it based on the mercurial version?
>

The Mercurial history for Firefox only goes back to 2008, but the Git
history includes the CVS history, so it goes all the way back to 1999 or
so. (Maybe there's some hg repo out there that has a version of the CVS
history now, I'm not sure.)


>   -Magnus
>
> On 09-10-2018 19:32, Kartikaya Gupta wrote:
> > Have you given any thought as to what would happen to the git mirror
> > of the comm-central repo that currently lives at
> > https://github.com/mozilla/releases-comm-central ? I ask because
> > searchfox uses this git repo to do the indexing of comm-central and
> > that might break with these changes. If you care about still having
> > searchfox indexing available for comm-central after this change it
> > would be good to continue having some sort of git mirror. If it's a
> > brand new git mirror with different hashes we'll have to rebuild the
> > blame repo and any comm-central permalinks (in bugs, etc.) will likely
> > break. Or we can try harder to keep permalinks working by mapping the
> > old hashes to the new ones but that's more work.
> >
> > Cheers,
> > kats
> > On Tue, Oct 9, 2018 at 11:48 AM Ben Bucksch <[hidden email]>
> wrote:
> >> I like this idea. I'm in favor.
> >>
> >> If I understood correctly, you're proposing to take a mozilla-central
> >> (or a mozilla-unified) repo, put comm-central checkout into the comm/
> >> subdirectory, and push that to the branch "comm/default". I think that's
> >> the right way to do things, yes.
> >>
> >> We do not even need buy-in from Firefox people, because hg is a
> >> distributed system, so we can simply have our own repository that
> >> contains a copy of mozilla-central plus comm/ (from former
> comm-central).
> >>
> >> I'm also in favor of re-adding CVS history. It's important for us, more
> >> important than for Firefox, due to the longevity of our project.
> >>
> >> As for "which combination of mozilla and thunderbird is building and
> >> working?", I would suggest to make a branch comm/bleeding where we merge
> >> mozilla-central immediately and completely without tests. So, if there
> >> are API changes in Mozilla and Thunderbird hasn't been updated yet, it
> >> will be broken. (Code-wise, this is what we currently get, by checking
> >> out mozilla-central and comm-central independently.) So, you have the
> >> option of easily continuing to check out the latest and bleeding edge.
> >> Then, once we tested the combination and we confirmed that it's working,
> >> we merge comm/bleeding into comm/default, and it becomes the new
> >> official Thunderbird development branch. This way, fixing an API
> >> breakage becomes a hg commit that fixes the API and hg merge operation.
> >>
> >> The main advantage of this setup is that it's clear which version of
> >> Mozilla works with which version of the Thunderbid code, down to the
> >> commit level. And we capture this information using the version control
> >> system itself, but with some build scripts.
> >>
> >> That should dramatically simplify development and build scripts.
> >>
> >> Ben
> >>
> >>
> >> Am 10/9/2018 um 10:06 AM schrieb Magnus Melin:
> >>
> >>> I'd like feedback on the below plan for having Thunderbird in a single
> >>> repository.
> >>> Let's have follow-ups go to dev-planning
> >>>
> >>> TL;DR version: planning to have current comm-central live as a branch
> >>> instead of a separate repository.
> >>>
> >>> ---
> >>>
> >>> Background
> >>> ----------------
> >>>
> >>> When the Mozilla move from CVS to hg took place in 2008, the code
> >>> related to Thunderbird was not included in mozilla-central and instead
> >>> placed in the comm-central repository. Under the hood Thunderbird uses
> >>> the mozilla-central code, so to build it currently has to check out
> >>> both repositories. This is not ideal, for many many reasons.
> >>>
> >>> I wrote [an email to
> >>> maildev](
> http://lists.thunderbird.net/pipermail/maildev_lists.thunderbird.net/2017-November/000875.html
> )
> >>> last year listing some of the drawbacks of this setup - but
> >>> essentially what it comes down to is that version control was not
> >>> meant to be used like this and it makes workflows error prone, in
> >>> addition to having to do various hacks in the build automation.
> >>> Enabling autoland, phabricator and other tools used by Mozilla core
> >>> could also be done much more easily for comm-central with a single
> >>> repository.
> >>>
> >>> The mail spurred some related discussion on git vs hg and importing of
> >>> history, but basically there was agreement among the comm-central
> >>> developers to go ahead with this change.
> >>>
> >>> Thunderbird has now migrated its build automation to Taskcluster, and
> >>> also moved to building with mozilla-central as topdir, with a comm/
> >>> sub-folder containing comm-central code. (We used to instead have a
> >>> mozilla subdir checkout of mozilla-central as child the comm-central
> >>> checkout.)
> >>>
> >>> In 2014 there was [a proposal to merge comm-central code into
> >>> mozilla-central](
> https://groups.google.com/forum/#!msg/mozilla.dev.planning/P8oHKTCWSiQ/76ECituhK-IJ
> )
> >>> but this was rejected since the powers that be did not want 1) the
> >>> confusion about what code belongs to what project, 2) pull/tree size,
> >>> and 3) it was seen as irrelevant to Firefox. The proposal below has
> >>> none of these obstacles, and depending on the physical repository
> >>> location, may not change anything at all for a Firefox developer.
> >>>
> >>> I think it's now time to revisit the single repository idea and pull
> >>> that through in one way or the other. So, on to the proposal. For
> >>> reference, on disk comm/ is on currently 122 MB (+ comm/.hg 192 MB),
> >>> so size-wise it is comparatively small, with a mozilla-central
> >>> checkout weighing in at 2.3 GB. Including the relevant CVS history,
> >>> the comm/.hg would be 317 MB in total.
> >>>
> >>> How
> >>> ------
> >>>
> >>> To capture the essence of what Thunderbird is, we would make
> >>> comm-central a named branch of the mozilla-central. Let's call this
> >>> branch "comm/default".
> >>>
> >>> Initially, we'd pull in a hg converted version of the current
> >>> comm-central default branch, putting all the files into a comm
> >>> sub-folder in the base directory of the comm/default branch. This is
> >>> the way that the Thunderbird build system already wants the code laid
> >>> out. With the comm sub-folder it will be clear to everyone involved
> >>> which code is on mozilla-central code and which is not. It is possible
> >>> to import the CVS history for the related directories too while we're
> >>> at it, but it adds some size. Importing related CVS history only adds
> >>> around 125 MB extra so I'm inclined to include it. With the new
> >>> comm/default branch pulled in, we merge the curent mozilla-central
> >>> default branch to comm/default. The resulting code is now the same as
> >>> status quo (of the both repos pulled in), just that it's all version
> >>> controlled in the same repository.
> >>>
> >>> Once this is all set up, Thunderbird development work would move on on
> >>> the comm/default branch, and there would be periodical merges with the
> >>> (mozilla-central) default branch. These merges could be automated if
> >>> wanted.
> >>>
> >>> Quite often it's the case that Thunderbird code needs to be adjusted
> >>> to build, due to changes in mozilla-central code. For cases of known
> >>> incoming bustage Thunderbird now would have the possibility of waiting
> >>> to do the merge until a fix is available. I would suggest never to
> >>> back out bustage causing mozilla-central changesets from the
> >>> comm/default branch, but to detect other incoming bustage by doing
> >>> builds from a comm/band-aid branch (branced from comm/default). The
> >>> details on this can be discussed later though, not to derail
> >>> discussion about the main issue here. There are many alternatives, and
> >>> it's also completely possible to do what we currently do.
> >>>
> >>> Where
> >>> ---------
> >>>
> >>> In the earlier discussions i's become clear there is a bunch of
> >>> confusion around what is a branch and what is a repository, so please
> >>> note the difference. Aa repository is the physical location where the
> >>> history of a project is stored. Every repository does not have to
> >>> carry every branch. I.e., the mozilla-central repository could carry
> >>> the comm/default branch, or the comm/default branch could exist only
> >>> in a comm-central2 repository somewhere else. In all cases it's
> >>> trivial for developers who have mozilla-central checked out, to add
> >>> the comm/default branch to their local checkout.
> >>>
> >>> So for the location there are at least three options:
> >>>
> >>> A: branch in the mozilla-central repository
> >>> B: branch in the mozilla-unified repository
> >>> C: branch in a repository elsewhere
> >>>
> >>> Not to disrupt operations too much I think it would be preferable to
> >>> use option B - create the branch in the mozilla-unified repository. It
> >>> does depend a bit on what plans Mozilla has for these repositories.
> >>>
> >>> Creating the branch in mozilla-unified of course needs buy-in from the
> >>> Mozilla hg people, so please let me hear your opinions.
> >>>
> >>> What about history?
> >>> --------------------------
> >>>
> >>> For the proposed approach (going through `hg convert`) commits are
> >>> preserved, but AFAIK it's not possible to preserve actual commit
> >>> hashes. `hg convert` will graft the original commits, and this adds
> >>> the original hash as an extra field to the new commit (use `hg log
> >>> --debug` to see it). I don't think this is such a big problem. The
> >>> original commits would be linked in the pushlog, the same way there is
> >>> "converted from" for mozilla-central-cvs, like
> >>>
> https://hg.mozilla.org/experimental/mozilla-central-cvs/rev/ab117d946959.
> >>> I don't see them e.g. in
> >>>
> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/rev/8f8b0933b611
> >>> so perhaps there is some server side feature that needs turning on?
> >>>
> >>> Can I try it?
> >>> ---------------
> >>>
> >>> You can check
> >>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/ which
> >>> carries the comm/default branch.
> >>>
> >>> In your mozilla tree, pull in the branch like this:
> >>>
> >>>       hg pull -u -b comm/default
> >>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central
> >>>
> >>> To completely remove it again, use `hg strip "branch(comm/default)`
> >>>
> >>> How did you do it?
> >>> ------------------------
> >>>
> >>>      CVS_GIT_REPO=~/tmp/mozilla-cvs-history/
> >>>      CC_HG_REPO=~/Code/tb/mozilla/comm/
> >>>
> >>>      git clone https://github.com/ehsan/mozilla-cvs-history.git
> >>> $CVS_GIT_REPO
> >>>
> >>>      echo default comm/default | tee branchmap.txt
> >>>      echo rename . comm | tee filemap.txt cc0.txt
> >>>
> >>>      hg --cwd=$CC_HG_REPO up 0
> >>>      (cd $CC_HG_REPO && ls -d */ | cut -f1 -d'/' | sed 's/^/include /')
> >>>>> cc0.txt
> >>>      hg convert --filemap=cc0.txt --branchmap=branchmap.txt
> >>> $CVS_GIT_REPO single-repo/
> >>>
> >>>      CVS_TIP=`hg id --cwd=single-repo -i -r tip --debug`
> >>>      CC_0=`hg id --cwd=$CC_HG_REPO -i -r 0 --debug`
> >>>      echo $CC_0 $CVS_TIP | tee splicemap.txt
> >>>      # e4f4569d451a5e0d12a6aa33ebd916f979dd8faa
> >>> 93dd94bad983ce2eff5f228719c349b78a8a7f7d
> >>>
> >>>      hg convert --config convert.hg.saverev=True
> >>> --splicemap=splicemap.txt --filemap=filemap.txt
> >>> --branchmap=branchmap.txt $CC_HG_REPO single-repo/
> >>>
> >>> For fun, see all the history down to 1999 is there, like
> >>>
> >>>      hg log comm/mailnews/mime/src/mimei.cpp
> >>>
> >>> Then go on and add the default branch from mozilla-central and do the
> >>> merge
> >>>
> >>>      cd single-repo
> >>>      hg pull -f -b default $CC_HG_REPO/..
> >>>
> >>>      hg up comm/default
> >>>      hg merge default && hg commit -m "Merge default to comm/default
> >>> branch"
> >>>
> >>>      hg push -f -b comm/default --new-branch thunderbird-central-push
> >>>
> >>> ---
> >>>
> >>> Thoughts, comments, feedback appreciated!
> >>>
> >>>   -Magnus
> >>>
> >>> _______________________________________________
> >>> tb-planning mailing list
> >>> [hidden email]
> >>> https://mail.mozilla.org/listinfo/tb-planning
> >> _______________________________________________
> >> dev-planning mailing list
> >> [hidden email]
> >> https://lists.mozilla.org/listinfo/dev-planning
> > _______________________________________________
> > dev-planning mailing list
> > [hidden email]
> > https://lists.mozilla.org/listinfo/dev-planning
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning