Keeping Mozilla and Adobe nanojit versions in sync.
This is a long email. Sorry, it needs to be a bit long.
Some of you may be aware, if you've been following nanojit development,
that Adobe and Mozilla are diverged. There are two copies of nanojit
embedded in independently-evolving repository histories: the Tamarin
history and the Tracemonkey history.
This is a historical accident, not a matter of policy. We had every
intention to be doing high-frequency / fine-granularity merges, simply a
lack of knowhow about how to do so *easily* and a lack of time to pursue
the correct technique. We've been trading high-priority patches for the
past nine months manually, and gradually diverging on other matters.
We have enough time now to pay attention to the issue again and try to
get it right. Previously -- as long ago as last year -- Edwin and I
tried forming a nanojit-only history with two repositories: one for
Adobe and one for Mozilla. Our hope, at the time, was that these two
repositories would hold tip revisions that were identical to the nanojit
subdirectories embedded in the Tamarin and Tracemonkey repositories,
respectively. We failed in that attempt for two reasons:
1. We did not start from merged repositories, but were playing
catch-up with the ongoing development in both Tamarin and
2. We were manually transporting patches and applying them by hand
or munging them via handmade scripts from one repository to another.
These two factors made it too awkward to regularly merge and we quickly
fell behind. I'm hoping we can find a way around these failure modes
presently, and have spent a little while hunting around for
alternatives. I think I have a way to do it, but it requires a little
discussion and a fair amount of buy-in from the development groups
involved. And thankfully, nobody else.
I'm proposing we pick a flag day both the Tamarin team and the
Tracemonkey team can agree on and simultaneously do the following:
1. Merge. Really really merge. Get everyone on the same IRC channel,
stop feature work, and set about resolving (with bug references,
regression testing and discussion) every remaining difference
between the two repositories, until we're identical. There is, as
far as I know, no fundamental disagreement on technical choices
between the teams, just a bunch of drift.
2. Re-seed the nanojit-only history Edwin and I attempted before,
but starting with this merged state.
3. Using 'hg convert', along with a splicemap and a filemap, splice
the nanojit-only history onto the merged (therefore identical) tips
of both the Tracemonkey and Tamarin repositories.
4. Add incoming hooks to the Tracemonkey and Tamarin repositories
(and local precommit hooks to all the local developer machines on
each team) *prohibiting* local checkins to the nanojit/
subdirectory. All changes to nanojit henceforth get made on the
nanojit-only history and copied to the Tracemonkey and Tamarin
repositories with 'hg convert'.
I've tested this sort of setup, and it can be made to work. It's not
even terribly hard. You wind up with a splicemap file like:
and you wind up making day-to-day nanojit changes like so:
~/src/nanojit-only$ hg commit --message="Some nanojit bug."
~/src/nanojit-only$ cd ..
~/src$ hg convert \
--source-type=hg --dest-type=hg \
--filemap=filemap --splicemap=splicemap \
--config convert.hg.saverev=True \
~/src$ cd tracemonkey/js/src/obj
~/src/tracemonkey/js/src/obj$ hg up
~/src/tracemonkey/js/src/obj$ make check
~/src/tracemonkey/js/src/obj$ hg push
(Replace Tracemonkey-specific paths with Tamarin paths in Tamarin's
I'll grant, this is not *totally* simple, but it's simple enough that it
can be internalized and hooked to reject incorrect actions. And
nobody outside our two development groups needs to care that this is
happening; it doesn't require any new machinery.
Every alternative I looked at involves adding repository metadata or
non-hg tools, which means convincing *all* hg users in the Adobe and
Mozilla development communities to install new code and learn to use it,
just to carry on doing what they're doing today. This is a steep cost I
wanted to avoid. The proposal above only affects a handful of
people. I'm happy to be pointed to more that this isn't true for. I just
discovered that 'convert' does splicing, file-renaming and hg-to-hg
operation recently. Those are the important bits you need to automate
1. Does the above proposal make sense, seem reasonable, suit
everyone's preferences, etc. etc.? Do you hate it? Can you think of
any general ways to improve it?
2. The 'hg convert' command does keep a mapping file associating
imported and local revision IDs. It may make more sense to check
this file in and commit it after each import, such that we all work
from the same associations; or it may make more sense to keep a
local copy of it elsewhere. I'm not sure. I believe it can be
reconstructed relatively easily in any case (particularly if
original-revision IDs get projected into the synthesized revisions,
as in my example). Any thoughts?
3. The proposal above involves sticking with *two* nanojit-only
repositories. This still permits the two repositories (within the
shared nanojit-only history) to diverge, and requires periodic pulls
from one to the other to absorb one another's work. And it will
necessarily involve a history with two evolving heads, two
"lineages". Would it makes more sense to try having only *one*
nanojit-only repository, that we insist remains functional for
*both* embeddings, both Tracemonkey and Tamarin?
4. If it's agreeable, what's a good flag-day to do it on?
Thanks for the read. Any further thoughts you have on the matter would
be great. Crossposting this to tamarin-devel and
Re: Keeping Mozilla and Adobe nanojit versions in sync.
On Jul 1, 11:41 am, Graydon Hoare <[hidden email]> wrote:
> 1. Merge. Really really merge. Get everyone on the same IRC channel,
> stop feature work, and set about resolving (with bug references,
> regression testing and discussion) every remaining difference
> between the two repositories, until we're identical. There is, as
> far as I know, no fundamental disagreement on technical choices
> between the teams, just a bunch of drift.
It'd be good to know for sure (as much as possible) if this is really
true before doing a whole lot of work...
> 4. Add incoming hooks to the Tracemonkey and Tamarin repositories
> (and local precommit hooks to all the local developer machines on
> each team) *prohibiting* local checkins to the nanojit/
> subdirectory. All changes to nanojit henceforth get made on the
> nanojit-only history and copied to the Tracemonkey and Tamarin
> repositories with 'hg convert'.
Why are the local hooks needed? Seems like we wouldn't want to
require it -- someone will forget to add them, and if they can then
stuff things up that is bad.
> and you wind up making day-to-day nanojit changes like so:
Will it be possible to end up with inconsistent nanojit and TM/Tamarin
> 3. The proposal above involves sticking with *two* nanojit-only
> repositories. This still permits the two repositories (within the
> shared nanojit-only history) to diverge, and requires periodic pulls
> from one to the other to absorb one another's work. And it will
> necessarily involve a history with two evolving heads, two
> "lineages". Would it makes more sense to try having only *one*
> nanojit-only repository, that we insist remains functional for
> *both* embeddings, both Tracemonkey and Tamarin?
That seems preferable to me, except that then any change to Nanojit's
interfaces will require that both TM and Tamarin be updated, which
Re: Keeping Mozilla and Adobe nanojit versions in sync.
Awkward though this is, if you can reply to [hidden email] as
well, it'll help keep the thread consolidated.
Nicholas Nethercote wrote:
> On Jul 1, 11:41 am, Graydon Hoare <[hidden email]> wrote:
>> There is, as
>> far as I know, no fundamental disagreement on technical choices
>> between the teams, just a bunch of drift.
> It'd be good to know for sure (as much as possible) if this is really
> true before doing a whole lot of work...
Of course. Though it's not entirely possible to know in advance, some of
the point of this message is to fish around for dissenting opinions from
either team. We may also stumble into things we differ on that need
resolving only *during* such a merge.
> Why are the local hooks needed? Seems like we wouldn't want to
> require it -- someone will forget to add them, and if they can then
> stuff things up that is bad.
Only to prevent you committing something you won't be allowed to push
anyway. Repository-side hooks *and* local hooks. You can skip the local
hook if you trust yourself. The server won't trust you either way.
> Will it be possible to end up with inconsistent nanojit and TM/Tamarin
Yes, but only transiently. I'm not proposing they move in lock-step, but
rather that entry into Tracemonkey or Tamarin requires passing through
the nanojit repository. Not necessarily merging it mind you; not unless
we require (also via repository hook) that it remain single-headed, the
way Tracemonkey or Mozilla-central is hooked. If you want maximally
synchronous operation, that's probably as close as we can get it.
There is a topologically different, but logically quite similar option,
which is to do the 'hg convert' operations directly from the Tracemonkey
repository to the Tamarin repository (and vice-versa). I fear this might
produce a slightly uglier merge history, but I'm not certain. I can poke
around with it if that sounds superior.
> That seems preferable to me, except that then any change to Nanojit's
> interfaces will require that both TM and Tamarin be updated, which
> seems problematic.
Again, I wasn't suggesting perfect lock-step, just some degree of causal
relationship between "doing our own work" and "exchanging work". In all
cases, this part is more "policy" than "mechanism", and would be hook
enforced on the repository side. I think we could pick any of 3 settings
for the dial:
1. You have to absorb a change to Tamarin's nanojit directory
immediately when it's pushed. Pushing to one causes a
corresponding convert + push to the other. Total lock-step.
2. You can't commit anything via the shared nanojit history,
hence can't commit to Tracemonkey's js/src/nanojit directory,
until you absorb changes from Tamarin's nanojit directory. But
you can wait and do nothing, if you need to talk it over more
to come to a consensus.
3. You can't (easily) merge any Tamarin changes outside the order they
occur there, so a change in Tamarin you want to delay will hold up
all other Tamarin changes you want to absorb.
I was thinking #3, but you're sounding more like you're leaning toward
#2, and I could certainly get behind that also, if others agree. It
requires more consensus and leaves less room for divergence, but that
also means less possibility for having to process ugly merges.
[re-cross-posted to mozilla.dev.tech.js-engine for those not on
On 09-07-06 10:22 AM, Edwin Smith wrote:
>> 3. ... Would it makes more sense to try having only *one*
>> nanojit-only repository, that we insist remains functional for
>> *both* embeddings, both Tracemonkey and Tamarin?
> Before you wrote this I have had thoughts taking shape about proposing
> a single (and single-headed) central nanojit repository, that includes
> nanojit that we have now, plus the assembler, tests that use the assembler,
> docs, etc. With a single headed lineage for "standalone nanojit", its very
> clear to all players what "nanojit" is, exactly what codebase the docs
> are describing, etc.
Yeah. Particularly if there's a testsuite (assembly + execution tests)
in there so there is a build-and-check target we can confirm against.
> Thinking out loud: if we had that one lineage, we can still allow
> (carefully managed, transient) divergence from it. It sounds like hg
> convert can still let us merge its history in TM and TR (and maybe any
> other project that might want to use it).
To some extent yes, but it requires care, and depends a bit on what we
wind up doing before and after each 'convert'.
'hg convert' works by appending partly-munged / synthetic csets. Those
csets are not patches, and it has no history to merge with. It just
drops the new state of files onto the end of the splice point. The csets
are statements of the form "file X changed to have content-id Y". This
turns into a clobber when you use 'convert' to append to a tree in which
file X has a pre-change state that differs from the one you were expecting.
So suppose we have a tracemonkey lineage like this:
And we convert an NJ history with a splicemap saying to splice it on to
TR3. We then get something like this:
Suppose we now do a little more non-NJ work in tracemonkey. Call it
"local divergence" if you like, or even just making non-NJ changes to
the tracemonkey tree. Mozilla-ish changes, things related to a web
browser, etc. We tack those on the end here:
And a little while later we fix an upstream bug in NJ and want to
'convert' it into this repository. Here is where we have to be careful
what we do. We have four options:
We convert with a new splicemap that tacks on to the front. So we wind
up with this:
This *clobbers* the state of any files in tracemonkey under nanojit/
with their upstream nanojit-repository state. If we had local divergence
in nanojit/, it's lost. If we only had differences outside nanojit/,
We convert with the existing splicemap (in fact we don't even need a
splicemap, we just reuse the .shamap generated from the first import).
So we wind up with this:
And so on. Every time we do a convert in this scenario we extend the
'convert lineage' and have to 'hg merge' it forward to our local tip. We
wind up with a 2-lineage *history* throughout tracemonkey and
mozilla-central repositories. One lineage for upstream nanojit, one
lineage for everything else. Merged before any pushes, of course. But
the future will always look like this:
If we're OK with this kind of history appearance, it's more honest. But
it's "ugly" for many people who dislike DVCSs. So we have...
In this case we do the same as option #2 but after converting, we
*rebase* rather than merge. This is a similar merge-operation to 'hg
merge' except that it writes its result to a surprising location:
In theory, this looks like the best outcome. It's tidy and we can
preserve local divergence. The difficulty is that we have to be very
careful to do exactly the right thing each time: we can't reuse a
.shamap or anything from before, but we are also not using a splicemap
that corresponds to tip. On the next convert, we want to splicemap on to
NJ3 even though it might not be tip, then rebase forwards. There is a
lot of "looking at glog and picking out targets" to do manually in this
approach. More like old gnarly CVS -j -j merges. Less easy to teach all
the developers involved to do. And if you get it wrong you get a total
rats nest. And so we have...
In this case we do the same as option #2, but after some number of
repeated merges from upstream we decide we don't want a doubled history
lineage so we get rid of any local divergence (making commits upstream
until local and upstream are identical), and then perform a 'convert
with an splicemap pointing to our tip, clobbering the
(identical-content) files. Then we are back in case #1. Looks like so:
As you can see, we can sort of mix-and-match policy here, but it's
easiest all around if we stick to a minimal-divergence setup. Possibly
everyone doing #1, no local divergence, just splicing onto tip no matter
what and making local adjustments *outside* the common nanojit/ files.
> if the single nanojit repo includes a reference embedding and
> tests, then both TR and TM can incrementally ensure that reference
> embedding and test suite adequately tests the library so when it's
> embedded it actually works. We also can create a series of benchmark
> tests for avoiding regressions in micro-optimizations.
Agreed. I think the more testing and benchmarking machinery we can put
inside a shared repo, the better.
> Lastly, I wonder if there's a way we can put a process in place before
> we actually arrive at the single merged point? Would this be easier
> with a single-headed proces
Oh ho ho! You have a very fine point! It seems that 'hg convert' can be
used, with a selective filemap, to split *out* the nanojit-only history
of both the tracemonkey and the tamarin trees. So we might be able to do
something involving splitting out the two histories and rebasing one on
the other, or just 'hg merge'ing them. Or something. It might be better
than hand-migrating them, and/or cause a lesser flag-day.
I'll play around with this more. Can you shop this idea around with
anyone in Adobe who may not have seen this email yet, see if they concur
or have objections to trying again?
Re: Keeping Mozilla and Adobe nanojit versions in sync.
On 09-07-06 03:10 PM, Graydon Hoare wrote:
> On 09-07-06 10:22 AM, Edwin Smith wrote:
>> Lastly, I wonder if there's a way we can put a process in place before
>> we actually arrive at the single merged point? Would this be easier
>> with a single-headed proces
> Oh ho ho! You have a very fine point! It seems that 'hg convert' can be
> used, with a selective filemap, to split *out* the nanojit-only history
> of both the tracemonkey and the tamarin trees. So we might be able to do
> something involving splitting out the two histories and rebasing one on
> the other, or just 'hg merge'ing them. Or something. It might be better
> than hand-migrating them, and/or cause a lesser flag-day.
> I'll play around with this more. Can you shop this idea around with
> anyone in Adobe who may not have seen this email yet, see if they concur
> or have objections to trying again?
Followup: I've extracted both the tamarin-redux and tracemonkey
nanojit-specific lineages. You guys have 128 revs that touch nanojit
since september, when nanojit first shows up in that repo; and we have
409 revs since last june, when we first copied nanojit out of
The last place we *claimed* to be merged was around October 6. You have
rev 87330c022cd1 (Oct 6) claiming it's sync'ed to TM tip, and we have
rev 53072c29a4fe (Oct 11) claiming it's sync'ed to tamarin-redux tip.
So what I did was splice the filtered-and-renamed history of
tamarin-redux's changes to nanojit, starting from your sync-point's
first child, on to the filtered-and-renamed history of our sync-point,
as a new synthetic child rev. This produces a 2-headed repository with a
common point supposedly at our last merge, that doesn't cleanly merge at
the moment, but at least appears to start at the right historical
divergence-point and have sensible 'hg blame' for each of the changes
along the way.
I can rebuild this repo again with slight differences if you like (say,
attaching our lineage to your merge-point as a child, or having hg
annotate the synthetic revs with the source revision IDs they came
from?) but otherwise I think this might be a reasonable repository to
carry on our earlier attempts at merging in. I can actually -- I think
-- continue importing day-to-day changes occuring in the tamarin-redux
and tracemonkey repositories into this shared one, while we work on
Then when it finally gets to a merged state, we can flag-day it, reverse
polarity, change the direction of imports, ask IT to set up a hook to
prevent multi-head states in the shared version, etc.
If I'm understanding this, it won't affect developers' private use of
patch queues --- it only affects the procedure for landing a patch, and
when that landing actually becomes visible in the full Mozilla or
Tamarin trees. Is that right?
If so, it seems like a great plan. I guess the most natural thing would
be to have a single common nanojit-only repository constrained to have
at most two heads, one for each context the tree gets spliced into.
(After a merge, it might have only one.) Since even an ideal version
control system wouldn't allow us to escape the usual negotiations
involved in sharing code between two contexts, we should allow ourselves
the current standard tool for managing such activity: a multi-headed
dev-tech-js-engine mailing list
[hidden email] https://lists.mozilla.org/listinfo/dev-tech-js-engine
On 09-07-07 08:59 AM, Edwin Smith wrote:
> Sounds like consensus on a single-headed single toplevel nanojit repo?
> i.e. never let it have two heads; dealing with divergence between that
> one head and an embedding (TM or TR or other) becomes an embedding-specific
> issue to manage? (that's my preference).
I'd prefer that too. It's hard enough to work out what to do about your
"owned" divergence. Figuring out what to do about divergence that's
motivated by *someone else's* policies and timelines is even more likely
get to stuck in inaction-paralysis.
> Could you create a user repo with that merged lineage?
Sure, I can upload it to a users/graydon_mozilla.com repo in a bit. I'm
going to toy with possibly-different arrangements first (filtering out
avmplus.h etc.) Though I'd recommend *not* making any commits to it.
Make non-build-breaking, merge-y commits to the tamarin-redux and
tracemonkey repos and repeatedly import them to the shared one, until
such time as we reverse the flow.
> could we make the merge effort simpler?
Not sure. Having them in shared-history state makes it somewhat simpler,
in the sense that kdiff3 can automatically reconcile eg. 244 of the 316
hunks differing in Assembler.cpp.
I'm not sure, however, if the remaining 72 hunks can be done
independently, or if it's best to work out which csets were involved and
port them feature-by-feature, or try to "big-bang" it ...
At least it's plausible to check for non-regression any way we do it, if
we make the commits land on our working/product repositories first.
That's better than the previous strategy.
It's unclear to me if there's ever a truly simple way to merge.
> a) take TM's NJ, add missing big animals from TR, test, declare victory
> b) take TR's NJ, add missing big animals from TM, test, declare victory
> (a) involves inspecting and manually merging the 100 or so changes from TR,
> whereas (b) requires doing that with 400+ TM changes. probably (a) is easier.
Both also possible. It's less the big animals that concern me, more the
hundreds of little one-liner differences. Easy to lose some (and worse,
harder to tell which variant ought to "win" during a merge).