Drivers meeting, 2006-03-21, 13:00-13:45
Present: Chris Hofmann (chofmann), David Baron (dbaron),
Mike Kaply (mkaply), Robert O'Callahan (roc),
Benjamin Smedberg (bsmedberg), Tim Rowley (tor),
Mike Schroepfer (schrep), Scott MacGregor (mscott)
mkaply: What do we want to do about regular drivers calls? Re-energize
drivers activity? And can we say something about what drivers is
responsible for? (Relative to product teams, etc.)
mkaply: We did agree last week that when issues get brought up, we'll
work harder to drive things to completion; I'll be the point person for
that for now.
mkaply: Any feedback based on the minutes from last week, esp. from
those not here then?
roc: Key question: is drivers about Gecko or about more than that?
mkaply: We need to be able to drive Gecko changes into Firefox release
if it's important for Gecko as a platform. Maybe drivers owns ensuring
that core Gecko changes are in the right releases?
roc: It's not clear that the people who own the product actually want
that to be the case. It seems like the people here right now are all
mkaply: Should we put together a proposal for what we think drivers does
and put it out for review and comments?
bsmedberg: Have we even solved the issue about drivers and daily triage
/ release management.
mkaply: What came up last week is that a lot of that is falling to
(mostly employed) people on the project teams. We should be the people
to say what's important for the platform, although maybe it's not that
black and white.
mkaply: Do people think the drivers=Gecko / product team=app is the
distinction we need? For pushing platform.
bsmedberg: From the discussion on mozilla-general, I'm not sure if we
need a distinction between module owners and project teams.
mkaply: Should the Firefox product team be the ones to decide about
svg:textPath in Fx 2.0?
roc: No, it should still be drivers with the power to make that
decision (after discussion in public).
bsmedberg: So when you do have conflict between product team and
platform, or between multiple products, who solves it?
chofmann: One of the things drivers did a few years back was to gather
up all of the release plans (then, mostly Netscape, but other products
too), take all the input, and try to figure out a tree management plan
that helped meet most of those needs -- so people didn't have to go off
on long-lived branches. We should look at whether or not that's a role
that still needs to be served. We don't happen to be in a part of the
development cycle where that's happening. When that function is needed,
we need someone to perform that function to avoid branch nightmares and
many different versions of Gecko going out.
mkaply: Tree management seems to be a role that's fallen down a little
chofmann: It's a job that comes and goes, and somebody has to keep an
eye on that. In the past it's been a cooperative effort of people on
drivers; either because people say current plan isn't working, or
because people complain about lack of plan given incoming requests.
bsmedberg: If we say drivers has role of managing Gecko, I'm not sure
what's different about that from the module owners in Gecko. How is
resolving conflicts between multiple module owners in Gecko different
from resolving conflicts between module owner of Camino and something
roc: I think the debate about svg:textPath was below the level that the
product teams would care about (except maybe for footprint).
bsmedberg: I don't see any reason for drivers to limit itself to the
platform. It should have representatives from the platform and from the
various products, so conflicts can be resolved in that forum.
mscott: In the past, we've tried to make sure that it reflected leads
from various projects. It seems like we haven't been updating it that
well, e.g., a representative for Camino.
mkaply: Maybe a way to describe drivers is that it's a way for the
representatives of the different projects to get together? Maybe put
the leads of the major projects on drivers automatically?
mscott: I think that's good, esp. if you're trying to act as an
abitrator for escalation, it's good to have representation from across
bsmedberg: I think that's a good model. I think somebody expressed
concerns about that on mozilla-dev-general: that there might be some
module owners who are great module owners but uninterested in global
management. I think that's the right way to go.
roc: I was talking about some subset of Gecko module owners when I said
that, not the product owners.
mscott: I think both those things can happen on the drivers list. I
don't think it has to be just products vs. Gecko; all these things could
be resolved on drivers.
mkaply: That way drivers doesn't decide things in a vacuum; there's
input from the product teams, and we're not forcing things down their
throat; there has to be some buy-in. But there should also be consensus
that the platform is important -- e.g., that Firefox move to XULRunner.
mscott: There are a lot of people on drivers representing platform and
looking out for that.
mkaply: So what do we want to say about drivers and release management
/ bug triage? Divide by product areas, e.g., app vs. platform?
bsmedberg: I don't think the drivers group here should be in charge of
roc: Brendan's pretty adamant that they should be related.
bsmedberg: In any case, it's going to have to be a small subset of this
group of product and module owners, and you need vast amounts of time.
mkaply: Yep, time's a big issue. And the sad part is that it's a lot
faster with a bunch of people in Mountain View in a room rather than
having people on the phone. That's more work for Mountain View people,
a little unsure about abdicating that responsibility, but it's a lot
better with everybody in the room.
bsmedberg: If this drivers list we're talking about is actually in
charge of small-scale release management, it has to discuss security
bugs. And then you need a list for private stuff.
mkaply: We have the security list for that. Are any drivers not on
mscott: There could be if we expand drivers.
mkaply: I definitely think we need broader product representation, and I
think there needs to be some level of platform responsibility at the
drivers level, but we need buy-in from the project teams. mscott, how
do you feel if people say to you that Gecko change X has to go in to
mscott: It doesn't bother me; although usually that happens with me in a
room with a bunch of others doing triage.
mkaply: Do you think bug triage works better with people in a room?
mscott: Yes, seems to go quicker with people in a room looking at a
mkaply: chofmann, do you think in the future release managers will
mostly be Mozilla Corporation folks or people outside?
chofmann: I don't see any reason it has to be one way or the other.
schrep: It's not true for Camino, SeaMonkey. And I don't want a
situation where it's strictly Mozilla Corporation that does that; and if
it's their job I'd rather have it be a delegated rather than abdicated
mkaply: Now it seems that all the bugs are thrown at drivers, and
anybody who has time does approvals until the end of the release, when
people get in a room. I wonder how other open source projects solve the
triage problem for releases.
mscott: That's a good question.
bsmedberg: I think the only project comparable in scope is openoffice.
mkaply: ... and I bet the Sun developers all get in a room.
mkaply: Well, we need to figure out who does approvals and who doesn't.
A lot of people see that as a big role of drivers.
chofmann: If we change things, we may need some infrastructure changes,
e.g., redirecting email for approval requests that currently goes to
schrep: We already have that for Firefox2; we have that for the current
state with module owner approval.
chofmann: But that won't last into the endgame.
chofmann: We used to have open vs. frozen tree, with drivers in control
for the latter.
mkaply: So do we want to throw up proposals on the newsgroups and get
discussions/feedback on specific issues.
chofmann: Should we switch part of drivers tasks to a generic endgame
approvers list, and make part of drivers' responsibility be making sure
that the right people are on that list?
mkaply: We could designate people as owning certain types of approvals.
Perhaps even down to giving people responsibility for certain sets of
directories? Would that make things easier or harder?
mscott: To some extent we do that already for things that aren't shared;
in the endgame I approve Tb-specific changes on my own, but not platform
chofmann: It would be good if the endgame-approval function stayed as
stable as possible. We want people to know where to go. We could miss
security bug fixes because people don't understand the approval process.
mkaply: Maybe we need an approvers list separate from drivers.
chofmann: Maybe an approvers list, but let's not go to a
bonecho-approvers, camino-approvers, etc., etc.
mkaply: Agreed. And maybe we could encourage people, when they ask for
approval, to be specific about what product it's for.
chofmann: I think it's good to have some level of continuity on the
approvers list. In the past few years, we've drawn new people in, but
some things get dropped, perhaps going around the table to various
parties and seeing if they know of issues outside of the formal
nomination/approval process. In the cases where we've made shipping
mistakes, we've had to revisit those.
mscott: I think we're doing a better job with the blog now.
chofmann: Also things with l10n, etc., etc. Lots of little details.
chofmann: Sounds like recommendation is to set up an approvers list, and
break that out of drivers responsibility. And drivers work together to
make sure approvers is staffed appropriately.
mkaply: Overriding goal about platform?
<others>: I don't think we agreed on that.
mkaply: Meet again in 2 weeks at this time. Maybe get more people on
the phone next time, get to some conclusions.
L. David Baron <URL: http://dbaron.org/ >
Technical Lead, Layout & CSS, Mozilla Corporation
Something seems missing that I've written about at length in the
Drivers was always about global project management. Not just Gecko,
because we want the UA rv: to mean the same thing across products. Not
just user-facing products, because they have requirements on Gecko, and
they are affected by Gecko decisions.
Sure, most of the potential conflicts are resolved by normal module
owner and peer interactions. That's as it should be, and it should be
But not all conflicts end at that level.
If there is no such group as drivers, then we will fail to consolidate,
share, and perpetuate culture and knowledge about risk assessment,
platform strategy, and competitive product / platform features.
But drivers is not just a tea-party for leaderly types. In the best
case, it is a way to get a leg up on bad old bugs and design flaws that
need to be fixed early in the cycle, and whose final followup fixes need
to be tended late in the cycle when triage vultures are swooping low to
pick off newborns based on different risk/reward understandings, or
misunderstandings. In the best case, drivers are directing design and
longer-term implementation, to the extent that module owners do not make
globally optimal decisions.
I can cite examples for all of these points. Perhaps you all can too.
> roc: Key question: is drivers about Gecko or about more than that?
> mkaply: We need to be able to drive Gecko changes into Firefox release
> if it's important for Gecko as a platform. Maybe drivers owns ensuring
> that core Gecko changes are in the right releases?
> roc: It's not clear that the people who own the product actually want
> that to be the case. It seems like the people here right now are all
> Gecko people.
> bsmedberg: So when you do have conflict between product team and
> platform, or between multiple products, who solves it?
Ultimately, I think that as a project, in the general case, we should
try to persuade where possible, rather than use that power. Which isn't
to say that there isn't an ultimate override lurking somewhere, but if
something is the Right Thing it shouldn't be hard to convince the
Of course, as roc points out, the current split on drivers still skews
very heavily towards Gecko people, is there a real desire to bring more
people in from the app side? I'd be more than happy to step up, if
that's appropriate and welcome.
> roc: I think the debate about svg:textPath was below the level that the
> product teams would care about (except maybe for footprint).
Not knowing the context of the debate, I think I would have cared about
more than footprint, though if we're trying to hold a certain line on
footprint that would at least matter in our planning. Maybe that makes
me smarter and/or more uptight than the average app-focused bear?
Anything that presents significant risk (above the statistical noise
level), when the product-level planning is assuming stability and little
risk, is something in which the product teams should have an absolutely
> mkaply: So what do we want to say about drivers and release management
> / bug triage? Divide by product areas, e.g., app vs. platform?
> bsmedberg: I don't think the drivers group here should be in charge of
> roc: Brendan's pretty adamant that they should be related.
> bsmedberg: In any case, it's going to have to be a small subset of this
> group of product and module owners, and you need vast amounts of time.
> mkaply: Yep, time's a big issue. And the sad part is that it's a lot
> faster with a bunch of people in Mountain View in a room rather than
> having people on the phone. That's more work for Mountain View people,
> a little unsure about abdicating that responsibility, but it's a lot
> better with everybody in the room.
As someone who spent probably two hundred hours on triage calls for 1.5
last year, I don't know how much slower it really was than when I was in
the office. Though I think I was usually the only one on the phone, so
that could skew things.
Handing off day to day bits to a specific subset of mostly-MV people is
still likely the best way, as long as larger/scarier changes go back to
the whole group, or the larger group can overrule the triage decision
where needed. We ran into this late in Fx 1.5 where certain things were
rejected even though the risk was deemed low, and that caused some friction.
> bsmedberg: If this drivers list we're talking about is actually in
> charge of small-scale release management, it has to discuss security
> bugs. And then you need a list for private stuff.
> mkaply: We have the security list for that. Are any drivers not on
> mscott: There could be if we expand drivers.
Thinking of the current, fairly large security group, I'm not thinking
of anyone who isn't already a member that really fits into the context
of the driver model. If there is such a person, I'm sure they would
have the credibility to be quickly accepted into s-g.