Self-Introduction: Fergus Sullivan

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Self-Introduction: Fergus Sullivan

Fergus Sullivan
== Full name ==
Fergus Sullivan

== Your IRC nick on irc.mozilla.org ==
ferg2k

== City, Country ==
Sunnyvale, California, United States.

== Profession or Student status ==
Engineering Manager

== Company, School, or other affiliation ==
Yahoo

== What do you want to help out with? ==
Scaling for enterprise level use.
UI overhaul

== Historical qualifications ==
Master's degree in medieval archaeology.  Although I suspect that's  
not what the question is alluding to.  That said, it does help when  
digging through layers of modifications and additions.

I've been managing our Bugzilla instance for the last 4 years, and  
spent much of the last year coding, albeit mostly UI. Other members  
of my team are the hard-core sql guys, and ttk jocks.

== Anything else you'd like to say ==
Yahoo has been happily using Bugzilla for the last six years.  We now  
have a mandate from our top-level management to contribute to the  
open source effort.

We are currently using a customized version of 2.22.  We serve  
Bugzilla on FreeBSD 6, mysql 4.1, perl 5.6.  We use a mixture of  
MyISAM and InnoDB.  We have a dual-master database (we have our own  
proxy that can switch as needed) and a VIP handling 3 front-end and  
servers and search slaves.  We have a second colo for BCP with an  
equivalent architecture.  We have additional machines as reporting  
slaves.

Our database currently stands at about 1.5 million bugs and grows at  
about 50k per month.  We have 20,000 named accounts (one per  
employee, plus many internal mailing lists), of whom about 8,300 are  
active in any one month.  In terms of active users and new bugs  
opened, our load has increased 50% in the last year.  We have 2,400  
separate products in our DB.

Nobody outside of Yahoo has access to our database.  All our  
customers are internal.

We test against IE 6, IE 7, Firefox 2, Safari 2 and 3.  We currently  
also support Opera.  We support Mac 10.4, Windows 2000, XP, and  
Vista, FreeBSD 4 and 6, RHEL 4 and 5.

In addition to myself, we have two developers and one QA assigned  
full time on Bugzilla.  We have another two guys working full time on  
it at the moment, although they may move to other tasks in a few  
months time.

I have an open req for a visual designer to polish up the UI in  
Bugzilla, and then in our other internal tools.

The core functionality has mostly met our needs, but we have had to  
customize in the following areas.

Speed
  - The core SQL within Bugzilla did not scale to meet our needs.  We  
reached a tipping point at around 800,000 bugs.  The main cause was  
lock contention within the DB.  The causes and solutions would fill  
an academic paper.  We rewrote search.pm and greatly increased  
performance.  Some of the changes were trivial.  A two line change  
improved our "search for bugs where person X is cced" from 45 seconds  
to 0.4 seconds.  Other changes were vast.  The overall effect was  
search times have dropped from up to 45 seconds in typical usage to  
about six seconds.

Product Management
  - With 2,400 different products, the vanilla product chooser does  
not scale for us.  We have built an entirely different version.

UI
  - We have made many minor and a few major changes to the UI.  One  
interesting one is the use of a JavaScript datatable for client-side  
sorting of buglist results.  This greatly reduces server load.
  - We have unified comments and the bug activity table.  Both appear  
inline in the show bug page.
  - We are making increasing use of the YUI library.  These allow our  
fancier functions to work on all platforms.

Company-integration
  - We have many contact points with other developer tools within  
Yahoo.  Our Bugzilla interfaces with our CVS repository, with  
Perforce and with Subversion.
  - We integrate with our employee directory to show pictures of  
employees in bugs, and to provide other information.
  - We integrate with our build and release system, and with our  
change management process.


As you can tell, we take Bugzilla seriously here.  I'm looking  
forward to contributing to the bugzilla.org effort!

  /ferg

--
fergus sullivan | developer tools team | [hidden email] | o.  
408.349.6807 | m. 408.203.FERG
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Self-Introduction: Fergus Sullivan

Frédéric Buclin
> Yahoo has been happily using Bugzilla for the last six years.  We now
> have a mandate from our top-level management to contribute to the open
> source effort.

Great news!


> 0.4 seconds.  Other changes were vast.  The overall effect was search
> times have dropped from up to 45 seconds in typical usage to about six
> seconds.

Impressive! When will this be implemented in the Bugzilla source code? ;)


> As you can tell, we take Bugzilla seriously here.  I'm looking forward
> to contributing to the bugzilla.org effort!
>
>  /ferg

Let me address you a big "Welcome!" and I hope you will enjoy working
with us. :)


LpSolit
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Self-Introduction: Fergus Sullivan

Gervase Markham
In reply to this post by Fergus Sullivan
So much to ask about!

Fergus Sullivan wrote:
> == Anything else you'd like to say ==
> Yahoo has been happily using Bugzilla for the last six years.  We now
> have a mandate from our top-level management to contribute to the open
> source effort.

Wahey!

> We are currently using a customized version of 2.22.  

Do you have an upgrade plan?

> We serve Bugzilla
> on FreeBSD 6, mysql 4.1, perl 5.6.  We use a mixture of MyISAM and
> InnoDB.  

As in, different tables use different engines? Which uses which?

> We have a dual-master database (we have our own proxy that can
> switch as needed)

So MySQL supports this?

> and a VIP handling 3 front-end and servers and search
> slaves.  We have a second colo for BCP with an equivalent architecture.  
> We have additional machines as reporting slaves.

As in, they have read-only mirrors of the DB and are used for running
reports?

> Our database currently stands at about 1.5 million bugs and grows at
> about 50k per month.  

That's the biggest known Bugzilla database, by about 3x. GNOME is at
about 470,000.

> We have 20,000 named accounts (one per employee,
> plus many internal mailing lists), of whom about 8,300 are active in any
> one month.

LDAP? RADIUS? Or a mirror of your internal employee database into the
Bugzilla DB?

> In terms of active users and new bugs opened, our load has
> increased 50% in the last year.  We have 2,400 separate products in our DB.

Red Hat has 7814 components in a single product, but I don't know of
anyone who has that many products.

> In addition to myself, we have two developers and one QA assigned full
> time on Bugzilla.  We have another two guys working full time on it at
> the moment, although they may move to other tasks in a few months time.

Presumably you have a testing installation?

> Speed
>  - The core SQL within Bugzilla did not scale to meet our needs.  We
> reached a tipping point at around 800,000 bugs.  The main cause was lock
> contention within the DB.  The causes and solutions would fill an
> academic paper.  We rewrote search.pm and greatly increased
> performance.  Some of the changes were trivial.  A two line change
> improved our "search for bugs where person X is cced" from 45 seconds to
> 0.4 seconds.  Other changes were vast.  The overall effect was search
> times have dropped from up to 45 seconds in typical usage to about six
> seconds.

I'm certainly looking forward to seeing these! Do you have any idea of
how well the changes fit with our changes between 2.22 and now?

Was it only search that was a problem?

> Product Management
>  - With 2,400 different products, the vanilla product chooser does not
> scale for us.  We have built an entirely different version.

Have you been eyeing Classifications with interest, or do they not help?
How do you do the UI for this?

> UI
>  - We have made many minor and a few major changes to the UI.  One
> interesting one is the use of a JavaScript datatable for client-side
> sorting of buglist results.  This greatly reduces server load.

I use a Greasemonkey script for this. I agree, it needs to be a core
feature.

>  - We have unified comments and the bug activity table.  Both appear
> inline in the show bug page.

Do you find this useful in practice?

Gerv
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Self-Introduction: Fergus Sullivan

Frédéric Buclin
Gervase Markham a écrit :
>> In terms of active users and new bugs opened, our load has increased
>> 50% in the last year.  We have 2,400 separate products in our DB.
>
> Red Hat has 7814 components in a single product, but I don't know of
> anyone who has that many products.


Mandriva had 32,000 products not so long ago.
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Self-Introduction: Fergus Sullivan

Gervase Markham
Frédéric Buclin wrote:
> Mandriva had 32,000 products not so long ago.

Yeah, I checked http://bugzilla.mandriva.com/query.cgi - they now only
have 9. They must have had a pretty drastic reorganisation :-)

Gerv
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Self-Introduction: Fergus Sullivan

Max Kanat-Alexander
In reply to this post by Fergus Sullivan
On Wed, 29 Aug 2007 11:31:33 -0700 Fergus Sullivan
<[hidden email]> wrote:
> == Anything else you'd like to say ==
> Yahoo has been happily using Bugzilla for the last six years.  We
> now have a mandate from our top-level management to contribute to
> the open source effort.

        Yay! :-) Welcome!

> We use a mixture of MyISAM and InnoDB.

        Yeah, we do the same thing upstream now (In 3.1).

> We have a dual-master database (we have our own  proxy that can
> switch as needed)

        Wow, dual master for MySQL, eh? I assume by "our own" means
that your wrote it yourself? What are you using to dual-master them?

>  We have a second colo for BCP

        BCP?

> Our database currently stands at about 1.5 million bugs and grows at  
> about 50k per month.

        That makes you, as far as I'm aware, the second largest
Bugzilla installation in the world, and because nobody knows anything
about the largest, you may be the largest.

> I have an open req for a visual designer to polish up the UI in  
> Bugzilla, and then in our other internal tools.

        We'd definitely love that. Have you seen any of our upstream
work on that?

> The main cause was lock contention within the DB.

        Ha! I *knew* that would happen on large installations!! I feel
vindicated. :-) It's just that it's nearly impossible to discover that
in a test environment.

        Anyhow, upstream is going wholly into transactions, which
should resolve it.

> A two line change improved our "search for bugs where person X is
> cced" from 45 seconds to 0.4 seconds.

        I seem to recall making a similar fix upstream, also. :-)

>  Other changes were vast.  The
> overall effect was search times have dropped from up to 45 seconds in
> typical usage to about six seconds.

        You'll see even more improvement with mod_perl, too. :-) That
six seconds is probably a lot of page-load and compile time.

> Product Management
>   - With 2,400 different products, the vanilla product chooser does  
> not scale for us.  We have built an entirely different version.

        Yeah, that's a well-known thing. Our idea was to AJAX it. What
did you guys do?

>   - We have made many minor and a few major changes to the UI.  One  
> interesting one is the use of a JavaScript datatable for client-side  
> sorting of buglist results.  This greatly reduces server load.

        Yeah, a definitely common request:

        https://bugzilla.mozilla.org/show_bug.cgi?id=151686

>   - We have unified comments and the bug activity table.  Both
> appear inline in the show bug page.

        Also a common request:

        https://bugzilla.mozilla.org/show_bug.cgi?id=11368

>   - We are making increasing use of the YUI library.  These allow
> our fancier functions to work on all platforms.

        We might be willing to consider that upstream, but we'd have to
see that it was a technical advantage over other solutions, and that the
licensing allowed it to be shipped with Bugzilla.

        Of course, within Yahoo it's the obvious choice, but for
upstream we have to have technical proof that it's the best choice. :-)

>   - We have many contact points with other developer tools within  
> Yahoo.  Our Bugzilla interfaces with our CVS repository, with  
> Perforce and with Subversion.

        This is probably shameless self-promotion, but you might be
interested in VCI. :-)

        http://search.cpan.org/dist/VCI/

>   - We integrate with our employee directory to show pictures of  
> employees in bugs, and to provide other information.

        That's neat. We just added a feature to show a picture for
groups, but pictures for individuals might be handy as well. Launchpad
does something like that.

> As you can tell, we take Bugzilla seriously here.  I'm looking  
> forward to contributing to the bugzilla.org effort!

        Wow, and we're looking forward to your contributions! :-)

        -Max
--
http://www.everythingsolved.com/
Competent, Friendly Bugzilla Services. And Everything Else, too.
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Self-Introduction: Fergus Sullivan

David Marshall-5
In reply to this post by Gervase Markham
Note: I work for Fergus and will compose a self-introduction soon...

One of the things I asked about when deciding to come work for Yahoo!  
was whether we would be sharing changes with Mozilla.  The idea got a  
very warm reception.  The only reason we haven't been sharing is  
because our code isn't quite up to snuff and there have been greater  
priorities.  We are finally starting to settle down into a steady  
state of code joy, so there has been more recent discussion of how  
and when to contribute.

On Aug 29, 2007, at 12:19 PM, Gervase Markham wrote:

> So much to ask about!
>
> Fergus Sullivan wrote:
>> == Anything else you'd like to say ==
>> Yahoo has been happily using Bugzilla for the last six years.  We  
>> now have a mandate from our top-level management to contribute to  
>> the open source effort.
>
> Wahey!
>
>> We are currently using a customized version of 2.22.
>
> Do you have an upgrade plan?

Yes, but it's slow going.  (I have been at Yahoo! since January, so  
my information may be slightly incorrect.)  There have been competing  
factions in the past that have pushed for either forking our Bugzilla  
from b.m.o, followed by at least one effort to defork.  We have  
wrestled with this question at length, and what we want to do is to  
make all the changes that made 2.22 -> 3.0 happen, but it's a large  
effort.

>
>> We serve Bugzilla on FreeBSD 6, mysql 4.1, perl 5.6.  We use a  
>> mixture of MyISAM and InnoDB.
>
> As in, different tables use different engines? Which uses which?

Basically, everything that doesn't need a full-text index is in  
InnoDB.  We are exploring how to use InnoDB for all the tables.

>
>> We have a dual-master database (we have our own proxy that can  
>> switch as needed)
>
> So MySQL supports this?

Er, no, it doesn't... :)  read "We have our own proxy" again.

>
>> and a VIP handling 3 front-end and servers and search slaves.  We  
>> have a second colo for BCP with an equivalent architecture.  We  
>> have additional machines as reporting slaves.
>
> As in, they have read-only mirrors of the DB and are used for  
> running reports?
>

Yes, we give RO-access to  people who want to run their own reports.  
It beats having to tell someone why we aren't going to add their  
spiffy feature.

>> Our database currently stands at about 1.5 million bugs and grows  
>> at about 50k per month.
>
> That's the biggest known Bugzilla database, by about 3x. GNOME is  
> at about 470,000.
>
>> We have 20,000 named accounts (one per employee, plus many  
>> internal mailing lists), of whom about 8,300 are active in any one  
>> month.
>
> LDAP? RADIUS? Or a mirror of your internal employee database into  
> the Bugzilla DB?

We have several databases from which we extract information for our  
own database.

>
>> In terms of active users and new bugs opened, our load has  
>> increased 50% in the last year.  We have 2,400 separate products  
>> in our DB.
>
> Red Hat has 7814 components in a single product, but I don't know  
> of anyone who has that many products.
>
>> In addition to myself, we have two developers and one QA assigned  
>> full time on Bugzilla.  We have another two guys working full time  
>> on it at the moment, although they may move to other tasks in a  
>> few months time.
>
> Presumably you have a testing installation?

sort of...

>
>> Speed
>>  - The core SQL within Bugzilla did not scale to meet our needs.  
>> We reached a tipping point at around 800,000 bugs.  The main cause  
>> was lock contention within the DB.  The causes and solutions would  
>> fill an academic paper.  We rewrote search.pm and greatly  
>> increased performance.  Some of the changes were trivial.  A two  
>> line change improved our "search for bugs where person X is cced"  
>> from 45 seconds to 0.4 seconds.  Other changes were vast.  The  
>> overall effect was search times have dropped from up to 45 seconds  
>> in typical usage to about six seconds.
>
> I'm certainly looking forward to seeing these! Do you have any idea  
> of how well the changes fit with our changes between 2.22 and now?
>

I think they will fit in reasonably well, although it remains to be  
seen whether anyone who doesn't have a million rows in bugs would  
notice any particular improvement.

> Was it only search that was a problem?

I have found a couple of places (unknown whether they're Mozilla or  
Yahoo! code) that would gather information about products/components/
etc one row at a time with separate queries.  For most installations,  
it's probably not a noticeable performance hit to do this.  For us,  
it was a killer, and I have altered those to hit the database once.

>
>> Product Management
>>  - With 2,400 different products, the vanilla product chooser does  
>> not scale for us.  We have built an entirely different version.
>
> Have you been eyeing Classifications with interest, or do they not  
> help? How do you do the UI for this?

We use classifications, but I don't think it's well integrated in our  
product-choosing workflow.  (Although I'm not a typical Bugzilla  
user, of course.)

>
>> UI
>>  - We have made many minor and a few major changes to the UI.  One  
>> interesting one is the use of a JavaScript datatable for client-
>> side sorting of buglist results.  This greatly reduces server load.
>
> I use a Greasemonkey script for this. I agree, it needs to be a  
> core feature.
>
>>  - We have unified comments and the bug activity table.  Both  
>> appear inline in the show bug page.
>
> Do you find this useful in practice?

This is still new, so we don't have a lot of feedback about it yet.  
I think it will be very useful, especially as we allow users to  
collapse this detail.

>
> Gerv
> -
> To view or change your list settings, click here:
> <http://bugzilla.org/cgi-bin/mj_wwwusr?user=dmarshal@...>

-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Self-Introduction: Fergus Sullivan

Olav Vitters
In reply to this post by Fergus Sullivan
On Wed, Aug 29, 2007 at 11:31:33AM -0700, Fergus Sullivan wrote:
> == Full name ==
> Fergus Sullivan

Welcome!

> Speed
>  - The core SQL within Bugzilla did not scale to meet our needs.  We
> reached a tipping point at around 800,000 bugs.  The main cause was lock
> contention within the DB.  The causes and solutions would fill an academic
> paper.  We rewrote search.pm and greatly increased performance.  Some of
> the changes were trivial.  A two line change improved our "search for bugs
> where person X is cced" from 45 seconds to 0.4 seconds.  Other changes were
> vast.  The overall effect was search times have dropped from up to 45
> seconds in typical usage to about six seconds.

I'm very interested in all speed improvements.

> Product Management
>  - With 2,400 different products, the vanilla product chooser does not
> scale for us.  We have built an entirely different version.
>
> UI
>  - We have made many minor and a few major changes to the UI.  One
> interesting one is the use of a JavaScript datatable for client-side
> sorting of buglist results.  This greatly reduces server load.

Please push upstream!

>  - We have unified comments and the bug activity table.  Both appear inline
> in the show bug page.

Same, very welcome upstream.

>  - We are making increasing use of the YUI library.  These allow our
> fancier functions to work on all platforms.

No idea what that is.

--
Regards,
Olav
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Self-Introduction: Fergus Sullivan

Kristis Makris-2
In reply to this post by Max Kanat-Alexander
On Wed, 2007-08-29 at 12:48 -0700, Max Kanat-Alexander wrote:
> >   - We have many contact points with other developer tools within  
> > Yahoo.  Our Bugzilla interfaces with our CVS repository, with  
> > Perforce and with Subversion.
>
>         This is probably shameless self-promotion, but you might be
> interested in VCI. :-)
>
>         http://search.cpan.org/dist/VCI/ 

Why didn't I know about VCI for so long ?? Mmm... yum! Thanks Max.

Could you elaborate on how Bugzilla "interfaces" with the SCM
repositories ? Do your repositories also interface with Bugzilla ? How
do you handle change management in such a big organization ? Could
Scmbug be of help ?

http://freshmeat.net/projects/scmbug/


signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Self-Introduction: Fergus Sullivan

Max Kanat-Alexander
On Wed, 29 Aug 2007 13:44:30 -0700 Kristis Makris
<[hidden email]> wrote:
> Why didn't I know about VCI for so long ?? Mmm... yum! Thanks Max.

        Yeah, you're welcome! :-) I figured you'd be a big consumer of
it. :-)

        You didn't know about it for so long because I just wrote
it. :-)

        -Max
--
http://www.everythingsolved.com/
Competent, Friendly Bugzilla Services. And Everything Else, too.
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Self-Introduction: Fergus Sullivan

Pankaj Garg
In reply to this post by Olav Vitters
I second the idea of looking into YUI (Yahoo User Interface Library)  
as the means of providing a more Web 2.0 and AJAX'y interface to  
Bugzilla. It comes with a BSD license, and so should be compatible (I  
think) with the Mozilla license.

I've found the abstractions for XML/HTTP calls and various UI  
elements (Panels, Dialogs, DataTable, etc.) very well-thought out and  
simple to use.

The same version that is distributed as Open Source is also used in  
all Yahoo properties so it is very stable and robust.

We had updated Bug 325501 (https://bugzilla.mozilla.org/show_bug.cgi?
id=325501) with this suggestion a few weeks ago.

Pankaj

P.S. I'll be posting a self-introduction soon.

On Aug 29, 2007, at 12:57 PM, Olav Vitters wrote:

>>  - We are making increasing use of the YUI library.  These allow our
>> fancier functions to work on all platforms.
>
> No idea what that is.
>
> --
> Regards,
> Olav
> -

  --
  Pankaj K Garg                         [hidden email]
  1684 Nightingale Avenue     408-373-4027
  Suite 201                                  408-733-2737(fax)
  Sunnyvale, CA 94087            http://www.zeesource.net
                Your Open Source Partner


-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Self-Introduction: Fergus Sullivan

Myk Melez
Pankaj K Garg wrote:
I second the idea of looking into YUI (Yahoo User Interface Library) as the means of providing a more Web 2.0 and AJAX'y interface to Bugzilla.
Thirded.  I used YUI for a recent project, and although I had some issues with it, I was impressed overall with the architecture and design.

It comes with a BSD license, and so should be compatible (I think) with the Mozilla license.
Gerv'll know more (and correct my presumption), but my understanding is that YUI's BSD license is indeed compatible with (presumably because more liberal than) the MPL license that Bugzilla uses.

-myk

Reply | Threaded
Open this post in threaded view
|

Re: Self-Introduction: Fergus Sullivan

Fergus Sullivan
In reply to this post by Fergus Sullivan
Many thanks to each of you for the very kind welcome.  We're looking  
forward to working with you guys.

There are a couple of open questions that we haven't yet covered, so  
there's more info below.

   /ferg


== How do we have dual-master sql proxies? ==
We have our own secret sauce for this.  It's in use across Yahoo and  
is central to our BCP/HA planning.  It's proprietary, and not  
currently on our list to release to open source.  I have, however,  
prodded its owners about that.  Essentially it allows us to switch  
from Master A to Master B the moment Master A fails a health check.  
We put the two masters in different server farms so as to reduce  
their vulnerability.

== What is BCP/HA? ==
Business Continuity Planning is a means to ensure that if our primary  
colo goes down we can switch to another as needed.  It assumes a  
certain amount of downtime as dns records are switched and boxes are  
spun up.  This could be anywhere from 30 minutes to a couple of hours.

High Availability means that the service should never go down, and  
automatically switch to a backup server farm as needed.

In both cases, the systems exist to allow for natural disasters,  
power outages, war, insurrection, comets or acts of god.  Read your  
insurance policy's exclusion policy for more...

In both cases, data in each location needs to be up to date.

== How many Bugzilla servers does Yahoo have ==
A couple of years ago we had 1.  Then we bumped up to three.  
Currently we're at one db master and three slaves which share the  
http and search load.  These sit behind a load-balancer.

Our next hardware revision will bump us to one master and four slaves  
in each of two server farms.  One farm is on the US west coast, the  
other on the east.  The primary goal of this is HA, not server load.

We are evaluating the need for mirrors local to our users in Asia and  
Europe.

We have one read-only mirror where we let users run their own raw sql  
queries.

And we have "The Bug Hole", which is where your query is sent if it  
requires a full table scan.  This type of query was blocking other  
users and hogging our search resources.

Incoming mail has always been a part of our instance and shares one  
of the other servers.

== What's this YUI thing, and why will Ferg be harping on about it  
for the next six months? ==
YUI is the Yahoo User Interface libraries.  These were released under  
the FreeBSD license a year and a half ago.  They are widely used,  
both at Yahoo and outside.  Airline sites like southwest.com and  
northwest.com make heavy use of them.

The YUI libraries include widgets like datatables, grid management,  
dialog management, CSS levelling, trees, drag and drop etc.  They are  
fully functional cross-platform and cross-browser.

I joke with the YUI guys that their work makes bad engineers look  
good.  My point is that a little time with the libraries can make the  
trickiest tasks easy.  For example, we decided to pipe the content of  
buglist.cgi into a datatable so that we could sort it client-side.  
It took one of our guys two days to get it working.

The YUI team sit across the aisle from us.  When we need a new  
feature, we wander over and ask.  We dreamed up the idea of how cool  
it would be to be able to edit a bug list in place, like an excel  
sheet.  We walked over to the YUI guys and asked them if it could be  
done.  And lo, inline editing in the datatable was born.

Here are some of my favorite examples, and how we could use them in  
Bugzilla.

Datatable - inline editing of the buglist.  Browse around to see  
examples of client-side sorting and pagination goodies.  Don't  
neglect the client-side pagination with browser history.
http://developer.yahoo.com/yui/examples/datatable/dt_cellediting.html

TreeView - Dependencies anyone?
http://developer.yahoo.com/yui/examples/treeview/default_tree.html

Calendar and Autocomplete - We already use these in our Bugzilla code  
Yahoo
http://developer.yahoo.com/yui/examples/calendar/quickstart.html

Grids - For clean layout
http://developer.yahoo.com/yui/examples/calendar/quickstart.html

There are many, many other uses besides.  For us, use of the YUI  
libraries has been the best thing ever.  The only catch bugzilla.org  
needs to consider is that if we all go with YUI we'll never want to  
turn back.

== Test harnesses ==
As dwm mentioned, we have a suite of tests that use an internal Yahoo  
tool.  We try hard to keep our tests up to date with a 100% pass rate.

We have a number of test machines.  These are mainly for load testing  
in the form of log replays.

== The Yahoo Product Chooser ==
With 2,500 products we needed an optimized product chooser.  To be  
honest, I'm not at all happy with its current form, so we may well  
change it in the near future.  We're thinking of something analogous  
to an address book module in a mail client.  Essentially we want to  
make it easier to select one or many products, and steer people away  
from searching 'all'.

== When can Yahoo start offering its patches for review ==
It depends.  We have an internal legal process to approve  
engagement.  That'll take a few weeks to complete.  We should be able  
to offer minor patches soon after.

The larger patches will be tricky, as they work for us right now with  
the rest of our code base.  Remember our code is based on 2.22 and  
chunks of it are modified.  That said, our Search.pm is a thing of  
beauty...


-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Self-Introduction: Fergus Sullivan

Dave Miller
In reply to this post by David Marshall-5
David Marshall wrote on 8/29/07 3:49 PM:
> On Aug 29, 2007, at 12:19 PM, Gervase Markham wrote:
>>> We have a dual-master database (we have our own proxy that can switch
>>> as needed)
>>
>> So MySQL supports this?
>
> Er, no, it doesn't... :)  read "We have our own proxy" again.

Actually, it does.  Sort of.  We've been looking into it for Mozilla.
It's by no means easy to implement, and there's a few nasty
give-and-take hacks you have to use (like each autoincrement key gets
ranges assigned in advance per-server, and the masters have to
periodically coordinate with each other on reassigning new ranges as
they run out - which means the bug IDs won't necessarily line up with
the order the bugs are filed, and so forth).  It's difficult enough to
pull off that there's a few commercial products for rather huge $$$ that
are supposed to make it easy. ;)

--
Dave Miller                                   http://www.justdave.net/
System Administrator, Mozilla Corporation      http://www.mozilla.com/
Project Leader, Bugzilla Bug Tracking System  http://www.bugzilla.org/
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Self-Introduction: Fergus Sullivan

Gervase Markham
In reply to this post by Fergus Sullivan
Fergus Sullivan wrote:
> == How do we have dual-master sql proxies? ==
> We have our own secret sauce for this.  It's in use across Yahoo and is
> central to our BCP/HA planning.  It's proprietary, and not currently on
> our list to release to open source.

I think the interest expressed here was more "hey, cool" rather than "we
need this for Bugzilla". Of course, a free software release would be
great too.

> Calendar and Autocomplete - We already use these in our Bugzilla code Yahoo
> http://developer.yahoo.com/yui/examples/calendar/quickstart.html
>
> Grids - For clean layout
> http://developer.yahoo.com/yui/examples/calendar/quickstart.html

Wrong URL?

> == When can Yahoo start offering its patches for review ==
> It depends.  We have an internal legal process to approve engagement.  
> That'll take a few weeks to complete.  We should be able to offer minor
> patches soon after.
>
> The larger patches will be tricky, as they work for us right now with
> the rest of our code base.  Remember our code is based on 2.22 and
> chunks of it are modified.  That said, our Search.pm is a thing of
> beauty...

Code thrown over the wall is still better than no code at all. Perhaps
you might consider posting a giant diff of your code compared to 2.22,
as well as pursuing a parallel strategy of submitting small, clean
patches against the trunk?

Gerv

-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Self-Introduction: Fergus Sullivan

Gervase Markham
In reply to this post by Pankaj Garg
Pankaj K Garg wrote:
> I second the idea of looking into YUI (Yahoo User Interface Library) as
> the means of providing a more Web 2.0 and AJAX'y interface to Bugzilla.
> It comes with a BSD license, and so should be compatible (I think) with
> the Mozilla license.

Recently, several frameworks of this sort have been developed. They all
seem very cool and have capabilities far beyond what we want or need.
Unless other Bugzilla developers have a strong preference for another
one, if the Yahoo! team have YUI-based improvements to Bugzilla, that
swings it for me.

The only thing is that we shouldn't end up using two!

BTW, there are no problems from a licensing point of view.

Gerv
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Self-Introduction: Fergus Sullivan

Gregary Hendricks
In reply to this post by Fergus Sullivan
On Thu, 2007-08-30 at 10:26 +0100, Gervase Markham wrote:
> Recently, several frameworks of this sort have been developed. They all
> seem very cool and have capabilities far beyond what we want or need.
> Unless other Bugzilla developers have a strong preference for another
> one, if the Yahoo! team have YUI-based improvements to Bugzilla, that
> swings it for me.
>

I second this. I have been using Dojo for Testopia but I am fed up with
it. I am looking for a new tool that can do the basic ajax without the
overhead and YUI seems to have alot better command of the situation.

> The only thing is that we shouldn't end up using two!
>

I am ready and willing to switch.

Greg

-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Self-Introduction: Fergus Sullivan

Colin Ogilvie
In reply to this post by Gervase Markham
Gervase Markham wrote:
> Recently, several frameworks of this sort have been developed. They
> all seem very cool and have capabilities far beyond what we want or
> need. Unless other Bugzilla developers have a strong preference for
> another one, if the Yahoo! team have YUI-based improvements to
> Bugzilla, that swings it for me.
>
> The only thing is that we shouldn't end up using two!
>
> BTW, there are no problems from a licensing point of view.

Shouldn't we look to evaluate different libraries / frameworks based on
what we actually want to do with AJAX/DHTML/JavaScript rather than just
picking YUI because the Yahoo! team have used it?

There may be other frameworks that provide the functionality that we
need / want to use and it seems a bit naive to just use the first one
suggested.

Colin
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Self-Introduction: Fergus Sullivan

byron-2
> Shouldn't we look to evaluate different libraries / frameworks based on
> what we actually want to do with AJAX/DHTML/JavaScript rather than just
> picking YUI because the Yahoo! team have used it?

i don't think so -- we need willing developers more than we need the
correct javascript library.

yui is very good anyhow :)


-b

begin-base64 644 signature.gif
R0lGODlhbQAHAIAAAABPo////ywAAAAAbQAHAAACfAxuGAnch+Bibkn7FL1p
XgVl4Ig1jjlZRoqybgun2Cur5uOunq7u/Ipq7WIyIc7XG9JquEgumPzdlhTf
h0O83kDJaXEm8mRHwXKJy5sac7qYOpT+gtv0n+0ujQOfdqh16caWt0foBViH
N1PRMXimiLUGt3ElVimlgbllWAAAOw==
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Self-Introduction: Fergus Sullivan

Dave Miller
In reply to this post by Gregary Hendricks
Gregary Hendricks wrote on 8/30/07 6:09 PM:

> On Thu, 2007-08-30 at 10:26 +0100, Gervase Markham wrote:
>> Recently, several frameworks of this sort have been developed. They all
>> seem very cool and have capabilities far beyond what we want or need.
>> Unless other Bugzilla developers have a strong preference for another
>> one, if the Yahoo! team have YUI-based improvements to Bugzilla, that
>> swings it for me.
>
> I second this. I have been using Dojo for Testopia but I am fed up with
> it. I am looking for a new tool that can do the basic ajax without the
> overhead and YUI seems to have alot better command of the situation.

After seeing it mentioned here, I grabbed YUI and I've been using it on
an internal project at Mozilla for the last couple days, and I'm
suitably impressed as well.  Seems to be well thought out at first glance.

--
Dave Miller                                   http://www.justdave.net/
System Administrator, Mozilla Corporation      http://www.mozilla.com/
Project Leader, Bugzilla Bug Tracking System  http://www.bugzilla.org/
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
12