Proposed upcoming mirror changes

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Proposed upcoming mirror changes

Justin Fitzhugh
We'd like to announce some proposed changes to how we mirror and
distribute our various products and builds.  Our current mirror size is
approaching 500gb - much too large and costly to ask our mirrors to
support.  Thus, we have come up with a new and improved mirror strategy
to help alleviate the space and bandwidth issues associated with
supporting such a large mirror site while retaining our ability to
distribute our software in a quick and efficient manner.

We will take comments until 6/8/06 with implementation scheduled for
6/9/06 assuming no major objections.  Please take the time to read
through the changes and let us know if you have any concerns with the
proposed plan.  Note - there should be no end-user effecting changes.
All of these modifications will be transparent to the user.

---
*  There will be 2 rsync modules/data repositories:
          o Releases - this will be all releases which have been
released with the last year. This includes rc's, beta's and alpha's. All
bouncer files will be contained within this module.
          o FTP - this will contain all of the releases module plus all
old releases (beta's, alpha's and rc's) and any nightly builds.
ftp.mozilla.org and archive.mozilla.org will go to this data set.

    * Anyone who would like to mirror Releases will *have* to mirror
from releases-rsync.mozilla.org (TDS and OSL - as most should be doing
now). We haven't enforced this and there are a few people who are still
mirroring off surf (i.e. stage). This has a few bad ramifications around
virus scanning, bandwidth, etc - we'll just start enforcing that people
use the correct rsync mirrors.

    * Master mirrors (OSL and TDS) and standard mirrors won't rsync the
FTP module, instead only the releases module. This will mean that all
ftp.mozilla.org traffic will come directly to Mozilla. Our main mirrors
do not want to keep this much data online and we can easily handle
traffic. Any high-traffic builds/releases should be in the releases module.
---

We are in discussions with our major mirror sites to make sure the
changes are implemented with minimal downstream impact.  Again, please
let me know if you have any questions/concerns/issues with the new setup.

--
Justin Fitzhugh
Director, IT - Mozilla Corporation
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Proposed upcoming mirror changes

Robert Kaiser
Justin Fitzhugh schrieb:
>     * Master mirrors (OSL and TDS) and standard mirrors won't rsync the
> FTP module, instead only the releases module. This will mean that all
> ftp.mozilla.org traffic will come directly to Mozilla. Our main mirrors
> do not want to keep this much data online and we can easily handle
> traffic. Any high-traffic builds/releases should be in the releases module.

Does this mean we should make releases download pageslink to
releases.mozilla.org instead of ftp.mozilla.org (same for any bouncer
URLs that still might point to ftp.m.o)?
At least the SeaMonkey project links to ftp.m.o on release download
pages, but as I'm maintaining those pages, I can s/ftp./releases./ there
if it's the way to go... (Of course, the bouncer URLs have to be changed
by someone who has access to the bouncer DB)

Robert Kaiser
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Proposed upcoming mirror changes

Ruediger Lahl
In reply to this post by Justin Fitzhugh
Justin Fitzhugh wrote:

> We are in discussions with our major mirror sites to make sure the
> changes are implemented with minimal downstream impact.  Again, please
> let me know if you have any questions/concerns/issues with the new setup.

For users like me, who work with Tinderbox-builds, it would be nice to
have a text-file for each build, which only contains the build-ID
from the current build on the server.
Checking the file-date/time is not very useful, because all the
different timezones and the nearly two hours cycle of new SM-builds.

No one can say, which of this builds is actual, if you find them on
different servers:
seamonkey-1.5a.en-US.win32.zip             11223kb 07.06.2006  12:56:00
seamonkey-1.5a.en-US.win32.zip             11206kb 07.06.2006  10:58:00

A .build-file makes that easy.

This can look like that, on a ftp-server:

seamonkey-1.5a.en-US.win32.zip             11206kb 07.06.2006  10:58:00
seamonkey-1.5a.en-US.win32.build               1kb 07.06.2006  10:58:00
seamonkey-1.5a.en-US.win32.installer.exe     253kb 07.06.2006  10:58:00
seamonkey-1.5a.en-US.win32.installer.build     1kb 07.06.2006  10:58:00
...

The *.build files are the text-files that containing the build-ID.
Easily this can be a renamed build.dtd from the toolkit.jar (in TB- and
SM-Trunk. For FF this file don't exists).
--
best regards
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Proposed upcoming mirror changes

Robert Kaiser
Ruediger Lahl schrieb:
> Justin Fitzhugh wrote:
>
>> We are in discussions with our major mirror sites to make sure the
>> changes are implemented with minimal downstream impact.  Again, please
>> let me know if you have any questions/concerns/issues with the new setup.
>
> For users like me, who work with Tinderbox-builds, it would be nice to
> have a text-file for each build, which only contains the build-ID
> from the current build on the server.

This thread is all about not needind this any more, as there will be
only one ftp.mozilla.org server which will carry the tinderbox builds in
the future, the mirrors won't have those and the nightlies any more,
only the releases.

Robert Kaiser
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Proposed upcoming mirror changes

Mike Connor-4
In reply to this post by Ruediger Lahl
Ruediger Lahl wrote:

> Justin Fitzhugh wrote:
>
>  
>> We are in discussions with our major mirror sites to make sure the
>> changes are implemented with minimal downstream impact.  Again, please
>> let me know if you have any questions/concerns/issues with the new setup.
>>    
>
> For users like me, who work with Tinderbox-builds, it would be nice to
> have a text-file for each build, which only contains the build-ID
> from the current build on the server.
> Checking the file-date/time is not very useful, because all the
> different timezones and the nearly two hours cycle of new SM-builds.
>
> No one can say, which of this builds is actual, if you find them on
> different servers:
> seamonkey-1.5a.en-US.win32.zip             11223kb 07.06.2006  12:56:00
> seamonkey-1.5a.en-US.win32.zip             11206kb 07.06.2006  10:58:00
>
> A .build-file makes that easy.
>
> This can look like that, on a ftp-server:
>
> seamonkey-1.5a.en-US.win32.zip             11206kb 07.06.2006  10:58:00
> seamonkey-1.5a.en-US.win32.build               1kb 07.06.2006  10:58:00
> seamonkey-1.5a.en-US.win32.installer.exe     253kb 07.06.2006  10:58:00
> seamonkey-1.5a.en-US.win32.installer.build     1kb 07.06.2006  10:58:00
> ...
>
> The *.build files are the text-files that containing the build-ID.
> Easily this can be a renamed build.dtd from the toolkit.jar (in TB- and
> SM-Trunk. For FF this file don't exists).
>  
All nightlies/tinderbox builds will live on ftp.mozilla.org, which would
stop being the mirror round-robin, so there should be one set of dates.

-- MIke
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Proposed upcoming mirror changes

Ruediger Lahl
In reply to this post by Robert Kaiser
Robert Kaiser wrote:

> This thread is all about not needind this any more,

*Arrgh* I think I should read with more attention.

> as there will be only one ftp.mozilla.org server which will carry the
> tinderbox builds in the future, the mirrors won't have those and the
> nightlies any more, only the releases.

Okay, thank you (and Mike) for clarify.
--
best regards
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Proposed upcoming mirror changes

Justin Fitzhugh
In reply to this post by Robert Kaiser
Hey Robert - see inline...

Robert Kaiser wrote:

> Does this mean we should make releases download pageslink to
> releases.mozilla.org instead of ftp.mozilla.org (same for any bouncer
> URLs that still might point to ftp.m.o)?

Nope, bouncer links (you know they are using bouncer if they have
download.mozilla.org in the URL) redirect to various mirrors and are not
effected by this change.  Any links to files within your releases/
directory *should* be pointing at releases.mozilla.org - anything else
to ftp.mozilla.org.

That said, if you do have files in bouncer (which seamonkey does), we'd
much prefer you use bouncer URLs as opposed to linking directly to
releases/ftp.

Thanks,
-Justin
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Proposed upcoming mirror changes

Robert Kaiser
Justin Fitzhugh schrieb:
> Nope, bouncer links (you know they are using bouncer if they have
> download.mozilla.org in the URL) redirect to various mirrors and are not
> effected by this change.  Any links to files within your releases/
> directory *should* be pointing at releases.mozilla.org - anything else
> to ftp.mozilla.org.
>
> That said, if you do have files in bouncer (which seamonkey does), we'd
> much prefer you use bouncer URLs as opposed to linking directly to
> releases/ftp.

We only have our main files (primary en-US builds of tier-1 platforms)
in bouncer right now, and we are pointing to bouncer for those, of
course. All other files currently direct to ftp.m.o on our release
pages, so I'll probably should go through them and redirect them to
releases.m.o

Thanks for making this clear.

Robert Kaiser
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Proposed upcoming mirror changes

Matthias Wallner
In reply to this post by Justin Fitzhugh
Justin Fitzhugh schrieb:

> We are in discussions with our major mirror sites to make sure the
> changes are implemented with minimal downstream impact.  Again, please
> let me know if you have any questions/concerns/issues with the new setup.

This is probably a non-issue since those files are outdated, but
http://releases.mozilla.org/pub/mozilla.org/mozilla/releases/
only has 1.7.x and higher
the README points to
http://archive.mozilla.org/pub/mozilla/releases/
but they only have Betas there

the real (old) versions are only available at
http://ftp.mozilla.org/pub/mozilla.org/mozilla/releases/

so please either change the README to point to ftp.m.o, or put the stuff
from ftp to archive (where the old stuff belongs IMO)

Matthias
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Proposed upcoming mirror changes

nrthomas
In reply to this post by Justin Fitzhugh
Will there be any change to the automatic moving of older nightly
builds from ftp.m.o. to archive.m.o ?

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Proposed upcoming mirror changes

Robert Kaiser
[hidden email] schrieb:
> Will there be any change to the automatic moving of older nightly
> builds from ftp.m.o. to archive.m.o ?

 From what I heard, the plan is to make archive == ftp, so that move
won't make a lot of sense after the work mentioned in this thread has
been done, AFAIK.

Robert Kaiser
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning