Reopening the trunk: planning

classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Reopening the trunk: planning

Boris Zbarsky
I think we're getting close to reopening the trunk.  The leak regression is
under control.  We're still gated on the startup regression, but hopefully we'll
get somewhere with it in the next day or so.

What I'd like to avoid is mad landing-madness when the trunk opens.  If nothing
else because it could put us right back into the "perf regression for causes
unknown" boat.  Especially because we want the thread manager changes to land.

So what I would like to propose is that when we're happy with the state of the
trunk performance tests, we keep it closed and darin lands his thread manager
patch.  We then wait for tinderboxen to cycle.  I'd like at least two cycles
here on all our perf tinderboxen; we're definitely expecting this patch to have
effect on perf across the board; we're just not sure in which directions in all
cases.  ;)  If something dire happens to the perf numbers we deal with it,
hopefully.  After we're happy with the state of post-darin-landing perf, we
reopen the trunk to general landings, but avoid landing all at once.  How, I'm
not sure.  Ideas would be nice here.

Thoughts?

-Boris
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Reopening the trunk: planning

Peter van der Woude
1. It would help if either GAIUS (stop building non-places for a day)
or PACIFICA would just do 1 type of build so that we have a new build
every 30-45 minutes.

2. Close the tree again at the first sign of orange or red (too bad if
it's a mid-checkin)

3. If it's going to be as bad as you fear, make checkins require
sheriffs approval, which should include a timeframe when that person is
allowed to check in.

I know this is a pretty time consuming, but it's better than spending
hours/days finding a regressor.

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Reopening the trunk: planning

Gervase Markham
In reply to this post by Boris Zbarsky
Boris Zbarsky wrote:
> After we're
> happy with the state of post-darin-landing perf, we reopen the trunk to
> general landings, but avoid landing all at once.  How, I'm not sure.
> Ideas would be nice here.

chofmann used to have a "branch landing tool", which was basically a
queue for big landings. Can we have a wiki page fulfilling the same
purpose? Identify the non-trivial landings, add them to the page, and
people can move their landing into a slot, with the slots being (say) 3
hours apart over the next week.

Gerv
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Reopening the trunk: planning

Robert Kaiser
In reply to this post by Boris Zbarsky
Boris Zbarsky schrieb:
> So what I would like to propose is that when we're happy with the state
> of the trunk performance tests, we keep it closed and darin lands his
> thread manager patch.  We then wait for tinderboxen to cycle.

Wasn't a "carpool" something like that in the old days? ;-)

Like Gerv said, reviving the idea behind the "branch landing tool", i.e.
having a list of major things to land, with bugs, explanation and state,
would probably be good...

Robert Kaiser
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Reopening the trunk: planning

Daniel Cater
In reply to this post by Boris Zbarsky
Boris Zbarsky wrote:

> I think we're getting close to reopening the trunk.  The leak regression
> is under control.  We're still gated on the startup regression, but
> hopefully we'll get somewhere with it in the next day or so.
>
> What I'd like to avoid is mad landing-madness when the trunk opens.  If
> nothing else because it could put us right back into the "perf
> regression for causes unknown" boat.  Especially because we want the
> thread manager changes to land.
>
> So what I would like to propose is that when we're happy with the state
> of the trunk performance tests, we keep it closed and darin lands his
> thread manager patch.  We then wait for tinderboxen to cycle.  I'd like
> at least two cycles here on all our perf tinderboxen; we're definitely
> expecting this patch to have effect on perf across the board; we're just
> not sure in which directions in all cases.  ;)  If something dire
> happens to the perf numbers we deal with it, hopefully.  After we're
> happy with the state of post-darin-landing perf, we reopen the trunk to
> general landings, but avoid landing all at once.  How, I'm not sure.  
> Ideas would be nice here.
>
> Thoughts?
>
> -Boris

Could the trunk sync up with the branch first? In a rush for a2 and the closure
of the trunk a few checkins ignored the "bake on-trunk first" rule. Before any
more diverging takes place, would it be a good idea to get those checkins on
trunk, and then close it for the threads patch? After that's been sorted then it
can open again for normal business, but hopefully not to a stampede of patches
all at once ;-)

I'm not particularly bothered, but it seems to make sense to me.

Dan.
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Reopening the trunk: planning

Boris Zbarsky
In reply to this post by Gervase Markham
Gervase Markham wrote:
> chofmann used to have a "branch landing tool", which was basically a
> queue for big landings. Can we have a wiki page fulfilling the same
> purpose? Identify the non-trivial landings

Actually, even "trivial" patches can often affect the perf tests, so for perf
test purposes we really need to deal with most checkins, not the non-trivial ones.

-Boris
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Reopening the trunk: planning

Boris Zbarsky
In reply to this post by Daniel Cater
Daniel Cater wrote:
> Could the trunk sync up with the branch first? In a rush for a2 and the
> closure of the trunk a few checkins ignored the "bake on-trunk first"
> rule. Before any more diverging takes place, would it be a good idea to
> get those checkins on trunk, and then close it for the threads patch?

Quite frankly, I feel that that's the problem of the people who ignored the
"bake on trunk first" rule.  Darin's patch is about 400KB after being gzipped;
making him merge it yet again to more changes is something that, imo, should be
avoided.

I could perhaps be persuaded otherwise given the exact list of checkins that
have landed on branch and still need to land on trunk.  Without that list,
making a call on this is hard.  If the list is very short and the patches are
very safe (such that we could land them, cycle quickly, be guaranteed to not
regress any perf metrics, and then land darin), it might be worth it.  Otherwise
these checkins should just get metered along with everything else.

-Boris
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Reopening the trunk: planning

Darin Fisher-2
Some things that people should anticipate with this branch landing:

- BeOS bustage
- OS/2 bustage (mkaply is working on a fix)
- bustage in other widget ports (e.g., xlib)
- Seamonkey is not as well tested as Firefox and Thunderbird -- and I
only tested it on Linux w/ the GTK1 widget toolkit.
- Camino not tested by me, but I believe that Mark tested it as he
developed the Cocoa port of the widget changes

There may be some things that I missed, so please bear with me as I
try to land this branch.  It unfortunately impacts all of our ports
because it affects widget code.  I tried to give advance notice to
port maintainers (see my post to m.d.platform many moons ago), but I
haven't had much response from port maintainers (other than from
mkaply).  I suspect that people simply missed my note, and I probably
should have made more noise about this, but I've been heads down just
trying to get this working solid for Firefox and Thunderbird.  I hope
everyone will be understanding as this is likely to be a bit bumpy.

For reference, here's the bug for this:
https://bugzilla.mozilla.org/show_bug.cgi?id=326273

More info here:
http://wiki.mozilla.org/XPCOM:nsIThreadManager

I plan on posting a follow-up to m.d.platform with more information
about the API impact of these changes.

Regards,
-Darin


On 5/10/06, Boris Zbarsky <[hidden email]> wrote:

> Daniel Cater wrote:
> > Could the trunk sync up with the branch first? In a rush for a2 and the
> > closure of the trunk a few checkins ignored the "bake on-trunk first"
> > rule. Before any more diverging takes place, would it be a good idea to
> > get those checkins on trunk, and then close it for the threads patch?
>
> Quite frankly, I feel that that's the problem of the people who ignored the
> "bake on trunk first" rule.  Darin's patch is about 400KB after being gzipped;
> making him merge it yet again to more changes is something that, imo, should be
> avoided.
>
> I could perhaps be persuaded otherwise given the exact list of checkins that
> have landed on branch and still need to land on trunk.  Without that list,
> making a call on this is hard.  If the list is very short and the patches are
> very safe (such that we could land them, cycle quickly, be guaranteed to not
> regress any perf metrics, and then land darin), it might be worth it.  Otherwise
> these checkins should just get metered along with everything else.
>
> -Boris
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Reopening the trunk: planning

beltzner
In reply to this post by Gervase Markham
On 5/10/06, Gervase Markham <[hidden email]> wrote:
> queue for big landings. Can we have a wiki page fulfilling the same
> purpose? Identify the non-trivial landings, add them to the page, and

Ask and ye shall recieve:

http://wiki.mozilla.org/Runway
http://wiki.mozilla.org/Runway/Queue

I just threw that together, but it seems to serve the purpose. Please
feel free to discuss in #developers or on this list. Eventually we
might want to get some JS thrown into the templates to allow us to
dynamically update status based on tinderbox readings or something.

cheers,
mike

--
/ mike beltzner / user experience lead / mozilla corporation /
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Reopening the trunk: planning

Mike Connor-4
Mike Beltzner wrote:

> On 5/10/06, Gervase Markham <[hidden email]> wrote:
>> queue for big landings. Can we have a wiki page fulfilling the same
>> purpose? Identify the non-trivial landings, add them to the page, and
>
> Ask and ye shall recieve:
>
> http://wiki.mozilla.org/Runway
> http://wiki.mozilla.org/Runway/Queue
>
> I just threw that together, but it seems to serve the purpose. Please
> feel free to discuss in #developers or on this list. Eventually we
> might want to get some JS thrown into the templates to allow us to
> dynamically update status based on tinderbox readings or something.
So, this works, assuming the sheriff is committed to making time to plan
this stuff.  I think its important to get as much as possible on this
runway, but I think it'll bottleneck until we figure out a way out of
the multitinderbox hole, since full cycles take much longer than
necessary.  More on this later, but I have some ideas around what we
really need ASAP for tinderbox coverage.

-- MIke
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Reopening the trunk: planning

Gervase Markham
In reply to this post by Gervase Markham
Mike Beltzner wrote:
> http://wiki.mozilla.org/Runway
> http://wiki.mozilla.org/Runway/Queue

Sounds good.

As for Boris's point, if any patch could affect perf, then it seems
easier just to tell smaller (read: more easily mergeable) patches to
wait, rather than try and manage them using the Runway.

Do we yet have any idea of how many large patches are involved here? If
we merge, say, 2 or 3 per day, how long are we going to be using air
traffic control?

Gerv
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Reopening the trunk: planning

Mike Beltzner
In reply to this post by Mike Connor-4
On 10-May-06, at 1:59 PM, Mike Connor wrote:

>> Ask and ye shall recieve:
>>
>> http://wiki.mozilla.org/Runway
>> http://wiki.mozilla.org/Runway/Queue
>>
> [...]
> So, this works, assuming the sheriff is committed to making time to  
> plan this stuff.  I think its important to get as much as possible  
> on this runway, but I think it'll bottleneck until we

Agreed ... the idea is that as many "and then:" fields can be added  
once order is committed. Really, the runway and the queue overlap a  
little, but one is under control of the sheriff and the other is just  
a dumping zone for patches that are ready. I'd suggested in  
#developers that instead of a queue we just use a keyword which we  
can search against in BZ, but shaver pointed out that the pilot's  
availability to check in and monitor the patch is pretty important  
data for the sheriff to be able to get at quickly.

cheers,
mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Reopening the trunk: planning

beltzner
In reply to this post by beltzner
On 5/10/06, Pam Greene <[hidden email]> wrote:
> Is the runway for every trunk patch?  Only "non-major" ones?  Or only
> "non-trivial" ones (a stronger restriction)?  And is it a permanent plan
> (which seems wise for major patches, perhaps reasonable for minor ones, but
> overly cumbersome for every single patch), or only until the flurry of
> activity settles down?
>
> Just looking for some clarification.

I believe the way it would work is that whenever the tree is closed,
people would have to look to the runway for landing clearance. When
the tree is open, there are no restrictions on landing.

Maybe I'm wrong, though.

--
/ mike beltzner / user experience lead / mozilla corporation /
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Reopening the trunk: planning

Mike Beltzner
In reply to this post by beltzner
As the mechanisms and priorities for trunk landings are worked out, I'd also like to think about what we need to do on branch to satisfy these two goals:

 1. Get to a stable A2 that we can release
 2. Merge back to trunk to avoid the "aviary effect"

There are currently four bugs blocking Fx2 Alpha 2, so obviously I'd like to see them land on the branch, but think the best time to do that will be after we merge the other patches that landed on the branch over the past little while. IMO those fixed1.8.1 patches not yet landed should be landed on trunk first, then we can go about adding the components and squashing bugs as neccessary in order to execute a quick A3 that includes DOMstorage and SafeBrowsing.

But I'm new at this. So, is this horribly wrongthinking of me?

cheers,
mike
-----Original Message-----
From: "Mike Beltzner" <[hidden email]>
Date: Wed, 10 May 2006 13:16:22
To:"Gervase Markham" <[hidden email]>
Cc:[hidden email]
Subject: Re: Reopening the trunk: planning

On 5/10/06, Gervase Markham <[hidden email]> wrote:
> queue for big landings. Can we have a wiki page fulfilling the same
> purpose? Identify the non-trivial landings, add them to the page, and

Ask and ye shall recieve:

http://wiki.mozilla.org/Runway
http://wiki.mozilla.org/Runway/Queue

I just threw that together, but it seems to serve the purpose. Please
feel free to discuss in #developers or on this list. Eventually we
might want to get some JS thrown into the templates to allow us to
dynamically update status based on tinderbox readings or something.

cheers,
mike

--
/ mike beltzner / user experience lead / mozilla corporation /
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Reopening the trunk: planning

Mike Beltzner
That should have said "as the mechanisms... are getting worked out"
-----Original Message-----
From: "Mike Beltzner" <[hidden email]>
Date: Wed, 10 May 2006 20:24:17
To:"dev.planning" <[hidden email]>
Subject: Re: Reopening the trunk: planning

As the mechanisms and priorities for trunk landings are worked out, I'd also like to think about what we need to do on branch to satisfy these two goals:

 1. Get to a stable A2 that we can release
 2. Merge back to trunk to avoid the "aviary effect"

There are currently four bugs blocking Fx2 Alpha 2, so obviously I'd like to see them land on the branch, but think the best time to do that will be after we merge the other patches that landed on the branch over the past little while. IMO those fixed1.8.1 patches not yet landed should be landed on trunk first, then we can go about adding the components and squashing bugs as neccessary in order to execute a quick A3 that includes DOMstorage and SafeBrowsing.

But I'm new at this. So, is this horribly wrongthinking of me?

cheers,
mike
-----Original Message-----
From: "Mike Beltzner" <[hidden email]>
Date: Wed, 10 May 2006 13:16:22
To:"Gervase Markham" <[hidden email]>
Cc:[hidden email]
Subject: Re: Reopening the trunk: planning

On 5/10/06, Gervase Markham <[hidden email]> wrote:
> queue for big landings. Can we have a wiki page fulfilling the same
> purpose? Identify the non-trivial landings, add them to the page, and

Ask and ye shall recieve:

http://wiki.mozilla.org/Runway
http://wiki.mozilla.org/Runway/Queue

I just threw that together, but it seems to serve the purpose. Please
feel free to discuss in #developers or on this list. Eventually we
might want to get some JS thrown into the templates to allow us to
dynamically update status based on tinderbox readings or something.

cheers,
mike

--
/ mike beltzner / user experience lead / mozilla corporation /
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning


_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Reopening the trunk: planning

Mike Connor-4
In reply to this post by Mike Beltzner
I don't think its wrongthinking.  I have no visibility into how any of the remaining patches interact with Darin's huge trunk patch, but getting trunk and branch back in sync would be a huge relief to me personally and avoid fallout down the road.

- Mike
   

-----Original Message-----
From: "Mike Beltzner" <[hidden email]>
Date: Wed, 10 May 2006 20:24:17
To:"dev.planning" <[hidden email]>
Subject: Re: Reopening the trunk: planning

As the mechanisms and priorities for trunk landings are worked out, I'd also like to think about what we need to do on branch to satisfy these two goals:

 1. Get to a stable A2 that we can release
 2. Merge back to trunk to avoid the "aviary effect"

There are currently four bugs blocking Fx2 Alpha 2, so obviously I'd like to see them land on the branch, but think the best time to do that will be after we merge the other patches that landed on the branch over the past little while. IMO those fixed1.8.1 patches not yet landed should be landed on trunk first, then we can go about adding the components and squashing bugs as neccessary in order to execute a quick A3 that includes DOMstorage and SafeBrowsing.

But I'm new at this. So, is this horribly wrongthinking of me?

cheers,
mike
-----Original Message-----
From: "Mike Beltzner" <[hidden email]>
Date: Wed, 10 May 2006 13:16:22
To:"Gervase Markham" <[hidden email]>
Cc:[hidden email]
Subject: Re: Reopening the trunk: planning

On 5/10/06, Gervase Markham <[hidden email]> wrote:
> queue for big landings. Can we have a wiki page fulfilling the same
> purpose? Identify the non-trivial landings, add them to the page, and

Ask and ye shall recieve:

http://wiki.mozilla.org/Runway
http://wiki.mozilla.org/Runway/Queue

I just threw that together, but it seems to serve the purpose. Please
feel free to discuss in #developers or on this list. Eventually we
might want to get some JS thrown into the templates to allow us to
dynamically update status based on tinderbox readings or something.

cheers,
mike

--
/ mike beltzner / user experience lead / mozilla corporation /
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning


_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Reopening the trunk: planning

Benjamin Smedberg
In reply to this post by beltzner
Mike Beltzner wrote:

> I believe the way it would work is that whenever the tree is closed,
> people would have to look to the runway for landing clearance. When
> the tree is open, there are no restrictions on landing.

I think we're overengineering this...

The current plan as discussed on #developers with Darin and Stuart is for
the trunk to stay closed tonight (May 10). Darin will clear the
threadmanager patch and get good baseline perf numbers. Then Darin (and
perhaps Stuart) will look on the runway page and ask in #developers what
other large patches need to land, and will grant landing clearance for those
as appropriate.

I will reopen the tree for general checkins tomorrow morning, and clamp down
if too much is landing at once (but it doesn't look like that will be a big
problem).

--BDS
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Reopening the trunk: planning

Mike Connor-4
Benjamin Smedberg wrote:
> Mike Beltzner wrote:
>
>> I believe the way it would work is that whenever the tree is closed,
>> people would have to look to the runway for landing clearance. When
>> the tree is open, there are no restrictions on landing.
>
> I think we're overengineering this...

I think we aren't disciplined enough right now, and we need to figure
out a way of managing risky checkins.  With the free for all, sooner or
later the wheels fall off, and then we're here again.  If the tree is
closed, we need to manage stuff and catch up major landings.  If the
tree is open, we still need a way of clearing the decks for major landings.

Back when we did daily smoketests, we had the ability to just leave the
tree closed while something major landed.  Is that worthwhile to
resurrect in some form?  Maybe have 2-3 windows a week reserved for
doing something like this?

> The current plan as discussed on #developers with Darin and Stuart is
> for the trunk to stay closed tonight (May 10). Darin will clear the
> threadmanager patch and get good baseline perf numbers. Then Darin
> (and perhaps Stuart) will look on the runway page and ask in
> #developers what other large patches need to land, and will grant
> landing clearance for those as appropriate.
>
> I will reopen the tree for general checkins tomorrow morning, and
> clamp down if too much is landing at once (but it doesn't look like
> that will be a big problem).

Are we sure on that timeline?  Seems like a 400k gzipped patch would
absolutely require more time than just one evening to shake out enough
to start landing the backlog of patches, but I do have a great deal of
faith in Darin!

-- Mike


_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Reopening the trunk: planning

Benjamin Smedberg
In reply to this post by Benjamin Smedberg
Mike Connor wrote:

> Back when we did daily smoketests, we had the ability to just leave the
> tree closed while something major landed.  Is that worthwhile to
> resurrect in some form?  Maybe have 2-3 windows a week reserved for
> doing something like this?

If somebody has something risky/needing perftests, they can and should ask
the sheriff on duty to close the tree for the landing. It's not a big deal,
and doesn't need complicated overhead. A day's notice here would be sufficient.

> Are we sure on that timeline?  Seems like a 400k gzipped patch would
> absolutely require more time than just one evening to shake out enough
> to start landing the backlog of patches, but I do have a great deal of
> faith in Darin!

If things look bad, we'll leave the tree closed longer (or backout, but that
seems unlikely given the magnitude of the change and the testing it has
already had).

--BDS
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: Reopening the trunk: planning

L. David Baron
In reply to this post by Benjamin Smedberg
On Wednesday 2006-05-10 17:26 -0400, Benjamin Smedberg wrote:
> Mike Beltzner wrote:
> >I believe the way it would work is that whenever the tree is closed,
> >people would have to look to the runway for landing clearance. When
> >the tree is open, there are no restrictions on landing.
>
> I think we're overengineering this...

Agreed.

I think the big problem is that we need to be quicker to:
 * close the tree for bustage or regressions
 * back people out of they break the rules, such as:
   * checking in on orange or red
   * checking in on branch before trunk.

-David

--
L. David Baron                                <URL: http://dbaron.org/ >
           Technical Lead, Layout & CSS, Mozilla Corporation

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning

attachment0 (196 bytes) Download Attachment
12