Extension architecture help required

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Extension architecture help required

Gervase Markham
Hi,

In my Sync extension:
http://bzr.mozilla.org/bugzilla/extensions/sync/
I am using the bug_end_of_update hook to do syncing once a bug is
updated. However, in the current implementations of sync clients, I sync
asynchronously, to avoid blocking the main execution. So what happens is
that a job gets added to the job queue which does the sync for that bug,
and some time later the job gets pulled off and run.

One of the job parameters is the bug ID, because the jobqueue system
Stores (using Storable) its parameters, and it doesn't like Storing the
entire updated bug object. So I pass the ID and recreate the object
using new Bugzilla::Bug($id);

This is normally fine, because the job queue comes with a bit of a delay
built in. However, bug_end_of_update runs inside a transaction. If the
job gets pulled off the queue and run before the transaction completes
(which is perhaps more likely if a bug is part of a large batch update,
all of which share a transaction - see process_bug.cgi) then when the
syncing job recreates the bug object, it gets stale data. And the wrong
data gets sent to the remote system.

How can I avoid this? Ideally I want a generic solution, such that the
"something's changed" event isn't even triggered until the database has
been updated. But there's no hook for that, and there isn't an obvious
place to add one. Otherwise, is there some way my code can wait for the
transaction to complete? Problem is, it would need to do so in each
individual syncing job implementation, which is an implementation detail
I'd hope to hide.

Ideas welcome!

Gerv
_______________________________________________
dev-apps-bugzilla mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-bugzilla
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Extension architecture help required

glob
hey gerv,

Gervase Markham wrote:
In my Sync extension:
http://bzr.mozilla.org/bugzilla/extensions/sync/
I am using the bug_end_of_update hook to do syncing once a bug is
updated. 
i have an extension which is designed to push updates to other systems which currently poll bugzilla for updates (pulse, elastic search caches, metrics, etc).
https://github.com/globau/bugzilla-push
it's extremely close to completion, and is a q2 goal.

i deal with the issue you've encountered by performing serialisation of the bug at the time the update is made, in my case to JSON.  these are inserted into a custom table which a custom daemon polls and sends.  i don't use the jobqueue system because that doesn't guarantee the order of the messages, which is important when you're sending update messages (especially when a remote system is down).

since i already have a working system for sending bug updates which deals correctly with ordering of the updates, as well as queuing messages when the remote system is down, another option would be for the update-pushing part of your extension to utilise my bugzilla-push extension and be dropped in as another connector.  see https://github.com/globau/bugzilla-push/tree/master/Push/lib/Connector



-byron
--
byron - <a class="moz-txt-link-freetext" href="irc:glob">irc:glob - bugzilla.mozilla.org team -

Reply | Threaded
Open this post in threaded view
|

Re: Extension architecture help required

Max Kanat-Alexander
        Yeah, what Byron says sounds good. Also, another option would be to add
a hook to Bugzilla::DB::bz_commit_transaction. You could store things in
$self in the extension during bug_end_of_update and trigger them when
the transaction was committed. $self in an extension is an object that's
re-created for every page call, so you're fairly safe there.

        -Max

On 04/29/2012 07:27 PM, Byron Jones wrote:

> hey gerv,
>
> Gervase Markham wrote:
>> In my Sync extension:
>> http://bzr.mozilla.org/bugzilla/extensions/sync/
>> I am using the bug_end_of_update hook to do syncing once a bug is
>> updated.
> i have an extension which is designed to push updates to other systems
> which currently poll bugzilla for updates (pulse, elastic search caches,
> metrics, etc).
> https://github.com/globau/bugzilla-push
> it's extremely close to completion, and is a q2 goal.
>
> i deal with the issue you've encountered by performing serialisation of
> the bug at the time the update is made, in my case to JSON. these are
> inserted into a custom table which a custom daemon polls and sends. i
> don't use the jobqueue system because that doesn't guarantee the order
> of the messages, which is important when you're sending update messages
> (especially when a remote system is down).
>
> since i already have a working system for sending bug updates which
> deals correctly with ordering of the updates, as well as queuing
> messages when the remote system is down, another option would be for the
> update-pushing part of your extension to utilise my bugzilla-push
> extension and be dropped in as another connector. see
> https://github.com/globau/bugzilla-push/tree/master/Push/lib/Connector
>
>
>
> -byron
> --
> byron - irc:glob - bugzilla.mozilla.org team -
>


--
Max Kanat-Alexander
Chief Architect, Community Lead, and Release Manager
Bugzilla Project
http://www.bugzilla.org/
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>
Reply | Threaded
Open this post in threaded view
|

Re: Extension architecture help required

Gervase Markham
In reply to this post by glob
On 07/05/12 23:12, Max Kanat-Alexander wrote:
>     Yeah, what Byron says sounds good. Also, another option would be to
> add a hook to Bugzilla::DB::bz_commit_transaction. You could store
> things in $self in the extension during bug_end_of_update and trigger
> them when the transaction was committed. $self in an extension is an
> object that's re-created for every page call, so you're fairly safe there.

In the end, I just checked for a timestamp mismatch in the job itself
and called $job->declined() if one occurred. That reschedules it after 5
minutes.

Gerv

_______________________________________________
dev-apps-bugzilla mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-bugzilla
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=lists@...>