How to determine topmost?

classic Classic list List threaded Threaded
7 messages Options
sbq
Reply | Threaded
Open this post in threaded view
|

How to determine topmost?

sbq
Greetings,
 
I am working on this problem:  the user hovers the mouse over a web page
displayed in Firefox and presses F8.  What is the topmost object under
the mouse?  To put it another way, if the user clicked the mouse exactly
where it is hovering, which object would receive the click event?
 
To get concrete, the web page is http://www.weather.com and the mouse is
hovering over the radio button "Site" in the The Weather Channel search
box near the top of the web page.  My current algorithm yields three
items as being contenders for this topmost item:
 
1. The background <img>, which is currently a picture of the sun over
ski tracks in snow.
2. The radio button, <input type="radio">
3. The <div> that contains the text box and "Search" button right under
the radio buttons.
 
I have verified using Firebug that the HTML for these items covers the
mouse.  At different parts of the at-spi tree for the web page, these
three objects contain the mouse.  I do not see any way of further
refining these choices automatically.  I would like to know if I am
overlooking anything.
 
My algorithm recursively searches the at-spi tree for nodes that contain
the mouse (x,y).  If a node is a leaf node it is a candidate (numbers 1
and 2 above).  If a parent node in the tree contains the (x,y) but no
descendents of that node contain (x,y), then that parent node is a
candidate (number 3 above).
 
Is there information available to at-spi that could help me deduce that
the radio-button is "topmost"?
 
I believe that if all 3 items above had onclick="..." handlers and I
clicked on the radio button, only the radio button's onclick handler
would be called.  So my algorithm is not as good as Firefox's + the
underlying GUI system.
 
Is there a way for an accessible application to get the same result as
Firefox without actually clicking the item?
 
-Sam
 
 
PS here is an outline of the HTML that yields the result above:
 
...
<div id="parent">
    ...
    <div id="hbBannerDiv"...>
        <div id="headerBanner">
            <img id="mainHeaderImage" .../>   <==  contains (x,y)
        </div>
    </div>
    ...
    <form>
        ...
        <div id="hbSearchBox">
            <div id="googlesearchContent">
                <div class="c1">
                    .... <input id=headerSearchR2" type="radio">   <==
containx (x,y)
                </div>
                <div class="c2">      <== containx (x,y)
                </div>
            </div>
        </div>
        ...
    </form>
    ...
</div>
 
_______________________________________________
dev-accessibility mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-accessibility
Reply | Threaded
Open this post in threaded view
|

Re: How to determine topmost?

Aaron Leventhal-3
Sam,

Both MSAA and AT-SPI have a way to get the accessible object at a point.
This is much faster than searching the entire tree recursively. In MSAA
there is a method which clients can use, called
AccessibleObjectFromPoint(). I don't recall what it's called for AT-SPI,
but look for something called "ChildAtPoint()" or something like that.

- Aaron

On 1/6/2009 6:59 PM, Quiring, Sam wrote:

> Greetings,
>
> I am working on this problem:  the user hovers the mouse over a web page
> displayed in Firefox and presses F8.  What is the topmost object under
> the mouse?  To put it another way, if the user clicked the mouse exactly
> where it is hovering, which object would receive the click event?
>
> To get concrete, the web page is http://www.weather.com and the mouse is
> hovering over the radio button "Site" in the The Weather Channel search
> box near the top of the web page.  My current algorithm yields three
> items as being contenders for this topmost item:
>
> 1. The background<img>, which is currently a picture of the sun over
> ski tracks in snow.
> 2. The radio button,<input type="radio">
> 3. The<div>  that contains the text box and "Search" button right under
> the radio buttons.
>
> I have verified using Firebug that the HTML for these items covers the
> mouse.  At different parts of the at-spi tree for the web page, these
> three objects contain the mouse.  I do not see any way of further
> refining these choices automatically.  I would like to know if I am
> overlooking anything.
>
> My algorithm recursively searches the at-spi tree for nodes that contain
> the mouse (x,y).  If a node is a leaf node it is a candidate (numbers 1
> and 2 above).  If a parent node in the tree contains the (x,y) but no
> descendents of that node contain (x,y), then that parent node is a
> candidate (number 3 above).
>
> Is there information available to at-spi that could help me deduce that
> the radio-button is "topmost"?
>
> I believe that if all 3 items above had onclick="..." handlers and I
> clicked on the radio button, only the radio button's onclick handler
> would be called.  So my algorithm is not as good as Firefox's + the
> underlying GUI system.
>
> Is there a way for an accessible application to get the same result as
> Firefox without actually clicking the item?
>
> -Sam
>
>
> PS here is an outline of the HTML that yields the result above:
>
> ...
> <div id="parent">
>      ...
>      <div id="hbBannerDiv"...>
>          <div id="headerBanner">
>              <img id="mainHeaderImage" .../>    <==  contains (x,y)
>          </div>
>      </div>
>      ...
>      <form>
>          ...
>          <div id="hbSearchBox">
>              <div id="googlesearchContent">
>                  <div class="c1">
>                      ....<input id=headerSearchR2" type="radio">    <==
> containx (x,y)
>                  </div>
>                  <div class="c2">       <== containx (x,y)
>                  </div>
>              </div>
>          </div>
>          ...
>      </form>
>      ...
> </div>
>

_______________________________________________
dev-accessibility mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-accessibility
Reply | Threaded
Open this post in threaded view
|

Re: How to determine topmost?

Steve Lee-3
Sam,

getAccessibleAtPoint

Accerciser is a great reference app (though it is python)

Take a look at _inspectUnderMouse and _getChildAccAtCoords
http://www.koders.com/python/fid8F2A47CCB9FACC23867491E2BC6729B8EFCDCB2F.aspx?s=_inspectUnderMouse#L25

Steve

2009/1/7 Aaron Leventhal <[hidden email]>:

> Sam,
>
> Both MSAA and AT-SPI have a way to get the accessible object at a point.
> This is much faster than searching the entire tree recursively. In MSAA
> there is a method which clients can use, called AccessibleObjectFromPoint().
> I don't recall what it's called for AT-SPI, but look for something called
> "ChildAtPoint()" or something like that.
>
> - Aaron
>
> On 1/6/2009 6:59 PM, Quiring, Sam wrote:
>>
>> Greetings,
>>
>> I am working on this problem:  the user hovers the mouse over a web page
>> displayed in Firefox and presses F8.  What is the topmost object under
>> the mouse?  To put it another way, if the user clicked the mouse exactly
>> where it is hovering, which object would receive the click event?
>>
>> To get concrete, the web page is http://www.weather.com and the mouse is
>> hovering over the radio button "Site" in the The Weather Channel search
>> box near the top of the web page.  My current algorithm yields three
>> items as being contenders for this topmost item:
>>
>> 1. The background<img>, which is currently a picture of the sun over
>> ski tracks in snow.
>> 2. The radio button,<input type="radio">
>> 3. The<div>  that contains the text box and "Search" button right under
>> the radio buttons.
>>
>> I have verified using Firebug that the HTML for these items covers the
>> mouse.  At different parts of the at-spi tree for the web page, these
>> three objects contain the mouse.  I do not see any way of further
>> refining these choices automatically.  I would like to know if I am
>> overlooking anything.
>>
>> My algorithm recursively searches the at-spi tree for nodes that contain
>> the mouse (x,y).  If a node is a leaf node it is a candidate (numbers 1
>> and 2 above).  If a parent node in the tree contains the (x,y) but no
>> descendents of that node contain (x,y), then that parent node is a
>> candidate (number 3 above).
>>
>> Is there information available to at-spi that could help me deduce that
>> the radio-button is "topmost"?
>>
>> I believe that if all 3 items above had onclick="..." handlers and I
>> clicked on the radio button, only the radio button's onclick handler
>> would be called.  So my algorithm is not as good as Firefox's + the
>> underlying GUI system.
>>
>> Is there a way for an accessible application to get the same result as
>> Firefox without actually clicking the item?
>>
>> -Sam
>>
>>
>> PS here is an outline of the HTML that yields the result above:
>>
>> ...
>> <div id="parent">
>>     ...
>>     <div id="hbBannerDiv"...>
>>         <div id="headerBanner">
>>             <img id="mainHeaderImage" .../>    <==  contains (x,y)
>>         </div>
>>     </div>
>>     ...
>>     <form>
>>         ...
>>         <div id="hbSearchBox">
>>             <div id="googlesearchContent">
>>                 <div class="c1">
>>                     ....<input id=headerSearchR2" type="radio">    <==
>> containx (x,y)
>>                 </div>
>>                 <div class="c2">       <== containx (x,y)
>>                 </div>
>>             </div>
>>         </div>
>>         ...
>>     </form>
>>     ...
>> </div>
>>
>
> _______________________________________________
> dev-accessibility mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-accessibility
>



--
Steve Lee
Open Source Assistive Technology Software and Accessibility
fullmeasure.co.uk
_______________________________________________
dev-accessibility mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-accessibility
sbq
Reply | Threaded
Open this post in threaded view
|

RE: How to determine topmost?

sbq
Steve,  Aaron,

Thanks for your responses.  Unfortunately, Accerciser proves my point --
the routine Steve pointed to, _inspectUnderMouse implements the
recursive search.

Aaron, in a past life I used MSAA and yes, AccessibleObjectFromPoint()
is the perfect routine for what I'm trying to do.  It would be great if
AT-SPI had a similiar routine.  I'm already calling AT-SPI's
AccessibleComponent_getAccessibleAtPoint().  The problem is that routine
takes a component as a parameter and returns only the direct child of
the component that is at the point, if there is one.  So you can only
descend into the tree one level at a time.

If you look at the code for _inspectUnderMouse from Accerciser to which
Steve Lee pointed, you see that it searches the entire tree of
applications to find the window under the mouse.  Look at the loop
beginning:

for app in desktop
   ...

That is the same loop I'm using.  The loop calls _getChildAccAtCoords,
defined right below _inspectUnderMouse. It implements the recusive
search.

Interestingly, _inspectUnderMouse calls
wnck_screen.get_windows_stacked(), a routine that is not part of AT-SPI,
to get the Z order.  It compares the name of the windows returned by
get_windows_stacked() to the name of the AT-SPI frame for each app.
This algorithm can fail if two windows on the desktop have the same
name.  Dave Bolter had earlier suggested consulting the wnck library for
Z order.  Samuel Thibault complained that there is no perfect way to
match windows from AT-SPI to windows from Xlib (wnck is built on Xlib).

One last item, _getChildAccAtCoords does not allow for the possibility
that there may be multiple Accessibles covering the same point, like I'm
seeing in Firefox as described in my email below.  I'll copy their exact
implementation and report back on if it gets the right answer.

-Sam

-----Original Message-----
From:
dev-accessibility-bounces+sam.quiring=[hidden email]
[mailto:dev-accessibility-bounces+sam.quiring=[hidden email]
a.org] On Behalf Of Steve Lee
Sent: Wednesday, January 07, 2009 1:40 AM
To: Aaron Leventhal
Cc: [hidden email]
Subject: Re: How to determine topmost?

Sam,

getAccessibleAtPoint

Accerciser is a great reference app (though it is python)

Take a look at _inspectUnderMouse and _getChildAccAtCoords
http://www.koders.com/python/fid8F2A47CCB9FACC23867491E2BC6729B8EFCDCB2F
.aspx?s=_inspectUnderMouse#L25

Steve

2009/1/7 Aaron Leventhal <[hidden email]>:
> Sam,
>
> Both MSAA and AT-SPI have a way to get the accessible object at a
point.
> This is much faster than searching the entire tree recursively. In
> MSAA there is a method which clients can use, called
AccessibleObjectFromPoint().

> I don't recall what it's called for AT-SPI, but look for something
> called "ChildAtPoint()" or something like that.
>
> - Aaron
>
> On 1/6/2009 6:59 PM, Quiring, Sam wrote:
>>
>> Greetings,
>>
>> I am working on this problem:  the user hovers the mouse over a web
>> page displayed in Firefox and presses F8.  What is the topmost object

>> under the mouse?  To put it another way, if the user clicked the
>> mouse exactly where it is hovering, which object would receive the
click event?
>>
>> To get concrete, the web page is http://www.weather.com and the mouse

>> is hovering over the radio button "Site" in the The Weather Channel
>> search box near the top of the web page.  My current algorithm yields

>> three items as being contenders for this topmost item:
>>
>> 1. The background<img>, which is currently a picture of the sun over
>> ski tracks in snow.
>> 2. The radio button,<input type="radio"> 3. The<div>  that contains
>> the text box and "Search" button right under the radio buttons.
>>
>> I have verified using Firebug that the HTML for these items covers
>> the mouse.  At different parts of the at-spi tree for the web page,
>> these three objects contain the mouse.  I do not see any way of
>> further refining these choices automatically.  I would like to know
>> if I am overlooking anything.
>>
>> My algorithm recursively searches the at-spi tree for nodes that
>> contain the mouse (x,y).  If a node is a leaf node it is a candidate
>> (numbers 1 and 2 above).  If a parent node in the tree contains the
>> (x,y) but no descendents of that node contain (x,y), then that parent

>> node is a candidate (number 3 above).
>>
>> Is there information available to at-spi that could help me deduce
>> that the radio-button is "topmost"?
>>
>> I believe that if all 3 items above had onclick="..." handlers and I
>> clicked on the radio button, only the radio button's onclick handler
>> would be called.  So my algorithm is not as good as Firefox's + the
>> underlying GUI system.
>>
>> Is there a way for an accessible application to get the same result
>> as Firefox without actually clicking the item?
>>
>> -Sam
>>
>>
>> PS here is an outline of the HTML that yields the result above:
>>
>> ...
>> <div id="parent">
>>     ...
>>     <div id="hbBannerDiv"...>
>>         <div id="headerBanner">
>>             <img id="mainHeaderImage" .../>    <==  contains (x,y)
>>         </div>
>>     </div>
>>     ...
>>     <form>
>>         ...
>>         <div id="hbSearchBox">
>>             <div id="googlesearchContent">
>>                 <div class="c1">
>>                     ....<input id=headerSearchR2" type="radio">
<==

>> containx (x,y)
>>                 </div>
>>                 <div class="c2">       <== containx (x,y)
>>                 </div>
>>             </div>
>>         </div>
>>         ...
>>     </form>
>>     ...
>> </div>
>>
>
> _______________________________________________
> dev-accessibility mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-accessibility
>



--
Steve Lee
Open Source Assistive Technology Software and Accessibility
fullmeasure.co.uk _______________________________________________
dev-accessibility mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-accessibility
_______________________________________________
dev-accessibility mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-accessibility
Reply | Threaded
Open this post in threaded view
|

Re: How to determine topmost?

Steve Lee-3
2009/1/7 Quiring, Sam <[hidden email]>:
> Thanks for your responses.  Unfortunately, Accerciser proves my point --
> the routine Steve pointed to, _inspectUnderMouse implements the
> recursive search.

Yes I spotted that and am not sure of the exact reason, most likely In
X there is no single global tree and it is up to the Window Manager to
do what it likes with active apps. In Win32 it's all one big
monolithic hierarchy really.

I suggest you join us on #accessibility and ask Eitan himself (
eeejay) while he's in our timezone

Steve

> Aaron, in a past life I used MSAA and yes, AccessibleObjectFromPoint()
> is the perfect routine for what I'm trying to do.  It would be great if
> AT-SPI had a similiar routine.  I'm already calling AT-SPI's
> AccessibleComponent_getAccessibleAtPoint().  The problem is that routine
> takes a component as a parameter and returns only the direct child of
> the component that is at the point, if there is one.  So you can only
> descend into the tree one level at a time.
>
> If you look at the code for _inspectUnderMouse from Accerciser to which
> Steve Lee pointed, you see that it searches the entire tree of
> applications to find the window under the mouse.  Look at the loop
> beginning:
>
> for app in desktop
>   ...
>
> That is the same loop I'm using.  The loop calls _getChildAccAtCoords,
> defined right below _inspectUnderMouse. It implements the recusive
> search.
>
> Interestingly, _inspectUnderMouse calls
> wnck_screen.get_windows_stacked(), a routine that is not part of AT-SPI,
> to get the Z order.  It compares the name of the windows returned by
> get_windows_stacked() to the name of the AT-SPI frame for each app.
> This algorithm can fail if two windows on the desktop have the same
> name.  Dave Bolter had earlier suggested consulting the wnck library for
> Z order.  Samuel Thibault complained that there is no perfect way to
> match windows from AT-SPI to windows from Xlib (wnck is built on Xlib).
>
> One last item, _getChildAccAtCoords does not allow for the possibility
> that there may be multiple Accessibles covering the same point, like I'm
> seeing in Firefox as described in my email below.  I'll copy their exact
> implementation and report back on if it gets the right answer.
>
> -Sam
>
> -----Original Message-----
> From:
> dev-accessibility-bounces+sam.quiring=[hidden email]
> [mailto:dev-accessibility-bounces+sam.quiring=[hidden email]
> a.org] On Behalf Of Steve Lee
> Sent: Wednesday, January 07, 2009 1:40 AM
> To: Aaron Leventhal
> Cc: [hidden email]
> Subject: Re: How to determine topmost?
>
> Sam,
>
> getAccessibleAtPoint
>
> Accerciser is a great reference app (though it is python)
>
> Take a look at _inspectUnderMouse and _getChildAccAtCoords
> http://www.koders.com/python/fid8F2A47CCB9FACC23867491E2BC6729B8EFCDCB2F
> .aspx?s=_inspectUnderMouse#L25
>
> Steve
>
> 2009/1/7 Aaron Leventhal <[hidden email]>:
>> Sam,
>>
>> Both MSAA and AT-SPI have a way to get the accessible object at a
> point.
>> This is much faster than searching the entire tree recursively. In
>> MSAA there is a method which clients can use, called
> AccessibleObjectFromPoint().
>> I don't recall what it's called for AT-SPI, but look for something
>> called "ChildAtPoint()" or something like that.
>>
>> - Aaron
>>
>> On 1/6/2009 6:59 PM, Quiring, Sam wrote:
>>>
>>> Greetings,
>>>
>>> I am working on this problem:  the user hovers the mouse over a web
>>> page displayed in Firefox and presses F8.  What is the topmost object
>
>>> under the mouse?  To put it another way, if the user clicked the
>>> mouse exactly where it is hovering, which object would receive the
> click event?
>>>
>>> To get concrete, the web page is http://www.weather.com and the mouse
>
>>> is hovering over the radio button "Site" in the The Weather Channel
>>> search box near the top of the web page.  My current algorithm yields
>
>>> three items as being contenders for this topmost item:
>>>
>>> 1. The background<img>, which is currently a picture of the sun over
>>> ski tracks in snow.
>>> 2. The radio button,<input type="radio"> 3. The<div>  that contains
>>> the text box and "Search" button right under the radio buttons.
>>>
>>> I have verified using Firebug that the HTML for these items covers
>>> the mouse.  At different parts of the at-spi tree for the web page,
>>> these three objects contain the mouse.  I do not see any way of
>>> further refining these choices automatically.  I would like to know
>>> if I am overlooking anything.
>>>
>>> My algorithm recursively searches the at-spi tree for nodes that
>>> contain the mouse (x,y).  If a node is a leaf node it is a candidate
>>> (numbers 1 and 2 above).  If a parent node in the tree contains the
>>> (x,y) but no descendents of that node contain (x,y), then that parent
>
>>> node is a candidate (number 3 above).
>>>
>>> Is there information available to at-spi that could help me deduce
>>> that the radio-button is "topmost"?
>>>
>>> I believe that if all 3 items above had onclick="..." handlers and I
>>> clicked on the radio button, only the radio button's onclick handler
>>> would be called.  So my algorithm is not as good as Firefox's + the
>>> underlying GUI system.
>>>
>>> Is there a way for an accessible application to get the same result
>>> as Firefox without actually clicking the item?
>>>
>>> -Sam
>>>
>>>
>>> PS here is an outline of the HTML that yields the result above:
>>>
>>> ...
>>> <div id="parent">
>>>     ...
>>>     <div id="hbBannerDiv"...>
>>>         <div id="headerBanner">
>>>             <img id="mainHeaderImage" .../>    <==  contains (x,y)
>>>         </div>
>>>     </div>
>>>     ...
>>>     <form>
>>>         ...
>>>         <div id="hbSearchBox">
>>>             <div id="googlesearchContent">
>>>                 <div class="c1">
>>>                     ....<input id=headerSearchR2" type="radio">
> <==
>>> containx (x,y)
>>>                 </div>
>>>                 <div class="c2">       <== containx (x,y)
>>>                 </div>
>>>             </div>
>>>         </div>
>>>         ...
>>>     </form>
>>>     ...
>>> </div>
>>>
>>
>> _______________________________________________
>> dev-accessibility mailing list
>> [hidden email]
>> https://lists.mozilla.org/listinfo/dev-accessibility
>>
>
>
>
> --
> Steve Lee
> Open Source Assistive Technology Software and Accessibility
> fullmeasure.co.uk _______________________________________________
> dev-accessibility mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-accessibility
>



--
Steve Lee
Open Source Assistive Technology Software and Accessibility
fullmeasure.co.uk
_______________________________________________
dev-accessibility mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-accessibility
sbq
Reply | Threaded
Open this post in threaded view
|

RE: How to determine topmost?

sbq
In reply to this post by sbq
Steve, Aaron,

I wrote an algorithm to use AccessibleComponent_getAccessibleAtPoint()
and compared it to what I've been getting (described in previous email,
below).  The result appears to be better in two ways.  

Using getAccessibleAtPoint

    1. there is only one result (I was getting three) and, so far, it is
the desirable result.
    2. is a lot faster because it lets me ignore huge chunks of the
tree.

There is one node in the Mozilla AT-SPI tree that has three children
containing the mouse.  getAccessibleAtPoint() selects the most
interesting child.

Thanks for the pointer to _inspectUnderMouse.

-Sam


 

-----Original Message-----
From:
dev-accessibility-bounces+sam.quiring=[hidden email]
[mailto:dev-accessibility-bounces+sam.quiring=[hidden email]
a.org] On Behalf Of Quiring, Sam
Sent: Wednesday, January 07, 2009 7:33 AM
To: Steve Lee; Aaron Leventhal
Cc: [hidden email]
Subject: RE: How to determine topmost?

Steve,  Aaron,

Thanks for your responses.  Unfortunately, Accerciser proves my point --
the routine Steve pointed to, _inspectUnderMouse implements the
recursive search.

Aaron, in a past life I used MSAA and yes, AccessibleObjectFromPoint()
is the perfect routine for what I'm trying to do.  It would be great if
AT-SPI had a similiar routine.  I'm already calling AT-SPI's
AccessibleComponent_getAccessibleAtPoint().  The problem is that routine
takes a component as a parameter and returns only the direct child of
the component that is at the point, if there is one.  So you can only
descend into the tree one level at a time.

If you look at the code for _inspectUnderMouse from Accerciser to which
Steve Lee pointed, you see that it searches the entire tree of
applications to find the window under the mouse.  Look at the loop
beginning:

for app in desktop
   ...

That is the same loop I'm using.  The loop calls _getChildAccAtCoords,
defined right below _inspectUnderMouse. It implements the recusive
search.

Interestingly, _inspectUnderMouse calls
wnck_screen.get_windows_stacked(), a routine that is not part of AT-SPI,
to get the Z order.  It compares the name of the windows returned by
get_windows_stacked() to the name of the AT-SPI frame for each app.
This algorithm can fail if two windows on the desktop have the same
name.  Dave Bolter had earlier suggested consulting the wnck library for
Z order.  Samuel Thibault complained that there is no perfect way to
match windows from AT-SPI to windows from Xlib (wnck is built on Xlib).

One last item, _getChildAccAtCoords does not allow for the possibility
that there may be multiple Accessibles covering the same point, like I'm
seeing in Firefox as described in my email below.  I'll copy their exact
implementation and report back on if it gets the right answer.

-Sam

-----Original Message-----
From:
dev-accessibility-bounces+sam.quiring=[hidden email]
[mailto:dev-accessibility-bounces+sam.quiring=[hidden email]
a.org] On Behalf Of Steve Lee
Sent: Wednesday, January 07, 2009 1:40 AM
To: Aaron Leventhal
Cc: [hidden email]
Subject: Re: How to determine topmost?

Sam,

getAccessibleAtPoint

Accerciser is a great reference app (though it is python)

Take a look at _inspectUnderMouse and _getChildAccAtCoords
http://www.koders.com/python/fid8F2A47CCB9FACC23867491E2BC6729B8EFCDCB2F
.aspx?s=_inspectUnderMouse#L25

Steve

2009/1/7 Aaron Leventhal <[hidden email]>:
> Sam,
>
> Both MSAA and AT-SPI have a way to get the accessible object at a
point.
> This is much faster than searching the entire tree recursively. In
> MSAA there is a method which clients can use, called
AccessibleObjectFromPoint().

> I don't recall what it's called for AT-SPI, but look for something
> called "ChildAtPoint()" or something like that.
>
> - Aaron
>
> On 1/6/2009 6:59 PM, Quiring, Sam wrote:
>>
>> Greetings,
>>
>> I am working on this problem:  the user hovers the mouse over a web
>> page displayed in Firefox and presses F8.  What is the topmost object

>> under the mouse?  To put it another way, if the user clicked the
>> mouse exactly where it is hovering, which object would receive the
click event?
>>
>> To get concrete, the web page is http://www.weather.com and the mouse

>> is hovering over the radio button "Site" in the The Weather Channel
>> search box near the top of the web page.  My current algorithm yields

>> three items as being contenders for this topmost item:
>>
>> 1. The background<img>, which is currently a picture of the sun over
>> ski tracks in snow.
>> 2. The radio button,<input type="radio"> 3. The<div>  that contains
>> the text box and "Search" button right under the radio buttons.
>>
>> I have verified using Firebug that the HTML for these items covers
>> the mouse.  At different parts of the at-spi tree for the web page,
>> these three objects contain the mouse.  I do not see any way of
>> further refining these choices automatically.  I would like to know
>> if I am overlooking anything.
>>
>> My algorithm recursively searches the at-spi tree for nodes that
>> contain the mouse (x,y).  If a node is a leaf node it is a candidate
>> (numbers 1 and 2 above).  If a parent node in the tree contains the
>> (x,y) but no descendents of that node contain (x,y), then that parent

>> node is a candidate (number 3 above).
>>
>> Is there information available to at-spi that could help me deduce
>> that the radio-button is "topmost"?
>>
>> I believe that if all 3 items above had onclick="..." handlers and I
>> clicked on the radio button, only the radio button's onclick handler
>> would be called.  So my algorithm is not as good as Firefox's + the
>> underlying GUI system.
>>
>> Is there a way for an accessible application to get the same result
>> as Firefox without actually clicking the item?
>>
>> -Sam
>>
>>
>> PS here is an outline of the HTML that yields the result above:
>>
>> ...
>> <div id="parent">
>>     ...
>>     <div id="hbBannerDiv"...>
>>         <div id="headerBanner">
>>             <img id="mainHeaderImage" .../>    <==  contains (x,y)
>>         </div>
>>     </div>
>>     ...
>>     <form>
>>         ...
>>         <div id="hbSearchBox">
>>             <div id="googlesearchContent">
>>                 <div class="c1">
>>                     ....<input id=headerSearchR2" type="radio">
<==

>> containx (x,y)
>>                 </div>
>>                 <div class="c2">       <== containx (x,y)
>>                 </div>
>>             </div>
>>         </div>
>>         ...
>>     </form>
>>     ...
>> </div>
>>
>
> _______________________________________________
> dev-accessibility mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-accessibility
>



--
Steve Lee
Open Source Assistive Technology Software and Accessibility
fullmeasure.co.uk _______________________________________________
dev-accessibility mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-accessibility
_______________________________________________
dev-accessibility mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-accessibility
_______________________________________________
dev-accessibility mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-accessibility
Reply | Threaded
Open this post in threaded view
|

Re: How to determine topmost?

Steve Lee-3
Excellent!

2009/1/22 Quiring, Sam <[hidden email]>:

> Steve, Aaron,
>
> I wrote an algorithm to use AccessibleComponent_getAccessibleAtPoint()
> and compared it to what I've been getting (described in previous email,
> below).  The result appears to be better in two ways.
>
> Using getAccessibleAtPoint
>
>    1. there is only one result (I was getting three) and, so far, it is
> the desirable result.
>    2. is a lot faster because it lets me ignore huge chunks of the
> tree.
>
> There is one node in the Mozilla AT-SPI tree that has three children
> containing the mouse.  getAccessibleAtPoint() selects the most
> interesting child.
>
> Thanks for the pointer to _inspectUnderMouse.
>
> -Sam
>
>
>
>
> -----Original Message-----
> From:
> dev-accessibility-bounces+sam.quiring=[hidden email]
> [mailto:dev-accessibility-bounces+sam.quiring=[hidden email]
> a.org] On Behalf Of Quiring, Sam
> Sent: Wednesday, January 07, 2009 7:33 AM
> To: Steve Lee; Aaron Leventhal
> Cc: [hidden email]
> Subject: RE: How to determine topmost?
>
> Steve,  Aaron,
>
> Thanks for your responses.  Unfortunately, Accerciser proves my point --
> the routine Steve pointed to, _inspectUnderMouse implements the
> recursive search.
>
> Aaron, in a past life I used MSAA and yes, AccessibleObjectFromPoint()
> is the perfect routine for what I'm trying to do.  It would be great if
> AT-SPI had a similiar routine.  I'm already calling AT-SPI's
> AccessibleComponent_getAccessibleAtPoint().  The problem is that routine
> takes a component as a parameter and returns only the direct child of
> the component that is at the point, if there is one.  So you can only
> descend into the tree one level at a time.
>
> If you look at the code for _inspectUnderMouse from Accerciser to which
> Steve Lee pointed, you see that it searches the entire tree of
> applications to find the window under the mouse.  Look at the loop
> beginning:
>
> for app in desktop
>   ...
>
> That is the same loop I'm using.  The loop calls _getChildAccAtCoords,
> defined right below _inspectUnderMouse. It implements the recusive
> search.
>
> Interestingly, _inspectUnderMouse calls
> wnck_screen.get_windows_stacked(), a routine that is not part of AT-SPI,
> to get the Z order.  It compares the name of the windows returned by
> get_windows_stacked() to the name of the AT-SPI frame for each app.
> This algorithm can fail if two windows on the desktop have the same
> name.  Dave Bolter had earlier suggested consulting the wnck library for
> Z order.  Samuel Thibault complained that there is no perfect way to
> match windows from AT-SPI to windows from Xlib (wnck is built on Xlib).
>
> One last item, _getChildAccAtCoords does not allow for the possibility
> that there may be multiple Accessibles covering the same point, like I'm
> seeing in Firefox as described in my email below.  I'll copy their exact
> implementation and report back on if it gets the right answer.
>
> -Sam
>
> -----Original Message-----
> From:
> dev-accessibility-bounces+sam.quiring=[hidden email]
> [mailto:dev-accessibility-bounces+sam.quiring=[hidden email]
> a.org] On Behalf Of Steve Lee
> Sent: Wednesday, January 07, 2009 1:40 AM
> To: Aaron Leventhal
> Cc: [hidden email]
> Subject: Re: How to determine topmost?
>
> Sam,
>
> getAccessibleAtPoint
>
> Accerciser is a great reference app (though it is python)
>
> Take a look at _inspectUnderMouse and _getChildAccAtCoords
> http://www.koders.com/python/fid8F2A47CCB9FACC23867491E2BC6729B8EFCDCB2F
> .aspx?s=_inspectUnderMouse#L25
>
> Steve
>
> 2009/1/7 Aaron Leventhal <[hidden email]>:
>> Sam,
>>
>> Both MSAA and AT-SPI have a way to get the accessible object at a
> point.
>> This is much faster than searching the entire tree recursively. In
>> MSAA there is a method which clients can use, called
> AccessibleObjectFromPoint().
>> I don't recall what it's called for AT-SPI, but look for something
>> called "ChildAtPoint()" or something like that.
>>
>> - Aaron
>>
>> On 1/6/2009 6:59 PM, Quiring, Sam wrote:
>>>
>>> Greetings,
>>>
>>> I am working on this problem:  the user hovers the mouse over a web
>>> page displayed in Firefox and presses F8.  What is the topmost object
>
>>> under the mouse?  To put it another way, if the user clicked the
>>> mouse exactly where it is hovering, which object would receive the
> click event?
>>>
>>> To get concrete, the web page is http://www.weather.com and the mouse
>
>>> is hovering over the radio button "Site" in the The Weather Channel
>>> search box near the top of the web page.  My current algorithm yields
>
>>> three items as being contenders for this topmost item:
>>>
>>> 1. The background<img>, which is currently a picture of the sun over
>>> ski tracks in snow.
>>> 2. The radio button,<input type="radio"> 3. The<div>  that contains
>>> the text box and "Search" button right under the radio buttons.
>>>
>>> I have verified using Firebug that the HTML for these items covers
>>> the mouse.  At different parts of the at-spi tree for the web page,
>>> these three objects contain the mouse.  I do not see any way of
>>> further refining these choices automatically.  I would like to know
>>> if I am overlooking anything.
>>>
>>> My algorithm recursively searches the at-spi tree for nodes that
>>> contain the mouse (x,y).  If a node is a leaf node it is a candidate
>>> (numbers 1 and 2 above).  If a parent node in the tree contains the
>>> (x,y) but no descendents of that node contain (x,y), then that parent
>
>>> node is a candidate (number 3 above).
>>>
>>> Is there information available to at-spi that could help me deduce
>>> that the radio-button is "topmost"?
>>>
>>> I believe that if all 3 items above had onclick="..." handlers and I
>>> clicked on the radio button, only the radio button's onclick handler
>>> would be called.  So my algorithm is not as good as Firefox's + the
>>> underlying GUI system.
>>>
>>> Is there a way for an accessible application to get the same result
>>> as Firefox without actually clicking the item?
>>>
>>> -Sam
>>>
>>>
>>> PS here is an outline of the HTML that yields the result above:
>>>
>>> ...
>>> <div id="parent">
>>>     ...
>>>     <div id="hbBannerDiv"...>
>>>         <div id="headerBanner">
>>>             <img id="mainHeaderImage" .../>    <==  contains (x,y)
>>>         </div>
>>>     </div>
>>>     ...
>>>     <form>
>>>         ...
>>>         <div id="hbSearchBox">
>>>             <div id="googlesearchContent">
>>>                 <div class="c1">
>>>                     ....<input id=headerSearchR2" type="radio">
> <==
>>> containx (x,y)
>>>                 </div>
>>>                 <div class="c2">       <== containx (x,y)
>>>                 </div>
>>>             </div>
>>>         </div>
>>>         ...
>>>     </form>
>>>     ...
>>> </div>
>>>
>>
>> _______________________________________________
>> dev-accessibility mailing list
>> [hidden email]
>> https://lists.mozilla.org/listinfo/dev-accessibility
>>
>
>
>
> --
> Steve Lee
> Open Source Assistive Technology Software and Accessibility
> fullmeasure.co.uk _______________________________________________
> dev-accessibility mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-accessibility
> _______________________________________________
> dev-accessibility mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-accessibility
>



--
Steve Lee
Open Source Assistive Technology Software and Accessibility
fullmeasure.co.uk
_______________________________________________
dev-accessibility mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-accessibility