This discussion is archived
11 Replies Latest reply: Jun 10, 2011 3:06 AM by 867880 RSS

Problem to get accessible context by using getAccessibleContextAt

843807 Newbie
Currently Being Moderated
Hi, I'm writing program in c# that uses Java Access Bridge in order to get accessible object from java application.
For my purpose I use p/invoke for calling methods from WindowsAccessBridge.dll.
The flow of my actions is:
First I get window handle (hwnd) from win32 function WindowFromPoint
Next I get accessible context from getAccessibleContextFromHwnd
Next I try to get accessible context at mouse position by calling the getAccessibleContextAt function, I pass it accessible context I got from the widow handle and the mouse position, but the value that I get is always 0 (its IntPtr.Zero).

I will appreciate any help.
Thanks
  • 1. Re: Problem to get accessible context by using getAccessibleContextAt
    843807 Newbie
    Currently Being Moderated
    I may be able to answer your question, but it generates another question.

    On a completely clean Windows XP machine, I've installed jre-1_5_0_09-windows-i586-p.exe. On top of this I've installed accessbridge-2_0.exe and run JavaFeret over a small sample application.

    I've enabled the menu option UpdateSettings\Update with F1 and as I press F1 with the mouse over the sample application, I get a full list of properties for the item the mouse is over. This feature must use getAccessibleContextAt under the covers.

    If I now upgrade to accessbridge-2_0_1.exe and run the same test, I get no information (other than the Java virtual machine information).

    Two screen shots of my tests are here:
    http://www.zen37830.zen.co.uk/v2.0.0.png
    http://www.zen37830.zen.co.uk/v2.0.1.png

    It appears to therefore be a bug in v2.0.1 of the access bridge. My problem is I cannot find where to report this. Is this the right place?
  • 2. Re: Problem to get accessible context by using getAccessibleContextAt
    843807 Newbie
    Currently Being Moderated
    I also used JavaFeret but in my case without UpdateSettings\Update checked.
    I just used mouse track and in this case they get accessible context from MouseEntered event, may be they do the same in your case.

    I didn't try the accessbridge-2_0.exe version I directly installed accessbridge-2_0_1.exe, so you may be right and there is a bug in this version I will check it tomorrow.

    I don't know if this is the right place to report about the bug.

    Thank you for your replay.
    I'll let you know if it will help me.
  • 3. Re: Problem to get accessible context by using getAccessibleContextAt
    843807 Newbie
    Currently Being Moderated
    Yes you were right, in accessbridge-2_0.exe it works.
  • 4. Re: Problem to get accessible context by using getAccessibleContextAt
    843807 Newbie
    Currently Being Moderated
    I think it probably needs to be logged here:
    http://bugs.sun.com/bugdatabase/
    (The link is found half-way down the http://java.sun.com/ page)

    Unfortunately all I get is a 'Your request has timed out.' error message. I'll keep trying through the day.
  • 5. Re: Problem to get accessible context by using getAccessibleContextAt
    843807 Newbie
    Currently Being Moderated
    Bug logged with the following reply:

    ************************************************
    Dear Java Developer,

    Thank you for your interest in improving the quality of Java Technology.

    Your report has been assigned an internal review ID of 824507, which is NOT visible on the Sun Developer Network (SDN).

    Please be aware that the large volume of reports we receive sometimes prevents us from responding individually to each message.

    We currently have a three week average response time. If the information is determined to be a new Bug or RFE, or a duplicate of a known Bug or RFE, you will receive a followup email containing a seven digit bug number. You may search for, view, or vote for this bug in the Bug Database at http://bugs.sun.com/.
  • 6. Re: Problem to get accessible context by using getAccessibleContextAt
    843807 Newbie
    Currently Being Moderated
    Hi,

    Did you end up finding a solution for this problem? Im trying to do something very similar and would like to know if there are any work arounds.

    Regards,

    Mark
  • 7. Re: Problem to get accessible context by using getAccessibleContextAt
    843807 Newbie
    Currently Being Moderated
    I'm having the same problem and after confirming it was a bug rather than my code, I found this topic through Google. I take it there was never a follow-up to the bug report?
  • 8. Re: Problem to get accessible context by using getAccessibleContextAt
    843807 Newbie
    Currently Being Moderated
    I get the impression that Java Access Bridge has basically been abandoned by Sun, given that the latest release basically doesn't work at all, and bug reports never make it into the actual database. Fair comment?

    Currently it seems that the only way to use Java Access Bridge is to find an old download link to V2.0, since the non-working 2.0.1 is the only version that can be officially downloaded via the Sun site.
  • 9. Re: Problem to get accessible context by using getAccessibleContextAt
    843807 Newbie
    Currently Being Moderated
    I am also having this issue. Does anyone have a link to the old 2.0 version? OR maybe by chance, has Sun fixed it and/or made a work around?
  • 10. Re: Problem to get accessible context by using getAccessibleContextAt
    867880 Newbie
    Currently Being Moderated
    I find it interesting that this function, is not working, I was planning on trying this to improve the performance of my current approach. I do however have a solution to the problem. This happens to be my existing approach and I will post the code here tommorrow. What you can do to get the accessible context at the global mouse point is:

    1) Retrieve the entire accessible tree from the root accessible context by re cursing the child, IAccessibleContexts.
    2) During the recursion add this to a generic list of List<IAccessibleContexts>.
    3) Sort the list by adding a simple comparer, to sort the bounding rectangles from smallest to largest
    4) Write a lambda expression to find(e=>e.Bounds.Contains(MousePoint))
    5) The net result of this will be the smallest accessible control under the X,Y



    I have this code currently working on bridge 2.0.1 and I am testing it on SAP for Java version 7.2. I find that the associative component tree helps with the realization of control recognition. Example Bridge will return a role type of "label", by walking the tree I am able to see that it's parent, parent ect may be a treeview, in which case I can identify a node. In other cases it may be a grid, in which case it is a row or cell.

    I do however have customers complaining that on very busy screens (screens with grids and lots of visual controls) the control recognition takes to long. Hence my wanting to investigate the native getAccessibleContextAt(),

    Remember however the Java Bridge can only provide the accessible information that is exposed in the target application.
  • 11. Re: Problem to get accessible context by using getAccessibleContextAt
    867880 Newbie
    Currently Being Moderated
    As I explained yesterday, I would provide the code to my previous post:

    Some Structures that will be required in C#


    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)]
    public struct AccessibleContextInfo
    {
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = WABAPI.MAX_STRING_SIZE)]
    public string name;          // the AccessibleName of the object
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = WABAPI.MAX_STRING_SIZE)]
    public string description;     // the AccessibleDescription of the object

    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = WABAPI.SHORT_STRING_SIZE)]
    public string role;     // localized AccesibleRole string
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = WABAPI.SHORT_STRING_SIZE)]
    public string role_en_US;     // AccesibleRole string in the en_US locale
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = WABAPI.SHORT_STRING_SIZE)]
    public string states;     // localized AccesibleStateSet string (comma separated)
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = WABAPI.SHORT_STRING_SIZE)]
    public string states_en_US; // AccesibleStateSet string in the en_US locale (comma separated)

    public Int32 indexInParent;               // index of object in parent
    public Int32 childrenCount;               // # of children, if any

    public Int32 x;                         // screen coords in pixels
    public Int32 y;                         // "
    public Int32 width;                    // pixel width of object
    public Int32 height;                    // pixel height of object

    public Boolean accessibleComponent;     // flags for various additional
    public Boolean accessibleAction;               // Java Accessibility interfaces
    public Boolean accessibleSelection;          // FALSE if this object doesn't
    public Boolean accessibleText;               // implement the additional interface
    // in question

    // BOOL accessibleValue; // old BOOL indicating whether AccessibleValue is supported
    public Boolean accessibleInterfaces;          // new bitfield containing additional interface flags
    }

    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)]
    private struct AccessibleTextInfo
    {
    public Int32 charCount;                              // # of characters in this text object
    public Int32 caretIndex;                         // index of caret
    public Int32 indexAtPoint;
    }

    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)]
    private struct AccessibleTextItemsInfo
    {
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 1)]
    public String letter;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = WABAPI.SHORT_STRING_SIZE)]
    public String word;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = WABAPI.MAX_STRING_SIZE)]
    public String sentence;
    }

    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)]
    private struct AccessibleTextAttributesInfo
    {
    public Boolean bold;
    public Boolean italic;
    public Boolean underline;
    public Boolean strikethrough;
    public Boolean superscript;
    public Boolean subscript;

    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = WABAPI.SHORT_STRING_SIZE)]
    public String backgroundColor;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = WABAPI.SHORT_STRING_SIZE)]
    public String foregroundColor;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = WABAPI.SHORT_STRING_SIZE)]
    public String fontFamily;
    public Int32 fontSize;

    public Int32 alignment;
    public Int32 bidiLevel;

    public Single firstLineIndent;
    public Single leftIndent;
    public Single rightIndent;
    public Single lineSpacing;
    public Single spaceAbove;
    public Single spaceBelow;

    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = WABAPI.MAX_STRING_SIZE)]
    public String fullAttributesString;
    }


    ======> THEN A SIMPLE STATIC CLASS WITH THE METHODS TO CALL FROM THE BRIDGE

    public static class WABAPI
    {
    public const String WinAccessBridgeDll = "windowsaccessbridge.dll";
    public const Int32 MAX_STRING_SIZE = 1024;
    public const Int32 SHORT_STRING_SIZE = 256;

    [DllImport(WinAccessBridgeDll, SetLastError = true, ThrowOnUnmappableChar = true, CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Auto)]
    public extern static unsafe Int32 isJavaWindow(IntPtr hwnd);

    [DllImport(WinAccessBridgeDll, SetLastError = true, ThrowOnUnmappableChar = true, CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Auto)]
    public extern static void Windows_run();

    [DllImport(WinAccessBridgeDll, SetLastError = true, ThrowOnUnmappableChar = true, CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Auto)]
    public extern static unsafe Boolean getAccessibleContextFromHWND(void* window, out Int32 vmID, out IntPtr ac);

    [DllImport(WinAccessBridgeDll, SetLastError = true, ThrowOnUnmappableChar = true, CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Auto)]
    public extern static unsafe Boolean getAccessibleContextInfo(Int32 vmID, IntPtr accessibleContext, IntPtr acInfo);

    [DllImport(WinAccessBridgeDll, SetLastError = true, ThrowOnUnmappableChar = true, CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Auto)]
    public extern static unsafe IntPtr getAccessibleChildFromContext(Int32 vmID, IntPtr ac, Int32 index);

    [DllImport(WinAccessBridgeDll, SetLastError = true, ThrowOnUnmappableChar = true, CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Auto)]
    public extern static unsafe Boolean getAccessibleTextInfo(Int32 vmID, IntPtr AccessibleContext, IntPtr textInfo, Int32 x, Int32 y);

    [DllImport(WinAccessBridgeDll, SetLastError = true, ThrowOnUnmappableChar = true, CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Auto)]
    public extern static unsafe Boolean getAccessibleTextItems(Int32 vmID, IntPtr AccessibleContext, IntPtr textItems, Int32 index);

    [DllImport(WinAccessBridgeDll, SetLastError = true, ThrowOnUnmappableChar = true, CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Auto)]
    public extern static unsafe Boolean getAccessibleTextAttributes(Int32 vmID, IntPtr AccessibleContext, Int32 index, IntPtr attributes);

    [DllImport(WinAccessBridgeDll, SetLastError = true, ThrowOnUnmappableChar = true, CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Auto)]
    public extern static unsafe Boolean getAccessibleContextWithFocus(void* window, out Int32 vmID, out IntPtr ac);
    }


    =========> MY OWN DATA CLASSES TO MAKE THINGS EASIER ON MYSELF

    public class AccessibleTreeItem
    {
    public String name;          // the AccessibleName of the object
    public String description;     // the AccessibleDescription of the object
    public String role;     // localized AccesibleRole string
    public String role_en_US;     // AccesibleRole string in the en_US locale
    public String states;     // localized AccesibleStateSet string (comma separated)
    public String states_en_US; // AccesibleStateSet string in the en_US locale (comma separated)
    public Int32 indexInParent;               // index of object in parent
    public Int32 childrenCount;               // # of children, if any
    public Int32 x;                         // screen coords in pixel
    public Int32 y;                         // "
    public Int32 width;                    // pixel width of object
    public Int32 height;                    // pixel height of object
    public Boolean accessibleComponent;     // flags for various additional
    public Boolean accessibleAction;               // Java Accessibility interfaces
    public Boolean accessibleSelection;          // FALSE if this object doesn't
    public Boolean accessibleText;               // implement the additional interface
    public Boolean accessibleInterfaces;

    public List<AccessibleTreeItem> children;

    public AccessibleTreeItem()
    {
    children = new List<AccessibleTreeItem>();
    }

    public AccessibleTreeItem(AccessibleContextInfo accessibleContextInfo)
    : this()
    {
    this.name = accessibleContextInfo.name;     // the AccessibleName of the object
    this.description = accessibleContextInfo.description;     // the AccessibleDescription of the object
    this.role = accessibleContextInfo.role;     // localized AccesibleRole string
    this.role_en_US = accessibleContextInfo.role_en_US;     // AccesibleRole string in the en_US locale
    this.states = accessibleContextInfo.states;     // localized AccesibleStateSet string (comma separated)
    this.states_en_US = accessibleContextInfo.states_en_US; // AccesibleStateSet string in the en_US locale (comma separated)
    this.indexInParent = accessibleContextInfo.indexInParent;               // index of object in parent
    this.childrenCount = accessibleContextInfo.childrenCount;               // # of children, if any
    this.x = accessibleContextInfo.x;                         // screen coords in pixel
    this.y = accessibleContextInfo.y;                         // "
    this.width = accessibleContextInfo.width;                    // pixel width of object
    this.height = accessibleContextInfo.height;                    // pixel height of object
    this.accessibleComponent = accessibleContextInfo.accessibleComponent;     // flags for various additional
    this.accessibleAction = accessibleContextInfo.accessibleAction;               // Java Accessibility interfaces
    this.accessibleSelection = accessibleContextInfo.accessibleSelection;          // FALSE if this object doesn't
    this.accessibleText = accessibleContextInfo.accessibleText;               // implement the additional interface
    this.accessibleInterfaces = accessibleContextInfo.accessibleInterfaces;
    }

    public Rectangle Bounds
    {
    get
    {
    return new Rectangle(this.x, this.y, this.width, this.height);
    }
    }

    public Int32 SquarePixels
    {
    get { return this.x * this.y; }
    }
    }


    =========> Item Comparer

    internal class ItemComparer : IComparer<AccessibleTreeItem>
    {
    #region IComparer<IReader> Members

    public int Compare(AccessibleTreeItem x, AccessibleTreeItem y)
    {
    if (x.SquarePixels < y.SquarePixels)
    {
    return 1;
    }
    else if (x.SquarePixels > y.SquarePixels)
    {
    return -1;
    }
    else
    {
    return 0;
    }
    }

    #endregion
    }
    #endregion
    }

    =========> Now get the componet tree

    private List<AccessibleTreeItem> screenContents = new List<AccessibleTreeItem>();

    private AccessibleTreeItem InternalGetComponentTree(IntPtr hWnd, out Int32 vmID)
    {
    AccessibleTreeItem accessibleTreeItem = new AccessibleTreeItem();
    vmID = 0;
    if (WABAPI.isJavaWindow(hWnd) == 1)
    {
    unsafe
    {
    IntPtr acPtr;

    if (WABAPI.getAccessibleContextFromHWND(hWnd.ToPointer(), out vmID, out acPtr))
    {

    AccessibleContextInfo ac = new AccessibleContextInfo();
    accessibleTreeItem = GetAccessibleContextInfo(vmID, acPtr, out ac, null); <<<< RECURSION SEED

    return accessibleTreeItem;
    }
    }
    }

    return null;
    }



    private AccessibleTreeItem GetAccessibleContextInfo(Int32 vmID, IntPtr ac, out AccessibleContextInfo acInfo, AccessibleTreeItem parentItem)
    {
    unsafe
    {
    // Allocate global memory space for the size of AccessibleContextInfo and store the address in acPtr
    IntPtr acPtr = Marshal.AllocHGlobal(Marshal.SizeOf(new AccessibleContextInfo()));
    try
    {

    Marshal.StructureToPtr(new AccessibleContextInfo(), acPtr, true);
    if (WABAPI.getAccessibleContextInfo(vmID, ac, acPtr))
    {
    acInfo = (AccessibleContextInfo)Marshal.PtrToStructure(acPtr, typeof(AccessibleContextInfo));
    if (!ReferenceEquals(acInfo, null))
    {
    AccessibleTreeItem newItem = BuildAccessibleTree(acInfo, parentItem);

    if (!ReferenceEquals(newItem, null))
    {
    for (int i = 0; i < acInfo.childrenCount; i++)
    {
    if (acInfo.role_en_US != "unknown" && acInfo.states_en_US.Contains("visible")) // Note the optomization here, I found this get me to an acceptable speed
    {
    AccessibleContextInfo childAc = new AccessibleContextInfo();
    IntPtr childContext = WABAPI.getAccessibleChildFromContext(vmID, ac, i);
    GetAccessibleContextInfo(vmID, childContext, out childAc, newItem);
    }
    }
    }

    return newItem;
    }
    }
    else
    {
    acInfo = new AccessibleContextInfo();
    }
    }
    finally
    {
    if (acPtr != IntPtr.Zero)
    Marshal.FreeHGlobal(acPtr);
    }
    }
    return null;
    }


    private AccessibleTreeItem BuildAccessibleTree(AccessibleContextInfo acInfo, AccessibleTreeItem parentItem)
    {
    if (!ReferenceEquals(acInfo, null))
    {
    AccessibleTreeItem item = new AccessibleTreeItem(acInfo);
    if (!ReferenceEquals(parentItem, null))
    {
    screenContents.Add(item);
    parentItem.children.Add(item);
    }
    return item;
    }
    return null;
    }


    ====================> You would want to create a global mouse hook, and recycle your list after every click ...

    And you getContextInfoAt is now = to:

    screenContents.Sort(new ItemComparer());
    item = screenContents.Find(e => e.Bounds.Contains(controlInfo.Point));

    Hope this gets everyone around the problem listed in this post