sci-gui-agent-benchmark

Files

David Chang 96e2c6ee34 ver Feb20th

fixed server/main.py
fixed several sheetcopilot tasks
fixed several calc metrics, including float difference of sheet_data and
conditional formatting recognition

2024-02-20 17:11:25 +08:00

main.py

ver Feb20th

2024-02-20 17:11:25 +08:00

pyxcursor.py

Fix the cursor logo placement for ubuntu

2023-12-28 15:37:18 +08:00

README.md

ver Jan15thv4

2024-01-15 18:27:29 +08:00

requirements.txt

ver Jan15th

2024-01-15 11:49:43 +08:00

README.md

About the Converted Accessibility Tree

For several applications like Firefox or Thunderbird, you should first enable

gsettings set org.gnome.desktop.interface toolkit-accessibility true

to see their accessibility tree.

Example of AT

An example of a node:

<section xmlns:attr="uri:deskat:attributes.at-spi.gnome.org" attr:class="subject" st:enabled="true" cp:screencoord="(1525, 169)", cp:windowcoord="(342, 162)", cp:size="(327, 21)">
    歡迎使用新的 Outlook.com 帳戶
</section>

An example of a tree:

<desktop-frame ...>
    <application name="Thunderbird" ...>
        ... <!-- nodes of windows -->
    </application>
    ...
</desktop-frame>

Useful attributes

name - shows the name of application, title of window, or name of some component
attr:class - somewhat the same role as class in HTML
attr:id - somewhat the same role as id in HTML
cp:screencoord - absolute coordinator on the screen
cp:windowcoord - relative coordinator in the window
cp:size - the size

Also several states like st:enabled and st:visible can be indicated. A full state list is available at https://gitlab.gnome.org/GNOME/pyatspi2/-/blob/master/pyatspi/state.py?ref_type=heads.

How to use it in evaluation

See example thunderbird/12086550-11c0-466b-b367-1d9e75b3910e.json and function check_accessibility_tree in metrics/general.py. You can use CSS selector or XPath to reference a target nodes. You can also check its text contents.

An example of a CSS selector:

application[name=Thunderbird] page-tab-list[attr|id="tabmail-tabs"]>page-tab[name="About Profiles"]

This selector will select the page tab of profile manager in Thunderbird (if open).

For usage of CSS selector: https://www.w3.org/TR/selectors-3/. For usage of XPath: https://www.w3.org/TR/xpath-31/.

Manual check

You can use accerciser to check the accessibility tree on GNOME VM.

sudo apt install accerciser

Additional Installation

Activating the window manager control requires the installation of wmctrl:

sudo apt install wmctrl

To enable recording in the virtual machine, you need to install ffmpeg:

sudo apt install ffmpeg