Commit Graph

76 Commits

Author SHA1 Message Date
FredWuCZ
e9dbc3c374 Update docs 2024-10-22 20:42:27 +08:00
FredWuCZ
b46b6f0649 Clean up 2024-10-18 18:47:10 +08:00
FredWuCZ
9e86f160e7 Capture cursor on Windows 2024-10-18 18:44:53 +08:00
Timothyxxx
50a887ebb6 Fix the get terminal output error caused by namespace parameter changes 2024-08-23 16:12:27 +08:00
Junli Wang
9ee006226e Improve accessibility functionality (#64)
* Initial commit

* fetching accessibility of dock
2024-08-02 17:39:21 +08:00
HappySix
a961d2276d Improve efficiency of getting accessibility tree (#57)
* Initial commit

* Accelarate a11y tree acquisition

* Clean code

* Leave todos for Windows and macOS

* Prepare for support of win and macos

* Prepare for support of win and macos

* Minor fix

* add preliminary support for macos

* fix subtle bugs

* Clean the windows a11y tree getter code

* Clean the windows a11y tree getter code

* Intermediate version

* Intermediate version

* Update

* adding support for macos

* Delete dummy

* Delete dummy

* add bounding box for pruning

* Delete dummy

* FIX: enable a11y tree fetching on Windows

* Move the requirement depency item place

* FIX: remove "jsonify" from return value of get a11y tree

* FIX: change print into logger, decompose functions in _create_pywinauto_node

* Update

* remove redundant nodes on macos

* remove reliance of pywinauto

* Clean

* Fix bugs for pywinauto a11y_tree acquisition

* FEAT: only fetch active windows, skip repeated nodes

* CHORE: clean code, add comments on time-consuming part, add connection logger.info

* FEAT: 1. add attrs including class_name, id, count... 2. use multithread to accelerate

* FIX: add code of getting writable properties

* Clean, update the max_depth for macOS

* FIX: get all active windows

* Accelarate child nodes in macOS

* FEAT: get all active windows, add timing when testing

* CHORE: remove print

* Clean and finalize

* Clean and finalize

---------

Co-authored-by: Timothyxxx <384084775@qq.com>
Co-authored-by: Junli Wang <ltnsxdxd@gmail.com>
Co-authored-by: YangJL2003 <yangjl22@mails.tsinghua.edu.cn>
2024-07-30 14:07:29 +08:00
Danyang Zhang
b2d407544d ver Jun28th (#51)
reduced time to acquire AT from at-spi from ~5 s to ~3.5 s
  * refined acquisition of States and Attributes
  * canceled acquisition of Component when node is not showing and
    visible according to the document
    (https://gnome.pages.gitlab.gnome.org/at-spi2-core/libatspi/iface.Component.html)
  * pruned windowcoord and parentcoord, merged size and screencoord into
    one at-spi invocation
  * canceled acquisition of Document
  * refined acquisition of RoleName
2024-06-28 15:17:47 +08:00
Tianbao Xie
fffa8f8da6 Refactoring VMware Integration and Implementing AWS Support (#44)
* Initailize aws support

* Add README for the VM server

* Refactor OSWorld for supporting more cloud services.

* Initialize vmware and aws implementation v1, waiting for verification

* Initlize files for azure, gcp and virtualbox support

* Debug on the VMware provider

* Fix on aws interface mapping

* Fix instance type

* Refactor

* Clean

* hk region; debug

* Fix lock

* Remove print

* Remove key_name requirements when allocating aws vm

* Clean README

---------

Co-authored-by: XinyuanWangCS <xywang626@gmail.com>
2024-06-15 20:52:29 +08:00
Timothyxxx
10adaf6f4d Merge remote-tracking branch 'origin/main' 2024-03-25 18:03:11 +08:00
Timothyxxx
172123ab2c Support downsampling; Fix bugs in windows a11y tree; Add a11y_tree trim 2024-03-25 18:02:48 +08:00
Fangyu Lei
0aab0e3745 Update requirements.txt 2024-03-25 15:11:25 +08:00
David Chang
15e01e7ccc ver Mar20thv2
fixed bugs in server/main.py (_create_pywinauto_node and
  get_screen_size)
finished migration of a few task configs to Windows
fixed bug in python.py
2024-03-20 22:22:57 +08:00
David Chang
6149061621 ver Mar20th
fixed a bug in _create_pywinauto_node
2024-03-20 14:25:09 +08:00
Timothyxxx
eeae1442cd Add execute timeout to server; Fix error examples 2024-03-18 20:42:57 +08:00
David Chang
e166106b6a ver Mar15th
added an option to keep buttons without text information but with an
image for SoM setting
2024-03-15 22:46:14 +08:00
David Chang
f6b96165e2 Merge branch 'zdy' 2024-03-14 22:40:27 +08:00
David Chang
587a89fa7f ver Mar13thv2
added a new service file with replacable arguments
2024-03-13 19:42:49 +08:00
Timothyxxx
c2aa009ed8 Update server script, baseline and running script 2024-03-13 15:04:19 +08:00
David Chang
3a0ab526c8 ver Mar13th
service file for server/main.py
2024-03-13 12:25:46 +08:00
David Chang
f08fa4912c ver Mar10th
changed AT element filtering
2024-03-10 18:03:02 +08:00
David Chang
96e2c6ee34 ver Feb20th
fixed server/main.py
fixed several sheetcopilot tasks
fixed several calc metrics, including float difference of sheet_data and
conditional formatting recognition
2024-02-20 17:11:25 +08:00
rhythmcao
3184f091a8 fix error in long video recording in server/main.py 2024-02-02 07:24:26 +08:00
David Chang
13cd053b1a Merge branch 'zdy' 2024-01-30 21:14:57 +08:00
David Chang
14dbc708a4 ver Jan30thv2
debugged on windows platform with new _create_pywinauto_node function
migrated example task from calc to excel
2024-01-30 21:09:53 +08:00
David Chang
da306376da ver Jan30th
updated function to get AT on Windows
2024-01-30 20:06:58 +08:00
Timothyxxx
d5d9fc56de Fix minor bugs of get_terminal output caused by a11y tree depth 2024-01-30 18:48:00 +08:00
BlankCheng
80e08a2ff8 Expand user directory on server /launch and /execute' 2024-01-29 21:52:27 +08:00
BlankCheng
7d2d8c855e Merge main 2024-01-29 21:51:26 +08:00
BlankCheng
284d6fb379 Add human operation time log 2024-01-29 21:42:16 +08:00
David Chang
297a080284 ver Jan27thv2
updated get_accessibility_tree and added a series of try-catch
2024-01-27 21:29:20 +08:00
David Chang
b7e3e1f7d0 ver Jan26thv2
fixed bugs by incomplete merging
2024-01-26 23:40:20 +08:00
David Chang
773c5ed40c ver Jan26thv4
updated linearized_accessibility_tree to add a column of "text"
removed replacement chars like uFFFC in thunderbird
2024-01-26 23:29:09 +08:00
David Chang
78c7b702b3 ver Jan26th
fixed the too long time consuming of massy AT nodes
added special handling for libreoffice calc
2024-01-26 12:30:41 +08:00
rhythmcao
f194fb8d75 add multi_apps; update chrome utilities 2024-01-25 13:53:19 +08:00
Timothyxxx
303c89411e Fix bugs of a11y tree when the tag is empty 2024-01-23 14:37:54 +08:00
Timothyxxx
7e61ab7e53 Minor fix to support "~/" like part in path parameter 2024-01-22 22:55:15 +08:00
David Chang
21314346c5 ver Jan18thv4
added a comment about accessing AT on windows
2024-01-18 22:22:51 +08:00
David Chang
00deae465a ver Jan18thv3
updated get_accessibility_tree in server/main.py to give place for other
  os-es and desktops
2024-01-18 21:40:49 +08:00
David Chang
ffc4c32bac ver Jan17th
updated the existing task configs
2024-01-17 17:27:08 +08:00
David Chang
82798bff6f ver Jan15thv4
fixed errors in server/README
2024-01-15 18:27:29 +08:00
Timothyxxx
c68796e842 Fix minor bugs 2024-01-15 13:52:13 +08:00
Timothyxxx
1141232d80 Merge remote-tracking branch 'origin/main'
# Conflicts:
#	desktop_env/controllers/setup.py
2024-01-15 13:51:11 +08:00
Timothyxxx
24169a65d0 Accomplish the exp scripts v1; Add video recording and trajectory recording of desktop agent; Fix minor bugs 2024-01-15 13:49:48 +08:00
David Chang
fc289a3427 Merge branch 'main' into zdy 2024-01-15 12:12:05 +08:00
David Chang
b9d8e6c631 ver Jan15th
attachment task of thunderbird
2024-01-15 11:49:43 +08:00
David Chang
59fdd9f1a2 ver Jan14th
setup method for Thunderbird composing tasks
2024-01-14 23:16:54 +08:00
Timothyxxx
d52b692ee5 Finish loading the vscode examples v1; Improve on the infra: Add accessibility tree into the observation; Add activate window function, etc 2024-01-14 18:30:49 +08:00
rhythmcao
d4116458ff 1. fix quote and \ characters in execute_command ; 2. add terminal output text as extra observation ; 3. move get_vm_*() to reset() 2024-01-12 18:09:05 +08:00
David Chang
127a101994 Merge branch 'main' into zdy 2024-01-11 23:02:00 +08:00
David Chang
27eaf2f5d5 ver Jan11th
finally set up a simple task, or which should be simple
2024-01-11 20:03:33 +08:00