Commit Graph

69 Commits

Author SHA1 Message Date
Tianbao Xie
4e11eafd1d Robust Evaluation, Blocking File Open, Grader Sensitivity, and LibreOffice Writer Fixes (#217)
* Refactor evaluator structure in LibreOffice Writer example JSON to support multiple expected and result files, enhancing evaluation flexibility.

* Update instance type to t3.large and add VNC access URL logging for allocated VMs, enhancing remote access capabilities.

* Update instance type to t3.large and add VNC access URL logging for allocated VMs, enhancing remote access capabilities.

* Update time format in get_vm_file function to include hours, minutes, and seconds for more precise file naming with time suffix.

* More delay for 936321ce-5236-426a-9a20-e0e3c5dc536f; support one more potential solutions.

* Enhance SetupController with configurable retry limit and improved error handling for file opening requests. Introduce new function to compare unique training records, and update logging for better debugging. Adjust JSON examples for evaluation to support multiple expected and result files.

* Clean debug code

---------

Co-authored-by: yuanmengqi <yuanmengqi@mail.ustc.edu.cn>
2025-06-16 21:37:19 +08:00
MillanK
983283a86a patch: minor bug fixes for evaluator and task configurations, documentation update (#121)
* fix: /cursor_position api return format fix

* chore: update README.md to remove deprecated command

* fix: add base score for evaluators and minor bug fixes

* fix: add base score for setup configurations

---------

Co-authored-by: Jiaqi Deng <jiaqideng@Jiaqis-MacBook-Pro.local>
2025-01-18 22:25:18 +08:00
Tianbao Xie
9d6879d334 Fix chromium command for M-chip MacBook device 2024-11-29 20:00:01 +08:00
Pierre Carrier
1754f195b0 fix(server): run on non-Windows python (#94) 2024-11-06 15:18:13 +08:00
FredWuCZ
e9dbc3c374 Update docs 2024-10-22 20:42:27 +08:00
FredWuCZ
b46b6f0649 Clean up 2024-10-18 18:47:10 +08:00
FredWuCZ
9e86f160e7 Capture cursor on Windows 2024-10-18 18:44:53 +08:00
Timothyxxx
50a887ebb6 Fix the get terminal output error caused by namespace parameter changes 2024-08-23 16:12:27 +08:00
Junli Wang
9ee006226e Improve accessibility functionality (#64)
* Initial commit

* fetching accessibility of dock
2024-08-02 17:39:21 +08:00
HappySix
a961d2276d Improve efficiency of getting accessibility tree (#57)
* Initial commit

* Accelarate a11y tree acquisition

* Clean code

* Leave todos for Windows and macOS

* Prepare for support of win and macos

* Prepare for support of win and macos

* Minor fix

* add preliminary support for macos

* fix subtle bugs

* Clean the windows a11y tree getter code

* Clean the windows a11y tree getter code

* Intermediate version

* Intermediate version

* Update

* adding support for macos

* Delete dummy

* Delete dummy

* add bounding box for pruning

* Delete dummy

* FIX: enable a11y tree fetching on Windows

* Move the requirement depency item place

* FIX: remove "jsonify" from return value of get a11y tree

* FIX: change print into logger, decompose functions in _create_pywinauto_node

* Update

* remove redundant nodes on macos

* remove reliance of pywinauto

* Clean

* Fix bugs for pywinauto a11y_tree acquisition

* FEAT: only fetch active windows, skip repeated nodes

* CHORE: clean code, add comments on time-consuming part, add connection logger.info

* FEAT: 1. add attrs including class_name, id, count... 2. use multithread to accelerate

* FIX: add code of getting writable properties

* Clean, update the max_depth for macOS

* FIX: get all active windows

* Accelarate child nodes in macOS

* FEAT: get all active windows, add timing when testing

* CHORE: remove print

* Clean and finalize

* Clean and finalize

---------

Co-authored-by: Timothyxxx <384084775@qq.com>
Co-authored-by: Junli Wang <ltnsxdxd@gmail.com>
Co-authored-by: YangJL2003 <yangjl22@mails.tsinghua.edu.cn>
2024-07-30 14:07:29 +08:00
Danyang Zhang
b2d407544d ver Jun28th (#51)
reduced time to acquire AT from at-spi from ~5 s to ~3.5 s
  * refined acquisition of States and Attributes
  * canceled acquisition of Component when node is not showing and
    visible according to the document
    (https://gnome.pages.gitlab.gnome.org/at-spi2-core/libatspi/iface.Component.html)
  * pruned windowcoord and parentcoord, merged size and screencoord into
    one at-spi invocation
  * canceled acquisition of Document
  * refined acquisition of RoleName
2024-06-28 15:17:47 +08:00
Timothyxxx
172123ab2c Support downsampling; Fix bugs in windows a11y tree; Add a11y_tree trim 2024-03-25 18:02:48 +08:00
David Chang
15e01e7ccc ver Mar20thv2
fixed bugs in server/main.py (_create_pywinauto_node and
  get_screen_size)
finished migration of a few task configs to Windows
fixed bug in python.py
2024-03-20 22:22:57 +08:00
David Chang
6149061621 ver Mar20th
fixed a bug in _create_pywinauto_node
2024-03-20 14:25:09 +08:00
Timothyxxx
eeae1442cd Add execute timeout to server; Fix error examples 2024-03-18 20:42:57 +08:00
David Chang
e166106b6a ver Mar15th
added an option to keep buttons without text information but with an
image for SoM setting
2024-03-15 22:46:14 +08:00
Timothyxxx
c2aa009ed8 Update server script, baseline and running script 2024-03-13 15:04:19 +08:00
David Chang
f08fa4912c ver Mar10th
changed AT element filtering
2024-03-10 18:03:02 +08:00
David Chang
96e2c6ee34 ver Feb20th
fixed server/main.py
fixed several sheetcopilot tasks
fixed several calc metrics, including float difference of sheet_data and
conditional formatting recognition
2024-02-20 17:11:25 +08:00
rhythmcao
3184f091a8 fix error in long video recording in server/main.py 2024-02-02 07:24:26 +08:00
David Chang
13cd053b1a Merge branch 'zdy' 2024-01-30 21:14:57 +08:00
David Chang
14dbc708a4 ver Jan30thv2
debugged on windows platform with new _create_pywinauto_node function
migrated example task from calc to excel
2024-01-30 21:09:53 +08:00
David Chang
da306376da ver Jan30th
updated function to get AT on Windows
2024-01-30 20:06:58 +08:00
Timothyxxx
d5d9fc56de Fix minor bugs of get_terminal output caused by a11y tree depth 2024-01-30 18:48:00 +08:00
BlankCheng
80e08a2ff8 Expand user directory on server /launch and /execute' 2024-01-29 21:52:27 +08:00
BlankCheng
7d2d8c855e Merge main 2024-01-29 21:51:26 +08:00
BlankCheng
284d6fb379 Add human operation time log 2024-01-29 21:42:16 +08:00
David Chang
297a080284 ver Jan27thv2
updated get_accessibility_tree and added a series of try-catch
2024-01-27 21:29:20 +08:00
David Chang
b7e3e1f7d0 ver Jan26thv2
fixed bugs by incomplete merging
2024-01-26 23:40:20 +08:00
David Chang
773c5ed40c ver Jan26thv4
updated linearized_accessibility_tree to add a column of "text"
removed replacement chars like uFFFC in thunderbird
2024-01-26 23:29:09 +08:00
David Chang
78c7b702b3 ver Jan26th
fixed the too long time consuming of massy AT nodes
added special handling for libreoffice calc
2024-01-26 12:30:41 +08:00
rhythmcao
f194fb8d75 add multi_apps; update chrome utilities 2024-01-25 13:53:19 +08:00
Timothyxxx
303c89411e Fix bugs of a11y tree when the tag is empty 2024-01-23 14:37:54 +08:00
Timothyxxx
7e61ab7e53 Minor fix to support "~/" like part in path parameter 2024-01-22 22:55:15 +08:00
David Chang
21314346c5 ver Jan18thv4
added a comment about accessing AT on windows
2024-01-18 22:22:51 +08:00
David Chang
00deae465a ver Jan18thv3
updated get_accessibility_tree in server/main.py to give place for other
  os-es and desktops
2024-01-18 21:40:49 +08:00
David Chang
ffc4c32bac ver Jan17th
updated the existing task configs
2024-01-17 17:27:08 +08:00
Timothyxxx
c68796e842 Fix minor bugs 2024-01-15 13:52:13 +08:00
Timothyxxx
1141232d80 Merge remote-tracking branch 'origin/main'
# Conflicts:
#	desktop_env/controllers/setup.py
2024-01-15 13:51:11 +08:00
Timothyxxx
24169a65d0 Accomplish the exp scripts v1; Add video recording and trajectory recording of desktop agent; Fix minor bugs 2024-01-15 13:49:48 +08:00
David Chang
fc289a3427 Merge branch 'main' into zdy 2024-01-15 12:12:05 +08:00
David Chang
59fdd9f1a2 ver Jan14th
setup method for Thunderbird composing tasks
2024-01-14 23:16:54 +08:00
Timothyxxx
d52b692ee5 Finish loading the vscode examples v1; Improve on the infra: Add accessibility tree into the observation; Add activate window function, etc 2024-01-14 18:30:49 +08:00
rhythmcao
d4116458ff 1. fix quote and \ characters in execute_command ; 2. add terminal output text as extra observation ; 3. move get_vm_*() to reset() 2024-01-12 18:09:05 +08:00
David Chang
127a101994 Merge branch 'main' into zdy 2024-01-11 23:02:00 +08:00
David Chang
27eaf2f5d5 ver Jan11th
finally set up a simple task, or which should be simple
2024-01-11 20:03:33 +08:00
Timothyxxx
287876affc Merge remote-tracking branch 'origin/main'
# Conflicts:
#	desktop_env/evaluators/getters/__init__.py
#	desktop_env/evaluators/metrics/__init__.py
#	requirements.txt
2024-01-10 23:20:49 +08:00
Timothyxxx
49ece15ac3 VLC v1 finished, improve on instructions, improve on infra 2024-01-10 23:18:30 +08:00
David Chang
cebae4b183 Merge branch 'main' into zdy 2024-01-10 22:16:25 +08:00
David Chang
cf5d480f44 ver Jan10th
new Thunderbird task config
2024-01-10 17:36:59 +08:00