Commit Graph

212 Commits

Author SHA1 Message Date
Timothyxxx
20b1d950a0 FIx corner cases (val connection in chrome when using playwright, and action parsing for agent, and accessibility tree xml handling) 2024-01-16 22:00:01 +08:00
Timothyxxx
186bf2e97c Implement heuristic cutting on the accessibility tree to get the important nodes; Finish accessibility tree text agent 2024-01-16 16:43:32 +08:00
Timothyxxx
48a86d36cf Minor updates 2024-01-16 12:15:21 +08:00
Timothyxxx
6336a31419 Merge remote-tracking branch 'origin/main' 2024-01-16 11:58:35 +08:00
Timothyxxx
8efa692951 Add raw accessibility-tree based prompting method (but the tokens are too large); Minor fix some small bugs 2024-01-16 11:58:23 +08:00
tsuky_chen
91823b1bcd Merge branch 'main' of https://github.com/ztjhz/DesktopEnv 2024-01-16 01:07:54 +08:00
tsuky_chen
9fe3a5db3b update libreoffice impress eval 2024-01-16 01:07:40 +08:00
Timothyxxx
28d8c0c528 Merge remote-tracking branch 'origin/main' 2024-01-15 21:58:45 +08:00
Timothyxxx
493b719821 Add gemini agent implementation; Add missed requirements; Minor fix some small bugs 2024-01-15 21:58:33 +08:00
David Chang
5dc633393f Merge branch 'zdy' 2024-01-15 17:08:38 +08:00
David Chang
4c87e11864 ver Jan15thv3
scheduled closing?
2024-01-15 16:46:34 +08:00
David Chang
00922923ee ver Jan15thv2
thunderbird example w.r.t. unified folder
2024-01-15 15:56:01 +08:00
Timothyxxx
c68796e842 Fix minor bugs 2024-01-15 13:52:13 +08:00
Timothyxxx
1141232d80 Merge remote-tracking branch 'origin/main'
# Conflicts:
#	desktop_env/controllers/setup.py
2024-01-15 13:51:11 +08:00
Timothyxxx
24169a65d0 Accomplish the exp scripts v1; Add video recording and trajectory recording of desktop agent; Fix minor bugs 2024-01-15 13:49:48 +08:00
David Chang
a9e9ebbc80 ver Jan15th
merged zdy into main
2024-01-15 12:13:44 +08:00
David Chang
fc289a3427 Merge branch 'main' into zdy 2024-01-15 12:12:05 +08:00
David Chang
b9d8e6c631 ver Jan15th
attachment task of thunderbird
2024-01-15 11:49:43 +08:00
tsuky_chen
7ffb5de551 Merge branch 'main' of https://github.com/ztjhz/DesktopEnv 2024-01-15 01:32:36 +08:00
tsuky_chen
f44995cb92 update libreoffice impress example 2024-01-15 01:32:22 +08:00
rhythmcao
69b0514f99 fix error in pyautogui.typewrite() 2024-01-14 23:53:31 +08:00
Timothyxxx
f153a4c253 Add 'WAIT', 'FAIL', 'DONE' to the action space; Debug basic prompting-based GPT-4 and Gemini agents; Initialize experiments script; 2024-01-14 23:36:19 +08:00
David Chang
59fdd9f1a2 ver Jan14th
setup method for Thunderbird composing tasks
2024-01-14 23:16:54 +08:00
Timothyxxx
d52b692ee5 Finish loading the vscode examples v1; Improve on the infra: Add accessibility tree into the observation; Add activate window function, etc 2024-01-14 18:30:49 +08:00
Timothyxxx
2228f346a9 Fix minor bugs caused from merging in setupcontroller; Initialize vscode example loading 2024-01-14 00:51:26 +08:00
Siheng Zhao
347160a35f update vsc 2024-01-13 23:20:36 +08:00
Timothyxxx
0242657b86 Minor fix on gimp examples name 2024-01-13 23:13:11 +08:00
Timothyxxx
57a41a279c Resolve conflicts 2024-01-13 22:58:20 +08:00
Timothyxxx
a1c3e4c294 Finish Chrome example loading v1 2024-01-13 22:56:50 +08:00
Siheng Zhao
f274193265 Merge branch 'main' of github.com:ztjhz/DesktopEnv 2024-01-13 18:14:31 +08:00
Siheng Zhao
105fd35683 implement action replay for vscode and gimp evaluation 2024-01-13 17:53:13 +08:00
David Chang
2f6974a59a Merge branch 'zdy' 2024-01-13 17:17:51 +08:00
David Chang
005b054a0b Merge branch 'main' into zdy 2024-01-13 17:15:32 +08:00
tsuky_chen
136b52c876 eval gimp compare pics 2024-01-13 01:49:46 +08:00
tsuky_chen
1a5aa39611 Merge branch 'main' of https://github.com/ztjhz/DesktopEnv 2024-01-13 01:46:02 +08:00
tsuky_chen
a841f27a9d update gimp examples eval 2024-01-13 01:45:46 +08:00
David Chang
d4192d3d9c ver Jan12thv3
debugged
2024-01-13 00:06:11 +08:00
David Chang
e08df57129 ver Jan12thv2
sqlite3 metric
2024-01-12 23:07:00 +08:00
Timothyxxx
bc88ee0c41 Minor fix of the logic of vm ip get 2024-01-12 21:18:59 +08:00
rhythmcao
d4116458ff 1. fix quote and \ characters in execute_command ; 2. add terminal output text as extra observation ; 3. move get_vm_*() to reset() 2024-01-12 18:09:05 +08:00
Timothyxxx
186df65683 Merge remote-tracking branch 'origin/main'
# Conflicts:
#	desktop_env/controllers/setup.py
#	desktop_env/evaluators/metrics/utils.py
2024-01-12 17:30:15 +08:00
Timothyxxx
5a93a32958 Update on Chrome examples; Refactor on logic of controlling 2024-01-12 17:24:47 +08:00
Siheng Zhao
d6f694da1c update GIMP examples and eval 2024-01-12 16:36:02 +08:00
Siheng Zhao
b3109bf856 update vscode eval extension: eval-0.0.1.vsix 2024-01-12 16:14:41 +08:00
tsuky_chen
85f64e884c 2 init gimp examples 2024-01-12 16:10:10 +08:00
tsuky_chen
e79235f568 2 init gimp example 2024-01-12 16:07:55 +08:00
David Chang
5160619783 ver Jan12th
quickly fixed two thunderbird examples
2024-01-12 12:19:23 +08:00
David Chang
5b0b1ff9f2 Merge branch 'zdy' 2024-01-11 23:04:21 +08:00
David Chang
127a101994 Merge branch 'main' into zdy 2024-01-11 23:02:00 +08:00
David Chang
3c04872dcf ver Jan11thv2
a new Thunderbird example w.r.t. email filter
2024-01-11 22:43:38 +08:00