|
|
b75f6bf341
|
feat: 增强任务步骤注入与a11y状态表达，提升树形交互稳定性
- 打通 metadata.steps 传递链路，将任务步骤注入 agent 预测上下文
- 优化 a11y tree 线性化输出：使用中心坐标并新增 states 列（expanded/collapsed/selected 等）
- 放宽可保留节点条件，保留无文本输入类控件（edit/textfield/searchbox 等）
- 强化输出约束：单轮仅允许动作代码或 WAIT/DONE/FAIL，禁止动作与 DONE 同轮返回
- 补充 avogadro 示例步骤：展开 aromatics 并选择 benzene.cjson
|
2026-02-26 18:56:53 +08:00 |
|
|
|
07e66490dd
|
feat: 增强科研软件的 a11y tree 支持
- 扩展 heuristic_retrieve.py 白名单以覆盖科研软件 GUI 框架:
- 新增 prefix 规则: sunawt (Java Swing), qt5q/qt6q (Qt), ovito, pymol,
contentspanel, wx (wxWidgets), afx (MFC), thunderrt (VB6)
- 新增 endswith 规则: edit, widget, box, dialog, view, frame, menuitem,
menubar, toolbar, tabitem, treeitem, window
- 新增 Qt 控件和 Win32 控件的精确匹配
- 在 agent.py 中添加原始 a11y tree 的调试日志
- 修复 run.py 中 agent 初始化缺少 platform='windows' 的问题
- 添加 NO_PROXY 绕过本地/VM IP (兼容 Clash 全局代理)
- lib_run_single.py 中应用启动等待时间增加到 15 秒
- 新增 test_each_domain_a11y_tree.json (每个域一个任务用于 a11y 验证)
|
2026-02-26 15:04:28 +08:00 |
|
Tianbao Xie
|
a156f8a3d6
|
Modify the namespace of a11y tree (#62)
|
2024-07-25 20:20:34 +08:00 |
|
Timothyxxx
|
9c75df5dce
|
Clean code; Refactor environment to pass screenshot content instead of path
|
2024-04-13 23:34:01 +08:00 |
|
Timothyxxx
|
172123ab2c
|
Support downsampling; Fix bugs in windows a11y tree; Add a11y_tree trim
|
2024-03-25 18:02:48 +08:00 |
|
Timothyxxx
|
3ce7636abd
|
Fix one multi_app example; remove some broken examples; Support downsampling
|
2024-03-21 22:05:16 +08:00 |
|
David Chang
|
4df088e2ad
|
ver Mar19thv2
supplemented at info back for som setting
|
2024-03-19 18:41:55 +08:00 |
|
David Chang
|
b5d58b8ecd
|
ver Mar19th
a tiny fix
|
2024-03-19 17:43:34 +08:00 |
|
David Chang
|
13e6040148
|
ver Mar15thv2
fixed a bug
|
2024-03-15 22:50:22 +08:00 |
|
David Chang
|
e166106b6a
|
ver Mar15th
added an option to keep buttons without text information but with an
image for SoM setting
|
2024-03-15 22:46:14 +08:00 |
|
David Chang
|
0c9c2f214a
|
ver Mar11thv2
minor adjustment
|
2024-03-11 22:45:16 +08:00 |
|
David Chang
|
e95e8e55ea
|
ver Mar11th
updated filter_nodes
|
2024-03-11 12:33:47 +08:00 |
|
David Chang
|
f08fa4912c
|
ver Mar10th
changed AT element filtering
|
2024-03-10 18:03:02 +08:00 |
|
David Chang
|
d8a497a417
|
ver Jan29th
updated the position of SoM marks
|
2024-01-29 21:49:53 +08:00 |
|
David Chang
|
eef5158663
|
ver Jan26thv3
fixed bug caused by an empty node.text
remove nodes whose name and text are all empty
|
2024-01-26 23:49:15 +08:00 |
|
Timothyxxx
|
f88331416c
|
Refactor baselines code implementations
|
2024-01-20 18:55:21 +08:00 |
|
Timothyxxx
|
09f3e776ae
|
Initialize all baselines: screenshot, a11y tree, both, SoM, SeeAct
|
2024-01-20 00:13:46 +08:00 |
|
Timothyxxx
|
20b1d950a0
|
FIx corner cases (val connection in chrome when using playwright, and action parsing for agent, and accessibility tree xml handling)
|
2024-01-16 22:00:01 +08:00 |
|
Timothyxxx
|
186bf2e97c
|
Implement heuristic cutting on the accessibility tree to get the important nodes; Finish accessibility tree text agent
|
2024-01-16 16:43:32 +08:00 |
|
