lzy/sci-gui-agent-benchmark
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add FAQ

Browse Source
This commit is contained in:
Tianbao Xie
2024-04-09 11:11:20 +08:00
committed by GitHub
parent 76b821426e
commit 62fd0094b9
1 changed files with 12 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
README.md
View File

@@ -121,6 +121,18 @@ Please start by reading through the [agent interface](https://github.com/xlang-a
Correctly implement the agent interface and import your customized version in the `run.py` file. Correctly implement the agent interface and import your customized version in the `run.py` file.
Afterward, you can execute a command similar to the one in the previous section to run the benchmark on your agent. Afterward, you can execute a command similar to the one in the previous section to run the benchmark on your agent.
## FAQ
### What are the running times and costs under different settings?
| Setting | Expected Time* | Budget Cost (Full Test Set/Small Test Set) |
| ------------------------------ | -------------- | ------------------------------------------ |
| GPT-4V (screenshot) | 10h | $100 ($10) |
| Gemini-ProV (screenshot) | 15h | $0 ($0) |
| Claude-3 Opus (screenshot) | 15h | $150 ($15) |
| GPT-4V (a11y tree, SoM, etc.) | 30h | $500 ($50) |
\*No environment parallelism. Calculated in April 2024.
## Citation ## Citation
If you find this environment useful, please consider citing our work: If you find this environment useful, please consider citing our work:
``` ```