lzy/sci-gui-agent-benchmark
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update readme

Browse Source
This commit is contained in:
Jing Hua
2023-11-02 09:44:02 +08:00
parent 4c17a41b0d
commit a3706004d0
1 changed files with 10 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
README.md
View File

@@ -3,16 +3,16 @@
## Setup guide
1. Download OS image
1. Download kubuntu from https://kubuntu.org/getkubuntu/
2. Download ubuntu from https://ubuntu.com/download/desktop
1. If mac OS, use https://cdimage.ubuntu.com/jammy/daily-live/current/jammy-desktop-arm64.iso
3. Download Windows, TODO
4. Download MacOS, TODO
1. Download kubuntu from <https://kubuntu.org/getkubuntu/>
2. Download ubuntu from <https://ubuntu.com/download/desktop>
1. If mac OS, use <https://cdimage.ubuntu.com/jammy/daily-live/current/jammy-desktop-arm64.iso>
3. Download Windows from <https://www.microsoft.com/en-au/software-download/windows10ISO>
4. ~~Download MacOS~~ (Not possible to download legally)
2. Setup virtual machine
1. Create `Host Only Adapter` and add it to the network adapter in the settings
3. Set up bridge for connecting to VM
1. Option 1: Install [xdotool](https://github.com/jordansissel/xdotool) on VM
2. Option 2: Install [mouse]() TODO
2. Option 2: Install [mouse](https://github.com/boppreh/mouse/)
4. Set up SSH server on VM: https://averagelinuxuser.com/ssh-into-virtualbox/
1. `sudo apt install openssh-server`
2. `sudo systemctl enable ssh --now`
@@ -26,11 +26,14 @@
6. Get screenshot
1. `scp user@192.168.7.128:~/screenshot.png screenshot.png`
2. `rm -rf ~/screenshot.png`
7. Set up python and install [mouse](https://github.com/boppreh/mouse/) and [keyboard](https://github.com/jordansissel/xdotool)
## Road map (Proposed)
- [ ] Explore VMWare, and whether it can be connected and control through mouse package
- [ ] Explore Windows and MacOS, whether it can be installed
- [x] Explore Windows and MacOS, whether it can be installed
- MacOS is closed source and cannot be legally installed
- Windows is available legally and can be installed
- [ ] Build gym-like python interface for controlling the VM
- [ ] Recording of actions (mouse movement, click, keyboard) for human to annotate, and we can replay it
- [ ] This part may be conflict with work from [Aran Komatsuzaki](https://twitter.com/arankomatsuzaki) team, a.k.a. [Duck AI](https://duckai.org/)