DesktopEnv: An Environment towards Human-like Computer Task Mastery
Setup guide
For members of the team
- Download OS image
- Download kubuntu from https://kubuntu.org/getkubuntu/
- Download ubuntu from https://ubuntu.com/download/desktop
- Download Windows from https://www.microsoft.com/en-au/software-download/windows10ISO
Download MacOS(Not possible to download legally)
- Setup virtual machine
- Create
Host Only Adapterand add it to the network adapter in the settings
- Create
- Set up bridge for connecting to VM
- Set up SSH server on VM | Guide
- Install screenshot tool (in vm)
sudo apt install imagemagick-6.q16hdriDISPLAY=:0 import -window root screenshot.png
- Get screenshot
scp user@192.168.7.128:~/screenshot.png screenshot.pngrm -rf ~/screenshot.png
- Set up python and install mouse and keyboard
For users of the environment
todo
Road map (Proposed)
- Explore VMWare, and whether it can be connected and control through mouse package
- Explore Windows and MacOS, whether it can be installed
- MacOS is closed source and cannot be legally installed
- Windows is available legally and can be installed
- Build gym-like python interface for controlling the VM
- Make configuration much easier from code perspective
- README
- Make it easier to install the dependencies
- Make it easier to install the VM
- Make it easier to set up the VM
- Recording of actions (mouse movement, click, keyboard) for human to annotate, and we can replay it
- This part may be conflict with work from Aran Komatsuzaki team, a.k.a. Duck AI
- Build a simple task, e.g. open a browser, open a website, click on a button, and close the browser
- Set up a pipeline and build agents implementation (zero-shot) for the task
- Start to design on which tasks inside the DesktopENv to focus on, start to wrap up the environment to be public
- Start to annotate the examples for training and testing