Download OS image
1. Download kubuntu from https://kubuntu.org/getkubuntu/
2. Download ubuntu from https://ubuntu.com/download/desktop
  1. If mac OS, use https://cdimage.ubuntu.com/jammy/daily-live/current/jammy-desktop-arm64.iso
3. Download Windows from https://www.microsoft.com/en-au/software-download/windows10ISO
4. ~~Download MacOS~~ (Not possible to download legally)
Setup virtual machine
1. Create Host Only Adapter and add it to the network adapter in the settings
Set up bridge for connecting to VM
1. Option 1: Install xdotool on VM
2. Option 2: Install mouse
Set up SSH server on VM: https://averagelinuxuser.com/ssh-into-virtualbox/
1. sudo apt install openssh-server
2. sudo systemctl enable ssh --now
3. sudo ufw disable (disable firewall - safe for local network, otherwise sudo ufw allow ssh)
4. ip a - find ip address
5. ssh username@<ip_address>
6. On host, run ssh-copy-id <username>@<ip_address>
Install screenshot tool (in vm)
1. sudo apt install imagemagick-6.q16hdri
2. DISPLAY=:0 import -window root screenshot.png
Get screenshot
1. scp user@192.168.7.128:~/screenshot.png screenshot.png
2. rm -rf ~/screenshot.png
Set up python and install mouse and keyboard

Road map (Proposed)

Explore VMWare, and whether it can be connected and control through mouse package
Explore Windows and MacOS, whether it can be installed
- MacOS is closed source and cannot be legally installed
- Windows is available legally and can be installed
Build gym-like python interface for controlling the VM
Recording of actions (mouse movement, click, keyboard) for human to annotate, and we can replay it
- This part may be conflict with work from Aran Komatsuzaki team, a.k.a. Duck AI
Build a simple task, e.g. open a browser, open a website, click on a button, and close the browser
Set up a pipeline and build agents implementation (zero-shot) for the task
Start to design on which tasks inside the DesktopENv to focus on, start to wrap up the environment to be public
Start to annotate the examples for training and testing