Add todos and roadmap

.gitignore

@@ -1,4 +1,163 @@
-*.iso
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
 .venv
-__pycache__
+env/
-*.png
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# PyCharm
+**/.idea/**/*
+
+# Mac OS
+.DS_Store
+
+# data
+**/data/**/*
+!**/utils/data/**/*
+
+# tmp files
+**/tmp/**/*
+api_key.py
+tmp.*
+
+## Server logging
+**/.logging/**/*
+
+# DB cache
+**/.db_cache/**/*
+
+**/debugging/**/*
+
+# embedding repo
+instructor-embedding
+
+# plugin cache
+**/static/**/*
+
+# frontend cache
+frontend/node_modules/
+frontend/.next/
+frontend/.idea

README.md

@@ -1,12 +1,32 @@
-# Setup guide
+#  DesktopEnv: A Learning Environment for Human-like Computer Task Mastery
 
-1. Download kubuntu from https://kubuntu.org/getkubuntu/
+## Setup guide
+
+1. Download OS image
+   1. Download kubuntu from https://kubuntu.org/getkubuntu/
+   2. Download ubuntu from https://ubuntu.com/download/desktop
+      1. If mac OS, use https://cdimage.ubuntu.com/jammy/daily-live/current/jammy-desktop-arm64.iso
+   3. Download Windows, TODO
+   4. Download MacOS, TODO
 2. Setup virtual machine
    1. Create `Host Only Adapter` and add it to the network adapter in the settings
-3. Install [xdotool](https://github.com/jordansissel/xdotool) on VM
+3. Set up bridge for connecting to VM
+   1. Option 1: Install [xdotool](https://github.com/jordansissel/xdotool) on VM
+   2. Option 2: Install [mouse]() TODO
 4. Set up SSH server on VM: https://averagelinuxuser.com/ssh-into-virtualbox/
    1. `sudo apt install openssh-server`
    2. `sudo systemctl enable ssh --now`
    3. `sudo ufw disable` (disable firewall - safe for local network, otherwise `sudo ufw allow ssh`)
    4. `ip a` - find ip address
    5. ssh username@<ip_address>
+
+## Road map (Proposed)
+- [ ] Explore VMWare, and whether it can be connected and control through mouse package
+- [ ] Explore Windows and MacOS, whether it can be installed
+- [ ] Build gym-like python interface for controlling the VM
+- [ ] Recording of actions (mouse movement, click, keyboard) for human to annotate, and we can replay it
+  - [ ] This part may be conflict with work from [Aran Komatsuzaki](https://twitter.com/arankomatsuzaki) team, a.k.a. [Duck AI](https://duckai.org/)
+- [ ] Build a simple task, e.g. open a browser, open a website, click on a button, and close the browser
+- [ ] Set up a pipeline and build agents implementation (zero-shot) for the task
+- [ ] Start to design on which tasks inside the DesktopENv to focus on, start to wrap up the environment to be public
+- [ ] Start to annotate the examples for training and testing