23 lines
999 B
Markdown
23 lines
999 B
Markdown
# DesktopEnv: An Environment towards Human-like Computer Task Mastery
|
|
|
|
## Setup guide
|
|
|
|
### For members of the team
|
|
todo
|
|
|
|
### For users of the environment
|
|
todo
|
|
|
|
## Road map (Proposed)
|
|
|
|
- [x] Explore VMWare, and whether it can be connected and control through mouse package
|
|
- [x] Explore Windows and MacOS, whether it can be installed
|
|
- MacOS is closed source and cannot be legally installed
|
|
- Windows is available legally and can be installed
|
|
- [x] Build gym-like python interface for controlling the VM
|
|
- [x] Recording of actions (mouse movement, click, keyboard) for humans to annotate, and we can replay it and compress it
|
|
- [x] Build a simple task, e.g. open a browser, open a website, click on a button, and close the browser
|
|
- [x] Set up a pipeline and build agents implementation (zero-shot) for the task
|
|
- [x] Start to design on which tasks inside the DesktopENv to focus on, start to wrap up the environment to be public
|
|
- [x] Start to annotate the examples for ~~training~~ and testing
|