DesktopEnv: An Environment towards Human-like Computer Task Mastery

Setup guide

For members of the team

todo

For users of the environment

todo

Road map of infra (Proposed)

• Explore VMWare, and whether it can be connected and control through mouse package
• Explore Windows and MacOS, whether it can be installed
  • MacOS is closed source and cannot be legally installed
  • Windows is available legally and can be installed
• Build gym-like python interface for controlling the VM
• Recording of actions (mouse movement, click, keyboard) for humans to annotate, and we can replay it and compress it
• Build a simple task, e.g. open a browser, open a website, click on a button, and close the browser
• Set up a pipeline and build agents implementation (zero-shot) for the task
• Start to design on which tasks inside the DesktopENv to focus on, start to wrap up the environment to be public
• Start to annotate the examples for training and testing
• Error handling during file passing and file opening, etc.
• Add accessibility tree from the OS into the observation space
• Add pre-process and post-process action support for benchmarking setup and evaluation
• Multiprocess support, this can enable the reinforcement learning to be more efficient