fffa8f8da6
* Initailize aws support * Add README for the VM server * Refactor OSWorld for supporting more cloud services. * Initialize vmware and aws implementation v1, waiting for verification * Initlize files for azure, gcp and virtualbox support * Debug on the VMware provider * Fix on aws interface mapping * Fix instance type * Refactor * Clean * hk region; debug * Fix lock * Remove print * Remove key_name requirements when allocating aws vm * Clean README --------- Co-authored-by: XinyuanWangCS <xywang626@gmail.com>
35 lines
2.3 KiB
Markdown
35 lines
2.3 KiB
Markdown
# Road Map
|
|
Here we provide a high-level road map for the project. We will update this road map as we make progress.
|
|
If you are interested in contributing to the project, please check the [CONTRIBUTING.md](CONTRIBUTING.md) for more details.
|
|
|
|
## Road Map for Environment Infrastructure
|
|
|
|
- [x] Explore VMWare, and whether it can be connected and controlled through the mouse package
|
|
- [x] Explore Windows and MacOS, whether it can be installed
|
|
- MacOS is closed source and cannot be legally installed
|
|
- Windows is available legally and can be installed
|
|
- [x] Build a gym-like Python interface for controlling the VM
|
|
- [x] Recording of actions (mouse movement, click, keyboard) for humans to annotate, and we can replay it and compress it
|
|
- [x] Build a simple task, e.g. open a browser, open a website, click on a button, and close the browser
|
|
- [x] Set up a pipeline and build agent implementation (zero-shot) for the task
|
|
- [x] Start to design which tasks inside the DesktopENv to focus on, start to wrap up the environment to be public
|
|
- [x] Start to annotate the examples for ~~training~~ and testing
|
|
- [x] Error handling during file passing and file opening, etc.
|
|
- [x] Add accessibility tree from the OS into the observation space
|
|
- [x] Add pre-process and post-process action support for benchmarking setup and evaluation
|
|
- [x] Experiment logging and visualization system
|
|
- [x] Add more tasks, maybe scale to 300 for v1.0.0, and create a dynamic leaderboard
|
|
- [x] Multiprocess support, can enable reinforcement learning to be more efficient
|
|
- [x] Add support for automatic VM download and configuration, enable auto-scaling management
|
|
- [x] VPN setup doc for those who need it
|
|
- [x] Support running on platforms that have nested virtualization, e.g. Google Cloud, AWS, etc.
|
|
- [ ] Prepare for the first release of Windows vm image for the environment
|
|
- [ ] Be able to run without virtual machine platform VMware Pro, e.g. VirtualBox, or other platforms
|
|
|
|
|
|
## Road Map of Annotation Tool
|
|
- [ ] Improve the annotation tool base on DuckTrack, and make it more robust which aligns on accessibility tree
|
|
- [ ] Annotate the steps of doing the task
|
|
- [ ] Crawl all resources we explored from the internet, and make it easy to access
|
|
- [ ] Set up ways for the crowdsourcing/community to contribute new examples
|