diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..306e6da --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,30 @@ +# Road Map +Here we provide a high-level road map for the project. We will update this road map as we make progress. +If you are interested in contributing to the project, please check the [CONTRIBUTING.md](CONTRIBUTING.md) for more details. + +## Road Map for Environment Infrastructure + +- [x] Explore VMWare, and whether it can be connected and control through mouse package +- [x] Explore Windows and MacOS, whether it can be installed + - MacOS is closed source and cannot be legally installed + - Windows is available legally and can be installed +- [x] Build gym-like python interface for controlling the VM +- [x] Recording of actions (mouse movement, click, keyboard) for humans to annotate, and we can replay it and compress it +- [x] Build a simple task, e.g. open a browser, open a website, click on a button, and close the browser +- [x] Set up a pipeline and build agents implementation (zero-shot) for the task +- [x] Start to design on which tasks inside the DesktopENv to focus on, start to wrap up the environment to be public +- [x] Start to annotate the examples for ~~training~~ and testing +- [x] Error handling during file passing and file opening, etc. +- [x] Add accessibility tree from the OS into the observation space +- [x] Add pre-process and post-process action support for benchmarking setup and evaluation +- [x] Experiment logging and visualization system +- [x] Add more tasks, maybe scale to 300 for v1.0.0, and create a dynamic leaderboard +- [x] Multiprocess support, this can enable the reinforcement learning to be more efficient +- [ ] Support running on platform that have nested virtualization, e.g. Google Cloud, AWS, etc. + + +## Road Map of Annotation Tool +- [ ] Improve the annotation tool base on DuckTrack, make it more robust which align on accessibility tree +- [ ] Annotate the steps of doing the task +- [ ] Crawl all resources we explored from the internet, and make it easy to access +- [ ] Set up ways for community to contribute new examples