Clean Code; Refactor README

README.md

@@ -1,10 +1,4 @@
-# OSWorld: Open-Ended Tasks in Real Computer Environments
-
-<p align="center">
-    <img src="desktop_env/assets/icon.jpg" alt="Logo" width="80px">
-    <br>
-    <b>SLOGAN</b>
-</p>
+# OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
 
 <p align="center">
 <a href="">Website</a> •
@@ -14,74 +8,65 @@
 ![Overview]()
 
 ## Updates
-- 2024-03-01: We released our [paper](), [environment code](), [dataset](), and [project page](). Check it out!
+- 2024-03-28: We released our [paper](), [environment and benchmark](), and [project page](https://os-world.github.io/). Check it out!
 
 ## Install
-1. Install VMWare and configure `vmrun` command:
-Please refer to [guidance](https://docs.google.com/document/d/1KBdeZwmZs2Vi_Wsnngb3Wf1-RiwMMpXTftwMqP2Ztak/edit#heading=h.uh0x0tkl7fuw)
+1. Install VMWare and configure `vmrun` command, and verify by:
+```bash
+vmrun -T ws list
+```
 
 2. Install the environment package, download the examples and the virtual machine image.
-For x86_64 Linux or Windows, you can install the environment package and download the examples and the virtual machine image by running the following commands:
+For x86_64 CPU Linux or Windows, you can install the environment package and download the examples and the virtual machine image by running the following commands:
+Remove the `nogui` parameter if you want to see what happens in the virtual machine.
 ```bash
-git clone https://github.com/xlang-ai/DesktopEnv
-cd DesktopEnv
+git clone https://github.com/xlang-ai/OSWorld
+cd OSWorld
 pip install -r requirements.txt
-gdown https://drive.google.com/drive/folders/1HX5gcf7UeyR-2UmiA15Q9U-
-Wr6E6Gio8 -O Ubuntu --folder
+gdown https://drive.google.com/drive/folders/1HX5gcf7UeyR-2UmiA15Q9U-Wr6E6Gio8 -O Ubuntu --folder
 vmrun -T ws start "Ubuntu/Ubuntu.vmx" nogui
 vmrun -T ws snapshot "Ubuntu/Ubuntu.vmx" "init_state"
 ```
 
+For Apple-chip macOS, you should install the specially prepared virtual machine image by running the following commands:
+```bash
+gdown https://drive.google.com/drive/folders/xxx -O Ubuntu --folder
+vmrun -T fusion start "Ubuntu/Ubuntu.vmx"
+vmrun -T fusion snapshot "Ubuntu/Ubuntu.vmx" "init_state"
+```
+
 ## Quick Start
 Run the following minimal example to interact with the environment:
 ```python
-import json
 from desktop_env.envs.desktop_env import DesktopEnv
 
-with open("evaluation_examples/examples/gimp/f723c744-e62c-4ae6-98d1-750d3cd7d79d.json", "r", encoding="utf-8") as f:
-    example = json.load(f)
-
+example = {
+  "id": "94d95f96-9699-4208-98ba-3c3119edf9c2",
+  "instruction": "I want to install Spotify on my current system. Could you please help me?",
+  "config": [{"type": "execute", "parameters": {"command": ["python","-c","import pyautogui; import time; pyautogui.click(960, 540); time.sleep(0.5);"]}}], "evaluator": {"func": "check_include_exclude", "result": {"type": "vm_command_line","command": "which spotify"}, "expected": {"type": "rule","rules": {"include": ["spotify"], "exclude": ["not found"]}}}
+}
 env = DesktopEnv(
-    path_to_vm=r"path_to_vm",
-    action_space="computer_13",
+    path_to_vm="Ubuntu/Ubuntu.vmx",
+    action_space="pyautogui",
     task_config=example
 )
-observation = env.reset()
-
-observation, reward, done, info = env.step({"action_type": "CLICK", "parameters": {"button": "right", "num_clicks": 1}})
+obs = env.reset()
+obs, reward, done, info = env.step("pyautogui.rightClick()")
 ```
 
-## Annotation Tool Usage
-We provide an annotation tool to help you annotate the examples.
+## Run Benchmark
+### Run the Baseline Agent
+If you want to run the baseline agent we use in our paper, you can run the following command as an example:
+```bash
 
-## Agent Usage
-We provide a simple agent to interact with the environment. You can use it as a starting point to build your own agent.
+```
 
-## Road map of infra (Proposed)
+### Run Evaluation of Your Agent
+Please first read through the [agent interface](https://github.com/xlang-ai/OSWorld/mm_agents/README.md) and the [environment interface](https://github.com/xlang-ai/OSWorld/desktop_env/README.md). 
+And implement the agent interface correctly and import you customized one in the `run.py` file.
+Then, you can run the following command to evaluate your agent:
 
-- [x] Explore VMWare, and whether it can be connected and control through mouse package
-- [x] Explore Windows and MacOS, whether it can be installed
-  - MacOS is closed source and cannot be legally installed
-  - Windows is available legally and can be installed
-- [x] Build gym-like python interface for controlling the VM
-- [x] Recording of actions (mouse movement, click, keyboard) for humans to annotate, and we can replay it and compress it
-- [x] Build a simple task, e.g. open a browser, open a website, click on a button, and close the browser
-- [x] Set up a pipeline and build agents implementation (zero-shot) for the task
-- [x] Start to design on which tasks inside the DesktopENv to focus on, start to wrap up the environment to be public
-- [x] Start to annotate the examples for ~~training~~ and testing
-- [x] Error handling during file passing and file opening, etc.
-- [x] Add accessibility tree from the OS into the observation space
-- [x] Add pre-process and post-process action support for benchmarking setup and evaluation
-- [ ] Multiprocess support, this can enable the reinforcement learning to be more efficient
-- [ ] Experiment logging and visualization system
-- [ ] Add more tasks, maybe scale to 300 for v1.0.0, and create a dynamic leaderboard
 
-## Road map of benchmark, tools and resources (Proposed)
-- [ ] Improve the annotation tool base on DuckTrack, make it more robust which align on accessibility tree
-- [ ] Annotate the steps of doing the task
-- [ ] Build a website for the project
-- [ ] Crawl all resources we explored from the internet, and make it easy to access
-- [ ] Set up ways for community to contribute new examples
 
 ## Citation
 If you find this environment useful, please consider citing our work: