From d7aa3047f2fc5a772a40c699ef87ab1ee7701705 Mon Sep 17 00:00:00 2001 From: Tianbao Xie <47296835+Timothyxxx@users.noreply.github.com> Date: Tue, 2 Apr 2024 20:11:04 +0800 Subject: [PATCH] Update README.md --- README.md | 31 ++++++++++++++----------------- 1 file changed, 14 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 8bf660f..0181495 100644 --- a/README.md +++ b/README.md @@ -5,22 +5,18 @@ Paper

-[![OSWorld](https://os-world.github.io/static/videos/main.mp4)](https://os-world.github.io/static/videos/main.mp4) - -![Overview]() - ## Updates -- 2024-03-28: We released our [paper](), [environment and benchmark](), and [project page](https://os-world.github.io/). Check it out! +- 2024-04-04: We released our [paper](), [environment and benchmark](https://github.com/xlang-ai/OSWorld), and [project page](https://os-world.github.io/). Check it out! ## Install ### Non-virtualized platform Suppose you are on a system that has not yet been virtualized, meaning you are not on an AWS, Azure, or k8s virtualized environment. -Otherwise, refer to the [virtualized platform](https://github.com/xlang-ai/OSWorld?tab=readme-ov-file##virtualized-platform) part. +Otherwise, refer to the [virtualized platform](https://github.com/xlang-ai/OSWorld?tab=readme-ov-file#virtualized-platform) part. 1. Install [VMware Work Station Pro](https://www.vmware.com/products/workstation-pro/workstation-pro-evaluation.html) (for Apple Chips, it should be [VMware Fusion](https://www.vmware.com/go/getfusion)) and configure `vmrun` command, and verify successful installation by: ```bash vmrun -T ws list ``` -If the installation along with the environment variable set are successful, you will see the message showing the current running virtual machines. +If the installation along with the environment variable set is successful, you will see the message showing the current running virtual machines. 2. Install the environment package, and download the examples and the virtual machine image. For x86_64 CPU Linux or Windows, you can install the environment package and download the examples and the virtual machine image by running the following commands: @@ -50,9 +46,12 @@ Run the following minimal example to interact with the environment: from desktop_env.envs.desktop_env import DesktopEnv example = { - "id": "94d95f96-9699-4208-98ba-3c3119edf9c2", - "instruction": "I want to install Spotify on my current system. Could you please help me?", - "config": [{"type": "execute", "parameters": {"command": ["python","-c","import pyautogui; import time; pyautogui.click(960, 540); time.sleep(0.5);"]}}], "evaluator": {"func": "check_include_exclude", "result": {"type": "vm_command_line","command": "which spotify"}, "expected": {"type": "rule","rules": {"include": ["spotify"], "exclude": ["not found"]}}} + "id": "94d95f96-9699-4208-98ba-3c3119edf9c2", + "instruction": "I want to install Spotify on my current system. Could you please help me?", + "config": [{"type": "execute", "parameters": { + "command": ["python", "-c", "import pyautogui; import time; pyautogui.click(960, 540); time.sleep(0.5);"]}}], + "evaluator": {"func": "check_include_exclude", "result": {"type": "vm_command_line", "command": "which spotify"}, + "expected": {"type": "rule", "rules": {"include": ["spotify"], "exclude": ["not found"]}}} } env = DesktopEnv( path_to_vm="Ubuntu/Ubuntu.vmx", @@ -65,17 +64,15 @@ obs, reward, done, info = env.step("pyautogui.rightClick()") ## Run Benchmark ### Run the Baseline Agent -If you want to run the baseline agent we use in our paper, you can run the following command as an example: +If you want to run the baseline agent we use in our paper, you can run the following command to run under the GPT-4V pure-screenshot setting as an example: ```bash - +python run.py --path_to_vm Ubuntu/Ubuntu.vmx --headless --observation_type screenshot --model gpt-4-vision-preview ``` ### Run Evaluation of Your Agent -Please first read through the [agent interface](https://github.com/xlang-ai/OSWorld/mm_agents/README.md) and the [environment interface](https://github.com/xlang-ai/OSWorld/desktop_env/README.md). -And implement the agent interface correctly and import you customized one in the `run.py` file. -Then, you can run the following command to evaluate your agent: - - +Please first read through the [agent interface](https://github.com/xlang-ai/OSWorld/blob/main/mm_agents/README.md) and the [environment interface](https://github.com/xlang-ai/OSWorld/blob/main/desktop_env/README.md). +Implement the agent interface correctly and import your customized one in the `run.py` file. +Then, you can run a similar command as the previous section to run the benchmark on your agent. ## Citation If you find this environment useful, please consider citing our work: