Update docs

README.md

@@ -32,6 +32,7 @@
 
 
 ## 📢 Updates
+- 2024-10-22: We supported Docker for hosting virtual machines on virtualized platforms. Check below for detailed instructions!
 - 2024-06-15: We refactor the code of environment part to decompose VMware Integration, and start to support other platforms such as VitualBox, AWS, Azure, etc. Hold tight!
 - 2024-04-11: We released our [paper](https://arxiv.org/abs/2404.07972), [environment and benchmark](https://github.com/xlang-ai/OSWorld), and [project page](https://os-world.github.io/). Check it out!
 
@@ -76,6 +77,9 @@ See [AWS_GUIDELINE](https://github.com/xlang-ai/OSWorld/blob/main/desktop_env/pr
 #### On your Azure
 We have finished the support for Azure but not yet fully tested.
 
+### On Docker
+See [DOCKER_GUIDELINE](https://github.com/xlang-ai/OSWorld/blob/main/desktop_env/providers/docker/DOCKER_GUIDELINE.md) for using Docker to host virtual machines.
+
 #### Others
 We are working on supporting more 👷. Please hold tight!
 

desktop_env/providers/docker/DOCKER_GUIDELINE.md

@@ -0,0 +1,27 @@
+# Configuration of Docker
+
+---
+
+Welcome to the Docker VM Management documentation.
+
+## Prerequisite: Check if your machine supports KVM
+
+We recommend running the VM with KVM support. To check if your hosting platform supports KVM, run
+
+```
+egrep -c '(vmx|svm)' /proc/cpuinfo
+```
+
+on Linux. If the return value is greater than zero, the processor should be able to support KVM.
+
+> **Note**: macOS hosts generally do not support KVM.
+
+## Install Docker
+
+If your hosting platform supports graphical user interface (GUI), you may refer to [Install Docker Desktop on Linux](https://docs.docker.com/desktop/install/linux/) or [Install Docker Desktop on Windows](https://docs.docker.com/desktop/install/windows-install/) based on your OS.  Otherwise, you may [Install Docker Engine](https://docs.docker.com/engine/install/).
+
+## Running Experiments
+
+Add the following arguments when initializing `DesktopEnv`: 
+- `provider`: `docker`
+- `os_type`: `Ubuntu` or `Windows`, depending on the OS of the VM

desktop_env/server/main.py

@@ -180,16 +180,19 @@ def capture_screen_with_cursor():
 
             return (cursor, hotspot)
 
-        cursor, (hotspotx, hotspoty) = get_cursor()
-
         ratio = ctypes.windll.shcore.GetScaleFactorForDevice(0) / 100
 
         img = ImageGrab.grab(bbox=None, include_layered_windows=True)
 
-        pos_win = win32gui.GetCursorPos()
+        try:
-        pos = (round(pos_win[0]*ratio - hotspotx), round(pos_win[1]*ratio - hotspoty))
+            cursor, (hotspotx, hotspoty) = get_cursor()
 
-        img.paste(cursor, pos, cursor)
+            pos_win = win32gui.GetCursorPos()
+            pos = (round(pos_win[0]*ratio - hotspotx), round(pos_win[1]*ratio - hotspoty))
+
+            img.paste(cursor, pos, cursor)
+        except:
+            pass
 
         img.save(file_path)
     elif user_platform == "Linux":