diff --git a/desktop_env/README.md b/desktop_env/README.md deleted file mode 100644 index e69de29..0000000 diff --git a/desktop_env/server/README.md b/desktop_env/server/README.md index 6d1dccd..5962dd2 100644 --- a/desktop_env/server/README.md +++ b/desktop_env/server/README.md @@ -41,11 +41,26 @@ Detailed instructions for each of these requirements will be provided in the fol Make a new VM with the Ubuntu 20.04 LTS image. +### How to install Ubuntu Desktop (package: ubuntu-desktop) with GNOME desktop environment on Ubuntu 22.04 system. + +```bash +sudo apt udpate +sudo apt install ubuntu-desktop +sudo systemctl set-default graphical.target +``` + ### Account Credentials Download the iso file from the [Ubuntu website](https://ubuntu.com/download/alternative-downloads) and install it in the VM. -The default username should be `user` and the password should be `password` when you are asked to set up the account. +Using GUI: +The default username should be `user` and the password should be `password` when you are asked to set up the account. Give the user sudo permission. + +Using Command Line: +```bash +sudo adduser user +usermod -aG sudo user +``` ### Installation and Auto-login Setup @@ -76,58 +91,275 @@ sudo systemctl restart gdm3 After setting up automatic login, the system will boot directly into the desktop environment without requiring password input, which enables seamless startup experience for automated testing environments. +### VNC Configuration + +1. Install x11vnc +``` +sudo apt update +sudo apt install x11vnc +``` + +2. Install noVNC +``` +sudo snap install novnc +``` + +3. Create system services for x11vnc and novnc: +- Go to directory cd `/etc/systemd/user/` +- Write a file `novnc.service` with the following content: +``` +[Unit] +Description=noVNC Service +After=x11vnc.service network.target snap.novnc.daemon.service +Wants=x11vnc.service +[Service] +Type=simple +ExecStart=/snap/bin/novnc --vnc localhost:5900 --listen 5910 +Restart=on-failure +RestartSec=3 +Environment=DISPLAY=:0 +Environment=XAUTHORITY=/home/user/.Xauthority +Environment=SNAP_COOKIE=/run/snap.cookie +Environment=SNAP_NAME=novnc +Environment=SNAP_REVISION=current +[Install] +WantedBy=default.target +``` +Write a file `x11vnc.service` with the following content: +``` +[Unit] +Description=X11 VNC Server +After=display-manager.service network.target +Wants=display-manager.service +[Service] +Type=simple +ExecStart=x11vnc -display :0 -rfbport 5900 -forever +User=user +Restart=on-failure +RestartSec=3 +Environment=DISPLAY=:0 +Environment=XAUTHORITY=/home/user/.Xauthority +[Install] +WantedBy=default.target +``` + +4. Enable both services: +``` +systemctl --user daemon-reload +systemctl --user enable novnc.service +systemctl --user enable x11vnc.service +systemctl --user start x11vnc.service +systemctl --user start novnc.service +``` + +5. Allow VNC port: +Expose 5910 port in the firewall and any other security tools you are using. + +6. Access the VNC server: +Connect to the VNC via `http://[Instance IP]:5910/vnc.html` + +### Display Configuration + +1. Install dummy video driver: +``` +sudo apt-get install xserver-xorg-video-dummy +``` + +Go to `/etc/X11/` and create a file named `xorg.conf` with the following content: +``` +Section "ServerLayout" + Identifier "X.org Configured" + Screen 0 "Screen0" 0 0 + InputDevice "Mouse0" "CorePointer" + InputDevice "Keyboard0" "CoreKeyboard" +EndSection + +Section "Files" + ModulePath "/usr/lib/xorg/modules" + FontPath "/usr/share/fonts/X11/misc" + FontPath "/usr/share/fonts/X11/cyrillic" + FontPath "/usr/share/fonts/X11/100dpi/:unscaled" + FontPath "/usr/share/fonts/X11/75dpi/:unscaled" + FontPath "/usr/share/fonts/X11/Type1" + FontPath "/usr/share/fonts/X11/100dpi" + FontPath "/usr/share/fonts/X11/75dpi" + FontPath "built-ins" +EndSection + +Section "Module" + Load "glx" +EndSection + +Section "InputDevice" + Identifier "Keyboard0" + Driver "kbd" +EndSection + +Section "InputDevice" + Identifier "Mouse0" + Driver "mouse" + Option "Protocol" "auto" + Option "Device" "/dev/input/mice" + Option "ZAxisMapping" "4 5 6 7" +EndSection + +Section "Monitor" + Identifier "Monitor0" + VendorName "Monitor Vendor" + ModelName "Monitor Model" + HorizSync 28.0-80.0 + VertRefresh 48.0-75.0 +EndSection + +Section "Device" + ### Available Driver options are:- + ### Values: : integer, : float, : "True"/"False", + ### : "String", : " Hz/kHz/MHz", + ### : "%" + ### [arg]: arg optional + #Option "SWcursor" # [] + #Option "kmsdev" # + #Option "ShadowFB" # [] + #Option "AccelMethod" # + #Option "PageFlip" # [] + #Option "ZaphodHeads" # + #Option "DoubleShadow" # [] + #Option "Atomic" # [] + #Option "VariableRefresh" # [] + #Option "UseGammaLUT" # [] + #Option "AsyncFlipSecondaries" # [] + Identifier "Card0" + Driver "modesetting" + BusID "PCI:0:30:0" + VideoRam 256000 +EndSection + +Section "Screen" + Identifier "Screen0" + Device "Device0" + Monitor "Monitor0" + DefaultDepth 24 + SubSection "Display" + Depth 24 + Modes "1920x1080" + EndSubSection +EndSection +``` + +2. In the same directory as the previous step, go to its sub-directory `xorg.conf.d` , create a file named `10-dummy.conf` with the following content: +``` +Section "Device" + Identifier "DummyDevice" + Driver "dummy" + VideoRam 32768 +EndSection + +Section "Monitor" + Identifier "DummyMonitor" + HorizSync 28.0-80.0 + VertRefresh 48.0-75.0 + Modeline "1920x1080" 172.80 1920 2048 2248 2576 1080 1083 1088 1120 +EndSection + +Section "Screen" + Identifier "DummyScreen" + Device "DummyDevice" + Monitor "DummyMonitor" + DefaultDepth 24 + SubSection "Display" + Depth 24 + Modes "1920x1080" + EndSubSection +EndSection +``` + +3. Reload the display manager: +``` +sudo systemctl restart display-manager +``` ### Set up the OSWorld server service in VM +Upload the OSWorld server to the home directory (/home/user) of user (via scp or git clone). 1. Copy the `main.py` and `pyxcursor.py` and to the `/home/user-name` where the `user-name` is your username of the ubuntu, here we make it `user` as default. If you customize the path of placing these files in this step, you should change the parameters in the service file we will mention later accordingly. 2. First please set up the environment: - ```shell - pip install -r requirements.txt - ``` - if you customize the environment in this step, you should change the parameters in the service file we will mention later accordingly. +```shell +sudo apt install python +pip install -r requirements.txt +sudo apt-get install python3-tk python3-dev +sudo apt install gnome-screenshot +sudo apt install wmctrl +sudo apt install ffmpeg +sudo apt install socat +sudo apt install xclip +``` -3. Copy the `osworld_server.service` to the systemd configuration directory at `/etc/systemd/system/`: - ```shell - sudo cp osworld_server.service /etc/systemd/system/ - ``` +If you encounter an error about python not being found, run: +``` +sudo ln -s /usr/bin/python3 /usr/bin/python +``` - Reload the systemd daemon to recognize the new service: - ```shell - sudo systemctl daemon-reload - ``` +if you customize the environment in this step, you should change the parameters in the service file we will mention later accordingly. - Enable the service to start on boot: - ```shell - sudo systemctl enable osworld_server.service - ``` +3. Due to some configuration issues, you need to modify the `osworld_server.service` file: +1) In our released version, the X server is set to :1, but the default X server is actually :0. You need to modify the `osworld_server.service` file to change the `DISPLAY` variable from `:1` to `:0`. +Change the following line: +``` +Environment="DISPLAY=:1" +``` +to +``` +Environment="DISPLAY=:0" +``` +2) Need to add environment variables to enable DBUS to change wallpaper. +Change the following line: +``` +Environment="DISPLAY=:0" +``` +to +``` +Environment="DISPLAY=:0;DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus" +``` - Start the service: - ```shell - sudo systemctl start osworld_server.service - ``` +4. Copy the `osworld_server.service` to the systemd configuration directory at `/etc/systemd/system/`: +```shell +sudo cp osworld_server.service /etc/systemd/system/ +``` - Verify the service is running correctly: - ```shell - sudo systemctl status osworld_server.service - ``` +Reload the systemd daemon to recognize the new service: +```shell +sudo systemctl daemon-reload +``` - You should see output indicating the service is active and running. If there are errors, review the logs with `journalctl -xe` for further troubleshooting. +Enable the service to start on boot: +```shell +sudo systemctl enable osworld_server.service +``` - If you need to make adjustments to the service configuration, you can edit the `/etc/systemd/system/osworld_server.service` file: - ```shell - sudo nano /etc/systemd/system/osworld_server.service - ``` - - After making changes, reload the daemon and restart the service: - ```shell - sudo systemctl - ``` +Start the service: +```shell +sudo systemctl start osworld_server.service +``` - - - - +Verify the service is running correctly: +```shell +sudo systemctl status osworld_server.service +``` + +You should see output indicating the service is active and running. If there are errors, review the logs with `journalctl -xe` for further troubleshooting. + +If you need to make adjustments to the service configuration, you can edit the `/etc/systemd/system/osworld_server.service` file: +```shell +sudo nano /etc/systemd/system/osworld_server.service +``` + +After making changes, reload the daemon and restart the service: +```shell +sudo systemctl daemon-reload +sudo systemctl enable osworld_server.service +sudo systemctl start osworld_server.service +``` ### Accessibility Tree Support @@ -159,9 +391,6 @@ You can run the following command to check if **Xorg** is being used: echo $XDG_SESSION_TYPE ``` - - - ### System Service Management (Optional) The automatic software update service can interfere with benchmark examples. To disable this service, you can refer to the https://www.makeuseof.com/disable-automatic-updates-in-ubuntu/ for the solution. @@ -227,29 +456,11 @@ To extract relevant internal information and configurations from the VS Code env # Set "Always save as" to "Microsoft Word 2007-2013 XML (.docx)" # Repeat similar steps for Calc (.xlsx) and Impress (.pptx) ``` -2. GIMP Startup Settings: -```bash -# Open GIMP -# Go to Edit -> Preferences -> Interface -# Under "Window Management": -# Uncheck "Show tips on startup" -# Under "File Saving": -# Uncheck "Show warning when saving images that will result in information loss" -``` -3. Chrome password requirement removal: +2. Chrome password requirement removal: Chrome requests a password input when first opened after system startup, which can interfere with our experiments. Here's how to disable this feature: ```bash -Using Terminal -# Remove the default keyring -rm -rf ~/.local/share/keyrings/* - -# Create empty keyring -echo -n "" >> ~/.local/share/keyrings/login.keyring - -# Restart Chrome after applying changes - -# Or just prevent Chrome from using keyring +# Prevent Chrome from using keyring mkdir -p ~/.local/share/keyrings touch ~/.local/share/keyrings/login.keyring ``` @@ -257,9 +468,21 @@ touch ~/.local/share/keyrings/login.keyring Or you can use any ways to disable the keyring service, which will prevent Chrome from requesting a password input. - ### Network Configuration +#### Firewall Configuration + +In OSWorld, we need the following ports to be open: +``` +server_port = 5000 +chromium_port = 9222 +vnc_port = 8006 +vlc_port = 8080 +novnc_port = 5910 +``` + +Please open the corresponding ports in the firewall and any other security tools you are using. + #### socat Installation Ensure `socat` is installed to enable port forwarding. diff --git a/desktop_env/server/requirements.txt b/desktop_env/server/requirements.txt index cfee21b..8594af2 100644 --- a/desktop_env/server/requirements.txt +++ b/desktop_env/server/requirements.txt @@ -8,3 +8,4 @@ numpy lxml pygame pywinauto +pyastpi2