lzy/sci-gui-agent-benchmark
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update desktop environment server configuration and documentation

Browse Source 
- Enhance README.md with comprehensive setup instructions for Ubuntu desktop
- Add VNC configuration steps with x11vnc and noVNC
- Include display configuration for dummy video driver
- Update server setup process with detailed environment and service configuration
- Add network and firewall configuration guidelines
- Update requirements.txt with pyastpi2 dependency
- Remove empty README.md in desktop_env directory
This commit is contained in:
Timothyxxx
2025-02-15 23:40:27 +08:00
parent 339a13e1d5
commit 5f6497afda
3 changed files with 287 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files

0
desktop_env/README.md
View File

349
desktop_env/server/README.md
View File

@@ -41,11 +41,26 @@ Detailed instructions for each of these requirements will be provided in the fol
Make a new VM with the Ubuntu 20.04 LTS image. Make a new VM with the Ubuntu 20.04 LTS image.
### How to install Ubuntu Desktop (package: ubuntu-desktop) with GNOME desktop environment on Ubuntu 22.04 system.
```bash
sudo apt udpate
sudo apt install ubuntu-desktop
sudo systemctl set-default graphical.target
```
### Account Credentials ### Account Credentials
Download the iso file from the [Ubuntu website](https://ubuntu.com/download/alternative-downloads) and install it in the VM. Download the iso file from the [Ubuntu website](https://ubuntu.com/download/alternative-downloads) and install it in the VM.
The default username should be `user` and the password should be `password` when you are asked to set up the account.
Using GUI:
The default username should be `user` and the password should be `password` when you are asked to set up the account. Give the user sudo permission.
Using Command Line:
```bash
sudo adduser user
usermod -aG sudo user
```
### Installation and Auto-login Setup ### Installation and Auto-login Setup
@@ -76,58 +91,275 @@ sudo systemctl restart gdm3
After setting up automatic login, the system will boot directly into the desktop environment without requiring password input, which enables seamless startup experience for automated testing environments. After setting up automatic login, the system will boot directly into the desktop environment without requiring password input, which enables seamless startup experience for automated testing environments.
### VNC Configuration
1. Install x11vnc
```
sudo apt update
sudo apt install x11vnc
```
2. Install noVNC
```
sudo snap install novnc
```
3. Create system services for x11vnc and novnc:
- Go to directory cd `/etc/systemd/user/`
- Write a file `novnc.service` with the following content:
```
[Unit]
Description=noVNC Service
After=x11vnc.service network.target snap.novnc.daemon.service
Wants=x11vnc.service
[Service]
Type=simple
ExecStart=/snap/bin/novnc --vnc localhost:5900 --listen 5910
Restart=on-failure
RestartSec=3
Environment=DISPLAY=:0
Environment=XAUTHORITY=/home/user/.Xauthority
Environment=SNAP_COOKIE=/run/snap.cookie
Environment=SNAP_NAME=novnc
Environment=SNAP_REVISION=current
[Install]
WantedBy=default.target
```
Write a file `x11vnc.service` with the following content:
```
[Unit]
Description=X11 VNC Server
After=display-manager.service network.target
Wants=display-manager.service
[Service]
Type=simple
ExecStart=x11vnc -display :0 -rfbport 5900 -forever
User=user
Restart=on-failure
RestartSec=3
Environment=DISPLAY=:0
Environment=XAUTHORITY=/home/user/.Xauthority
[Install]
WantedBy=default.target
```
4. Enable both services:
```
systemctl --user daemon-reload
systemctl --user enable novnc.service
systemctl --user enable x11vnc.service
systemctl --user start x11vnc.service
systemctl --user start novnc.service
```
5. Allow VNC port:
Expose 5910 port in the firewall and any other security tools you are using.
6. Access the VNC server:
Connect to the VNC via `http://[Instance IP]:5910/vnc.html`
### Display Configuration
1. Install dummy video driver:
```
sudo apt-get install xserver-xorg-video-dummy
```
Go to `/etc/X11/` and create a file named `xorg.conf` with the following content:
```
Section "ServerLayout"
Identifier "X.org Configured"
Screen 0 "Screen0" 0 0
InputDevice "Mouse0" "CorePointer"
InputDevice "Keyboard0" "CoreKeyboard"
EndSection
Section "Files"
ModulePath "/usr/lib/xorg/modules"
FontPath "/usr/share/fonts/X11/misc"
FontPath "/usr/share/fonts/X11/cyrillic"
FontPath "/usr/share/fonts/X11/100dpi/:unscaled"
FontPath "/usr/share/fonts/X11/75dpi/:unscaled"
FontPath "/usr/share/fonts/X11/Type1"
FontPath "/usr/share/fonts/X11/100dpi"
FontPath "/usr/share/fonts/X11/75dpi"
FontPath "built-ins"
EndSection
Section "Module"
Load "glx"
EndSection
Section "InputDevice"
Identifier "Keyboard0"
Driver "kbd"
EndSection
Section "InputDevice"
Identifier "Mouse0"
Driver "mouse"
Option "Protocol" "auto"
Option "Device" "/dev/input/mice"
Option "ZAxisMapping" "4 5 6 7"
EndSection
Section "Monitor"
Identifier "Monitor0"
VendorName "Monitor Vendor"
ModelName "Monitor Model"
HorizSync 28.0-80.0
VertRefresh 48.0-75.0
EndSection
Section "Device"
### Available Driver options are:-
### Values: <i>: integer, <f>: float, <bool>: "True"/"False",
### <string>: "String", <freq>: "<f> Hz/kHz/MHz",
### <percent>: "<f>%"
### [arg]: arg optional
#Option "SWcursor" # [<bool>]
#Option "kmsdev" # <str>
#Option "ShadowFB" # [<bool>]
#Option "AccelMethod" # <str>
#Option "PageFlip" # [<bool>]
#Option "ZaphodHeads" # <str>
#Option "DoubleShadow" # [<bool>]
#Option "Atomic" # [<bool>]
#Option "VariableRefresh" # [<bool>]
#Option "UseGammaLUT" # [<bool>]
#Option "AsyncFlipSecondaries" # [<bool>]
Identifier "Card0"
Driver "modesetting"
BusID "PCI:0:30:0"
VideoRam 256000
EndSection
Section "Screen"
Identifier "Screen0"
Device "Device0"
Monitor "Monitor0"
DefaultDepth 24
SubSection "Display"
Depth 24
Modes "1920x1080"
EndSubSection
EndSection
```
2. In the same directory as the previous step, go to its sub-directory `xorg.conf.d` , create a file named `10-dummy.conf` with the following content:
```
Section "Device"
Identifier "DummyDevice"
Driver "dummy"
VideoRam 32768
EndSection
Section "Monitor"
Identifier "DummyMonitor"
HorizSync 28.0-80.0
VertRefresh 48.0-75.0
Modeline "1920x1080" 172.80 1920 2048 2248 2576 1080 1083 1088 1120
EndSection
Section "Screen"
Identifier "DummyScreen"
Device "DummyDevice"
Monitor "DummyMonitor"
DefaultDepth 24
SubSection "Display"
Depth 24
Modes "1920x1080"
EndSubSection
EndSection
```
3. Reload the display manager:
```
sudo systemctl restart display-manager
```
### Set up the OSWorld server service in VM ### Set up the OSWorld server service in VM
Upload the OSWorld server to the home directory (/home/user) of user (via scp or git clone).
1. Copy the `main.py` and `pyxcursor.py` and to the `/home/user-name` where the `user-name` is your username of the ubuntu, here we make it `user` as default. If you customize the path of placing these files in this step, you should change the parameters in the service file we will mention later accordingly. 1. Copy the `main.py` and `pyxcursor.py` and to the `/home/user-name` where the `user-name` is your username of the ubuntu, here we make it `user` as default. If you customize the path of placing these files in this step, you should change the parameters in the service file we will mention later accordingly.
2. First please set up the environment: 2. First please set up the environment:
```shell ```shell
pip install -r requirements.txt sudo apt install python
``` pip install -r requirements.txt
if you customize the environment in this step, you should change the parameters in the service file we will mention later accordingly. sudo apt-get install python3-tk python3-dev
sudo apt install gnome-screenshot
sudo apt install wmctrl
sudo apt install ffmpeg
sudo apt install socat
sudo apt install xclip
```
3. Copy the `osworld_server.service` to the systemd configuration directory at `/etc/systemd/system/`: If you encounter an error about python not being found, run:
```shell ```
sudo cp osworld_server.service /etc/systemd/system/ sudo ln -s /usr/bin/python3 /usr/bin/python
``` ```
Reload the systemd daemon to recognize the new service: if you customize the environment in this step, you should change the parameters in the service file we will mention later accordingly.
```shell
sudo systemctl daemon-reload
```
Enable the service to start on boot: 3. Due to some configuration issues, you need to modify the `osworld_server.service` file:
```shell 1) In our released version, the X server is set to :1, but the default X server is actually :0. You need to modify the `osworld_server.service` file to change the `DISPLAY` variable from `:1` to `:0`.
sudo systemctl enable osworld_server.service Change the following line:
``` ```
Environment="DISPLAY=:1"
```
to
```
Environment="DISPLAY=:0"
```
2) Need to add environment variables to enable DBUS to change wallpaper.
Change the following line:
```
Environment="DISPLAY=:0"
```
to
```
Environment="DISPLAY=:0;DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus"
```
Start the service: 4. Copy the `osworld_server.service` to the systemd configuration directory at `/etc/systemd/system/`:
```shell ```shell
sudo systemctl start osworld_server.service sudo cp osworld_server.service /etc/systemd/system/
``` ```
Verify the service is running correctly: Reload the systemd daemon to recognize the new service:
```shell ```shell
sudo systemctl status osworld_server.service sudo systemctl daemon-reload
``` ```
You should see output indicating the service is active and running. If there are errors, review the logs with `journalctl -xe` for further troubleshooting. Enable the service to start on boot:
```shell
sudo systemctl enable osworld_server.service
```
If you need to make adjustments to the service configuration, you can edit the `/etc/systemd/system/osworld_server.service` file: Start the service:
```shell ```shell
sudo nano /etc/systemd/system/osworld_server.service sudo systemctl start osworld_server.service
``` ```
After making changes, reload the daemon and restart the service:
```shell
sudo systemctl
```
<!-- vimc: call SyntaxRange#Include('```xml', '```', 'xml', 'NonText'): --> Verify the service is running correctly:
<!-- vimc: call SyntaxRange#Include('```css', '```', 'css', 'NonText'): --> ```shell
<!-- vimc: call SyntaxRange#Include('```sh', '```', 'sh', 'NonText'): --> sudo systemctl status osworld_server.service
<!-- vimc: call SyntaxRange#Include('```bash', '```', 'sh', 'NonText'): --> ```
You should see output indicating the service is active and running. If there are errors, review the logs with `journalctl -xe` for further troubleshooting.
If you need to make adjustments to the service configuration, you can edit the `/etc/systemd/system/osworld_server.service` file:
```shell
sudo nano /etc/systemd/system/osworld_server.service
```
After making changes, reload the daemon and restart the service:
```shell
sudo systemctl daemon-reload
sudo systemctl enable osworld_server.service
sudo systemctl start osworld_server.service
```
### Accessibility Tree Support ### Accessibility Tree Support
@@ -159,9 +391,6 @@ You can run the following command to check if **Xorg** is being used:
echo $XDG_SESSION_TYPE echo $XDG_SESSION_TYPE
``` ```
### System Service Management (Optional) ### System Service Management (Optional)
The automatic software update service can interfere with benchmark examples. To disable this service, you can refer to the https://www.makeuseof.com/disable-automatic-updates-in-ubuntu/ for the solution. The automatic software update service can interfere with benchmark examples. To disable this service, you can refer to the https://www.makeuseof.com/disable-automatic-updates-in-ubuntu/ for the solution.
@@ -227,29 +456,11 @@ To extract relevant internal information and configurations from the VS Code env
# Set "Always save as" to "Microsoft Word 2007-2013 XML (.docx)" # Set "Always save as" to "Microsoft Word 2007-2013 XML (.docx)"
# Repeat similar steps for Calc (.xlsx) and Impress (.pptx) # Repeat similar steps for Calc (.xlsx) and Impress (.pptx)
``` ```
2. GIMP Startup Settings: 2. Chrome password requirement removal:
```bash
# Open GIMP
# Go to Edit -> Preferences -> Interface
# Under "Window Management":
# Uncheck "Show tips on startup"
# Under "File Saving":
# Uncheck "Show warning when saving images that will result in information loss"
```
3. Chrome password requirement removal:
Chrome requests a password input when first opened after system startup, which can interfere with our experiments. Here's how to disable this feature: Chrome requests a password input when first opened after system startup, which can interfere with our experiments. Here's how to disable this feature:
```bash ```bash
Using Terminal # Prevent Chrome from using keyring
# Remove the default keyring
rm -rf ~/.local/share/keyrings/*
# Create empty keyring
echo -n "" >> ~/.local/share/keyrings/login.keyring
# Restart Chrome after applying changes
# Or just prevent Chrome from using keyring
mkdir -p ~/.local/share/keyrings mkdir -p ~/.local/share/keyrings
touch ~/.local/share/keyrings/login.keyring touch ~/.local/share/keyrings/login.keyring
``` ```
@@ -257,9 +468,21 @@ touch ~/.local/share/keyrings/login.keyring
Or you can use any ways to disable the keyring service, which will prevent Chrome from requesting a password input. Or you can use any ways to disable the keyring service, which will prevent Chrome from requesting a password input.
### Network Configuration ### Network Configuration
#### Firewall Configuration
In OSWorld, we need the following ports to be open:
```
server_port = 5000
chromium_port = 9222
vnc_port = 8006
vlc_port = 8080
novnc_port = 5910
```
Please open the corresponding ports in the firewall and any other security tools you are using.
#### socat Installation #### socat Installation
Ensure `socat` is installed to enable port forwarding. Ensure `socat` is installed to enable port forwarding.

1
desktop_env/server/requirements.txt
View File

@@ -8,3 +8,4 @@ numpy
lxml lxml
pygame pygame
pywinauto pywinauto
pyastpi2