191 lines
6.9 KiB
Plaintext
191 lines
6.9 KiB
Plaintext
# Cameras
|
||
|
||
Here we describe how to set up and use a camera with LeRobot. We support different ways of capturing videos in LeRobot, such as using a phone camera, an integrated laptop camera, an external webcam, or an Intel realsense camera.
|
||
## Set up Cameras
|
||
|
||
There are three ways to connect and use a camera with LeRobot:
|
||
1. Use [Camera Class](./setup_cameras?use+phone=Mac#use-opencvcamera), which allows you to use any camera: usb, realsense, laptop webcam
|
||
2. Use [iPhone camera](./setup_cameras?use+phone=Mac#use-your-phone) with MacOS
|
||
3. Use [Phone camera](./setup_cameras?use+phone=Linux#use-your-phone) on Linux
|
||
|
||
### Use Camera Class
|
||
|
||
In LeRobot, you can efficiently record frames from most cameras using either the OpenCVCamera class or the RealSenseCamera class. For more details on compatibility for the OpenCVCamera class, see [Video I/O with OpenCV Overview](https://docs.opencv.org/4.x/d0/da7/videoio_overview.html).
|
||
|
||
To instantiate a camera, you need a camera index. When you only have one camera like a webcam of a laptop, the camera index is usually `0` but it might differ, and the camera index might change if you reboot your computer or re-plug your camera. This behavior depends on your operating system.
|
||
|
||
To find the camera indices, run the following script:
|
||
```bash
|
||
python lerobot/find_cameras.py list-cameras
|
||
```
|
||
|
||
The output will look something like this if you have two cameras connected:
|
||
```
|
||
--- Detected Cameras ---
|
||
Camera #0:
|
||
Name: OpenCV Camera @ 0
|
||
Type: OpenCV
|
||
Id: 0
|
||
Backend api: AVFOUNDATION
|
||
Default stream profile:
|
||
Format: 16.0
|
||
Width: 1920
|
||
Height: 1080
|
||
Fps: 15.0
|
||
--------------------
|
||
Camera #1:
|
||
Name: OpenCV Camera @ 1
|
||
Type: OpenCV
|
||
Id: 1
|
||
Backend api: AVFOUNDATION
|
||
Default stream profile:
|
||
Format: 16.0
|
||
Width: 1920
|
||
Height: 1080
|
||
Fps: 1.0
|
||
--------------------
|
||
```
|
||
|
||
> [!WARNING]
|
||
> On , you could get this error: `Error finding RealSense cameras: failed to set power state`, this can be solved by running the same command with `sudo` permissions.
|
||
|
||
### Use your phone
|
||
<hfoptions id="use phone">
|
||
<hfoption id="Mac">
|
||
|
||
To use your iPhone as a camera on macOS, enable the Continuity Camera feature:
|
||
- Ensure your Mac is running macOS 13 or later, and your iPhone is on iOS 16 or later.
|
||
- Sign in both devices with the same Apple ID.
|
||
- Connect your devices with a USB cable or turn on Wi-Fi and Bluetooth for a wireless connection.
|
||
|
||
For more details, visit [Apple support](https://support.apple.com/en-gb/guide/mac-help/mchl77879b8a/mac).
|
||
|
||
Your iPhone should be detected automatically when running the camera setup script in the next section.
|
||
|
||
</hfoption>
|
||
<hfoption id="Linux">
|
||
|
||
If you want to use your phone as a camera on Linux, follow these steps to set up a virtual camera
|
||
|
||
1. *Install `v4l2loopback-dkms` and `v4l-utils`*. Those packages are required to create virtual camera devices (`v4l2loopback`) and verify their settings with the `v4l2-ctl` utility from `v4l-utils`. Install them using:
|
||
```python
|
||
sudo apt install v4l2loopback-dkms v4l-utils
|
||
```
|
||
2. *Install [DroidCam](https://droidcam.app) on your phone*. This app is available for both iOS and Android.
|
||
3. *Install [OBS Studio](https://obsproject.com)*. This software will help you manage the camera feed. Install it using [Flatpak](https://flatpak.org):
|
||
```python
|
||
flatpak install flathub com.obsproject.Studio
|
||
```
|
||
4. *Install the DroidCam OBS plugin*. This plugin integrates DroidCam with OBS Studio. Install it with:
|
||
```python
|
||
flatpak install flathub com.obsproject.Studio.Plugin.DroidCam
|
||
```
|
||
5. *Start OBS Studio*. Launch with:
|
||
```python
|
||
flatpak run com.obsproject.Studio
|
||
```
|
||
6. *Add your phone as a source*. Follow the instructions [here](https://droidcam.app/obs/usage). Be sure to set the resolution to `640x480`.
|
||
7. *Adjust resolution settings*. In OBS Studio, go to `File > Settings > Video`. Change the `Base(Canvas) Resolution` and the `Output(Scaled) Resolution` to `640x480` by manually typing it in.
|
||
8. *Start virtual camera*. In OBS Studio, follow the instructions [here](https://obsproject.com/kb/virtual-camera-guide).
|
||
9. *Verify the virtual camera setup*. Use `v4l2-ctl` to list the devices:
|
||
```python
|
||
v4l2-ctl --list-devices
|
||
```
|
||
You should see an entry like:
|
||
```
|
||
VirtualCam (platform:v4l2loopback-000):
|
||
/dev/video1
|
||
```
|
||
10. *Check the camera resolution*. Use `v4l2-ctl` to ensure that the virtual camera output resolution is `640x480`. Change `/dev/video1` to the port of your virtual camera from the output of `v4l2-ctl --list-devices`.
|
||
```python
|
||
v4l2-ctl -d /dev/video1 --get-fmt-video
|
||
```
|
||
You should see an entry like:
|
||
```
|
||
>>> Format Video Capture:
|
||
>>> Width/Height : 640/480
|
||
>>> Pixel Format : 'YUYV' (YUYV 4:2:2)
|
||
```
|
||
|
||
Troubleshooting: If the resolution is not correct you will have to delete the Virtual Camera port and try again as it cannot be changed.
|
||
|
||
If everything is set up correctly, you can proceed with the rest of the tutorial.
|
||
|
||
</hfoption>
|
||
</hfoptions>
|
||
|
||
## Use Cameras
|
||
|
||
Below are two examples, demonstrating how to work with the API.
|
||
|
||
- **Asynchronous frame capture** using an OpenCV-based camera
|
||
- **Color and depth capture** using an Intel RealSense camera
|
||
|
||
### Asynchronous OpenCV Camera
|
||
|
||
This snippet shows how to:
|
||
|
||
* Construct an `OpenCVCameraConfig` with your desired FPS, resolution, color mode, and rotation.
|
||
* Instantiate and connect an `OpenCVCamera`, performing a warm-up read.
|
||
* Read frames asynchronously in a loop via `async_read(timeout_ms)`.
|
||
|
||
```python
|
||
from lerobot.common.cameras.opencv.configuration_opencv import OpenCVCameraConfig
|
||
from lerobot.common.cameras.opencv.camera_opencv import OpenCVCamera
|
||
from lerobot.common.cameras.configs import ColorMode, Cv2Rotation
|
||
|
||
config = OpenCVCameraConfig(
|
||
index_or_path=0,
|
||
fps=15,
|
||
width=1920,
|
||
height=1080,
|
||
color_mode=ColorMode.RGB,
|
||
rotation=Cv2Rotation.NO_ROTATION
|
||
)
|
||
|
||
camera = OpenCVCamera(config)
|
||
camera.connect(do_warmup_read=True)
|
||
|
||
try:
|
||
for i in range(10):
|
||
frame = camera.async_read(timeout_ms=1000)
|
||
print(f"Async frame {i} shape:", frame.shape)
|
||
finally:
|
||
camera.disconnect()
|
||
```
|
||
|
||
### Intel RealSense Camera (Color + Depth)
|
||
|
||
This snippet shows how to:
|
||
|
||
* Create a `RealSenseCameraConfig` specifying your camera’s serial number and enabling depth.
|
||
* Instantiate and connect a `RealSenseCamera` with warm-up.
|
||
* Capture a color frame via `read()` and a depth map via `read_depth()`.
|
||
|
||
```python
|
||
from lerobot.common.cameras.intel.configuration_realsense import RealSenseCameraConfig
|
||
from lerobot.common.cameras.intel.camera_realsense import RealSenseCamera
|
||
from lerobot.common.cameras.configs import ColorMode, Cv2Rotation
|
||
|
||
config = RealSenseCameraConfig(
|
||
serial_number="233522074606",
|
||
fps=15,
|
||
width=640,
|
||
height=480,
|
||
color_mode=ColorMode.RGB,
|
||
use_depth=True,
|
||
rotation=Cv2Rotation.NO_ROTATION
|
||
)
|
||
|
||
camera = RealSenseCamera(config)
|
||
camera.connect(do_warmup_read=True)
|
||
|
||
try:
|
||
color_frame = camera.read()
|
||
depth_map = camera.read_depth()
|
||
print("Color frame shape:", color_frame.shape)
|
||
print("Depth map shape:", depth_map.shape)
|
||
finally:
|
||
camera.disconnect()
|
||
```
|