98 lines
3.7 KiB
Markdown
98 lines
3.7 KiB
Markdown
# DuckTrack
|
|
|
|
This is the repository for the DuckAI DuckTrack app which records all keyboard and mouse input as well as the screen for use in a multimodal computer interaction dataset.
|
|
|
|
## Installation & Setup
|
|
|
|
### Download Application
|
|
|
|
<!-- TODO: add prebuilt applications in github releases -->
|
|
Download the pre-built application for your system [here](https://github.com/TheDuckAI/DuckTrack/releases/).
|
|
|
|
Make sure you have OBS downloaded with the following configuration:
|
|
1. Have a screen capture source recording your whole main screen.
|
|
2. Enable desktop audio and mute microphone.
|
|
3. Make sure the default websocket is enabled.
|
|
|
|
More detailed instructions for OBS setup and installation located [here](OBS_SETUP.md).
|
|
|
|
If you are on MacOS, make sure to enable to the following Privacy & Security permissions before running the app:
|
|
|
|
1. Accessibility (for playing back actions)
|
|
2. Input Monitoring (for reading keyboard inputs)
|
|
|
|
Make sure to accept all other security permission dialogues to ensure that the app works properly.
|
|
|
|
### Build from source
|
|
|
|
Have Python >=3.11.
|
|
|
|
Clone this repo and `cd` into it:
|
|
```bash
|
|
$ git clone https://github.com/TheDuckAI/DuckTrack
|
|
$ cd DuckTrack
|
|
```
|
|
|
|
Install the dependencies for this project:
|
|
```bash
|
|
$ pip install -r requirements.txt
|
|
```
|
|
|
|
Build the application:
|
|
```bash
|
|
$ python3 build.py
|
|
```
|
|
|
|
The built application should be located in the generated `dist` directory. After this, follow the remaining relevant setup instructions.
|
|
|
|
## Running the App
|
|
|
|
You can run the app like any other desktop app on your computer. If you decided to not download the app or build it from source, just run `python main.py` and it should work the same. You will be interacting with the app through an app tray icon or a small window.
|
|
|
|
### Recording
|
|
|
|
From the app tray or GUI, you can start and stop a recording as well as pause and resume a recording. Pausing and resuming is important for when you want to hide sensitive information like credit card of login credentials. You can optionally name your recording and give it a description upon stopping a recording. You can also view your recordings by pressing the "Show Recordings" option.
|
|
|
|
### Playback
|
|
|
|
You can playback a recording, i.e. simulate the series of events from the recording, by pressing "Play Latest Recording", which plays the latest created recording, or by pressing "Play Custom Recording", which lets you choose a recording to play. You can easily replay the most recently played recording by pressing "Replay Recording".
|
|
|
|
To stop the app mid-playback, just press `shift`+`esc` on your keyboard.
|
|
|
|
### Misc
|
|
|
|
To quit the app, you just press the "Quit" option.
|
|
|
|
## Recording Format
|
|
|
|
Recordings are stored in `Documents/DuckTrack_Recordings`. Each recording is a directory containing:
|
|
|
|
1. `events.jsonl` file - sequence of all computer actions that happened. A sample event may look like this:
|
|
```json
|
|
{"time_stamp": 1234567.89, "action": "move", "x": 69.0, "y": 420.0}
|
|
```
|
|
1. `metadata.json` - stores metadata about the computer that made the recording
|
|
2. `README.md` - stores the description for the recording
|
|
3. MP4 file - the screen recording from OBS of the recording.
|
|
|
|
Here is a [sample recording](example) for further reference.
|
|
|
|
## Technical Overview
|
|
|
|
<!-- maybe put a nice graphical representation of the app here -->
|
|
|
|
*TDB*
|
|
|
|
## Known Bugs
|
|
|
|
- After doing lots of playbacks on macOS, a segfault will occur.
|
|
- Mouse movement is not captured when the current application is using raw input, i.e. video games.
|
|
- OBS may not open in the background properly on some Linux machines.
|
|
|
|
## Things To Do
|
|
|
|
- Add logging
|
|
- Testing
|
|
- CI (with builds and testing)
|
|
- Add way to hide/show window from the app tray (and it saves that as a preference?)
|
|
- Make saving preferences a thing generally, like with natural scrolling too |