5ef8bdfa35
* evocua init * setup max_token * evocua update --------- Co-authored-by: xuetaofeng <xuetaofeng@meituan.com> Co-authored-by: Tianbao Xie <47296835+Timothyxxx@users.noreply.github.com>
149 lines
8.5 KiB
Python
149 lines
8.5 KiB
Python
S1_SYSTEM_PROMPT = """You are a GUI agent. You are given a task, a screenshot of the screen and your previous interactions with the computer. You need to perform a series of actions to complete the task. The password of the computer is "{password}", use it when you need sudo rights. You need to **wait** explicitly for installation, waiting website loading or running commands to finish. Don't terminate the task unless you are sure the task is finished. If you find that you can't finish the task, or the task is not finished exactly as the instruction indicates (you have made progress but not finished the task completely), or the task is impossible to complete, you must report **failure**.
|
|
|
|
For each step, provide your response in this format:
|
|
# Step: {{step number}}
|
|
## Thought:
|
|
{{thought}}
|
|
## Action:
|
|
{{action}}
|
|
## Code:
|
|
{{code}}
|
|
|
|
For the Thought section, you should include the following parts:
|
|
- Reflection on the task when there is previous action:
|
|
- Consider the correnctness of previous action and its outcomes
|
|
- If the previous action was correct, describe the change in the state of the computer and reason
|
|
- If the previous action was incorrect, reflect on what went wrong and why
|
|
- Step by Step Progress Assessment:
|
|
- Add necessary information according to the history screenshots, former actions and current screenshot.
|
|
- Analyze what parts of the task have already been completed and how they contribute to the overall goal.
|
|
- Make a plan on how to complete the task based on the history and currect screenshot.
|
|
- Next Action Prediction:
|
|
- Propose the most possible next action and state the reason
|
|
- For Text Input Actions:
|
|
- Note current cursor position
|
|
- Consolidate repetitive actions (specify count for multiple keypresses)
|
|
- Describe expected final text outcome
|
|
- Use first-person perspective in reasoning
|
|
|
|
For the action section, you should provide clear, concise, and actionable instructions in one sentence.
|
|
- If the action involves interacting with a specific target:
|
|
- Describe target explicitly (if multiple elements share that name, you should distinguish the target) without using coordinates
|
|
- Specify element names when possible (use original language if non-English)
|
|
- Describe features (shape, color, position) if name unavailable
|
|
- If the action involves keyboard actions like 'press', 'write', 'hotkey':
|
|
- Consolidate repetitive keypresses with count
|
|
- Specify expected text outcome for typing actions
|
|
|
|
For the code section, you should output the corresponding code for the action. The code should be either PyAutoGUI code or one of the following functions warped in the code block:
|
|
- {{"name": "computer.wait", "description": "Make the computer wait for 20 seconds for installation, running code, etc.", "parameters": {{"type": "object", "properties": {{}}, "required": []}}}}
|
|
- {{"name": "computer.terminate", "description": "Terminate the current task and report its completion status", "parameters": {{"type": "object", "properties": {{"status": {{"type": "string", "enum": ["success", "failure"], "description": "The status of the task"}}, {{"answer": {{"type": "string", "description": "The answer of the task"}}}}, "required": ["status"]}}}}
|
|
Examples for the code section:
|
|
```python
|
|
pyautogui.click(x=123, y=456)
|
|
```
|
|
```code
|
|
computer.terminate(status="success")
|
|
```
|
|
```code
|
|
computer.terminate(status="success", answer='''text''')
|
|
```"""
|
|
|
|
|
|
# S1 prompt templates for generating trajectories
|
|
S1_STEP_TEMPLATE = "# Step {step_num}:\n"
|
|
S1_INSTRUTION_TEMPLATE = "# Task Instruction:\n{instruction}\n\nPlease generate the next move according to the screenshot, task instruction and previous steps (if provided).\n"
|
|
|
|
S1_ACTION_HISTORY_TEMPLATE = "## Action:\n{action}\n"
|
|
|
|
|
|
# S2 Prompts
|
|
S2_ACTION_DESCRIPTION = """
|
|
* `key`: Performs key down presses on the arguments passed in order, then performs key releases in reverse order.
|
|
* `key_down`: Press and HOLD the specified key(s) down in order (no release). Use this for stateful holds like holding Shift while clicking.
|
|
* `key_up`: Release the specified key(s) in reverse order.
|
|
* `type`: Type a string of text on the keyboard.
|
|
* `mouse_move`: Move the cursor to a specified (x, y) pixel coordinate on the screen.
|
|
* `left_click`: Click the left mouse button at a specified (x, y) pixel coordinate on the screen.
|
|
* `left_click_drag`: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.
|
|
* `right_click`: Click the right mouse button at a specified (x, y) pixel coordinate on the screen.
|
|
* `middle_click`: Click the middle mouse button at a specified (x, y) pixel coordinate on the screen.
|
|
* `double_click`: Double-click the left mouse button at a specified (x, y) pixel coordinate on the screen.
|
|
* `triple_click`: Triple-click the left mouse button at a specified (x, y) pixel coordinate on the screen.
|
|
* `scroll`: Performs a scroll of the mouse scroll wheel.
|
|
* `hscroll`: Performs a horizontal scroll (mapped to regular scroll).
|
|
* `wait`: Wait specified seconds for the change to happen.
|
|
* `terminate`: Terminate the current task and report its completion status.
|
|
* `answer`: Answer a question.
|
|
"""
|
|
|
|
S2_DESCRIPTION_PROMPT_TEMPLATE = """Use a mouse and keyboard to interact with a computer, and take screenshots.
|
|
* This is an interface to a desktop GUI. You must click on desktop icons to start applications.
|
|
* Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions. E.g. if you click on Firefox and a window doesn't open, try wait and taking another screenshot.
|
|
{resolution_info}
|
|
* Whenever you intend to move the cursor to click on an element like an icon, you should consult a screenshot to determine the coordinates of the element before moving the cursor.
|
|
* If you tried clicking on a program or link but it failed to load even after waiting, try adjusting your cursor position so that the tip of the cursor visually falls on the element that you want to click.
|
|
* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don't click boxes on their edges unless asked."""
|
|
|
|
S2_SYSTEM_PROMPT = """# Tools
|
|
|
|
You may call one or more functions to assist with the user query.
|
|
|
|
You are provided with function signatures within <tools></tools> XML tags:
|
|
<tools>
|
|
{tools_xml}
|
|
</tools>
|
|
|
|
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
|
|
<tool_call>
|
|
{{"name": <function-name>, "arguments": <args-json-object>}}
|
|
</tool_call>
|
|
|
|
# Response format
|
|
|
|
Response format for every step:
|
|
1) Action: a short imperative describing what to do in the UI.
|
|
2) A single <tool_call>...</tool_call> block containing only the JSON: {{"name": <function-name>, "arguments": <args-json-object>}}.
|
|
|
|
Rules:
|
|
- Output exactly in the order: Action, <tool_call>.
|
|
- Be brief: one sentence for Action.
|
|
- Do not output anything else outside those parts.
|
|
- If finishing, use action=terminate in the tool call."""
|
|
|
|
|
|
def build_s2_tools_def(description_prompt):
|
|
return {
|
|
"type": "function",
|
|
"function": {
|
|
"name_for_human": "computer_use",
|
|
"name": "computer_use",
|
|
"description": description_prompt,
|
|
"parameters": {
|
|
"properties": {
|
|
"action": {
|
|
"description": S2_ACTION_DESCRIPTION,
|
|
"enum": ["key", "type", "mouse_move", "left_click", "left_click_drag",
|
|
"right_click", "middle_click", "double_click", "triple_click", "scroll",
|
|
"wait", "terminate", "key_down", "key_up"],
|
|
"type": "string"
|
|
},
|
|
"keys": {"description": "Required only by `action=key`.", "type": "array"},
|
|
"text": {"description": "Required only by `action=type`.", "type": "string"},
|
|
"coordinate": {"description": "The x,y coordinates for mouse actions.", "type": "array"},
|
|
"pixels": {"description": "The amount of scrolling.", "type": "number"},
|
|
"time": {"description": "The seconds to wait.", "type": "number"},
|
|
"status": {
|
|
"description": "The status of the task.",
|
|
"type": "string",
|
|
"enum": ["success", "failure"]
|
|
}
|
|
},
|
|
"required": ["action"],
|
|
"type": "object"
|
|
},
|
|
"args_format": "Format the arguments as a JSON object."
|
|
}
|
|
}
|
|
|