lzy/sci-gui-agent-benchmark
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor compare_python_pure_text function for improved normalization and error handling. Update JSON example to clarify instruction for extracting Python code from Colab, changing output file names for consistency.

Browse Source
This commit is contained in:
yuanmengqi
2025-07-03 13:50:21 +00:00
parent bdaf37e0e5
commit cb4bed20a0
2 changed files with 69 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
evaluation_examples/examples/multi_apps/dd60633f-2c72-42ba-8547-6f2c8cb0fdb0.json
View File

@@ -1,7 +1,7 @@
{
"id": "dd60633f-2c72-42ba-8547-6f2c8cb0fdb0",
"snapshot": "multiapps",
"instruction": "I ran some simple code demos on the currently open google colab, and I think the effect is pretty good. Please help me extract the code in all code boxes, merge it into a \"task.py\" file and store it in the local Home directory.",
"instruction": "Please extract all Python code and comments from Karpathy's GPT colab code cells (skip markdown parts), merge into \"gpt_dev_pure_code.py\" in Home directory. Include all Python code and # comments from code cells, but exclude markdown docstrings and file headers.",
"source": "authors",
"config": [
{
@@ -27,7 +27,7 @@
"type": "chrome_open_tabs",
"parameters": {
"urls_to_open": [
"https://colab.research.google.com/github/stanfordnlp/dspy/blob/main/intro.ipynb#scrollTo=L1BHAoL_LRd7"
"https://colab.research.google.com/drive/1JMLa53HDuA-i7ZBmqV7ZnA3c_fvtXnx-?usp=sharing#scrollTo=h5hjCcLDr2WC"
]
}
}
@@ -42,14 +42,15 @@
"func": "compare_python_pure_text",
"result": {
"type": "vm_file",
"path": "/home/user/colab.py",
"dest": "colab.py"
"path": "/home/user/gpt_dev_pure_code.py",
"dest": "gpt_dev_pure_code.py"
},
"expected": {
"type": "cloud_file",
"path": "https://huggingface.co/datasets/xlangai/ubuntu_osworld_file_cache/resolve/main/multi_apps/dd60633f-2c72-42ba-8547-6f2c8cb0fdb0/Colab%20Gold.py",
"dest": "colab_Gold.py"
"path": "https://huggingface.co/datasets/xlangai/ubuntu_osworld_file_cache/resolve/main/multi_apps/dd60633f-2c72-42ba-8547-6f2c8cb0fdb0/gpt_dev_pure_code_gold.py",
"dest": "gpt_dev_pure_code_gold.py"
}
},
"proxy": true
"proxy": true,
"env_may_change": true
}