[
    {
        "id": 1,
        "chunk": "# ChemDFM: A Large Language Foundation Model for Chemistry  \n\nZihan Zhao1 ∗ Da ${{\\bf{M}}{\\bf{a}}^{1}}^{*}$ Lu Chen1,2 † Liangtai Sun1 Zihao Li3 Yi Xia2 Bo Chen2 Hongshen ${\\bf X}{\\bf u}^{1}$ Zichen Zhu1 Su Zhu4 Shuai Fan4 Guodong Shen2 Kai $\\mathbf{Y}\\mathbf{u}^{1,2\\dag}$ Xin Chen2 †  \n\n$^1{\\cal X}$ -LANCE Lab, Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence, SJTU AI Institute Shanghai Jiao Tong University, Shanghai, China 2Suzhou Laboratory, Suzhou, China   \n3Shanghai Key Laboratory for Molecular Engineering of Chiral Drugs School of Chemistry and Chemical Engineering Shanghai Jiao Tong University, Shanghai, China 4AI Speech Co, .Ltd., Suzhou, China {zhao_mengxin, chenlusz, kai.yu}@sjtu.edu.cn",
        "category": " Abstract"
    },
    {
        "id": 2,
        "chunk": "# Abstract  \n\nArtificial intelligence (AI) has played an increasingly important role in chemical research. However, most models currently used in chemistry are specialist models that require training and tuning for specific tasks. A more generic and efficient solution would be an AI model that could address many tasks and support free-form dialogue in the broad field of chemistry. In its utmost form, such a generalist AI chemist could be referred to as Chemical General Intelligence. Large language models (LLMs) have recently logged tremendous success in the general domain of natural language processing, showing emerging task generalization and freeform dialogue capabilities. However, domain knowledge of chemistry is largely missing when training general-domain LLMs. The lack of such knowledge greatly hinders the performance of generalist LLMs in the field of chemistry. To this end, we develop ChemDFM, a pioneering LLM for chemistry trained on 34B tokens from chemical literature and textbooks, and fine-tuned using 2.7M instructions. As a result, it can understand and reason with chemical knowledge in free-form dialogue. Quantitative evaluations show that ChemDFM significantly surpasses most representative open-source LLMs. It outperforms GPT-4 on a great portion of chemical tasks, despite the substantial size difference. We have open-sourced the inference codes, evaluation datasets, and model weights of ChemDFM on Huggingface3.",
        "category": " Abstract"
    },
    {
        "id": 3,
        "chunk": "# 1 Introduction  \n\nWith the rapid development of artificial intelligence (AI), utilizing AI to assist chemical research has garnered increasing attention [Wang et al., 2023b, Back et al., 2024]. Various AI models have been developed for tasks such as property prediction [Zhou et al., 2022, Wu et al., 2023b, Chen et al., 2023], molecular captioning and generation [Xu et al., 2021, Edwards et al., 2022, Perron et al., 2022,  \n\nDu et al., 2024, Lu et al., 2024], and reaction predictions [Schwaller et al., 2020, Wang et al., 2021, Han et al., 2024]. Since BERT [Devlin et al., 2019] and GPT [Radford et al.], efforts have been made to fine-tune pre-trained models for specific chemical tasks [Zhou et al., 2022, Edwards et al., 2022, Liu et al., 2023, Luo et al., 2023, Zhang et al., 2024]. However, these models are typically trained on a meticulously curated dataset to solve a designated task in a particular scenario, leading to a one-to-one relationship between models and tasks. Once out of that specific scenario, they are often not useful, even for highly related tasks. A more attractive and practical AI system should be capable of handling a wide range of chemical tasks under real-world scenarios and conducting free-form human-AI collaborations. Such an AI system necessitates a comprehensive array of chemical competencies, coupled with the ability to comprehend and reason in both chemical and natural languages. This would enable it to work as a research assistant or even collaborator alongside human researchers. This could be an essential step towards eventually achieving Chemical Artificial General Intelligence.  \n\nIn pursuit of a highly integrated AI system for a broad range of chemical challenges, recent advancements in large language models (LLMs) [Du et al., 2022, Touvron et al., 2023a, Xu et al., 2023] brought great new hopes. Numerous studies have demonstrated the remarkable competencies of LLMs in natural language understanding and task generalization [Wei et al., 2021, Xu et al., 2023], deductive reasoning [Wei et al., 2022, Kojima et al., 2022], and tool utilization [Schick et al., 2023, Qin et al., 2024]. These made LLMs shine in traditional natural language processing tasks and accomplish problems that were previously unimaginable and unsolvable, such as handling tasks in unknown scenarios or conducting free-form dialogues with humans. These inherent strengths underscore the viability of employing LLMs as AI-driven research collaborators in the field of chemistry.  \n\nDifferent from general domains, tasks in chemical domains necessitate models to possess additional chemical comprehension capabilities for understanding and reasoning over chemical-specialized language and knowledge. This hinders general domain LLMs from excelling in chemical tasks as they often lack in-depth chemical knowledge [Kristiadi et al., 2024]. For example, molecules are a vital component of the chemical world. Although molecules can be conveyed through naturallanguage-like notations such as SMILES (Simplified Molecular Input Line Entry System), IUPAC names, and molecular formulas, their meanings and intrinsic structures are entirely different from those in natural language. CO represents carbon monoxide in chemistry, not Colorado, while Co represents Cobalt, not a company, and (CO) as part of a SMILES typically represents the carbonyl group. The lack of understanding of these molecular notations severely limits the applicability and performance of general domain LLMs in solving chemistry problems. Therefore, we believe that equipping general-domain LLMs with rich chemical knowledge of task-specific chemical models, as illustrated in Figure 1, is vital for developing LLMs useful in the field of chemistry.  \n\nIn this work, we propose ChemDFM, a Dialogue Foundation Model for Chemistry. ChemDFM takes advantage of the pre-trained LLaMa-13B model [Touvron et al., 2023a], an open-source general-domain LLM, and is further specialized in chemistry through two phases: 1) Domain Pre-training, where the model harvests the chemical knowledge from research articles and textbooks, and 2) Instruction Tuning, where the model familiarizes itself with chemical language and patterns, especially molecule notations. Each phase uses an extensive and diverse collection of chemical data: 1) nearly 34B tokens from over 3.8M chemical pa  \n\n![](images/e597be304841969d07748eab5d2d77d7a4af53c53b6cd55446c415ac69270f28.jpg)  \nFigure 1: Scheme to obtain an LLM for chemistry, through using chemical domain knowledge to train a general-domain LLM.  \n\npers and 1.4K textbooks in chemistry used in Phase I, and 2) over 2.7M instructions crafted from various chemical databases in Phase II. Apart from chemical data, we also incorporated a substantial amount of general-domain data in both phases to make sure that ChemDFM maintains comprehension and reasoning capabilities of natural language while acquiring new chemical knowledge. As a result, ChemDFM can simultaneously handle a wide range of chemical tasks and convey free-form dialogues using the language of chemists, enabling human-AI collaboration in chemical research.  \n\nA series of experiments have been conducted to evaluate the prowess of ChemDFM, including molecule recognition, molecule design, molecular property prediction, and reaction analysis. The results show that ChemDFM achieves advanced performances, surpassing typical open-source LLMs.  \n\n![](images/43ac0a392eaec7b515bfa72edd952364bea2b6d9fd781002d21027e1db76982a.jpg)  \nFigure 2: a) Two-step training procedure to obtain ChemDFM. The icons are generated by the SDXL model provided by Stability $\\bar{\\mathrm{AI}^{5}}$ . b) Various types of tasks ChemDFM is capable of.  \n\nIt even outperforms GPT-4 on most tasks, despite the notable difference in model size. We further compared the performance between ChemDFM and the baseline LLMs in free-form unseen scenarios analogues to real-world scenarios. The test examples were constructed based on the latest chemical papers to avoid possible data leakage. The results show that ChemDFM can generate answers that are more accurate and relevant to the specific questions. These findings suggest that ChemDFM, capable of handling a broad range of chemical tasks and reasoning in both chemical and natural languages, can indeed serve as an AI assistant in chemical research.",
        "category": " Introduction"
    },
    {
        "id": 4,
        "chunk": "# 2 ChemDFM  \n\nAs outlined in Figure 2, ChemDFM is trained based on LLaMa, a general domain LLM. Domain knowledge of chemistry is instilled in ChemDFM in two steps: Domain Pre-training and Instruction Tuning. Through this two-stage specialization process, ChemDFM “learned” chemistry and gained abilities such as molecule recognition and reaction prediction. The training process is presented below and evaluations of ChemDFM’s capability are elaborated in the next section.",
        "category": " Results and discussion"
    },
    {
        "id": 5,
        "chunk": "# 2.1 Domain Pre-training  \n\nData used to train general-domain LLMs must contain knowledge covering a wide range of topics. Such broadness is often accompanied by sacrifices of deepness in each field. While models trained on such data have successfully gained strong natural language understanding and reasoning capabilities, they often fall short when it comes to in-depth specialized knowledge. The lack of domain knowledge is partially responsible for the well-known “hallucination” problem [Huang et al., 2023]. To alleviate this problem, we collected a corpus of data rich in chemical knowledge for domain pre-training, primarily from the two most authoritative sources for chemical knowledge: textbooks and published papers. Textbooks represent the widely accepted knowledge and basic principles of chemistry while published papers offer more details and more up-to-date chemical knowledge, some of which have not been incorporated into textbooks. Specifically, we selected 1.4K chemistry books from LibreTexts6 and Gold Books7 and collected 3.9M open-access papers in chemistry-related topics before January 2022. After further pre-processing and deduplication, we obtained 49M tokens from the textbooks and 34B tokens from the published research articles. To maintain the LLM’s general-domain knowledge and capabilities, we also included highly selective data in the general field, including Wikipedia, Arxiv, Books, StackExchange, GitHub code, WuDao Corpora [Yuan et al., 2021], etc. More details of domain pre-training are available in Appendix Section A.1.",
        "category": " Materials and methods"
    },
    {
        "id": 6,
        "chunk": "# 2.2 Instruction Tuning  \n\nThe data for the chemical instruction tuning dataset comprises two main categories: chemical knowledge presented in natural language and specialized molecular notations. A dataset containing over 1M question-answering pairs specialized in chemistry was constructed for instruction turning to enhance the model’s capability to process chemistry-related natural language. These data were generated from two sources. The first one is established question-answering datasets, including ARC [Clark et al., 2018], PIQA [Bisk et al., 2020], SciQ [Welbl et al., 2017], and HendrycksTest [Hendrycks et al., 2021]. The other source of questions is middle school exams. We collected open-source exam questions from the Internet and constructed question-answer pairs (with key points or problem-solving strategies when available) for the instruction tuning of ChemDFM.  \n\nWhile natural languages such as English or German are generally descriptive and highly versatile, they are often not the best media to convey chemical knowledge. For example, it is often much easier and more comprehensible to draw the molecular structure of a complicated organic molecule than to describe it using natural language. Generations of chemists have developed many specialized notations, such as molecular formulas and Simplified Molecular Input Line Entry System (SMILES) [Weininger, 1988] notation. This represents a key challenge for LLMs to understand chemistry. A key goal of the instruction tuning stage was to tackle this challenge by familiarizing ChemDFM with the specialized notations. In training ChemDFM, we chose SMILES, one of the most popular notations of molecules, as the main representation for molecules. It uses a sequence of letters to present a molecule, retaining rich structural information such as molecular configuration and chirality in most cases. In addition, its text-like data structure makes it highly compatible with LLMs.  \n\nTo help the model comprehend SMILES, three kinds of molecular data were used: 1. Molecule description (MD) and text-based molecule design (TBMD). Our dataset includes all the SMILES-description pairs from PubChem8, a web-scale chemical database that contains more than 100M compounds. The model was instructed to generate descriptions of given molecules or reversely, generate molecule(s) that match a description. We duplicated samples with descriptions longer than two sentences to further enhance the data quality. 2. Molecular property prediction (MPP). The model was instructed to predict the properties of a given molecule. This data was constructed based on the widely used molecular property prediction benchmark, MoleculeNet [Wu et al., 2018]. 3. Reaction completion (RC). The model was also instructed to complete chemical reactions in which one or more reactants/products were masked randomly. The reactions were sampled from USPTO [Lowe, 2012], the largest open-source chemical reaction database.  \n\nTable 1: Itemized list of our instruction tuning dataset. MD: Molecule Description, TBMD: Text-Based Molecule Design, MPP: Molecular Property Prediction, RC: Reaction Completion, MNA: Molecular Notation Alignment.   \n\n\n<html><body><table><tr><td>Data Type</td><td># samples</td><td>Data Source</td></tr><tr><td>QAs from Datasets</td><td>131K</td><td>ARC, PIQA, SciQ HendrycksTest</td></tr><tr><td>QAs from Exam</td><td>915K</td><td>Internet</td></tr><tr><td>MD</td><td>576K</td><td>PubChem</td></tr><tr><td>TBMD</td><td>576K</td><td>PubChem</td></tr><tr><td>MPP</td><td>102K</td><td>MoleculeNet</td></tr><tr><td>RC</td><td>300K</td><td>USPTO</td></tr><tr><td>MNA</td><td>120K</td><td>PubChem</td></tr></table></body></html>",
        "category": " Materials and methods"
    },
    {
        "id": 7,
        "chunk": "# [Round 0]  \n\nHuman:  Chemical reaction equations are typically presented as follows: \"reactant1.reactant2.reactant3...>reagent1.reagent2.reagent3...>product1.product2.product3...\". Each substance, be it a reactant, reagent, or product, is represented using the SMILES notation. You will be given an incomplete chemical reaction equation with missing parts showcased as _\". These voids may stand for one or several substances. Based on the available information in the equation, please predict what the missing substances could be. In your response, list only the missing elements without introducing any additional information.  \n\n![](images/14041f91fd2d33f9b69c098b9d5c8e7dc5f699e97dabaef9af9dc0402c52a9ce.jpg)  \n\n![](images/62d846bcfb24b22410c626e5e3ca70042519043b2e845ecbd1947d0b31e399fe.jpg)  \nFigure 3: Representative question used in instruction tuning.  \n\nIn addition to SMILES, we indirectly include two other widely used notations of molecules, IUPAC names and molecular formulas. We instructed the model to translate between the three notations, e.g. predicting SMILES of a molecule given its IUPAC name and vice versa, allowing it to understand these alternatives. This kind of data is called Molecular Notation Alignment (MNA) in this work.  \n\nTable 1 lists the itemized entries of our instruction tuning dataset. All the data samples take the form of (prompt, returns) tuples, where the prompt is composed of the dialogue format, instructions, and example inputs, and the returns are the expected outputs. Such an example is presented in Figure 3. To diversify the natural language instructions, we used GPT-4 to rephrase instructions for all tasks. The number of different instructions for each task ranges from 20 to 200.  \n\nTo maintain the advanced natural language comprehension abilities, we also included a substantial amount of general domain data for the instruction-tuning of ChemDFM. The ratio of data from the chemical domain to the general domain is roughly 1:2. The instruction tuning of ChemDFM is a full-parameter tuning process with more details in Appendix Section A.2.",
        "category": " Materials and methods"
    },
    {
        "id": 8,
        "chunk": "# 3 Evaluations  \n\nTo assess ChemDFM’s capability in chemistry, we compared its performance against several generalist LLM models: GPT-4 [OpenAI, 2023], LLaMa-2 [Touvron et al., 2023b] and Galactica [Taylor et al., 2022], as they represent very large generalist LLMs, medium-sized generalist LLMs and LLMs tuned for science, respectively. We used ChemLLMBench [Guo et al., 2023] for quantitative evaluation of ChemDFM’s ability in chemistry and then carried out qualitative analyses of ChemDFM’s free-form collaboration capacity, focusing on its superior chemistry-related conversation processing power.",
        "category": " Results and discussion"
    },
    {
        "id": 9,
        "chunk": "# 3.1 Quantitative Evaluation  \n\nChemLLMBench is made of a series of chemical tasks covering a wide range of chemistry-related topics. The standard form of evaluation was conducted on 100 instances randomly sampled from the respective test sets of the tasks. To ensure a fair comparison, we used the same 100 samples when comparing different LLMs, unless otherwise specified. Some non-LLM task-specific models were used for comparisons when available. Detailed explanations of the tasks and the prompt format for ChemDFM can be found in Appendix Section B. Specifically, the quantitative evaluation tasks can be categorized into the following four groups.  \n\n1) Molecule recognition. There are two series of tasks in ChemLLMBench that directly assess the capability to recognize molecules: name prediction and molecule captioning. In the name prediction tasks, a model is asked to translate between different notations for molecules, including SMILES, IUPAC name, and molecular formula. Specifically, it consists of four tasks: SMILES to IUPAC name translation (S2I), IUPAC name to SMILES translation (I2S), SMILES to Molecular Formula translation (S2MF), and IUPAC name to Molecular Formula transla  \n\nTable 2: Accuracy scores in name prediction tasks. Baseline results are from Guo et al. [2023]. S2I: SMILES to IUPAC names translation, I2S: IUPAC names to SMILES translation, S2MF: SMILES to molecule formulas translation, I2MF: IUPAC names to molecule formulas translation.   \n\n\n<html><body><table><tr><td>Model S2I↑</td><td>I2S↑ S2MF↑</td><td>I2MF↑</td></tr><tr><td colspan=\"3\">Task-specific specialist models</td></tr><tr><td>STOUT 55.0</td><td>70.0</td><td></td></tr><tr><td>LLM-based generalist models</td><td></td><td></td></tr><tr><td colspan=\"3\"></td></tr><tr><td>GPT-4 0 LLaMa2-13B-chat 0</td><td>1.2 8.6</td><td>8.4</td></tr><tr><td>Galactica-30B 0</td><td>0 1.0</td><td>0</td></tr><tr><td></td><td>0 0</td><td>0</td></tr><tr><td>ChemDFM-13B 4.0</td><td>11.0 73.0</td><td>51.0</td></tr></table></body></html>  \n\ntion (I2MF). For IUPAC names and SMILES, we normalized the predictions before calculating the accuracy scores, while for molecular formulas, only exact matches are considered correct answers. The molecule captioning tasks further require LLMs to not only recognize the molecule present by a given SMILES notation but also generate a brief description of it using natural language. In these tasks, traditional captioning metrics like BLUE, ROUGE, and METEOR are used to assess the model’s performance on a test set of ChEBI-20 [Edwards et al., 2021].  \n\nBenchmark performance of different models on these two molecule recognition tasks is reported in Table 2 and Table 3, respectively. Table 2 shows that most LLMs, including GPT-4, can hardly complete name prediction tasks, indicating a limited understanding of molecules and ChemDFM outperforms open-source LLMs by a significant margin across all these tasks. This outstanding performance of ChemDFM proves its robust molecule recognition capabilities and validates the effectiveness of our specialization process.  \n\nTable 3: Benchmark results of different models in molecule captioning tasks. †: results from Guo et al. [2023]. \\*: reproduced results.   \n\n\n<html><body><table><tr><td>Model</td><td>BLEU-2个</td><td>BLEU-4↑</td><td>ROUGE-1↑</td><td>ROUGE-2↑</td><td>ROUGE-L↑</td><td>METEOR↑</td></tr><tr><td colspan=\"7\">Task-specific specialistmodels</td></tr><tr><td>Text+Chem T5 [Christofidellis et al., 2023]</td><td>0.625</td><td>0.542</td><td>0.682</td><td>0.543</td><td>0.622</td><td>0.648</td></tr><tr><td>MolXPT [Liu et al., 2023]</td><td>0.594</td><td>0.505</td><td>0.660</td><td>0.511</td><td>0.597</td><td>0.626</td></tr><tr><td>InstructMol [Cao et al., 2023a]</td><td>0.475</td><td>0.371</td><td>0.566</td><td>0.394</td><td>0.502</td><td>0.509</td></tr><tr><td>Mol-Instruction [Fang et al., 2023]</td><td>0.249</td><td>0.171</td><td>0.331</td><td>0.203</td><td>0.289</td><td>0.271</td></tr><tr><td colspan=\"7\">LLM-based generalist models</td></tr><tr><td>GPT-4 (10-shot)</td><td>0.464</td><td>0.365</td><td>0.545</td><td>0.362</td><td>0.459</td><td>0.519</td></tr><tr><td>GPT-4 (0-shot)</td><td>0.062</td><td>0.013</td><td>0.192</td><td>0.040</td><td>0.125</td><td>0.209</td></tr><tr><td>LLaMa-2-13B-chat (10-shot)</td><td>0.197</td><td>0.140</td><td>0.331</td><td>0.193</td><td>0.265</td><td>0.372</td></tr><tr><td>Galactica-30B (10-shot)*</td><td>0.114</td><td>0.055</td><td>0.334</td><td>0.189</td><td>0.330</td><td>0.187</td></tr><tr><td>Galactica-30B (0-shot)</td><td>0.008</td><td>0.002</td><td>0.019</td><td>0.004</td><td>0.015</td><td>0.043</td></tr><tr><td>ChemDFM-13B (0-shot)</td><td>0.321</td><td>0.265</td><td>0.490</td><td>0.374</td><td>0.483</td><td>0.402</td></tr></table></body></html>  \n\nTable 4: Benchmark results of different models in text-based molecule design tasks. †: results from Guo et al. [2023]. \\*: 10-shot results   \n\n\n<html><body><table><tr><td>Model</td><td>Exact↑</td><td>BLEU↑</td><td>Dis↓</td><td>Validity↑</td><td>MACCS↑</td><td>RDK↑</td><td>Morgan↑</td></tr><tr><td colspan=\"8\">Task-specifc specialist models</td></tr><tr><td>MolXPT [Liu et al., 2023]</td><td>21.5</td><td>-</td><td></td><td>98.3</td><td>0.859</td><td>0.757</td><td>0.667</td></tr><tr><td>Text+Chem T5 [Christofidellis et al.,2023]</td><td>32.2</td><td>0.853</td><td>16.87</td><td>94.3</td><td>0.901</td><td>0.816</td><td>0.757</td></tr><tr><td>Mol-Instruction [Fang et al., 2023]</td><td>0.2</td><td>0.345</td><td>41.4</td><td>100</td><td>0.412</td><td>0.231</td><td>0.147</td></tr><tr><td colspan=\"8\">LLM-based generalist models</td></tr><tr><td>GPT-4**</td><td>17.4</td><td>0.816</td><td>21.2</td><td>88.8</td><td>0.867</td><td>0.738</td><td>0.672</td></tr><tr><td>LLaMa-2-13B-chat**</td><td>2.0</td><td>0.626</td><td>34.0</td><td>78.2</td><td>0.679</td><td>0.568</td><td>0.454</td></tr><tr><td>Galactica-30B</td><td>0.0</td><td>0.004</td><td>2738</td><td>95.6</td><td>0.233</td><td>0.109</td><td>0.053</td></tr><tr><td>ChemDFM-13B</td><td>45.0</td><td>0.874</td><td>9.9</td><td>98.0</td><td>0.922</td><td>0.871</td><td>0.798</td></tr></table></body></html>  \n\nIn molecule captioning tasks (as shown in Table 3), ChemDFM also performs far superior to open-source LLMs. The results denote that ChemDFM not only recognizes molecules but also infers their underlying chemical essence and nature. It is worth noting the drastic drop in GPT4’s performance from the ten-shot setting to the zero-shot setting, which indicates that GPT-4 thrives mostly on its extraordinary natural language capabilities to learn from given exemplars while its inherent molecule recognition capability is relatively fragile. Comparatively, ChemDFM achieves comparable performance without the help of exemplars, demonstrating its intrinsic molecule recognition capability.  \n\n2) Text-based molecule design. To evaluate LLM’s efficiency in making qualified molecule designs, ChemLLMBench reverses the molecule captioning tasks and asks the models to generate molecules based on their descriptions. Specifically, in the text-based molecule design task, models are asked to predict the SMILES of the molecule that fits the given description. Two sets of metrics are utilized to measure the performance of these tasks. The first set measures the text-based similarity of the predicted SMILES compared to the golden SMILES, which includes exact match, BLUE, and Levenshtein distance. The second set of metrics measures the chemical similarity of the predicted molecules to the golden molecules, including the validity of the predicted SMILES and the FTS (fingerprint Tanimoto Similarity) [Tanimoto, 1958] in terms of MACCS [und David Metzener, 1988], $\\mathrm{R}\\mathrm{\\bar{D}K}^{\\mathrm{\\bar{g}}}$ , Morgan [Morgan, 1965].  \n\nAs shown in Table 4, ChemDFM outperforms not only the generalist LLMs but also the traditional task-specific specialist models across almost all metrics, which is both surprising and promising. Considering that task-specific specialist models were evaluated on the entire test set, whereas the performance of ChemDFM was initially assessed on only 100 samples, we further evaluated ChemDFM on the complete test set to align with the task-specific models for a fair comparison. The results, shown in Table 7 of the Appendix, further validate the advantage of ChemDFM. The results from Table 4 and 7 unveil two key superiorities of ChemDFM over other models. On the one hand, ChemDFM has effectively established a relationship between SMILES notations and the chemical nature of compounds in our model, which other LLMs lack. On the other hand, ChemDFM benefits from the solid natural language comprehension capabilities inherited from LLaMa, which taskspecific specialist models lack. Altogether, ChemDFM constructs a more comprehensive knowledge system in chemistry, which helps it surpass both generalist and task-specific specialist models.  \n\nTable 5: AUC-ROC scores [Bradley, 1997] of different models in molecular property prediction tasks. Avg: average. †: reproduced results (The results of GPT-4 were obtained in January 2024).   \n\n\n<html><body><table><tr><td>Model</td><td>BACE↑</td><td>BBBP↑</td><td>ClinTox↑</td><td>HIV↑</td><td>Tox21↑</td><td>Avg↑</td></tr><tr><td colspan=\"7\">Task-specific specialist models</td></tr><tr><td>Uni-Mol [Zhou et al., 2022]</td><td>85.7</td><td>72.9</td><td>91.9</td><td>80.8</td><td>79.6</td><td>82.2</td></tr><tr><td>MolXPT [Liu et al., 2023]</td><td>88.4</td><td>80.0</td><td>95.3</td><td>78.1</td><td>77.1</td><td>83.8</td></tr><tr><td>InstructMol [Cao et al., 2023a]</td><td>85.9</td><td>64.0</td><td></td><td>74.0</td><td></td><td></td></tr><tr><td colspan=\"7\">LLM-based generalist models</td></tr><tr><td>GPT-4</td><td>62.5</td><td>61.5</td><td>51.6</td><td>65.9</td><td>55.2</td><td>59.3</td></tr><tr><td>LLaMa-2-13B-chat</td><td>26.0</td><td>60.3</td><td>45.7</td><td>29.0</td><td>51.7</td><td>42.5</td></tr><tr><td>Galactica-30B [Taylor et al., 2022]</td><td>72.7</td><td>59.6</td><td>82.2</td><td>75.9</td><td>68.5</td><td>71.8</td></tr><tr><td>ChemDFM-13B</td><td>78.4</td><td>66.7</td><td>89.9</td><td>73.6</td><td>79.8</td><td>77.7</td></tr></table></body></html>  \n\nTable 6: Accuracy scores of different models in reaction prediction and retrosynthesis tasks. B-H: Buchwald-Hartwig dataset [Ahneman et al., 2018]. Suzuki: Suzuki-Miyaura dataset [Reizman et al., 2016]. YP: Yield Prediction, RP: Reactant Prediction, RS: Reagent Selection, Retro: Retrosynthesis. †: results from Guo et al. [2023]. Please refer to Table $9{\\sim}12$ in the Appendix for complete results.   \n\n\n<html><body><table><tr><td>Model</td><td>YP↑ RP↑</td><td>Retro↑</td><td>RS↑</td></tr><tr><td colspan=\"4\">task-specific specialist models</td></tr><tr><td>UAGNN [Kwon et al., 2022] Chemformer [Irwin et al., 2022]</td><td>96.1</td><td>53.6</td><td>一</td></tr><tr><td colspan=\"4\">93.8 LLM-based generalist models</td></tr><tr><td>GPT-4</td><td>78.2 23.0</td><td>11.4</td><td>45.3</td></tr><tr><td>LLaMa-2-13B-chat</td><td>0.7 3.2</td><td>0.0</td><td>16.0</td></tr><tr><td>Galactica (30B)</td><td>0.4</td><td>1.6</td><td>8.0</td></tr><tr><td>ChemDFM-13B</td><td>81.0</td><td>49.0 12.0</td><td>23.7</td></tr></table></body></html>  \n\n3) Molecular property prediction. The molecular property prediction tasks in ChemLLMBench consist of five tasks from the MoleculeNet [Wu et al., 2018], including BACE, BBBP, HIV, ClinTox, and Tox21. Among them, BACE and BBBP each contain a single balanced binary classification task. HIV contains a single unbalanced binary classification task. ClinTox and Tox21 comprise two and twenty-one unbalanced binary classification tasks, respectively. To address the severe label imbalance in these tasks, the Area Under the Curve of the Receiver Operating Characteristic (AUCROC) metric [Bradley, 1997] was introduced. To better assess the molecular property prediction, we adopted a scaffold-vertical manner for data splitting. Specifically, the molecules from the DeepChem library [Ramsundar et al., 2019] were first grouped based on their Bemis-Murcko scaffold [Bemis and Murcko, 1996] representations. The datasets were then split into training and test sets according to these groups. This method ensures that no molecule sharing the same scaffold would appear in both the training set and the test set. While avoiding information leaking due to mere similarity of molecules, this method also significantly increases the difficulty of the tasks, making the assessment more challenging and meaningful. The results listed in Table 5 show that ChemDFM consistently outperforms other LLMs in all but one molecular property prediction task.  \n\n4) Reaction prediction and retrosynthesis. ChemLLMBench includes four types of tasks targeted at evaluating models’ capability of reaction understanding: Yield Prediction (YP), Reaction Prediction (RP), Reagent Selection (RS), and Retrosynthesis (Retro). The yield prediction tasks ask models to predict whether the given reaction is a high-yield reaction and are constructed based on two High-Throughput experimentation (HTE) datasets: the Buchwald-Hartwig dataset [Ahneman et al., 2018] and the Suzuki-Miyaura dataset [Reizman et al., 2016]. The reaction prediction tasks ask models to predict the product of the given reaction. The USPTO-MIT dataset [Jin et al., 2017] was used to construct these tasks. The retrosynthesis tasks focus on predicting the reactants of the given reactions and are constructed based on the USPTO-50K dataset [Schneider et al., 2016]. The reagent selection tasks focus on selecting the appropriate reactants, solvents, or ligands that lead to a higher yield of the reaction from a list of candidates based on the dataset proposed by Perera et al. [2018]. Accuracy is utilized to measure the performance.  \n\n![](images/4e9036adc0f62b95a01ca275bf678fc24c7ecebf53daac6436545b5bceaba0bd.jpg)  \nFigure 4: Examples of paper reading. Answers from ChemDFM are compared with GPT and the base model LLaMa. Correct and relevant information in the replies is marked in green, correct but irrelevant information in yellow, and wrong information in red. Key points of the answer are marked in bold. Full details and more examples are elaborated in Appendix Section C.1.  \n\nAs depicted in Table 6, ChemDFM’s performance significantly exceeds open-source LLMs in all the reaction-related tasks. When compared to GPT-4, ChemDFM achieved superior performances on 3 out of 4 tasks, but lagged behind on reagent selection tasks. A closer inspection reveals that these reagent selection tasks are multiple-choice questions where models are asked to directly copy the correct SMILES from the candidates listed in the questions. Compared with generative questions such as reaction prediction and retrosynthesis, multiple-choice questions alleviate the models’ burden of generating molecules from scratch. It seems that GPT-4 can indeed better follow the instructions and directly copy corresponding SMILES, while ChemDFM often tries to generate new answers.",
        "category": " Results and discussion"
    },
    {
        "id": 10,
        "chunk": "# 3.2 LLM-Based Research Assistant  \n\nTo function as a competent AI assistant researcher, an LLM needs not only strong chemistry skills, but also language skills to comprehend, reason, and communicate with human researchers, primarily in natural language. In the following, we test ChemDFM in two typical scenarios faced by chemistry researchers: reading papers and designing experiments, both of which demand expertise in chemical and natural language skills.  \n\nReading literature and other technical papers is an indispensable part of a researcher’s daily routine. Oftentimes, researchers come across new concepts or expressions that can hinder their understanding of the material. An  \n\n![](images/1bf70d2587f420b24870361d209b6bd07353f7b8e42616fb7c45af2636596f85.jpg)  \nFigure 5: Example showing ChemDFM as an assistant researcher in the design of experiment through free-form dialogue. Key points of the answer are marked in bold. More examples can be found in Appendix Section C.2.  \n\nLLM-based reading partner or assistant can provide instant explanations and answers to such questions. In Figure 4, we compare the answers generated by ChemDFM with those from other LLMs. We have provided three example questions, with more examples in Appendix Section C.1, which are generally consistent with the analysis below. To prevent information leakage, the questions were constructed from chemistry papers published in 2023 only. Since ChemDFM only learned from papers published before 2022, this approach ensures ChemDFM has not learned the answers during training and simulates ChemDFM’s performance as a reading partner or tutor when reading new papers. Q1 represents a question of widely known domain knowledge. All LLMs including ChemDFM provide good answers. However, when questions involve new molecules and reactions (Q2 [Yin et al., 2023] & Q3 [Dargo et al., 2023]), the performances differ. Specifically, LLaMa-2 and Galactica primarily rely on retrieving knowledge from memory, which can result in numerous knowledge points that are correct but irrelevant or even misleading in the context of the questions. GPT-4 shows a primary level of ability to answer questions based on the provided molecules and/or reactions. It effectively answers Q2 but struggles with more complex questions involving complicated molecules such as Q3. In Q3, GPT-4 fails to fully recognize the underlying chemical aspects of the question and proposes methods that could violate the molecule’s catalytic activity. It is also worth noticing that as GPT-4 is a closed-source LLM, it is uncertain whether the literature used to construct the questions is included in GPT-4’s training corpus. Therefore, these \"new papers\" may not be new to GPT-4. In contrast to other LLMs, ChemDFM shows the ability to integrate memory-based knowledge while considering the situation described in the questions, providing key points that are highly relevant to the question. In terms of accuracy, relevance, and overall quality of the answers, ChemDFM largely outperforms other LLMs including GPT-4, demonstrating a better understanding of molecules and reactions, especially in the example of Q3. Apart from presenting key points, ChemDFM also endeavors to expand on its explanation and elaborate on the mechanism of the queried reactions or the proposed solutions, although this occasionally leads to inaccurate answers, as seen in the cases of Q1 and Q2. Please refer to Appendix Section C.1 for a more detailed analysis.  \n\nA knowledgeable discussion partner who is always available and patient would be invaluable for researchers, particularly in the stage of hypothesis generation and design of experiment (DOE). Figure 5 illustrates a scenario inspired by Yin et al. [2023] that showcases ChemDFM’s potential to assist researchers in free-form dialogues as an AI research partner. In this example, a human researcher aimed to selectively oxidize one of the two carbonyl groups of a molecule. The initial solution given by ChemDFM would lead to the oxidation of both carbonyl groups. However, after being alerted and challenged by the human researcher, ChemDFM acknowledged the mistake and proposed two possible strategies: using a weaker oxidation agent/condition or introducing a protecting group. Upon the researcher’s decision to use a protecting group, ChemDFM provided detailed recommendations, including a feasible agent and reaction condition. Throughout the dialogue, ChemDFM exhibited promising capabilities in comprehension (Round 1), error correction (Round 2), and detailing (Round 3), showcasing its efficacy in mastering both chemical and natural language. More examples can be found in Appendix Section C.2.",
        "category": " Results and discussion"
    },
    {
        "id": 11,
        "chunk": "# 4 Related Work  \n\nThere have been several pioneering studies focusing on leveraging LLMs to solve chemical problems. These works typically adopt one of two general strategies. The first one treats LLMs as powerful base models for multi-task training, neglecting their greatest strength in natural language understanding and reasoning [Christofidellis et al., 2023, Fang et al., 2023, Cao et al., 2023a, Zheng et al., 2023, Kim et al., 2024, Yu et al., 2024]. Consequently, the models devised under this framework are confined to solving the specific tasks on which they were trained, losing the ability to tackle unseen tasks or conduct free-form human-AI collaborations. The other strategy exploits LLMs’ strong natural language understanding and reasoning abilities, using them directly to handle complex chemical tasks described in natural language [Hatakeyama-Sato et al., 2023, Cao et al., 2023b, Boiko et al., 2023, Yoshikawa et al., 2023, M. Bran et al., 2024, Ruan et al., 2024]. However, most of them suffer from the fact that generalist LLMs lack an inherent understanding of chemical language and knowledge [Kristiadi et al., 2024]. We argue that an LLM useful in chemistry must learn and reason with both general-domain knowledge and chemical knowledge. In this work, we tried to achieve this by equipping general-domain LLMs with rich chemical knowledge of task-specific chemical models and obtained promising results.  \n\nNotably, this strategy has been successfully applied to develop specialist LLMs for several other scientific domains. For example, Med-PaLM [Singhal et al., 2023] and PMC-LLaMa [Wu et al., 2023a] are specialized LLMs for biology and medicine. Similarly, ChatDoctor [Li et al., 2023] and DrugChat [Liang et al., 2023] also offer LLMs specifically for the medicine field but focus on medical inquiries and drug discoveries. Other domain-specific LLMs have endeavored include education [Dan et al., 2023], materials science [Xie et al., 2023], and geography [Deng et al., 2023]. It is worth noting that most of these works only focus on natural language. Domain-specific languages, which differ significantly from natural languages, such as SMILES in chemistry, are often overlooked.",
        "category": " Introduction"
    },
    {
        "id": 12,
        "chunk": "# 5 Conclusion  \n\nIn summary, this paper introduces ChemDFM, a specialist LLM that evolves from a generalist LLM through pre-training and instruction tuning using domain knowledge in chemistry. Quantitative evaluations show ChemDFM’s strong comprehension of molecular notations and reasoning capabilities for chemical knowledge, resulting in excellent performance in a wide range of chemical tasks such as molecular design and reaction analysis. In scenarios such as paper reading and experimental design, ChemDFM shows great potential in wielding chemical and natural languages to assist researchers through dialogue-based, free-form human-AI collaborations.",
        "category": " Conclusions"
    },
    {
        "id": 13,
        "chunk": "# Acknowledgments and Disclosure of Funding  \n\nThis work was supported by the National Science and Technology Major Project 2023ZD0120703, the China NSFC Projects (U23B2057, 62106142 and 62120106006), and the Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102).",
        "category": " References"
    },
    {
        "id": 14,
        "chunk": "# References  \n\nDerek T Ahneman, Jesús G Estrada, Shishi Lin, Spencer D Dreher, and Abigail G Doyle. Predicting reaction performance in c–n cross-coupling using machine learning. Science, 360(6385):186–190, 2018.  \n\nSeoin Back, Alán Aspuru-Guzik, Michele Ceriotti, Ganna Gryn’ova, Bartosz Grzybowski, Geun Ho Gu, Jason Hein, Kedar Hippalgaonkar, Rodrigo Hormázabal, Yousung Jung, et al. Accelerated chemical science with ai. Digital Discovery, 3(1):23–33, 2024.  \n\nGuy W Bemis and Mark A Murcko. The properties of known drugs. 1. molecular frameworks. Journal of medicinal chemistry, 39(15):2887–2893, 1996.  \n\nYonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI, volume 34, pages 7432–7439, 2020.  \n\nDaniil A Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical research with large language models. Nature, 624(7992):570–578, 2023.  \n\nAndrew P Bradley. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern recognition, 30(7):1145–1159, 1997.  \n\nHe Cao, Zijing Liu, Xingyu Lu, Yuan Yao, and Yu Li. Instructmol: Multi-modal integration for building a versatile and reliable molecular assistant in drug discovery. arXiv preprint arXiv:2311.16208, 2023a.  \n\nZhonglin Cao, Rishikesh Magar, Yuyang Wang, and Amir Barati Farimani. Moformer: self-supervised transformer model for metal–organic framework property prediction. Journal of the American Chemical Society, 145(5):2958–2967, 2023b.  \n\nMarta Catellani, Franco Frignani, and Armando Rangoni. A complex catalytic cycle leading to a regioselective synthesis of o, o’-disubstituted vinylarenes. ChemInform, 28(16), 1997.  \n\nTianlang Chen, Shengjie Luo, Di He, Shuxin Zheng, Tie-Yan Liu, and Liwei Wang. GeoMFormer: A general architecture for geometric molecular representation learning. In NeurIPS 2023 AI for Science Workshop, 2023. URL https://openreview.net/forum?id $\\c=$ s0UNtuuqU5.   \nDimitrios Christofidellis, Giorgio Giannone, Jannis Born, Ole Winther, Teodoro Laino, and Matteo Manica. Unifying molecular and textual representations via multi-task language modelling. In ICML, 2023.   \nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.   \nYuhao Dan, Zhikai Lei, Yiyang Gu, Yong Li, Jianghao Yin, Jiaju Lin, Linhao Ye, Zhiyan Tie, Yougen Zhou, Yilei Wang, Aimin Zhou, Ze Zhou, Qin Chen, Jie Zhou, Liang He, and Xipeng Qiu. Educhat: A large-scale language model-based chatbot system for intelligent education. arXiv preprint arXiv:2308.02773, 2023.   \nGyula Dargo, David Kis, Martin Gede, Sushil Kumar, Jozsef Kupai, and Gyorgy Szekely. Mesesamol, a bio-based and versatile polar aprotic solvent for organic synthesis and depolymerization. Chemical Engineering Journal, page 144365, 2023.   \nCheng Deng, Tianhang Zhang, Zhongmou He, Yi Xu, Qiyuan Chen, Yuanyuan Shi, Luoyi Fu, Weinan Zhang, Xinbing Wang, Chenghu Zhou, Zhouhan Lin, and Junxian He. K2: A foundation language model for geoscience knowledge understanding and utilization. arXiv preprint arXiv:2306.05064, 2023.   \nDB Dess and JC Martin. Readily accessible 12-i-5 oxidant for the conversion of primary and secondary alcohols to aldehydes and ketones. The Journal of Organic Chemistry, 48(22):4155– 4156, 1983.   \nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the NAACL, 2019. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423.   \nYuanqi Du, Arian R Jamasb, Jeff Guo, Tianfan Fu, Charles Harris, Yingheng Wang, Chenru Duan, Pietro Liò, Philippe Schwaller, and Tom L Blundell. Machine learning-aided generative molecular design. Nature Machine Intelligence, pages 1–16, 2024.   \nZhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. GLM: General language model pretraining with autoregressive blank infilling. In Proceedings of the ACL, 2022. doi: 10.18653/v1/2022.acl-long.26. URL https://aclanthology.org/2022. acl-long.26.   \nCarl Edwards, ChengXiang Zhai, and Heng Ji. Text2Mol: Cross-modal molecule retrieval with natural language queries. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021. doi: 10.18653/v1/2021.emnlp-main.47. URL https: //aclanthology.org/2021.emnlp-main.47.   \nCarl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. Translation between molecules and natural language. In Proceedings of the EMNLP, 2022. doi: 10.18653/v1/ 2022.emnlp-main.26. URL https://aclanthology.org/2022.emnlp-main.26.   \nYin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Mol-instructions: A large-scale biomolecular instruction dataset for large language models. arXiv preprint arXiv:2306.08018, 2023.   \nTaicheng Guo, Kehan Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. What can large language models do in chemistry? a comprehensive benchmark on eight tasks. arXiv preprint arXiv:2305.18365, 2023.   \nYuqiang Han, Xiaoyang Xu, Chang-Yu Hsieh, Keyan Ding, Hongxia Xu, Renjun Xu, Tingjun Hou, Qiang Zhang, and Huajun Chen. Retrosynthesis prediction with an iterative string editing model. Nature Communications, 15(1):6404, 2024.  \n\nYu Hao, Zi-Hao Li, Zhi-Gang Ma, Ru-Xin Liu, Rui-Tian Ge, Quan-Zhe Li, Tong-Mei Ding, and Shu-Yu Zhang. Axially chiral styrene-based organocatalysts and their application in asymmetric cascade michael/cyclization reaction. Chemical Science, 14(35):9496–9502, 2023.  \n\nKan Hatakeyama-Sato, Naoki Yamane, Yasuhiko Igarashi, Yuta Nabae, and Teruaki Hayakawa. Prompt engineering of gpt-4 for chemical research: what can/cannot be done? Science and Technology of Advanced Materials: Methods, (1):2260300, 2023.   \nDan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ICLR, 2021.   \nLei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023. URL https://arxiv.org/abs/2311.05232.   \nRoss Irwin, Spyridon Dimitriadis, Jiazhen He, and Esben Jannik Bjerrum. Chemformer: a pre-trained transformer for computational chemistry. Machine Learning: Science and Technology, 3(1): 015022, 2022.   \nWengong Jin, Connor Coley, Regina Barzilay, and Tommi Jaakkola. Predicting organic reaction outcomes with weisfeiler-lehman network. Advances in neural information processing systems, 30, 2017.   \nSeongmin Kim, Yousung Jung, and Joshua Schrier. Large language models for inorganic synthesis predictions. Journal of the American Chemical Society, 2024.   \nMilton C Kloetzel. The diels-alder reactions with maleic anhydride. Org React, 4:1–59, 1948.   \nTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. NeurIPS, 35:22199–22213, 2022.   \nAgustinus Kristiadi, Felix Strieth-Kalthoff, Marta Skreta, Pascal Poupart, Alán Aspuru-Guzik, and Geoff Pleiss. A sober look at llms for material discovery: Are they actually good for bayesian optimization over molecules? arXiv preprint arXiv:2402.05015, 2024.   \nYoungchun Kwon, Dongseon Lee, Youn-Suk Choi, and Seokho Kang. Uncertainty-aware prediction of chemical reaction yields with graph neural networks. Journal of Cheminformatics, 14:1–10, 2022.   \nYunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Cureus, 15(6), 2023.   \nYouwei Liang, Ruiyi Zhang, Li Zhang, and Pengtao Xie. Drugchat: Towards enabling chatgpt-like capabilities on drug molecule graphs. arXiv preprint arXiv:2309.03907, 2023.   \nZequn Liu, Wei Zhang, Yingce Xia, Lijun Wu, Shufang Xie, Tao Qin, Ming Zhang, and Tie-Yan Liu. MolXPT: Wrapping molecules with text for generative pre-training. In Proceedings of the ACL, 2023. doi: 10.18653/v1/2023.acl-short.138. URL https://aclanthology.org/2023. acl-short.138.   \nIlya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. URL https://openreview.net/forum?id $\\c=$ Bkg6RiCqY7.   \nDaniel Mark Lowe. Extraction of chemical structures and reactions from the literature. PhD thesis, University of Cambridge, 2012.   \nShuqi Lu, Lin Yao, Xi Chen, Hang Zheng, Di He, and Guolin Ke. 3d molecular generation via virtual dynamics. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=QvipGVdE6L.  \n\nShengjie Luo, Tianlang Chen, Yixian Xu, Shuxin Zheng, Tie-Yan Liu, Liwei Wang, and Di He. One transformer can understand both 2d & 3d molecular data. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id= vZTp1oPV3PC.  \n\nAndres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Augmenting large language models with chemistry tools. Nature Machine Intelligence, pages 1–11, 2024.   \nH. L. Morgan. The generation of a unique machine description for chemical structures-a technique developed at chemical abstracts service. Journal of Chemical Documentation, 5(2):107–113, 1965. doi: 10.1021/c160017a018. URL https://doi.org/10.1021/c160017a018.   \nOpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.   \nDamith Perera, Joseph W Tucker, Shalini Brahmbhatt, Christopher J Helal, Ashley Chong, William Farrell, Paul Richardson, and Neal W Sach. A platform for automated nanomole-scale reaction screening and micromole-scale synthesis in flow. Science, 359(6374):429–434, 2018.   \nQuentin Perron, Olivier Mirguet, Hamza Tajmouati, Adam Skiredj, Anne Rojas, Arnaud Gohier, Pierre Ducrot, Marie-Pierre Bourguignon, Patricia Sansilvestri-Morel, Nicolas Do Huu, et al. Deep generative models for ligand-based de novo design applied to multi-parametric optimization. Journal of Computational Chemistry, 43(10):692–703, 2022.   \nYujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. ToolLLM: Facilitating large language models to master $16000+$ real-world APIs. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id $\\c=$ dHng2O0Jjr.   \nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training.   \nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’20. IEEE Press, 2020. ISBN 9781728199986.   \nBharath Ramsundar, Peter Eastman, Patrick Walters, Vijay Pande, Karl Leswing, and Zhenqin Wu. Deep Learning for the Life Sciences. O’Reilly Media, 2019. https://www.amazon.com/ Deep-Learning-Life-Sciences-Microscopy/dp/1492039837.   \nBrandon J Reizman, Yi-Ming Wang, Stephen L Buchwald, and Klavs F Jensen. Suzuki–miyaura cross-coupling optimization enabled by automated feedback. Reaction chemistry & engineering, 1 (6):658–666, 2016.   \nYixiang Ruan, Chenyin Lu, Ning Xu, Jian Zhang, Jun Xuan, Jianzhang Pan, Qun Fang, Hanyu Gao, Xiaodong Shen, Ning Ye, et al. Accelerated end-to-end chemical synthesis development with large language models, 2024.   \nTimo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.   \nNadine Schneider, Nikolaus Stiefl, and Gregory A Landrum. What’s what: The (nearly) definitive guide to reaction role assignment. Journal of chemical information and modeling, 56(12):2336– 2346, 2016.   \nPhilippe Schwaller, Riccardo Petraglia, Valerio Zullo, Vishnu H Nair, Rico Andreas Haeuselmann, Riccardo Pisoni, Costas Bekas, Anna Iuliano, and Teodoro Laino. Predicting retrosynthetic pathways using transformer-based models and a hyper-graph exploration strategy. Chemical science, 11(12):3316–3325, 2020.  \n\nKaran Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023.  \n\nTaffee T Tanimoto. Elementary mathematical theory of classification and prediction. Journal of Biomedical Science and Engineering, 1958.   \nRoss Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022.   \nHugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.   \nHugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.   \nJohn W. Ratcliff und David Metzener. Pattern matching: The gestalt approach. Dr. Dobb’s Journal, 1988.   \nAi-Fang Wang, Jin-Miao Tian, Xiao-Jing Zhao, Zi-Hao Li, Ye Zhang, Ka Lu, Hong Wang, Shu-Yu Zhang, Yong-Qiang Tu, Tong-Mei Ding, et al. Asymmetric intramolecular hydroalkylation of internal olefin with cycloalkanone to directly access polycyclic systems. Angewandte Chemie International Edition, 62(39):e202308858, 2023a.   \nHanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47–60, 2023b.   \nXiaorui Wang, Yuquan Li, Jiezhong Qiu, Guangyong Chen, Huanxiang Liu, Benben Liao, Chang-Yu Hsieh, and Xiaojun Yao. Retroprime: A diverse, plausible and transformer-based method for single-step retrosynthesis predictions. Chemical Engineering Journal, 420:129845, 2021.   \nJason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In ICLR, 2021.   \nJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 35: 24824–24837, 2022.   \nDavid Weininger. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28(1):31–36, 1988.   \nJohannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209, 2017.   \nChaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-llama: Towards building open-source language models for medicine. arXiv preprint arXiv:2304.14454, 2023a.   \nFang Wu, Dragomir Radev, and Stan Z Li. Molformer: Motif-based transformer on 3d heterogeneous molecular graphs. In Proceedings of the AAAI, volume 37, pages 5312–5320, 2023b.   \nZhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9(2):513–530, 2018.   \nTong Xie, Yuwei Wan, Wei Huang, Zhenyu Yin, Yixuan Liu, Shaozhou Wang, Qingyuan Linghu, Chunyu Kit, Clara Grazian, Wenjie Zhang, Imran Razzak, and Bram Hoex. Darwin series: Domain specific large language models for natural science. arXiv preprint arXiv:2308.13565, 2023.  \n\nCanwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. In Proceedings of the EMNLP, 2023. doi: 10.18653/ v1/2023.emnlp-main.385. URL https://aclanthology.org/2023.emnlp-main.385.  \n\nMinkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. In International Conference on Learning Representations, 2021.   \nZhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, Zhongzhu Zhou, Michael Wyatt, Molly Smith, Lev Kurilenko, Heyang Qin, Masahiro Tanaka, Shuai Che, Shuaiwen Leon Song, and Yuxiong He. DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales. arXiv preprint arXiv:2308.01320, 2023.   \nJun-Jie Yin, Yun-Peng Wang, Jun Xue, Feng-Fan Zhou, Xing-Qian Shan, Rong Zhu, Kun Fang, Lei Shi, Shu-Yu Zhang, Si-Hua Hou, et al. Total syntheses of polycyclic diterpenes phomopsene, methyl phomopsenonate, and iso-phomopsene via reorganization of c–c single bonds. Journal of the American Chemical Society, 145(39):21170–21175, 2023.   \nNaruki Yoshikawa, Marta Skreta, Kourosh Darvish, Sebastian Arellano-Rubach, Zhi Ji, Lasse Bjørn Kristensen, Andrew Zou Li, Yuchi Zhao, Haoping Xu, Artur Kuramshin, et al. Large language models for chemistry robotics. Autonomous Robots, 47(8):1057–1086, 2023.   \nBotao Yu, Frazier N Baker, Ziqi Chen, Xia Ning, and Huan Sun. Llasmol: Advancing large language models for chemistry with a large-scale, comprehensive, high-quality instruction tuning dataset. arXiv preprint arXiv:2402.09391, 2024.   \nSha Yuan, Hanyu Zhao, Zhengxiao Du, Ming Ding, Xiao Liu, Yukuo Cen, Xu Zou, Zhilin Yang, and Jie Tang. Wudaocorpora: A super large-scale chinese corpora for pre-training language models. AI Open, 2:65–68, 2021.   \nJuzheng Zhang, Yatao Bian, Yongqiang Chen, and Quanming Yao. Unimot: Unified molecule-text language model with discrete token representation. arXiv preprint arXiv:2408.00863, 2024.   \nZhiling Zheng, Ali H Alawadhi, Saumil Chheda, S Ephraim Neumann, Nakul Rampal, Shengchao Liu, Ha L Nguyen, Yen-hsu Lin, Zichao Rong, J Ilja Siepmann, et al. Shaping the water-harvesting behavior of metal–organic frameworks aided by fine-tuned gpt models. Journal of the American Chemical Society, 145(51):28284–28295, 2023.   \nGengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. In ICLR, 2022.   \nQing-Bo Zhuang, Jin-Rui Tian, Ka Lu, Xiao-Ming Zhang, Fu-Min Zhang, Yong-Qiang Tu, Rong Fan, Zhi-Hao Li, and Yu-Dong Zhang. Catalytic asymmetric polycyclization of tertiary enamides with silyl enol ethers: Total synthesis of (-)-cephalocyclidin a. Journal of the American Chemical Society, 145(49):26550–26556, 2023.",
        "category": " References"
    },
    {
        "id": 15,
        "chunk": "# A Experimental Setups",
        "category": " Materials and methods"
    },
    {
        "id": 16,
        "chunk": "# A.1 Domain Pre-training  \n\nChemDFM is pre-trained using the popular framework Megatron-DeepSpeed10 with Zero-2 [Rajbhandari et al., 2020] optimization technique based on LLaMa-13B [Touvron et al., 2023a]. We train ChemDFM using AdamW [Loshchilov and Hutter, 2019] with $(\\beta_{1},\\beta_{2})=(0.9,0.95)$ . During training, our model deals with 4M tokens per batch with a maximum sequence length of 6K. The maximum learning rate is 5e-5 under the cosine learning rate scheduler.  \n\n[Round 0]  \n\n![](images/c700e6755bb9fdc455f61ae5b9aa86769df120f0d6c5cb8d43ddeb0abff1cdc4.jpg)  \nFigure 6: Prompt format of the name prediction tasks",
        "category": " Materials and methods"
    },
    {
        "id": 17,
        "chunk": "# A.2 Instruction Tuning  \n\nTo fully exploit the capabilities of the pre-trained model, we employed full-parameter tuning during the instruction tuning stage. The popular framework Deepspeed-Chat [Yao et al., 2023] is leveraged with the Zero-3 optimization technique. We set the learning rate to 1e-5 with a global batch size of 256. To encourage the model to focus more on responding to the requirements rather than memorizing the patterns in prompts, we performed gradient back-propagation only on the tokens of the returns. Specifically, the loss function of our instruction tuning is  \n\n$$\n\\mathcal{L}=-\\frac{1}{|\\mathcal{D}|}\\sum_{i=1}^{|\\mathcal{D}|}\\sum_{j=1}^{n_{i}}l o g\\mathrm{P}(r_{j}|\\mathrm{prompt}_{i},r_{1},r_{2},...,r_{j-1}),\n$$  \n\nwhere $|\\mathcal D|$ is the size of the instruction tuning dataset and $\\mathtt{r e t u n r s}_{i}=(r_{1},r_{2},...,r_{n_{i}})$ . We train ChemDFM using AdamW with $(\\beta_{1},\\beta_{2})=(0.9,0.95)$ and a cosine learning rate scheduler.",
        "category": " Materials and methods"
    },
    {
        "id": 18,
        "chunk": "# B More Details about ChemLLMBench Evaluations",
        "category": " Results and discussion"
    },
    {
        "id": 19,
        "chunk": "# B.1 Molecule Recognition",
        "category": " Introduction"
    },
    {
        "id": 20,
        "chunk": "# B.1.1 Task Introduction  \n\nThe name prediction tasks take advantage of the different notations of molecules, including SMILES, IUPAC name, and molecular formula, and ask the models to translate between them. Specifically, it consists of four tasks: SMILES to IUPAC name translation (S2I), IUPAC name to SMILES translation (I2S), SMILES to Molecular Formula translation (S2MF), and IUPAC name to Molecular Formula translation (I2MF). For IUPAC names and SMILES, we normalized the predictions before calculating the accuracy scores, while for molecular formulas, only exact matches are considered correct answers.  \n\nThe molecule captioning tasks further require the LLMs to not only recognize what the molecule given by SMILES is but also understand the basic chemical nature of the molecule so as to generate a brief description of it. Specifically, ChemLLMBench leverages the test set of ChEBI-20 [Edwards et al., 2021] for this task. To measure the performance of this task, ChemLLMBench utilizes a series of traditional captioning metrics, including BLUE, ROUGE, and METEOR.",
        "category": " Introduction"
    },
    {
        "id": 21,
        "chunk": "# B.1.2 Prompt Format  \n\nFor the name prediction tasks, we use a simpler prompt compared with that introduced in Guo et al. [2023]. An example is shown in Figure 6  \n\nFor the molecule captioning task, we use the same prompt introduced in Guo et al. [2023].  \n\nTable 7: Benchmark full test-set evaluation results of different models in text-based molecule design tasks. The best results among specialist and generalist models are highlighted in bold, respectively. $^\\dagger$ : reproducing results.   \n\n\n<html><body><table><tr><td>Model</td><td>Exact↑</td><td>BLUE↑</td><td>Dis↓</td><td>Validity↑</td><td>MACCS↑</td><td>RDK↑</td><td>Morgan↑</td></tr><tr><td colspan=\"8\">task-specific specialist models</td></tr><tr><td>MolXPT [Liu et al., 2023]</td><td>21.5</td><td></td><td>-</td><td>98.3</td><td>0.859</td><td>0.757</td><td>0.667</td></tr><tr><td>Text+Chem T5 [Christofidellis et al., 2023]</td><td>32.2</td><td>0.853</td><td>16.87</td><td>94.3</td><td>0.901</td><td>0.816</td><td>0.757</td></tr><tr><td>Mol-Instruction [Fang et al., 2023]</td><td>0.2</td><td>0.345</td><td>41.4</td><td>100</td><td>0.412</td><td>0.231</td><td>0.147</td></tr><tr><td colspan=\"8\">LLM-based generalist models</td></tr><tr><td>Galactica-30B (10-shot)</td><td>0.3</td><td>0.295</td><td>64.3</td><td>82.2</td><td>0.356</td><td>0.239</td><td>0.186</td></tr><tr><td>ChemDFM-13B</td><td>43.2</td><td>0.839</td><td>16.9</td><td>97.6</td><td>0.901</td><td>0.829</td><td>0.759</td></tr></table></body></html>",
        "category": " Results and discussion"
    },
    {
        "id": 22,
        "chunk": "# B.2 Text-Based Molecule Design",
        "category": " Introduction"
    },
    {
        "id": 23,
        "chunk": "# B.2.1 Task Introduction  \n\nThe test set of ChEBI-20 is also exploited for this task in ChemLLMBench. Models are asked to predict the SMILES of the molecule that fits the given description. Two kinds of metrics are utilized to measure the performance of this task. The first set of metrics measures the text-based similarity of the predicted SMILES compared to the golden SMILES, which includes exact match, BLUE, and Levenshtein distance. The second set of metrics measures the chemical similarity of the predicted molecules compared to the golden molecules. That is mainly composed of the validity of the predicted SMILES and the FTS (fingerprint Tanimoto Similarity) [Tanimoto, 1958] in terms of MACCS [und David Metzener, 1988], RDK11, Morgan [Morgan, 1965].",
        "category": " Results and discussion"
    },
    {
        "id": 24,
        "chunk": "# B.2.2 Prompt Format  \n\nWe use the same prompt introduced in Guo et al. [2023].",
        "category": " References"
    },
    {
        "id": 25,
        "chunk": "# B.2.3 Additional Results  \n\nTo achieve a fair comparison with task-specific specialist models, we evaluate the performance of ChemDFM on the full test set of ChEBI-20 on this task. The results are illustrated in Table 7. ChemDFM surpasses the performance of the advanced specialist models on the major metrics while achieving comparable performance on others. Specifically, ChemDFM outperforms the specialist models on exact match scores and all three FTS-based similarity scores, which indicates that ChemDFM can make more reliable predictions based on the descriptions compared with specialist models.",
        "category": " Results and discussion"
    },
    {
        "id": 26,
        "chunk": "# B.3 Molecular Property Prediction",
        "category": " Results and discussion"
    },
    {
        "id": 27,
        "chunk": "# B.3.1 Task Introduction  \n\nThe molecular property prediction tasks in ChemLLMBench consist of five tasks from MoleculeNet benchmark [Wu et al., 2018], including BACE, BBBP, HIV, ClinTox, and Tox21. Among them, BACE and BBBP are each a balanced binary classification task. HIV is an unbalanced binary classification task. ClinTox and Tox21 comprise two and twenty-one unbalanced binary classification tasks, respectively.",
        "category": " Introduction"
    },
    {
        "id": 28,
        "chunk": "# B.3.2 Prompt Format  \n\nWe use the same prompts introduced in Guo et al. [2023].",
        "category": " References"
    },
    {
        "id": 29,
        "chunk": "# B.3.3 Additional Results  \n\nDuring evaluations, we leverage a popular and more challenging dataset split provided by DeepChem library [Ramsundar et al., 2019]. We reproduce the results of the baseline models, including GPT-4, LLaMa-2-13B-chat, and Galactica (30B). Apart from the results in the Quantitative Evaluation  \n\nTable 8: AUC-ROC scores [Bradley, 1997] of different models under different settings in molecular property prediction tasks. †reproducing results (The results of GPT-4 were obtained in January 2024).   \n\n\n<html><body><table><tr><td>Model</td><td>BACE↑</td><td>BBBP↑</td><td>ClinTox↑</td><td>HIV↑</td><td>Tox21个</td></tr><tr><td colspan=\"6\">LLM-based generalist models</td></tr><tr><td>GPT-4 (0-shot)</td><td>62.5</td><td>61.5</td><td>51.6</td><td>65.9</td><td>55.2</td></tr><tr><td>GPT-4 (8-shot)</td><td>45.9</td><td>61.8</td><td>59.3</td><td>50.8</td><td>60.6</td></tr><tr><td>LLaMa-2-13B-chat (O-shot)</td><td>26.0</td><td>60.3</td><td>45.7</td><td>29.0</td><td>51.7</td></tr><tr><td>LLaMa-2-13B-chat (8-shot)</td><td>72.9</td><td>52.3</td><td>42.1</td><td>70.8</td><td>45.9</td></tr><tr><td>Galactica-30B [Taylor et al., 2022]</td><td>72.7</td><td>59.6</td><td>82.2</td><td>75.9</td><td>68.5</td></tr><tr><td>ChemDFM-13B (0-shot)</td><td>78.4</td><td>66.7</td><td>89.9</td><td>73.6</td><td>79.8</td></tr><tr><td>ChemDFM-13B (8-shot)</td><td>81.7</td><td>67.9</td><td>85.3</td><td>73.3</td><td>76.7</td></tr></table></body></html>  \n\n[Round 0]  \n\n![](images/ec8badf8bd1d6c94ed620987af271b1f52c9830ceaf535f73e6cc7a0861dd81e.jpg)  \nFigure 7: Prompt format of the reaction prediction and retrosynthesis tasks  \n\nSection of the main text, we also conduct few-shot experiments. The results are shown in Table 8. It is worth noticing that the performances under the few-shot setting are not always better than those under the zero-shot setting. That may be a result of the scaffold-vertical dataset split we use in our experiments. Because under the scaffold-vertical setting, the exemplars provided by the training split may be less helpful for the test samples.",
        "category": " Results and discussion"
    },
    {
        "id": 30,
        "chunk": "# B.4 Reaction Prediction and Retrosynthesis",
        "category": " Introduction"
    },
    {
        "id": 31,
        "chunk": "# B.4.1 Task Introduction  \n\nIn ChemLLMBench, there are four types of tasks targeted at evaluating models’ capabilities of reaction understanding. The yield prediction tasks ask models to predict whether the given reaction is a high-yield reaction and are constructed based on two High-Throughput experimentation (HTE) datasets: the Buchwald-Hartwig dataset [Ahneman et al., 2018] and the Suzuki-Miyaura dataset [Reizman et al., 2016]. The reaction prediction task asks the model to predict the product of the given reaction. ChemLLMBench utilizes the USPTO-MIT dataset [Jin et al., 2017] for this task. The reagent selection tasks focus on selecting the reagent that can maximize the yield of the reaction from a list of candidates. ChemLLMBench constructs three reagent selection tasks based on the dataset proposed by Perera et al. [2018]. The retrosynthesis task focuses on predicting the reactants of the given reactions and is constructed based on the USPTO-50K dataset [Schneider et al., 2016]. Accuracy is utilized to measure the performances except for the ligand selection task which uses top $50\\%$ accuracy.  \n\nTable 9: Accuracy scores of different models in yield prediction tasks. B-H and Suzuki stand for the Buchwald-Hartwig dataset and the Suzuki-Miyaura dataset, respectively. $\\dagger$ : results from Guo et al. [2023]. \\*: 8-shot results   \n\n\n<html><body><table><tr><td>Model</td><td>B-H Suzuki</td></tr><tr><td>task-specific specialist models</td><td></td></tr><tr><td>UAGNN [Kwon et al., 2022] 96.5</td><td>95.7</td></tr><tr><td>LLM-based generalist models</td><td></td></tr><tr><td>80.0 GPT-4+*</td><td>76.4</td></tr><tr><td>LLaMa-2-13B-chat 0.8</td><td>0.6</td></tr><tr><td>Galactica-30B 0.0</td><td>0.8</td></tr><tr><td>ChemDFM-13B 82.7</td><td>79.3</td></tr></table></body></html>  \n\nTable 10: Benchmark results of different models in reaction prediction tasks. †: results from Guo et al. [2023].   \n\n\n<html><body><table><tr><td>Model</td><td>Accuracy</td><td>Validity</td></tr><tr><td>task-specific specialist models Chemformer [Irwin et al., 2022]</td><td>93.8</td><td>100</td></tr><tr><td>Mol-Instruction [Fang et al., 2023] InstructMol [Cao et al., 2023a]</td><td>4.5 53.6</td><td>100 100</td></tr><tr><td>LLM-based generalist models</td><td></td><td></td></tr><tr><td>GPT-4 (20-shot)</td><td>23.0</td><td>93.0</td></tr><tr><td>LLaMa-2-13B-chat (20-shot)</td><td>3.2</td><td>72.2</td></tr><tr><td>Galactica-30B (5-shot)</td><td>3.6</td><td>94.8</td></tr><tr><td>ChemDFM-13B (0-shot)</td><td>49.0</td><td>98.0</td></tr></table></body></html>",
        "category": " Results and discussion"
    },
    {
        "id": 32,
        "chunk": "# B.4.2 Prompt Format  \n\nWe reformat the prompt provided by Guo et al. [2023] using the SMILES notations for reactions.   \nSpecifically, the examples of our prompts are illustrated in Figure 7.",
        "category": " Materials and methods"
    },
    {
        "id": 33,
        "chunk": "# B.4.3 Additional Results  \n\nThe complete results for the yield prediction tasks, the reaction prediction task, the reagent selection tasks, and the retrosynthesis tasks are shown in Table 9, Table 10, Table 12, and Table 11, respectively.",
        "category": " Results and discussion"
    },
    {
        "id": 34,
        "chunk": "# C More Qualitative Analysis",
        "category": " Results and discussion"
    },
    {
        "id": 35,
        "chunk": "# C.1 Paper Reading  \n\nWe first test the models with questions that only involve known knowledge (Figure 8).  \n\nQ-S1 (Q1) is an example of knowledge-intense questions. Models only need to memorize the details and mechanisms of Catellani-type reactions [Catellani et al., 1997] to answer the question correctly. The key point of the answer to this question is “regioselectivity”. While Galactica can hardly answer the question and LLaMa-2 misses the key point of the answer, ChemDFM accurately captures the key point to answer the question and provides a comprehensive answer. GPT-4 gives the best reply as it not only points out “regioselectivity” but also gives the result of the regioselectivity of norbornene. ChemDFM is the only model that tries to provide a detailed description of the mechanism behind the reaction. However, it makes minor mistakes when doing so.  \n\nQ-S2 asks for the regioselectivity of the Diels-Alder reaction [Kloetzel, 1948]. Only ChemDFM successfully answers the key points to this question, which is the result of the regioselectivity. GPT-4 provides a detailed introduction to the Diels-Alder reaction and regioselectivity but fails to answer the specific regioselectivity of the Diels-Alder reaction, while LLaMa-2 only gives the factors that could influence the regioselectivity. They do not answer the question.  \n\n![](images/59a44d9af45e1385b57e8284808895042aac6b9e1e2ae743f533a571806916d9.jpg)  \nFigure 8: Examples of paper reading where only widely known domain knowledge is involved. correct and relevant information in the replies is marked in green, correct but irrelevant information in yellow, and wrong information in red. Key points of the answer are marked in bold.  \n\n![](images/cd4a3f98bfc6ebb0c769837c4e394bdd81592a986d1b491cee8ebf5bf105e20c.jpg)  \nFigure 9: Examples of paper reading where new molecules and reactions are involved. correct and relevant information in the replies is marked in green, correct but irrelevant information in yellow, and wrong information in red. Key points of the answer are marked in bold.  \n\nTable 11: Benchmark results of different models in retrosynthesis tasks. $\\dagger$ : results from Guo et al. [2023].   \n\n\n<html><body><table><tr><td>Model</td><td>Accuracy</td><td>Validity</td></tr><tr><td>task-specific specialist models</td><td></td><td></td></tr><tr><td>Chemformer [Irwin et al., 2022]</td><td>53.6</td><td>100</td></tr><tr><td>LLM-based generalist models</td><td></td><td></td></tr><tr><td>GPT-4 (5-shot)</td><td>11.4</td><td>89.0</td></tr><tr><td>LLaMa-2-13B-chat (20-shot)</td><td>0.0</td><td>72.8</td></tr><tr><td>Galactica-30B (5-shot)</td><td>1.6</td><td>94.8</td></tr><tr><td>ChemDFM-13B (0-shot)</td><td>12.0</td><td>91.0</td></tr></table></body></html>  \n\nTable 12: Benchmark results of different models in reagent selection tasks. We report the result in accuracy scores except for Ligand Selection where we report the top $50\\%$ accuracy score. $\\dagger$ : results from Guo et al. [2023].   \n\n\n<html><body><table><tr><td>Model</td><td>Reactant</td><td>Solvent</td><td>Ligand</td></tr><tr><td colspan=\"4\">LLM-based generalist models</td></tr><tr><td>GPT-4</td><td>29.9</td><td>52.6</td><td>53.4</td></tr><tr><td>LLaMa-2-13B-chat+</td><td>14.5</td><td>5.0</td><td>28.4</td></tr><tr><td>Galactica-30B</td><td>10.7</td><td>10.4</td><td>3.0</td></tr><tr><td>ChemDFM-13B</td><td>24.0</td><td>12.0</td><td>35.0</td></tr></table></body></html>  \n\nAs for Q-S3, ChemDFM, Galactica, and GPT-4, all capture the key point to the answer (“the oxidation of alcohols to aldehydes and ketones”), while ChemDFM and GPT-4 further answer more properties of the Dess-Martin periodinane [Dess and Martin, 1983]. LLaMa-2, on the other hand, gives numerous wrong arguments and misses the key points.  \n\nThen, we ask the models about new molecules and new reactions which are published after January 2022. In this way, we can ensure minimal risk of data leakage and evaluate the models’ capability to handle unforeseen situations. The results are shown in Figure 9 and Figure 10.  \n\nQ-S4 (Q2) is constructed based on Yin et al. [2023]. Because the reaction mentioned in the question is a novel instance, models need to correctly identify the reaction and discover the mechanisms of it before answering the question. In practice, Galactica successfully identifies the key point of the answer, “deprotonate”, but fails to provide other useful information. LLaMa-2, in its reply, fails to identify the reaction mentioned in the question. Most of the information about NaH in its reply is correct but irrelevant to the reaction. GPT-4 identifies the key point of the answer but only gives a rough description of the mechanism of how it works. ChemDFM not only correctly identifies the key point of the answer but also provides an almost correct description of the mechanism.  \n\nQ-S5 is also constructed based on Yin et al. [2023]. All the models can recognize the DIBAL-H as a reducing agent, which is existing knowledge. However, only ChemDFM successfully identifies the reaction site of the new molecule, indicating its strong capabilities to handle unforeseen situations where new molecules and reactions are involved. The main mistake that ChemDFM makes is providing the wrong IUPAC name, which is a challenging task for LLMs even as a separate task (see Table 2 in the main text).  \n\nQ-S6 is constructed based on Wang et al. [2023a] and asks directly for the mechanism of the given reaction. Among the answers, the answer of ChemDFM is the most precise. Galactica and LLaMa-2 give nearly no correct information. Although GPT-4’s answer contains the correct reaction process, it also contains auxiliary processes that do not happen during the reaction, which masks the whole mechanism predicted by GPT-4 wrong. ChemDFM answers the correct reaction process with no excess. The only mistakes ChemDFM makes are again providing the wrong IUPAC names, which is a challenging task for LLMs even as a separate task (see Table 2 in the main text).  \n\nWe also ask several questions focusing more on molecules and less on reactions.  \n\n![](images/0ba025f64fdbbed6fecb072ff5358201f2d3ea5573aa840fb503956782eade8b.jpg)  \nFigure 10: Examples of paper reading where new molecules and reactions are involved. correct and relevant information in the replies is marked in green, correct but irrelevant information in yellow, and wrong information in red. Key points of the answer are marked in bold.  \n\nQ-S7 (Q3), constructed based on Dargo et al. [2023], focus on the modification of catalyst molecules. The molecule mentioned in the question is a novel instance and models need to infer the chemical properties of that molecule to answer the question. The key point of the answer is “introducing electron-withdrawing groups on the aromatic rings” as this method has the potential to increase the acidity while keeping the catalytic ability of the molecule. Among the LLMs, only ChemDFM successfully answers the key point, while others either fail to provide any specific solutions or give wrong solutions which will damage the catalytic ability of the molecule.  \n\nQ-S8, constructed based on Hao et al. [2023], focus on the modification of chiral environment. In the given molecule, there are two chiral centers. GPT-4 identifies the point chiral center and provides candidate methods that are not all correct. The other three models identify the axial chirality which is challenging to identify with only the SMILES notation. Among the three models, Galactica gives almost no detailed method to change the chiral environment, LLaMa-2 gives one correct method with more wrong ones, and ChemDFM provides two correct methods one of which is targeting specifically the axial chirality.  \n\nQ-S9 is constructed on Wang et al. [2023a]. It asks for the coordinate sites between the given molecule and palladium. There are a total of three coordinate sites. GPT-4 and ChemDFM each identify one of them, while Galactica and LLaMa-2 fail to identify any.",
        "category": " Results and discussion"
    },
    {
        "id": 36,
        "chunk": "# C.2 Dialogue-Based Human-AI Collaboration  \n\nWe demonstrate two more examples of dialogue-based human-AI collaboration based on ChemDFM here. The dialogues are also inspired by the recently published papers to minimize the risk of data leakage and evaluate ChemDFM’s capability to handle unforeseen situations during human-AI collaboration.  \n\nThe dialogue shown in Figure 11 is inspired by Yin et al. [2023]. During the dialogue, the human researcher first asks for the role of LiCl in the given reaction. ChemDFM successfully identifies",
        "category": " Results and discussion"
    },
    {
        "id": 37,
        "chunk": "# Round 1",
        "category": " Introduction"
    },
    {
        "id": 38,
        "chunk": "# What is the role of LiCl in the process of generating O=C1C=C2C3(CCC2)CC4(C)CCC(C)(C)C4C31C from O=C1C(C(OC)=O)=C2C3(CCC2)CC4(C)CCC(C)(C)C4C31C?  \n\nThe role of LiCl in this process is to act as a catalyst for the reaction. LiCl is added to a solution of the reactants in dry acetone, and it helps to facilitate the Diels-Alder reaction between the diene and dienophile moieties of the two starting materials. In this reaction LiCl serves as a Lewis acid, coordinating with the carbonyl oxygen of one of the starting materials and increasing the electrophilicity of the dienophile. This makes it more reactive towards the diene, which is then able to add to the dienophile to form the final product  \n\n![](images/39abf54487cadbc2d05716ff1c394138499d66535e6b47461839ea997bde1978.jpg)",
        "category": " Results and discussion"
    },
    {
        "id": 39,
        "chunk": "# Round 2  \n\nBut it is not Diels-Alder reaction. The ester group on reactant leaves to form product.  \n\nYou are correct, I apologize for the mistake in my previous response. The process you described is not a Diels-Alder reaction, but rather a decarbonylation reaction. In this reaction, the LiCl catalyst helps to facilitate the loss of the carbonyl group from one of the starting materials, allowing the decarbonylated product to be formed. The exact mechanism by which LiCl promotes this reaction is not entirely clear, but it is believed to involve the formation of a carbanion intermediate that can undergo betaelimination to remove the carbonyl group.  \n\n![](images/32f40bc26c5ec5652af712cf8897f26927d5bbfbc0308c27e93448dd42590257.jpg)",
        "category": " Results and discussion"
    },
    {
        "id": 40,
        "chunk": "# Round 3  \n\nAfter using LiCl, a lithium salt intermediate product was obtained. What should be added next to obtain the final product?  \n\nAfter using LiCl to promote the decarbonylation reaction, the resulting intermediate product will contain a lithium salt. To obtain the final product, the lithium salt can be hydrolyzed by adding water to the reaction mixture. This will cause the lithium salt to precipitate out as a solid, and the resulting supernatant can be isolated and purified to obtain the final product. Alternatively, the lithium salt can be converted back into one of the starting materials by treating it with a proton source, such as acid.  \n\n![](images/e8e5c0a6d163c034951a307ad053d818b863b6dcf81376db155e2685ca380ea3.jpg)  \n\nFigure 11: Example showing ChemDFM as an assistant researcher in the design of experiment through free-form dialogue. Key points of the answer are marked in bold and wrong information in red. The dialogue is inspired by Yin et al. [2023].",
        "category": " Results and discussion"
    },
    {
        "id": 41,
        "chunk": "# Round 1  \n\nWhat intramolecular reactions will occur when AIBN and Bu3SnH is added to CC(C)(C)[Si](C)(C)OC1(c2cc3c(cc2Br)OCO3)C(=O)N2CCCC23C=C(OC(=O)c2ccccc2)C(=O)C13?  \n\nWhen AIBN and Bu3SnH is added to CC(C)(C)[Si](C)(C)OC1(c2cc3c(cc2Br)OCO3)C $\\scriptstyle(=0)$ )N2CCCC23C $\\scriptstyle:=($ C(OC(=O)c2ccccc2)C(=O)C13, an intramolecular reaction will occur to form a new ring. The reaction involves the formation of a new C-C bond between the carbon adjacent to the carbonyl group and the carbon adjacent to the silicon group in the starting material. This reaction is called the \"ene\" reaction.  \n\n![](images/313f2a6444bce2ddc08b38f2ee1bd54f6ce22569cc4d39053beb9c3391a293ba.jpg)",
        "category": " Results and discussion"
    },
    {
        "id": 42,
        "chunk": "# Round 2  \n\nBut this is a radical debromination reaction.  \n\nYou are correct, I apologize for the error in my previous response. The reaction between AIBN and Bu3SnH in the presence of CC(C)(C)[Si](C)(C)OC1(c2cc3c(cc2Br)OCO3)C(=O)N2CCCC23C=C(OC(=O)c2ccccc2)C(=O)C13 will result in a debromination reaction, not an ene reaction. The debromination reaction will occur at the beta-position of the enone moiety resulting in the formation of a new C-C bond and the release of bromine.  \n\n![](images/5ff1af1dc2427389ca3e13ccb45d531c24b5a82e11b1abb2d5c507c79d884aaa.jpg)",
        "category": " Results and discussion"
    },
    {
        "id": 43,
        "chunk": "# Round 3  \n\nSubsequently, TBAF is added, and what is its purpose?  \n\nTBAF (tetrabutylammonium fluoride) is a source of fluoride ion and will be added to the reaction mixture after the debromination reaction. The purpose of adding TBAF is to hydrolyze the silicon-containing group in the product, thereby removing the protecting group and revealing the alcohol functional group. This step is necessary to allow for further functionalization of the molecule.  \n\n![](images/a99599a94a5e451c8b0ab884fa7e24754e8aa65b4f99c40538afcc5095056686.jpg)  \n\nFigure 12: Example showing ChemDFM as an assistant researcher in the design of experiment through free-form dialogue. Key points of the answer are marked in bold and wrong information in red. The dialogue is inspired by Zhuang et al. [2023].  \n\nthe LiCl as a catalyst while misjudging the type of the reaction. To correct the answer, the human researcher points out the key error in the answer with some important details of the reaction (which can be easily discovered by comparing the product with the reactant). ChemDFM then corrects its mistake with even more details about the reaction process. To further validate whether ChemDFM fully understands the unforeseen reaction, we continue to ask about the post-processing procedure which is necessary to get the final product. ChemDFM successfully captures the key point to the question and gives the correct answer.  \n\nThe dialogue shown in Figure 12 is inspired by Zhuang et al. [2023]. ChemDFM first gives a partially correct answer to the question from the human researcher where it misjudges the position of the newly formed C-C bond and the type of the reaction. With the help of human correction, ChemDFM then realizes the mistakes and corrects them. Then the human researcher further asks about the next reaction that is conducted in Zhuang et al. [2023] without clarifying the current molecule composition of the system or restating the previous reaction. ChemDFM can infer this information from the dialogue history and correctly answer the question.  \n\nIn these dialogues, ChemDFM shows promising capabilities in handling unforeseen situations, error correction, and inferring information from dialogue history. These capabilities can be attributed to the fact that ChemDFM comprehends both natural language and chemical language. This allows a universal language protocol established between ChemDFM and human researchers, enabling meaningful human-AI collaborations.",
        "category": " Results and discussion"
    }
]