[NeurIPS 2023 Datasets and Benchmarks Track] ChemLLMBench ⚛

The official repository of "What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks". https://arxiv.org/abs/2305.18365

🆕 News

[Dec 2023] We updated all test dataset we used and we also add prompts we used (in log format) in the folder of each task. Please see data/ folder for more details!
[Sep 2023] Our paper has been accepted to NeurIPS 2023 Datasets and Benchmarks Track!
[Sep 2023] We released the second version (v2) of our paper, we added extra LLMs (GPT-4, GPT-3.5, Davinci-003, LLama2, Galactica) experiments; more baselines, and more investigations on SELFIES, and label interpretation!
[May 2023] We released the first version (v1) of our paper! Very glad to share our investigations and insights about LLM in chemistry!

💡 Tasks Overview

📌 Prompt

The followings are our prompt used in the paper. It's extremely easy to try your own designed prompt! Only need to change the prompt in the Jupyter code of each task and then we can see the results and performance.

Zero-shot Prompt

ICL Prompt

📊 Dataset

The datasets of some tasks are already uploaded in this repository. Becuase of the size limit, please download these datasets according to the link. After downloading these datasets, please move these datasets to the corresponding folder and then you can run our Jupyter code of each task.

Dataset	Link	Reference
USPTO_Mixed	download	https://github.com/MolecularAI/Chemformer
USPTO-50k	download	https://github.com/MolecularAI/Chemformer
ChEBI-20	download	https://github.com/blender-nlp/MolT5
Suzuki-miyaura	download	https://github.com/seokhokang/reaction_yield_nn
Butchward-Hariwig	download	https://github.com/seokhokang/reaction_yield_nn
BBBP,BACE,HIV,Tox21,Clintox	download	https://github.com/hwwang55/MolR
PubChem	download	https://github.com/ChemFoundationModels/ChemLLMBench/blob/main/data/name_prediction/llm_test.csv

🤗 Cite us

@misc{guo2023gpt,
      title={What indeed can GPT models do in chemistry? A comprehensive benchmark on eight tasks}, 
      author={Taicheng Guo and Kehan Guo and Bozhao Nan and Zhenwen Liang and Zhichun Guo and Nitesh V. Chawla and Olaf Wiest and Xiangliang Zhang},
      year={2023},
      eprint={2305.18365},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

🤗 Contact us

Taicheng Guo: tguo2@nd.edu

Xiangliang Zhang: xzhang33@nd.edu