Article
GPT-4.1 Sets the Standard in Automated Experiment Design Using Novel Python Libraries
Future Internet
— 2025 — MDPI
Key information
Authors:
Published in
September 8, 2025
Abstract
Large language models (LLMs) have advanced rapidly as tools for automating code generation in scientific research, yet their ability to interpret and use unfamiliar Python APIs for complex computational experiments remains poorly characterized. This study systematically benchmarks a selection of state-of-the-art LLMs in generating functional Python code for two increasingly challenging scenarios: conversational data analysis with the ParShift library, and synthetic data generation and clustering using pyclugen and scikit-learn. Both experiments use structured, zero-shot prompts specifying detailed requirements but omitting in-context examples. Model outputs are evaluated quantitatively for functional correctness and prompt compliance over multiple runs, and qualitatively by analyzing the errors produced when code execution fails. Results show that only a small subset of models consistently generate correct, executable code. GPT-4.1 achieved a 100% success rate across all runs in both experimental tasks, whereas most other models succeeded in fewer than half of the runs, with only Grok-3 and Mistral-Large approaching comparable performance. In addition to benchmarking LLM performance, this approach helps identify shortcomings in third-party libraries, such as unclear documentation or obscure implementation bugs. Overall, these findings highlight current limitations of LLMs for end-to-end scientific automation and emphasize the need for careful prompt design, comprehensive library documentation, and continued advances in language model capabilities.
Publication details
Authors in the community:
Nuno Fachada
ist145239
Carlos Miguel da Costa Fernandes
ist46419
Publication version
CVoR - Corrected Version of Record
Publisher
MDPI
Link to the publisher's version
https://www.mdpi.com/1999-5903/17/9/412
Title of the publication container
Future Internet
First page or article number
412
Volume
17
Issue
9
ISSN
1999-5903
Fields of Science and Technology (FOS)
computer-and-information-sciences - Computer and information sciences
Keywords
- large language models
- llms
- code quality
- code generation
- Python libraries
Publication language (ISO code)
eng - English
Rights type:
Open access
Creative Commons license
CC-BY - Attribution
Financing entity
Fundação para a Ciência e a Tecnologia
Identifier for the funding entity: https://doi.org/10.13039/501100001871
Type of identifier of the funding entity: Crossref Funder
Number for the project, award or grant: UIDB/04111/2020
Financing entity
Fundação para a Ciência e a Tecnologia
Identifier for the funding entity: https://doi.org/10.13039/501100001871
Type of identifier of the funding entity: Crossref Funder
Number for the project, award or grant: UIDB/00066/2020
Financing entity
Fundação para a Ciência e a Tecnologia
Identifier for the funding entity: https://doi.org/10.13039/501100001871
Type of identifier of the funding entity: Crossref Funder
Number for the project, award or grant: UIDB/00408/2020
Financing entity
Fundação para a Ciência e a Tecnologia
Identifier for the funding entity: https://doi.org/10.13039/501100001871
Type of identifier of the funding entity: Crossref Funder
Number for the project, award or grant: CEECINST/00002/2021/CP2788/CT0001