Assessing LLM-Generated Python Code: Proficiency Analysis and Detection of Student Usage

10/31/2024

This thesis investigates the effects of a LLM, specifically chatbots powered by them like ChatGPT and Gemini, on computer science students. It explores how these tools influence their learning processes and comprehension of programming concepts. As LLMs become increasingly integrated into educational environments, they offer significant advantages such as enhanced accessibility to code examples and debugging assistance. However, this widespread adoption also introduces challenges, primarily concerning the potential for students to rely excessively on these tools at the expense of developing a deep, foundational understanding of programming principles. This research critically examines the interactions between students and LLMs, assessing how these models facilitate or hinder genuine learning and problem-solving capabilities. By exploring the nuances of prompt engineering to test the limits of code generation and the impact of LLM-generated code on student work, this study seeks to understand the broader educational implications of these technologies. Additionally, this work introduces a tool for analyzing assessments using CodeBERT or Moss, which assesses the similarity between student code and solutions produced by LLMs. The findings reveal variances in the performance of different LLMs and underscore the necessity of integrating prompt engineering to optimize LLM outputs effectively. This research contributes to highlighting the potential risks of their misuse.

Authors in the community:

• 

  A

  Aylton Fernandes da Silva

  ist186390

Supervisors of this institution:

• 

  M

  Maria Luísa Torres Ribeiro Marques da Silva Coheur

  ist24630

• 

  A

  Alberto Abad Gareta

  ist90700

electrical-engineering-electronic-engineering-information-engineering - Electrical engineering, electronic engineering, information engineering

eng - English

Instituto Superior Técnico