Scaling Down Code: Balancing Data Volume and Intelligence in Code Generation Models

https://doi.org/10.5281/zenodo.18091368

Abstract

The performance of models that perform code generation primarily depends on both the number of data points they are trained on and how well those data points represent functionally correct input code. This research looks at the interactions between various sizes of CodeSearchNet-Python datasets and transformer-based models that have been trained using fine-tuning methods using cross-entropy loss functions. The study also looks at the method used to tokenize the training set and the characteristics and performance of the models' decoders, and whether or not these models have the same type of performance for varying numbers of data extracted from CodeSearchNet-Python. All evaluations used HumanEval data as the performance criterion and reported performance using metrics such as ROUGE and BLEU which measure both functionally-correct and coherent outputs. The results show that as the number of records grows in increments across different dataset sizes, the performance levels of the models fluctuate as a result of the growing volume of usable data; thus producing critical insights into opportunities and risks associated with training data size for optimally efficient trainable resources. Additionally, they show evidence that the effort expended on ensuring that there is a sufficient degree of quality in the preparation (distillation) of training data has a significant impact on addressing the issue of scale.