Introduction

In this repo, I try finetuning an LLM for commit message generation. I am using the Qwen/Qwen2.5-Coder-1.5B-Instruct model, using HuggingFace to download it and for tokenizing inputs and outputs.

Getting Started

To start using this repo, I would recommend creating an environment and installing the libraries in requirements.txt:

pip install -r requirements.txt

This repo also uses PyTorch, but is not included in the requirements.txt, since download changes depending on CPU, GPU, and GPU model.

Instructions to download PyTorch can be found here.

*NOTE if running this in a google colab, there is a specific install given at the start of finetune.ipynb. This script specifically installs unsloth without dependencies to avoid the package downloading its own cpu pytorch. The script also uses specific versions of packages to work with the colab packages.

Optimizations

To reduce the training time, constant changes and improvements were made to training configs. The reductions are shown in timings.txt and helped to reduce the training from 60 hours to 12 hours. There are very likely more improvements to be made, depending on the device used for training, but the current reductions make the training much more manageable.

Results

The finetune.ipynb contains all results as well as a final analysis of the results. Overall, the finetuning improved the model by making commit messages more concise and coincided with the given commit messages in terms of meaning and context much more than the base model. Exact numbers are in the notebook.