This repository contains scripts and resources for fine-tuning the Google T5 Mini model. Due to the limitations of the NVIDIA RTX 2070 GPU, QLoRA (Quantized Low-Rank Adaptation) is used to make the fine-tuning process more memory-efficient while maintaining performance.
The RTX 2070 GPU, with its 8 GB memory, poses challenges for traditional fine-tuning methods, especially with models like T5. QLoRA offers a solution by reducing the memory requirements, enabling effective fine-tuning on limited hardware.
-
EDA.ipynb: Notebook for exploratory data analysis, providing insights into the dataset before fine-tuning. -
benchmark.py: Script to benchmark the fine-tuned model and evaluate performance improvements. -
dataimport.py: Handles data import and preprocessing, preparing it for training. -
fine-tune.py: Script for fine-tuning the T5 Mini model using QLoRA, optimized for the RTX 2070 GPU.
Given the memory limitations of the RTX 2070, QLoRA is employed to reduce the memory footprint of the fine-tuning process without compromising on model accuracy.
- Model Selection: Starting with the T5 Mini model for its balance of performance and efficiency.
- Data Preparation: Data is cleaned and preprocessed with
dataimport.py. - Fine-Tuning: The model is fine-tuned using
fine-tune.py, utilizing QLoRA for efficiency. - Benchmarking: Performance is evaluated with
benchmark.py.
- Python 3.8+
- PyTorch
- Hugging Face Transformers
- NVIDIA RTX 2070 GPU or similar
Clone the repository and install the required dependencies:
git clone https://github.com/yourusername/your-repo-name.git
cd your-repo-name
pip install -r requirements.txt- Prepare data with
dataimport.py. - Fine-tune the model using
fine-tune.py. - Benchmark the results with
benchmark.py.
Performance metrics and results from the fine-tuning process will be documented here.
This project is licensed under the MIT License.
This version is more direct and less descriptive, focusing on key information and instructions.