detection with BERT.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Competition Overview\n",
    "DataHack is a datathon hosted by Club Sientifique de l'Esi (CSE) community in The Higher National School of Computer Science (ESI ex INI) Algiers for 2023/2024 season.\n",
    "### Description\n",
    "In this Challenge, you are tasked with developing machine learning models to identify spam and non-spam (genuine) reviews from Chicago hotels and restaurants. The dataset consists of reviews and metadata, and the goal is to create models that accurately classify reviews as either spam (Y) or non-spam (N).  \n",
    "[**Link to the competetion**](https://www.kaggle.com/competitions/DATAHACK-Collective-Opinion-Spam-Detection/overview)\n",
    "\n",
    "#### **Note** :\n",
    "in this notebook i've kept only the reviews columns in order to use BERT , later on in theis competition i've tried encoding the reviews column with Word2Vec and used XGBoost which gave me way better results (because i exploited the other columns)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\akeba\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\tqdm\\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from transformers import BertTokenizer, BertForSequenceClassification, AdamW\n",
    "from torch.utils.data import DataLoader, Dataset\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code defines a custom PyTorch Dataset class named `CustomDataset` for processing textual data in a datathon competition. The class takes three parameters during initialization: `dataframe` (presumably containing text data and corresponding labels), `tokenizer` (likely a pre-trained tokenizer from the Hugging Face Transformers library), and `max_length` (the maximum length of input sequences to be processed). \n",
    "\n",
    "The `__len__` method returns the length of the dataset, while the `__getitem__` method retrieves a sample from the dataset at a given index. For each sample, it retrieves the text and label from the dataframe, encodes the text using the provided tokenizer with specified parameters (truncation, padding, and maximum length), and returns a dictionary containing `'input_ids'` (tokenized input sequence), `'attention_mask'` (mask indicating which tokens are padding and which are not), and `'labels'` (the corresponding label converted to a PyTorch tensor of type long).\n",
    "\n",
    "All in the purpose of facilitating the preparation of data , ensuring that text inputs are properly tokenized and formatted as required by the model architecture.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomDataset(Dataset):\n",
    "    def __init__(self, dataframe, tokenizer, max_length):\n",
    "        self.data = dataframe\n",
    "        self.tokenizer = tokenizer\n",
    "        self.max_length = max_length\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.data)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        text = self.data.iloc[idx]['review']\n",
    "        label = self.data.iloc[idx]['label']\n",
    "        encoding = self.tokenizer(text, truncation=True, padding='max_length', max_length=self.max_length, return_tensors='pt')\n",
    "        return {\n",
    "            'input_ids': encoding['input_ids'].flatten(),\n",
    "            'attention_mask': encoding['attention_mask'].flatten(),\n",
    "            'labels': torch.tensor(label, dtype=torch.long)\n",
    "        }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### loading trainning data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>Date</th>\n",
       "      <th>review ID</th>\n",
       "      <th>reviewer ID</th>\n",
       "      <th>product ID</th>\n",
       "      <th>rating_Helpful</th>\n",
       "      <th>rating_Thanks</th>\n",
       "      <th>rating_LoveThis</th>\n",
       "      <th>rating_OhNo</th>\n",
       "      <th>reviews</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>5/17/2009</td>\n",
       "      <td>0dFa6egshOwhusL8aSMw-Q</td>\n",
       "      <td>8GC6cFcby0stKarnzL9i2w</td>\n",
       "      <td>dKcO9OQ44RPRlkWe-vToFA</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>Just got back from Shaw's. Great oysters. They...</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>10/25/2011</td>\n",
       "      <td>htQgJ_Z0ADA_QHeKthfeFw</td>\n",
       "      <td>88KSdQ5IMdpCkOidmq1udg</td>\n",
       "      <td>NkOir65b_YAAQVlJR_zmJA</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Back from friday &amp; saturday nite stays in King...</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>8/23/2009</td>\n",
       "      <td>2RsvT8p0SuAC25bhAi3EIw</td>\n",
       "      <td>bMKlvA-zWF4jU3OJCVbVlA</td>\n",
       "      <td>cQnY_VneZisfUAqcbuEuKg</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>It is a beautiful Saturday afternoon and my wi...</td>\n",
       "      <td>N</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>10/28/2011</td>\n",
       "      <td>LM-zONQMUNnAuf6NBISrow</td>\n",
       "      <td>9DMoXd0afrTIdpcwcDDVsw</td>\n",
       "      <td>WBU0yq9J8qiYQfI_fh2P1Q</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>A friend told me about this place but I have t...</td>\n",
       "      <td>N</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>6/18/2010</td>\n",
       "      <td>-DoQeDcNYFdmhOYcgx2MjQ</td>\n",
       "      <td>PyUn2FeMuLdmyB6xxMe4NA</td>\n",
       "      <td>-pO0hsi0xlF4FwqLGJUizg</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>I went to Uncommon Ground for brunch on a Sund...</td>\n",
       "      <td>N</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id        Date               review ID             reviewer ID  \\\n",
       "0   0   5/17/2009  0dFa6egshOwhusL8aSMw-Q  8GC6cFcby0stKarnzL9i2w   \n",
       "1   1  10/25/2011  htQgJ_Z0ADA_QHeKthfeFw  88KSdQ5IMdpCkOidmq1udg   \n",
       "2   2   8/23/2009  2RsvT8p0SuAC25bhAi3EIw  bMKlvA-zWF4jU3OJCVbVlA   \n",
       "3   3  10/28/2011  LM-zONQMUNnAuf6NBISrow  9DMoXd0afrTIdpcwcDDVsw   \n",
       "4   4   6/18/2010  -DoQeDcNYFdmhOYcgx2MjQ  PyUn2FeMuLdmyB6xxMe4NA   \n",
       "\n",
       "               product ID  rating_Helpful  rating_Thanks  rating_LoveThis  \\\n",
       "0  dKcO9OQ44RPRlkWe-vToFA               0              0                0   \n",
       "1  NkOir65b_YAAQVlJR_zmJA               0              0                0   \n",
       "2  cQnY_VneZisfUAqcbuEuKg               0              0                0   \n",
       "3  WBU0yq9J8qiYQfI_fh2P1Q               0              1                1   \n",
       "4  -pO0hsi0xlF4FwqLGJUizg               0              2                0   \n",
       "\n",
       "   rating_OhNo                                            reviews Label  \n",
       "0            4  Just got back from Shaw's. Great oysters. They...     Y  \n",
       "1            2  Back from friday & saturday nite stays in King...     Y  \n",
       "2            5  It is a beautiful Saturday afternoon and my wi...     N  \n",
       "3            5  A friend told me about this place but I have t...     N  \n",
       "4            5  I went to Uncommon Ground for brunch on a Sund...     N  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data = pd.read_csv(\"./data/training_data.csv\")  \n",
    "train_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### loading testing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>Date</th>\n",
       "      <th>review ID</th>\n",
       "      <th>reviewer ID</th>\n",
       "      <th>product ID</th>\n",
       "      <th>rating_Helpful</th>\n",
       "      <th>rating_Thanks</th>\n",
       "      <th>rating_LoveThis</th>\n",
       "      <th>rating_OhNo</th>\n",
       "      <th>reviews</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>11/25/2007</td>\n",
       "      <td>EpUIAOmCal3KLpwfRPwaSw</td>\n",
       "      <td>y5-7amVLpxYyg3EsaV_nSw</td>\n",
       "      <td>tW2jfL-qMccAYZSghPBbHA</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>Great pizza, good location, and a parking lot....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>3/16/2009</td>\n",
       "      <td>WP8YNEOrIYkA-JD1pj4SoA</td>\n",
       "      <td>8wtvJvvxDehPtYgep_525Q</td>\n",
       "      <td>LMaoM2Ue2BR_HI9ba3JsZg</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>This is my favorite place in Chicago. The food...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>11/9/2009</td>\n",
       "      <td>fIklWlw56IGRosS</td>\n",
       "      <td>LdBKnXa6JeePsi_SVuSdHQ</td>\n",
       "      <td>O6uWHgJzylSjWjPSJKGhnQ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>A few friends and I were visiting our other fr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>5/12/2010</td>\n",
       "      <td>7wVIW6OChqj4Y4y7OiuLVw</td>\n",
       "      <td>w2mghTRdP5THiUPSpxVl9g</td>\n",
       "      <td>DXwSYgiXqIVNdO9dazel6w</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>How do I put this politely without offending m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>8/26/2012</td>\n",
       "      <td>GdMImdnQta4l3AkQILj2HA</td>\n",
       "      <td>oq6T6FcKl0TA9LV_970_8Q</td>\n",
       "      <td>p9aMkgTdOKhsjkkv4G0QBw</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Traveling through Chicago for business, they b...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id        Date               review ID             reviewer ID  \\\n",
       "0   0  11/25/2007  EpUIAOmCal3KLpwfRPwaSw  y5-7amVLpxYyg3EsaV_nSw   \n",
       "1   1   3/16/2009  WP8YNEOrIYkA-JD1pj4SoA  8wtvJvvxDehPtYgep_525Q   \n",
       "2   2   11/9/2009         fIklWlw56IGRosS  LdBKnXa6JeePsi_SVuSdHQ   \n",
       "3   3   5/12/2010  7wVIW6OChqj4Y4y7OiuLVw  w2mghTRdP5THiUPSpxVl9g   \n",
       "4   4   8/26/2012  GdMImdnQta4l3AkQILj2HA  oq6T6FcKl0TA9LV_970_8Q   \n",
       "\n",
       "               product ID  rating_Helpful  rating_Thanks  rating_LoveThis  \\\n",
       "0  tW2jfL-qMccAYZSghPBbHA               0              0                0   \n",
       "1  LMaoM2Ue2BR_HI9ba3JsZg               0              0                0   \n",
       "2  O6uWHgJzylSjWjPSJKGhnQ               0              0                0   \n",
       "3  DXwSYgiXqIVNdO9dazel6w               1              0                1   \n",
       "4  p9aMkgTdOKhsjkkv4G0QBw               0              0                0   \n",
       "\n",
       "   rating_OhNo                                            reviews  \n",
       "0            4  Great pizza, good location, and a parking lot....  \n",
       "1            5  This is my favorite place in Chicago. The food...  \n",
       "2            4  A few friends and I were visiting our other fr...  \n",
       "3            3  How do I put this politely without offending m...  \n",
       "4            3  Traveling through Chicago for business, they b...  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data = pd.read_csv('./data/testing_data.csv')\n",
    "test_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### keeping only the reviews column and dropping ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "columns_to_drop = ['id', 'Date', 'review ID','reviewer ID','product ID','rating_Helpful','rating_Thanks','rating_LoveThis','rating_OhNo']\n",
    "test_data.drop(columns=columns_to_drop, inplace=True)\n",
    "train_data.drop(columns=columns_to_drop, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>reviews</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Great pizza, good location, and a parking lot....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>This is my favorite place in Chicago. The food...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A few friends and I were visiting our other fr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>How do I put this politely without offending m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Traveling through Chicago for business, they b...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             reviews\n",
       "0  Great pizza, good location, and a parking lot....\n",
       "1  This is my favorite place in Chicago. The food...\n",
       "2  A few friends and I were visiting our other fr...\n",
       "3  How do I put this politely without offending m...\n",
       "4  Traveling through Chicago for business, they b..."
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>reviews</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Just got back from Shaw's. Great oysters. They...</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Back from friday &amp; saturday nite stays in King...</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>It is a beautiful Saturday afternoon and my wi...</td>\n",
       "      <td>N</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A friend told me about this place but I have t...</td>\n",
       "      <td>N</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>I went to Uncommon Ground for brunch on a Sund...</td>\n",
       "      <td>N</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             reviews Label\n",
       "0  Just got back from Shaw's. Great oysters. They...     Y\n",
       "1  Back from friday & saturday nite stays in King...     Y\n",
       "2  It is a beautiful Saturday afternoon and my wi...     N\n",
       "3  A friend told me about this place but I have t...     N\n",
       "4  I went to Uncommon Ground for brunch on a Sund...     N"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code initializes a BERT (Bidirectional Encoder Representations from Transformers) model for sequence classification using the `BertForSequenceClassification` class from the Hugging Face Transformers library. The model is loaded with pre-trained weights from the 'bert-base-uncased' model checkpoint, which is a BERT model trained on uncased English text. \n",
    "\n",
    "The `num_labels` parameter is set to 2, indicating that the model will be used for binary classification tasks, where the target labels are either 0 or 1. This parameter specifies the number of output labels the model will predict.\n",
    "\n",
    "This cell sets up the neural network model architecture for performing sequence classification tasks on text data, leveraging the power of pre-trained BERT embeddings for feature extraction and classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\akeba\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\huggingface_hub\\file_download.py:149: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\\Users\\akeba\\.cache\\huggingface\\hub\\models--bert-base-uncased. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.\n",
      "To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development\n",
      "  warnings.warn(message)\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code cell is responsible for training and evaluating the BERT-based sequence classification model on the provided data. \n",
    "\n",
    "First, it initializes parameters such as `batch_size` (number of samples in each batch), `max_length` (maximum length of input sequences), `epochs` (number of training epochs), and `learning_rate` (initial learning rate for the optimizer).\n",
    "\n",
    "Then, it prepares the training dataset using the `CustomDataset` class initialized earlier, creates a DataLoader to iterate over batches of data during training, and initializes the AdamW optimizer with the specified learning rate.\n",
    "\n",
    "During training, it iterates over each epoch and each batch of data, computes the model's loss, performs backpropagation, and updates the model's parameters.\n",
    "\n",
    "After training, it evaluates the model's performance on the test set using the same data processing steps as in training, calculates the accuracy of the model's predictions compared to the ground truth labels, and prints the test accuracy.\n",
    "\n",
    "This cell effectively trains the BERT model for sequence classification and evaluates its performance on unseen test data, providing insights into the model's effectiveness in classifying text data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 8\n",
    "max_length = 128\n",
    "epochs = 3\n",
    "learning_rate = 2e-5\n",
    "\n",
    "\n",
    "train_dataset = CustomDataset(train_data, tokenizer, max_length)\n",
    "train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\n",
    "\n",
    "\n",
    "optimizer = AdamW(model.parameters(), lr=learning_rate)\n",
    "total_steps = len(train_loader) * epochs\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    model.train()\n",
    "    total_loss = 0\n",
    "\n",
    "    for batch in train_loader:\n",
    "        input_ids = batch['input_ids']\n",
    "        attention_mask = batch['attention_mask']\n",
    "        labels = batch['labels']\n",
    "\n",
    "        optimizer.zero_grad()\n",
    "        outputs = model(input_ids, attention_mask=attention_mask, labels=labels)\n",
    "        loss = outputs.loss\n",
    "        total_loss += loss.item()\n",
    "\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "    avg_train_loss = total_loss / len(train_loader)\n",
    "    print(f\"Epoch {epoch + 1}/{epochs}, Average Training Loss: {avg_train_loss:.4f}\")\n",
    "\n",
    "# Evaluation on test set\n",
    "test_dataset = CustomDataset(test_data, tokenizer, max_length)\n",
    "test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)\n",
    "\n",
    "model.eval()\n",
    "all_preds = []\n",
    "all_labels = []\n",
    "\n",
    "for batch in test_loader:\n",
    "    input_ids = batch['input_ids']\n",
    "    attention_mask = batch['attention_mask']\n",
    "    labels = batch['labels']\n",
    "\n",
    "    with torch.no_grad():\n",
    "        outputs = model(input_ids, attention_mask=attention_mask)\n",
    "    \n",
    "    logits = outputs.logits\n",
    "    preds = torch.argmax(logits, dim=1).cpu().numpy()\n",
    "    all_preds.extend(preds)\n",
    "    all_labels.extend(labels.cpu().numpy())\n",
    "\n",
    "accuracy = accuracy_score(all_labels, all_preds)\n",
    "print(f\"Test Accuracy: {accuracy:.4f}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}