GraderGPT_SimpleNotebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: Prepping Submissions & Custom Criteria Descriptions\n",
    "\n",
    "You should only need to run the code blocks in Part 1 once for an assignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install pdf2docx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build custom file for manual criterion descriptions\n",
    "\n",
    "After you populate the `custom_description` column, save the file manually again with the 'UTF-8' encoding.\n",
    "For me on Windows, if I don't explicitly state that, it won't load properly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from simpleHelper import *\n",
    "\n",
    "pathToAssgnCSV = os.path.join('data','MOVESCI_110_FA_2023_2060736_assignmentsJK.csv')\n",
    "pathToGradingsCSV = os.path.join('data','MOVESCI_110_FA_2023_2060736_gradings.csv')\n",
    "pathToRubricsCSV = os.path.join('data','MOVESCI_110_FA_2023_2060736_rubrics.csv')\n",
    "\n",
    "pathToCriterionCSV = os.path.join('data','2060736_criterion.csv')\n",
    "\n",
    "\n",
    "makeBlankCriterionFile(pathToAssgnCSV, pathToGradingsCSV, pathToRubricsCSV, pathToCriterionCSV)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate text versions of submission files\n",
    "\n",
    "You will need to run this notebook cell to convert the submissions into a format usable by the notebook.\n",
    "\n",
    "The `data/submissions_xxxxxx` should contain the unzipped submission folders that are exported from Canvas. The folder name can be anything, you just need to specify it in the `originalSubmissionFolder` variable. You will also need the Assignment ID in variable `assignmentID`.\n",
    "\n",
    "A folder for converted submissions is automatically made called `data/Converted submissions_xxxxxx` where xxxxxx is the Assignment ID."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from simpleHelper import *\n",
    "\n",
    "originalSubmissionFolder = os.path.join('data', 'submissions_2060736')\n",
    "assignmentID = 2060736\n",
    "\n",
    "convertSubmissions(originalSubmissionFolder, assignmentID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2: Getting ChatGPT responses for submissions\n",
    "\n",
    "You need to provide the full file path to each of the 4 CSV files. You also need to specify your OpenAI key details, the course code, and the assignment ID.\n",
    "The `courseName` variable simply is the real name of the course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from simpleHelper import *\n",
    "from tqdm import tqdm\n",
    "\n",
    "pathToAssgnCSV = os.path.join('data','coursename_courseid_assignmentsJK.csv')\n",
    "pathToGradingsCSV = os.path.join('data','coursename_courseid_gradings.csv')\n",
    "pathToRubricsCSV = os.path.join('data','coursename_courseid__rubrics.csv')\n",
    "\n",
    "pathToCriterionCSV = os.path.join('data','courseid__criterion.csv')\n",
    "\n",
    "courseCode = 626158\n",
    "assignmentID = 2060736\n",
    "courseName = 'Movement Science'\n",
    "\n",
    "overWriteSave = False\n",
    "\n",
    "customDescMode = True\n",
    "critDescDF = pd.read_csv(pathToCriterionCSV).drop_duplicates()\n",
    "\n",
    "saveFolder = 'data/saves'\n",
    "errorFolder = 'data/error'\n",
    "if not os.path.exists(saveFolder):\n",
    "        os.mkdir(saveFolder)\n",
    "if not os.path.exists(errorFolder):\n",
    "        os.mkdir(errorFolder)\n",
    "    \n",
    "gradeRubricAssignmentDF = getGRAData(pathToAssgnCSV, pathToGradingsCSV, pathToRubricsCSV)\n",
    "print(gradeRubricAssignmentDF)\n",
    "print(f\"gradeRubricAssignmentDF size {gradeRubricAssignmentDF.shape[0]}\")\n",
    "print(assignmentID)\n",
    "\n",
    "gradeRubricAssignmentDF = gradeRubricAssignmentDF[gradeRubricAssignmentDF['assignment_id']==assignmentID]\n",
    "\n",
    "print(f\"gradeRubricAssignmentDF size {gradeRubricAssignmentDF.shape[0]}\")\n",
    "\n",
    "for index, row in (pbar := tqdm(gradeRubricAssignmentDF.iterrows(), total=gradeRubricAssignmentDF.shape[0])):\n",
    "    pbar.set_description(f\"Processing: {assignmentID}-{row['submitter_id']}\")\n",
    "    if checkIfSaved(row['assignment_id'], row['submitter_id'], saveFolder, errorFolder) and not overWriteSave:\n",
    "        # print(f\"Already saved: {assignmentID}-{row['submitter_id']}\")\n",
    "        continue\n",
    "    else:\n",
    "        dataDict, runSuccess = processGRARow(row, courseName, customDescMode, critDescDF)\n",
    "        saveOutputasPickle(dataDict, runSuccess, saveFolder, errorFolder)\n",
    "\n",
    "print(f\"gradeRubricAssignmentDF size {gradeRubricAssignmentDF.shape[0]}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install \"pandas<2.0.0\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3: Analyzing the results and making charts and tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resultsDF = convertPicklesToDF(saveFolder)\n",
    "errorDF = convertPicklesToDF(errorFolder)\n",
    "\n",
    "excelFolder = 'data/excelJK'\n",
    "if not os.path.exists(excelFolder):\n",
    "        os.mkdir(excelFolder)\n",
    "chartFolder = 'data/chartsJK'\n",
    "if not os.path.exists(chartFolder):\n",
    "        os.mkdir(chartFolder)\n",
    "\n",
    "saveName = f'{courseName}-{assignmentID}'\n",
    "        \n",
    "mergedCriterionData = getCriterionDataDF(resultsDF, saveName, excelFolder)\n",
    "\n",
    "# getScoreSpread(resultsDF, saveName, chartFolder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "meanDiffDF, meanDiffPercentDF = saveGraderPeerGPTMeanScoreDiff(resultsDF, saveName, excelFolder)\n",
    "\n",
    "display(meanDiffPercentDF)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from simpleHelper import buildFullInfoDF\n",
    "#fullInfoDF, rubricOrderDict = buildFullInfoDF(gradeRubricAssignmentDF, resultsDF, saveName, excelFolder)\n",
    "print(\"sss\")\n",
    "critDataDF = pd.DataFrame()\n",
    "for index,row in resultsDF.iterrows():\n",
    "    criterionData = row['data_peerGPT']\n",
    "    for col in ['submitter_id', 'assignment_id', 'grader_id']:\n",
    "        criterionData[col] = row[col]\n",
    "    critDataDF = pd.concat([critDataDF, criterionData])\n",
    "allCritDF = critDataDF.drop(['mastery_points','ignore_for_scoring','title','peerGPT_criterion_id','description_grade'],\n",
    "                        axis=1, errors='ignore')\n",
    "\n",
    "print(f\"allCritDF size {allCritDF.shape[0]}\")\n",
    "\n",
    "meanInfoList = []\n",
    "for group in allCritDF.groupby(['assignment_id','criterion_id','grader_id']):\n",
    "    meanInfoList.append({'assignment_id':group[0][0], 'criterion_id':group[0][1], 'grader_id':group[0][2], \\\n",
    "                        'Grader Mean':group[1]['points_grade'].mean(), \\\n",
    "                        'Grader Std. Dev.':group[1]['points_grade'].std(), \\\n",
    "                        'peerGPT Mean':group[1]['peerGPT_criterion_score'].mean(), \\\n",
    "                        'peerGPT Std. Dev.':group[1]['peerGPT_criterion_score'].std(), \\\n",
    "                        # 'Correlation Score':group[1]['peerGPT_criterion_score'].corr(group[1]['points_grade']), \\\n",
    "                        })\n",
    "meanInfoDF = pd.DataFrame(meanInfoList)\n",
    "meanInfoDF['Mean Difference'] = meanInfoDF['peerGPT Mean'] - meanInfoDF['Grader Mean']\n",
    "print(f\"gradeRubricAssignmentDF size {gradeRubricAssignmentDF.shape[0]}\")\n",
    "assignmentDF = gradeRubricAssignmentDF[['assignment_id', 'assignment_title']].drop_duplicates()\n",
    "print(f\"assignmentDF size {assignmentDF.shape[0]}\")\n",
    "\n",
    "baseInfoDF = allCritDF[['assignment_id', 'criterion_id', 'description_rubric', 'points_rubric']].drop_duplicates()\n",
    "baseInfoDF = baseInfoDF.merge(assignmentDF, on='assignment_id')\n",
    "print(f\"baseInfoDF size {baseInfoDF.shape[0]}\")\n",
    "\n",
    "globalMeanList = [{'assignment_id':group[0][0], 'criterion_id':group[0][1], \\\n",
    "                'All Graders Mean':group[1]['points_grade'].mean(), \\\n",
    "                'All Graders Std. Dev.':group[1]['points_grade'].std(), \\\n",
    "                'Global peerGPT Mean':group[1]['peerGPT_criterion_score'].mean(), \\\n",
    "                'Global peerGPT Std. Dev.':group[1]['peerGPT_criterion_score'].std()} \\\n",
    "                    for group in allCritDF.groupby(['assignment_id', 'criterion_id'])]\n",
    "globalMeanDF = pd.DataFrame(globalMeanList)\n",
    "\n",
    "baseInfoDF = baseInfoDF.merge(globalMeanDF, on=['assignment_id', 'criterion_id'])\n",
    "\n",
    "fullInfoDF = meanInfoDF.merge(baseInfoDF, on=['assignment_id', 'criterion_id'])\n",
    "fullInfoDF['Mean Difference %'] = 100*fullInfoDF['Mean Difference'].div(fullInfoDF['points_rubric'])\n",
    "print(fullInfoDF)\n",
    "fullInfoDF['Grader Mean Diff. %'] = 100*(fullInfoDF['All Graders Mean'] - fullInfoDF['Grader Mean']).div(fullInfoDF['points_rubric'])\n",
    "print(fullInfoDF)\n",
    "print(f\"sSaving file at: {os.path.join(excelFolder, saveName+' - Grader Difference Table.xlsx')}\")\n",
    "fullInfoDF.to_excel(os.path.join(excelFolder, saveName+' - Grader Difference Table.xlsx'))\n",
    "\n",
    "rubricInfo = gradeRubricAssignmentDF[['assignment_id', 'data_rubric']].drop_duplicates('assignment_id').reset_index(drop=True)\n",
    "rubricOrderDict = {}\n",
    "for index, row in rubricInfo.iterrows():\n",
    "    rubricOrderDict[row['assignment_id']] = pd.DataFrame(row['data_rubric'])['description'].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "getZScoreAndCI(fullInfoDF, saveName, excelFolder, confidence=0.93)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "getMeanDiffPercentCharts(fullInfoDF, rubricOrderDict, courseName, chartFolder)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}