06_LA_CAG.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# No.1 accuracy in multiform table extraction \n",
    "- Convert documents to maximize RAG performance \n",
    "- LangChain provides powerful tools for text splitting and vectorization\n",
    "\n",
    "\n",
    "![Layout Analyzer](./figures/la.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip3 install -qU  markdownify  langchain-upstage  requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "%load_ext dotenv\n",
    "%dotenv\n",
    "# UPSTAGE_API_KEY"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Layout Analyzer](./figures/solar_sample.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_upstage import UpstageLayoutAnalysisLoader\n",
    "\n",
    "\n",
    "layzer = UpstageLayoutAnalysisLoader(\"pdfs/solar_sample.pdf\", output_type=\"html\")\n",
    "# For improved memory efficiency, consider using the lazy_load method to load documents page by page.\n",
    "docs = layzer.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table id='0' style='font-size:14px'><tr><td>Model</td><td>Size</td><td>Type</td><td>H6 (Avg.)</td><td>ARC</td><td>HellaSwag</td><td>MMLU</td><td>TruthfulQA</td><td>Winogrande</td><td>GSM8K</td></tr><tr><td>SOLAR 10.7B-Instruct</td><td>⇠ 11B</td><td>Alignment-tuned</td><td>74.20</td><td>71.08</td><td>88.16</td><td>66.21</td><td>71.43</td><td>83.58</td><td>64.75</td></tr><tr><td>Qwen 72B</td><td>⇠ 72B</td><td>Pretrained</td><td>73.60</td><td>65.19</td><td>85.94</td><td>77.37</td><td>60.19</td><td>82.48</td><td>70.43</td></tr><tr><td>Mixtral 8x7B-Instruct-v0.1</td><td>⇠ 47B</td><td>Instruction-tuned</td><td>72.62</td><td>70.22</td><td>87.63</td><td>71.16</td><td>64.58</td><td>81.37</td><td>60.73</td></tr><tr><td>Yi 34B-200K</td><td>⇠ 34B</td><td>Pretrained</td><td>70.81</td><td>65.36</td><td>85.58</td><td>76.06</td><td>53.64</td><td>82.56</td><td>61.64</td></tr><tr><td>Yi 34B</td><td>⇠ 34B</td><td>Pretrained</td><td>69.42</td><td>64.59</td><td>85.69</td><td>76.35</td><td>56.23</td><td>83.03</td><td>50.64</td></tr><tr><td>Mixtral 8x7B-v0.1</td><td>⇠ 47B</td><td>Pretrained</td><td>68.42</td><td>66.04</td><td>86.49</td><td>71.82</td><td>46.78</td><td>81.93</td><td>57.47</td></tr><tr><td>Llama 2 70B</td><td>⇠ 70B</td><td>Pretrained</td><td>67.87</td><td>67.32</td><td>87.33</td><td>69.83</td><td>44.92</td><td>83.74</td><td>54.06</td></tr><tr><td>Falcon 180B</td><td>⇠ 180B</td><td>Pretrained</td><td>67.85</td><td>69.45</td><td>88.86</td><td>70.50</td><td>45.47</td><td>86.90</td><td>45.94</td></tr><tr><td>SOLAR 10.7B</td><td>⇠ 11B</td><td>Pretrained</td><td>66.04</td><td>61.95</td><td>84.60</td><td>65.48</td><td>45.04</td><td>83.66</td><td>55.50</td></tr><tr><td>Qwen 14B</td><td>⇠ 14B</td><td>Pretrained</td><td>65.86</td><td>58.28</td><td>83.99</td><td>67.70</td><td>49.43</td><td>76.80</td><td>58.98</td></tr><tr><td>Mistral 7B-Instruct-v0.2</td><td>⇠ 7B</td><td>Instruction-tuned</td><td>65.71</td><td>63.14</td><td>84.88</td><td>60.78</td><td>68.26</td><td>77.19</td><td>40.03</td></tr><tr><td>Yi 34B-Chat</td><td>⇠ 34B</td><td>Instruction-tuned</td><td>65.32</td><td>65.44</td><td>84.16</td><td>74.90</td><td>55.37</td><td>80.11</td><td>31.92</td></tr><tr><td>Mistral 7B</td><td>⇠ 7B</td><td>Pretrained</td><td>60.97</td><td>59.98</td><td>83.31</td><td>64.16</td><td>42.15</td><td>78.37</td><td>37.83</td></tr></table><br><p id='1' style='font-size:16px'>Table 2: Evaluation results in the Open LLM Leaderboard for SOLAR 10.7B and SOLAR 10.7B-Instruct along with<br>other top-performing models. We report the scores for the six tasks mentioned in Sec. 4.1 along with the H6 score<br>(average of six tasks). We also report the size of the models in units of billions of parameters. The type indicates the<br>training stage of the model and is chosen from {Pretrained, Instruction-tuned, Alignment-tuned}. Models based on<br>SOLAR 10.7B are colored purple. The best scores for H6 and the individual tasks are shown in bold.</p><p id='2' style='font-size:20px'>MetaMathQA ( Yu et al. , 2023 ) dataset.</p><br><p id='3' style='font-size:20px'>We reformatted the instruction datasets with an<br>Alpaca-styled chat template. For datasets such as<br>OpenOrca, which are derived from FLAN ( Long-<br>pre et al. , 2023 ), we ﬁlter data that overlaps with<br>the benchmark datasets (see Tab. 8 in Appendix. C<br>for more information). The alignment datasets<br>are in the {prompt, chosen, rejected} triplet for-<br>mat. We preprocess the alignment datasets follow-<br>ing Zephyr ( Tunstall et al. , 2023 ). We use Data-<br>verse ( Park et al. , 2024 ) for data preprocessing.</p><br><p id='4' style='font-size:20px'>Evaluation. In the HuggingFace Open LLM<br>Leaderboard ( Beeching et al. , 2023 ), six types of<br>evaluation methods are presented: ARC ( Clark<br>et al. , 2018 ), HellaSWAG ( Zellers et al. , 2019 ),<br>MMLU ( Hendrycks et al. , 2020 ), TruthfulQA ( Lin<br>et al. , 2022 ), Winogrande ( Sakaguchi et al. , 2021 ),<br>and GSM8K ( Cobbe et al. , 2021 ). We utilize these<br>datasets as benchmarks for evaluation and also re-<br>port the average scores for the six tasks, e.g., H6.<br>We either submit directly to the Open LLM Leader-<br>board or utilize Evalverse ( Kim et al. , 2024b ) for<br>running evaluations locally.</p><br><p id='5' style='font-size:18px'>Model merging. Model merging methods such<br>as Yadav et al. ( 2023 ) can boost model perfor-<br>mance without further training. We merge some<br>of the models that we trained in both the instruc-<br>tion and alignment tuning stages. We implement<br>our own merging methods although popular open<br>source also exist such as MergeKit 3 .</p><br><p id='6' style='font-size:22px'>4.2 Main Results</p><br><p id='7' style='font-size:20px'>We present evaluation results for our SOLAR<br>10.7B and SOLAR 10.7B-Instruct models along</p><br><p id='8' style='font-size:14px'>3 https://github.com/cg123/mergekit</p><br><p id='9' style='font-size:20px'>with other top-performing models in Tab. 2 . SO-<br>LAR 10.7B ou"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import display, HTML\n",
    "\n",
    "display(HTML(docs[0].page_content[:5000]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.prompts import PromptTemplate\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_upstage import ChatUpstage\n",
    "\n",
    "\n",
    "llm = ChatUpstage()\n",
    "\n",
    "prompt_template = PromptTemplate.from_template(\n",
    "    \"\"\"\n",
    "    Please provide most correct answer from the following context. \n",
    "    Think step by step and look the html tags and table values carefully to provide the most correct answer.\n",
    "    If the answer is not present in the context, please write \"The information is not present in the context.\"\n",
    "    ---\n",
    "    Question: {question}\n",
    "    ---\n",
    "    Context: {Context}\n",
    "    \"\"\"\n",
    ")\n",
    "chain = prompt_template | llm | StrOutputParser()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Table 2 presents the evaluation results for the Open LLM Leaderboard, which includes SOLAR 10.7B and SOLAR 10.7B-Instruct along with other top-performing models. The table reports the scores for six tasks mentioned in Sec. 4.1, along with the H6 score (average of six tasks). The table also provides information on the size of the models in billions of parameters and the type of training stage, which can be chosen from {Pretrained, Instruction-tuned, Alignment-tuned}. Models based on SOLAR 10.7B are colored purple, and the best scores for H6 and the individual tasks are shown in bold.'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.invoke({\"question\": \"Explain Table 2?\", \"Context\": docs})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The MMLU scores of SOLAR 10.7B is 65.48.'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.invoke({\"question\": \"What is MMLU scores of SOLAR 10.7B?\", \"Context\": docs})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'MMLU scores of Mistral 7B-Instruct-v0.2 is 60.78.'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.invoke({\"question\": \"What is MMLU scores of Mistral 7B-Instruct-v0.2?\", \"Context\": docs})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Excercise \n",
    "Sometimes, even if we provide a table in Markdown or HTML format, the Large Language Model (LLM) may not extract the information correctly. How can you fix this issue?\n",
    "\n",
    "Hint: Consider using CoT, a few-shot learning approach or a divide and conquer strategy. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}