language/use-cases/document-summarization/summarization_large_documents_langchain.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "uxCkB_DXTHzf"
   },
   "outputs": [],
   "source": [
    "# Copyright 2023 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Hny4I-ODTIS6"
   },
   "source": [
    "# Text Summarization of Large Documents using LangChain 🦜🔗\n",
    "\n",
    "> **NOTE:** This notebook uses the PaLM generative model, which will reach its [discontinuation date in October 2024](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text#model_versions). Please refer to [this updated notebook](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/document_processing.ipynb) for a version which uses the latest Gemini model.\n",
    "\n",
    "<table align=\"left\">\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/language/use-cases/document-summarization/summarization_large_documents_langchain.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Run in Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/language/use-cases/document-summarization/summarization_large_documents_langchain.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/language/use-cases/document-summarization/summarization_large_documents_langchain.ipynb\">\n",
    "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
    "    </a>\n",
    "  </td>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| | |\n",
    "|-|-|\n",
    "|Author(s) | [Polong Lin](https://github.com/polong-lin) |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-nLS57E2TO5y"
   },
   "source": [
    "## Overview\n",
    "\n",
    "Text summarization is an NLP task that creates a concise and informative summary of a longer text. LLMs can be used to create summaries of news articles, research papers, technical documents, and other types of text.\n",
    "\n",
    "Summarizing large documents can be challenging. To create summaries, you need to apply summarization strategies to your indexed documents. You have already seen some of these strategies in the previous notebooks. If you haven't completed it, it is recommended to do so to have a basic understanding of how to summarize large documents.\n",
    "\n",
    "In this notebook, you will use LangChain, a framework for developing LLM applications, to apply some summarization strategies. The notebook covers several examples of how to summarize large documents."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iXsvgIuwTPZw"
   },
   "source": [
    "### Objective\n",
    "\n",
    "In this tutorial, you learn how to use LangChain with PaLM API to summarize large documents by working through the following examples:\n",
    "\n",
    "- Stuffing method\n",
    "- MapReduce method\n",
    "- Refine method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "skXAu__iqks_"
   },
   "source": [
    "### Costs\n",
    "\n",
    "This tutorial uses billable components of Google Cloud:\n",
    "- Vertex AI Generative AI Studio\n",
    "\n",
    "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mvKl-BtQTRiQ"
   },
   "source": [
    "## Getting Started"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PwFMpIMrTV_4"
   },
   "source": [
    "### Install Vertex AI SDK & Other dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8aP6JVlZkS-m"
   },
   "outputs": [],
   "source": [
    "!sudo apt -y -qq install tesseract-ocr\n",
    "!sudo apt -y -qq install libtesseract-dev\n",
    "!sudo apt-get -y -qq install poppler-utils #required by PyPDF2 for page count and other pdf utilities\n",
    "!sudo apt-get -y -qq install python-dev libxml2-dev libxslt1-dev antiword unrtf poppler-utils pstotext tesseract-ocr flac ffmpeg lame libmad0 libsox-fmt-mp3 sox libjpeg-dev swig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oDmNq5__Trl4"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    USER = \"--user\"\n",
    "else:\n",
    "    USER = \"\"\n",
    "\n",
    "! pip3 install {USER} --upgrade pytesseract pypdf PyPDF2 textract langchain transformers google-cloud-aiplatform --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jWwtjLV5TY6H"
   },
   "source": [
    "**Colab only**: Run the following cell to restart the kernel. For Vertex AI Workbench you can restart the terminal using the button on top."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XRvKdaPDTznN"
   },
   "outputs": [],
   "source": [
    "# Automatically restart kernel after installs so that your environment can access the new packages\n",
    "import IPython\n",
    "\n",
    "app = IPython.Application.instance()\n",
    "app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "opUxT_k5TdgP"
   },
   "source": [
    "### Authenticating your notebook environment\n",
    "\n",
    "- If you are using **Colab** to run this notebook, run the cell below and continue.\n",
    "- If you are using **Vertex AI Workbench**, check out the setup instructions [here](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/setup-env)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vbNgv4q1T2Mi"
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "if \"google.colab\" in sys.modules:\n",
    "    from google.colab import auth\n",
    "\n",
    "    auth.authenticate_user()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n5fXfvzhTkYN"
   },
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Colab only:** Run the following cell to initialize the Vertex AI SDK. For Vertex AI Workbench, you don't need to run this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GjSsu6cmUdEx"
   },
   "outputs": [],
   "source": [
    "import vertexai\n",
    "\n",
    "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
    "REGION = \"us-central1\"  # @param {type:\"string\"}\n",
    "\n",
    "vertexai.init(project=PROJECT_ID, location=REGION)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cRkcfnQMT9vD"
   },
   "outputs": [],
   "source": [
    "import urllib\n",
    "import warnings\n",
    "from pathlib import Path as p\n",
    "\n",
    "import pandas as pd\n",
    "from langchain import PromptTemplate\n",
    "from langchain.chains.summarize import load_summarize_chain\n",
    "from langchain.document_loaders import PyPDFLoader\n",
    "from langchain.llms import VertexAI\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DAGaTjPVTmhP"
   },
   "source": [
    "### Import models\n",
    "\n",
    "You load the pre-trained text generation model called `text-bison`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ITUmZiNZcMUW"
   },
   "outputs": [],
   "source": [
    "vertex_llm_text = VertexAI(model_name=\"text-bison\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cKG-ZTJ_02wq"
   },
   "source": [
    "## Summarization with Large Documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HZkLDRTjTcfm"
   },
   "source": [
    "### Preparing data files\n",
    "\n",
    "To begin, you will need to download a few files that are required for the summarizing tasks below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7H0zINHpTaSu"
   },
   "outputs": [],
   "source": [
    "data_folder = p.cwd() / \"data\"\n",
    "p(data_folder).mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "pdf_url = \"https://services.google.com/fh/files/misc/practitioners_guide_to_mlops_whitepaper.pdf\"\n",
    "pdf_file = str(p(data_folder, pdf_url.split(\"/\")[-1]))\n",
    "\n",
    "urllib.request.urlretrieve(pdf_url, pdf_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JELITHdBhnf0"
   },
   "source": [
    "### Extract text from the PDF\n",
    "\n",
    "You use an `PdfReader` to extract the text from our scanned documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "x3INtovxreI_"
   },
   "outputs": [],
   "source": [
    "pdf_loader = PyPDFLoader(pdf_file)\n",
    "pages = pdf_loader.load_and_split()\n",
    "print(pages[3].page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZDVwBFSjZ7ws"
   },
   "source": [
    "## Method 1: Stuffing\n",
    "\n",
    "Stuffing is the simplest method to pass data to a language model. It \"stuffs\" text into the prompt as context in a way that all of the relevant information can be processed by the model to get what you want.\n",
    "\n",
    "In LangChain, you can use `StuffDocumentsChain` as part of the `load_summarize_chain` method. What you need to do is setting `stuff` as `chain_type` of your chain."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uhEi-XqKnv2v"
   },
   "source": [
    "### Prompt design with `Stuffing` chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "B-ljajUen1YO"
   },
   "outputs": [],
   "source": [
    "prompt_template = \"\"\"Write a concise summary of the following text delimited by triple backquotes.\n",
    "              Return your response in bullet points which covers the key points of the text.\n",
    "              ```{text}```\n",
    "              BULLET POINT SUMMARY:\n",
    "  \"\"\"\n",
    "\n",
    "prompt = PromptTemplate(template=prompt_template, input_variables=[\"text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "N5aVrDWkJs3Y"
   },
   "source": [
    "### Retrying\n",
    "Initiate a chain using `stuff` method and process three pages document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "N_hoizIgObe9"
   },
   "outputs": [],
   "source": [
    "stuff_chain = load_summarize_chain(vertex_llm_text, chain_type=\"stuff\", prompt=prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Q1_zwxwgTnlV"
   },
   "outputs": [],
   "source": [
    "three_pages = pages[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0jEUfOn7UFI2"
   },
   "outputs": [],
   "source": [
    "three_pages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QnXUwWxkrLu4"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    print(stuff_chain.run(three_pages))\n",
    "except Exception as e:\n",
    "    print(\n",
    "        \"The code failed since it won't be able to run inference on such a huge context and throws this exception: \",\n",
    "        e,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xKb_fBEedZqu"
   },
   "source": [
    "As you can see, with the `stuff` method, you can summarize the entire document content with a single API call passing all data at once.\n",
    "\n",
    "Depending on the context length of LLM, the `stuff` method would not work as it result in a prompt larger than the context length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RtgemmBzkddX"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    print(stuff_chain.run(pages))\n",
    "except Exception as e:\n",
    "    print(\n",
    "        \"The code failed since it won't be able to run inference on such a huge context and throws this exception: \",\n",
    "        e,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Vtp21WX3T7d_"
   },
   "source": [
    "As expected, the code returns the expection message."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hqZrKM32h-o2"
   },
   "source": [
    "### Considerations\n",
    "\n",
    "The `stuffing` method is a way to summarize text by feeding the entire document to a large language model (LLM) in a single call. This method has both pros and cons.\n",
    "\n",
    "The stuffing method only requires a single call to the LLM, which can be faster than other methods that require multiple calls. When summarizing text, the LLM has access to all the data at once, which can result in a better summary.\n",
    "\n",
    "But, LLMs have a context length, which is the maximum number of tokens that can be processed in a single call. If the document is longer than the context length, the stuffing method will not work. Also the stuffing method is not suitable for summarizing large documents, as it can be slow and may not produce a good summary.\n",
    "\n",
    "Let's explore other approaches to help deal with having longer text than context length limit of LLMs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RM3V1JARZ9-k"
   },
   "source": [
    "## Method 2: MapReduce\n",
    "\n",
    "The `MapReduce` method implements a multi-stage summarization. It is a technique for summarizing large pieces of text by first summarizing smaller chunks of text and then combining those summaries into a single summary.\n",
    "\n",
    "In LangChain, you can use `MapReduceDocumentsChain` as part of the `load_summarize_chain` method. What you need to do is setting `map_reduce` as `chain_type` of your chain."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lagLXEamlPY2"
   },
   "source": [
    "### Prompt design with `MapReduce` chain\n",
    "\n",
    "In our example, you have a 32-page document that you need to summarize.\n",
    "\n",
    "With LangChain, the `map_reduce` chain breaks the document down into 1024 token chunks max. Then it runs the initial prompt you define on each chunk to generate a summary of that chunk. In the example below, you use the following first stage or map prompt.\n",
    "\n",
    "```Write a concise summary of the following text delimited by triple backquotes. Return your response in bullet points which covers the key points of the text.\n",
    "'''{text}'''. BULLET POINT SUMMARY:```\n",
    "\n",
    "Once summaries for all of the chunks are generated, it runs a different prompt to combine those summaries into a single summary. In the example below, you use the following second stage or combine prompt.\n",
    "\n",
    "```Write a summary of the entire document that includes the main points from all of the individual summaries.```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "R6oHEtdSmsTn"
   },
   "outputs": [],
   "source": [
    "map_prompt_template = \"\"\"\n",
    "                      Write a summary of this chunk of text that includes the main points and any important details.\n",
    "                      {text}\n",
    "                      \"\"\"\n",
    "\n",
    "map_prompt = PromptTemplate(template=map_prompt_template, input_variables=[\"text\"])\n",
    "\n",
    "combine_prompt_template = \"\"\"\n",
    "                      Write a concise summary of the following text delimited by triple backquotes.\n",
    "                      Return your response in bullet points which covers the key points of the text.\n",
    "                      ```{text}```\n",
    "                      BULLET POINT SUMMARY:\n",
    "                      \"\"\"\n",
    "\n",
    "combine_prompt = PromptTemplate(\n",
    "    template=combine_prompt_template, input_variables=[\"text\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JXoz0uLDMoWD"
   },
   "source": [
    "### Generate summaries using MapReduce method\n",
    "\n",
    "After defining prompts, you initialize the associated `map_reduce_chain`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VRGJcBZeVdEa"
   },
   "outputs": [],
   "source": [
    "map_reduce_chain = load_summarize_chain(\n",
    "    vertex_llm_text,\n",
    "    chain_type=\"map_reduce\",\n",
    "    map_prompt=map_prompt,\n",
    "    combine_prompt=combine_prompt,\n",
    "    return_intermediate_steps=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-6fekDDr0hrJ"
   },
   "source": [
    "Then, you generate summaries using the chain. Notice that LangChain use a tokenizer (from transformer library) with 1024 token limit by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "uSC6w2TBV35q"
   },
   "outputs": [],
   "source": [
    "map_reduce_outputs = map_reduce_chain({\"input_documents\": pages})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "meH2ELuz2H46"
   },
   "source": [
    "After summaries are generated, you can validate them by organize input documents and associated output in a Pandas DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "r6FRSR7xRLew"
   },
   "outputs": [],
   "source": [
    "final_mp_data = []\n",
    "for doc, out in zip(\n",
    "    map_reduce_outputs[\"input_documents\"], map_reduce_outputs[\"intermediate_steps\"]\n",
    "):\n",
    "    output = {}\n",
    "    output[\"file_name\"] = p(doc.metadata[\"source\"]).stem\n",
    "    output[\"file_type\"] = p(doc.metadata[\"source\"]).suffix\n",
    "    output[\"page_number\"] = doc.metadata[\"page\"]\n",
    "    output[\"chunks\"] = doc.page_content\n",
    "    output[\"concise_summary\"] = out\n",
    "    final_mp_data.append(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dA9cnh8YaNbF"
   },
   "outputs": [],
   "source": [
    "pdf_mp_summary = pd.DataFrame.from_dict(final_mp_data)\n",
    "pdf_mp_summary = pdf_mp_summary.sort_values(\n",
    "    by=[\"file_name\", \"page_number\"]\n",
    ")  # sorting the dataframe by filename and page_number\n",
    "pdf_mp_summary.reset_index(inplace=True, drop=True)\n",
    "pdf_mp_summary.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "yA0eM1K3cvH2"
   },
   "outputs": [],
   "source": [
    "index = 3\n",
    "print(\"[Context]\")\n",
    "print(pdf_mp_summary[\"chunks\"].iloc[index])\n",
    "print(\"\\n\\n [Simple Summary]\")\n",
    "print(pdf_mp_summary[\"concise_summary\"].iloc[index])\n",
    "print(\"\\n\\n [Page number]\")\n",
    "print(pdf_mp_summary[\"page_number\"].iloc[index])\n",
    "print(\"\\n\\n [Source: file_name]\")\n",
    "print(pdf_mp_summary[\"file_name\"].iloc[index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ROrE1-HKpg7y"
   },
   "source": [
    "### Considerations\n",
    "\n",
    "With `MapReduce` method, the model is able to summarize a large paper by overcoming the context limit of `Stuffing` method with parallel processing.\n",
    "\n",
    "However, the `MapReduce` requires multiple calls to the model and potentially losing context between pages.\n",
    "\n",
    "To deal this challenge, you can try another method to summarize multiple pages at a time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lxdB-5PqgCf-"
   },
   "source": [
    "## Method 3: Refine\n",
    "\n",
    "The Refine method is an alternative method to deal with large document summarization. It works by first running an initial prompt on a small chunk of data, generating some output. Then, for each subsequent document, the output from the previous document is passed in along with the new document, and the LLM is asked to refine the output based on the new document.\n",
    "\n",
    "In LangChain, you can use `RefineDocumentsChain` as part of the load_summarize_chain method. What you need to do is setting `refine` as `chain_type` of your chain."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pjj2UZilDF4Q"
   },
   "source": [
    "### Prompt design with `Refine` chain\n",
    "\n",
    "With LangChain, the `refine` chain requires two prompts.\n",
    "\n",
    "The question prompt to generate the output for subsequent task. The refine prompt to refine the output based on the generated content.\n",
    "\n",
    "In this example, the question prompt is:\n",
    "\n",
    "```\n",
    "Please provide a summary of the following text.\n",
    "TEXT: {text}\n",
    "SUMMARY:\n",
    "```\n",
    "\n",
    "and the refine prompt is:\n",
    "\n",
    "```\n",
    "Write a concise summary of the following text delimited by triple backquotes.\n",
    "Return your response in bullet points which covers the key points of the text.\n",
    "```{text}```\n",
    "BULLET POINT SUMMARY:\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XiZX45Z5VTwS"
   },
   "outputs": [],
   "source": [
    "question_prompt_template = \"\"\"\n",
    "                  Please provide a summary of the following text.\n",
    "                  TEXT: {text}\n",
    "                  SUMMARY:\n",
    "                  \"\"\"\n",
    "\n",
    "question_prompt = PromptTemplate(\n",
    "    template=question_prompt_template, input_variables=[\"text\"]\n",
    ")\n",
    "\n",
    "refine_prompt_template = \"\"\"\n",
    "              Write a concise summary of the following text delimited by triple backquotes.\n",
    "              Return your response in bullet points which covers the key points of the text.\n",
    "              ```{text}```\n",
    "              BULLET POINT SUMMARY:\n",
    "              \"\"\"\n",
    "\n",
    "refine_prompt = PromptTemplate(\n",
    "    template=refine_prompt_template, input_variables=[\"text\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-USlaSPbM0rs"
   },
   "source": [
    "### Generate summaries using Refine method\n",
    "\n",
    "After you define prompts, you initiate a summarization chain using `refine` chain type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_-Sv3HO1U3hi"
   },
   "outputs": [],
   "source": [
    "refine_chain = load_summarize_chain(\n",
    "    vertex_llm_text,\n",
    "    chain_type=\"refine\",\n",
    "    question_prompt=question_prompt,\n",
    "    refine_prompt=refine_prompt,\n",
    "    return_intermediate_steps=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f9EZCDK-MQJH"
   },
   "source": [
    "Then, you use the summarization chain to summarize document using Refine method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KHwwab7vXNa1"
   },
   "outputs": [],
   "source": [
    "refine_outputs = refine_chain({\"input_documents\": pages})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eUqpki5EMYEr"
   },
   "source": [
    "Below you can see the resulting summaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7j5cUGStZ5WF"
   },
   "outputs": [],
   "source": [
    "final_refine_data = []\n",
    "for doc, out in zip(\n",
    "    refine_outputs[\"input_documents\"], refine_outputs[\"intermediate_steps\"]\n",
    "):\n",
    "    output = {}\n",
    "    output[\"file_name\"] = p(doc.metadata[\"source\"]).stem\n",
    "    output[\"file_type\"] = p(doc.metadata[\"source\"]).suffix\n",
    "    output[\"page_number\"] = doc.metadata[\"page\"]\n",
    "    output[\"chunks\"] = doc.page_content\n",
    "    output[\"concise_summary\"] = out\n",
    "    final_refine_data.append(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "N_7Mm9cEmGOV"
   },
   "outputs": [],
   "source": [
    "pdf_refine_summary = pd.DataFrame.from_dict(final_refine_data)\n",
    "pdf_refine_summary = pdf_mp_summary.sort_values(\n",
    "    by=[\"file_name\", \"page_number\"]\n",
    ")  # sorting the dataframe by filename and page_number\n",
    "pdf_refine_summary.reset_index(inplace=True, drop=True)\n",
    "pdf_refine_summary.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jvLVCs8Gbwbw"
   },
   "outputs": [],
   "source": [
    "index = 3\n",
    "print(\"[Context]\")\n",
    "print(pdf_refine_summary[\"chunks\"].iloc[index])\n",
    "print(\"\\n\\n [Simple Summary]\")\n",
    "print(pdf_refine_summary[\"concise_summary\"].iloc[index])\n",
    "print(\"\\n\\n [Page number]\")\n",
    "print(pdf_refine_summary[\"page_number\"].iloc[index])\n",
    "print(\"\\n\\n [Source: file_name]\")\n",
    "print(pdf_refine_summary[\"file_name\"].iloc[index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7dwgbRTrM5Cb"
   },
   "source": [
    "### Considerations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1H0Y5pPcXbgm"
   },
   "source": [
    "In short, the Refine method for text summarization with LLMs can pull in more relevant context and may be less lossy than Map Reduce. However, it requires many more calls to the LLM than Stuffing, and these calls are not independent, meaning they cannot be parallelized. Additionally, there is some potential dependency on the ordering of the documents. Latest documents they might become more relevant as this method suffers from recency bias."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HAaWXncPMhv4"
   },
   "source": [
    "## Conclusion\n",
    "\n",
    "\n",
    "In this notebook you learn about different techniques to summarize long documents with LangChain and PaLM API. What you have seen in this notebook are only some of the possibilities you have. For example, there is another method called the Map-Rerank method which involves running an initial prompt on each chunk of data, which not only tries to complete a task but also gives a score for how certain it is in its answer. The responses are then ranked according to this score, and the highest score is returned.\n",
    "\n",
    "With that being said, it is important to highlight that depending on your needs you may consider to use pure Foundational model with a custom framework to build generative ai application.\n",
    "\n",
    "Here are some of the benefits of using a foundational model with a custom framework:\n",
    "\n",
    " - More flexibility to implement your application with different LLMs, prompting templates, document handling strategies and more.\n",
    "\n",
    " - More control to customize your generative applications based on your scenario.\n",
    "\n",
    " - Better performance to improve latency and scalability of your application.\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "summarization_large_documents_langchain.ipynb",
   "toc_visible": true
  },
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-11.m108",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-11:m108"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}