scrape_app_reviews.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6e7da697",
   "metadata": {},
   "source": [
    "**Importing Required Libraries**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "563e19a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "from google_play_scraper import Sort, reviews, app\n",
    "import json\n",
    "\n",
    "from pygments import highlight\n",
    "from pygments.lexers import JsonLexer\n",
    "from pygments.formatters import TerminalFormatter \n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a3ba3f3",
   "metadata": {},
   "source": [
    "**Let's use the apps again for reviews that fit the criteria from the Productivity category.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "dcf1581b",
   "metadata": {},
   "outputs": [],
   "source": [
    "app_packages = [\n",
    "  'com.jio.myjio',\n",
    "  'share.sharekaro.pro',\n",
    "  'com.speedifylink.fastest',\n",
    "  'com.mventus.selfcare.activity',\n",
    "  'com.document.reader.pdfreader.pdf',\n",
    "  'com.downlood.sav.whmedia',\n",
    "  'com.cambyte.okenscan',\n",
    "  'com.digilocker.android',\n",
    "  'com.nic.mparivahan',\n",
    "  'com.taymay.pdf.scanner',\n",
    "  'cn.wps.moffice_eng',\n",
    "  'com.documentreader.documentapp.filereader',\n",
    "  'com.lightning.safespeed',\n",
    "  'com.cv.docscanner',\n",
    "  'com.adobe.reader'\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2ba4033",
   "metadata": {},
   "source": [
    "**Let's scrape the reviews for each app. We can satisfy the first requirement by using the scraping package option to filter the review score. For the second, we'll sort the reviews by their helpfulness, which are the reviews that Google Play thinks are most important. Just in case, we'll get a subset from the newest, too.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8e305214",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████| 15/15 [02:54<00:00, 11.63s/it]\n"
     ]
    }
   ],
   "source": [
    "app_reviews = []\n",
    "for app_link in tqdm(app_packages):\n",
    "    for score in list(range(1, 6)):\n",
    "        for sort_order in [Sort.MOST_RELEVANT, Sort.NEWEST]:\n",
    "            try:\n",
    "                rvs, _ = reviews(\n",
    "                app_link,\n",
    "                lang='en',\n",
    "                country='in',\n",
    "                sort=sort_order,\n",
    "                count= 200 if score == 3 else 100,\n",
    "                filter_score_with=score\n",
    "                )\n",
    "                for r in rvs:\n",
    "                    r['sortOrder'] = 'most_relevant' if sort_order == Sort.MOST_RELEVANT else 'newest'\n",
    "                    r['appId'] = app_link\n",
    "                app_reviews.extend(rvs)\n",
    "            except:\n",
    "                print(\"App reviews not found(404).\")\n",
    "                "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecfa139e",
   "metadata": {},
   "source": [
    "**We got the reviews for all 15 apps. Let's write a helper function that prints JSON objects a bit better.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "11da542a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_json(json_object):\n",
    "    json_str = json.dumps(\n",
    "    json_object,\n",
    "    indent=2,\n",
    "    sort_keys=True,\n",
    "    default=str\n",
    "    )\n",
    "    print(highlight(json_str, JsonLexer(), TerminalFormatter())) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d3fdcf97",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m  \u001b[39;49;00m\u001b[94m\"appId\"\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m\u001b[33m\"com.jio.myjio\"\u001b[39;49;00m,\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m  \u001b[39;49;00m\u001b[94m\"at\"\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m\u001b[33m\"2022-04-23 11:13:17\"\u001b[39;49;00m,\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m  \u001b[39;49;00m\u001b[94m\"content\"\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m\u001b[33m\"Whenever there is bad weather or a power cut, there is no network in my area. Some of us are doing WFH, and I have missed many deadlines because of this. This problem has always been there but it appears that they never try to fix it. I've raised this issue many times before, but I got the same old scripted reply stating that I should change my setting. Time to switch it to Airtel, at least they don't have any network issue.\"\u001b[39;49;00m,\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m  \u001b[39;49;00m\u001b[94m\"repliedAt\"\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m\u001b[33m\"2022-04-24 18:11:04\"\u001b[39;49;00m,\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m  \u001b[39;49;00m\u001b[94m\"replyContent\"\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m\u001b[33m\"Hi, we desire to build your experience better and have made a note of your valuable feedback. For further assistance, please open MyJio app on your mobile, tap \\u2018JioCare: Help & support\\u2019 from the drop-down menu & select \\u2018Live chat\\u2019 to connect with our support team instantly. - Team MyJio\"\u001b[39;49;00m,\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m  \u001b[39;49;00m\u001b[94m\"reviewCreatedVersion\"\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m\u001b[33m\"7.0.06\"\u001b[39;49;00m,\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m  \u001b[39;49;00m\u001b[94m\"reviewId\"\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m\u001b[33m\"gp:AOqpTOEM0mkgJV_hBK1VPUU-KKyUNTU7pSSnx6qu1G7v5B6_BvCDsbE4ES4P6L2WqlAmnbUXkPfwUN5Upa7mbQ\"\u001b[39;49;00m,\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m  \u001b[39;49;00m\u001b[94m\"score\"\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m\u001b[34m1\u001b[39;49;00m,\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m  \u001b[39;49;00m\u001b[94m\"sortOrder\"\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m\u001b[33m\"most_relevant\"\u001b[39;49;00m,\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m  \u001b[39;49;00m\u001b[94m\"thumbsUpCount\"\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m\u001b[34m4944\u001b[39;49;00m,\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m  \u001b[39;49;00m\u001b[94m\"userImage\"\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m\u001b[33m\"https://play-lh.googleusercontent.com/a-/AOh14GgcaRSsh3gE8B-vPv9hEj-NVEjrKw-V-TtNjI4ZCA\"\u001b[39;49;00m,\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m  \u001b[39;49;00m\u001b[94m\"userName\"\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m\u001b[33m\"Monalisa Tonz\"\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "}\u001b[37m\u001b[39;49;00m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print_json(app_reviews[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "84829c33",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15012"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(app_reviews)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7dba6e2",
   "metadata": {},
   "source": [
    "**Let's save the reviews to a CSV file.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "090012e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "app_reviews_df = pd.DataFrame(app_reviews)\n",
    "app_reviews_df.to_csv('app_reviews.csv', index=None, header=True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "sentiment-venv",
   "language": "python",
   "name": "sentiment-venv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}