docs/source/notebooks/aijack_fedavg.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SROAZ9dO80s8"
   },
   "source": [
    "# FedAVG\n",
    "\n",
    "In this tutorial, you will learn how to simulate FedAVG, a representative scheme of Federated Learning, with AIJack. You can choose the single process or MPI as the backend. We will also demonstrate that you can add various defense methods to FedAVG with only a few additional lines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WcsHvpzSzaTK"
   },
   "source": [
    "While deep learning achieves substantial success in various areas, training deep learning models requires much data. Thus, acquiring high performance in deep learning while preserving privacy is challenging. One way to solve this problem is Federated Learning, where multiple clients collaboratively train a single global model without sharing their local dataset.\n",
    "\n",
    "The procedure of typical Federated Learning is as follows:\n",
    "\n",
    "```\n",
    "1. The central server initializes the global model.\n",
    "2. The server distributes global model to each client.\n",
    "3. Each client locally calculates the gradient of the loss function on their dataset.\n",
    "4. Each client sends the gradient to the server.\n",
    "5. The server aggregates the received gradients with some method (e.g., average) and updates the global model with the aggregated gradient.\n",
    "6. Repeat 2 ~ 5 until converge.\n",
    "```\n",
    "\n",
    "The mathematical notification when the aggregation is the weighted average is as follows:\n",
    "\n",
    "$$\n",
    "w_{t} \\leftarrow w_{t - 1} - \\eta \\sum_{c=1}^{C} \\frac{n_{c}}{N} \\nabla \\mathcal{l}(w_{t - 1}, X_{c}, Y_{c})\n",
    "$$\n",
    "\n",
    ", where $w_{t}$ is the parameter of the global model in $t$-th round, $\\nabla \\mathcal{l}(w_{t - 1}, X_{c}, Y_{c})$ is the gradient calculated on $c$-th client's dataset $((X_{c}, Y_{c}))$, $n_{c}$ is the number of $c$-th client's dataset, and N is the total number of samples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FNMDQuH49CBO"
   },
   "source": [
    "## Single Process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "J4s0w9rHwOd8"
   },
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "from mpi4py import MPI\n",
    "from torchvision import datasets, transforms\n",
    "\n",
    "from aijack.collaborative.fedavg import FedAVGClient, FedAVGServer, FedAVGAPI\n",
    "\n",
    "\n",
    "def evaluate_gloal_model(dataloader, client_id=-1):\n",
    "    def _evaluate_global_model(api):\n",
    "        test_loss = 0\n",
    "        correct = 0\n",
    "        with torch.no_grad():\n",
    "            for data, target in dataloader:\n",
    "                data, target = data.to(api.device), target.to(api.device)\n",
    "                if client_id == -1:\n",
    "                    output = api.server(data)\n",
    "                else:\n",
    "                    output = api.clients[client_id](data)\n",
    "                test_loss += F.nll_loss(\n",
    "                    output, target, reduction=\"sum\"\n",
    "                ).item()  # sum up batch loss\n",
    "                pred = output.argmax(\n",
    "                    dim=1, keepdim=True\n",
    "                )  # get the index of the max log-probability\n",
    "                correct += pred.eq(target.view_as(pred)).sum().item()\n",
    "\n",
    "        test_loss /= len(dataloader.dataset)\n",
    "        accuracy = 100.0 * correct / len(dataloader.dataset)\n",
    "        print(f\"Test set: Average loss: {test_loss}, Accuracy: {accuracy}\")\n",
    "\n",
    "    return _evaluate_global_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "idlkuHjjULVz"
   },
   "outputs": [],
   "source": [
    "training_batch_size = 64\n",
    "test_batch_size = 64\n",
    "num_rounds = 5\n",
    "lr = 0.001\n",
    "seed = 0\n",
    "client_size = 2\n",
    "criterion = F.nll_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "gsh_CjamwqfV"
   },
   "outputs": [],
   "source": [
    "def fix_seed(seed):\n",
    "    random.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    torch.manual_seed(seed)\n",
    "    torch.cuda.manual_seed_all(seed)\n",
    "    torch.backends.cudnn.deterministic = True\n",
    "\n",
    "\n",
    "def prepare_dataloader(num_clients, myid, train=True, path=\"\"):\n",
    "    transform = transforms.Compose(\n",
    "        [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]\n",
    "    )\n",
    "    if train:\n",
    "        dataset = datasets.MNIST(path, train=True, download=True, transform=transform)\n",
    "        idxs = list(range(len(dataset.data)))\n",
    "        random.shuffle(idxs)\n",
    "        idx = np.array_split(idxs, num_clients, 0)[myid - 1]\n",
    "        dataset.data = dataset.data[idx]\n",
    "        dataset.targets = dataset.targets[idx]\n",
    "        train_loader = torch.utils.data.DataLoader(\n",
    "            dataset, batch_size=training_batch_size\n",
    "        )\n",
    "        return train_loader\n",
    "    else:\n",
    "        dataset = datasets.MNIST(path, train=False, download=True, transform=transform)\n",
    "        test_loader = torch.utils.data.DataLoader(dataset, batch_size=test_batch_size)\n",
    "        return test_loader\n",
    "\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.ln = nn.Linear(28 * 28, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.ln(x.reshape(-1, 28 * 28))\n",
    "        output = F.log_softmax(x, dim=1)\n",
    "        return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "UX-5FPlbxs4z"
   },
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "fix_seed(seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 431,
     "referenced_widgets": [
      "917c709913bb480d8a8ed79af56fb295",
      "13a2bd97dedd4bf4a44f2e104542d230",
      "9371c7b66c6b49b4beba08e69103ab7e",
      "ff29cf94306a4c4a8c983e521afc6440",
      "651e8f5f44934b6c8541bfb6cc938fe0",
      "76943ff8de564fcbab773ddf0a6ca179",
      "6e98fe8dc36344549659336eb3b0ef22",
      "ff69f6236fa045bf92d9f27f7a626c8f",
      "9dba40a72c9540e798725b23baedb032",
      "74b3dde3960c4fa5a1e245be0efb1472",
      "734303b3d419406f955cad697a1fa9cd",
      "1080409ad599411c966907925e5a0363",
      "9a7ec62d5ee9470bb45bddd0d28c90d3",
      "47c3a994921c4d79aadf560b1c90c418",
      "a0d8d6dc678f43dbb8b5fff92e51bf15",
      "d9b39efe9f4c4fe999effa7c82d03980",
      "10d6959568df40ffb5ae5440f793bff4",
      "6db621afba5b43d5b05d0920074da6b0",
      "7ec62d738d6d4e90babd6b6ceb2aff7e",
      "69e07cd28ee0488783e1dc87ce960fde",
      "9d46cec1335645d69c893b48d9bc6b60",
      "cf4a153faba44c1dbeb6640ac5e0e128",
      "f9ce086b334647979cade9199c36cfbb",
      "be16173f879e4801b4e7198ad91f2ed1",
      "d45d5e218fa242f688cb51b7f9334195",
      "08de784042f54a928c7f99c5f5213488",
      "4d951249d31540f4a28135106e045dcf",
      "6f7c94eb66864b57b20df70fc4af28bb",
      "87631b21fc544ec28f9d75848d72c60f",
      "0be9231df6b449789d90b5b9452dc9a8",
      "7d35ecf53bd44c64acd2e6215011e0f2",
      "24a0beed8f9341559c7098d21a500f10",
      "5900cb2c91724b3e842152fea7378eb5",
      "4b98a9ff6a1b446590a01aea00b093d1",
      "99bb17593e1f48e4b5e9abe3612a1468",
      "a000e57d608f4abaa2d26fe986e44e3c",
      "637cd5cd789e466dafaa91c1b4f6b83c",
      "137edcab4054429db14dd9a65081315e",
      "e906b87f9bbf43fe8035fc6ffe3e5305",
      "e0abd3cd05124abca79702e4ed4b5691",
      "a1b47a6e45e54e06a969032c0da4a755",
      "61cce78a932c46edb60c815fb60461f8",
      "e51d54635a634dab91ea7cc5581d843a",
      "29ced398dee84743ac8bf52819874345"
     ]
    },
    "id": "zz_YjoioAcLD",
    "outputId": "afd8a1e1-5915-4ddb-b63b-c3e03ebf3306"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to MNIST/raw/train-images-idx3-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "917c709913bb480d8a8ed79af56fb295",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/9912422 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST/raw/train-images-idx3-ubyte.gz to MNIST/raw\n",
      "\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to MNIST/raw/train-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1080409ad599411c966907925e5a0363",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/28881 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST/raw/train-labels-idx1-ubyte.gz to MNIST/raw\n",
      "\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to MNIST/raw/t10k-images-idx3-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f9ce086b334647979cade9199c36cfbb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1648877 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST/raw/t10k-images-idx3-ubyte.gz to MNIST/raw\n",
      "\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to MNIST/raw/t10k-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4b98a9ff6a1b446590a01aea00b093d1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/4542 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST/raw/t10k-labels-idx1-ubyte.gz to MNIST/raw\n",
      "\n"
     ]
    }
   ],
   "source": [
    "local_dataloaders = [prepare_dataloader(client_size, c) for c in range(client_size)]\n",
    "test_dataloader = prepare_dataloader(client_size, -1, train=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UEzYT3tex0Nm",
    "outputId": "7962f7c2-a96e-48f6-c1f7-bd3f76dd0a9a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "communication 0, epoch 0: client-1 0.019623182541131972\n",
      "communication 0, epoch 0: client-2 0.019723439224561056\n",
      "Test set: Average loss: 0.7824367880821228, Accuracy: 83.71\n",
      "communication 1, epoch 0: client-1 0.01071754728158315\n",
      "communication 1, epoch 0: client-2 0.010851142065723737\n",
      "Test set: Average loss: 0.58545467877388, Accuracy: 86.49\n",
      "communication 2, epoch 0: client-1 0.008766427374879518\n",
      "communication 2, epoch 0: client-2 0.00891655088464419\n",
      "Test set: Average loss: 0.507768925857544, Accuracy: 87.54\n",
      "communication 3, epoch 0: client-1 0.007839484961827596\n",
      "communication 3, epoch 0: client-2 0.00799967499623696\n",
      "Test set: Average loss: 0.46477557654380797, Accuracy: 88.25\n",
      "communication 4, epoch 0: client-1 0.0072782577464977904\n",
      "communication 4, epoch 0: client-2 0.007445397683481375\n",
      "Test set: Average loss: 0.436919868183136, Accuracy: 88.63\n"
     ]
    }
   ],
   "source": [
    "clients = [FedAVGClient(Net().to(device), user_id=c) for c in range(client_size)]\n",
    "local_optimizers = [optim.SGD(client.parameters(), lr=lr) for client in clients]\n",
    "\n",
    "server = FedAVGServer(clients, Net().to(device))\n",
    "\n",
    "api = FedAVGAPI(\n",
    "    server,\n",
    "    clients,\n",
    "    criterion,\n",
    "    local_optimizers,\n",
    "    local_dataloaders,\n",
    "    num_communication=num_rounds,\n",
    "    custom_action=evaluate_gloal_model(test_dataloader),\n",
    ")\n",
    "api.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "smTKPDVx9Dt4"
   },
   "source": [
    "## MPI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pzKLDKNw820p",
    "outputId": "c4e43963-8a2a-47b9-9fff-0e4969a250d1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing mpi_FedAVG.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile mpi_FedAVG.py\n",
    "import random\n",
    "from logging import getLogger\n",
    "\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "from mpi4py import MPI\n",
    "from torchvision import datasets, transforms\n",
    "\n",
    "from aijack.collaborative import FedAVGClient, FedAVGServer, MPIFedAVGAPI, MPIFedAVGClientManager, MPIFedAVGServerManager\n",
    "\n",
    "logger = getLogger(__name__)\n",
    "\n",
    "training_batch_size = 64\n",
    "test_batch_size = 64\n",
    "num_rounds = 5\n",
    "lr = 0.001\n",
    "seed = 0\n",
    "\n",
    "\n",
    "def fix_seed(seed):\n",
    "    random.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    torch.manual_seed(seed)\n",
    "    torch.cuda.manual_seed_all(seed)\n",
    "    torch.backends.cudnn.deterministic = True\n",
    "\n",
    "\n",
    "def prepare_dataloader(num_clients, myid, train=True, path=\"\"):\n",
    "    transform = transforms.Compose(\n",
    "        [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]\n",
    "    )\n",
    "    if train:\n",
    "        dataset = datasets.MNIST(path, train=True, download=False, transform=transform)\n",
    "        idxs = list(range(len(dataset.data)))\n",
    "        random.shuffle(idxs)\n",
    "        idx = np.array_split(idxs, num_clients, 0)[myid - 1]\n",
    "        dataset.data = dataset.data[idx]\n",
    "        dataset.targets = dataset.targets[idx]\n",
    "        train_loader = torch.utils.data.DataLoader(\n",
    "            dataset, batch_size=training_batch_size\n",
    "        )\n",
    "        return train_loader\n",
    "    else:\n",
    "        dataset = datasets.MNIST(path, train=False, download=False, transform=transform)\n",
    "        test_loader = torch.utils.data.DataLoader(dataset, batch_size=test_batch_size)\n",
    "        return test_loader\n",
    "\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.ln = nn.Linear(28 * 28, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.ln(x.reshape(-1, 28 * 28))\n",
    "        output = F.log_softmax(x, dim=1)\n",
    "        return output\n",
    "\n",
    "\n",
    "def evaluate_gloal_model(dataloader):\n",
    "    def _evaluate_global_model(api):\n",
    "        test_loss = 0\n",
    "        correct = 0\n",
    "        with torch.no_grad():\n",
    "            for data, target in dataloader:\n",
    "                data, target = data.to(api.device), target.to(api.device)\n",
    "                output = api.party(data)\n",
    "                test_loss += F.nll_loss(\n",
    "                    output, target, reduction=\"sum\"\n",
    "                ).item()  # sum up batch loss\n",
    "                pred = output.argmax(\n",
    "                    dim=1, keepdim=True\n",
    "                )  # get the index of the max log-probability\n",
    "                correct += pred.eq(target.view_as(pred)).sum().item()\n",
    "\n",
    "        test_loss /= len(dataloader.dataset)\n",
    "        accuracy = 100.0 * correct / len(dataloader.dataset)\n",
    "        print(\n",
    "            f\"Round: {api.party.round}, Test set: Average loss: {test_loss}, Accuracy: {accuracy}\"\n",
    "        )\n",
    "\n",
    "    return _evaluate_global_model\n",
    "\n",
    "def main():\n",
    "    fix_seed(seed)\n",
    "\n",
    "    comm = MPI.COMM_WORLD\n",
    "    myid = comm.Get_rank()\n",
    "    size = comm.Get_size()\n",
    "\n",
    "    device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "    model = Net()\n",
    "    model = model.to(device)\n",
    "    optimizer = optim.SGD(model.parameters(), lr=lr)\n",
    "\n",
    "    mpi_client_manager = MPIFedAVGClientManager()\n",
    "    mpi_server_manager = MPIFedAVGServerManager()\n",
    "    MPIFedAVGClient = mpi_client_manager.attach(FedAVGClient)\n",
    "    MPIFedAVGServer = mpi_server_manager.attach(FedAVGServer)\n",
    "\n",
    "    if myid == 0:\n",
    "        dataloader = prepare_dataloader(size - 1, myid, train=False)\n",
    "        client_ids = list(range(1, size))\n",
    "        server = MPIFedAVGServer(comm, [1, 2], model)\n",
    "        api = MPIFedAVGAPI(\n",
    "            comm,\n",
    "            server,\n",
    "            True,\n",
    "            F.nll_loss,\n",
    "            None,\n",
    "            None,\n",
    "            num_rounds,\n",
    "            1,\n",
    "            custom_action=evaluate_gloal_model(dataloader),\n",
    "            device=device\n",
    "        )\n",
    "    else:\n",
    "        dataloader = prepare_dataloader(size - 1, myid, train=True)\n",
    "        client = MPIFedAVGClient(comm, model, user_id=myid)\n",
    "        api = MPIFedAVGAPI(\n",
    "            comm,\n",
    "            client,\n",
    "            False,\n",
    "            F.nll_loss,\n",
    "            optimizer,\n",
    "            dataloader,\n",
    "            num_rounds,\n",
    "            1,\n",
    "            device=device\n",
    "        )\n",
    "    api.run()\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UDBFHwAsvCv3",
    "outputId": "1dccd74a-5f4e-4aa1-cae2-5e29f2e0bf2b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "communication 0, epoch 0: client-3 0.019996537216504413\n",
      "communication 0, epoch 0: client-2 0.02008056694070498\n",
      "Round: 1, Test set: Average loss: 0.7860309104919434, Accuracy: 82.72\n",
      "communication 1, epoch 0: client-3 0.010822976715366046\n",
      "communication 1, epoch 0: client-2 0.010937693453828494\n",
      "Round: 2, Test set: Average loss: 0.5885528886795044, Accuracy: 86.04\n",
      "communication 2, epoch 0: client-2 0.008990796900788942\n",
      "communication 2, epoch 0: client-3 0.008850129560629527\n",
      "Round: 3, Test set: Average loss: 0.5102099328994751, Accuracy: 87.33\n",
      "communication 3, epoch 0: client-3 0.00791173183619976\n",
      "communication 3, epoch 0: client-2 0.008069112183650334\n",
      "Round: 4, Test set: Average loss: 0.4666414333820343, Accuracy: 88.01\n",
      "communication 4, epoch 0: client-2 0.007512268128991127\n",
      "communication 4, epoch 0: client-3 0.007343090359369914\n",
      "Round: 5, Test set: Average loss: 0.4383064950466156, Accuracy: 88.65\n"
     ]
    }
   ],
   "source": [
    "!sudo mpiexec -np 3 --allow-run-as-root python /content/mpi_FedAVG.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "nBPih2JPaxPJ"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "machine_shape": "hm",
   "provenance": []
  },
  "gpuClass": "standard",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "caa2b01f75ba60e629eaa9e4dabde0c46b243c9a0484934eeb17ad8b3fc9c91a"
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {}
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}