16_Coroutines/16_Coroutines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 协程\n",
    "\n",
    "相较于Python的其他特性，协程的资料较少（在阅读这一章之前，本人之前从未有意识的使用过）。从句法上来看协程与生成器类似，这两者均是包含yield关键字的“函数”。不同的是，协程中yield通常出现在表达式的右侧 —— 可以产出值也可以不产出值。协程可能会从调用方接收数据，这一过程由调用方主动使用.send()方法提供\n",
    "\n",
    "从功能上来看，协程中的yield关键字是一种流程控制工具，以实现协作式多任务。\n",
    "\n",
    "## 协程的基本行为\n",
    "\n",
    "Python中协程的正常运作依赖于生成器\n",
    "1. 协程使用生成器函数定义，即定义体中包含yield关键字\n",
    "2. 对于协程，yield关键字在表达式中使用，并且能够从调用方接收数据，默认值为None\n",
    "3. 协程的运作方式类似于生成器 —— 首先需要得到生成器对象，并且使用next()函数调用该对象后会运行到第一个yield语句\n",
    "4. 当程序运行到定义体末尾后，生成器同样会抛出StopIteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'generator'>\n",
      "GEN_CREATED\n",
      "-> coroutine started\n",
      "GEN_SUSPENDED\n",
      "-> coroutine received:  1024\n"
     ]
    },
    {
     "ename": "StopIteration",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mStopIteration\u001b[0m                             Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Local\\Temp/ipykernel_2504/784055863.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     11\u001b[0m \u001b[0mnext\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmy_coro\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     12\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mgetgeneratorstate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmy_coro\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 13\u001b[1;33m \u001b[0mmy_coro\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m1024\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mStopIteration\u001b[0m: "
     ]
    }
   ],
   "source": [
    "from inspect import getgeneratorstate\n",
    "\n",
    "def simple_coroutine():\n",
    "    print(\"-> coroutine started\")\n",
    "    x = yield\n",
    "    print(\"-> coroutine received: \", x)\n",
    "\n",
    "my_coro = simple_coroutine()\n",
    "print(type(my_coro))\n",
    "print(getgeneratorstate(my_coro))\n",
    "next(my_coro)\n",
    "print(getgeneratorstate(my_coro))\n",
    "my_coro.send(1024)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上述例子表明协程可能处于多种状态，这些状态可以由inspect.getgeneratorstate()函数获得：\n",
    "* GEN_CREATED：等待开始执行\n",
    "* GEN_RUNNING：解释器正在执行\n",
    "* GEN_SUSPENDED：在yield表达式处暂停\n",
    "* GEN_CLOSED：执行结束\n",
    "\n",
    "值得注意的是，仅当生成器处于GEN_SUSPENDED状态时才可以使用.send()传递数据。若在GEN_CREATED状态时传递则会抛出TypeError: can't send non-None value to a just-started generator；若在GEN_CLOSED阶段传递则会抛出StopIteration。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 协程的执行过程\n",
    "\n",
    "协程的执行过程需要着重理解。关键在于暂停的位置以及.send()传递数据的时机。\n",
    "\n",
    "实际上也就一点：对于生成器，若调用next()函数，则**该生成器会运行到下一个yield语句处（包含yield语句），.send()语句传递数据在此之后**\n",
    "\n",
    "下述例子描述了协程的基本执行过程，本书的图16-1对协程的执行过程有很清晰的描述"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "开始执行\n",
      "-> started: a = :  14\n",
      "14\n",
      "第一次send：\n",
      "-> Received: b = :  28\n",
      "42\n",
      "第二次send：\n",
      "-> Received: c = :  99\n"
     ]
    },
    {
     "ename": "StopIteration",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mStopIteration\u001b[0m                             Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Local\\Temp/ipykernel_12684/3842126806.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     12\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmy_coro2\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m28\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     13\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"第二次send：\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 14\u001b[1;33m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmy_coro2\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m99\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mStopIteration\u001b[0m: "
     ]
    }
   ],
   "source": [
    "def sample_coro2(a):\n",
    "    print(\"-> started: a = : \", a)\n",
    "    b = yield a\n",
    "    print(\"-> Received: b = : \", b)\n",
    "    c = yield a + b\n",
    "    print(\"-> Received: c = : \", c)\n",
    "\n",
    "my_coro2 = sample_coro2(14)\n",
    "print(\"开始执行\")\n",
    "print(next(my_coro2))\n",
    "print(\"第一次send：\")\n",
    "print(my_coro2.send(28))\n",
    "print(\"第二次send：\")\n",
    "print(my_coro2.send(99))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 预激协程\n",
    "\n",
    "协程使用前需要进行预激，即让生成器处于GEN_SUSPENDED状态。\n",
    "\n",
    "预激协程最为朴素的方法就是通过next()函数，使得生成器运行到第一个yield位置。为了简化起见，还可以通过装饰器预激协程。具体来说，通过实现一个装饰器自动运行next()函数。\n",
    "\n",
    "用于预激协程的装饰器需要自行实现，本书给出了一个例子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10.0\n",
      "20.0\n",
      "30.0\n"
     ]
    }
   ],
   "source": [
    "from functools import wraps\n",
    "\n",
    "def coroutine(func):\n",
    "    @wraps(func)\n",
    "    def primer(*args, **kwargs):\n",
    "        # 创建生成器\n",
    "        gen = func(*args, **kwargs)\n",
    "        # 预激\n",
    "        next(gen)\n",
    "        # 返回生成器\n",
    "        return gen\n",
    "    return primer\n",
    "\n",
    "@coroutine\n",
    "def averager():\n",
    "    total = 0.0\n",
    "    count = 0\n",
    "    average = None\n",
    "    while True:\n",
    "        term = yield average\n",
    "        total += term\n",
    "        count += 1\n",
    "        average = total/count\n",
    "\n",
    "coro_avg = averager()\n",
    "print(coro_avg.send(10))\n",
    "print(coro_avg.send(30))\n",
    "print(coro_avg.send(50))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 终止协程和异常处理\n",
    "\n",
    "协程中没有处理的异常会向上冒泡至调用方，并且终止该协程。\n",
    "\n",
    "上述过程实际上暗示了某种终止协程的方法：通过传递某些特殊的数据，让协程报错后自动终止。例如传递None、StopIteration等\n",
    "\n",
    "更“温和”的做法是通过两个方法显式的传递异常给协程：\n",
    "1. throw\n",
    ".throw(exc_type[, exc_value[, traceback]])，显式传递错误给协程，若协程处理了该错误则会自动运行到下一个yield位置并正常返回产出的值；否则该异常会向上冒泡至调用方\n",
    "2. close\n",
    "强制协程在暂停处抛出GeneratorExit，若协程没有处理该错误则协程终止并且调用方不会报错\n",
    "\n",
    "close方法可以终止协程，throw方法则可以显式的传递特定错误给协程"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 协程的返回值\n",
    "\n",
    "协程可以使用return返回值，这一特性是Python3.3新增的，对于Python3.3之前的版本，若同时使用yield和return会抛出句法错误。\n",
    "\n",
    "具体来说，当协程正常终止后（抛出StopIteration），return的内容会自动赋值到StopIteration的一个属性，通过处理该错误能够得到返回值。下述是一个例子。该例子中，当传递None给协程时，该协程会退出while循环并且返回一个包含均值和项数的字典，在主程序里通过try/except捕捉StopIteration错误能够得到返回值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'count': 3, 'average': 30.0}\n"
     ]
    }
   ],
   "source": [
    "def averager():\n",
    "    total = 0.0\n",
    "    count = 0\n",
    "    average = None\n",
    "    while True:\n",
    "        term = yield\n",
    "        if term is None:\n",
    "            break\n",
    "        total += term\n",
    "        count += 1\n",
    "        average = total / count\n",
    "    return {\"count\": count, \"average\": average}\n",
    "\n",
    "coro_avg = averager()\n",
    "next(coro_avg)\n",
    "coro_avg.send(10)\n",
    "coro_avg.send(30)\n",
    "coro_avg.send(50)\n",
    "try:\n",
    "    coro_avg.send(None)\n",
    "except StopIteration as exc:\n",
    "    result = exc.value\n",
    "    print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 协程中的yield from\n",
    "\n",
    "上述获取返回值的方法略别扭，好比在while循环中手动使用迭代器获取元素，完了还需要处理StopIteration。与for循环会自动生成并使用迭代器一样，yield from会自动处理StopIteration并返回结果。\n",
    "\n",
    "为了描述yield from，PEP380中使用了三个术语：\n",
    "\n",
    "1. 委派生成器\n",
    "    包含yield from <iterable>表达式的生成器函数\n",
    "2. 子生成器\n",
    "    从yield from <iterable>的<iterable>中获取的生成器\n",
    "3. 调用方\n",
    "    调用委派生成器的主程序\n",
    "\n",
    "本书的图16-2描述了上述三个术语之间的关系以及数据/消息在这些对象之间的传递：\n",
    "\n",
    "调用方能够使用.send、.throw和.close传递数据并控制委派生成器，委派生成器能够返回数据给调用方；委派生成器则可以进一步控制子生成器，同时委派生成器也会设法处理子生成器抛出的错误。\n",
    "\n",
    "下面是使用yield from的一个例子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'group_1': {'count': 5, 'average': 56.0}, 'group_2': {'count': 5, 'average': 10.6}}\n",
      "GEN_SUSPENDED\n",
      "GEN_CLOSED\n"
     ]
    }
   ],
   "source": [
    "from inspect import getgeneratorstate\n",
    "\n",
    "# 子生成器\n",
    "def averager():\n",
    "    total = 0.0\n",
    "    count = 0\n",
    "    average = None\n",
    "    while True:\n",
    "        term = yield\n",
    "        if term is None:\n",
    "            break\n",
    "        total += term\n",
    "        count += 1\n",
    "        average = total/count\n",
    "    return {\"count\": count, \"average\": average}\n",
    "\n",
    "\n",
    "# 委派生成器\n",
    "def grouper(results, key):\n",
    "    while True:\n",
    "        results[key] = yield from averager()    \n",
    "\n",
    "\n",
    "# 调用方\n",
    "def main():\n",
    "    test_data = {\"group_1\": [10, 30, 50, 90, 100], \"group_2\":[8, 10, 9, 7, 19]}\n",
    "    results = {}\n",
    "    for key, values in test_data.items():\n",
    "        group = grouper(results, key)\n",
    "        next(group)\n",
    "        for value in values:\n",
    "            group.send(value)\n",
    "        group.send(None)\n",
    "    print(results)\n",
    "\n",
    "    print(getgeneratorstate(group))\n",
    "    group.close()\n",
    "    print(getgeneratorstate(group))\n",
    "\n",
    "main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上述程序执行了相当多的操作，对于test_data中的每一个键值对：\n",
    "\n",
    "1. 新建grouper实例group，并传入记录数据的results和键key。group是一个委派生成器\n",
    "2. 使用next(group)预激委派生成器并使得子生成器在yield暂停，同时group也在yield from位置暂停\n",
    "3. 逐个传入数据，调用方传递数据给group，group传递数据给子生成器。值得注意的是虽然子生成器在获取到数据后会运行到下一个yield位置，group依然停留在yield from的位置并没有运行委派生成器里的赋值语句\n",
    "4. 调用方的内层for循环执行完毕后，发送None给group，group传递None给子生成器，子生成器抛出StopIteration并执行return；group处理StopIteration并运行委派生成器中的赋值语句，然后**运行至下一个yield from位置**\n",
    "5. 运行调用方的外层for循环，并重新执行上述1~4步。前一个group实例以及其子生成器被自动回收\n",
    "\n",
    "yield from的行为可以归纳为如下几点：\n",
    "1. 子生成器产出的值均会直接传递给调用方\n",
    "2. 调用方使用send()传递给委派生成器的值均会直接传递给子生成器。若传递的值不是None则调用子生成器的send()方法，否则调用\\_\\_next\\_\\_()方法。若该调用抛出StopIteration，则委派生成器恢复运行，任何其他异常均会向上冒泡传递给委派生成器\n",
    "3. 子生成器退出时，子生成器的return expr会触发StopIteration(expr)\n",
    "4. yield from表达式的值是子生成器终止时传给StopIteration异常的第一个参数\n",
    "5. 调用方传递给委派生成器的异常，除了GeneratorExit外，均会传递给子生成器的throw()方法。若调用throw()方法抛出StopIteration，则委派生成器恢复运行；其他异常则会由子生成器向上冒泡至委派生成器\n",
    "6. 若调用方传递GeneratorExit异常给委派生成器，或者调用close()方法。若子生成器有close()方法则调用该方法。若调用close()方法导致异常，则向上冒泡该错误至委派生成器；否则委派生成器抛出GeneratorExit\n",
    "7. **存疑**，从实践来看，委托生成器在运行到定义体结束时也会抛出StopIteration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结\n",
    "\n",
    "1. 从句法上来看，协程和生成器类似，这两者均包含yield关键字。不同的是，协程中yield通常出现在表达式的右侧。协程可以从调用方接收数据，这一过程由调用方主动使用.send()实现\n",
    "2. Python中协程的运作方式类似于生成器 —— 首先需要得到生成器对象，并且使用next()函数调用该对象后运行到第一个yield语句（预激）。当程序运行到定义体末尾，协程会抛出StopIteration\n",
    "3. 从类型（type函数）来看，协程在Python中是一种特性。若查看协程的定义体的类型，会发现仍是生成器（class 'generator'）\n",
    "4. 协程的状态有如下几种：等待开始执行（GEN_CREATED）、正在执行（GEN_RUNNING）、在yield表达式位置暂停（GEN_SUSPENDED）以及执行结束（GEN_CLOSED）。这些状态可由inspect.getgeneratorstate()函数获得。仅有当协程处于GEN_SUSPENDED时才可以使用.send传递数据\n",
    "5. 协程的执行过程并不复杂，实际上就是：next()函数会使得生成器运行到下一个yield语句位置（yield右侧的内容会返回给调用方），send()传递数据则在yield语句之后进行（yield左侧的赋值语句）\n",
    "6. 协程使用前需要进行预激活，即使得协程运行到yield位置并暂停，此时生成器处于GEN_SUSPENDED状态。除了手动调用next方法运行到yield位置外，还可以使用装饰器自动执行预激\n",
    "7. 协程可以通过throw和close来终止协程。其中throw可以显示的传递特定错误给协程，close方法则可以直接终止协程\n",
    "8. Python3.3后，协程可以使用return返回值，具体来说，当协程正常终止后（抛出StopIteration），return的内容会自动赋值到StopIteration的一个属性，通过处理该错误能够得到返回值\n",
    "9. 除了直接使用yield构建协程外，还可以通过yield from执行更为复杂和规范化的操作\n",
    "10. 基于yield from的协程功能由三个组件共同实现，其分别是：调用方、委派生成器以及子生成器。其中，委派生成器实际上是一种“通道”，其连接了调用方和子生成器，为两者之间的数据以及控制信息的传递提供了规范化的途径。本书的图16-2很好的描述了这三者之间的关系\n",
    "11. yield from有非常复杂的语义，但是总体来说还是能够理清的。基本上就是调用方驱动委派生成器，委派生成器驱动子生成器。数据的传递依靠send()函数、子生成器的产出以及return，基本的控制信息则依靠throw()函数、close()函数以及send(None)等。本书的示例16-19使用伪代码描述了核心语句（委派生成器中RESULT = yield from EXPR语句）的执行过程。这一份伪代码对理解yield from有很大帮助"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.7 ('FlyAi_Pytorch1_5Python37')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "19f15d24debae226a793fdaae111f52c1e7583ee4f15ab643269646087f243d5"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}