-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathatom.xml
609 lines (409 loc) · 576 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>花解语</title>
<subtitle>求知若饥，虚心若愚</subtitle>
<link href="/atom.xml" rel="self"/>
<link href="https://vincent507cpu.github.io/"/>
<updated>2020-12-29T21:53:21.540Z</updated>
<id>https://vincent507cpu.github.io/</id>
<author>
<name>翟文嘉</name>
</author>
<generator uri="https://hexo.io/">Hexo</generator>
<entry>
<title>[阿里云天池] 自然语言处理训练营 2</title>
<link href="https://vincent507cpu.github.io/2020/12/29/%E9%98%BF%E9%87%8C%E4%BA%91%E5%A4%A9%E6%B1%A0-%E8%87%AA%E7%84%B6%E8%AF%AD%E8%A8%80%E5%A4%84%E7%90%86%E8%AE%AD%E7%BB%83%E8%90%A5-2/"/>
<id>https://vincent507cpu.github.io/2020/12/29/%E9%98%BF%E9%87%8C%E4%BA%91%E5%A4%A9%E6%B1%A0-%E8%87%AA%E7%84%B6%E8%AF%AD%E8%A8%80%E5%A4%84%E7%90%86%E8%AE%AD%E7%BB%83%E8%90%A5-2/</id>
<published>2020-12-29T21:52:47.000Z</published>
<updated>2020-12-29T21:53:21.540Z</updated>
<content type="html"><![CDATA[<h1 id="文本表示方法-Part1"><a href="#文本表示方法-Part1" class="headerlink" title="文本表示方法 Part1"></a>文本表示方法 Part1</h1><p>在机器学习算法的训练过程中，假设给定 $N$ 个样本，每个样本有 $M$ 个特征，这样组成了 $N × M$ 的样本矩阵，然后完成算法的训练和预测。同样的在计算机视觉中可以将图片的像素看作特征，每张图片看作 $hight×width×3$的特征图，一个三维的矩阵来进入计算机进行计算。</p><a id="more"></a><p>但是在自然语言领域，上述方法却不可行：文本是不定长度的。文本表示成计算机能够运算的数字或向量的方法一般称为词嵌入（Word Embedding）方法。词嵌入将不定长的文本转换到定长的空间内，是文本分类的第一步。</p><h2 id="One-hot"><a href="#One-hot" class="headerlink" title="One-hot"></a>One-hot</h2><p>这里的One-hot与数据挖掘任务中的操作是一致的，即将每一个单词使用一个离散的向量表示。具体将每个字/词编码一个索引，然后根据索引进行赋值。</p><h2 id="Bag-of-Words"><a href="#Bag-of-Words" class="headerlink" title="Bag of Words"></a>Bag of Words</h2><p>Bag of Words（词袋表示），也称为Count Vectors，每个文档的字/词可以使用其出现次数来进行表示。<br>在 sklearn 中可以直接 <code>CountVectorizer</code> 来实现这一步骤：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> sklearn.feature_extraction.text <span class="keyword">import</span> CountVectorizer</span><br><span class="line">corpus = [</span><br><span class="line"> <span class="string">'This is the first document.'</span>,</span><br><span class="line"> <span class="string">'This document is the second document.'</span>,</span><br><span class="line"> <span class="string">'And this is the third one.'</span>,</span><br><span class="line"> <span class="string">'Is this the first document?'</span>,</span><br><span class="line">]</span><br><span class="line">vectorizer = CountVectorizer()</span><br><span class="line">vectorizer.fit_transform(corpus).toarray()</span><br></pre></td></tr></table></figure><h2 id="N-gram"><a href="#N-gram" class="headerlink" title="N-gram"></a>N-gram</h2><p>N-gram与Count Vectors类似，不过加入了相邻单词组合成为新的单词，并进行计数。</p><h2 id="TF-IDF"><a href="#TF-IDF" class="headerlink" title="TF-IDF"></a>TF-IDF</h2><p>TF-IDF 分数由两部分组成：第一部分是词语频率（Term Frequency），第二部分是逆文档频率（Inverse Document Frequency）。其中计算语料库中文档总数除以含有该词语的文档数量，然后再取对数就是逆文档频率。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">TF(t)= 该词语在当前文档出现的次数 / 当前文档中词语的总数</span><br><span class="line">IDF(t)= log_e（文档总数 / 出现该词语的文档总数）</span><br></pre></td></tr></table></figure><h1 id="基于机器学习的文本分类"><a href="#基于机器学习的文本分类" class="headerlink" title="基于机器学习的文本分类"></a>基于机器学习的文本分类</h1><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Count Vectors + RidgeClassifier</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> sklearn.feature_extraction.text <span class="keyword">import</span> CountVectorizer</span><br><span class="line"><span class="keyword">from</span> sklearn.linear_model <span class="keyword">import</span> RidgeClassifier</span><br><span class="line"><span class="keyword">from</span> sklearn.metrics <span class="keyword">import</span> f1_score</span><br><span class="line"></span><br><span class="line">train_df = pd.read_csv(<span class="string">'../data/train_set.csv'</span>, sep=<span class="string">'\t'</span>, nrows=<span class="number">15000</span>)</span><br><span class="line"></span><br><span class="line">vectorizer = CountVectorizer(max_features=<span class="number">3000</span>)</span><br><span class="line">train_test = vectorizer.fit_transform(train_df[<span class="string">'text'</span>])</span><br><span class="line"></span><br><span class="line">clf = RidgeClassifier()</span><br><span class="line">clf.fit(train_test[:<span class="number">10000</span>], train_df[<span class="string">'label'</span>].values[:<span class="number">10000</span>])</span><br><span class="line"></span><br><span class="line">val_pred = clf.predict(train_test[<span class="number">10000</span>:])</span><br><span class="line">print(f1_score(train_df[<span class="string">'label'</span>].values[<span class="number">10000</span>:], val_pred, average=<span class="string">'macro'</span>))</span><br><span class="line"><span class="comment"># 0.65</span></span><br></pre></td></tr></table></figure><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># TF-IDF + RidgeClassifier</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> sklearn.feature_extraction.text <span class="keyword">import</span> TfidfVectorizer</span><br><span class="line"><span class="keyword">from</span> sklearn.linear_model <span class="keyword">import</span> RidgeClassifier</span><br><span class="line"><span class="keyword">from</span> sklearn.metrics <span class="keyword">import</span> f1_score</span><br><span class="line"></span><br><span class="line">train_df = pd.read_csv(<span class="string">'../data/train_set.csv'</span>, sep=<span class="string">'\t'</span>, nrows=<span class="number">15000</span>)</span><br><span class="line"></span><br><span class="line">tfidf = TfidfVectorizer(ngram_range=(<span class="number">1</span>,<span class="number">3</span>), max_features=<span class="number">3000</span>)</span><br><span class="line">train_test = tfidf.fit_transform(train_df[<span class="string">'text'</span>])</span><br><span class="line"></span><br><span class="line">clf = RidgeClassifier()</span><br><span class="line">clf.fit(train_test[:<span class="number">10000</span>], train_df[<span class="string">'label'</span>].values[:<span class="number">10000</span>])</span><br><span class="line"></span><br><span class="line">val_pred = clf.predict(train_test[<span class="number">10000</span>:])</span><br><span class="line">print(f1_score(train_df[<span class="string">'label'</span>].values[<span class="number">10000</span>:], val_pred, average=<span class="string">'macro'</span>))</span><br><span class="line"><span class="comment"># 0.87</span></span><br></pre></td></tr></table></figure><h1 id="作业"><a href="#作业" class="headerlink" title="作业"></a>作业</h1><ol><li>尝试改变TF-IDF的参数，并验证精度<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># TF-IDF (6000 words)+ RidgeClassifier</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> sklearn.feature_extraction.text <span class="keyword">import</span> TfidfVectorizer</span><br><span class="line"><span class="keyword">from</span> sklearn.linear_model <span class="keyword">import</span> RidgeClassifier</span><br><span class="line"><span class="keyword">from</span> sklearn.metrics <span class="keyword">import</span> f1_score</span><br><span class="line"></span><br><span class="line">train_df = pd.read_csv(<span class="string">'./data/train_set.csv'</span>, sep=<span class="string">'\t'</span>, nrows=<span class="number">15000</span>)</span><br><span class="line"></span><br><span class="line">tfidf = TfidfVectorizer(ngram_range=(<span class="number">1</span>,<span class="number">3</span>), max_features=<span class="number">6000</span>)</span><br><span class="line">train_test = tfidf.fit_transform(train_df[<span class="string">'text'</span>])</span><br><span class="line"></span><br><span class="line">clf = RidgeClassifier()</span><br><span class="line">clf.fit(train_test[:<span class="number">10000</span>], train_df[<span class="string">'label'</span>].values[:<span class="number">10000</span>])</span><br><span class="line"></span><br><span class="line">val_pred = clf.predict(train_test[<span class="number">10000</span>:])</span><br><span class="line">print(f1_score(train_df[<span class="string">'label'</span>].values[<span class="number">10000</span>:], val_pred, average=<span class="string">'macro'</span>))</span><br></pre></td></tr></table></figure></li><li>尝试使用其他机器学习模型，完成训练和验证</li></ol>]]></content>
<summary type="html">
<h1 id="文本表示方法-Part1"><a href="#文本表示方法-Part1" class="headerlink" title="文本表示方法 Part1"></a>文本表示方法 Part1</h1><p>在机器学习算法的训练过程中，假设给定 $N$ 个样本，每个样本有 $M$ 个特征，这样组成了 $N × M$ 的样本矩阵，然后完成算法的训练和预测。同样的在计算机视觉中可以将图片的像素看作特征，每张图片看作 $hight×width×3$的特征图，一个三维的矩阵来进入计算机进行计算。</p>
</summary>
<category term="NLP" scheme="https://vincent507cpu.github.io/tags/NLP/"/>
</entry>
<entry>
<title>[阿里云天池] 自然语言处理训练营 1</title>
<link href="https://vincent507cpu.github.io/2020/12/29/%E9%98%BF%E9%87%8C%E4%BA%91%E5%A4%A9%E6%B1%A0-%E8%87%AA%E7%84%B6%E8%AF%AD%E8%A8%80%E5%A4%84%E7%90%86%E8%AE%AD%E7%BB%83%E8%90%A5-1/"/>
<id>https://vincent507cpu.github.io/2020/12/29/%E9%98%BF%E9%87%8C%E4%BA%91%E5%A4%A9%E6%B1%A0-%E8%87%AA%E7%84%B6%E8%AF%AD%E8%A8%80%E5%A4%84%E7%90%86%E8%AE%AD%E7%BB%83%E8%90%A5-1/</id>
<published>2020-12-29T16:50:52.000Z</published>
<updated>2020-12-29T16:51:51.406Z</updated>
<content type="html"><![CDATA[<h1 id="数据读取"><a href="#数据读取" class="headerlink" title="数据读取"></a>数据读取</h1><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line">train_df = pd.read_csv(<span class="string">'./data/train_set.csv'</span>, sep=<span class="string">'\t'</span>, nrows=<span class="number">100</span>)</span><br><span class="line">train_df.head()</span><br></pre></td></tr></table></figure><table><thead><tr><th></th><th>label</th><th>text</th></tr></thead><tbody><tr><td>0</td><td>2</td><td>2967 6758 339 2021 1854 3731 4109 3792 4149 15…</td></tr><tr><td>1</td><td>11</td><td>4464 486 6352 5619 2465 4802 1452 3137 5778 54…</td></tr><tr><td>2</td><td>3</td><td>7346 4068 5074 3747 5681 6093 1777 2226 7354 6…</td></tr><tr><td>3</td><td>2</td><td>7159 948 4866 2109 5520 2490 211 3956 5520 549…</td></tr><tr><td>4</td><td>3</td><td>3646 3055 3055 2490 4659 6065 3370 5814 2465 5…</td></tr></tbody></table><a id="more"></a><h1 id="数据分析"><a href="#数据分析" class="headerlink" title="数据分析"></a>数据分析</h1><p>在读取完成数据集后，我们还可以对数据集进行数据分析的操作。虽然对于非结构数据并不需要做很多的数据分析，但通过数据分析还是可以找出一些规律的。</p><p>此步骤我们读取了所有的训练集数据，在此我们通过数据分析希望得出以下结论：</p><ul><li>赛题数据中，新闻文本的长度是多少？</li><li>赛题数据的类别分布是怎么样的，哪些类别比较多？</li><li>赛题数据中，字符分布是怎么样的？<h2 id="句子长度分析"><a href="#句子长度分析" class="headerlink" title="句子长度分析"></a>句子长度分析</h2></li></ul><p>在赛题数据中每行句子的字符使用空格进行隔开，所以可以直接统计单词的个数来得到每个句子的长度。统计并如下：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">%pylab inline</span><br><span class="line">train_df[<span class="string">'text_len'</span>] = train_df[<span class="string">'text'</span>].apply(<span class="keyword">lambda</span> x: len(x.split(<span class="string">' '</span>)))</span><br><span class="line">print(train_df[<span class="string">'text_len'</span>].describe())</span><br><span class="line"></span><br><span class="line">Populating the interactive namespace <span class="keyword">from</span> numpy <span class="keyword">and</span> matplotlib</span><br><span class="line">count <span class="number">100.000000</span></span><br><span class="line">mean <span class="number">872.320000</span></span><br><span class="line">std <span class="number">923.138191</span></span><br><span class="line">min <span class="number">64.000000</span></span><br><span class="line"><span class="number">25</span>% <span class="number">359.500000</span></span><br><span class="line"><span class="number">50</span>% <span class="number">598.000000</span></span><br><span class="line"><span class="number">75</span>% <span class="number">1058.000000</span></span><br><span class="line">max <span class="number">7125.000000</span></span><br><span class="line">Name: text_len, dtype: float64</span><br></pre></td></tr></table></figure><h2 id="新闻类别分布"><a href="#新闻类别分布" class="headerlink" title="新闻类别分布"></a>新闻类别分布</h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">train_df[<span class="string">'label'</span>].value_counts().plot(kind=<span class="string">'bar'</span>)</span><br><span class="line">plt.title(<span class="string">'News class count'</span>)</span><br><span class="line">plt.xlabel(<span class="string">"category"</span>)</span><br></pre></td></tr></table></figure><p>在数据集中标签的对应的关系如下：{‘科技’: 0, ‘股票’: 1, ‘体育’: 2, ‘娱乐’: 3, ‘时政’: 4, ‘社会’: 5, ‘教育’: 6, ‘财经’: 7, ‘家居’: 8, ‘游戏’: 9, ‘房产’: 10, ‘时尚’: 11, ‘彩票’: 12, ‘星座’: 13}</p><p>从统计结果可以看出，赛题的数据集类别分布存在较为不均匀的情况。在训练集中科技类新闻最多，其次是股票类新闻，最少的新闻是星座新闻。</p><h2 id="字符分布统计"><a href="#字符分布统计" class="headerlink" title="字符分布统计"></a>字符分布统计</h2><p>接下来可以统计每个字符出现的次数，首先可以将训练集中所有的句子进行拼接进而划分为字符，并统计每个字符的个数。</p><p>从统计结果中可以看出，在训练集中总共包括6869个字，其中编号3750的字出现的次数最多，编号3133的字出现的次数最少。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> collections <span class="keyword">import</span> Counter</span><br><span class="line">all_lines = <span class="string">' '</span>.join(list(train_df[<span class="string">'text'</span>]))</span><br><span class="line">word_count = Counter(all_lines.split(<span class="string">" "</span>))</span><br><span class="line">word_count = sorted(word_count.items(), key=<span class="keyword">lambda</span> d:d[<span class="number">1</span>], reverse = <span class="literal">True</span>)</span><br><span class="line"></span><br><span class="line">print(len(word_count)) <span class="comment"># 2405</span></span><br><span class="line"></span><br><span class="line">print(word_count[<span class="number">0</span>]) <span class="comment"># ('3750', 3702)</span></span><br><span class="line"></span><br><span class="line">print(word_count[<span class="number">-1</span>]) <span class="comment"># ('5034', 1)</span></span><br></pre></td></tr></table></figure><p>这里还可以根据字在每个句子的出现情况，反推出标点符号。下面代码统计了不同字符在句子中出现的次数，其中字符3750，字符900和字符648在20w新闻的覆盖率接近99%，很有可能是标点符号。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> collections <span class="keyword">import</span> Counter</span><br><span class="line">train_df[<span class="string">'text_unique'</span>] = train_df[<span class="string">'text'</span>].apply(<span class="keyword">lambda</span> x: <span class="string">' '</span>.join(list(set(x.split(<span class="string">' '</span>)))))</span><br><span class="line">all_lines = <span class="string">' '</span>.join(list(train_df[<span class="string">'text_unique'</span>]))</span><br><span class="line">word_count = Counter(all_lines.split(<span class="string">" "</span>))</span><br><span class="line">word_count = sorted(word_count.items(), key=<span class="keyword">lambda</span> d:int(d[<span class="number">1</span>]), reverse = <span class="literal">True</span>)</span><br><span class="line"></span><br><span class="line">print(word_count[:<span class="number">5</span>]) <span class="comment"># [('900', 99), ('3750', 99), ('648', 96), ('7399', 87), ('2109', 86)]</span></span><br></pre></td></tr></table></figure><h1 id="数据分析的结论"><a href="#数据分析的结论" class="headerlink" title="数据分析的结论"></a>数据分析的结论</h1><p>通过上述分析我们可以得出以下结论：</p><ol><li>赛题中每个新闻包含的字符个数平均为1000个，还有一些新闻字符较长；</li><li>赛题中新闻类别分布不均匀，科技类新闻样本量接近4w，星座类新闻样本量不到1k；</li><li>赛题总共包括7000-8000个字符；</li></ol><p>通过数据分析，我们还可以得出以下结论：</p><ol><li>每个新闻平均字符个数较多，可能需要截断；</li><li>由于类别不均衡，会严重影响模型的精度；</li></ol><h1 id="本章作业"><a href="#本章作业" class="headerlink" title="本章作业"></a>本章作业</h1><ol><li>假设字符3750，字符900和字符648是句子的标点符号，请分析赛题每篇新闻平均由多少个句子构成？<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">train_df[<span class="string">'length'</span>] = train_df[<span class="string">'text'</span>].apply(<span class="keyword">lambda</span> x: x.count(<span class="string">'3750'</span>) + x.count(<span class="string">'900'</span>) + x.count(<span class="string">'648'</span>))</span><br><span class="line">print(sum(train_df[<span class="string">'length'</span>]) / <span class="number">100</span>)</span><br></pre></td></tr></table></figure></li><li>统计每类新闻中出现次数对多的字符<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line">lst = [[] <span class="keyword">for</span> _ <span class="keyword">in</span> range(<span class="number">14</span>)]</span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> range(len(train_df)):</span><br><span class="line"> lst[train_df[<span class="string">'label'</span>][i]].extend(train_df[<span class="string">'text'</span>][i].split())</span><br><span class="line">lst_freq = list(map(Counter, lst))</span><br><span class="line">lst_freq = [sorted(lst.items(), key=<span class="keyword">lambda</span> d:int(d[<span class="number">1</span>]), reverse = <span class="literal">True</span>) <span class="keyword">for</span> lst <span class="keyword">in</span> lst_freq]</span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> range(len(lst_freq)):</span><br><span class="line"> print(lst_freq[i][<span class="number">0</span>])</span><br><span class="line"> </span><br><span class="line">(<span class="string">'3750'</span>, <span class="number">610</span>)</span><br><span class="line">(<span class="string">'3750'</span>, <span class="number">531</span>)</span><br><span class="line">(<span class="string">'3750'</span>, <span class="number">956</span>)</span><br><span class="line">(<span class="string">'3750'</span>, <span class="number">239</span>)</span><br><span class="line">(<span class="string">'3750'</span>, <span class="number">78</span>)</span><br><span class="line">(<span class="string">'3750'</span>, <span class="number">193</span>)</span><br><span class="line">(<span class="string">'3750'</span>, <span class="number">491</span>)</span><br><span class="line">(<span class="string">'3750'</span>, <span class="number">214</span>)</span><br><span class="line">(<span class="string">'3750'</span>, <span class="number">68</span>)</span><br><span class="line">(<span class="string">'3750'</span>, <span class="number">51</span>)</span><br><span class="line">(<span class="string">'3750'</span>, <span class="number">152</span>)</span><br><span class="line">(<span class="string">'3750'</span>, <span class="number">102</span>)</span><br><span class="line">(<span class="string">'4464'</span>, <span class="number">59</span>)</span><br><span class="line">(<span class="string">'648'</span>, <span class="number">6</span>)</span><br></pre></td></tr></table></figure></li></ol>]]></content>
<summary type="html">
<h1 id="数据读取"><a href="#数据读取" class="headerlink" title="数据读取"></a>数据读取</h1><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line">train_df = pd.read_csv(<span class="string">'./data/train_set.csv'</span>, sep=<span class="string">'\t'</span>, nrows=<span class="number">100</span>)</span><br><span class="line">train_df.head()</span><br></pre></td></tr></table></figure>
<table>
<thead>
<tr>
<th></th>
<th>label</th>
<th>text</th>
</tr>
</thead>
<tbody><tr>
<td>0</td>
<td>2</td>
<td>2967 6758 339 2021 1854 3731 4109 3792 4149 15…</td>
</tr>
<tr>
<td>1</td>
<td>11</td>
<td>4464 486 6352 5619 2465 4802 1452 3137 5778 54…</td>
</tr>
<tr>
<td>2</td>
<td>3</td>
<td>7346 4068 5074 3747 5681 6093 1777 2226 7354 6…</td>
</tr>
<tr>
<td>3</td>
<td>2</td>
<td>7159 948 4866 2109 5520 2490 211 3956 5520 549…</td>
</tr>
<tr>
<td>4</td>
<td>3</td>
<td>3646 3055 3055 2490 4659 6065 3370 5814 2465 5…</td>
</tr>
</tbody></table>
</summary>
<category term="NLP" scheme="https://vincent507cpu.github.io/tags/NLP/"/>
</entry>
<entry>
<title>[DSU&阿里云天池] Python训练营 Task 4</title>
<link href="https://vincent507cpu.github.io/2020/12/28/DSU-%E9%98%BF%E9%87%8C%E4%BA%91%E5%A4%A9%E6%B1%A0-Python%E8%AE%AD%E7%BB%83%E8%90%A5-Task-4/"/>
<id>https://vincent507cpu.github.io/2020/12/28/DSU-%E9%98%BF%E9%87%8C%E4%BA%91%E5%A4%A9%E6%B1%A0-Python%E8%AE%AD%E7%BB%83%E8%90%A5-Task-4/</id>
<published>2020-12-28T15:36:48.000Z</published>
<updated>2020-12-29T01:23:08.607Z</updated>
<content type="html"><![CDATA[<h1 id="数据载入"><a href="#数据载入" class="headerlink" title="数据载入"></a>数据载入</h1><h2 id="pd-read-csv"><a href="#pd-read-csv" class="headerlink" title="pd.read_csv"></a><code>pd.read_csv</code></h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">pd.read_csv(filepath, sep, names)</span><br></pre></td></tr></table></figure><ul><li><code>filepath</code>：待读取文件的路径</li><li><code>sep</code>：CSV 文件的 delimiter</li><li><code>names</code>：列名的列表<a id="more"></a><h2 id="df-merge"><a href="#df-merge" class="headerlink" title="df.merge"></a><code>df.merge</code></h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">df.merge(left, right)</span><br></pre></td></tr></table></figure></li><li><code>left</code>, <code>right</code>：合并的两个数据框<h2 id="df-head"><a href="#df-head" class="headerlink" title="df.head"></a><code>df.head</code></h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">df.head(n=<span class="number">5</span>)</span><br></pre></td></tr></table></figure>查看最上面的 n 行内容。</li></ul><hr><p><strong>数据载入的要点</strong>：需要将所需数据合并在一起。</p><h1 id="数据清洗"><a href="#数据清洗" class="headerlink" title="数据清洗"></a>数据清洗</h1><h2 id="pd-DataFrame"><a href="#pd-DataFrame" class="headerlink" title="pd.DataFrame"></a><code>pd.DataFrame</code></h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">pd.DataFrame(df, columes)</span><br></pre></td></tr></table></figure><p><code>df</code>：目标数据框<br><code>columns</code>：提取的列</p><h2 id="df-shape"><a href="#df-shape" class="headerlink" title="df.shape"></a><code>df.shape</code></h2><p>获得数据框的维度</p><h2 id="df-info"><a href="#df-info" class="headerlink" title="df.info()"></a><code>df.info()</code></h2><p>打印列表的简单信息</p><h2 id="df-fillna"><a href="#df-fillna" class="headerlink" title="df.fillna"></a><code>df.fillna</code></h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">df.fillna(value, inplace)</span><br></pre></td></tr></table></figure><ul><li><code>value</code>：用来填补缺失值的值</li><li><code>inplace</code>：是否在原地完成操作<h2 id="df-satype"><a href="#df-satype" class="headerlink" title="df.satype"></a><code>df.satype</code></h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">df.astype(dtype)</span><br></pre></td></tr></table></figure></li><li><code>dtype</code>：指定的数据类型<h2 id="df-describe"><a href="#df-describe" class="headerlink" title="df.describe"></a><code>df.describe</code></h2>打印数据框的统计信息</li></ul><hr><p><strong>数据清洗的要点</strong>：检查数据是否有缺失值，如果有，需要采取相应的行为（填补或删除）。</p><h1 id="数据分析"><a href="#数据分析" class="headerlink" title="数据分析"></a>数据分析</h1><h2 id="df-groupby"><a href="#df-groupby" class="headerlink" title="df.groupby"></a><code>df.groupby</code></h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">df.groupby(by)</span><br></pre></td></tr></table></figure><ul><li><code>by</code>：确定整合成一组的标准<h2 id="df-sum"><a href="#df-sum" class="headerlink" title="df.sum"></a><code>df.sum</code></h2>求和<h2 id="df-sort-values"><a href="#df-sort-values" class="headerlink" title="df.sort_values"></a><code>df.sort_values</code></h2><figure class="highlight plain"><figcaption><span>ascending)</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">```</span><br><span class="line">- `by`：排序依据的行或列</span><br><span class="line">- `ascending`：`True` 则升序排列</span><br><span class="line">## `df.value_counts`</span><br><span class="line">返回不同行的计数</span><br><span class="line">***</span><br><span class="line">**数据分析的要点**：通过合并分组、排序、计数等手段，获得数据的信息。</span><br><span class="line"># 数据可视化</span><br><span class="line">## `df.plot(kind)`</span><br><span class="line">```py</span><br><span class="line">df.plot(kind)</span><br></pre></td></tr></table></figure><code>kind</code>：可视化类型</li><li><code>line</code>：折线图</li><li><code>bar</code>：柱状图</li><li><code>hist</code>：直方图</li><li><code>box</code>：箱型图</li><li><code>kde</code>：密度图</li><li><code>pie</code>：pie 图</li><li><code>scatter</code>：散点图</li></ul><hr><p><strong>数据可视化的要点</strong>：根据不同的数据结构和需求选择合适的可视化方法。</p>]]></content>
<summary type="html">
<h1 id="数据载入"><a href="#数据载入" class="headerlink" title="数据载入"></a>数据载入</h1><h2 id="pd-read-csv"><a href="#pd-read-csv" class="headerlink" title="pd.read_csv"></a><code>pd.read_csv</code></h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">pd.read_csv(filepath, sep, names)</span><br></pre></td></tr></table></figure>
<ul>
<li><code>filepath</code>：待读取文件的路径</li>
<li><code>sep</code>：CSV 文件的 delimiter</li>
<li><code>names</code>：列名的列表
</summary>
<category term="Python" scheme="https://vincent507cpu.github.io/tags/Python/"/>
</entry>
<entry>
<title>[经验总结]一行代码完成一个任务</title>
<link href="https://vincent507cpu.github.io/2020/12/26/%E7%BB%8F%E9%AA%8C%E6%80%BB%E7%BB%93-%E4%B8%80%E8%A1%8C%E4%BB%A3%E7%A0%81%E5%AE%8C%E6%88%90%E4%B8%80%E4%B8%AA%E4%BB%BB%E5%8A%A1/"/>
<id>https://vincent507cpu.github.io/2020/12/26/%E7%BB%8F%E9%AA%8C%E6%80%BB%E7%BB%93-%E4%B8%80%E8%A1%8C%E4%BB%A3%E7%A0%81%E5%AE%8C%E6%88%90%E4%B8%80%E4%B8%AA%E4%BB%BB%E5%8A%A1/</id>
<published>2020-12-27T03:33:05.000Z</published>
<updated>2020-12-27T03:34:55.804Z</updated>
<content type="html"><![CDATA[<p>2020 年的最后一周写一点轻松的东西，熟练使用这些技巧会给日常工作减负不少：本文中会提到几个使用一行代码可以完成简单任务的方法。</p><a id="more"></a><h1 id="替代-if-语句：Ternary-Operator"><a href="#替代-if-语句：Ternary-Operator" class="headerlink" title="替代 if 语句：Ternary Operator"></a>替代 <code>if</code> 语句：Ternary Operator<a href="https://book.pythontips.com/en/latest/ternary_operators.html" target="_blank" rel="noopener" title="Ternary Operator"></a></h1><h2 id="标准用法"><a href="#标准用法" class="headerlink" title="标准用法"></a>标准用法</h2><p>先来看一个简单的 <code>if</code> 语句：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">if</span> <span class="number">5</span> > <span class="number">3</span>:</span><br><span class="line"> print(<span class="string">'5 is more than 3'</span>)</span><br><span class="line"><span class="keyword">else</span>:</span><br><span class="line"> print(<span class="string">'5 is less than 3'</span>)</span><br><span class="line"><span class="comment"># '5 is more than 3'</span></span><br></pre></td></tr></table></figure><p>怎么用一行代码完成同样的任务呢？有请 Ternary Operator：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">True_expression <span class="keyword">if</span> True_condition <span class="keyword">else</span> False_expression</span><br></pre></td></tr></table></figure><p>意思是这样的：判断 <code>True_condition</code> 是否成立，如果成立，执行 <code>True_expression</code>；如果不成立，执行 <code>False_expression</code>。那么上面的 <code>if</code> 语句可以写成</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="string">'5 is more than 3'</span> <span class="keyword">if</span> <span class="number">5</span> > <span class="number">3</span> <span class="keyword">else</span> <span class="string">'5 is less than 3'</span></span><br><span class="line"><span class="comment"># '5 is more than 3'</span></span><br></pre></td></tr></table></figure><h2 id="Ternary-Operator-的变体"><a href="#Ternary-Operator-的变体" class="headerlink" title="Ternary Operator 的变体"></a>Ternary Operator 的变体</h2><ol><li>Shorthanded Ternary<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">expression <span class="keyword">or</span> another_expression</span><br></pre></td></tr></table></figure>如果 <code>expression</code> 为 <code>True</code>，那么 <code>another_expression</code> 则不会执行；反之则会执行。比如：<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="literal">True</span> <span class="keyword">or</span> <span class="string">'something'</span></span><br><span class="line"><span class="comment"># True</span></span><br><span class="line"><span class="literal">False</span> <span class="keyword">or</span> <span class="string">'something'</span></span><br><span class="line"><span class="comment">#'something'</span></span><br></pre></td></tr></table></figure>因为 <code>None</code> 等于 <code>False</code>，这可以用于检查一个函数是否有输出：<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">output = <span class="literal">None</span></span><br><span class="line">msg = output <span class="keyword">or</span> <span class="string">'No data returned'</span></span><br><span class="line"><span class="comment"># 'No data returned'</span></span><br></pre></td></tr></table></figure>也可以用来设定一个函数的动态变量：<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">display_name</span><span class="params">(name, default=None)</span>:</span></span><br><span class="line"> displayeded_name = dafault <span class="keyword">or</span> name</span><br><span class="line"> print(displayed_name)</span><br><span class="line">display_name(<span class="string">'Jim'</span>) <span class="comment"># 'Jim'</span></span><br><span class="line">display_name(<span class="string">'Tom'</span>, <span class="string">'anonymous123'</span>) <span class="comment"># 'anonymous123'</span></span><br></pre></td></tr></table></figure></li><li>这个变体同样巧妙地应用了 <code>False == 0</code> 和 <code>True == 1</code> 这两个性质。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">(expr_if_False, expr_if_true)[True_or_False]</span><br></pre></td></tr></table></figure>这里 <code>(expr_if_False, expr_if_true)</code> 用了 tuple，但是也可以用 list。也可以用 <code>bool()</code> 函数将所有变量转换为 <code>0</code> 或 <code>1</code>，因为 <code>False == 0</code>，当 <code>True_or_False</code> 为假时会取第一个元素；当 <code>True_or_False</code> 为真时会取第二个元素。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">output = <span class="string">'blabla'</span></span><br><span class="line">[<span class="string">'Not have an output'</span>, <span class="string">'Has an output'</span>][bool(output)]</span><br><span class="line"><span class="comment"># 'Has an output'</span></span><br></pre></td></tr></table></figure><h1 id="替代-def-语句：lambda-匿名函数"><a href="#替代-def-语句：lambda-匿名函数" class="headerlink" title="替代 def 语句：lambda 匿名函数"></a>替代 <code>def</code> 语句：<code>lambda</code> 匿名函数</h1>再来看一个简单的自定义函数：<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">add</span><span class="params">(x, y)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> x + y</span><br><span class="line">add(<span class="number">1</span>, <span class="number">2</span>) <span class="comment"># 3</span></span><br></pre></td></tr></table></figure>这个简单的函数可以使用 <code>lambda</code> 匿名函数完成。<code>lambda</code> 匿名函数必须在一行内完成，语法为：<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">lambda</span> arguments: expression</span><br></pre></td></tr></table></figure>那么上面的两数相加的函数也可以写成：<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">add = <span class="keyword">lambda</span> x, y: x + y</span><br><span class="line">add(<span class="number">1</span>, <span class="number">2</span>) <span class="comment"># 3</span></span><br></pre></td></tr></table></figure><h1 id="一行代码生成一个容器：解析式"><a href="#一行代码生成一个容器：解析式" class="headerlink" title="一行代码生成一个容器：解析式"></a>一行代码生成一个容器：解析式</h1><h2 id="基础列表解析式"><a href="#基础列表解析式" class="headerlink" title="基础列表解析式"></a>基础列表解析式</h2>假如我们希望写一个函数将一个列表中的数字平方在返回一个新列表：<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">square</span><span class="params">(lst)</span>:</span></span><br><span class="line"> new_lst = []</span><br><span class="line"> <span class="keyword">for</span> num <span class="keyword">in</span> lst:</span><br><span class="line"> new_lst.append(num**<span class="number">2</span>)</span><br><span class="line"> <span class="keyword">return</span> new_lst</span><br></pre></td></tr></table></figure>这么一个简单的函数居然要用 5 行代码，尴尬症都要犯了。解析式来救场！解析式又分列表解析式、集合解析式和字典解析式，以列表解析式为例，语法为：<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[expression <span class="keyword">for</span> var <span class="keyword">in</span> iterable <span class="keyword">if</span> condition]</span><br></pre></td></tr></table></figure>所以我们可以将上面的函数改写为：<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[var**<span class="number">2</span> <span class="keyword">for</span> var <span class="keyword">in</span> lst]</span><br></pre></td></tr></table></figure>一行搞定！要注意的是，只有当元素符合判定条件（<code>if</code>）时才会被处理，解析式里没有 <code>else</code>。<h2 id="嵌套列表解析式"><a href="#嵌套列表解析式" class="headerlink" title="嵌套列表解析式"></a><a href="https://spapas.github.io/2016/04/27/python-nested-list-comprehensions/" target="_blank" rel="noopener" title="嵌套列表解析式">嵌套列表解析式</a></h2>更复杂的列表解析式包含多个变量：<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[expression <span class="keyword">for</span> var1 <span class="keyword">in</span> iterable1 <span class="keyword">if</span> condition1 <span class="keyword">for</span> var2 <span class="keyword">in</span> iterable2 <span class="keyword">if</span> condition2]</span><br></pre></td></tr></table></figure>我们应该怎么写呢？根据解析式的 PEP202 文档，<blockquote><p>It is proposed to allow conditional construction of list literals using for and if clauses. <em>They would nest in the same way for loops and if statements nest now.</em></p></blockquote></li></ol><p>如果我们想写一个 <code>for</code> 循环来做同样的事情，我们可能这样写：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">for</span> var1 <span class="keyword">in</span> iterable1:</span><br><span class="line"> <span class="keyword">if</span> condition1:</span><br><span class="line"> <span class="keyword">for</span> var2 <span class="keyword">in</span> iterable2:</span><br><span class="line"> <span class="keyword">if</span> condition2:</span><br><span class="line"> expression</span><br></pre></td></tr></table></figure><p>使用解析式，我们只需要将正常写法最后的表达式写在最前面，之后依次把嵌套循环的代码依次写在同一行就行了。我们来看一个终极例子：</p><blockquote><p>有一个嵌套单词列表，如果单词有至少两个字母，则返回一个包含所有字母与它所在单词的列表的索引的新列表。</p></blockquote><p>如果我们用常规方法，可能会这么写：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">new_lst = []</span><br><span class="line"><span class="keyword">for</span> idx, lst <span class="keyword">in</span> enumerate(nested_lst):</span><br><span class="line"> <span class="keyword">for</span> word <span class="keyword">in</span> lst:</span><br><span class="line"> <span class="keyword">if</span> len(word) >= <span class="number">2</span>:</span><br><span class="line"> <span class="keyword">for</span> letter <span class="keyword">in</span> word:</span><br><span class="line"> new_lst.append((letter, idx))</span><br></pre></td></tr></table></figure><p>使用列表解析式可以这么写：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">strings = [ [<span class="string">'foo'</span>, <span class="string">'bar'</span>], [<span class="string">'baz'</span>, <span class="string">'taz'</span>], [<span class="string">'w'</span>, <span class="string">'koko'</span>] ]</span><br><span class="line">[ (letter, idx) <span class="keyword">for</span> idx, lst <span class="keyword">in</span> enumerate(strings) <span class="keyword">for</span> word <span class="keyword">in</span> lst <span class="keyword">if</span> len(word)><span class="number">2</span> <span class="keyword">for</span> letter <span class="keyword">in</span> word]</span><br><span class="line"><span class="comment"># [('f', 0), ('o', 0), ('o', 0), ('b', 0), ('a', 0), ('r', 0), ('b', 1), ('a', 1), ('z', 1), ('t', 1), ('a', 1), ('z', 1), ('k', 2), ('o', 2), ('k', 2), ('o', 2)]</span></span><br></pre></td></tr></table></figure><h2 id="集合解析式与字典解析式"><a href="#集合解析式与字典解析式" class="headerlink" title="集合解析式与字典解析式"></a>集合解析式与字典解析式</h2><p>集合解析式与列表解析式差不多，区别仅仅是将 <code>[]</code> 换成了 <code>{}</code>。字典解析式与列表解析式的区别在于将 <code>var</code> 换成了键值对<code>key, val</code>。比如在 NLP 应用中，需要生成一个词与索引的 lookup 字典：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">id2word = {idx:word <span class="keyword">for</span> (idx, word) <span class="keyword">in</span> enumerate(corpus)}</span><br><span class="line">word2id = {word:id <span class="keyword">for</span> (id, word) <span class="keyword">in</span> id2word.items()}</span><br></pre></td></tr></table></figure><h1 id="替代-yield-语句：生成器"><a href="#替代-yield-语句：生成器" class="headerlink" title="替代 yield 语句：生成器"></a>替代 <code>yield</code> 语句：生成器</h1><p><code>yield</code> 语句是一种一边循环一边计算的机制，将一个函数中的 <code>return</code> 替换成 <code>yield</code>，使用 <code>next</code> 调用该函数：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">generator</span><span class="params">(x)</span>:</span></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(x):</span><br><span class="line"> <span class="keyword">yield</span> i</span><br><span class="line">x = generator(<span class="number">5</span>)</span><br><span class="line">x <span class="comment"># <generator object generator at 0x7fee19239c80></span></span><br><span class="line">next(x) <span class="comment"># 0</span></span><br><span class="line">next(x) <span class="comment"># 1</span></span><br><span class="line">...</span><br></pre></td></tr></table></figure><p>也可以使用一行代码将一个简单的 <code>yield</code> 语句实现，这就是生成器。生成器与列表解析式的语法一样，只是用 <code>()</code> 代替了 <code>[]</code>。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">x = (i <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">5</span>))</span><br><span class="line">x</span><br><span class="line">next(x) <span class="comment"># 0</span></span><br><span class="line">...</span><br></pre></td></tr></table></figure><hr><p>本文中的技巧仅可用于一些简单的任务，如果任务比较复杂，这些技巧要么无法完成，要么可读性大幅下降而变得难以理解与维护。</p>]]></content>
<summary type="html">
<p>2020 年的最后一周写一点轻松的东西，熟练使用这些技巧会给日常工作减负不少：本文中会提到几个使用一行代码可以完成简单任务的方法。</p>
</summary>
<category term="经验总结" scheme="https://vincent507cpu.github.io/tags/%E7%BB%8F%E9%AA%8C%E6%80%BB%E7%BB%93/"/>
</entry>
<entry>
<title>[DSU&阿里云天池] Python训练营 Task3</title>
<link href="https://vincent507cpu.github.io/2020/12/24/DSU-%E9%98%BF%E9%87%8C%E4%BA%91%E5%A4%A9%E6%B1%A0-Python%E8%AE%AD%E7%BB%83%E8%90%A5-Task3/"/>
<id>https://vincent507cpu.github.io/2020/12/24/DSU-%E9%98%BF%E9%87%8C%E4%BA%91%E5%A4%A9%E6%B1%A0-Python%E8%AE%AD%E7%BB%83%E8%90%A5-Task3/</id>
<published>2020-12-24T20:44:26.000Z</published>
<updated>2020-12-26T02:20:00.652Z</updated>
<content type="html"><![CDATA[<h1 id="函数与-Lambda-表达式"><a href="#函数与-Lambda-表达式" class="headerlink" title="函数与 Lambda 表达式"></a>函数与 Lambda 表达式</h1><h2 id="函数"><a href="#函数" class="headerlink" title="函数"></a>函数</h2><h3 id="函数的定义"><a href="#函数的定义" class="headerlink" title="函数的定义"></a>函数的定义</h3><ol><li>函数以 <code>def</code> 关键词开头，后接函数名和圆括号 <code>()</code>。</li><li>函数执行的代码以冒号起始，并且缩进。</li><li><code>return [表达式]</code> 结束函数，选择性地返回一个值给调用方。不带表达式的 return 相当于返回 <code>None</code> 。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">functionname</span><span class="params">(parameters)</span>:</span></span><br><span class="line"> <span class="string">"函数_文档字符串"</span></span><br><span class="line"> function_suite</span><br><span class="line"> <span class="keyword">return</span> [expression]</span><br></pre></td></tr></table></figure><a id="more"></a><h3 id="函数的调用"><a href="#函数的调用" class="headerlink" title="函数的调用"></a>函数的调用</h3><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">printme</span><span class="params">(str)</span>:</span></span><br><span class="line"> print(str)</span><br><span class="line">printme(<span class="string">"我要调用用户自定义函数!"</span>) <span class="comment"># 我要调用用户自定义函数! </span></span><br><span class="line">printme(<span class="string">"再次调用同一函数"</span>) <span class="comment"># 再次调用同一函数</span></span><br><span class="line">temp = printme(<span class="string">'hello'</span>) <span class="comment"># hello</span></span><br><span class="line">print(temp) <span class="comment"># None</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">add</span><span class="params">(a, b)</span>:</span></span><br><span class="line"> print(a + b)</span><br><span class="line">add(<span class="number">1</span>, <span class="number">2</span>) <span class="comment"># 3</span></span><br><span class="line">add([<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>], [<span class="number">4</span>, <span class="number">5</span>, <span class="number">6</span>]) <span class="comment"># [1, 2, 3, 4, 5, 6]</span></span><br></pre></td></tr></table></figure><h3 id="函数文档"><a href="#函数文档" class="headerlink" title="函数文档"></a>函数文档</h3><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">MyFirstFunction</span><span class="params">(name)</span>:</span></span><br><span class="line"> <span class="string">"函数定义过程中 name 是形参"</span> </span><br><span class="line"> <span class="comment"># 因为 Ta 只是一个形式，表示占据一个参数位置 </span></span><br><span class="line"> print(<span class="string">'传递进来的 {0} 叫做实参，因为 Ta 是具体的参数值!'</span>.format(name))</span><br><span class="line"> </span><br><span class="line">MyFirstFunction(<span class="string">'老马的程序人生'</span>)</span><br><span class="line"><span class="comment"># 传递进来的 老马的程序人生 叫做实参，因为Ta是具体的参数值!</span></span><br><span class="line"></span><br><span class="line">print(MyFirstFunction.__doc__) </span><br><span class="line"><span class="comment"># 函数定义过程中name是形参</span></span><br><span class="line"></span><br><span class="line">help(MyFirstFunction)</span><br><span class="line"><span class="comment"># Help on function MyFirstFunction in module __main__: </span></span><br><span class="line"><span class="comment"># MyFirstFunction(name)</span></span><br><span class="line"><span class="comment"># 函数定义过程中name是形参</span></span><br></pre></td></tr></table></figure><h3 id="函数参数"><a href="#函数参数" class="headerlink" title="函数参数"></a>函数参数</h3></li><li>位置参数<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">functionname</span><span class="params">(arg1)</span>:</span> </span><br><span class="line"> <span class="string">"函数_文档字符串"</span></span><br><span class="line"> function_suite</span><br><span class="line"> <span class="keyword">return</span> [expression]</span><br></pre></td></tr></table></figure><code>arg1</code> - 位置参数 ，这些参数在调用函数 (call function) 时位置要固定。</li><li>默认参数<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">functionname</span><span class="params">(arg1, arg2=v)</span>:</span> </span><br><span class="line"> <span class="string">"函数_文档字符串"</span></span><br><span class="line"> function_suite</span><br><span class="line"> <span class="keyword">return</span> [expression]</span><br></pre></td></tr></table></figure></li></ol><ul><li><code>arg2 = v</code> - 默认参数 = 默认值，调用函数时，默认参数的值如果没有传入，则被认为是默认值。</li><li>默认参数一定要放在位置参数后面，不然程序会报错。</li><li>Python 允许函数调用时参数的顺序与声明时不一致，因为 Python 解释器能够用参数名匹配参数值。</li></ul><ol start="3"><li>可变参数<br>可变参数就是传入的参数个数是可变的，可以是 0, 1, 2 到任意个，是不定长的参数。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">functionname</span><span class="params">(arg1, arg2=v, *args)</span>:</span> <span class="string">"函数_文档字符串"</span></span><br><span class="line"> function_suite</span><br><span class="line"> <span class="keyword">return</span> [expression]</span><br></pre></td></tr></table></figure></li></ol><ul><li><code>*args</code> - 可变参数，可以是从零个到任意个，自动组装成元组。 </li><li>加了星号(*)的变量名会存放所有未命名的变量参数。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">printinfo</span><span class="params">(arg1, *args)</span>:</span></span><br><span class="line"> print(arg1)</span><br><span class="line"> <span class="keyword">for</span> var <span class="keyword">in</span> args:</span><br><span class="line"> print(var)</span><br><span class="line">printinfo(<span class="number">10</span>) <span class="comment"># 10</span></span><br><span class="line">printinfo(<span class="number">70</span>, <span class="number">60</span>, <span class="number">50</span>)</span><br><span class="line"><span class="comment"># 70 </span></span><br><span class="line"><span class="comment"># 60 </span></span><br><span class="line"><span class="comment"># 50</span></span><br></pre></td></tr></table></figure></li></ul><ol start="4"><li>关键字参数<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">functionname</span><span class="params">(arg1, arg2=v, *args, **kw)</span>:</span> </span><br><span class="line"> <span class="string">"函数_文档字符串"</span></span><br><span class="line"> function_suite</span><br><span class="line"> <span class="keyword">return</span> [expression]</span><br></pre></td></tr></table></figure><code>**kw</code> - 关键字参数，可以是从零个到任意个，自动组装成字典。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">printinfo</span><span class="params">(arg1, *args, **kwargs)</span>:</span></span><br><span class="line"> print(arg1)</span><br><span class="line"> print(args)</span><br><span class="line"> print(kwargs)</span><br><span class="line">printinfo(<span class="number">70</span>, <span class="number">60</span>, <span class="number">50</span>)</span><br><span class="line"><span class="comment"># 70</span></span><br><span class="line"><span class="comment"># (60, 50)</span></span><br><span class="line"><span class="comment"># {}</span></span><br><span class="line">printinfo(<span class="number">70</span>, <span class="number">60</span>, <span class="number">50</span>, a=<span class="number">1</span>, b=<span class="number">2</span>)</span><br><span class="line"><span class="comment"># 70</span></span><br><span class="line"><span class="comment"># (60, 50)</span></span><br><span class="line"><span class="comment"># {'a': 1, 'b': 2}</span></span><br></pre></td></tr></table></figure></li><li>命名关键字参数<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">functionname</span><span class="params">(arg1, arg2=v, *args, *, nkw, **kw)</span>:</span></span><br><span class="line"> <span class="string">"函数_文档字符串"</span></span><br><span class="line"> function_suite</span><br><span class="line"> <span class="keyword">return</span> [expression]</span><br></pre></td></tr></table></figure></li></ol><ul><li><code>*, nkw</code> - 命名关键字参数，用户想要输入的关键字参数，定义方式是在nkw 前面加个分隔符 <code>*</code>。</li><li>如果要限制关键字参数的名字，就可以用「命名关键字参数」</li><li>使用命名关键字参数时，要特别注意不能缺少参数名。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">printinfo</span><span class="params">(arg1, *, nkw, **kwargs)</span>:</span></span><br><span class="line"> print(arg1)</span><br><span class="line"> print(nkw)</span><br><span class="line"> print(kwargs)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">printinfo(<span class="number">70</span>, nkw=<span class="number">10</span>, a=<span class="number">1</span>, b=<span class="number">2</span>)</span><br><span class="line"><span class="comment"># 70</span></span><br><span class="line"><span class="comment"># 10</span></span><br><span class="line"><span class="comment"># {'a': 1, 'b': 2}</span></span><br><span class="line"></span><br><span class="line">printinfo(<span class="number">70</span>, <span class="number">10</span>, a=<span class="number">1</span>, b=<span class="number">2</span>)</span><br><span class="line"><span class="comment"># TypeError: printinfo() takes 1 positional argument but 2 were given</span></span><br></pre></td></tr></table></figure></li><li>没有写参数名 <code>nwk</code>，因此 <code>10</code> 被当成「位置参数」，而原函数只有 1 个位置函数，现在调用了 2 个，因此程序会报错。<h3 id="参数组合"><a href="#参数组合" class="headerlink" title="参数组合"></a>参数组合</h3>在 Python 中定义函数，可以用位置参数、默认参数、可变参数、命名关键字参数和关键字参数，这 5 种参数中的 4 个都可以一起使用，但是注意，参数定义的顺序必须是：</li><li>位置参数、默认参数、可变参数和关键字参数。</li><li>位置参数、默认参数、命名关键字参数和关键字参数。</li></ul><p>要注意定义可变参数和关键字参数的语法：</p><ul><li><code>*args</code> 是可变参数，<code>args</code> 接收的是一个 <code>tuple</code></li><li><code>**kw</code> 是关键字参数，<code>kw</code> 接收的是一个 <code>dict</code></li></ul><p>命名关键字参数是为了限制调用者可以传入的参数名，同时可以提供默认值。定义命名关键字参数不要忘了写分隔符 <code>*</code>，否则定义的是位置参数。</p><h2 id="函数的返回值"><a href="#函数的返回值" class="headerlink" title="函数的返回值"></a>函数的返回值</h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">back</span><span class="params">()</span>:</span></span><br><span class="line"> <span class="keyword">return</span> [<span class="number">1</span>, <span class="string">'小马的程序人生'</span>, <span class="number">3.14</span>]</span><br><span class="line">print(back()) <span class="comment"># [1, '小马的程序人生', 3.14]</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">back</span><span class="params">()</span>:</span></span><br><span class="line"> <span class="keyword">return</span> <span class="number">1</span>, <span class="string">'小马的程序人生'</span>, <span class="number">3.14</span></span><br><span class="line">print(back()) <span class="comment"># (1, '小马的程序人生', 3.14)</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">printme</span><span class="params">(str)</span>:</span></span><br><span class="line"> print(str)</span><br><span class="line">temp = printme(<span class="string">'hello'</span>) <span class="comment"># hello</span></span><br><span class="line">print(temp) <span class="comment"># None</span></span><br><span class="line">print(type(temp)) <span class="comment"># <class 'NoneType'></span></span><br></pre></td></tr></table></figure><h2 id="变量的作用域"><a href="#变量的作用域" class="headerlink" title="变量的作用域"></a>变量的作用域</h2><ul><li>Python 中，程序的变量并不是在哪个位置都可以访问的，访问权限决定于这个变量是在哪里赋值的。</li><li>定义在函数内部的变量拥有局部作用域，该变量称为局部变量。</li><li>定义在函数外部的变量拥有全局作用域，该变量称为全局变量。</li><li>局部变量只能在其被声明的函数内部访问，而全局变量可以在整个程序范围内访问。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">discounts</span><span class="params">(price, rate)</span>:</span></span><br><span class="line"> final_price = price * rate</span><br><span class="line"> <span class="keyword">return</span> final_price</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">old_price = float(input(<span class="string">'请输入原价:'</span>)) <span class="comment"># 98</span></span><br><span class="line">rate = float(input(<span class="string">'请输入折扣率:'</span>)) <span class="comment"># 0.9</span></span><br><span class="line">new_price = discounts(old_price, rate)</span><br><span class="line">print(<span class="string">'打折后价格是:%.2f'</span> % new_price) <span class="comment"># 88.20</span></span><br></pre></td></tr></table></figure></li><li>当内部作用域想修改外部作用域的变量时，就要用到 <code>global</code> 和 <code>nonlocal</code> 关键字了。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">num = <span class="number">1</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">fun1</span><span class="params">()</span>:</span></span><br><span class="line"> <span class="keyword">global</span> num <span class="comment"># 需要使用 global 关键字声明</span></span><br><span class="line"> print(num) <span class="comment"># 1</span></span><br><span class="line"> num = <span class="number">123</span></span><br><span class="line"> print(num) <span class="comment"># 123</span></span><br><span class="line"></span><br><span class="line">fun1()</span><br><span class="line">print(num) <span class="comment"># 123</span></span><br></pre></td></tr></table></figure><h3 id="内嵌函数"><a href="#内嵌函数" class="headerlink" title="内嵌函数"></a>内嵌函数</h3><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">outer</span><span class="params">()</span>:</span></span><br><span class="line"> print(<span class="string">'outer函数在这被调用'</span>)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">inner</span><span class="params">()</span>:</span></span><br><span class="line"> print(<span class="string">'inner函数在这被调用'</span>)</span><br><span class="line"></span><br><span class="line"> inner() <span class="comment"># 该函数只能在outer函数内部被调用</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">outer()</span><br><span class="line"><span class="comment"># outer函数在这被调用</span></span><br><span class="line"><span class="comment"># inner函数在这被调用</span></span><br></pre></td></tr></table></figure><h3 id="闭包"><a href="#闭包" class="headerlink" title="闭包"></a>闭包</h3></li><li>是函数式编程的一个重要的语法结构，是一种特殊的内嵌函数。</li><li>如果在一个内部函数里对外层非全局作用域的变量进行引用，那么内部函数就被认为是闭包。</li><li>通过闭包可以访问外层非全局作用域的变量，这个作用域称为闭包作用域。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">funX</span><span class="params">(x)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">funY</span><span class="params">(y)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> x * y</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> funY</span><br><span class="line"></span><br><span class="line">i = funX(<span class="number">8</span>)</span><br><span class="line">print(type(i)) <span class="comment"># <class 'function'></span></span><br><span class="line">print(i(<span class="number">5</span>)) <span class="comment"># 40</span></span><br></pre></td></tr></table></figure>闭包的返回值通常是函数。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">make_counter</span><span class="params">(init)</span>:</span></span><br><span class="line"> counter = [init]</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">inc</span><span class="params">()</span>:</span> counter[<span class="number">0</span>] += <span class="number">1</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">dec</span><span class="params">()</span>:</span> counter[<span class="number">0</span>] -= <span class="number">1</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">get</span><span class="params">()</span>:</span> <span class="keyword">return</span> counter[<span class="number">0</span>]</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">reset</span><span class="params">()</span>:</span> counter[<span class="number">0</span>] = init</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> inc, dec, get, reset</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">inc, dec, get, reset = make_counter(<span class="number">0</span>)</span><br><span class="line">inc()</span><br><span class="line">inc()</span><br><span class="line">inc()</span><br><span class="line">print(get()) <span class="comment"># 3</span></span><br><span class="line">dec()</span><br><span class="line">print(get()) <span class="comment"># 2</span></span><br><span class="line">reset()</span><br><span class="line">print(get()) <span class="comment"># 0</span></span><br></pre></td></tr></table></figure>如果要修改闭包作用域中的变量则需要 <code>nonlocal</code> 关键字<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">outer</span><span class="params">()</span>:</span></span><br><span class="line"> num = <span class="number">10</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">inner</span><span class="params">()</span>:</span></span><br><span class="line"> <span class="keyword">nonlocal</span> num <span class="comment"># nonlocal关键字声明</span></span><br><span class="line"> num = <span class="number">100</span></span><br><span class="line"> print(num)</span><br><span class="line"></span><br><span class="line"> inner()</span><br><span class="line"> print(num)</span><br><span class="line"></span><br><span class="line">outer()</span><br><span class="line"><span class="comment"># 100</span></span><br><span class="line"><span class="comment"># 100</span></span><br></pre></td></tr></table></figure><h3 id="递归"><a href="#递归" class="headerlink" title="递归"></a>递归</h3>如果一个函数在内部调用自身本身，这个函数就是递归函数。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 利用循环</span></span><br><span class="line">n = <span class="number">5</span></span><br><span class="line"><span class="keyword">for</span> k <span class="keyword">in</span> range(<span class="number">1</span>, <span class="number">5</span>):</span><br><span class="line"> n = n * k</span><br><span class="line">print(n) <span class="comment"># 120</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 利用递归</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">factorial</span><span class="params">(n)</span>:</span></span><br><span class="line"> <span class="keyword">if</span> n == <span class="number">1</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="number">1</span></span><br><span class="line"> <span class="keyword">return</span> n * factorial(n - <span class="number">1</span>)</span><br><span class="line"></span><br><span class="line">print(factorial(<span class="number">5</span>)) <span class="comment"># 120</span></span><br></pre></td></tr></table></figure>设置递归的层数，Python默认递归层数为 <code>100</code><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> sys</span><br><span class="line">sys.setrecursionlimit(<span class="number">1000</span>)</span><br></pre></td></tr></table></figure><h2 id="Lambda-表达式"><a href="#Lambda-表达式" class="headerlink" title="Lambda 表达式"></a><code>Lambda</code> 表达式</h2><h3 id="匿名函数的定义"><a href="#匿名函数的定义" class="headerlink" title="匿名函数的定义"></a>匿名函数的定义</h3>在 Python 里有两类函数：</li><li>第一类：用 def 关键词定义的正规函数</li><li>第二类：用 lambda 关键词定义的匿名函数</li></ul><p>Python 使用 lambda 关键词来创建匿名函数，而非def关键词，它没有函数名，其语法结构下：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">lambda</span> argument_list: expression</span><br></pre></td></tr></table></figure><ul><li><code>lambda</code> - 定义匿名函数的关键词。</li><li><code>argument_list</code> - 函数参数，它们可以是位置参数、默认参数、关键字参数，和正规函数里的参数类型一样。</li><li><code>:</code> - 冒号，在函数参数和表达式中间要加个冒号。</li><li><code>expression</code> - 只是一个表达式，输入函数参数，输出一些值。</li></ul><p>注意：</p><ul><li><p><code>expression</code> 中没有 return 语句，因为 lambda 不需要它来返回，表达式本身结果就是返回值。</p></li><li><p>匿名函数拥有自己的命名空间，且不能访问自己参数列表之外或全局命名空间里的参数。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">sqr</span><span class="params">(x)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> x ** <span class="number">2</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">print(sqr)</span><br><span class="line"><span class="comment"># <function sqr at 0x000000BABD3A4400></span></span><br><span class="line"></span><br><span class="line">y = [sqr(x) <span class="keyword">for</span> x <span class="keyword">in</span> range(<span class="number">10</span>)]</span><br><span class="line">print(y)</span><br><span class="line"><span class="comment"># [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]</span></span><br><span class="line"></span><br><span class="line">lbd_sqr = <span class="keyword">lambda</span> x: x ** <span class="number">2</span></span><br><span class="line">print(lbd_sqr)</span><br><span class="line"><span class="comment"># <function <lambda> at 0x000000BABB6AC1E0></span></span><br><span class="line"></span><br><span class="line">y = [lbd_sqr(x) <span class="keyword">for</span> x <span class="keyword">in</span> range(<span class="number">10</span>)]</span><br><span class="line">print(y)</span><br><span class="line"><span class="comment"># [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">sumary = <span class="keyword">lambda</span> arg1, arg2: arg1 + arg2</span><br><span class="line">print(sumary(<span class="number">10</span>, <span class="number">20</span>)) <span class="comment"># 30</span></span><br><span class="line"></span><br><span class="line">func = <span class="keyword">lambda</span> *args: sum(args)</span><br><span class="line">print(func(<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>)) <span class="comment"># 15</span></span><br></pre></td></tr></table></figure><h3 id="匿名函数的应用"><a href="#匿名函数的应用" class="headerlink" title="匿名函数的应用"></a>匿名函数的应用</h3><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">f</span><span class="params">(x)</span>:</span></span><br><span class="line"> y = []</span><br><span class="line"> <span class="keyword">for</span> item <span class="keyword">in</span> x:</span><br><span class="line"> y.append(item + <span class="number">10</span>)</span><br><span class="line"> <span class="keyword">return</span> y</span><br><span class="line"></span><br><span class="line">x = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>]</span><br><span class="line">f(x)</span><br><span class="line">print(x)</span><br><span class="line"><span class="comment"># [1, 2, 3]</span></span><br></pre></td></tr></table></figure><p>匿名函数 常常应用于函数式编程的高阶函数 (high-order function)中，主要有两种形式：</p></li><li><p>参数是函数 (filter, map)</p></li><li><p>返回值是函数 (closure)<br>如，在 <code>filter</code> 和 <code>map</code> 函数中的应用：</p></li><li><p><code>filter(function, iterable)</code> 过滤序列，过滤掉不符合条件的元素，返回一个迭代器对象，如果要转换为列表，可以使用 <code>list()</code> 来转换。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">odd = <span class="keyword">lambda</span> x: x % <span class="number">2</span> == <span class="number">1</span></span><br><span class="line">templist = filter(odd, [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>, <span class="number">6</span>, <span class="number">7</span>, <span class="number">8</span>, <span class="number">9</span>])</span><br><span class="line">print(list(templist)) <span class="comment"># [1, 3, 5, 7, 9]</span></span><br></pre></td></tr></table></figure></li><li><p><code>map(function, *iterables)</code> 根据提供的函数对指定序列做映射。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">m1 = map(<span class="keyword">lambda</span> x: x ** <span class="number">2</span>, [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>])</span><br><span class="line">print(list(m1)) </span><br><span class="line"><span class="comment"># [1, 4, 9, 16, 25]</span></span><br><span class="line"></span><br><span class="line">m2 = map(<span class="keyword">lambda</span> x, y: x + y, [<span class="number">1</span>, <span class="number">3</span>, <span class="number">5</span>, <span class="number">7</span>, <span class="number">9</span>], [<span class="number">2</span>, <span class="number">4</span>, <span class="number">6</span>, <span class="number">8</span>, <span class="number">10</span>])</span><br><span class="line">print(list(m2)) </span><br><span class="line"><span class="comment"># [3, 7, 11, 15, 19]</span></span><br></pre></td></tr></table></figure><h1 id="类与对象"><a href="#类与对象" class="headerlink" title="类与对象"></a>类与对象</h1><h2 id="对象-属性-方法"><a href="#对象-属性-方法" class="headerlink" title="对象 = 属性 + 方法"></a>对象 = 属性 + 方法</h2><p>对象是类的实例。换句话说，类主要定义对象的结构，然后我们以类为模板创建对象。类不但包含方法定义，而且还包含所有实例共享的数据。</p></li><li><p>封装：信息隐蔽技术</p></li></ul><p>我们可以使用关键字 <code>class</code> 定义 Python 类，关键字后面紧跟类的名称、分号和类的实现。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Turtle</span>:</span> <span class="comment"># Python中的类名约定以大写字母开头</span></span><br><span class="line"> <span class="string">"""关于类的一个简单例子"""</span></span><br><span class="line"> <span class="comment"># 属性</span></span><br><span class="line"> color = <span class="string">'green'</span></span><br><span class="line"> weight = <span class="number">10</span></span><br><span class="line"> legs = <span class="number">4</span></span><br><span class="line"> shell = <span class="literal">True</span></span><br><span class="line"> mouth = <span class="string">'大嘴'</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># 方法</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">climb</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(<span class="string">'我正在很努力的向前爬...'</span>)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">run</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(<span class="string">'我正在飞快的向前跑...'</span>)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">bite</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(<span class="string">'咬死你咬死你!!'</span>)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">eat</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(<span class="string">'有得吃，真满足...'</span>)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">sleep</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(<span class="string">'困了，睡了，晚安，zzz'</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">tt = Turtle()</span><br><span class="line">print(tt)</span><br><span class="line"><span class="comment"># <__main__.Turtle object at 0x0000007C32D67F98></span></span><br><span class="line"></span><br><span class="line">print(type(tt))</span><br><span class="line"><span class="comment"># <class '__main__.Turtle'></span></span><br><span class="line"></span><br><span class="line">print(tt.__class__)</span><br><span class="line"><span class="comment"># <class '__main__.Turtle'></span></span><br><span class="line"></span><br><span class="line">print(tt.__class__.__name__)</span><br><span class="line"><span class="comment"># Turtle</span></span><br><span class="line"></span><br><span class="line">tt.climb()</span><br><span class="line"><span class="comment"># 我正在很努力的向前爬...</span></span><br><span class="line"></span><br><span class="line">tt.run()</span><br><span class="line"><span class="comment"># 我正在飞快的向前跑...</span></span><br><span class="line"></span><br><span class="line">tt.bite()</span><br><span class="line"><span class="comment"># 咬死你咬死你!!</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Python类也是对象。它们是type的实例</span></span><br><span class="line">print(type(Turtle))</span><br><span class="line"><span class="comment"># <class 'type'></span></span><br></pre></td></tr></table></figure><ul><li>继承：子类自动共享父类之间数据和方法的机制<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">MyList</span><span class="params">(list)</span>:</span></span><br><span class="line"> <span class="keyword">pass</span></span><br><span class="line"></span><br><span class="line">lst = MyList([<span class="number">1</span>, <span class="number">5</span>, <span class="number">2</span>, <span class="number">7</span>, <span class="number">8</span>])</span><br><span class="line">lst.append(<span class="number">9</span>)</span><br><span class="line">lst.sort()</span><br><span class="line">print(lst)</span><br><span class="line"><span class="comment"># [1, 2, 5, 7, 8, 9]</span></span><br></pre></td></tr></table></figure></li><li>多态：不同对象对同一方法响应不同的行动<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Animal</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">run</span><span class="params">(self)</span>:</span></span><br><span class="line"> <span class="keyword">raise</span> AttributeError(<span class="string">'子类必须实现这个方法'</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">People</span><span class="params">(Animal)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">run</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(<span class="string">'人正在走'</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Pig</span><span class="params">(Animal)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">run</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(<span class="string">'pig is walking'</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Dog</span><span class="params">(Animal)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">run</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(<span class="string">'dog is running'</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">func</span><span class="params">(animal)</span>:</span></span><br><span class="line"> animal.run()</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">func(Pig())</span><br><span class="line"><span class="comment"># pig is walking</span></span><br></pre></td></tr></table></figure><h2 id="self"><a href="#self" class="headerlink" title="self"></a>self</h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Test</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">prt</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(self)</span><br><span class="line"> print(self.__class__)</span><br><span class="line"></span><br><span class="line">t = Test()</span><br><span class="line">t.prt()</span><br><span class="line"><span class="comment"># <__main__.Test object at 0x000000BC5A351208></span></span><br><span class="line"><span class="comment"># <class '__main__.Test'></span></span><br></pre></td></tr></table></figure>类的方法与普通的函数只有一个特别的区别 —— 它们必须有一个额外的第一个参数名称（对应于该实例，即该对象本身），按照惯例它的名称是 <code>self</code>。在调用方法时，我们无需明确提供与参数 <code>self</code> 相对应的参数。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Ball</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">setName</span><span class="params">(self, name)</span>:</span></span><br><span class="line"> self.name = name</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">kick</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(<span class="string">"我叫%s,该死的，谁踢我..."</span> % self.name)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">a = Ball()</span><br><span class="line">a.setName(<span class="string">"球A"</span>)</span><br><span class="line">b = Ball()</span><br><span class="line">b.setName(<span class="string">"球B"</span>)</span><br><span class="line">c = Ball()</span><br><span class="line">c.setName(<span class="string">"球C"</span>)</span><br><span class="line">a.kick()</span><br><span class="line"><span class="comment"># 我叫球A,该死的，谁踢我...</span></span><br><span class="line">b.kick()</span><br><span class="line"><span class="comment"># 我叫球B,该死的，谁踢我...</span></span><br></pre></td></tr></table></figure><h2 id="魔术方法"><a href="#魔术方法" class="headerlink" title="魔术方法"></a>魔术方法</h2>类有一个名为 <code>__init__(self[, param1, param2...])</code> 的魔法方法，该方法在类实例化时会自动调用。<h2 id="公有和私有"><a href="#公有和私有" class="headerlink" title="公有和私有"></a>公有和私有</h2>在 Python 中定义私有变量只需要在变量名或函数名前加上 <code>__</code> 两个下划线，那么这个函数或变量就会为私有的了。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">JustCounter</span>:</span></span><br><span class="line"> __secretCount = <span class="number">0</span> <span class="comment"># 私有变量</span></span><br><span class="line"> publicCount = <span class="number">0</span> <span class="comment"># 公开变量</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">count</span><span class="params">(self)</span>:</span></span><br><span class="line"> self.__secretCount += <span class="number">1</span></span><br><span class="line"> self.publicCount += <span class="number">1</span></span><br><span class="line"> print(self.__secretCount)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">counter = JustCounter()</span><br><span class="line">counter.count() <span class="comment"># 1</span></span><br><span class="line">counter.count() <span class="comment"># 2</span></span><br><span class="line">print(counter.publicCount) <span class="comment"># 2</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Python的私有为伪私有</span></span><br><span class="line">print(counter._JustCounter__secretCount) <span class="comment"># 2 </span></span><br><span class="line">print(counter.__secretCount) </span><br><span class="line"><span class="comment"># AttributeError: 'JustCounter' object has no attribute '__secretCount'</span></span><br></pre></td></tr></table></figure>类的私有方法实例<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Site</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, name, url)</span>:</span></span><br><span class="line"> self.name = name <span class="comment"># public</span></span><br><span class="line"> self.__url = url <span class="comment"># private</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">who</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(<span class="string">'name : '</span>, self.name)</span><br><span class="line"> print(<span class="string">'url : '</span>, self.__url)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__foo</span><span class="params">(self)</span>:</span> <span class="comment"># 私有方法</span></span><br><span class="line"> print(<span class="string">'这是私有方法'</span>)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">foo</span><span class="params">(self)</span>:</span> <span class="comment"># 公共方法</span></span><br><span class="line"> print(<span class="string">'这是公共方法'</span>)</span><br><span class="line"> self.__foo()</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">x = Site(<span class="string">'老马的程序人生'</span>, <span class="string">'https://blog.csdn.net/LSGO_MYP'</span>)</span><br><span class="line">x.who()</span><br><span class="line"><span class="comment"># name : 老马的程序人生</span></span><br><span class="line"><span class="comment"># url : https://blog.csdn.net/LSGO_MYP</span></span><br><span class="line"></span><br><span class="line">x.foo()</span><br><span class="line"><span class="comment"># 这是公共方法</span></span><br><span class="line"><span class="comment"># 这是私有方法</span></span><br><span class="line"></span><br><span class="line">x.__foo()</span><br><span class="line"><span class="comment"># AttributeError: 'Site' object has no attribute '__foo'</span></span><br></pre></td></tr></table></figure><h2 id="继承"><a href="#继承" class="headerlink" title="继承"></a>继承</h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">DerivedClassName</span><span class="params">(BaseClassName)</span>:</span></span><br><span class="line"> statement<span class="number">-1</span></span><br><span class="line"> .</span><br><span class="line"> .</span><br><span class="line"> .</span><br><span class="line"> statement-N</span><br></pre></td></tr></table></figure>BaseClassName（基类名）必须与派生类定义在一个作用域内。除了类，还可以用表达式，基类定义在另一个模块中时这一点非常有用。如果子类中定义与父类同名的方法或属性，则会自动覆盖父类对应的方法或属性。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 类定义</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">people</span>:</span></span><br><span class="line"> <span class="comment"># 定义基本属性</span></span><br><span class="line"> name = <span class="string">''</span></span><br><span class="line"> age = <span class="number">0</span></span><br><span class="line"> <span class="comment"># 定义私有属性,私有属性在类外部无法直接进行访问</span></span><br><span class="line"> __weight = <span class="number">0</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># 定义构造方法</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, n, a, w)</span>:</span></span><br><span class="line"> self.name = n</span><br><span class="line"> self.age = a</span><br><span class="line"> self.__weight = w</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">speak</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(<span class="string">"%s 说: 我 %d 岁。"</span> % (self.name, self.age))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 单继承示例</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">student</span><span class="params">(people)</span>:</span></span><br><span class="line"> grade = <span class="string">''</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, n, a, w, g)</span>:</span></span><br><span class="line"> <span class="comment"># 调用父类的构函</span></span><br><span class="line"> people.__init__(self, n, a, w)</span><br><span class="line"> self.grade = g</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 覆写父类的方法</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">speak</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(<span class="string">"%s 说: 我 %d 岁了，我在读 %d 年级"</span> % (self.name, self.age, self.grade))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">s = student(<span class="string">'小马的程序人生'</span>, <span class="number">10</span>, <span class="number">60</span>, <span class="number">3</span>)</span><br><span class="line">s.speak()</span><br><span class="line"><span class="comment"># 小马的程序人生 说: 我 10 岁了，我在读 3 年级</span></span><br></pre></td></tr></table></figure>注意：如果上面的程序去掉：<code>people.__init__(self, n, a, w)</code>，则输出：说: <code>我 0 岁了，我在读 3 年级</code>，因为子类的构造方法继承了父类的变量。解决该问题可用以下两种方式：</li><li>调用未绑定的父类方法 <code>Fish.__init__(self)</code><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Shark</span><span class="params">(Fish)</span>:</span> <span class="comment"># 鲨鱼</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self)</span>:</span></span><br><span class="line"> Fish.__init__(self)</span><br><span class="line"> self.hungry = <span class="literal">True</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">eat</span><span class="params">(self)</span>:</span></span><br><span class="line"> <span class="keyword">if</span> self.hungry:</span><br><span class="line"> print(<span class="string">"吃货的梦想就是天天有得吃！"</span>)</span><br><span class="line"> self.hungry = <span class="literal">False</span></span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> print(<span class="string">"太撑了，吃不下了！"</span>)</span><br><span class="line"> self.hungry = <span class="literal">True</span></span><br></pre></td></tr></table></figure></li><li>使用 <code>super</code> 函数 <code>super().__init__()</code><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Shark</span><span class="params">(Fish)</span>:</span> <span class="comment"># 鲨鱼</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self)</span>:</span></span><br><span class="line"> super().__init__()</span><br><span class="line"> self.hungry = <span class="literal">True</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">eat</span><span class="params">(self)</span>:</span></span><br><span class="line"> <span class="keyword">if</span> self.hungry:</span><br><span class="line"> print(<span class="string">"吃货的梦想就是天天有得吃！"</span>)</span><br><span class="line"> self.hungry = <span class="literal">False</span></span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> print(<span class="string">"太撑了，吃不下了！"</span>)</span><br><span class="line"> self.hungry = <span class="literal">True</span></span><br></pre></td></tr></table></figure><h2 id="类、类对象和实例对象"><a href="#类、类对象和实例对象" class="headerlink" title="类、类对象和实例对象"></a>类、类对象和实例对象</h2></li><li>类对象：创建一个类，其实也是一个对象也在内存开辟了一块空间，称为类对象，类对象只有一个。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">A</span><span class="params">(object)</span>:</span></span><br><span class="line"> <span class="keyword">pass</span></span><br></pre></td></tr></table></figure></li><li>实例对象：就是通过实例化类创建的对象，称为实例对象，实例对象可以有多个。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">A</span><span class="params">(object)</span>:</span></span><br><span class="line"> <span class="keyword">pass</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 实例化对象 a、b、c都属于实例对象。</span></span><br><span class="line">a = A()</span><br><span class="line">b = A()</span><br><span class="line">c = A()</span><br></pre></td></tr></table></figure></li><li>类属性：类里面方法外面定义的变量称为类属性。类属性所属于类对象并且多个实例对象之间共享同一个类属性，说白了就是类属性所有的通过该类实例化的对象都能共享。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">A</span><span class="params">()</span>:</span></span><br><span class="line"> a = <span class="number">0</span> <span class="comment">#类属性</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, xx)</span>:</span></span><br><span class="line"> A.a = xx <span class="comment">#使用类属性可以通过 （类名.类属性）调用。</span></span><br></pre></td></tr></table></figure></li><li>实例属性：实例属性和具体的某个实例对象有关系，并且一个实例对象和另外一个实例对象是不共享属性的，说白了实例属性只能在自己的对象里面使用，其他的对象不能直接使用，因为 <code>self</code> 是谁调用，它的值就属于该对象。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 创建类对象</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Test</span><span class="params">(object)</span>:</span></span><br><span class="line"> class_attr = <span class="number">100</span> <span class="comment"># 类属性</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self)</span>:</span></span><br><span class="line"> self.sl_attr = <span class="number">100</span> <span class="comment"># 实例属性</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">func</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(<span class="string">'类对象.类属性的值:'</span>, Test.class_attr) <span class="comment"># 调用类属性</span></span><br><span class="line"> print(<span class="string">'self.类属性的值'</span>, self.class_attr) <span class="comment"># 相当于把类属性 变成实例属性</span></span><br><span class="line"> print(<span class="string">'self.实例属性的值'</span>, self.sl_attr) <span class="comment"># 调用实例属性</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">a = Test()</span><br><span class="line">a.func()</span><br><span class="line"></span><br><span class="line"><span class="comment"># 类对象.类属性的值: 100</span></span><br><span class="line"><span class="comment"># self.类属性的值 100</span></span><br><span class="line"><span class="comment"># self.实例属性的值 100</span></span><br><span class="line"></span><br><span class="line">b = Test()</span><br><span class="line">b.func()</span><br><span class="line"></span><br><span class="line"><span class="comment"># 类对象.类属性的值: 100</span></span><br><span class="line"><span class="comment"># self.类属性的值 100</span></span><br><span class="line"><span class="comment"># self.实例属性的值 100</span></span><br><span class="line"></span><br><span class="line">a.class_attr = <span class="number">200</span></span><br><span class="line">a.sl_attr = <span class="number">200</span></span><br><span class="line">a.func()</span><br><span class="line"></span><br><span class="line"><span class="comment"># 类对象.类属性的值: 100</span></span><br><span class="line"><span class="comment"># self.类属性的值 200</span></span><br><span class="line"><span class="comment"># self.实例属性的值 200</span></span><br><span class="line"></span><br><span class="line">b.func()</span><br><span class="line"></span><br><span class="line"><span class="comment"># 类对象.类属性的值: 100</span></span><br><span class="line"><span class="comment"># self.类属性的值 100</span></span><br><span class="line"><span class="comment"># self.实例属性的值 100</span></span><br><span class="line"></span><br><span class="line">Test.class_attr = <span class="number">300</span></span><br><span class="line">a.func()</span><br><span class="line"></span><br><span class="line"><span class="comment"># 类对象.类属性的值: 300</span></span><br><span class="line"><span class="comment"># self.类属性的值 200</span></span><br><span class="line"><span class="comment"># self.实例属性的值 200</span></span><br><span class="line"></span><br><span class="line">b.func()</span><br><span class="line"><span class="comment"># 类对象.类属性的值: 300</span></span><br><span class="line"><span class="comment"># self.类属性的值 300</span></span><br><span class="line"><span class="comment"># self.实例属性的值 100</span></span><br></pre></td></tr></table></figure><h2 id="绑定"><a href="#绑定" class="headerlink" title="绑定"></a>绑定</h2>Python 严格要求方法需要有实例才能被调用，这种限制其实就是 Python 所谓的绑定概念。</li></ul><p>Python 对象的数据属性通常存储在名为 <code>.__ dict__</code> 的字典中，我们可以直接访问 <code>__dict__</code>，或利用 Python 的内置函数 <code>vars()</code> 获取 <code>.__ dict__</code>。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">CC</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">setXY</span><span class="params">(self, x, y)</span>:</span></span><br><span class="line"> self.x = x</span><br><span class="line"> self.y = y</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">printXY</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(self.x, self.y)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">dd = CC()</span><br><span class="line">print(dd.__dict__)</span><br><span class="line"><span class="comment"># {}</span></span><br><span class="line"></span><br><span class="line">print(vars(dd))</span><br><span class="line"><span class="comment"># {}</span></span><br><span class="line"></span><br><span class="line">print(CC.__dict__)</span><br><span class="line"><span class="comment"># {'__module__': '__main__', 'setXY': <function CC.setXY at 0x000000C3473DA048>, 'printXY': <function CC.printXY at 0x000000C3473C4F28>, '__dict__': <attribute '__dict__' of 'CC' objects>, '__weakref__': <attribute '__weakref__' of 'CC' objects>, '__doc__': None}</span></span><br><span class="line"></span><br><span class="line">dd.setXY(<span class="number">4</span>, <span class="number">5</span>)</span><br><span class="line">print(dd.__dict__)</span><br><span class="line"><span class="comment"># {'x': 4, 'y': 5}</span></span><br><span class="line"></span><br><span class="line">print(vars(CC))</span><br><span class="line"><span class="comment"># {'__module__': '__main__', 'setXY': <function CC.setXY at 0x000000632CA9B048>, 'printXY': <function CC.printXY at 0x000000632CA83048>, '__dict__': <attribute '__dict__' of 'CC' objects>, '__weakref__': <attribute '__weakref__' of 'CC' objects>, '__doc__': None}</span></span><br><span class="line"></span><br><span class="line">print(CC.__dict__)</span><br><span class="line"><span class="comment"># {'__module__': '__main__', 'setXY': <function CC.setXY at 0x000000632CA9B048>, 'printXY': <function CC.printXY at 0x000000632CA83048>, '__dict__': <attribute '__dict__' of 'CC' objects>, '__weakref__': <attribute '__weakref__' of 'CC' objects>, '__doc__': None}</span></span><br></pre></td></tr></table></figure><h2 id="一些相关的内置函数（BIF）"><a href="#一些相关的内置函数（BIF）" class="headerlink" title="一些相关的内置函数（BIF）"></a>一些相关的内置函数（BIF）</h2><ul><li><code>issubclass(class, classinfo)</code> 方法用于判断参数 class 是否是类型参数 classinfo 的子类。</li><li>一个类被认为是其自身的子类。</li><li><code>classinfo</code> 可以是类对象的元组，只要 class 是其中任何一个候选类的子类，则返回 True。</li><li>isinstance(object, classinfo) 方法用于判断一个对象是否是一个已知的类型，类似type()。</li><li>type()不会认为子类是一种父类类型，不考虑继承关系。</li><li>isinstance()会认为子类是一种父类类型，考虑继承关系。</li><li>如果第一个参数不是对象，则永远返回False。</li><li>如果第二个参数不是类或者由类对象组成的元组，会抛出一个TypeError异常。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">a = <span class="number">2</span></span><br><span class="line">print(isinstance(a, int)) <span class="comment"># True</span></span><br><span class="line">print(isinstance(a, str)) <span class="comment"># False</span></span><br><span class="line">print(isinstance(a, (str, int, list))) <span class="comment"># True</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">A</span>:</span></span><br><span class="line"> <span class="keyword">pass</span></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">B</span><span class="params">(A)</span>:</span></span><br><span class="line"> <span class="keyword">pass</span></span><br><span class="line"></span><br><span class="line">print(isinstance(A(), A)) <span class="comment"># True</span></span><br><span class="line">print(type(A()) == A) <span class="comment"># True</span></span><br><span class="line">print(isinstance(B(), A)) <span class="comment"># True</span></span><br><span class="line">print(type(B()) == A) <span class="comment"># False</span></span><br></pre></td></tr></table></figure></li><li><code>hasattr(object, name)</code> 用于判断对象是否包含对应的属性。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Coordinate</span>:</span></span><br><span class="line"> x = <span class="number">10</span></span><br><span class="line"> y = <span class="number">-5</span></span><br><span class="line"> z = <span class="number">0</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">point1 = Coordinate()</span><br><span class="line">print(hasattr(point1, <span class="string">'x'</span>)) <span class="comment"># True</span></span><br><span class="line">print(hasattr(point1, <span class="string">'y'</span>)) <span class="comment"># True</span></span><br><span class="line">print(hasattr(point1, <span class="string">'z'</span>)) <span class="comment"># True</span></span><br><span class="line">print(hasattr(point1, <span class="string">'no'</span>)) <span class="comment"># False</span></span><br></pre></td></tr></table></figure></li><li><code>getattr(object, name[, default])</code> 用于返回一个对象属性值。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">A</span><span class="params">(object)</span>:</span></span><br><span class="line"> bar = <span class="number">1</span></span><br><span class="line"></span><br><span class="line">a = A()</span><br><span class="line">print(getattr(a, <span class="string">'bar'</span>)) <span class="comment"># 1</span></span><br><span class="line">print(getattr(a, <span class="string">'bar2'</span>, <span class="number">3</span>)) <span class="comment"># 3</span></span><br><span class="line">print(getattr(a, <span class="string">'bar2'</span>))</span><br><span class="line"><span class="comment"># AttributeError: 'A' object has no attribute 'bar2'</span></span><br></pre></td></tr></table></figure></li><li><code>setattr(object, name, value)</code> 对应函数 <code>getattr()</code>，用于设置属性值，该属性不一定是存在的。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">A</span><span class="params">(object)</span>:</span></span><br><span class="line"> bar = <span class="number">1</span></span><br><span class="line"></span><br><span class="line">a = A()</span><br><span class="line">print(getattr(a, <span class="string">'bar'</span>)) <span class="comment"># 1</span></span><br><span class="line">setattr(a, <span class="string">'bar'</span>, <span class="number">5</span>)</span><br><span class="line">print(a.bar) <span class="comment"># 5</span></span><br><span class="line">setattr(a, <span class="string">"age"</span>, <span class="number">28</span>)</span><br><span class="line">print(a.age) <span class="comment"># 28</span></span><br></pre></td></tr></table></figure></li><li><code>delattr(object, name)</code> 用于删除属性。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Coordinate</span>:</span></span><br><span class="line"> x = <span class="number">10</span></span><br><span class="line"> y = <span class="number">-5</span></span><br><span class="line"> z = <span class="number">0</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">point1 = Coordinate()</span><br><span class="line"></span><br><span class="line">print(<span class="string">'x = '</span>, point1.x) <span class="comment"># x = 10</span></span><br><span class="line">print(<span class="string">'y = '</span>, point1.y) <span class="comment"># y = -5</span></span><br><span class="line">print(<span class="string">'z = '</span>, point1.z) <span class="comment"># z = 0</span></span><br><span class="line"></span><br><span class="line">delattr(Coordinate, <span class="string">'z'</span>)</span><br><span class="line"></span><br><span class="line">print(<span class="string">'--删除 z 属性后--'</span>) <span class="comment"># --删除 z 属性后--</span></span><br><span class="line">print(<span class="string">'x = '</span>, point1.x) <span class="comment"># x = 10</span></span><br><span class="line">print(<span class="string">'y = '</span>, point1.y) <span class="comment"># y = -5</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 触发错误</span></span><br><span class="line">print(<span class="string">'z = '</span>, point1.z)</span><br><span class="line"><span class="comment"># AttributeError: 'Coordinate' object has no attribute 'z'</span></span><br></pre></td></tr></table></figure><h1 id="魔术方法-1"><a href="#魔术方法-1" class="headerlink" title="魔术方法"></a>魔术方法</h1>魔法方法的第一个参数应为 <code>cls</code>（类方法） 或者 <code>self</code>（实例方法）。</li><li><code>cls</code>：代表一个类的名称</li><li><code>self</code>：代表一个实例对象的名称<h2 id="基本的魔法方法"><a href="#基本的魔法方法" class="headerlink" title="基本的魔法方法"></a>基本的魔法方法</h2></li><li><code>__init__(self[, ...])</code> 构造器，当一个实例被创建的时候调用的初始化方法。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Rectangle</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, x, y)</span>:</span></span><br><span class="line"> self.x = x</span><br><span class="line"> self.y = y</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">getPeri</span><span class="params">(self)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> (self.x + self.y) * <span class="number">2</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">getArea</span><span class="params">(self)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> self.x * self.y</span><br><span class="line"></span><br><span class="line">rect = Rectangle(<span class="number">4</span>, <span class="number">5</span>)</span><br><span class="line">print(rect.getPeri()) <span class="comment"># 18</span></span><br><span class="line">print(rect.getArea()) <span class="comment"># 20</span></span><br></pre></td></tr></table></figure></li><li><code>__str__(self)</code>:<ul><li>当你打印一个对象的时候，触发 <code>__str__</code></li><li>当你使用 <code>%s</code> 格式化的时候，触发 <code>__str__</code></li><li><code>str</code> 强转数据类型的时候，触发 <code>__str__</code></li></ul></li><li><code>__repr__(self)</code>：<ul><li><code>repr</code> 是 <code>str</code> 的备胎</li><li>有 <code>__str__</code> 的时候执行 <code>__str__</code>，没有实现 <code>__str__</code> 的时候，执行 <code>__repr__</code></li><li><code>repr(obj)</code> 内置函数对应的结果是 <code>__repr__</code> 的返回值</li><li>当你使用 <code>%r</code> 格式化的时候 触发 <code>__repr__</code><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Cat</span>:</span></span><br><span class="line"> <span class="string">"""定义一个猫类"""</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, new_name, new_age)</span>:</span></span><br><span class="line"> <span class="string">"""在创建完对象之后 会自动调用, 它完成对象的初始化的功能"""</span></span><br><span class="line"> self.name = new_name</span><br><span class="line"> self.age = new_age</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__str__</span><span class="params">(self)</span>:</span></span><br><span class="line"> <span class="string">"""返回一个对象的描述信息"""</span></span><br><span class="line"> <span class="keyword">return</span> <span class="string">"名字是:%s , 年龄是:%d"</span> % (self.name, self.age)</span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__repr__</span><span class="params">(self)</span>:</span></span><br><span class="line"> <span class="string">"""返回一个对象的描述信息"""</span></span><br><span class="line"> <span class="keyword">return</span> <span class="string">"Cat:(%s,%d)"</span> % (self.name, self.age)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">eat</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(<span class="string">"%s在吃鱼...."</span> % self.name)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">drink</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(<span class="string">"%s在喝可乐..."</span> % self.name)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">introduce</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(<span class="string">"名字是:%s, 年龄是:%d"</span> % (self.name, self.age))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 创建了一个对象</span></span><br><span class="line">tom = Cat(<span class="string">"汤姆"</span>, <span class="number">30</span>)</span><br><span class="line">print(tom) <span class="comment"># 名字是:汤姆 , 年龄是:30</span></span><br><span class="line">print(str(tom)) <span class="comment"># 名字是:汤姆 , 年龄是:30</span></span><br><span class="line">print(repr(tom)) <span class="comment"># Cat:(汤姆,30)</span></span><br><span class="line">tom.eat() <span class="comment"># 汤姆在吃鱼....</span></span><br><span class="line">tom.introduce() <span class="comment"># 名字是:汤姆, 年龄是:30</span></span><br></pre></td></tr></table></figure><h2 id="算术运算符"><a href="#算术运算符" class="headerlink" title="算术运算符"></a>算术运算符</h2></li></ul></li><li><code>__add__(self, other)</code> 定义加法的行为：<code>+</code></li><li><code>__sub__(self, other)</code> 定义减法的行为：<code>-</code><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">MyClass</span>:</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, height, weight)</span>:</span></span><br><span class="line"> self.height = height</span><br><span class="line"> self.weight = weight</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 两个对象的长相加，宽不变.返回一个新的类</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__add__</span><span class="params">(self, others)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> MyClass(self.height + others.height, self.weight + others.weight)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 两个对象的宽相减，长不变.返回一个新的类</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__sub__</span><span class="params">(self, others)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> MyClass(self.height - others.height, self.weight - others.weight)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 说一下自己的参数</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">intro</span><span class="params">(self)</span>:</span></span><br><span class="line"> print(<span class="string">"高为"</span>, self.height, <span class="string">" 重为"</span>, self.weight)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">()</span>:</span></span><br><span class="line"> a = MyClass(height=<span class="number">10</span>, weight=<span class="number">5</span>)</span><br><span class="line"> a.intro()</span><br><span class="line"></span><br><span class="line"> b = MyClass(height=<span class="number">20</span>, weight=<span class="number">10</span>)</span><br><span class="line"> b.intro()</span><br><span class="line"></span><br><span class="line"> c = b - a</span><br><span class="line"> c.intro()</span><br><span class="line"></span><br><span class="line"> d = a + b</span><br><span class="line"> d.intro()</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> main()</span><br><span class="line"></span><br><span class="line"><span class="comment"># 高为 10 重为 5</span></span><br><span class="line"><span class="comment"># 高为 20 重为 10</span></span><br><span class="line"><span class="comment"># 高为 10 重为 5</span></span><br><span class="line"><span class="comment"># 高为 30 重为 15</span></span><br></pre></td></tr></table></figure></li><li><code>__mul__(self, other)</code> 定义乘法的行为：<code>*</code></li><li><code>__truediv__(self, other)</code> 定义真除法的行为：<code>/</code></li><li><code>__floordiv__(self, other)</code> 定义整数除法的行为：<code>//</code></li><li><code>__mod__(self, other)</code> 定义取模算法的行为：<code>%</code></li><li><code>__divmod__(self, other)</code> 定义当被 <code>divmod()</code> 调用时的行为</li><li><code>divmod(a, b)</code> 把除数和余数运算结果结合起来，返回一个包含商和余数的元组 <code>(a // b, a % b)</code>。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">print(divmod(<span class="number">7</span>, <span class="number">2</span>)) <span class="comment"># (3, 1)</span></span><br><span class="line">print(divmod(<span class="number">8</span>, <span class="number">2</span>)) <span class="comment"># (4, 0)</span></span><br></pre></td></tr></table></figure></li><li><code>__pow__(self, other[, module])</code> 定义当被 <code>power()</code> 调用或 <code>**</code> 运算时的行为</li><li><code>__lshift__(self, other)</code> 定义按位左移位的行为：<code><<</code></li><li><code>__rshift__(self, other)</code> 定义按位右移位的行为：<code>>></code></li><li><code>__and__(self, other)</code> 定义按位与操作的行为：<code>&</code></li><li><code>__xor__(self, other)</code> 定义按位异或操作的行为：<code>^</code></li><li><code>__or__(self, other)</code> 定义按位或操作的行为：<code>|</code><h2 id="增量赋值运算符"><a href="#增量赋值运算符" class="headerlink" title="增量赋值运算符"></a>增量赋值运算符</h2></li><li><code>__iadd__(self, other)</code> 定义赋值加法的行为：<code>+=</code></li><li><code>__isub__(self, other)</code> 定义赋值减法的行为：<code>-=</code></li><li><code>__imul__(self, other)</code> 定义赋值乘法的行为：<code>*=</code></li><li><code>__itruediv__(self, other)</code> 定义赋值真除法的行为：<code>/=</code></li><li><code>__ifloordiv__(self, other)</code> 定义赋值整数除法的行为：<code>//=</code></li><li><code>__imod__(self, other)</code> 定义赋值取模算法的行为：<code>%=</code></li><li><code>__ipow__(self, other[, modulo])</code> 定义赋值幂运算的行为：<code>**=</code></li><li><code>__ilshift__(self, other)</code> 定义赋值按位左移位的行为：<code><<=</code></li><li><code>__irshift__(self, other)</code> 定义赋值按位右移位的行为：<code>>>=</code></li><li><code>__iand__(self, other)</code> 定义赋值按位与操作的行为：<code>&=</code></li><li><code>__ixor__(self, other)</code> 定义赋值按位异或操作的行为：<code>^=</code></li><li><code>__ior__(self, other)</code> 定义赋值按位或操作的行为：<code>|=</code><h2 id="一元运算符"><a href="#一元运算符" class="headerlink" title="一元运算符"></a>一元运算符</h2></li><li><code>__neg__(self)</code> 定义正号的行为：<code>+x</code></li><li><code>__pos__(self)</code> 定义负号的行为：<code>-x</code></li><li><code>__abs__(self)</code> 定义当被 <code>abs()</code> 调用时的行为<h2 id="属性访问"><a href="#属性访问" class="headerlink" title="属性访问"></a>属性访问</h2></li><li><code>__getattr__(self, name)</code>: 定义当用户试图获取一个不存在的属性时的行为。</li><li><code>__getattribute__(self, name)</code>：定义当该类的属性被访问时的行为（先调用该方法，查看是否存在该属性，若不存在，接着去调用 <code>__getattr__</code>）。</li><li><code>__setattr__(self, name, value)</code>：定义当一个属性被设置时的行为。</li><li><code>__delattr__(self, name)</code>：定义当一个属性被删除时的行为。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">C</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__getattribute__</span><span class="params">(self, item)</span>:</span></span><br><span class="line"> print(<span class="string">'__getattribute__'</span>)</span><br><span class="line"> <span class="keyword">return</span> super().__getattribute__(item)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__getattr__</span><span class="params">(self, item)</span>:</span></span><br><span class="line"> print(<span class="string">'__getattr__'</span>)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__setattr__</span><span class="params">(self, key, value)</span>:</span></span><br><span class="line"> print(<span class="string">'__setattr__'</span>)</span><br><span class="line"> super().__setattr__(key, value)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__delattr__</span><span class="params">(self, item)</span>:</span></span><br><span class="line"> print(<span class="string">'__delattr__'</span>)</span><br><span class="line"> super().__delattr__(item)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">c = C()</span><br><span class="line">c.x</span><br><span class="line"><span class="comment"># __getattribute__</span></span><br><span class="line"><span class="comment"># __getattr__</span></span><br><span class="line"></span><br><span class="line">c.x = <span class="number">1</span></span><br><span class="line"><span class="comment"># __setattr__</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">del</span> c.x</span><br><span class="line"><span class="comment"># __delattr__</span></span><br></pre></td></tr></table></figure><h2 id="定制序列"><a href="#定制序列" class="headerlink" title="定制序列"></a>定制序列</h2></li><li>如果说你希望定制的容器是不可变的话，你只需要定义 <code>__len__()</code> 和 <code>__getitem__()</code> 方法。</li><li>如果你希望定制的容器是可变的话，除了 <code>__len__()</code> 和 <code>__getitem__()</code> 方法，你还需要定义 <code>__setitem__()</code> 和 <code>__delitem__()</code> 两个方法。</li></ul><p><strong>编写一个不可改变的自定义列表，要求记录列表中每个元素被访问的次数。</strong></p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">CountList</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, *args)</span>:</span></span><br><span class="line"> self.values = [x <span class="keyword">for</span> x <span class="keyword">in</span> args]</span><br><span class="line"> self.count = {}.fromkeys(range(len(self.values)), <span class="number">0</span>)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__len__</span><span class="params">(self)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> len(self.values)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__getitem__</span><span class="params">(self, item)</span>:</span></span><br><span class="line"> self.count[item] += <span class="number">1</span></span><br><span class="line"> <span class="keyword">return</span> self.values[item]</span><br><span class="line"></span><br><span class="line">c1 = CountList(<span class="number">1</span>, <span class="number">3</span>, <span class="number">5</span>, <span class="number">7</span>, <span class="number">9</span>)</span><br><span class="line">c2 = CountList(<span class="number">2</span>, <span class="number">4</span>, <span class="number">6</span>, <span class="number">8</span>, <span class="number">10</span>)</span><br><span class="line">print(c1[<span class="number">1</span>]) <span class="comment"># 3</span></span><br><span class="line">print(c2[<span class="number">2</span>]) <span class="comment"># 6</span></span><br><span class="line">print(c1[<span class="number">1</span>] + c2[<span class="number">1</span>]) <span class="comment"># 7</span></span><br><span class="line"></span><br><span class="line">print(c1.count)</span><br><span class="line"><span class="comment"># {0: 0, 1: 2, 2: 0, 3: 0, 4: 0}</span></span><br><span class="line"></span><br><span class="line">print(c2.count)</span><br><span class="line"><span class="comment"># {0: 0, 1: 1, 2: 1, 3: 0, 4: 0}</span></span><br></pre></td></tr></table></figure><ul><li><code>__len__(self)</code> 定义当被 <code>len()</code> 调用时的行为（返回容器中元素的个数）。</li><li><code>__getitem__(self, key)</code> 定义获取容器中元素的行为，相当于 <code>self[key]</code>。</li><li><code>__setitem__(self, key, value)</code> 定义设置容器中指定元素的行为，相当于<code>self[key] = value</code>。</li><li><code>__delitem__(self, key)</code> 定义删除容器中指定元素的行为，相当于 <code>del self[key]</code>。</li></ul><p><strong>编写一个可改变的自定义列表，要求记录列表中每个元素被访问的次数。</strong></p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">CountList</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, *args)</span>:</span></span><br><span class="line"> self.values = [x <span class="keyword">for</span> x <span class="keyword">in</span> args]</span><br><span class="line"> self.count = {}.fromkeys(range(len(self.values)), <span class="number">0</span>)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__len__</span><span class="params">(self)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> len(self.values)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__getitem__</span><span class="params">(self, item)</span>:</span></span><br><span class="line"> self.count[item] += <span class="number">1</span></span><br><span class="line"> <span class="keyword">return</span> self.values[item]</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__setitem__</span><span class="params">(self, key, value)</span>:</span></span><br><span class="line"> self.values[key] = value</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__delitem__</span><span class="params">(self, key)</span>:</span></span><br><span class="line"> <span class="keyword">del</span> self.values[key]</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">0</span>, len(self.values)):</span><br><span class="line"> <span class="keyword">if</span> i >= key:</span><br><span class="line"> self.count[i] = self.count[i + <span class="number">1</span>]</span><br><span class="line"> self.count.pop(len(self.values))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">c1 = CountList(<span class="number">1</span>, <span class="number">3</span>, <span class="number">5</span>, <span class="number">7</span>, <span class="number">9</span>)</span><br><span class="line">c2 = CountList(<span class="number">2</span>, <span class="number">4</span>, <span class="number">6</span>, <span class="number">8</span>, <span class="number">10</span>)</span><br><span class="line">print(c1[<span class="number">1</span>]) <span class="comment"># 3</span></span><br><span class="line">print(c2[<span class="number">2</span>]) <span class="comment"># 6</span></span><br><span class="line">c2[<span class="number">2</span>] = <span class="number">12</span></span><br><span class="line">print(c1[<span class="number">1</span>] + c2[<span class="number">2</span>]) <span class="comment"># 15</span></span><br><span class="line">print(c1.count)</span><br><span class="line"><span class="comment"># {0: 0, 1: 2, 2: 0, 3: 0, 4: 0}</span></span><br><span class="line">print(c2.count)</span><br><span class="line"><span class="comment"># {0: 0, 1: 0, 2: 2, 3: 0, 4: 0}</span></span><br><span class="line"><span class="keyword">del</span> c1[<span class="number">1</span>]</span><br><span class="line">print(c1.count)</span><br><span class="line"><span class="comment"># {0: 0, 1: 0, 2: 0, 3: 0}</span></span><br></pre></td></tr></table></figure><h1 id="迭代器"><a href="#迭代器" class="headerlink" title="迭代器"></a>迭代器</h1><ul><li>迭代是 Python 最强大的功能之一，是访问集合元素的一种方式。</li><li>迭代器是一个可以记住遍历的位置的对象。</li><li>迭代器对象从集合的第一个元素开始访问，直到所有的元素被访问完结束。</li><li>迭代器只能往前不会后退。</li><li>字符串，列表或元组对象都可用于创建迭代器：<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">links = {<span class="string">'B'</span>: <span class="string">'百度'</span>, <span class="string">'A'</span>: <span class="string">'阿里'</span>, <span class="string">'T'</span>: <span class="string">'腾讯'</span>}</span><br><span class="line"><span class="keyword">for</span> each <span class="keyword">in</span> iter(links):</span><br><span class="line"> print(<span class="string">'%s -> %s'</span> % (each, links[each]))</span><br><span class="line"><span class="comment"># B -> 百度</span></span><br><span class="line"><span class="comment"># A -> 阿里</span></span><br><span class="line"><span class="comment"># T -> 腾讯</span></span><br><span class="line"><span class="comment"># B -> 百度</span></span><br><span class="line"><span class="comment"># A -> 阿里</span></span><br><span class="line"><span class="comment"># T -> 腾讯</span></span><br></pre></td></tr></table></figure></li><li>迭代器有两个基本的方法：<code>iter()</code> 和 <code>next()</code>。</li><li><code>iter(object)</code> 函数用来生成迭代器。</li><li><code>next(iterator[, default])</code> 返回迭代器的下一个项目。</li><li><code>iterator</code> – 可迭代对象</li><li><code>default</code> – 可选，用于设置在没有下一个元素时返回该默认值，如果不设置，又没有下一个元素则会触发 <code>StopIteration</code> 异常。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">links = {<span class="string">'B'</span>: <span class="string">'百度'</span>, <span class="string">'A'</span>: <span class="string">'阿里'</span>, <span class="string">'T'</span>: <span class="string">'腾讯'</span>}</span><br><span class="line"></span><br><span class="line">it = iter(links)</span><br><span class="line"><span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> each = next(it)</span><br><span class="line"> <span class="keyword">except</span> StopIteration:</span><br><span class="line"> <span class="keyword">break</span></span><br><span class="line"> print(each)</span><br><span class="line"></span><br><span class="line"><span class="comment"># B</span></span><br><span class="line"><span class="comment"># A</span></span><br><span class="line"><span class="comment"># T</span></span><br><span class="line"></span><br><span class="line">it = iter(links)</span><br><span class="line">print(next(it)) <span class="comment"># B</span></span><br><span class="line">print(next(it)) <span class="comment"># A</span></span><br><span class="line">print(next(it)) <span class="comment"># T</span></span><br><span class="line">print(next(it)) <span class="comment"># StopIteration</span></span><br></pre></td></tr></table></figure>把一个类作为一个迭代器使用需要在类中实现两个魔法方法 <code>__iter__()</code> 与 <code>__next__()</code>。</li><li><code>__iter__(self)</code> 定义当迭代容器中的元素的行为，返回一个特殊的迭代器对象， 这个迭代器对象实现了 <code>__next__()</code> 方法并通过 <code>StopIteration</code> 异常标识迭代的完成。</li><li><code>__next__()</code> 返回下一个迭代器对象。</li><li><code>StopIteration</code> 异常用于标识迭代的完成，防止出现无限循环的情况，在 <code>__next__()</code> 方法中我们可以设置在完成指定循环次数后触发 <code>StopIteration</code> 异常来结束迭代。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Fibs</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, n=<span class="number">10</span>)</span>:</span></span><br><span class="line"> self.a = <span class="number">0</span></span><br><span class="line"> self.b = <span class="number">1</span></span><br><span class="line"> self.n = n</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__iter__</span><span class="params">(self)</span>:</span></span><br><span class="line"> <span class="keyword">return</span> self</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__next__</span><span class="params">(self)</span>:</span></span><br><span class="line"> self.a, self.b = self.b, self.a + self.b</span><br><span class="line"> <span class="keyword">if</span> self.a > self.n:</span><br><span class="line"> <span class="keyword">raise</span> StopIteration</span><br><span class="line"> <span class="keyword">return</span> self.a</span><br><span class="line"></span><br><span class="line">fibs = Fibs(<span class="number">100</span>)</span><br><span class="line"><span class="keyword">for</span> each <span class="keyword">in</span> fibs:</span><br><span class="line"> print(each, end=<span class="string">' '</span>)</span><br><span class="line"><span class="comment"># 1 1 2 3 5 8 13 21 34 55 89</span></span><br></pre></td></tr></table></figure><h1 id="生成器"><a href="#生成器" class="headerlink" title="生成器"></a>生成器</h1></li><li>在 Python 中，使用了 <code>yield</code> 的函数被称为生成器（generator）。</li><li>跟普通函数不同的是，生成器是一个返回迭代器的函数，只能用于迭代操作，更简单点理解生成器就是一个迭代器。</li><li>在调用生成器运行的过程中，每次遇到 <code>yield</code> 时函数会暂停并保存当前所有的运行信息，返回 <code>yield</code> 的值, 并在下一次执行 <code>next()</code> 方法时从当前位置继续运行。</li><li>调用一个生成器函数，返回的是一个迭代器对象。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">myGen</span><span class="params">()</span>:</span></span><br><span class="line"> print(<span class="string">'生成器执行！'</span>)</span><br><span class="line"> <span class="keyword">yield</span> <span class="number">1</span></span><br><span class="line"> <span class="keyword">yield</span> <span class="number">2</span></span><br><span class="line"> </span><br><span class="line">myG = myGen()</span><br><span class="line"><span class="keyword">for</span> each <span class="keyword">in</span> myG:</span><br><span class="line"> print(each)</span><br><span class="line"></span><br><span class="line"><span class="string">'''</span></span><br><span class="line"><span class="string">生成器执行！</span></span><br><span class="line"><span class="string">1</span></span><br><span class="line"><span class="string">2</span></span><br><span class="line"><span class="string">'''</span></span><br><span class="line"></span><br><span class="line">myG = myGen()</span><br><span class="line">print(next(myG)) </span><br><span class="line"><span class="comment"># 生成器执行！</span></span><br><span class="line"><span class="comment"># 1</span></span><br><span class="line"></span><br><span class="line">print(next(myG)) <span class="comment"># 2</span></span><br><span class="line">print(next(myG)) <span class="comment"># StopIteration</span></span><br></pre></td></tr></table></figure></li></ul>]]></content>
<summary type="html">
<h1 id="函数与-Lambda-表达式"><a href="#函数与-Lambda-表达式" class="headerlink" title="函数与 Lambda 表达式"></a>函数与 Lambda 表达式</h1><h2 id="函数"><a href="#函数" class="headerlink" title="函数"></a>函数</h2><h3 id="函数的定义"><a href="#函数的定义" class="headerlink" title="函数的定义"></a>函数的定义</h3><ol>
<li>函数以 <code>def</code> 关键词开头，后接函数名和圆括号 <code>()</code>。</li>
<li>函数执行的代码以冒号起始，并且缩进。</li>
<li><code>return [表达式]</code> 结束函数，选择性地返回一个值给调用方。不带表达式的 return 相当于返回 <code>None</code> 。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">functionname</span><span class="params">(parameters)</span>:</span></span><br><span class="line"> <span class="string">"函数_文档字符串"</span></span><br><span class="line"> function_suite</span><br><span class="line"> <span class="keyword">return</span> [expression]</span><br></pre></td></tr></table></figure>
</summary>
<category term="python" scheme="https://vincent507cpu.github.io/tags/python/"/>
</entry>
<entry>
<title>[DSU&阿里云天池] Python训练营 Task 2</title>
<link href="https://vincent507cpu.github.io/2020/12/21/DSU-%E9%98%BF%E9%87%8C%E4%BA%91%E5%A4%A9%E6%B1%A0-Python%E8%AE%AD%E7%BB%83%E8%90%A5-Task-2/"/>
<id>https://vincent507cpu.github.io/2020/12/21/DSU-%E9%98%BF%E9%87%8C%E4%BA%91%E5%A4%A9%E6%B1%A0-Python%E8%AE%AD%E7%BB%83%E8%90%A5-Task-2/</id>
<published>2020-12-22T02:30:03.000Z</published>
<updated>2020-12-23T16:23:45.345Z</updated>
<content type="html"><![CDATA[<p>[TOC]</p><h1 id="列表"><a href="#列表" class="headerlink" title="列表"></a>列表</h1><h2 id="定义"><a href="#定义" class="headerlink" title="定义"></a>定义</h2><p>列表是有序不定长集合语法为 <code>[元素 1, 元素 2, 元素 3,..., 元素 N]</code></p><h2 id="列表的创建"><a href="#列表的创建" class="headerlink" title="列表的创建"></a>列表的创建</h2><h3 id="创建一个普通列表"><a href="#创建一个普通列表" class="headerlink" title="创建一个普通列表"></a>创建一个普通列表</h3><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">a = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>] <span class="comment"># [1, 2, 3]</span></span><br><span class="line">b = list(<span class="number">4</span>, <span class="number">5</span>, <span class="number">6</span>) <span class="comment"># [4, 5, 6]</span></span><br></pre></td></tr></table></figure><h3 id="使用列表解析式"><a href="#使用列表解析式" class="headerlink" title="使用列表解析式"></a>使用列表解析式</h3><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[i <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">10</span>)] <span class="comment"># [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]</span></span><br></pre></td></tr></table></figure><p><strong>列表内的元素可以是不同类型。</strong></p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mix = [<span class="number">1</span>, <span class="string">'lsgo'</span>, <span class="number">3.14</span>, [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>]] <span class="comment"># mix = [1, 'lsgo', 3.14, [1, 2, 3]]</span></span><br></pre></td></tr></table></figure><a id="more"></a><h3 id="创建一个空列表。"><a href="#创建一个空列表。" class="headerlink" title="创建一个空列表。"></a>创建一个空列表。</h3><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">lst = [] <span class="comment"># []</span></span><br><span class="line">lst2 = list() <span class="comment"># []</span></span><br></pre></td></tr></table></figure><h3 id="列表的方法"><a href="#列表的方法" class="headerlink" title="列表的方法"></a>列表的方法</h3><ul><li>向列表结尾添加一个元素：<code>lst.append(obj)</code><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">lst = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>]</span><br><span class="line">lst.append(<span class="number">4</span>) <span class="comment"># [1, 2, 3, 4]</span></span><br></pre></td></tr></table></figure></li><li>向列表结尾添加多个元素：<code>lst.extend(obj)</code><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">lst = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>]</span><br><span class="line">lst.extend([<span class="number">4</span>, <span class="number">5</span>]) <span class="comment"># [1, 2, 3, 4, 5]</span></span><br></pre></td></tr></table></figure>注意 <code>append</code> 和 <code>extend</code> 的区别：<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">lst = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>]</span><br><span class="line">lst.append([<span class="number">4</span>, <span class="number">5</span>]) <span class="comment"># [1, 2, 3, [4, 5]]</span></span><br></pre></td></tr></table></figure></li><li>删除列表中的指定元素：<code>lst.remove(obj)</code><br><code>lst.remove()</code> 删除列表中的第一个匹配项。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">lst = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">1</span>]</span><br><span class="line">lst.remove(<span class="number">1</span>) <span class="comment"># [2, 3, 1]</span></span><br></pre></td></tr></table></figure></li><li>删除列表中指定位置的元素：<code>lst.pop([index=-1])</code><br>通常省略<code>index=-1</code>，即移除最后一个值，并返回它。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">lst = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>]</span><br><span class="line">y = lst.pop()</span><br><span class="line">print(lst, y)</span><br><span class="line"><span class="comment"># [1, 2] 3</span></span><br></pre></td></tr></table></figure></li><li>在指定位置 <code>index</code> 插入元素：<code>lst.insert(index, obj)</code><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">lst = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>]</span><br><span class="line">lst.insert(<span class="number">0</span>, <span class="number">4</span>)</span><br><span class="line">print(lst)</span><br><span class="line"><span class="comment"># [4, 1, 2, 3]</span></span><br></pre></td></tr></table></figure><h3 id="获取列表中的元素"><a href="#获取列表中的元素" class="headerlink" title="获取列表中的元素"></a>获取列表中的元素</h3></li><li>通过元素的索引值，从列表获取单个元素，注意，列表索引值是从0开始的。</li><li>通过将索引指定为-1，可让Python返回最后一个列表元素，索引 -2 返回倒数第二个列表元素，以此类推。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">lst = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>]</span><br><span class="line">print(lst[<span class="number">0</span>])</span><br><span class="line"><span class="comment"># [1]</span></span><br></pre></td></tr></table></figure>切片的通用写法是 start : stop : step</li><li>情况 1 - “start :”</li></ul><p>以 step 为 1 (默认) 从编号 start 往列表尾部切片。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">lst = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>]</span><br><span class="line">print(lst[<span class="number">2</span>:])</span><br><span class="line"><span class="comment"># [3, 4, 5]</span></span><br></pre></td></tr></table></figure><ul><li>情况 2 - “: stop”</li></ul><p>以 step 为 1 (默认) 从列表头部往编号 stop 切片。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">lst = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>]</span><br><span class="line">print(lst[:<span class="number">2</span>])</span><br><span class="line"><span class="comment"># [1, 2]</span></span><br></pre></td></tr></table></figure><ul><li>情况 3 - “start : stop”</li></ul><p>以 step 为 1 (默认) 从编号 start 往编号 stop 切片。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">lst = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>]</span><br><span class="line">print(lst[<span class="number">1</span>:<span class="number">3</span>])</span><br><span class="line"><span class="comment"># [2, 3]</span></span><br></pre></td></tr></table></figure><ul><li>情况 4 - “start : stop : step”</li></ul><p>以具体的 step 从编号 start 往编号 stop 切片。注意最后把 step 设为 -1，相当于将列表反向排列。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">lst = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>]</span><br><span class="line">print(lst[<span class="number">1</span>:<span class="number">5</span>:<span class="number">2</span>])</span><br><span class="line"><span class="comment"># [2, 4]</span></span><br></pre></td></tr></table></figure><ul><li>情况 5 - “ : “<br>复制列表中的所有元素（浅拷贝）。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">lst1 = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>]</span><br><span class="line">lst2 = lst1[:]</span><br><span class="line">lst2[<span class="number">0</span>] = <span class="number">4</span></span><br><span class="line">print(lst1, lst2)</span><br><span class="line"><span class="comment"># [1, 2, 3] [4, 2, 3]</span></span><br></pre></td></tr></table></figure><h3 id="列表的常用操作符"><a href="#列表的常用操作符" class="headerlink" title="列表的常用操作符"></a>列表的常用操作符</h3></li><li>等号操作符：<code>==</code></li><li>连接操作符：<code>+</code></li><li>重复操作符：<code>*</code></li><li>成员关系操作符：<code>in</code>、<code>not in</code></li></ul><p>等号 <code>==</code>，只有成员、成员位置都相同时才返回 <code>True</code>。</p><p>列表拼接有两种方式，用加号 <code>+</code>和乘号 <code>*</code>，前者首尾拼接，后者复制拼接。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">list1 = [<span class="number">123</span>, <span class="number">456</span>]</span><br><span class="line">list2 = [<span class="number">456</span>, <span class="number">123</span>]</span><br><span class="line">list3 = [<span class="number">123</span>, <span class="number">456</span>]</span><br><span class="line"></span><br><span class="line">print(list1 == list2) <span class="comment"># False</span></span><br><span class="line">print(list1 == list3) <span class="comment"># True</span></span><br><span class="line"></span><br><span class="line">list4 = list1 + list2 <span class="comment"># extend()</span></span><br><span class="line">print(list4) <span class="comment"># [123, 456, 456, 123]</span></span><br><span class="line"></span><br><span class="line">list5 = list3 * <span class="number">3</span></span><br><span class="line">print(list5) <span class="comment"># [123, 456, 123, 456, 123, 456]</span></span><br><span class="line"></span><br><span class="line">list3 *= <span class="number">3</span></span><br><span class="line">print(list3) <span class="comment"># [123, 456, 123, 456, 123, 456]</span></span><br><span class="line"></span><br><span class="line">print(<span class="number">123</span> <span class="keyword">in</span> list3) <span class="comment"># True</span></span><br><span class="line">print(<span class="number">456</span> <span class="keyword">not</span> <span class="keyword">in</span> list3) <span class="comment"># False</span></span><br></pre></td></tr></table></figure><h3 id="列表的其它方法"><a href="#列表的其它方法" class="headerlink" title="列表的其它方法"></a>列表的其它方法</h3><ul><li><code>lst.count(obj)</code>：统计某个元素在列表中出现的次数<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">list1 = [<span class="number">123</span>, <span class="number">456</span>] * <span class="number">3</span> <span class="comment"># [123, 456, 123, 456, 123, 456]</span></span><br><span class="line">num = list1.count(<span class="number">123</span>)</span><br><span class="line">print(num) <span class="comment"># 3</span></span><br></pre></td></tr></table></figure></li><li><code>lst.reverse()</code>：从列表中找出某个值第一个匹配项的索引位置<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">list1 = [<span class="number">123</span>, <span class="number">456</span>] * <span class="number">5</span></span><br><span class="line">print(list1.index(<span class="number">123</span>)) <span class="comment"># 0</span></span><br><span class="line">print(list1.index(<span class="number">123</span>, <span class="number">1</span>)) <span class="comment"># 2</span></span><br><span class="line">print(list1.index(<span class="number">123</span>, <span class="number">3</span>, <span class="number">7</span>)) <span class="comment"># 4</span></span><br></pre></td></tr></table></figure></li><li><code>lst.index(obj)</code>：反向列表中元素<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">x = [<span class="number">123</span>, <span class="number">456</span>, <span class="number">789</span>]</span><br><span class="line">x.reverse() <span class="comment"># [789, 456, 123]</span></span><br></pre></td></tr></table></figure></li><li><code>lst.sort(key=None, rverse=False)</code>：对原列表进行排序。<ul><li>key – 主要是用来进行比较的元素，只有一个参数，具体的函数的参数就是取自于可迭代对象中，指定可迭代对象中的一个元素来进行排序。</li><li>reverse – 排序规则，reverse = True 降序， reverse = False 升序（默认）。</li><li>该方法没有返回值，但是会对列表的对象进行排序。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">x = [<span class="number">123</span>, <span class="number">456</span>, <span class="number">789</span>, <span class="number">213</span>]</span><br><span class="line">x.sort() <span class="comment"># [123, 213, 456, 789]</span></span><br><span class="line"></span><br><span class="line">x.sort(reverse=<span class="literal">True</span>) <span class="comment"># [789, 456, 213, 123]</span></span><br><span class="line"></span><br><span class="line">x.sort(key=<span class="keyword">lambda</span> a: a[<span class="number">0</span>]) <span class="comment"># [(1, 3), (2, 2), (3, 4), (4, 1)]</span></span><br></pre></td></tr></table></figure><h1 id="元组"><a href="#元组" class="headerlink" title="元组"></a>元组</h1>元组的语法：<code>(元素1, 元素2,元素3,...,元素N)</code><h2 id="创建和访问一个元组"><a href="#创建和访问一个元组" class="headerlink" title="创建和访问一个元组"></a>创建和访问一个元组</h2></li></ul></li><li>Python 的元组与列表类似，不同之处在于tuple被创建后就不能对其进行修改，类似字符串。</li><li>元组使用小括号，列表使用方括号。</li><li>元组与列表类似，也用整数来对它进行索引 (indexing) 和切片 (slicing)。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">t1 = (<span class="number">1</span>, <span class="number">2</span>, <span class="string">'a'</span>) <span class="comment"># (1, 2, 'a')</span></span><br><span class="line">t2 = (<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>, <span class="number">6</span>, <span class="number">7</span>)</span><br><span class="line">t2[<span class="number">1</span>] <span class="comment"># 2</span></span><br><span class="line">t2[<span class="number">4</span>:] <span class="comment"># (5, 6, 7)</span></span><br><span class="line">t2[:<span class="number">3</span>] <span class="comment"># (1, 2, 3)</span></span><br></pre></td></tr></table></figure></li><li>创建元组可以用小括号 ()，也可以什么都不用，为了可读性，建议还是用 ()。</li><li>元组中只包含一个元素时，需要在元素后面添加逗号，否则括号会被当作运算符使用。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">x = (<span class="number">1</span>)</span><br><span class="line">type(x) <span class="comment"># int</span></span><br><span class="line"></span><br><span class="line">x = ()</span><br><span class="line">type(x) <span class="comment"># tuple</span></span><br><span class="line"></span><br><span class="line">x = (<span class="number">1</span>, )</span><br><span class="line">type(x) <span class="comment"># tuple</span></span><br></pre></td></tr></table></figure><h2 id="更新和删除一个元组"><a href="#更新和删除一个元组" class="headerlink" title="更新和删除一个元组"></a>更新和删除一个元组</h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">num = (<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>)</span><br><span class="line">new = num[:<span class="number">2</span>] + (<span class="number">3</span>, ) + num[<span class="number">4</span>:] <span class="comment"># (1, 2, 3, 5)</span></span><br></pre></td></tr></table></figure>元组有不可更改 (immutable) 的性质，因此不能直接给元组的元素赋值，但是只要元组中的元素可更改 (mutable)，那么我们可以直接更改其元素，注意这跟赋值其元素不同。<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">t = (1, 2, 3, [4, 5, 6])</span><br><span class="line">t[3][0] = 9</span><br><span class="line">t # (1, 2, 3, [9, 5, 6])</span><br></pre></td></tr></table></figure><h2 id="元组相关的操作符"><a href="#元组相关的操作符" class="headerlink" title="元组相关的操作符"></a>元组相关的操作符</h2></li><li>等号操作符：<code>==</code></li><li>连接操作符：<code>+</code></li><li>重复操作符：<code>*</code></li><li>成员关系操作符：<code>in</code>、<code>not in</code></li></ul><p>等号 <code>==</code>，只有成员、成员位置都相同时才返回 <code>True</code>。</p><p>元组拼接有两种方式，用加号 <code>+</code> 和乘号 <code>*</code>，前者首尾拼接，后者复制拼接。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">t1 = (<span class="number">123</span>, <span class="number">456</span>)</span><br><span class="line">t2 = (<span class="number">456</span>, <span class="number">123</span>)</span><br><span class="line">t3 = (<span class="number">123</span>, <span class="number">456</span>)</span><br><span class="line"></span><br><span class="line">print(t1 == t2) <span class="comment"># False</span></span><br><span class="line">print(t1 == t3) <span class="comment"># True</span></span><br><span class="line"></span><br><span class="line">t4 = t1 + t2</span><br><span class="line">print(t4) <span class="comment"># (123, 456, 456, 123)</span></span><br><span class="line"></span><br><span class="line">t5 = t3 * <span class="number">3</span></span><br><span class="line">print(t5) <span class="comment"># (123, 456, 123, 456, 123, 456)</span></span><br><span class="line"></span><br><span class="line">t3 *= <span class="number">3</span></span><br><span class="line">print(t3) <span class="comment"># (123, 456, 123, 456, 123, 456)</span></span><br><span class="line"></span><br><span class="line">print(<span class="number">123</span> <span class="keyword">in</span> t3) <span class="comment"># True</span></span><br><span class="line">print(<span class="number">456</span> <span class="keyword">not</span> <span class="keyword">in</span> t3) <span class="comment"># False</span></span><br></pre></td></tr></table></figure><h2 id="内置方法"><a href="#内置方法" class="headerlink" title="内置方法"></a>内置方法</h2><ul><li><code>count</code>：返回在元组中该元素出现几次<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">t = (<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">2</span>)</span><br><span class="line">t.count(<span class="number">2</span>) <span class="comment"># 2</span></span><br></pre></td></tr></table></figure></li><li><code>index</code>：找到指定元素在元组中的索引<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">t = (<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">2</span>)</span><br><span class="line">t.index(<span class="number">3</span>) <span class="comment"># 2</span></span><br></pre></td></tr></table></figure><h2 id="解压元组"><a href="#解压元组" class="headerlink" title="解压元组"></a>解压元组</h2>如果只想要元组其中几个元素，用通配符 <code>*</code>。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">t = (<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>)</span><br><span class="line">a, b, *rest, c = t</span><br><span class="line">print(a, b, c) <span class="comment"># 1 2 5</span></span><br><span class="line">peint(rest) <span class="comment"># 3, 4</span></span><br></pre></td></tr></table></figure><h1 id="字符串"><a href="#字符串" class="headerlink" title="字符串"></a>字符串</h1><h2 id="字符串的定义"><a href="#字符串的定义" class="headerlink" title="字符串的定义"></a>字符串的定义</h2></li><li>Python 中字符串被定义为引号之间的字符集合。</li><li>Python 支持使用成对的 单引号 或 双引号。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">print(<span class="number">1</span> + <span class="number">2</span>) <span class="comment"># 3</span></span><br><span class="line">print(<span class="string">'1'</span> + <span class="string">'2'</span>) <span class="comment"># 12</span></span><br></pre></td></tr></table></figure></li><li>常用的转义字符：<code>\\</code> 反斜杠符号，<code>\t</code> 横向制表符，<code>\n</code> 换行符<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">print(<span class="string">'let\'s go'</span>) <span class="comment"># let's go</span></span><br><span class="line">print(<span class="string">'C:\\temp'</span>) <span class="comment"># C:\temp</span></span><br></pre></td></tr></table></figure></li><li>原始字符串只需要在字符串前边加一个英文字母 <code>r</code> 即可。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(<span class="string">r'C:\temp'</span>) <span class="comment"># C:\temp</span></span><br></pre></td></tr></table></figure><h2 id="字符串的切片与拼接"><a href="#字符串的切片与拼接" class="headerlink" title="字符串的切片与拼接"></a>字符串的切片与拼接</h2></li><li>类似于元组具有不可修改性；</li><li>从 0 开始；</li><li>切片通常写成 <code>start:end</code> 这种形式，包括 <code>start</code> 索引对应的元素，不包括<code>end</code> 索引对应的元素；</li><li>索引值可正可负，正索引从 0 开始，从左往右；负索引从 -1 开始，从右往左。使用负数索引时，会从最后一个元素开始计数。最后一个元素的位置编号是 -1。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">s1 = <span class="string">'Hello World'</span></span><br><span class="line">s1[:<span class="number">5</span>] <span class="comment"># Hello</span></span><br></pre></td></tr></table></figure><h2 id="字符串的常用内置方法"><a href="#字符串的常用内置方法" class="headerlink" title="字符串的常用内置方法"></a>字符串的常用内置方法</h2></li><li><code>capitalize()</code>：将字符串的第一个字符转换为大写。</li><li><code>lower()</code>：转换字符串中所有大写字符为小写。</li><li><code>upper()</code>：转换字符串中的小写字母为大写。</li><li><code>swapcase()</code>：将字符串中大写转换为小写，小写转换为大写。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">s = <span class="string">'hello WORLD'</span></span><br><span class="line">s.capitalize() <span class="comment"># Hello WORLD</span></span><br><span class="line">s.lower() <span class="comment"># hello world</span></span><br><span class="line">s.upper() <span class="comment"># HELLO WORLD</span></span><br><span class="line">s.swapcase() <span class="comment"># HELLO world</span></span><br></pre></td></tr></table></figure></li><li><code>count(str, beg= 0,end=len(string))</code> 返回 <code>str</code> 在 string 里面出现的次数，如果 <code>beg</code> 或者 <code>end</code> 指定则返回指定范围内 <code>str</code> 出现的次数。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">s = <span class="string">'DAXIExiaoxie'</span></span><br><span class="line">s.count(<span class="string">'xi'</span>) <span class="comment"># 2</span></span><br></pre></td></tr></table></figure></li><li><code>endswith(suffix, beg=0, end=len(string))</code> 检查字符串是否以指定子字符串 <code>suffix</code> 结束，如果是，返回 <code>True</code>，否则返回 <code>False</code>。如果 <code>beg</code> 和 <code>end</code> 指定值，则在指定范围内检查。</li><li><code>startswith(substr, beg=0,end=len(string))</code> 检查字符串是否以指定子字符串 <code>substr</code> 开头，如果是，返回 <code>True</code>，否则返回 <code>False1</code>。如果 <code>beg</code> 和 <code>end</code> 指定值，则在指定范围内检查。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">s = <span class="string">'hello world'</span></span><br><span class="line">s.startswith(<span class="string">'he'</span>) <span class="comment"># True</span></span><br><span class="line">s.endswith(<span class="string">'LD'</span>) <span class="comment"># False</span></span><br></pre></td></tr></table></figure></li><li><code>find(str, beg=0, end=len(string))</code> 检测 <code>str</code> 是否包含在字符串中，如果指定范围 <code>beg</code> 和 <code>end</code>，则检查是否包含在指定范围内，如果包含，返回开始的索引值，否则返回 -1。</li><li><code>rfind(str, beg=0,end=len(string))</code> 类似于 <code>find()</code> 函数，不过是从右边开始查找。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">s = <span class="string">'hello world'</span></span><br><span class="line">s.find(<span class="string">'o'</span>) <span class="comment"># 4</span></span><br><span class="line">s.rfind(<span class="string">'o'</span>) <span class="comment"># 7</span></span><br></pre></td></tr></table></figure></li><li><code>isnumeric()</code> 如果字符串中只包含数字字符，则返回 <code>True</code>，否则返回 <code>False</code>。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">s = <span class="string">'1234'</span></span><br><span class="line">s.isnumeric() <span class="comment"># True</span></span><br><span class="line">s += <span class="string">'a'</span> <span class="comment"># '1234a'</span></span><br><span class="line">s.isnumeric() <span class="comment"># False</span></span><br></pre></td></tr></table></figure></li><li><code>ljust(width[, fillchar])</code> 返回一个原字符串左对齐，并使用fillchar（默认空格）填充至长度width的新字符串。</li><li><code>rjust(width[, fillchar])</code> 返回一个原字符串右对齐，并使用fillchar（默认空格）填充至长度width的新字符串。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">s = <span class="string">'abcd'</span></span><br><span class="line">print(s.ljust(<span class="number">8</span>, <span class="string">'0'</span>)) <span class="comment"># 'abcd0000'</span></span><br><span class="line">print(s.rjust(<span class="number">8</span>, <span class="string">'0'</span>)) <span class="comment"># '0000abcd'</span></span><br></pre></td></tr></table></figure></li><li>lstrip([chars]) 截掉字符串左边的空格或指定字符。</li><li>rstrip([chars]) 删除字符串末尾的空格或指定字符。</li><li>strip([chars]) 在字符串上执行lstrip()和rstrip()。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">s = <span class="string">' Hello World '</span></span><br><span class="line">s.lstrip() <span class="comment"># 'Hello World '</span></span><br><span class="line">s.rstrip() <span class="comment"># ' Hello World'</span></span><br><span class="line">s.strip() <span class="comment"># 'Hello World'</span></span><br></pre></td></tr></table></figure></li><li><code>partition(sub)</code> 找到子字符串 <code>sub</code>，把字符串分为一个三元组 <code>(pre_sub,sub,fol_sub)</code>，如果字符串中不包含 <code>sub</code> 则返回 <code>('原字符串','','')</code>。</li><li><code>rpartition(sub)</code> 类似于 <code>partition()</code> 方法，不过是从右边开始查找。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">s = <span class="string">'Hello World'</span></span><br><span class="line">s.partition(<span class="string">'o'</span>) <span class="comment"># ('Hell', 'o W', 'orld')</span></span><br><span class="line">s.rpartition(<span class="string">'o'</span>) <span class="comment"># ('Hello W', 'o', 'rld')</span></span><br></pre></td></tr></table></figure></li><li><code>replace(old, new [, max])</code> 把 将字符串中的 <code>old</code> 替换成 <code>new</code>，如果 <code>max</code> 指定，则替换不超过 <code>max</code> 次。<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">s = 'Hello World'</span><br><span class="line">s.replace('o', 'a', 1) # Hella World</span><br></pre></td></tr></table></figure></li><li><code>split(str="", num)</code> 不带参数默认是以空格为分隔符切片字符串，如果 <code>num</code> 参数有设置，则仅分隔 <code>num</code> 个子字符串，返回切片后的子字符串拼接的列表。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">s = <span class="string">'Hello World'</span></span><br><span class="line">s.split() <span class="comment"># ['Hello', 'World']</span></span><br><span class="line">s.split(<span class="string">'l'</span>, <span class="number">2</span>) <span class="comment"># ['He', '', 'o World']</span></span><br></pre></td></tr></table></figure><h2 id="字符串格式化"><a href="#字符串格式化" class="headerlink" title="字符串格式化"></a>字符串格式化</h2><code>f-string</code> 格式化函数<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">```</span><br><span class="line"><span class="comment"># 字典</span></span><br><span class="line"><span class="comment">## 可变类型与不可变类型</span></span><br><span class="line">- 序列是以连续的整数为索引，与此不同的是，字典以<span class="string">"关键字"</span>为索引，关键字可以是任意不可变类型，通常用字符串或数值。</span><br><span class="line">- 字典是 Python 唯一的一个 映射类型，字符串、元组、列表属于序列类型。</span><br><span class="line"></span><br><span class="line">那么如何快速判断一个数据类型 X 是不是可变类型的呢？两种方法：</span><br><span class="line">- 麻烦方法：用 `id(X)` 函数，对 `X` 进行某种操作，比较操作前后的 `id`，如果不一样，则 `X` 不可变，如果一样，则 X 可变。</span><br><span class="line">- 便捷方法：用 `hash(X)`，只要不报错，证明 `X` 可被哈希，即不可变，反过来不可被哈希，即可变。</span><br><span class="line">```py</span><br><span class="line">i = <span class="number">1</span></span><br><span class="line">print(id(i)) <span class="comment"># 4376351872</span></span><br><span class="line">i = i + <span class="number">2</span></span><br><span class="line">print(id(i)) <span class="comment"># 4376351936</span></span><br><span class="line"></span><br><span class="line">l = [<span class="number">1</span>, <span class="number">2</span>]</span><br><span class="line">print(id(l)) <span class="comment"># 140478358404224</span></span><br><span class="line">l.append(<span class="string">'Python'</span>)</span><br><span class="line">print(id(l)) <span class="comment"># 140478358404224</span></span><br></pre></td></tr></table></figure></li><li>整数 <code>i</code> 在加 1 之后的 <code>id</code> 和之前不一样，因此加完之后的这个 <code>i</code> (虽然名字没变)，但不是加之前的那个 <code>i</code> 了，因此整数是不可变类型。</li><li>列表 <code>l</code> 在附加 <code>'Python'</code> 之后的 <code>id</code> 和之前一样，因此列表是可变类型。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">print(hash(<span class="string">'Name'</span>)) <span class="comment"># 2936281444635265970</span></span><br><span class="line">print(hash((<span class="number">1</span>, <span class="number">2</span>, <span class="string">'Python'</span>))) <span class="comment"># 5769585943857102932</span></span><br><span class="line">print(hash([<span class="number">1</span>, <span class="number">2</span>, <span class="string">'Python'</span>]))</span><br><span class="line">---------------------------------------------------------------------------</span><br><span class="line">TypeError Traceback (most recent call last)</span><br><span class="line"><ipython-input<span class="number">-146</span>-f9149cd3bfae> <span class="keyword">in</span> <module></span><br><span class="line"> <span class="number">3</span> print(hash((<span class="number">1</span>, <span class="number">2</span>, <span class="string">'Python'</span>))) <span class="comment"># 1704535747474881831</span></span><br><span class="line"> <span class="number">4</span> </span><br><span class="line">----> 5 print(hash([1, 2, 'Python']))</span><br><span class="line"></span><br><span class="line">TypeError: unhashable type: <span class="string">'list'</span></span><br></pre></td></tr></table></figure></li><li>数值、字符和元组 都能被哈希，因此它们是不可变类型。</li><li>列表、集合、字典不能被哈希，因此它是可变类型。<h2 id="字典的定义"><a href="#字典的定义" class="headerlink" title="字典的定义"></a>字典的定义</h2></li></ul><p>字典 是无序的 键:值（<code>key:value</code>）对集合，键必须是互不相同的（在同一个字典之内）。</p><ul><li><code>dict</code> 内部存放的顺序和 <code>key</code> 放入的顺序是没有关系的。</li><li><code>dict</code> 查找和插入的速度极快，不会随着 <code>key</code> 的增加而增加，但是需要占用大量的内存。<br>字典 定义语法为 <code>{元素1, 元素2, ..., 元素n}</code></li><li>其中每一个元素是一个「键值对」– 键:值 (<code>key:value</code>)</li><li>关键点是大括号 <code>{}</code>,逗号 <code>,</code>和冒号 <code>:</code></li><li>大括号 – 把所有元素绑在一起</li><li>逗号 – 将每个键值对分开</li><li>冒号 – 将键和值分开<h2 id="创建和访问字典"><a href="#创建和访问字典" class="headerlink" title="创建和访问字典"></a>创建和访问字典</h2>如果我们取的键在字典中不存在，会直接报错KeyError。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">dct = {<span class="string">'a'</span>:<span class="string">'hello'</span>, <span class="string">'b'</span>:<span class="string">'world'</span>} <span class="comment"># {'a':'hello', 'b':'world'}</span></span><br><span class="line">dct[<span class="string">'c'</span>]</span><br><span class="line">---------------------------------------------------------------------------</span><br><span class="line">KeyError Traceback (most recent call last)</span><br><span class="line"><ipython-input<span class="number">-147</span><span class="number">-556</span>b5f4b8778> <span class="keyword">in</span> <module></span><br><span class="line"> <span class="number">1</span> dct = {<span class="string">'a'</span>:<span class="string">'hello'</span>, <span class="string">'b'</span>:<span class="string">'world'</span>} <span class="comment"># {'a':'hello', 'b':'world'}</span></span><br><span class="line">----> 2 dct['c']</span><br><span class="line"></span><br><span class="line">KeyError: <span class="string">'c'</span></span><br></pre></td></tr></table></figure></li><li><code>dict()</code> 创建一个空的字典。</li></ul><p>通过 <code>key</code> 直接把数据放入字典中，但一个 <code>key</code> 只能对应一个 <code>value</code>，多次对一个 <code>key</code> 放入 <code>value</code>，后面的值会把前面的值冲掉。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">dic = dict()</span><br><span class="line">dic[<span class="string">'a'</span>] = <span class="number">1</span></span><br><span class="line">dic[<span class="string">'b'</span>] = <span class="number">2</span></span><br><span class="line">dic[<span class="string">'c'</span>] = <span class="number">3</span></span><br><span class="line"></span><br><span class="line">print(dic) <span class="comment"># {'a': 1, 'b': 2, 'c': 3}</span></span><br><span class="line"></span><br><span class="line">dic[<span class="string">'a'</span>] = <span class="number">11</span></span><br><span class="line">print(dic) <span class="comment"># {'a': 11, 'b': 2, 'c': 3}</span></span><br><span class="line"></span><br><span class="line">dic[<span class="string">'d'</span>] = <span class="number">4</span></span><br><span class="line">print(dic) <span class="comment"># {'a': 11, 'b': 2, 'c': 3, 'd': 4}</span></span><br></pre></td></tr></table></figure><ul><li><code>dict(mapping)</code> 通过键值对创建字典<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">dic1 = dict([(<span class="string">'apple'</span>, <span class="number">4139</span>), (<span class="string">'peach'</span>, <span class="number">4127</span>), (<span class="string">'cherry'</span>, <span class="number">4098</span>)])</span><br><span class="line">print(dic1) <span class="comment"># {'cherry': 4098, 'apple': 4139, 'peach': 4127}</span></span><br></pre></td></tr></table></figure></li><li><code>dict(**kwargs)</code> 通过 <code>name=value</code> 方式创建字典<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">dic = dict(name=<span class="string">'Tom'</span>, age=<span class="number">10</span>)</span><br><span class="line">print(dic) <span class="comment"># {'name': 'Tom', 'age': 10}</span></span><br></pre></td></tr></table></figure><h2 id="字典的内置方法"><a href="#字典的内置方法" class="headerlink" title="字典的内置方法"></a>字典的内置方法</h2></li><li><code>dict.fromkeys(seq[, value])</code> 用于创建一个新字典，以序列 <code>seq</code> 中元素做字典的键，<code>value</code> 为字典所有键对应的初始值。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">seq = (<span class="string">'name'</span>, <span class="string">'age'</span>, <span class="string">'sex'</span>)</span><br><span class="line">dic1 = dict.fromkeys(seq)</span><br><span class="line">print(dic1)</span><br><span class="line"><span class="comment"># {'name': None, 'age': None, 'sex': None}</span></span><br><span class="line"></span><br><span class="line">dic2 = dict.fromkeys(seq, <span class="number">10</span>)</span><br><span class="line">print(dic2)</span><br><span class="line"><span class="comment"># {'name': 10, 'age': 10, 'sex': 10}</span></span><br></pre></td></tr></table></figure></li><li><code>dict.keys()</code> 返回一个可迭代对象，可以使用 <code>list()</code> 来转换为列表，列表为字典中的所有键。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">dic = {<span class="string">'Name'</span>: <span class="string">'lsgogroup'</span>, <span class="string">'Age'</span>: <span class="number">7</span>}</span><br><span class="line">print(dic.keys()) <span class="comment"># dict_keys(['Name', 'Age'])</span></span><br><span class="line">lst = list(dic.keys()) <span class="comment"># 转换为列表</span></span><br><span class="line">print(lst) <span class="comment"># ['Name', 'Age']</span></span><br></pre></td></tr></table></figure></li><li><code>dict.values()</code> 返回一个迭代器，可以使用 <code>list()</code> 来转换为列表，列表为字典中的所有值。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">dic = {<span class="string">'Sex'</span>: <span class="string">'female'</span>, <span class="string">'Age'</span>: <span class="number">7</span>, <span class="string">'Name'</span>: <span class="string">'Zara'</span>}</span><br><span class="line">print(dic.values()) <span class="comment"># dict_values(['female', 7, 'Zara'])</span></span><br><span class="line">print(list(dic.values()))<span class="comment"># [7, 'female', 'Zara']</span></span><br></pre></td></tr></table></figure></li><li><code>dict.items()</code> 以列表返回可遍历的 (键, 值) 元组数组。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">dic = {<span class="string">'Name'</span>: <span class="string">'Lsgogroup'</span>, <span class="string">'Age'</span>: <span class="number">7</span>}</span><br><span class="line">print(dic.items()) <span class="comment"># dict_items([('Name', 'Lsgogroup'), ('Age', 7)])</span></span><br><span class="line"></span><br><span class="line">print(tuple(dic.items())) <span class="comment"># (('Name', 'Lsgogroup'), ('Age', 7))</span></span><br><span class="line"></span><br><span class="line">print(list(dic.items())) <span class="comment"># [('Name', 'Lsgogroup'), ('Age', 7)]</span></span><br></pre></td></tr></table></figure></li><li><code>dict.get(key, default=None)</code> 返回指定键的值，如果值不在字典中返回默认值。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">dic = {<span class="string">'Name'</span>: <span class="string">'Lsgogroup'</span>, <span class="string">'Age'</span>: <span class="number">27</span>}</span><br><span class="line">print(<span class="string">"Age 值为 : %s"</span> % dic.get(<span class="string">'Age'</span>)) <span class="comment"># Age 值为 : 27</span></span><br><span class="line">print(<span class="string">"Sex 值为 : %s"</span> % dic.get(<span class="string">'Sex'</span>, <span class="string">"NA"</span>)) <span class="comment"># Sex 值为 : NA</span></span><br><span class="line">print(dic) <span class="comment"># {'Name': 'Lsgogroup', 'Age': 27}</span></span><br></pre></td></tr></table></figure></li><li><code>dict.setdefault(key, default=None)</code> 和 <code>get()</code> 方法 类似, 如果键不存在于字典中，将会添加键并将值设为默认值。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">dic = {<span class="string">'Name'</span>: <span class="string">'Lsgogroup'</span>, <span class="string">'Age'</span>: <span class="number">7</span>}</span><br><span class="line">print(<span class="string">"Age 键的值为 : %s"</span> % dic.setdefault(<span class="string">'Age'</span>, <span class="literal">None</span>)) <span class="comment"># Age 键的值为 : 7</span></span><br><span class="line">print(<span class="string">"Sex 键的值为 : %s"</span> % dic.setdefault(<span class="string">'Sex'</span>, <span class="literal">None</span>)) <span class="comment"># Sex 键的值为 : None</span></span><br><span class="line">print(dic) </span><br><span class="line"><span class="comment"># {'Age': 7, 'Name': 'Lsgogroup', 'Sex': None}</span></span><br></pre></td></tr></table></figure></li><li><code>dict.pop(key[,default])</code> 删除字典给定键 <code>key</code> 所对应的值，返回值为被删除的值。<code>key</code> 值必须给出。若 <code>key</code> 不存在，则返回 <code>default</code> 值。</li><li><code>del dict[key]</code> 删除字典给定键 <code>key</code> 所对应的值。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">dic1 = {<span class="number">1</span>: <span class="string">"a"</span>, <span class="number">2</span>: [<span class="number">1</span>, <span class="number">2</span>]}</span><br><span class="line">print(dic1.pop(<span class="number">1</span>), dic1) <span class="comment"># a {2: [1, 2]}</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 设置默认值，必须添加，否则报错</span></span><br><span class="line">print(dic1.pop(<span class="number">3</span>, <span class="string">"nokey"</span>), dic1) <span class="comment"># nokey {2: [1, 2]}</span></span><br></pre></td></tr></table></figure></li><li><code>dict.popitem()</code> 随机返回并删除字典中的一对键和值，如果字典已经为空，却调用了此方法，就报出 <code>KeyError</code> 异常。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">dic1 = {<span class="number">1</span>: <span class="string">"a"</span>, <span class="number">2</span>: [<span class="number">1</span>, <span class="number">2</span>]}</span><br><span class="line">print(dic1.popitem()) <span class="comment"># {2: [1, 2]}</span></span><br><span class="line">print(dic1) <span class="comment"># (1, 'a')</span></span><br></pre></td></tr></table></figure></li><li>dict.clear()用于删除字典内所有元素。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">dic = {<span class="string">'Name'</span>: <span class="string">'Zara'</span>, <span class="string">'Age'</span>: <span class="number">7</span>}</span><br><span class="line">print(<span class="string">"字典长度 : %d"</span> % len(dic)) <span class="comment"># 字典长度 : 2</span></span><br><span class="line">dic.clear()</span><br><span class="line">print(<span class="string">"字典删除后长度 : %d"</span> % len(dic)) <span class="comment"># 字典删除后长度 : 0</span></span><br></pre></td></tr></table></figure></li><li><code>dict.copy()</code> 返回一个字典的浅复制。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">dic1 = {<span class="string">'Name'</span>: <span class="string">'Lsgogroup'</span>, <span class="string">'Age'</span>: <span class="number">7</span>, <span class="string">'Class'</span>: <span class="string">'First'</span>}</span><br><span class="line">dic2 = dic1.copy()</span><br><span class="line">print(<span class="string">"dic2"</span>) <span class="comment"># {'Age': 7, 'Name': 'Lsgogroup', 'Class': 'First'}</span></span><br></pre></td></tr></table></figure></li><li><code>dict.update(dict2)</code> 把字典参数 <code>dict2</code> 的 <code>key:value</code> 对更新到字典 <code>dict</code> 里。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">dic = {<span class="string">'Name'</span>: <span class="string">'Lsgogroup'</span>, <span class="string">'Age'</span>: <span class="number">7</span>}</span><br><span class="line">dic2 = {<span class="string">'Sex'</span>: <span class="string">'female'</span>, <span class="string">'Age'</span>: <span class="number">8</span>}</span><br><span class="line">dic.update(dic2)</span><br><span class="line">print(dic) </span><br><span class="line">{<span class="string">'Name'</span>: <span class="string">'Lsgogroup'</span>, <span class="string">'Age'</span>: <span class="number">8</span>, <span class="string">'Sex'</span>: <span class="string">'female'</span>}</span><br></pre></td></tr></table></figure><h1 id="集合"><a href="#集合" class="headerlink" title="集合"></a>集合</h1>Python 中 <code>set</code> 与 <code>dict</code> 类似，也是一组 <code>key</code> 的集合，但不存储 <code>value</code>。由于<code>key</code> 不能重复，所以在 <code>set</code> 中，没有重复的 <code>key</code>。</li></ul><p>注意，<code>key</code> 为不可变类型，即可哈希的值。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">num = {}</span><br><span class="line">print(type(num)) <span class="comment"># <class 'dict'></span></span><br><span class="line">num = {<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>}</span><br><span class="line">print(type(num)) <span class="comment"># <class 'set'></span></span><br></pre></td></tr></table></figure><h2 id="集合的创建"><a href="#集合的创建" class="headerlink" title="集合的创建"></a>集合的创建</h2><ul><li><p>先创建对象再加入元素。</p></li><li><p>在创建空集合的时候只能使用 <code>s = set()</code>，因为 <code>s = {}</code> 创建的是空字典。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">basket = set()</span><br><span class="line">basket.add(<span class="string">'apple'</span>)</span><br><span class="line">basket.add(<span class="string">'banana'</span>)</span><br><span class="line">print(basket) <span class="comment"># {'banana', 'apple'}</span></span><br></pre></td></tr></table></figure></li><li><p>直接把一堆元素用花括号括起来 <code>{元素1, 元素2, ..., 元素n}</code>。</p></li><li><p>重复元素在 <code>set</code> 中会被自动被过滤。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">basket = {<span class="string">'apple'</span>, <span class="string">'orange'</span>, <span class="string">'apple'</span>, <span class="string">'pear'</span>, <span class="string">'orange'</span>, <span class="string">'banana'</span>}</span><br><span class="line">print(basket) <span class="comment"># {'banana', 'apple', 'pear', 'orange'}</span></span><br></pre></td></tr></table></figure></li><li><p>使用 <code>set(value)</code> 工厂函数，把列表或元组转换成集合。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">a = set(<span class="string">'abracadabra'</span>)</span><br><span class="line">print(a) </span><br><span class="line"><span class="comment"># {'r', 'b', 'd', 'c', 'a'}</span></span><br><span class="line"></span><br><span class="line">b = set((<span class="string">"Google"</span>, <span class="string">"Lsgogroup"</span>, <span class="string">"Taobao"</span>, <span class="string">"Taobao"</span>))</span><br><span class="line">print(b) </span><br><span class="line"><span class="comment"># {'Taobao', 'Lsgogroup', 'Google'}</span></span><br><span class="line"></span><br><span class="line">c = set([<span class="string">"Google"</span>, <span class="string">"Lsgogroup"</span>, <span class="string">"Taobao"</span>, <span class="string">"Google"</span>])</span><br><span class="line">print(c) </span><br><span class="line"><span class="comment"># {'Taobao', 'Lsgogroup', 'Google'}</span></span><br></pre></td></tr></table></figure><h2 id="访问集合中的值"><a href="#访问集合中的值" class="headerlink" title="访问集合中的值"></a>访问集合中的值</h2></li><li><p>可以使用 <code>len()</code> 內建函数得到集合的大小。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">s = set([<span class="string">'Google'</span>, <span class="string">'Baidu'</span>, <span class="string">'Taobao'</span>])</span><br><span class="line">print(len(s)) <span class="comment"># 3</span></span><br></pre></td></tr></table></figure></li><li><p>可以使用 <code>for</code> 把集合中的数据一个个读取出来。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">s = set([<span class="string">'Google'</span>, <span class="string">'Baidu'</span>, <span class="string">'Taobao'</span>])</span><br><span class="line"><span class="keyword">for</span> item <span class="keyword">in</span> s:</span><br><span class="line"> print(item) </span><br><span class="line"><span class="comment"># Baidu</span></span><br><span class="line"><span class="comment"># Google</span></span><br><span class="line"><span class="comment"># Taobao</span></span><br></pre></td></tr></table></figure></li><li><p>可以通过 <code>in</code> 或 <code>not in</code> 判断一个元素是否在集合中已经存在。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">s = set([<span class="string">'Google'</span>, <span class="string">'Baidu'</span>, <span class="string">'Taobao'</span>])</span><br><span class="line">print(<span class="string">'Taobao'</span> <span class="keyword">in</span> s) <span class="comment"># True</span></span><br><span class="line">print(<span class="string">'Facebook'</span> <span class="keyword">not</span> <span class="keyword">in</span> s) <span class="comment"># True</span></span><br></pre></td></tr></table></figure><h2 id="集合的内置方法"><a href="#集合的内置方法" class="headerlink" title="集合的内置方法"></a>集合的内置方法</h2></li><li><p><code>set.add(elmnt)</code> 用于给集合添加元素，如果添加的元素在集合中已存在，则不执行任何操作。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">fruits = {<span class="string">"apple"</span>, <span class="string">"banana"</span>, <span class="string">"cherry"</span>}</span><br><span class="line">fruits.add(<span class="string">"orange"</span>)</span><br><span class="line">print(fruits) </span><br><span class="line"><span class="comment"># {'orange', 'cherry', 'banana', 'apple'}</span></span><br><span class="line"></span><br><span class="line">fruits.add(<span class="string">"apple"</span>)</span><br><span class="line">print(fruits) </span><br><span class="line"><span class="comment"># {'orange', 'cherry', 'banana', 'apple'}</span></span><br></pre></td></tr></table></figure></li><li><p><code>set.update(set)</code> 用于修改当前集合，可以添加新的元素或集合到当前集合中，如果添加的元素在集合中已存在，则该元素只会出现一次，重复的会忽略。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">x = {<span class="string">"apple"</span>, <span class="string">"banana"</span>, <span class="string">"cherry"</span>}</span><br><span class="line">y = {<span class="string">"google"</span>, <span class="string">"baidu"</span>, <span class="string">"apple"</span>}</span><br><span class="line">x.update(y)</span><br><span class="line">print(x) <span class="comment"># {'cherry', 'banana', 'apple', 'google', 'baidu'}</span></span><br></pre></td></tr></table></figure></li><li><p><code>set.remove(item)</code> 用于移除集合中的指定元素。如果元素不存在，则会发生错误。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">fruits = {<span class="string">"apple"</span>, <span class="string">"banana"</span>, <span class="string">"cherry"</span>}</span><br><span class="line">fruits.remove(<span class="string">"banana"</span>)</span><br><span class="line">print(fruits) <span class="comment"># {'apple', 'cherry'}</span></span><br></pre></td></tr></table></figure></li><li><p><code>set.discard(value)</code> 用于移除指定的集合元素。<code>remove()</code> 方法在移除一个不存在的元素时会发生错误，而 <code>discard()</code> 方法不会。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">fruits = {<span class="string">"apple"</span>, <span class="string">"banana"</span>, <span class="string">"cherry"</span>}</span><br><span class="line">fruits.discard(<span class="string">"banana"</span>)</span><br><span class="line">print(fruits) <span class="comment"># {'apple', 'cherry'}</span></span><br></pre></td></tr></table></figure></li><li><p><code>set.pop()</code> 用于随机移除一个元素。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">fruits = {<span class="string">"apple"</span>, <span class="string">"banana"</span>, <span class="string">"cherry"</span>}</span><br><span class="line">x = fruits.pop()</span><br><span class="line">print(fruits) <span class="comment"># {'cherry', 'apple'}</span></span><br><span class="line">print(x) <span class="comment"># banana</span></span><br></pre></td></tr></table></figure><p>由于 set 是无序和无重复元素的集合，所以两个或多个 set 可以做数学意义上的集合操作。</p></li><li><p><code>set.intersection(set1, set2)</code> 返回两个集合的交集。</p></li><li><p><code>set1 & set2</code> 返回两个集合的交集。</p></li><li><p><code>set.intersection_update(set1, set2)</code> 交集，在原始的集合上移除不重叠的元素。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">a = set(<span class="string">'abracadabra'</span>)</span><br><span class="line">b = set(<span class="string">'alacazam'</span>)</span><br><span class="line">print(a) <span class="comment"># {'r', 'a', 'c', 'b', 'd'}</span></span><br><span class="line">print(b) <span class="comment"># {'c', 'a', 'l', 'm', 'z'}</span></span><br><span class="line"></span><br><span class="line">c = a.intersection(b)</span><br><span class="line">print(c) <span class="comment"># {'a', 'c'}</span></span><br><span class="line">print(a & b) <span class="comment"># {'c', 'a'}</span></span><br><span class="line">print(a) <span class="comment"># {'a', 'r', 'c', 'b', 'd'}</span></span><br><span class="line"></span><br><span class="line">a.intersection_update(b)</span><br><span class="line">print(a) <span class="comment"># {'a', 'c'}</span></span><br></pre></td></tr></table></figure></li><li><p><code>set.union(set1, set2)</code> 返回两个集合的并集。</p></li><li><p><code>set1 | set2</code> 返回两个集合的并集。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">a = set(<span class="string">'abracadabra'</span>)</span><br><span class="line">b = set(<span class="string">'alacazam'</span>)</span><br><span class="line">print(a) <span class="comment"># {'r', 'a', 'c', 'b', 'd'}</span></span><br><span class="line">print(b) <span class="comment"># {'c', 'a', 'l', 'm', 'z'}</span></span><br><span class="line"></span><br><span class="line">print(a | b) <span class="comment"># {'l', 'd', 'm', 'b', 'a', 'r', 'z', 'c'}</span></span><br><span class="line"></span><br><span class="line">c = a.union(b)</span><br><span class="line">print(c) <span class="comment"># {'c', 'a', 'd', 'm', 'r', 'b', 'z', 'l'}</span></span><br></pre></td></tr></table></figure></li><li><p><code>set.difference(set)</code> 返回集合的差集。</p></li><li><p><code>set1 - set2</code> 返回集合的差集。</p></li><li><p><code>set.difference_update(set)</code> 集合的差集，直接在原来的集合中移除元素，没有返回值。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">a = set(<span class="string">'abracadabra'</span>)</span><br><span class="line">b = set(<span class="string">'alacazam'</span>)</span><br><span class="line">print(a) <span class="comment"># {'r', 'a', 'c', 'b', 'd'}</span></span><br><span class="line">print(b) <span class="comment"># {'c', 'a', 'l', 'm', 'z'}</span></span><br><span class="line"></span><br><span class="line">c = a.difference(b)</span><br><span class="line">print(c) <span class="comment"># {'b', 'd', 'r'}</span></span><br><span class="line">print(a - b) <span class="comment"># {'d', 'b', 'r'}</span></span><br><span class="line"></span><br><span class="line">print(a) <span class="comment"># {'r', 'd', 'c', 'a', 'b'}</span></span><br><span class="line">a.difference_update(b)</span><br><span class="line">print(a) <span class="comment"># {'d', 'r', 'b'}</span></span><br></pre></td></tr></table></figure></li><li><p><code>set.issubset(set)</code> 判断集合是不是被其他集合包含，如果是则返回 <code>True</code>，否则返回 <code>False</code>。</p></li><li><p><code>set1 <= set2</code> 判断集合是不是被其他集合包含，如果是则返回 <code>True</code>，否则返回 <code>False</code>。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">x = {<span class="string">"a"</span>, <span class="string">"b"</span>, <span class="string">"c"</span>}</span><br><span class="line">y = {<span class="string">"f"</span>, <span class="string">"e"</span>, <span class="string">"d"</span>, <span class="string">"c"</span>, <span class="string">"b"</span>, <span class="string">"a"</span>}</span><br><span class="line">z = x.issubset(y)</span><br><span class="line">print(z) <span class="comment"># True</span></span><br><span class="line">print(x <= y) <span class="comment"># True</span></span><br></pre></td></tr></table></figure></li><li><p><code>set.issuperset(set)</code> 用于判断集合是不是包含其他集合，如果是则返回 <code>True</code>，否则返回 <code>False</code>。</p></li><li><p><code>set1 >= set2</code> 判断集合是不是包含其他集合，如果是则返回 <code>True</code>，否则返回 <code>False</code>。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">x = {<span class="string">"f"</span>, <span class="string">"e"</span>, <span class="string">"d"</span>, <span class="string">"c"</span>, <span class="string">"b"</span>, <span class="string">"a"</span>}</span><br><span class="line">y = {<span class="string">"a"</span>, <span class="string">"b"</span>, <span class="string">"c"</span>}</span><br><span class="line">z = x.issuperset(y)</span><br><span class="line">print(z) <span class="comment"># True</span></span><br><span class="line">print(x >= y) <span class="comment"># True</span></span><br></pre></td></tr></table></figure></li><li><p><code>set.isdisjoint(set)</code> 用于判断两个集合是不是不相交，如果是返回 <code>True</code>，否则返回 <code>False</code>。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">x = {<span class="string">"f"</span>, <span class="string">"e"</span>, <span class="string">"d"</span>, <span class="string">"c"</span>, <span class="string">"b"</span>}</span><br><span class="line">y = {<span class="string">"a"</span>, <span class="string">"b"</span>, <span class="string">"c"</span>}</span><br><span class="line">z = x.isdisjoint(y)</span><br><span class="line">print(z) <span class="comment"># False</span></span><br><span class="line"></span><br><span class="line">x = {<span class="string">"f"</span>, <span class="string">"e"</span>, <span class="string">"d"</span>, <span class="string">"m"</span>, <span class="string">"g"</span>}</span><br><span class="line">y = {<span class="string">"a"</span>, <span class="string">"b"</span>, <span class="string">"c"</span>}</span><br><span class="line">z = x.isdisjoint(y)</span><br><span class="line">print(z) <span class="comment"># True</span></span><br></pre></td></tr></table></figure><h2 id="不可变集合"><a href="#不可变集合" class="headerlink" title="不可变集合"></a>不可变集合</h2></li><li><p><code>frozenset([iterable])</code> 返回一个冻结的集合，冻结后集合不能再添加或删除任何元素。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">a = frozenset(range(<span class="number">10</span>)) <span class="comment"># 生成一个新的不可变集合</span></span><br><span class="line">print(a) <span class="comment"># frozenset({0, 1, 2, 3, 4, 5, 6, 7, 8, 9})</span></span><br><span class="line"></span><br><span class="line">b = frozenset(<span class="string">'lsgogroup'</span>)</span><br><span class="line">print(b) <span class="comment"># frozenset({'g', 's', 'p', 'r', 'u', 'o', 'l'})</span></span><br></pre></td></tr></table></figure><h1 id="序列"><a href="#序列" class="headerlink" title="序列"></a>序列</h1><h2 id="针对序列的内置函数"><a href="#针对序列的内置函数" class="headerlink" title="针对序列的内置函数"></a>针对序列的内置函数</h2></li><li><p><code>list(sub)</code> 把一个可迭代对象转换为列表。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">a = list()</span><br><span class="line">print(a) <span class="comment"># []</span></span><br><span class="line"></span><br><span class="line">b = <span class="string">'I Love LsgoGroup'</span></span><br><span class="line">b = list(b)</span><br><span class="line">print(b) </span><br><span class="line"><span class="comment"># ['I', ' ', 'L', 'o', 'v', 'e', ' ', 'L', 's', 'g', 'o', 'G', 'r', 'o', 'u', 'p']</span></span><br><span class="line"></span><br><span class="line">c = (<span class="number">1</span>, <span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">5</span>, <span class="number">8</span>)</span><br><span class="line">c = list(c)</span><br><span class="line">print(c) <span class="comment"># [1, 1, 2, 3, 5, 8]</span></span><br></pre></td></tr></table></figure></li><li><p><code>tuple(sub)</code> 把一个可迭代对象转换为元组。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">a = tuple()</span><br><span class="line">print(a) <span class="comment"># ()</span></span><br><span class="line"></span><br><span class="line">b = <span class="string">'I Love LsgoGroup'</span></span><br><span class="line">b = tuple(b)</span><br><span class="line">print(b) </span><br><span class="line"><span class="comment"># ('I', ' ', 'L', 'o', 'v', 'e', ' ', 'L', 's', 'g', 'o', 'G', 'r', 'o', 'u', 'p')</span></span><br><span class="line"></span><br><span class="line">c = [<span class="number">1</span>, <span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">5</span>, <span class="number">8</span>]</span><br><span class="line">c = tuple(c)</span><br><span class="line">print(c) <span class="comment"># (1, 1, 2, 3, 5, 8)</span></span><br></pre></td></tr></table></figure></li><li><p><code>str(obj)</code> 把 <code>obj</code> 对象转换为字符串</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">a = <span class="number">123</span></span><br><span class="line">a = str(a)</span><br><span class="line">print(a) <span class="comment"># 123</span></span><br></pre></td></tr></table></figure></li><li><p><code>len(s)</code> 返回对象（字符、列表、元组等）长度或元素个数。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">a = list()</span><br><span class="line">print(len(a)) <span class="comment"># 0</span></span><br><span class="line"></span><br><span class="line">b = (<span class="string">'I'</span>, <span class="string">' '</span>, <span class="string">'L'</span>, <span class="string">'o'</span>, <span class="string">'v'</span>, <span class="string">'e'</span>, <span class="string">' '</span>, <span class="string">'L'</span>, <span class="string">'s'</span>, <span class="string">'g'</span>, <span class="string">'o'</span>, <span class="string">'G'</span>, <span class="string">'r'</span>, <span class="string">'o'</span>, <span class="string">'u'</span>, <span class="string">'p'</span>)</span><br><span class="line">print(len(b)) <span class="comment"># 16</span></span><br><span class="line"></span><br><span class="line">c = <span class="string">'I Love LsgoGroup'</span></span><br><span class="line">print(len(c)) <span class="comment"># 16</span></span><br></pre></td></tr></table></figure></li><li><p><code>max(sub)</code> 返回序列或者参数集合中的最大值</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">print(max(<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>)) <span class="comment"># 5</span></span><br><span class="line">print(max([<span class="number">-8</span>, <span class="number">99</span>, <span class="number">3</span>, <span class="number">7</span>, <span class="number">83</span>])) <span class="comment"># 99</span></span><br><span class="line">print(max(<span class="string">'IloveLsgoGroup'</span>)) <span class="comment"># v</span></span><br></pre></td></tr></table></figure></li><li><p><code>min(sub)</code> 返回序列或参数集合中的最小值</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">print(min(<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>)) <span class="comment"># 1</span></span><br><span class="line">print(min([<span class="number">-8</span>, <span class="number">99</span>, <span class="number">3</span>, <span class="number">7</span>, <span class="number">83</span>])) <span class="comment"># -8</span></span><br><span class="line">print(min(<span class="string">'IloveLsgoGroup'</span>)) <span class="comment"># G</span></span><br></pre></td></tr></table></figure></li><li><p><code>sum(iterable[, start=0])</code> 返回序列 <code>iterable</code> 与可选参数 <code>start</code> 的总和。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">print(sum([<span class="number">1</span>, <span class="number">3</span>, <span class="number">5</span>, <span class="number">7</span>, <span class="number">9</span>])) <span class="comment"># 25</span></span><br><span class="line">print(sum([<span class="number">1</span>, <span class="number">3</span>, <span class="number">5</span>, <span class="number">7</span>, <span class="number">9</span>], <span class="number">10</span>)) <span class="comment"># 35</span></span><br><span class="line">print(sum((<span class="number">1</span>, <span class="number">3</span>, <span class="number">5</span>, <span class="number">7</span>, <span class="number">9</span>))) <span class="comment"># 25</span></span><br><span class="line">print(sum((<span class="number">1</span>, <span class="number">3</span>, <span class="number">5</span>, <span class="number">7</span>, <span class="number">9</span>), <span class="number">20</span>)) <span class="comment"># 45</span></span><br></pre></td></tr></table></figure></li><li><p><code>sorted(iterable, key=None, reverse=False)</code> 对所有可迭代的对象进行排序操作。</p><ul><li><code>iterable</code> – 可迭代对象。</li><li><code>key</code> – 主要是用来进行比较的元素，只有一个参数，具体的函数的参数就是取自于可迭代对象中，指定可迭代对象中的一个元素来进行排序。</li><li><code>reverse</code> – 排序规则，<code>reverse = True</code> 降序 ， <code>reverse = False</code> 升序（默认）。</li><li>返回重新排序的列表。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">x = [<span class="number">-8</span>, <span class="number">99</span>, <span class="number">3</span>, <span class="number">7</span>, <span class="number">83</span>]</span><br><span class="line">print(sorted(x)) <span class="comment"># [-8, 3, 7, 83, 99]</span></span><br><span class="line">print(sorted(x, reverse=<span class="literal">True</span>)) <span class="comment"># [99, 83, 7, 3, -8]</span></span><br><span class="line"></span><br><span class="line">t = ({<span class="string">"age"</span>: <span class="number">20</span>, <span class="string">"name"</span>: <span class="string">"a"</span>}, {<span class="string">"age"</span>: <span class="number">25</span>, <span class="string">"name"</span>: <span class="string">"b"</span>}, {<span class="string">"age"</span>: <span class="number">10</span>, <span class="string">"name"</span>: <span class="string">"c"</span>})</span><br><span class="line">x = sorted(t, key=<span class="keyword">lambda</span> a: a[<span class="string">"age"</span>])</span><br><span class="line">print(x)</span><br><span class="line"><span class="comment"># [{'age': 10, 'name': 'c'}, {'age': 20, 'name': 'a'}, {'age': 25, 'name': 'b'}]</span></span><br></pre></td></tr></table></figure></li></ul></li><li><p><code>reversed(seq)</code> 函数返回一个反转的迭代器。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">s = <span class="string">'lsgogroup'</span></span><br><span class="line">x = reversed(s)</span><br><span class="line">print(type(x)) <span class="comment"># <class 'reversed'></span></span><br><span class="line">print(x) <span class="comment"># <reversed object at 0x000002507E8EC2C8></span></span><br><span class="line">print(list(x))</span><br><span class="line"><span class="comment"># ['p', 'u', 'o', 'r', 'g', 'o', 'g', 's', 'l']</span></span><br><span class="line"></span><br><span class="line">t = (<span class="string">'l'</span>, <span class="string">'s'</span>, <span class="string">'g'</span>, <span class="string">'o'</span>, <span class="string">'g'</span>, <span class="string">'r'</span>, <span class="string">'o'</span>, <span class="string">'u'</span>, <span class="string">'p'</span>)</span><br><span class="line">print(list(reversed(t)))</span><br><span class="line"><span class="comment"># ['p', 'u', 'o', 'r', 'g', 'o', 'g', 's', 'l']</span></span><br><span class="line"></span><br><span class="line">r = range(<span class="number">5</span>, <span class="number">9</span>)</span><br><span class="line">print(list(reversed(r)))</span><br><span class="line"><span class="comment"># [8, 7, 6, 5]</span></span><br><span class="line"></span><br><span class="line">x = [<span class="number">-8</span>, <span class="number">99</span>, <span class="number">3</span>, <span class="number">7</span>, <span class="number">83</span>]</span><br><span class="line">print(list(reversed(x)))</span><br><span class="line"><span class="comment"># [83, 7, 3, 99, -8]</span></span><br></pre></td></tr></table></figure></li><li><p><code>enumerate(sequence, [start=0])</code> 用于将一个可遍历的数据对象(如列表、元组或字符串)组合为一个索引序列，同时列出数据和数据下标，一般用在 <code>for</code> 循环当中。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">seasons = [<span class="string">'Spring'</span>, <span class="string">'Summer'</span>, <span class="string">'Fall'</span>, <span class="string">'Winter'</span>]</span><br><span class="line">a = list(enumerate(seasons))</span><br><span class="line">print(a) </span><br><span class="line"><span class="comment"># [(0, 'Spring'), (1, 'Summer'), (2, 'Fall'), (3, 'Winter')]</span></span><br><span class="line"></span><br><span class="line">b = list(enumerate(seasons, <span class="number">1</span>))</span><br><span class="line">print(b) </span><br><span class="line"><span class="comment"># [(1, 'Spring'), (2, 'Summer'), (3, 'Fall'), (4, 'Winter')]</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> i, element <span class="keyword">in</span> a:</span><br><span class="line"> print(<span class="string">'{0},{1}'</span>.format(i, element))</span><br><span class="line"><span class="comment"># 0,Spring</span></span><br><span class="line"><span class="comment"># 1,Summer</span></span><br><span class="line"><span class="comment"># 2,Fall</span></span><br><span class="line"><span class="comment"># 3,Winter</span></span><br></pre></td></tr></table></figure></li></ul>]]></content>
<summary type="html">
<p>[TOC]</p>
<h1 id="列表"><a href="#列表" class="headerlink" title="列表"></a>列表</h1><h2 id="定义"><a href="#定义" class="headerlink" title="定义"></a>定义</h2><p>列表是有序不定长集合语法为 <code>[元素 1, 元素 2, 元素 3,..., 元素 N]</code></p>
<h2 id="列表的创建"><a href="#列表的创建" class="headerlink" title="列表的创建"></a>列表的创建</h2><h3 id="创建一个普通列表"><a href="#创建一个普通列表" class="headerlink" title="创建一个普通列表"></a>创建一个普通列表</h3><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">a = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>] <span class="comment"># [1, 2, 3]</span></span><br><span class="line">b = list(<span class="number">4</span>, <span class="number">5</span>, <span class="number">6</span>) <span class="comment"># [4, 5, 6]</span></span><br></pre></td></tr></table></figure>
<h3 id="使用列表解析式"><a href="#使用列表解析式" class="headerlink" title="使用列表解析式"></a>使用列表解析式</h3><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[i <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">10</span>)] <span class="comment"># [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]</span></span><br></pre></td></tr></table></figure>
<p><strong>列表内的元素可以是不同类型。</strong></p>
<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mix = [<span class="number">1</span>, <span class="string">'lsgo'</span>, <span class="number">3.14</span>, [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>]] <span class="comment"># mix = [1, 'lsgo', 3.14, [1, 2, 3]]</span></span><br></pre></td></tr></table></figure>
</summary>
<category term="python" scheme="https://vincent507cpu.github.io/tags/python/"/>
</entry>
<entry>
<title>[经验总结]取整与取余</title>
<link href="https://vincent507cpu.github.io/2020/12/19/%E7%BB%8F%E9%AA%8C%E6%80%BB%E7%BB%93-%E5%8F%96%E6%95%B4%E4%B8%8E%E5%8F%96%E4%BD%99/"/>
<id>https://vincent507cpu.github.io/2020/12/19/%E7%BB%8F%E9%AA%8C%E6%80%BB%E7%BB%93-%E5%8F%96%E6%95%B4%E4%B8%8E%E5%8F%96%E4%BD%99/</id>
<published>2020-12-19T23:58:43.000Z</published>
<updated>2020-12-20T00:05:36.432Z</updated>
<content type="html"><![CDATA[<p>最近开始复习、深挖 Python 基础知识，有机会深入探索一些以前没有想过的事情。</p><p>我们知道，Python 内置函数 <code>int</code> 和 <code>round</code> 可以把一个浮点数取整，比如</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span>int(<span class="number">1.1</span>)</span><br><span class="line"><span class="number">1</span></span><br><span class="line"><span class="meta">>>> </span>round(<span class="number">1.1</span>)</span><br><span class="line"><span class="number">1</span></span><br></pre></td></tr></table></figure><p>它们是如何工作的呢？</p><a id="more"></a><h1 id="int-函数"><a href="#int-函数" class="headerlink" title="int 函数"></a><code>int</code> 函数</h1><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">int(x, base=<span class="number">10</span>)</span><br></pre></td></tr></table></figure><p>本文中我们不关注一个数的进制，统一按十进制处理。我们来看一个例子：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">print(int(<span class="number">1.1</span>)) <span class="comment"># 1</span></span><br><span class="line">print(int(<span class="number">1.9</span>)) <span class="comment"># 1</span></span><br><span class="line">print(int(<span class="number">-1.1</span>)) <span class="comment"># -1</span></span><br><span class="line">print(int(<span class="number">-1.9</span>)) <span class="comment"># -1</span></span><br></pre></td></tr></table></figure><p>看起来 <code>int</code> 函数就是简单地截取小数点前面的数值。</p><h1 id="round-函数"><a href="#round-函数" class="headerlink" title="round 函数"></a><code>round</code> 函数</h1><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">round(x, ndigits=<span class="literal">None</span>)</span><br></pre></td></tr></table></figure><p>这里 <code>round</code> 的处理近似于四舍五入，我们先看几个没有争议的例子：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">print(round(<span class="number">1.1</span>)) <span class="comment"># 1</span></span><br><span class="line">print(round(<span class="number">1.9</span>)) <span class="comment"># 2</span></span><br><span class="line">print(round(<span class="number">-1.1</span>)) <span class="comment"># -1</span></span><br><span class="line">print(round(<span class="number">-1.9</span>)) <span class="comment"># -2</span></span><br></pre></td></tr></table></figure><p>现在有意思的时候到了，<code>.5</code> 应该如何进位呢？</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">print(round(<span class="number">1.5</span>)) <span class="comment"># 2</span></span><br><span class="line">print(round(<span class="number">2.5</span>)) <span class="comment"># 2</span></span><br><span class="line">print(round(<span class="number">-1.5</span>)) <span class="comment"># -2</span></span><br><span class="line">print(round(<span class="number">-2.5</span>)) <span class="comment"># -2</span></span><br></pre></td></tr></table></figure><p>根据<a href="https://docs.python.org/3/library/functions.html#round" target="_blank" rel="noopener" title="round 函数">官方文档</a>，rounding 会选择被 2 整除的数，所以 1.5 和 2.5 的 rounding 都是 2。当 <code>ndigits</code> 取非 0 值时，这个原则仍然适用。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">print(round(<span class="number">1.55</span>, <span class="number">1</span>)) <span class="comment"># 1.6</span></span><br><span class="line">print(round(<span class="number">1.45</span>, <span class="number">1</span>)) <span class="comment"># 1.4</span></span><br></pre></td></tr></table></figure><p>上述例子中的 <code>ndigits</code> 在为 0 时为 <code>None</code>，取 0 会有什么变化吗？</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">round(<span class="number">1.1</span>, ndigits=<span class="literal">None</span>) <span class="comment"># 1</span></span><br><span class="line">round(<span class="number">1.1</span>, <span class="number">0</span>) <span class="comment"># 1.0</span></span><br></pre></td></tr></table></figure><p>是有变化的！如果 <code>ndigits=None</code> 或者干脆省略，返回一个整数；如果 <code>ndigits=0</code>，返回一个带有一位小数点的整数。无独有偶，<code>numpy</code>，<code>TensorFlow</code> 和 <code>PyTorch</code> 也有 <code>numpy.round</code>，<code>tensorflow.math.round</code> 和 <code>torch.round</code> 与 Python 原生 <code>round</code> 函数对应，它们与原生函数有什么区别呢？</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">print(np.round(<span class="number">1.5</span>)) <span class="comment"># 2.0</span></span><br><span class="line">print(np.round(<span class="number">2.5</span>)) <span class="comment"># 2.0</span></span><br><span class="line"></span><br><span class="line">print(tf.round(tf.Variable(<span class="number">1.5</span>)).numpy()) <span class="comment"># 2.0</span></span><br><span class="line">print(tf.round(tf.Variable(<span class="number">1.5</span>)).numpy()) <span class="comment"># 2.0</span></span><br><span class="line"></span><br><span class="line">print(torch.round(torch.tensor(<span class="number">1.5</span>)).item()) <span class="comment"># 2.0</span></span><br><span class="line">print(torch.round(torch.tensor(<span class="number">2.5</span>)).item()) <span class="comment"># 2.0</span></span><br></pre></td></tr></table></figure><p>可以看到，numpy，TensorFlow 和 PyTorch 的 <code>round</code> 函数的工作原理与 Python 原生函数相同。顺便提一句，TensorFlow 和 PyTorch 的 <code>round</code> 函数只能取整，返回一个带有一位小数点的整数；numpy 的 <code>round</code> 函数与 Python 原生函数相同，但是变量名不是 <code>nsdigits</code> 而是 <code>decimals</code>。</p><h1 id="Python-里的相除取整（-）与相除取余（-）"><a href="#Python-里的相除取整（-）与相除取余（-）" class="headerlink" title="Python 里的相除取整（//）与相除取余（%）"></a>Python 里的相除取整（<code>//</code>）与相除取余（<code>%</code>）</h1><p>所谓的相除取整和相除取余很好理解，就是一个除法如果不能整除的话就分别取整除部分和余数部分：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">print(<span class="number">123</span> // <span class="number">10</span>) <span class="comment"># 12</span></span><br><span class="line">print(<span class="number">123</span> % <span class="number">10</span>) <span class="comment"># 3</span></span><br></pre></td></tr></table></figure><p>本来很简单的一件小事遇到负数就有意思了：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">print(<span class="number">-123</span> // <span class="number">10</span>) <span class="comment"># -13</span></span><br><span class="line">print(<span class="number">123</span> // <span class="number">-10</span>) <span class="comment"># -13</span></span><br><span class="line">print(<span class="number">-123</span> // <span class="number">-10</span>) <span class="comment"># 12</span></span><br><span class="line">print(<span class="number">-123</span> % <span class="number">10</span>) <span class="comment"># 7</span></span><br><span class="line">print(<span class="number">123</span> % <span class="number">-10</span>) <span class="comment"># -7</span></span><br><span class="line">print(<span class="number">-123</span> % <span class="number">-10</span>) <span class="comment"># -3</span></span><br></pre></td></tr></table></figure><p>这是怎么回事呢？<a href="https://blog.csdn.net/sun___M/article/details/83142126" target="_blank" rel="noopener" title="相除取整与相除取余"></a></p><h2 id="相除取整"><a href="#相除取整" class="headerlink" title="相除取整"></a>相除取整</h2><p>我们把几个除法的结果比较一下：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">print(<span class="number">123</span> / <span class="number">10</span>) <span class="comment"># 12.3</span></span><br><span class="line">print(<span class="number">123</span> // <span class="number">10</span>) <span class="comment"># 12</span></span><br><span class="line">print(int(<span class="number">123</span> / <span class="number">10</span>)) <span class="comment"># 12</span></span><br><span class="line">print(<span class="number">-123</span> / <span class="number">10</span>) <span class="comment"># -12.3</span></span><br><span class="line">print(<span class="number">-123</span> // <span class="number">10</span>) <span class="comment"># -13</span></span><br><span class="line">print(int(<span class="number">-123</span> / <span class="number">10</span>)) <span class="comment"># -12</span></span><br><span class="line">print(<span class="number">-123</span> / <span class="number">-10</span>) <span class="comment"># 12.3</span></span><br><span class="line">print(<span class="number">-123</span> // <span class="number">-10</span>) <span class="comment"># 12</span></span><br><span class="line">print(int(<span class="number">-123</span> / <span class="number">-10</span>)) <span class="comment"># 12</span></span><br></pre></td></tr></table></figure><p>在没有看函数源代码的情况下，我们可以大概说，<code>//</code> 操作为向下取整。如果想要一个负结果的向上取整的结果，可以使用 <code>int</code> 配合普通除法。负数除以一个负数与正数除以一个正数的结果相同。</p><h2 id="相除取余"><a href="#相除取余" class="headerlink" title="相除取余"></a>相除取余</h2><p>还是比较几个除法取余的结果：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">print(<span class="number">123</span> % <span class="number">10</span>) <span class="comment"># 3</span></span><br><span class="line">print(<span class="number">123</span> % <span class="number">-10</span>) <span class="comment"># -7</span></span><br><span class="line">print(<span class="number">-123</span> % <span class="number">10</span>) <span class="comment"># 7</span></span><br><span class="line">print(<span class="number">-123</span> % <span class="number">-10</span>) <span class="comment"># -3</span></span><br></pre></td></tr></table></figure><p>其实在 Python 中，取余的计算公式与别的语言并没有什么区别：<br>$$r = a - n * [a // n]$$<br>其中 <code>r</code> 是余数，<code>a</code> 是被除数，<code>n</code> 是除数。不过在 <code>a // n</code> 这一步，当 <code>a</code> 是负数的时候，上面提到会向下取整，所以有：<br>$$-123 % 10 = -123 - 10 * (-123 // 10) = -123 - 10 * (-13) = 7$$<br>其余的两个相除取余也可以按照此法推演出来。</p>]]></content>
<summary type="html">
<p>最近开始复习、深挖 Python 基础知识，有机会深入探索一些以前没有想过的事情。</p>
<p>我们知道，Python 内置函数 <code>int</code> 和 <code>round</code> 可以把一个浮点数取整，比如</p>
<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span>int(<span class="number">1.1</span>)</span><br><span class="line"><span class="number">1</span></span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>round(<span class="number">1.1</span>)</span><br><span class="line"><span class="number">1</span></span><br></pre></td></tr></table></figure>
<p>它们是如何工作的呢？</p>
</summary>
<category term="经验总结" scheme="https://vincent507cpu.github.io/tags/%E7%BB%8F%E9%AA%8C%E6%80%BB%E7%BB%93/"/>
</entry>
<entry>
<title>[DSU&阿里云天池] Python训练营 Task 1</title>
<link href="https://vincent507cpu.github.io/2020/12/19/DSU-%E9%98%BF%E9%87%8C%E4%BA%91%E5%A4%A9%E6%B1%A0-Python%E8%AE%AD%E7%BB%83%E8%90%A5-Task-1/"/>
<id>https://vincent507cpu.github.io/2020/12/19/DSU-%E9%98%BF%E9%87%8C%E4%BA%91%E5%A4%A9%E6%B1%A0-Python%E8%AE%AD%E7%BB%83%E8%90%A5-Task-1/</id>
<published>2020-12-19T14:25:04.000Z</published>
<updated>2020-12-21T16:25:30.461Z</updated>
<content type="html"><![CDATA[<p>[TOC]<br>今天参加了由阿里云天池开展的 Python 训练营，借这个机会回顾一下 Python 的基础知识，并深入一些以前没有注意到的点。本文为 task 1。</p><h1 id="变量、运算符与数据类型"><a href="#变量、运算符与数据类型" class="headerlink" title="变量、运算符与数据类型"></a>变量、运算符与数据类型</h1><h2 id="注释"><a href="#注释" class="headerlink" title="注释"></a>注释</h2><h3 id="单行注释"><a href="#单行注释" class="headerlink" title="单行注释"></a>单行注释</h3><p><code>#</code> 表示其后面的整行内容为注释，后面的所有文字会被忽略。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 这是一个注释</span></span><br><span class="line">print(<span class="string">'Hello Word!'</span>) <span class="comment"># '#'也可以放在一条代码的后面</span></span><br><span class="line">Hello Word!</span><br></pre></td></tr></table></figure><a id="more"></a><h3 id="多行注释"><a href="#多行注释" class="headerlink" title="多行注释"></a>多行注释</h3><p>使用 <code>''' ''''</code> 或 <code>""" """</code> 表示多行注释，在三个引号内的内容为注释。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="string">'''</span></span><br><span class="line"><span class="string">这是多行注释，用三个单引号</span></span><br><span class="line"><span class="string">这是多行注释，用三个单引号</span></span><br><span class="line"><span class="string">这是多行注释，用三个单引号</span></span><br><span class="line"><span class="string">'''</span></span><br><span class="line">print(<span class="string">'Hello China'</span>)</span><br><span class="line"><span class="comment"># Hello China</span></span><br><span class="line"></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">这是多行注释，用三个双引号</span></span><br><span class="line"><span class="string">这是多行注释，用三个双引号</span></span><br><span class="line"><span class="string">这是多行注释，用三个双引号</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line">print(<span class="string">'Hello China'</span>)</span><br><span class="line"><span class="comment"># Hello China</span></span><br></pre></td></tr></table></figure><h2 id="运算符"><a href="#运算符" class="headerlink" title="运算符"></a>运算符</h2><h3 id="算术运算符"><a href="#算术运算符" class="headerlink" title="算术运算符"></a>算术运算符</h3><p>算术运算符有 <code>+</code>（加），<code>-</code>（减），<code>*</code>（乘），<code>/</code>（除），<code>//</code>（除取整），<code>%</code>(除取余)和 <code>**</code>（幂次）。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">print(<span class="number">1</span> + <span class="number">2</span>) <span class="comment"># 3</span></span><br><span class="line">print(<span class="number">4</span> - <span class="number">3</span>) <span class="comment"># 1</span></span><br><span class="line">print(<span class="number">5</span> * <span class="number">6</span>) <span class="comment"># 30</span></span><br><span class="line">print(<span class="number">7</span> / <span class="number">8</span>) <span class="comment"># 0.875</span></span><br><span class="line">print(<span class="number">9</span> // <span class="number">10</span>) <span class="comment"># 0，因为 9 不够 1 个 10</span></span><br><span class="line">print(<span class="number">1</span> % <span class="number">2</span>) <span class="comment"># 1，因为 1 除以 2 余 1</span></span><br><span class="line">print(<span class="number">3</span> ** <span class="number">4</span>) <span class="comment"># 81，因为 3 的 4 次幂是 81</span></span><br></pre></td></tr></table></figure><p>相除取整和相除取余在涉及到负数的时候变得复杂，详情请见<a href="https://mp.weixin.qq.com/s?__biz=Mzg3OTIwODUzMQ==&mid=2247485633&idx=1&sn=9d29d82c929feabfed60c45898381a3a&chksm=cf06b9fdf87130eb49e334c806024bff962afafc12f33bfbf71aa5cb486197961df9be88116a&token=457736762&lang=zh_CN#rd" target="_blank" rel="noopener">这里</a>。</p><h3 id="赋值运算符"><a href="#赋值运算符" class="headerlink" title="赋值运算符"></a>赋值运算符</h3><p>赋值运算符为算术运算符后面加个 <code>=</code>，在进行运算以后将新值赋给原变量，即为原地操作（in-place）。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">a = <span class="number">1</span></span><br><span class="line">b = <span class="number">2</span></span><br><span class="line"></span><br><span class="line">a = a + b <span class="comment"># 现在 a 的值为 3</span></span><br><span class="line">a += b <span class="comment"># 新的值被原地赋给了 a</span></span><br><span class="line"></span><br><span class="line">c = <span class="number">3</span></span><br><span class="line">c += <span class="number">1</span> <span class="comment"># 4</span></span><br><span class="line">c -= <span class="number">2</span> <span class="comment"># 2</span></span><br><span class="line">c *= <span class="number">3</span> <span class="comment"># 6</span></span><br><span class="line">c /= <span class="number">4</span> <span class="comment"># 1.5</span></span><br><span class="line"></span><br><span class="line">d = <span class="number">4</span></span><br><span class="line">d **= <span class="number">2</span> <span class="comment"># 16</span></span><br><span class="line">d //= <span class="number">3</span> <span class="comment"># 5</span></span><br><span class="line">d %= <span class="number">4</span> = <span class="number">1</span></span><br></pre></td></tr></table></figure><h3 id="比较运算符"><a href="#比较运算符" class="headerlink" title="比较运算符"></a>比较运算符</h3><p>比较运算符有 <code>></code>（大于），<code><</code>（小于），<code>>=</code>（大于等于），<code><=</code>（小于等于），<code>==</code>（等于）和 <code>!=</code>（不等于），返回 <code>True</code> 或 <code>False</code>。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">print(<span class="number">1</span> > <span class="number">2</span>) <span class="comment"># False</span></span><br><span class="line">print(<span class="number">2</span> < <span class="number">4</span>) <span class="comment"># True</span></span><br><span class="line">print(<span class="number">5</span> == <span class="number">5</span>) <span class="comment"># True</span></span><br><span class="line">print(<span class="number">6</span> != <span class="number">6</span>) <span class="comment"># False</span></span><br></pre></td></tr></table></figure><h3 id="逻辑运算符"><a href="#逻辑运算符" class="headerlink" title="逻辑运算符"></a>逻辑运算符</h3><p>逻辑运算符有 <code>and</code>（和、与），<code>or</code>（或）和 <code>not</code>（非）。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">print(<span class="literal">True</span>) <span class="comment"># True</span></span><br><span class="line">print(<span class="literal">True</span> <span class="keyword">and</span> <span class="literal">False</span>) <span class="comment"># False</span></span><br><span class="line">print(<span class="literal">True</span> <span class="keyword">or</span> <span class="literal">False</span>) <span class="comment"># True</span></span><br><span class="line">print(<span class="keyword">not</span> <span class="literal">True</span>) <span class="comment"># False</span></span><br></pre></td></tr></table></figure><p>只有 <code>and</code> 两边同时为真才返回 <code>True</code>， 取余情况返回 <code>False</code>；只有 <code>or</code> 两边同时为假才返回 <code>False</code>，取余情况返回 <code>True</code>；<code>not</code> 返回与原判断结果相反的结果。</p><h3 id="其它操作符"><a href="#其它操作符" class="headerlink" title="其它操作符"></a>其它操作符</h3><p><code>is</code>（是），<code>in</code>（存在）</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="number">1</span> <span class="keyword">in</span> [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>] <span class="comment"># true</span></span><br><span class="line"><span class="string">'d'</span> <span class="keyword">in</span> <span class="string">'abc'</span> <span class="comment"># False</span></span><br></pre></td></tr></table></figure><h3 id="is-与-的区别"><a href="#is-与-的区别" class="headerlink" title="is 与 == 的区别"></a><code>is</code> 与 <code>==</code> 的区别</h3><ul><li>is, is not 对比的是两个变量的内存地址</li><li>==, != 对比的是两个变量的值</li><li>比较的两个变量，指向的都是地址不可变的类型（str等），那么is，is not 和 ==，！= 是完全等价的。</li><li>对比的两个变量，指向的是地址可变的类型（list，dict，tuple等），则两者是有区别的。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">a, b = <span class="string">'abc'</span>, <span class="string">'abc'</span></span><br><span class="line">print(a <span class="keyword">is</span> b, a == b) <span class="comment"># True True</span></span><br><span class="line">print(a <span class="keyword">is</span> <span class="keyword">not</span> b, a != b) <span class="comment"># False False</span></span><br><span class="line"></span><br><span class="line">a, b = [<span class="string">'abc'</span>], [<span class="string">'abc'</span>]</span><br><span class="line">print(a <span class="keyword">is</span> b, a == b) <span class="comment"># False True</span></span><br><span class="line">print(a <span class="keyword">is</span> <span class="keyword">not</span> b, a != b) <span class="comment"># True False</span></span><br></pre></td></tr></table></figure><h3 id="运算符的优先级"><a href="#运算符的优先级" class="headerlink" title="运算符的优先级"></a>运算符的优先级</h3>不同运算符的优先度不同，优先级从高到低为：<br><code>**</code> > <code>*, /, %, //</code> > <code>+-</code> > <code>&</code> > <code>>, >=, <, <=</code> > <code>==, !=</code> > <code>=, +=, -=, *=, /=, %=, //=</code> > <code>is, not is</code> > <code>in, not in</code> > <code>and, or, not</code><h2 id="变量与赋值"><a href="#变量与赋值" class="headerlink" title="变量与赋值"></a>变量与赋值</h2></li></ul><ol><li>在使用变量之前，需要对其先赋值。</li><li>变量名可以包括字母、数字、下划线、但变量名不能以数字开头。 </li><li>Python 变量名是大小写敏感的，foo != Foo。</li><li>同一行可以赋值多个变量，如 <code>a, b = 1, 2</code>，<code>a</code> 和 <code>b</code> 分别赋予了 1 和 2。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">a, b = <span class="number">1</span>, <span class="number">2</span></span><br><span class="line">c = a + b</span><br><span class="line">print(c) <span class="comment"># 3</span></span><br></pre></td></tr></table></figure><h2 id="数据类型与转换"><a href="#数据类型与转换" class="headerlink" title="数据类型与转换"></a>数据类型与转换</h2>常见的数据类型有：<code>str</code>（字符），<code>int</code>（整型），<code>float</code>（浮点型）和 <code>bool</code>（布尔型）。布尔变量只能是 <code>True</code> 和 <code>False</code>。除了直接给变量赋予 <code>True</code> 和 <code>False</code> 也可以用 <code>bool(X)</code> 让 Python 自行判断，<code>X</code> 可以是一个值（整型，浮点型或布尔型）也可以是一个容器（字符串，列表，元组，集合或字典），判断依据是：</li><li>对于数值，0 为 <code>False</code>，非 0 为 <code>True</code>；</li><li>对于容器，空容器为 <code>False</code>，非空容器为 <code>True</code>。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">print(bool(<span class="literal">False</span>), bool(<span class="literal">True</span>)) <span class="comment"># False, True</span></span><br><span class="line">print(bool(<span class="number">0</span>), bool(<span class="number">1</span>)) <span class="comment"># False, True</span></span><br><span class="line">print(bool(<span class="number">0.0</span>), bool(<span class="number">1.5</span>)) <span class="comment"># False, True</span></span><br><span class="line"></span><br><span class="line">print(bool(<span class="string">''</span>), bool(<span class="string">'abc'</span>)) <span class="comment"># False, True</span></span><br><span class="line">print(bool([]), bool([<span class="number">1</span>, <span class="number">2</span>])) <span class="comment"># False, True</span></span><br><span class="line">print(bool(()), bool((<span class="number">1</span>, <span class="number">2</span>))) <span class="comment"># False, True</span></span><br><span class="line">print(bool({}), bool({<span class="number">1</span>, <span class="number">2</span>})) <span class="comment"># False, True</span></span><br><span class="line">print(bool({}), bool({<span class="string">'a'</span>:<span class="number">1</span>})) <span class="comment"># False, True</span></span><br></pre></td></tr></table></figure></li></ol><p><strong>获取类型信息</strong>：</p><ul><li><code>type(X)</code>：返回 X 的类型信息</li><li><code>isinstance(var, type)</code>：返回 <code>var</code> 是不是 <code>type</code><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">print(type(<span class="number">1</span>)) <span class="comment"># <class 'int'></span></span><br><span class="line">print(type([<span class="number">1</span>, <span class="number">2</span>])) <span class="comment"># <class 'list'></span></span><br><span class="line"></span><br><span class="line">print(isinstance(<span class="number">1</span>, int)) <span class="comment"># True</span></span><br><span class="line">print(isinstance({<span class="number">1</span>, <span class="number">2</span>}), set) <span class="comment"># True</span></span><br></pre></td></tr></table></figure><h2 id="print-函数"><a href="#print-函数" class="headerlink" title="print() 函数"></a><code>print()</code> 函数</h2>函数的语法为<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">print(*objects, sep=<span class="string">' '</span>, end=<span class="string">'\n'</span>, file=sys.stdout, flush=<span class="literal">False</span>)</span><br></pre></td></tr></table></figure></li></ul><ol><li>将对象以字符串表示的方式格式化输出到流文件对象file里。其中所有非关键字参数都按 str() 方式进行转换为字符 串输出;</li><li>关键字参数 sep 是实现分隔符，比如多个参数输出时想要输出中间的分隔字符;</li><li>关键字参数 end 是输出结束时的字符，默认是换行符 \n ;</li><li>关键字参数 file 是定义流输出的文件，可以是标准的系统输出 sys.stdout ，也可以重定义为别的文件;</li><li>关键字参数 flush 是立即把内容输出到流文件，不作缓存。</li></ol><p>常用的变量为 <code>sep</code> 和 <code>end</code>。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 修改 sep 变量</span></span><br><span class="line">print(<span class="string">'apple'</span>, <span class="string">'banana'</span>) <span class="comment"># sep = ' '</span></span><br><span class="line"><span class="comment"># apple banana</span></span><br><span class="line">print(<span class="string">'apple'</span>, <span class="string">'banana'</span>, sep=<span class="string">'&'</span>) <span class="comment"># apple 和 banana 之间用 & 分隔</span></span><br><span class="line"><span class="comment"># apple&banana</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 修改 end 变量</span></span><br><span class="line">fruits = [<span class="string">'apple'</span>, <span class="string">'banana'</span>]</span><br><span class="line">print(<span class="string">"This is printed with default end"</span>)</span><br><span class="line"><span class="keyword">for</span> item <span class="keyword">in</span> fruits:</span><br><span class="line"> print(item)</span><br><span class="line"><span class="comment"># This is printed with default end</span></span><br><span class="line"><span class="comment"># apple</span></span><br><span class="line"><span class="comment"># fruit</span></span><br><span class="line"></span><br><span class="line">print(<span class="string">"This is printed with 'end='&''"</span>)</span><br><span class="line"><span class="keyword">for</span> item <span class="keyword">in</span> fruits:</span><br><span class="line"> print(item, end=<span class="string">'&'</span>)</span><br><span class="line"><span class="comment"># This is printed with 'end='&''</span></span><br><span class="line">apple&banana&</span><br></pre></td></tr></table></figure><h1 id="条件语句"><a href="#条件语句" class="headerlink" title="条件语句"></a>条件语句</h1><h2 id="if-语句"><a href="#if-语句" class="headerlink" title="if 语句"></a><code>if</code> 语句</h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">if</span> expression:</span><br><span class="line"> exp_true_action</span><br></pre></td></tr></table></figure><p>如果 <code>expression</code> 为真，则执行 <code>exp_true_action</code>；否则不会执行。<code>expression</code> 可为多重条件判断。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">if</span> <span class="number">2</span> > <span class="number">1</span> <span class="keyword">and</span> <span class="keyword">not</span> <span class="number">2</span> > <span class="number">3</span>:</span><br><span class="line"> print(<span class="string">'Correct!'</span>)</span><br><span class="line"><span class="comment"># Correct!</span></span><br></pre></td></tr></table></figure><h2 id="if-else-语句"><a href="#if-else-语句" class="headerlink" title="if-else 语句"></a><code>if-else</code> 语句</h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">if</span> expression:</span><br><span class="line"> exp_true_action</span><br><span class="line"><span class="keyword">else</span>:</span><br><span class="line"> exp_false_action</span><br></pre></td></tr></table></figure><p>如果 <code>expression</code> 为真，则执行 <code>exp_true_action</code>；否则执行 <code>exp_false_action</code>。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">color = <span class="string">'red'</span></span><br><span class="line"><span class="keyword">if</span> color == <span class="string">'blue'</span>:</span><br><span class="line"> print(<span class="string">'Color is blue.'</span>)</span><br><span class="line"><span class="keyword">else</span>:</span><br><span class="line"> print(<span class="string">'Color is not blue.'</span>)</span><br><span class="line"><span class="comment"># Color is not blue.</span></span><br></pre></td></tr></table></figure><h2 id="if-elif-else-语句"><a href="#if-elif-else-语句" class="headerlink" title="if-elif-else 语句"></a><code>if-elif-else</code> 语句</h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">if</span> expression1:</span><br><span class="line"> exp1_true_action</span><br><span class="line"><span class="keyword">elif</span> expression2:</span><br><span class="line"> exp2_true_action</span><br><span class="line"> .</span><br><span class="line"> .</span><br><span class="line"><span class="keyword">elif</span> expressionN:</span><br><span class="line"> expN_true_action</span><br><span class="line"><span class="keyword">else</span>:</span><br><span class="line"> exp_false_action</span><br></pre></td></tr></table></figure><p>分别判断哪一个 <code>expression</code> 为真，哪个为真就执行哪个动作；全为假则执行 <code>exp_false_action</code>。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">number = <span class="number">5</span></span><br><span class="line"><span class="keyword">if</span> number < <span class="number">3</span>:</span><br><span class="line"> print(<span class="string">'Number is smaller than 3.'</span>)</span><br><span class="line"><span class="keyword">elif</span> number < <span class="number">7</span>:</span><br><span class="line"> print(<span class="string">'Number is between 3 and 7.'</span>)</span><br><span class="line"><span class="keyword">else</span>:</span><br><span class="line"> print(<span class="string">'Number is greater than 7.'</span>)</span><br><span class="line"><span class="comment"># Number is between 3 and 7.</span></span><br></pre></td></tr></table></figure><h2 id="assert-语句"><a href="#assert-语句" class="headerlink" title="assert 语句"></a><code>assert</code> 语句</h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">assert</span> expression, text</span><br></pre></td></tr></table></figure><p>如果 <code>expression</code> 为假，则中断程序运行，抛出 <code>AssertionError</code> 异常，异常信息为 <code>text</code>。</p><h1 id="循环语句"><a href="#循环语句" class="headerlink" title="循环语句"></a>循环语句</h1><h2 id="while-循环"><a href="#while-循环" class="headerlink" title="while 循环"></a><code>while</code> 循环</h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">while</span> expression:</span><br><span class="line"> action</span><br></pre></td></tr></table></figure><p>如果 <code>expression</code> 为真，则执行 <code>action</code>，然后再判断 <code>expression</code> 是否为真，若还为真则再执行 <code>action</code>，再判断 <code>expression</code> 是否为真，…，直到 <code>expression</code> 为假，循环结束。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">i = <span class="number">0</span></span><br><span class="line"><span class="keyword">while</span> i < <span class="number">2</span>:</span><br><span class="line"> print(i)</span><br><span class="line"> i += <span class="number">1</span></span><br><span class="line"><span class="comment"># 0</span></span><br><span class="line"><span class="comment"># 1</span></span><br></pre></td></tr></table></figure><h2 id="while-else-循环"><a href="#while-else-循环" class="headerlink" title="while-else 循环"></a><code>while-else</code> 循环</h2><p>当 <code>while</code> 循环正常执行完的情况下，执行 <code>else</code> 输出，如果 <code>while</code> 循环中执行了跳出循环的语句，比如 <code>break</code>，将不执行 <code>else</code> 代码块的内容。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">count = <span class="number">0</span></span><br><span class="line"><span class="keyword">while</span> count <<span class="number">5</span>:</span><br><span class="line"> print(<span class="string">f'<span class="subst">{count}</span> is less than 5'</span>)</span><br><span class="line"> count += <span class="number">1</span></span><br><span class="line"><span class="keyword">else</span>:</span><br><span class="line"> print(<span class="string">f'<span class="subst">{count}</span> is not less than 5'</span>)</span><br><span class="line"><span class="comment"># count is less than 5</span></span><br><span class="line"><span class="comment"># count is less than 5</span></span><br><span class="line"><span class="comment"># count is less than 5</span></span><br><span class="line"><span class="comment"># count is less than 5</span></span><br><span class="line"><span class="comment"># count is less than 5</span></span><br><span class="line"><span class="comment"># count is not less than 5</span></span><br><span class="line">count = <span class="number">0</span></span><br><span class="line"><span class="keyword">while</span> count < <span class="number">5</span>:</span><br><span class="line"> print(<span class="string">f'<span class="subst">{count}</span> is less than 5'</span>)</span><br><span class="line"> count = <span class="number">6</span></span><br><span class="line"> <span class="keyword">break</span></span><br><span class="line"><span class="keyword">else</span>:</span><br><span class="line"> print(<span class="string">f'count is not less than 5'</span>)</span><br><span class="line"><span class="comment"># 0 is less than 5</span></span><br></pre></td></tr></table></figure><h2 id="for-循环"><a href="#for-循环" class="headerlink" title="for 循环"></a><code>for</code> 循环</h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">for</span> 迭代变量 <span class="keyword">in</span> 迭代器:</span><br><span class="line"> action</span><br></pre></td></tr></table></figure><p><code>for</code> 循环是迭代循环，在 Python 中相当于一个通用的序列迭代器，可以遍历任何有序序列，如 <code>str、list、tuple</code> 等，也可以遍历任何可迭代对象，如 <code>dict</code>。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">for</span> s <span class="keyword">in</span> <span class="string">'abc'</span>:</span><br><span class="line"> print(s)</span><br><span class="line"><span class="comment"># a</span></span><br><span class="line"><span class="comment"># b</span></span><br><span class="line"><span class="comment"># c</span></span><br></pre></td></tr></table></figure><h2 id="for-else-循环"><a href="#for-else-循环" class="headerlink" title="for-else 循环"></a><code>for-else</code> 循环</h2><p>当 <code>for</code> 循环正常执行完的情况下，执行 <code>else</code> 输出，如果 <code>for</code> 循环中执行了跳出循环的语句，比如 <code>break</code>，将不执行 <code>else</code> 代码块的内容，与 <code>while - else</code> 语句一样。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">2</span>):</span><br><span class="line"> print(i)</span><br><span class="line"><span class="keyword">else</span>:</span><br><span class="line"> print(<span class="string">'done'</span>)</span><br><span class="line"><span class="comment"># 0</span></span><br><span class="line"><span class="comment"># 1</span></span><br><span class="line"><span class="comment"># done</span></span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">2</span>):</span><br><span class="line"> <span class="keyword">if</span> i % <span class="number">2</span> == <span class="number">1</span>:</span><br><span class="line"> <span class="keyword">break</span></span><br><span class="line"> print(i)</span><br><span class="line"><span class="keyword">else</span>:</span><br><span class="line"> print(<span class="string">'done'</span>)</span><br><span class="line"><span class="comment"># 0</span></span><br></pre></td></tr></table></figure><h2 id="range-函数"><a href="#range-函数" class="headerlink" title="range 函数"></a><code>range</code> 函数</h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">range([start,] stop[, step=<span class="number">1</span>])</span><br></pre></td></tr></table></figure><p><code>range</code> 这个内置函数的作用是生成一个从 <code>start</code> 参数的值开始到 <code>stop</code>参数的值结束的数字序列，该序列包含 <code>start</code> 的值但不包含 <code>stop</code> 的值。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">1</span>, <span class="number">5</span>, <span class="number">2</span>):</span><br><span class="line"> print(i)</span><br><span class="line"><span class="comment"># 1</span></span><br><span class="line"><span class="comment"># 3</span></span><br></pre></td></tr></table></figure><h2 id="enumerate-函数"><a href="#enumerate-函数" class="headerlink" title="enumerate 函数"></a><code>enumerate</code> 函数</h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">enumerate(sequence, [start=<span class="number">0</span>])</span><br></pre></td></tr></table></figure><p>返回枚举对象，可与 <code>for</code> 循环连用。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">letters = [<span class="string">'a'</span>, <span class="string">'b'</span>, <span class="string">'c'</span>, <span class="string">'d'</span>]</span><br><span class="line">lst = list(enumerate(letters))</span><br><span class="line">print(lst)</span><br><span class="line"><span class="comment"># [(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd')]</span></span><br><span class="line"><span class="keyword">for</span> idx, letter <span class="keyword">in</span> enumerate(letters, <span class="number">1</span>):</span><br><span class="line"> print(idx, letter)</span><br><span class="line"><span class="comment"># 1 a</span></span><br><span class="line"><span class="comment"># 2 b</span></span><br><span class="line"><span class="comment"># 3 c</span></span><br><span class="line"><span class="comment"># 4 d</span></span><br></pre></td></tr></table></figure><h2 id="break-语句"><a href="#break-语句" class="headerlink" title="break 语句"></a><code>break</code> 语句</h2><p><code>break</code> 语句可以跳出当前所在层的循环，例子见上。</p><h2 id="continue-语句"><a href="#continue-语句" class="headerlink" title="continue 语句"></a><code>continue</code> 语句</h2><p><code>continue</code> 终止本轮循环并开始下一轮循环。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">3</span>):</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> i % <span class="number">2</span>:</span><br><span class="line"> <span class="keyword">continue</span></span><br><span class="line"> print(i)</span><br><span class="line"><span class="comment"># 1</span></span><br></pre></td></tr></table></figure><h2 id="pass-语句"><a href="#pass-语句" class="headerlink" title="pass 语句"></a><code>pass</code> 语句</h2><p><code>pass</code> 语句的意思是“不做任何事”，即不做任何操作，只起到占位的作用，其作用是为了保持程序结构的完整性。尽管 <code>pass</code> 语句不做任何操作，但如果暂时不确定要在一个位置放上什么样的代码，可以先放置一个 <code>pass</code> 语句，让代码可以正常运行。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">3</span>):</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> i % <span class="number">2</span>:</span><br><span class="line"> <span class="keyword">pass</span></span><br><span class="line"> print(i)</span><br><span class="line"><span class="comment"># 0</span></span><br><span class="line"><span class="comment"># 1</span></span><br><span class="line"><span class="comment"># 2</span></span><br></pre></td></tr></table></figure><h2 id="解析式"><a href="#解析式" class="headerlink" title="解析式"></a>解析式</h2><h3 id="列表解析式"><a href="#列表解析式" class="headerlink" title="列表解析式"></a>列表解析式</h3><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[expr <span class="keyword">for</span> val <span class="keyword">in</span> iterable [<span class="keyword">if</span> condition]]</span><br></pre></td></tr></table></figure><p>返回一个根据解析式条件创建的列表。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[i <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">5</span>) <span class="keyword">if</span> i % <span class="number">2</span> == <span class="number">1</span>]</span><br><span class="line"><span class="comment"># [1 ,3]</span></span><br></pre></td></tr></table></figure><h3 id="字典解析式"><a href="#字典解析式" class="headerlink" title="字典解析式"></a>字典解析式</h3><p>类似列表解析式，只是变量为键值对。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">{i:j <span class="keyword">for</span> i, j <span class="keyword">in</span> enumerate(<span class="string">'abcde'</span>)}</span><br><span class="line"><span class="comment"># {0: 'a', 1: 'b', 2: 'c', 3: 'd', 4: 'e'}</span></span><br></pre></td></tr></table></figure><h3 id="集合解析式"><a href="#集合解析式" class="headerlink" title="集合解析式"></a>集合解析式</h3><p>类似列表解析式。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">{i <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">5</span>) <span class="keyword">if</span> i % <span class="number">2</span> == <span class="number">1</span>}</span><br><span class="line"><span class="comment"># {1, 3}</span></span><br></pre></td></tr></table></figure><h3 id="生成式"><a href="#生成式" class="headerlink" title="生成式"></a>生成式</h3><p>类似列表解析式，只是返回一个迭代器对象。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">(i <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">3</span>) <span class="keyword">if</span> i % <span class="number">2</span> == <span class="number">1</span>)</span><br><span class="line"><span class="comment"># <generator object <genexpr> at 0x7fc3aad2b2d0></span></span><br></pre></td></tr></table></figure><h2 id="iter-与-next"><a href="#iter-与-next" class="headerlink" title="iter 与 next"></a><code>iter</code> 与 <code>next</code></h2><p><code>iter</code> 将一个可迭代对象转换为一个迭代器，可以使用 <code>next</code> 进行遍历。遍历结束以后若继续迭代，则抛出 <code>StopIteration</code> 异常。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">iterator = iter(list(range(<span class="number">3</span>)))</span><br><span class="line">print(iterator)</span><br><span class="line"><list_iterator object at <span class="number">0x7fc3aa756990</span>></span><br><span class="line">next(iterator)</span><br><span class="line"><span class="comment"># 0</span></span><br><span class="line">next(iterator)</span><br><span class="line"><span class="comment"># 1</span></span><br><span class="line">next(iterator)</span><br><span class="line"><span class="comment"># 2</span></span><br><span class="line">next(iterator)</span><br><span class="line">---------------------------------------------------------------------------</span><br><span class="line">StopIteration Traceback (most recent call last)</span><br><span class="line"><ipython-input<span class="number">-74</span><span class="number">-4</span>ce711c44abc> <span class="keyword">in</span> <module></span><br><span class="line">----> 1 next(iterator)</span><br><span class="line"></span><br><span class="line">StopIteration:</span><br></pre></td></tr></table></figure><h1 id="异常处理"><a href="#异常处理" class="headerlink" title="异常处理"></a>异常处理</h1><h2 id="Python-常用标准异常"><a href="#Python-常用标准异常" class="headerlink" title="Python 常用标准异常"></a>Python 常用标准异常</h2><ul><li>AssertionError：<code>Assertion</code> 的条件为 <code>False</code></li><li>AttributeError：尝试访问未知的对象属性</li><li>IndexError：索引超出序列的范围</li><li>KeyError：字典中查找一个不存在的关键字</li><li>NameError：尝试访问一个不存在的变量</li><li>TypeError：不同类型间的无效操作</li><li>ValueError：传入无效的参数<h2 id="try-except-语句"><a href="#try-except-语句" class="headerlink" title="try-except 语句"></a><code>try-except</code> 语句</h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">try</span>:</span><br><span class="line"> expression</span><br><span class="line"><span class="keyword">except</span> Exception[ <span class="keyword">as</span> reason]:</span><br><span class="line"> action</span><br></pre></td></tr></table></figure>try 语句按照如下方式工作：</li><li>首先，执行 <code>try</code> 子句（在关键字 <code>try</code> 和关键字 <code>except</code> 之间的语句）</li><li>如果没有异常发生，忽略 <code>except</code> 子句，<code>try</code> 子句执行后结束。</li><li>如果在执行 <code>try</code> 子句的过程中发生了异常，那么 <code>try</code> 子句余下的部分将被忽略。如果异常的类型和 <code>except</code> 之后的名称相符，那么对应的 <code>except</code> 子句将被执行。最后执行<code>try - except</code> 语句之后的代码。</li><li>如果一个异常没有与任何的 <code>except</code> 匹配，那么这个异常将会传递给上层的 <code>try</code> 中。</li><li>一个 <code>try</code> 语句可能包含多个 <code>except</code> 子句，分别来处理不同的特定的异常。最多只有一个分支会被执行。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">try</span>:</span><br><span class="line"> int(<span class="string">"abc"</span>)</span><br><span class="line"> s = <span class="number">1</span> + <span class="string">'1'</span></span><br><span class="line"> f = open(<span class="string">'test.txt'</span>)</span><br><span class="line"> print(f.read())</span><br><span class="line"> f.close()</span><br><span class="line"><span class="keyword">except</span> OSError <span class="keyword">as</span> error:</span><br><span class="line"> print(<span class="string">'打开文件出错\n原因是：'</span> + str(error))</span><br><span class="line"><span class="keyword">except</span> TypeError <span class="keyword">as</span> error:</span><br><span class="line"> print(<span class="string">'类型出错\n原因是：'</span> + str(error))</span><br><span class="line"><span class="keyword">except</span> ValueError <span class="keyword">as</span> error:</span><br><span class="line"> print(<span class="string">'数值出错\n原因是：'</span> + str(error))</span><br><span class="line"></span><br><span class="line"><span class="comment"># 数值出错</span></span><br><span class="line"><span class="comment"># 原因是：invalid literal for int() with base 10: 'abc'</span></span><br></pre></td></tr></table></figure><h2 id="try-except-finally-语句"><a href="#try-except-finally-语句" class="headerlink" title="try-except-finally 语句"></a><code>try-except-finally</code> 语句</h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">try</span>:</span><br><span class="line"> 检测范围</span><br><span class="line"><span class="keyword">except</span> Exception[<span class="keyword">as</span> reason]:</span><br><span class="line"> 出现异常后的处理代码</span><br><span class="line"><span class="keyword">finally</span>:</span><br><span class="line"> 无论如何都会被执行的代码</span><br></pre></td></tr></table></figure>不管<code>try</code>子句里面有没有发生异常，<code>finally</code>子句都会执行。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">divide</span><span class="params">(x, y)</span>:</span></span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> result = x / y</span><br><span class="line"> print(<span class="string">"result is"</span>, result)</span><br><span class="line"> <span class="keyword">except</span> ZeroDivisionError:</span><br><span class="line"> print(<span class="string">"division by zero!"</span>)</span><br><span class="line"> <span class="keyword">finally</span>:</span><br><span class="line"> print(<span class="string">"executing finally clause"</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">divide(<span class="number">2</span>, <span class="number">1</span>)</span><br><span class="line"><span class="comment"># result is 2.0</span></span><br><span class="line"><span class="comment"># executing finally clause</span></span><br><span class="line">divide(<span class="number">2</span>, <span class="number">0</span>)</span><br><span class="line"><span class="comment"># division by zero!</span></span><br><span class="line"><span class="comment"># executing finally clause</span></span><br><span class="line">divide(<span class="string">"2"</span>, <span class="string">"1"</span>)</span><br><span class="line"><span class="comment"># executing finally clause</span></span><br><span class="line">---------------------------------------------------------------------------</span><br><span class="line">TypeError Traceback (most recent call last)</span><br><span class="line"><ipython-input<span class="number">-79</span><span class="number">-16805</span>cf48925> <span class="keyword">in</span> <module></span><br><span class="line"> <span class="number">15</span> <span class="comment"># division by zero!</span></span><br><span class="line"> <span class="number">16</span> <span class="comment"># executing finally clause</span></span><br><span class="line">---> 17 divide("2", "1")</span><br><span class="line"> <span class="number">18</span> <span class="comment"># executing finally clause</span></span><br><span class="line"> <span class="number">19</span> <span class="comment"># TypeError: unsupported operand type(s) for /: 'str' and 'str'</span></span><br><span class="line"></span><br><span class="line"><ipython-input<span class="number">-79</span><span class="number">-16805</span>cf48925> <span class="keyword">in</span> divide(x, y)</span><br><span class="line"> <span class="number">1</span> <span class="function"><span class="keyword">def</span> <span class="title">divide</span><span class="params">(x, y)</span>:</span></span><br><span class="line"> <span class="number">2</span> <span class="keyword">try</span>:</span><br><span class="line">----> 3 result = x / y</span><br><span class="line"> <span class="number">4</span> print(<span class="string">"result is"</span>, result)</span><br><span class="line"> <span class="number">5</span> <span class="keyword">except</span> ZeroDivisionError:</span><br><span class="line"></span><br><span class="line">TypeError: unsupported operand type(s) <span class="keyword">for</span> /: <span class="string">'str'</span> <span class="keyword">and</span> <span class="string">'str'</span></span><br></pre></td></tr></table></figure><h2 id="try-except-else-语句"><a href="#try-except-else-语句" class="headerlink" title="try-except-else 语句"></a><code>try-except-else</code> 语句</h2><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">try</span>:</span><br><span class="line"> 检测范围</span><br><span class="line"><span class="keyword">except</span>:</span><br><span class="line"> 出现异常后的处理代码</span><br><span class="line"><span class="keyword">else</span>:</span><br><span class="line"> 如果没有异常执行这块代码</span><br></pre></td></tr></table></figure>如果 <code>except</code>语句没有执行，则继续执行 <code>else</code> 语句；若执行则跳过 <code>else</code> 语句。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">dict1 = {<span class="string">'a'</span>: <span class="number">1</span>, <span class="string">'b'</span>: <span class="number">2</span>, <span class="string">'v'</span>: <span class="number">22</span>}</span><br><span class="line"><span class="keyword">try</span>:</span><br><span class="line"> x = dict1[<span class="string">'y'</span>]</span><br><span class="line"><span class="keyword">except</span> KeyError:</span><br><span class="line"> print(<span class="string">'键错误'</span>)</span><br><span class="line"><span class="keyword">except</span> LookupError:</span><br><span class="line"> print(<span class="string">'查询错误'</span>)</span><br><span class="line"><span class="keyword">else</span>:</span><br><span class="line"> print(x)</span><br><span class="line"><span class="comment"># 键错误</span></span><br><span class="line">dict1 = {<span class="string">'a'</span>: <span class="number">1</span>, <span class="string">'b'</span>: <span class="number">2</span>, <span class="string">'v'</span>: <span class="number">22</span>}</span><br><span class="line"><span class="keyword">try</span>:</span><br><span class="line"> x = dict1[<span class="string">'a'</span>]</span><br><span class="line"><span class="keyword">except</span> KeyError:</span><br><span class="line"> print(<span class="string">'键错误'</span>)</span><br><span class="line"><span class="keyword">except</span> LookupError:</span><br><span class="line"> print(<span class="string">'查询错误'</span>)</span><br><span class="line"><span class="keyword">else</span>:</span><br><span class="line"> print(x)</span><br><span class="line"><span class="comment"># 1</span></span><br></pre></td></tr></table></figure><h2 id="raise-语句"><a href="#raise-语句" class="headerlink" title="raise 语句"></a><code>raise</code> 语句</h2>抛出一个错误异常。<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">raise NameError()</span><br><span class="line">---------------------------------------------------------------------------</span><br><span class="line">NameError Traceback (most recent call last)</span><br><span class="line"><ipython-input-80-1d90210dd9ab> in <module></span><br><span class="line">----> 1 raise NameError()</span><br><span class="line"></span><br><span class="line">NameError:</span><br></pre></td></tr></table></figure></li></ul>]]></content>
<summary type="html">
<p>[TOC]<br>今天参加了由阿里云天池开展的 Python 训练营，借这个机会回顾一下 Python 的基础知识，并深入一些以前没有注意到的点。本文为 task 1。</p>
<h1 id="变量、运算符与数据类型"><a href="#变量、运算符与数据类型" class="headerlink" title="变量、运算符与数据类型"></a>变量、运算符与数据类型</h1><h2 id="注释"><a href="#注释" class="headerlink" title="注释"></a>注释</h2><h3 id="单行注释"><a href="#单行注释" class="headerlink" title="单行注释"></a>单行注释</h3><p><code>#</code> 表示其后面的整行内容为注释，后面的所有文字会被忽略。</p>
<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 这是一个注释</span></span><br><span class="line">print(<span class="string">'Hello Word!'</span>) <span class="comment"># '#'也可以放在一条代码的后面</span></span><br><span class="line">Hello Word!</span><br></pre></td></tr></table></figure>
</summary>
<category term="基础" scheme="https://vincent507cpu.github.io/tags/%E5%9F%BA%E7%A1%80/"/>
</entry>
<entry>
<title>[工作站] 我的第一台个人深度学习工作站之配置环境篇</title>
<link href="https://vincent507cpu.github.io/2020/11/02/%E5%B7%A5%E4%BD%9C%E7%AB%99-%E6%88%91%E7%9A%84%E7%AC%AC%E4%B8%80%E5%8F%B0%E4%B8%AA%E4%BA%BA%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%B7%A5%E4%BD%9C%E7%AB%99%E4%B9%8B%E9%85%8D%E7%BD%AE%E7%8E%AF%E5%A2%83%E7%AF%87/"/>
<id>https://vincent507cpu.github.io/2020/11/02/%E5%B7%A5%E4%BD%9C%E7%AB%99-%E6%88%91%E7%9A%84%E7%AC%AC%E4%B8%80%E5%8F%B0%E4%B8%AA%E4%BA%BA%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%B7%A5%E4%BD%9C%E7%AB%99%E4%B9%8B%E9%85%8D%E7%BD%AE%E7%8E%AF%E5%A2%83%E7%AF%87/</id>
<published>2020-11-03T04:17:30.000Z</published>
<updated>2020-11-10T02:30:32.045Z</updated>
<content type="html"><![CDATA[<p>工作站的配置有两种方式：</p><ul><li>传统方法；</li><li>Docker。</li></ul><p>理论上 Docker 更好一点，因为 <code>nvidia-docker2</code> 是一个已经配置好的环境，不需要手动安装 CUDA 和 cuDNN，而且不需要了可以删除，更新也很方便。然而本文中将采用传统方法对工作站进行配置。使用 Docker 可以参考这篇<a href="https://cnvrg.io/how-to-setup-docker-and-nvidia-docker-2-0-on-ubuntu-18-04/" target="_blank" rel="noopener" title="How to setup Docker and Nvidia-Docker 2.0 on Ubuntu 18.04">文章</a>。</p><a id="more"></a><h2 id="安装系统"><a href="#安装系统" class="headerlink" title="安装系统"></a><a href="https://www.sysgeek.cn/install-ubuntu-20-04-lts-desktop/" target="_blank" rel="noopener" title="Ubuntu 20.04 LTS 桌面版详细安装指南">安装系统</a></h2><p>我选择的是 Ubuntu 20.04。我在 MacOS 上制作引导盘，方法在<a href="https://ubuntu.com/tutorials/create-a-usb-stick-on-macos#1-overview" target="_blank" rel="noopener" title="Create a bootable USB stick on macOS">这里</a>。一路安装都很简单，有几个建议：</p><ol><li>在选择安装模式的时候选择“Minimal Installation”，不安装更新和第三方软件，一会系统安装好了以后手动更新。<br><img src="https://mmbiz.qpic.cn/mmbiz_png/OuEFiapfBFkFUQhwosjXq2DdhTRka7burQzvdibLullKy8IIuv2m5KUx7icmaxjh5G4NDZOOTEK22NnzC6m3vIq5w/0?wx_fmt=png" alt="两个确认框都取消"></li><li>在分区时选择默认分区就好了，不需要 LVM，不要选择 ZFS。</li><li>在设置用户的时候选择 “Log in automatically”。</li></ol><h2 id="系统更新"><a href="#系统更新" class="headerlink" title="系统更新"></a>系统更新</h2><p>安装好了 Ubuntu 并设置好网络以后，我们要做的第一件事就是更新系统、安装基础组件。工作站配置好了以后就不要轻易更新了，以免破坏环境。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">>>> sudo apt-get update && sudo apt-get upgrade && sudo apt-get autoremove</span><br><span class="line">>>> sudo apt install build-essential vim git curl wget make</span><br></pre></td></tr></table></figure><h2 id="设置-SSH"><a href="#设置-SSH" class="headerlink" title="设置 SSH"></a><a href="https://www.linuxbabe.com/linux-server/setup-passwordless-ssh-login" target="_blank" rel="noopener" title="2 Simple Steps to Set up Passwordless SSH Login on Ubuntu">设置 SSH</a></h2><ol><li>安装 SSH：<code>sudo apt install Openssh-server</code></li><li>启动 SSH：<code>sudo /etc/init.d/ssh start</code></li><li>开机自动启动 SSH：<code>sudo systemctl enable ssh</code></li><li>在客户端生成 SSH 秘钥<br><code>ssh-keygen</code> 或 强秘钥 <code>ssh-keygen -t rsa -b 4096</code></li><li>在客户端上传秘钥至服务器：<code>ssh-copy-id remote-user@server-ip</code></li><li>关闭 SSH 密码登录 <code>sudo vim /etc/ssh/sshd_config</code><br>找到 <code>#PasswordAuthentication yes</code> 修改为 PasswordAuthentication no<br>找到 <code>ChallengeResponseAuthentication</code> 修改为 <code>ChallengeResponseAuthentication no</code></li><li>重启 SSH 服务<br><code>sudo service ssh restart</code> 或 <code>sudo systemctl restart ssh</code></li><li>（可选）备份 SSH key<br><code>cp ~/.ssh/id_rsa* /path/to/safe/location/</code></li></ol><p>设置好 SSH 以后，我们可以在个人电脑上登录工作站。</p><h2 id="设置防火墙"><a href="#设置防火墙" class="headerlink" title="设置防火墙"></a><a href="https://www.digitalocean.com/community/tutorials/initial-server-setup-with-ubuntu-20-04" target="_blank" rel="noopener" title="Initial Server Setup with Ubuntu 20.04">设置防火墙</a></h2><p>下面来设置一下防火墙，防止有坏人来捣乱。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">>>> sudo ufw allow OpenSSH</span><br><span class="line">>>> sudo ufw allow 22 <span class="comment"># 开放端口 22</span></span><br><span class="line">>>> ufw <span class="built_in">enable</span></span><br></pre></td></tr></table></figure><h2 id="安装显卡驱动"><a href="#安装显卡驱动" class="headerlink" title="安装显卡驱动"></a><a href="https://linuxconfig.org/how-to-install-the-nvidia-drivers-on-ubuntu-20-04-focal-fossa-linux" target="_blank" rel="noopener" title="How to install the NVIDIA drivers on Ubuntu 20.04 Focal Fossa Linux">安装显卡驱动</a></h2><p>Ubuntu 提供了一个很简单的命令来安装显卡驱动。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">>>> sudo ubuntu-drivers autoinstall</span><br><span class="line">>>> sudo reboot <span class="comment"># 重启工作站</span></span><br></pre></td></tr></table></figure><h2 id="安装-cuda-11-1"><a href="#安装-cuda-11-1" class="headerlink" title="安装 cuda 11.1"></a><a href="https://zhuanlan.zhihu.com/p/78002221" target="_blank" rel="noopener" title="Ubuntu下CUDA，CUDNN和Tensorflow配置">安装 cuda 11.1</a></h2><p>这里安装最新的 CUDA Toolkit 11.1。记得安装的时候把”安装驱动程序“取消。</p><ol><li>安装 cuda<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">>>> wget https://developer.download.nvidia.com/compute/cuda/11.1.1/local_installers/cuda_11.1.1_455.32.00_linux.run</span><br><span class="line">>>> sudo sh cuda_11.1.1_455.32.00_linux.run</span><br></pre></td></tr></table></figure></li><li>将以下内容加进 /etc/profile：<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">export</span> PATH=/usr/<span class="built_in">local</span>/cuda-11.1/bin:<span class="variable">$PATH</span></span><br><span class="line"><span class="built_in">export</span> LD_LIBRARY_PATH=/usr/<span class="built_in">local</span>/cuda-11.1/lib64<span class="variable">$LD_LIBRARY_PATH</span></span><br></pre></td></tr></table></figure></li><li>重启电脑 <code>sudo reboot</code></li><li>检查cuda 版本<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">>>> nvcc -V</span><br><span class="line">nvcc: NVIDIA (R) Cuda compiler driver</span><br><span class="line">Copyright (c) 2005-2020 NVIDIA Corporation</span><br><span class="line">Built on Mon_Oct_12_20:09:46_PDT_2020</span><br><span class="line">Cuda compilation tools, release 11.1, V11.1.105</span><br><span class="line">Build cuda_11.1.TC455_06.29190527_0</span><br></pre></td></tr></table></figure></li><li>测试<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">>>> cuda-install-samples-11.1.sh ~</span><br><span class="line">Copying samples to /home/nlp/NVIDIA_CUDA-11.1_Samples now...</span><br><span class="line">Finished copying samples.</span><br><span class="line">>>> <span class="built_in">cd</span> /home/nlp/NVIDIA_CUDA-11.1_Samples/</span><br><span class="line">>>> make <span class="comment"># 耗时约 20 分钟</span></span><br><span class="line">>>> ./1_Utilities/deviceQuery/deviceQuery <span class="comment"># 如果通过测试会显示如下信息</span></span><br><span class="line">./1_Utilities/deviceQuery/deviceQuery Starting...</span><br><span class="line"></span><br><span class="line"> CUDA Device Query (Runtime API) version (CUDART static linking)</span><br><span class="line"></span><br><span class="line">Detected 2 CUDA Capable device(s)</span><br><span class="line">...</span><br><span class="line">deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.0, CUDA Runtime Version = 11.1, NumDevs = 2</span><br><span class="line">Result = PASS</span><br></pre></td></tr></table></figure></li></ol><h2 id="安装-cuDNN-8-0-4"><a href="#安装-cuDNN-8-0-4" class="headerlink" title="安装 cuDNN 8.0.4"></a><a href="https://zhuanlan.zhihu.com/p/143429249" target="_blank" rel="noopener" title="简易记录：安装GPU驱动，CUDA和cuDNN">安装 cuDNN 8.0.4</a></h2><p>安装以前要先<a href="https://developer.nvidia.com/cudnn-download-survey" target="_blank" rel="noopener" title="NVIDIA Developer Program Membership Required">注册新用户</a>。</p><ol><li>在注册后，到 <code>https://developer.nvidia.com/rdp/cudnn-download</code> 下载 <code>libcudnn8_8.0.4.30-1+cuda11.1_amd64.deb</code>，<code>libcudnn8-dev_8.0.4.30-1+cuda11.1_amd64.deb</code> 和 <code>libcudnn8-samples_8.0.4.30-1+cuda11.1_amd64.deb</code>。可以在个人电脑上下载然后 <code>scp</code> 给工作站，下同。</li><li>解包：<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">>>> sudo dpkg -i libcudnn*</span><br></pre></td></tr></table></figure></li><li>检查 cudnn：<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">>>> cp -r /usr/src/cudnn_samples_v8/ <span class="variable">$HOME</span></span><br><span class="line">>>> <span class="built_in">cd</span> <span class="variable">$HOME</span>/cudnn_samples_v8/mnistCUDNN</span><br><span class="line">>>> make clean && make</span><br><span class="line">...</span><br><span class="line">>>> ./mnistCUDNN</span><br><span class="line">...</span><br><span class="line">Test passed!</span><br></pre></td></tr></table></figure><h2 id="安装-Miniconda"><a href="#安装-Miniconda" class="headerlink" title="安装 Miniconda"></a>安装 Miniconda</h2>没必要安装 Anaconda（用不到 GUI），安装 Miniconda 就可以了。</li><li>到 <code>https://docs.conda.io/en/latest/miniconda.html#linux-installers</code> 下载 Miniconda3 Linux 64-bit。运行<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">>>> bash Miniconda3-latest-Linux-x86_64.sh</span><br><span class="line">>>> <span class="built_in">source</span> ~/.bashrc 更新 bash 环境</span><br><span class="line">>>> conda update conda</span><br></pre></td></tr></table></figure></li><li>安装虚拟环境<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">>>> conda create --name nlp python=3.8</span><br></pre></td></tr></table></figure></li></ol><h2 id="配置远程-Jupyter-lab"><a href="#配置远程-Jupyter-lab" class="headerlink" title="配置远程 Jupyter lab"></a><a href="https://blog.csdn.net/starfish55555/article/details/96788672" target="_blank" rel="noopener" title="Ubuntu与windows下配置安装jupyter-notebook以及其开机自启、后台运行与远程访问">配置远程 Jupyter lab</a><a href="https://blog.csdn.net/tuzixini/article/details/79105482" target="_blank" rel="noopener" title="服务器 配置 Jupyter notebook 远程访问 (Ubuntu 14.04)"></a><a href="https://zhuanlan.zhihu.com/p/75524289" target="_blank" rel="noopener" title="Jupyter开发环境的进阶配置（Ubuntu 18.04）"></a></h2><p>安装 Jupyter lab 后可以选择 Jupyter lab 还是 Jupyter notebook，方法是在登录的域名后面加 “/lab?” 或者 “/tree?”。</p><ol><li>生成配置文件：<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">>>> pip install jupyterlab nodejs</span><br><span class="line">>>> jupyter lab --generate-config <span class="comment"># 生成配置文件</span></span><br><span class="line">Writing default config to: /home/nlp/.jupyter/jupyter_notebook_config.py</span><br></pre></td></tr></table></figure></li><li>设置登录密码：<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 首先进入 python 命令行</span></span><br><span class="line">>>> python3 </span><br><span class="line"><span class="comment"># 在命令行下输入</span></span><br><span class="line">>>> from notebook.auth import passwd; passwd()</span><br><span class="line"><span class="comment"># 按照提示输入密码，这是 jupyter 的登陆密码</span></span><br></pre></td></tr></table></figure></li><li>设置成功会出现形如下面的哈希（hash）密码, 保存好，下面会用到<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="string">'argon2:$argon2id$v=19$m=10240,t=10,p=8$mYbUFvU1Csiwz3UGlsRwEA$q7r2mSN5RbFwjbhZCew4fg'</span></span><br></pre></td></tr></table></figure></li><li>配置 Jupyter lab：<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">>>> sudo vim /home/nlp/.jupyter/jupyter_notebook_config.py <span class="comment"># 配置 Jupyter lab</span></span><br><span class="line">c.NotebookApp.ip = <span class="string">'*'</span></span><br><span class="line">c.NotebookApp.token = <span class="string">''</span></span><br><span class="line">c.NotebookApp.password = <span class="string">'argon2:$argon2id$v=19$m=10240,t=10,p=8$mYbUFvU1Csiwz3UGlsRwEA$q7r2mSN5RbFwjbhZCew4fg'</span></span><br><span class="line">c.NotebookApp.open_browser = False</span><br><span class="line">c.NotebookApp.notebook_dir = <span class="string">'/home/nlp/Documents'</span> <span class="comment"># 设置默认根目录</span></span><br><span class="line">c.NotebookApp.allow_remote_access = True</span><br><span class="line">c.NotebookApp.port = 8889 <span class="comment"># 设置端口</span></span><br></pre></td></tr></table></figure></li><li>防火墙开放端口<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">>>> sudo ufw allow 8889</span><br></pre></td></tr></table></figure></li><li>注册虚拟环境<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">>>> conda activate nlp</span><br><span class="line">>>> conda install ipykernel notebook</span><br><span class="line">>>> python -m ipykernel install --user --name nlp --display-name <span class="string">"NLP"</span></span><br></pre></td></tr></table></figure>设置以后在 SSH 状态中输入 “jupyter lab” 后在浏览器地址栏里输入 “域名:IP” 就可以启动 Jupyter lab 了。 因为工作站很少关机，在 ssh 以后输入 <code>nohup jupyter lab &</code> 就可以让 Jupyter lab 保持在后台运行。</li></ol><h2 id="设置-Jupyter-notebook-开机自动在后台启动（可选）"><a href="#设置-Jupyter-notebook-开机自动在后台启动（可选）" class="headerlink" title="设置 Jupyter notebook 开机自动在后台启动（可选）"></a><a href="https://medium.com/@datamove/setup-jupyter-notebook-server-to-start-up-on-boot-and-use-different-conda-environments-147b091b9a5f" target="_blank" rel="noopener" title="Setup Jupyter Notebook Server to Start Up on Boot and Use Different Conda Environments.">设置 Jupyter notebook 开机自动在后台启动</a>（可选）</h2><p>这里要注意是开机自动启动的是 Jupyter notebook，设定好以后就不能启动 Jupyter lab 了。我更喜欢 Jupyter lab，所以调试好以后又删掉了。反正工作站不关机，把 Jupyter lab 挂在后台也不麻烦。</p><ol><li>在 terminal 中输入<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">>>> sudo vim /lib/systemd/system/ipython-notebook.service</span><br></pre></td></tr></table></figure></li><li>在编辑器里粘贴如下：<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">[Unit]</span><br><span class="line"> Description=IPython notebook</span><br><span class="line">[Service]</span><br><span class="line"> Type=simple</span><br><span class="line"> PIDFile=/var/run/ipython-notebook.pid</span><br><span class="line"> <span class="comment"># 环境是 Jupyter 的默认环境，可以在编辑器内更改</span></span><br><span class="line">Environment=<span class="string">"PATH=/home/*用户名*/miniconda3/envs/*kernel 环境名*/bin:/home/*用户名*/miniconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"</span></span><br><span class="line"> ExecStart=/home/*用户名*/miniconda3/envs/*kernel 环境名*/bin/jupyter-notebook --no-browser --notebook-dir=/home/nlp/Documents --NotebookApp.token=*token* --ip=0.0.0.0</span><br><span class="line"> User=*用户名*</span><br><span class="line"> Group=*用户组名*</span><br><span class="line"> WorkingDirectory=/home/*用户名* <span class="comment"># 此处可根据需要自由设定</span></span><br><span class="line">[Install]</span><br><span class="line"> WantedBy=multi-user.target</span><br></pre></td></tr></table></figure></li><li>依次输入<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">>>> sudo systemctl daemon-reload</span><br><span class="line">>>> sudo systemctl <span class="built_in">enable</span> ipython-notebook</span><br><span class="line">Created symlink /etc/systemd/system/multi-user.target.wants/jupyter.service → /lib/systemd/system/jupyter.service.</span><br><span class="line">>>> sudo systemctl start ipython-notebook</span><br></pre></td></tr></table></figure></li><li>验证 Jupyter notebook 是否设置成功，输入<code>sudo systemctl status ipython-notebook</code>，成 成功的话会输出如下类似的信息：<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">● ipython-notebook.service - IPython notebook</span><br><span class="line"> Loaded: loaded (/lib/systemd/system/ipython-notebook.service; enabled; vendor preset: enabled)</span><br><span class="line"> Active: active (running) since Mon 2020-11-02 20:53:51 EST; 7s ago</span><br><span class="line"> Main PID: 3838 (jupyter-noteboo)</span><br><span class="line"> Tasks: 1 (<span class="built_in">limit</span>: 19026)</span><br><span class="line"> Memory: 60.1M</span><br><span class="line"> CGroup: /system.slice/ipython-notebook.service</span><br><span class="line"> └─3838 /home/nlp/miniconda3/envs/notebook_env/bin/python /home/nlp/miniconda3/envs/notebook_env/bin/jupyter-notebook --no-browser --notebook-dir=/home/nlp --NotebookApp.token=argon2:<span class="variable">$argon2id</span><span class="variable">$v</span>=1></span><br><span class="line"></span><br><span class="line">Nov 02 20:53:51 WORKSTATION systemd[1]: Started IPython notebook.</span><br><span class="line">Nov 02 20:53:51 WORKSTATION jupyter-notebook[3838]: [I 20:53:51.858 NotebookApp] [nb_conda_kernels] enabled, 3 kernels found</span><br><span class="line">Nov 02 20:53:52 WORKSTATION jupyter-notebook[3838]: [I 20:53:52.036 NotebookApp] Serving notebooks from <span class="built_in">local</span> directory: /home/nlp</span><br><span class="line">Nov 02 20:53:52 WORKSTATION jupyter-notebook[3838]: [I 20:53:52.037 NotebookApp] Jupyter Notebook 6.1.4 is running at:</span><br><span class="line">Nov 02 20:53:52 WORKSTATION jupyter-notebook[3838]: [I 20:53:52.037 NotebookApp] http://WORKSTATION:8889/?token=...</span><br><span class="line">Nov 02 20:53:52 WORKSTATION jupyter-notebook[3838]: [I 20:53:52.037 NotebookApp] or http://127.0.0.1:8889/?token=...</span><br><span class="line">Nov 02 20:53:52 WORKSTATION jupyter-notebook[3838]: [I 20:53:52.037 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).</span><br></pre></td></tr></table></figure></li></ol><h2 id="安装-Python-包、配置-Jupyter、Vim、IDE"><a href="#安装-Python-包、配置-Jupyter、Vim、IDE" class="headerlink" title="安装 Python 包、配置 Jupyter、Vim、IDE"></a>安装 Python 包、配置 Jupyter、Vim、IDE</h2><p>现在工作站基本上配置好啦！之后安装各种包，配置 Jupyter、Vim 和各种 IDE 就看各位的喜好啦。我用的 IDE 是 VS code（不想搞破解版 PyCharm），安装 Remote - SSH 就可以远程炼丹啦。</p><h2 id="Bonus-Benchmark"><a href="#Bonus-Benchmark" class="headerlink" title="Bonus: Benchmark"></a>Bonus: Benchmark</h2><p>工作站配置好了，来简单做一下 benchmark。作为对照的是 Google Colab，Colab 上的 GPU 是随机分配的，这次分配到的是 Tesla T4。脚本<a href="https://github.com/vincent507cpu/nlp_projects/blob/main/text%20classification/03%20transformers%20Colab%20GPU.ipynb" target="_blank" rel="noopener" title="Colab GPU">在此</a>。每个 epoch 用时大约 40 分钟。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">+-----------------------------------------------------------------------------+</span><br><span class="line">| NVIDIA-SMI 455.32.00 Driver Version: 418.67 CUDA Version: 10.1 |</span><br><span class="line">|-------------------------------+----------------------+----------------------+</span><br><span class="line">| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |</span><br><span class="line">| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |</span><br><span class="line">| | | MIG M. |</span><br><span class="line">|===============================+======================+======================|</span><br><span class="line">| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |</span><br><span class="line">| N/A 45C P8 11W / 70W | 0MiB / 15079MiB | 0% Default |</span><br><span class="line">| | | ERR! |</span><br><span class="line">+-------------------------------+----------------------+----------------------+</span><br></pre></td></tr></table></figure><p>然后分别使用一张 2060 和两张 2060 显卡在工作站上测试，脚本<a href="https://github.com/vincent507cpu/nlp_projects/blob/main/text%20classification/05%20transformers%20single%20GPU.ipynb" target="_blank" rel="noopener" title="single GPU">在此</a>。batch size 设置为 64 的话会爆显存，所以设置成了 32。单卡在运行时的显卡状态为</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">+-----------------------------------------------------------------------------+</span><br><span class="line">| NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |</span><br><span class="line">|-------------------------------+----------------------+----------------------+</span><br><span class="line">| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |</span><br><span class="line">| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |</span><br><span class="line">| | | MIG M. |</span><br><span class="line">|===============================+======================+======================|</span><br><span class="line">| 0 GeForce RTX 2060 Off | 00000000:08:00.0 On | N/A |</span><br><span class="line">| 90% 83C P2 163W / 170W | 5077MiB / 5926MiB | 96% Default |</span><br><span class="line">| | | N/A |</span><br><span class="line">+-------------------------------+----------------------+----------------------+</span><br><span class="line">| 1 GeForce RTX 2060 Off | 00000000:09:00.0 Off | N/A |</span><br><span class="line">| 46% 44C P8 7W / 170W | 9MiB / 5934MiB | 0% Default |</span><br><span class="line">| | | N/A |</span><br><span class="line">+-------------------------------+----------------------+----------------------+</span><br></pre></td></tr></table></figure><p>每个 epoch 用时大约 27 分钟。双卡在运行时的状态为</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">+-----------------------------------------------------------------------------+</span><br><span class="line">| NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |</span><br><span class="line">|-------------------------------+----------------------+----------------------+</span><br><span class="line">| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |</span><br><span class="line">| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |</span><br><span class="line">| | | MIG M. |</span><br><span class="line">|===============================+======================+======================|</span><br><span class="line">| 0 GeForce RTX 2060 Off | 00000000:08:00.0 On | N/A |</span><br><span class="line">| 92% 84C P2 162W / 170W | 5702MiB / 5926MiB | 93% Default |</span><br><span class="line">| | | N/A |</span><br><span class="line">+-------------------------------+----------------------+----------------------+</span><br><span class="line">| 1 GeForce RTX 2060 Off | 00000000:09:00.0 Off | N/A |</span><br><span class="line">| 88% 82C P2 161W / 170W | 3148MiB / 5934MiB | 34% Default |</span><br><span class="line">| | | N/A |</span><br><span class="line">+-------------------------------+----------------------+----------------------+</span><br></pre></td></tr></table></figure><p>每个 epoch 用时大约 17 分钟，脚本<a href="https://github.com/vincent507cpu/nlp_projects/blob/main/text%20classification/06%20transformers%20multiple%20GPU.ipynb" target="_blank" rel="noopener" title="multiple GPU">在此</a>。可以看到两张显卡的显存的利用不同（5702MiB vs 3146MiB），这种不平衡已经超过本文的讨论范围了。</p><hr><p>可以看到，Colab 唯一的优势在于显存比较大，算力被入门级的 2060 完爆。我的这台工作站待机时两个 GPU 功耗为 17W，24 小时待机也没有负担。根据谣言，3060 Ti 的性能与 2080 Super 相当，那么根据 Tom’s Hardware 的 <a href="https://www.tomshardware.com/reviews/gpu-hierarchy,4388.html" target="_blank" rel="noopener" title="GPU Benchmarks and Hierarchy: Graphics Cards Ranked">GPU 性能排行榜</a>，3060 Ti 相比 2060 预计有 50% 的性能提升，使用一张 3060 Ti 时每个 epoch 用时会在 20 分钟左右，相比 Colab 时间减少了一半。一个字，香！</p>]]></content>
<summary type="html">
<p>工作站的配置有两种方式：</p>
<ul>
<li>传统方法；</li>
<li>Docker。</li>
</ul>
<p>理论上 Docker 更好一点，因为 <code>nvidia-docker2</code> 是一个已经配置好的环境，不需要手动安装 CUDA 和 cuDNN，而且不需要了可以删除，更新也很方便。然而本文中将采用传统方法对工作站进行配置。使用 Docker 可以参考这篇<a href="https://cnvrg.io/how-to-setup-docker-and-nvidia-docker-2-0-on-ubuntu-18-04/" target="_blank" rel="noopener" title="How to setup Docker and Nvidia-Docker 2.0 on Ubuntu 18.04">文章</a>。</p>
</summary>
<category term="深度学习工作站" scheme="https://vincent507cpu.github.io/tags/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%B7%A5%E4%BD%9C%E7%AB%99/"/>
</entry>
<entry>
<title>[工作站] 我的第一台个人深度学习工作站之硬件篇</title>
<link href="https://vincent507cpu.github.io/2020/11/01/%E5%B7%A5%E4%BD%9C%E7%AB%99-%E6%88%91%E7%9A%84%E7%AC%AC%E4%B8%80%E5%8F%B0%E4%B8%AA%E4%BA%BA%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%B7%A5%E4%BD%9C%E7%AB%99%E4%B9%8B%E7%A1%AC%E4%BB%B6%E7%AF%87/"/>
<id>https://vincent507cpu.github.io/2020/11/01/%E5%B7%A5%E4%BD%9C%E7%AB%99-%E6%88%91%E7%9A%84%E7%AC%AC%E4%B8%80%E5%8F%B0%E4%B8%AA%E4%BA%BA%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%B7%A5%E4%BD%9C%E7%AB%99%E4%B9%8B%E7%A1%AC%E4%BB%B6%E7%AF%87/</id>
<published>2020-11-01T21:32:01.000Z</published>
<updated>2020-11-02T02:59:05.373Z</updated>
<content type="html"><![CDATA[<h1 id="工作站配置"><a href="#工作站配置" class="headerlink" title="工作站配置"></a>工作站配置</h1><p>我的计划是使用入门级显卡配一台双 GPU 工作站使用两年，预留两年之内升级一次的空间，在保证性能和可扩展性的前提下，够用就好。经过若干次修改，最终的配置为：</p><a id="more"></a><ul><li>CPU：Ryzen 3 3100</li></ul><p>可能很多人觉得现在都 2020 年了，还用 4 核 CPU 太落伍。其实我想买 3600，但是没货，不得已买了 3100（不想多花钱买 3600X）。在深度学习中 4 核 CPU 搭配 2 块 GPU 已经足够了。不过在考虑明年升级成 5600。</p><ul><li>散热器：九州风神 GAMMAXX 200T</li></ul><p>虽然 CPU 的原装散热器已经可以满足需要，我买它的原因是有折扣，就算为明年换 5600 提前买好了。</p><ul><li>主板：Asus Prime X570-Pro</li></ul><p>随便一块 X570 主板基本都有两条 PCI-E 插槽（ITX 小主板除外），不过通常是 16-4 通道的配置。虽然 PCI-E 4.0 下 x4 应该不是瓶颈，用起来心里还是有点不舒服。所以选择了支持 SLI 的主板中最便宜的一块。</p><ul><li>内存：OLOy DDR4 RAM 16GB (2x8GB) 3200 MHz </li></ul><p>OLOy 是一个新牌子，买它完全是因为便宜。<a href="https://www.tomshardware.com/features/oloy-ram-should-you-buy" target="_blank" rel="noopener" title="Should You Buy OLOy RAM?">Tom Hardware 认为它的产品质量还可以</a>，先买来试试。</p><ul><li>显卡：两块 EVGA 06G-P4-2066-KR GeForce RTX 2060 KO Gaming</li></ul><p>我的目标是 3060 Ti，但是还没有发布，所以先用两块 2060 当亮机卡。</p><ul><li>硬盘：Crucial P1 500GB</li></ul><p>这块硬盘的性能很一般，买它因为折扣很大（超过一半），而且再不济也是一块 NVMe 硬盘，性能与顶级 PCI-E 3.0 NVMe 硬盘最多差 10%，完全够用。</p><p>还有一块差不多 10 年前买的 3T 硬盘，里面有很多以前的文件，现在读不出来了，把外壳拿掉直接插电脑上看看能不能读出来。</p><ul><li>电源：Thermaltake ToughPower 750W 80 Plus Gold Semi Modular Power Supply</li></ul><p>买它的原因也是有折扣，而且 750W 带 5600 + 两块 3060 Ti 应该也没问题。</p><p>显示器、键盘、鼠标用现成的，机箱买的最便宜的，另外买了一个 150M 的 WiFi 接收器。这就是全部配件了：<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkGDIRyOlnx8uLyibZtHgZricicbxcnd3rovL3BCribuely2RckeeQzvoibo2xCMySB5rcKUzvv8HCeE2UQ/0?wx_fmt=jpeg" alt=""></p><p>关于配件的选择请参考：</p><ul><li><a href="https://mp.weixin.qq.com/s?__biz=Mzg3OTIwODUzMQ==&mid=2247485308&idx=1&sn=fc687999f159fed9614688e55d948c58&chksm=cf06b640f8713f56ff63c20acebd8c846a517be81a8fdbfae21062bc96819046630238dd0381&token=1239929154&lang=zh_CN#rd" target="_blank" rel="noopener">《2020 年 10 月的多 GPU 深度学习工作站配置指南》</a></li><li><a href="https://mp.weixin.qq.com/s?__biz=Mzg3OTIwODUzMQ==&mid=2247485265&idx=1&sn=9da954f0dd5c00884a6b3b92d7f2349a&chksm=cf06b66df8713f7b7ff0200a8e3b7ab324f3e6a72a0355cd5f9d0479cf802e4bb5df29d21ebc&token=1239929154&lang=zh_CN#rd" target="_blank" rel="noopener">《2020 年 10 月的单 GPU 深度学习工作站配置指南》</a><h1 id="升级计划"><a href="#升级计划" class="headerlink" title="升级计划"></a>升级计划</h1></li></ul><h2 id="第一次升级：2021-年上半年"><a href="#第一次升级：2021-年上半年" class="headerlink" title="第一次升级：2021 年上半年"></a>第一次升级：2021 年上半年</h2><p>把 CPU 升级到 5600，如果 3100 用的还行，就不升级了。</p><p>传说中 NVIDIA 计划在 12 月份发布 16GB 显存的 3070 Super，不过现在有消息说计划已经取消。那就把显卡换成两块 8GB 显存的 30 系列的最低款（目前可能是 3060 Ti）。内存升级到 32GB。</p><h2 id="第二次升级：2022-年下半年"><a href="#第二次升级：2022-年下半年" class="headerlink" title="第二次升级：2022 年下半年"></a>第二次升级：2022 年下半年</h2><p>如果两年以后进步很大，这台电脑不能满足需要了，就再升级一次。这应该是一台全新的工作站，目标是高性能，所有配件都要换，应该可以使用五年。</p><ul><li>CPU：Ryzen 9 5950 的下一代</li><li>内存：64 ~ 128 GB</li><li>显卡：2 块带水冷的 3090 显卡</li><li>硬盘：1 块 2T SSD 硬盘</li><li>电源：1200W 电源</li><li>机箱：具有风道设计的机箱</li></ul><h1 id="装机过程"><a href="#装机过程" class="headerlink" title="装机过程"></a>装机过程</h1><p>首先要了解需要连接哪些线，可以先看说明书。这台工作站没有任何灯效，所以少了一点麻烦。<br><img src="https://mmbiz.qpic.cn/mmbiz_png/OuEFiapfBFkGDIRyOlnx8uLyibZtHgZriciczCE9DMrTFk0RX1LTnvRy8LEFoyLvhbibaSRG7J4zaia8JicTt9Htcbjqg/0?wx_fmt=png" alt=""><br>主板上需要连接的线有：</p><ul><li>24 pin 主板供电（图中右侧 1）</li><li>8 pin CPU 供电（图中上侧 1）</li><li>4 pin CPU 风扇供电（图中上侧 3）</li><li>4 pin 机箱风扇供电（图中中部左侧 3）</li><li>机箱前面板 USB 3 接口（图中中部右侧 9）</li><li>机箱控制面板排针（图中下方右侧 14）</li><li>机箱前面板 USB 2.0 接口（图中下方中部 18）</li><li>机箱音频连接排针（虽然我不用工作站的声音，还是接上了。图中下方左侧 21）</li></ul><p>还有两张显卡需要两条 8 pin 供电线，与电源相连。这些插口都有防呆设计，插反了是插不进去的。</p><h2 id="CPU"><a href="#CPU" class="headerlink" title="CPU"></a>CPU</h2><p>先把拨杆拉开。主板的 CPU 插槽和 CPU 的一角上都有一个三角标记，对准了把 CPU 放下去，拉下拨杆。拉下拨杆的时候有一点阻力，稍微用力一点就可以了。<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkGDIRyOlnx8uLyibZtHgZricicjib6bJRUiajNISkZQTwvsNN3SRRic6cRmibcZusynv4HqqSIcd7fQEyNAg/0?wx_fmt=jpeg" alt=""><br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkGDIRyOlnx8uLyibZtHgZricicEIqyHQOzKkNib5fBfq4CBOccKQ5Lh0ulZNqiceUBc2YORr7ibx82pkOwQ/0?wx_fmt=jpeg" alt=""></p><h2 id="内存"><a href="#内存" class="headerlink" title="内存"></a>内存</h2><p>现在的电脑都是双通道内存设计，假如有 4 个内存插槽需要插两根内存，要以 1-3 或 2-4 这样插。扳开卡扣，看准了方向把内存插紧，听到“咔”的一声就插好了。插反了是插不进去的。<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkGDIRyOlnx8uLyibZtHgZricicSyXS2WrgXD807y9zSIeEoBMAzzuJ4WZjQic8hIQwHvNkh8o2l7kfHWQ/0?wx_fmt=jpeg" alt=""></p><h2 id="SSD-硬盘"><a href="#SSD-硬盘" class="headerlink" title="SSD 硬盘"></a>SSD 硬盘</h2><p>有的主板上有一些 SSD 插槽有扇热片，需要先把扇热片取下来。SSD 硬盘一般是 2280 尺寸，对应离插槽第 3 远的螺丝。先要拧一颗加高螺丝（用手就可以拧），然后将 SSD 硬盘插进去，拧上固定螺丝。扇热片上一般有散热胶贴纸，先把贴纸撕下来，再把扇热片放回去。<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkGDIRyOlnx8uLyibZtHgZricic2DP0jVKBE73ib6uv5KK1aRJVQKC5E1okzOiaZjUia23E1KdcfmR2NWoyQ/0?wx_fmt=jpeg" alt=""></p><h2 id="CPU-散热器"><a href="#CPU-散热器" class="headerlink" title="CPU 散热器"></a>CPU 散热器</h2><p>CPU 插槽的两边有两个卡扣，将 CPU 散热器卡在上面就可以了，需要用点力气。需要插第一根供电线了。<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkGDIRyOlnx8uLyibZtHgZricicC8iaBhatAibTgWvib0PcHHoWABWtWBrZWcmzvcjLnjc9Sib0G52cdSwHqg/0?wx_fmt=jpeg" alt=""></p><h2 id="把电源放进机箱里并固定"><a href="#把电源放进机箱里并固定" class="headerlink" title="把电源放进机箱里并固定"></a>把电源放进机箱里并固定</h2><p>在把电源放进机箱前可以把要用的线先接上。电源分全模组、半模组和非模组三种。非模组电源是所有线事先连接在电源上，全模组电源是所有线都根据需要连接，半模组电源介于二者之间。我的电源是半模组电源，需要再连接两条 8 pin 供电线。<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkGDIRyOlnx8uLyibZtHgZriciciabLYYOW9EGv7l3zLW1dzSHWHKXKYaliciaOzqKo7zDnLeD5UddsvKbtA/0?wx_fmt=jpeg" alt=""><br>现在可以把电源放进机箱了。<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkGDIRyOlnx8uLyibZtHgZricicNu3n87EzBWzBEv7VYib4POIZ6ia45lYErb0qB7aISAW8qGfg1JvYl6hg/0?wx_fmt=jpeg" alt=""></p><h2 id="把主板放进机箱并固定"><a href="#把主板放进机箱并固定" class="headerlink" title="把主板放进机箱并固定"></a>把主板放进机箱并固定</h2><p>把主板放进机箱前需要先看看那些螺丝孔需要基座螺丝，需要的话要先拧上。我的这张主板一共需要固定 9 颗螺丝，上中下各 3 颗。<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkGDIRyOlnx8uLyibZtHgZricic3sU0N2zia4l0AIv1kWonDdwFfU1Y1HBSoE0owQxkI0ZUbibnK3gAqpMQ/0?wx_fmt=jpeg" alt=""></p><h2 id="连接所有信号线和供电线"><a href="#连接所有信号线和供电线" class="headerlink" title="连接所有信号线和供电线"></a>连接所有信号线和供电线</h2><p>这应该是整个过程里最麻烦的一步了，然而说明书和接头上都有标记，耐心一点一根根插上就行了。</p><h2 id="插上显卡并连接供电"><a href="#插上显卡并连接供电" class="headerlink" title="插上显卡并连接供电"></a>插上显卡并连接供电</h2><p>首先把机箱上对应位置的挡板拿下去，把 PCI-E 插槽右侧的卡扣扳下去，把显卡插上听见“咔”的一声就行了。然后在显卡上插上对应数量的供电线。<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkGDIRyOlnx8uLyibZtHgZricicfrAbfbmeSpK2JDRogxttZ5za1Vy8Q7hUVCBFKR9WP8YvGHzOadP8gQ/0?wx_fmt=jpeg" alt=""></p><h2 id="理线"><a href="#理线" class="headerlink" title="理线"></a>理线</h2><p>这一步不是必须的，但是规整的内部空间有利于风道的畅通。一般电源和机箱会送几根 zip tie 或者尼龙扣，用它们把线捆在一起就可以了。正面的理线在上图里有，下面是背部的理线：<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkGDIRyOlnx8uLyibZtHgZricicTTbassaXdRFvoyVYaAKFlopmhdCsOClhPa1rYiaiaO0oo6UIKke8WcLA/0?wx_fmt=jpeg" alt=""><br>至此电脑就组装完了，很简单吧？在合上机箱盖之前可以先把电脑点亮测试一下：<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkHZxA0uGf9GCaiaFIDSmKk6y9mWic9JiawV9QSmK2PaLzFI6dictjicdal9O7I6VktremGYt3pykpAgqsg/0?wx_fmt=jpeg" alt=""><br>这绿油油的光好像魔兽世界西瘟疫之地上的绿光。如果连接到显示器上，应该可以看到开机自检（这里显示的 BIOS）：<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkGDIRyOlnx8uLyibZtHgZricicicAt8umrJTay2EXoib3Ficf2cMMvBehsLGKNsRr6icSKbTxib78pLs7Kxdw/0?wx_fmt=jpeg" alt=""><br>至此工作站就组装完毕了，下一篇来谈谈环境设置。</p>]]></content>
<summary type="html">
<h1 id="工作站配置"><a href="#工作站配置" class="headerlink" title="工作站配置"></a>工作站配置</h1><p>我的计划是使用入门级显卡配一台双 GPU 工作站使用两年，预留两年之内升级一次的空间，在保证性能和可扩展性的前提下，够用就好。经过若干次修改，最终的配置为：</p>
</summary>
<category term="深度学习工作站" scheme="https://vincent507cpu.github.io/tags/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%B7%A5%E4%BD%9C%E7%AB%99/"/>
</entry>
<entry>
<title>[工作站] 2020 年 10 月的多 GPU 深度学习工作站配置指南</title>
<link href="https://vincent507cpu.github.io/2020/10/10/%E5%B7%A5%E4%BD%9C%E7%AB%99-2020-%E5%B9%B4-10-%E6%9C%88%E7%9A%84%E5%A4%9A-GPU-%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%B7%A5%E4%BD%9C%E7%AB%99%E9%85%8D%E7%BD%AE%E6%8C%87%E5%8D%97/"/>
<id>https://vincent507cpu.github.io/2020/10/10/%E5%B7%A5%E4%BD%9C%E7%AB%99-2020-%E5%B9%B4-10-%E6%9C%88%E7%9A%84%E5%A4%9A-GPU-%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%B7%A5%E4%BD%9C%E7%AB%99%E9%85%8D%E7%BD%AE%E6%8C%87%E5%8D%97/</id>
<published>2020-10-11T01:58:39.000Z</published>
<updated>2020-10-25T00:42:15.938Z</updated>
<content type="html"><![CDATA[<p>本文接上一篇<a href="https://mp.weixin.qq.com/s?__biz=Mzg3OTIwODUzMQ==&mid=2247485265&idx=1&sn=9da954f0dd5c00884a6b3b92d7f2349a&chksm=cf06b66df8713f7b7ff0200a8e3b7ab324f3e6a72a0355cd5f9d0479cf802e4bb5df29d21ebc&token=1032059941&lang=zh_CN#rd" target="_blank" rel="noopener">《2020 年 10 月的单 GPU 深度学习工作站配置指南》</a>，探讨多 GPU 工作站的搭建。很多在单 GPU 工作站中不甚重要的因素在多 GPU 工作站中变得举足轻重。</p><a id="more"></a><p>本文主要参考了以下文章：</p><ul><li><a href="https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning/" target="_blank" rel="noopener" title="Which GPU(s) to Get for Deep Learning: My Experience and Advice for Using GPUs in Deep Learning">Which GPU(s) to Get for Deep Learning: My Experience and Advice for Using GPUs in Deep Learning</a></li><li><a href="https://timdettmers.com/2018/12/16/deep-learning-hardware-guide/" target="_blank" rel="noopener" title="A Full Hardware Guide to Deep Learning">A Full Hardware Guide to Deep Learning</a></li><li><a href="https://lambdalabs.com/blog/deep-learning-hardware-deep-dive-rtx-30xx/#blower-gpus" target="_blank" rel="noopener" title="Deep Learning Hardware Deep Dive – RTX 3090, RTX 3080, and RTX 3070">Deep Learning Hardware Deep Dive – RTX 3090, RTX 3080, and RTX 3070</a></li><li><a href="https://www.howtogeek.com/365215/what’s-the-difference-between-a-blower-and-an-open-air-gpu-cooler/" target="_blank" rel="noopener" title="What’s the Difference Between a Blower and an Open-Air GPU Cooler?">What’s the Difference Between a Blower and an Open-Air GPU Cooler?</a></li></ul><p>搭建多 GPU 工作站的要点是避免显卡过热与电源过载，其它很多方面与单 GPU 工作站的原则相似，没有提到的方面（包括显卡的选择）请参考<a href="https://mp.weixin.qq.com/s?__biz=Mzg3OTIwODUzMQ==&mid=2247485265&idx=1&sn=9da954f0dd5c00884a6b3b92d7f2349a&chksm=cf06b66df8713f7b7ff0200a8e3b7ab324f3e6a72a0355cd5f9d0479cf802e4bb5df29d21ebc&token=1032059941&lang=zh_CN#rd" target="_blank" rel="noopener">《2020 年 10 月的单 GPU 深度学习工作站配置指南》</a>。</p><h1 id="双-GPU-工作站"><a href="#双-GPU-工作站" class="headerlink" title="双 GPU 工作站"></a>双 GPU 工作站</h1><h2 id="PCI-E-带宽"><a href="#PCI-E-带宽" class="headerlink" title="PCI-E 带宽"></a>PCI-E 带宽</h2><p>随着 GPU 的增加，模型训练的并行程度和 GPU 之间的数据传输增加，PCI-E 带宽变得越来越重要。然而对于双 GPU 工作站来说，PCI-E 带宽的重要性仍然有限。已经有人对 PCI-E 3.0 下 x16 和 x8 通道进行过测试<a href="https://www.pugetsystems.com/labs/hpc/PCIe-X16-vs-X8-with-4-x-Titan-V-GPUs-for-Machine-Learning-1167/" target="_blank" rel="noopener" title="PCIe X16 vs X8 with 4 x Titan V GPUs for Machine Learning"></a>，结论是影响非常小。那么在 x4 甚至 x2 或 x1 时带宽对深度学习有影响吗？目前还不清楚。</p><p>一个现实是 CPU 拥有的 PCI-E 通道是有限的：</p><table><thead><tr><th>CPU</th><th>支持 PCI-E 等级</th><th>通道数</th></tr></thead><tbody><tr><td>Ryzen 3000/5000</td><td>4.0</td><td>24</td></tr><tr><td>Core</td><td>3.0</td><td>20</td></tr></tbody></table><p>而有限的通道中至少要给 NVME 存储器分配 4~8 个通道。既然 x8 通道对深度学习没什么影响，双 GPU 完全可以使用双 x8 通道。这里支持 PCI-E 4.0 的优势显示出来了，一个 x8 PCI-E 4.0 通道相当于一个 x16 PCI-E 3.0 通道（30 系列显卡才支持 PCI-E 4.0）。双 x8 PCI-E 通道并联被 NVIDIA 称为 SLI 技术，高端芯片组 X570 和 Z490 都支持 SLI，所以在买主板的时候留意是否支持 SLI 就可以了。根据我的经验，只要主板上的两个 PCI-E 插槽都有金属包装，很可能就支持 SLI：</p><p><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkHD2fLPmuYNV1VI4qqLibvOuFbMuB1RCg1GmLia97uvRib5E8oyiafyBcmn3aEDqax0VEpNhWtia1AzRmg/0?wx_fmt=jpeg" alt="GIGABYTE X570 AORUS PRO：请注意上面两个 PCI-E x16 插槽有金属包装"></p><p>支持 SLI 的主板上如果还有第三个 PCI-E x16 插槽，这个插槽的通道要么走主板芯片要么与第二个插槽平分通道。比如上图，如果在最上面的两个插槽的任意一个中插一张卡，则为 x16 通道；在上面两个插槽插两张卡，则为 8-8 通道；三个插槽都插卡，则为 8-8-4 或 8-4-4 通道。</p><p>X570 主板中支持双路 x8 PCI-E 的型号有：</p><ul><li>ASRock X570 Phantom Gaming X</li><li>ASRock X570 Creator</li><li>ASRock X570 TAICHI</li><li>ASUS PRIME X570-PRO</li><li>ASUS AMD AM4 ROG Crosshair VIII Hero</li><li>ASUS ROG Strix X570-E Gaming</li><li>ASUS ROG Strix X570-F Gaming</li><li>GIGABYTE X570 AORUS PRO</li><li>GIGABYTE X570 AORUS ULTRA</li><li>GIGABYTE X570 AORUS MASTER</li><li>GIGABYTE X570 AORUS XTREME</li><li>MSI MEG X570 ACE Gaming</li><li>MSI MEG X570 UNIFY</li><li>MSI MEG X570 GODLIKE</li></ul><p>MSI MEG X570 GODLIKE 有 4 个 x16 PCI-E 插槽，前三个可以以 8-4-4 通道数连接；第四个 PCI-E 插槽走主板芯片以 4 条通道连接（8-4-4-4）。</p><p><img src="https://mmbiz.qpic.cn/mmbiz_png/OuEFiapfBFkHu2GuyEJXjiaRyiaVPbc4m1nV5AiaVW1QqRkLgBlpiaU9SUZukK8IvzqiaOL0rdpJkwnDraCJeGcASITA/0?wx_fmt=png" alt="MSI MEG X570 GODLIKE"></p><p>Z490 主板中支持双路 x8 PCI-E 的型号有：</p><ul><li>ASRock Z490 Taichi</li><li>ASUS ProART Z490-CREATOR</li><li>ASUS ROG STRIX Z490-E GAMING</li><li>ASUS ROG MAXIMUS XII APEX</li><li>ASUS ROG MAXIMUS XII FORMULA</li><li>GIGABYTE Z490 VISION</li><li>GIGABYTE Z490 AORUS PRO AX</li><li>GIGABYTE Z490 AORUS ULTRA</li><li>GIGABYTE Z490 AORUS MASTER</li><li>GIGABYTE Z490 AORUS ULTRA</li><li>MSI MPG Z490 GAMING CARBON</li><li>MSI MEG Z490 UNIFY</li><li>MSI MEG Z490 ACE</li></ul><p>X570 和 Z490 芯片组是最高端的芯片组，比 B550 和 B460 贵一些；支持 SLI 的功能算是进阶设计，价格要更贵一些。</p><h2 id="CPU、内存、电源的选择"><a href="#CPU、内存、电源的选择" class="headerlink" title="CPU、内存、电源的选择"></a>CPU、内存、电源的选择</h2><ul><li>理论上 4 核 CPU 足够，如果有很多预处理任务也可以买 6 核的 3600 和 10400F 或者 8 核的 3700x 和 10700F，再多就没有必要。</li><li>内存的大小看实际需求和 pipeline 设计，要么不小于<strong>单卡显存 + 6~8G</strong>，要么不小于<strong>显存之和 + 6~8G</strong>。</li><li>如果使用 4 核 CPU 配两张 3070 显卡，可选 750W 或 850W 电源；如果使用 6 核 CPU 配两张 3080/3090 显卡，至少要使用 1000W 电源。</li></ul><h2 id="散热"><a href="#散热" class="headerlink" title="散热"></a>散热</h2><p>如果安装两块 3070，发热与两块 2080 Ti 差不多，散热应该不是大问题；如果安装两块 3080 或 3090，请参考下面的散热部分。</p><h1 id="三-GPU-工作站"><a href="#三-GPU-工作站" class="headerlink" title="三 GPU 工作站"></a>三 GPU 工作站</h1><h2 id="PCI-E-带宽-1"><a href="#PCI-E-带宽-1" class="headerlink" title="PCI-E 带宽"></a>PCI-E 带宽</h2><p>如果希望三张卡都有至少 x8 带宽，Core 和 Ryzen 就不能满足了，必须是 Core X-Series，Xeon，Threadripper 或者 EPYC。我对 Xeon 和 EPYC 完全不了解，此处略。</p><table><thead><tr><th>CPU</th><th>支持 PCI-E 等级</th><th>通道数</th></tr></thead><tbody><tr><td>Threadripper</td><td>4.0</td><td>64</td></tr><tr><td>10 代 Core X-Series</td><td>3.0</td><td>48</td></tr></tbody></table><p>若主板上有三个 PCI-E 插槽，Intel X299 和 AMD sTRX40 主板都支持 16-8-16 分配；若有第四个插槽，sTRX40 可以支持 16-8-16-8 分配，而 X299 支持 8-8-8-8 分配。此处 AMD 的优势又体现出来了，不要说 Threadripper 支持更多的 PCI-E 通道，而且 PCI-E 4.0 x8 已经相当于全速 PCI-E 3.0 x16。Threadripper 唯二的缺点是贵和功耗大（然而未必比 Core X-Series 的满载功耗更大）。</p><p>ASRock TRX40 TAICHI 主板支持 16-16-16 通道分配，是 Threadripper 的最佳搭配。</p><p><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkHD2fLPmuYNV1VI4qqLibvOuia5DwicJDdticKcaBkz2rbU04ApTqn5ntJxibSWX5Z0Mbem5yicNSGXTtcw/0?wx_fmt=jpeg" alt=""></p><h2 id="供电"><a href="#供电" class="headerlink" title="供电"></a>供电</h2><p>常见的 CPU 与 GPU 的热设计功率（TDP）为：</p><table><thead><tr><th>CPU</th><th>TDP</th></tr></thead><tbody><tr><td>Threadripper</td><td>280W</td></tr><tr><td>Core X-Series</td><td>165W</td></tr></tbody></table><p>新 30 系列 GPU 的热设计功率为：</p><table><thead><tr><th>GPU</th><th>TDP</th></tr></thead><tbody><tr><td>RTX 3090</td><td>350W</td></tr><tr><td>RTX 3080</td><td>320W</td></tr><tr><td>RTX 3070</td><td>220W</td></tr></tbody></table><p>如果使用 Core 10920X 搭配三块 3070，推荐 1000W 电源；其它搭配推荐 1500W 电源。</p><h2 id="散热-1"><a href="#散热-1" class="headerlink" title="散热"></a>散热</h2><p>GPU 到了三块，散热开始需要重视，不然显卡会因为过热自动降频。显卡的散热方式有风冷和水冷两种，风冷又分涡轮式散热（blower）和开放式两种（open-air）两种。</p><ul><li>开放式散热：由风扇吸入冷空气，冷空气在散热片上进行热交换，热空气在 GPU 的周围排出。<br><img src="https://mmbiz.qpic.cn/mmbiz_png/OuEFiapfBFkHu2GuyEJXjiaRyiaVPbc4m1njCgJuv2dI8kKZiapLjsJuriaxfGcChwGcYlthiaribgicrmO7ZoXNLuIvibg/0?wx_fmt=png" alt=""></li><li>涡轮式散热：整个 PCB 板被包裹起来，冷空气被风扇吸入后在散热片上进行热交换后在 GPU 后挡板处排出。<br><img src="https://mmbiz.qpic.cn/mmbiz_png/OuEFiapfBFkHu2GuyEJXjiaRyiaVPbc4m1nwK6j7GntzNUHFtVlibfvvicERAaROXGFyKXgrMwHu69rCica4DiapWEquQ/0?wx_fmt=png" alt=""></li><li>水冷散热：冷水被水泵抽到芯片上吸收芯片的热量，热水随后被抽到散热片与冷空气进行热交换。<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkHcWd8F0HgaTHaTkAP1opu0jlexAribzFEzmGCI0xtrN22pY5B5RpHrVFkVHrQB2p0OedGq5o9CqNg/0?wx_fmt=jpeg" alt=""></li></ul><p>使用开放式散热的显卡会面临热空气被其它显卡吸收的问题，会降低散热的效果，极端情况下会造成显卡过热自动降频，从而降低性能。如果显卡之间有超过 1 个 PCI-E 空位，则基本不会存在散热的问题，但是这样由于空间的限制可能仅可以使用双卡；对于三卡工作站而言，涡轮式散热显卡或水冷散热显卡是必需的，然而是否可行仍需实践。</p><h2 id="风道与机箱的选择"><a href="#风道与机箱的选择" class="headerlink" title="风道与机箱的选择"></a>风道与机箱的选择</h2><p>当使用了 3 块以上的 GPU 以后，机箱的风道变得很重要，否则热空气会在机箱内积累，一样会造成显卡过热。一款合适的深度学习服务器机箱应该有充足的内部空间和足够多放风扇的位置。我推荐两款机箱：</p><ul><li>Thermaltake Core X71<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkHu2GuyEJXjiaRyiaVPbc4m1nxzP0vIGpX1uvpa7hfIWPCst1McuSibTkRSXWIWM8ROCeXVcVQXSiaKBg/0?wx_fmt=jpeg" alt=""></li></ul><p>这个机箱的优点是可以装下足够多的风扇（上面 3 个，前面 2 个，下面 3 个，后面 1 个），非常适合多个水冷设备。</p><ul><li>Corsair Carbide Series Air 540<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkHu2GuyEJXjiaRyiaVPbc4m1nG5sYicnklDFYxRibBMbo5ucYaQR1wX0kicS2MQkxuTmpKAXLeBnIR1PNw/0?wx_fmt=jpeg" alt=""></li></ul><p>这个机箱的优点是内部空间非常充足，可以安装风扇的位置也不少（上面 3 个，前面 2 个，后面 1 个）。</p><h2 id="显卡选择"><a href="#显卡选择" class="headerlink" title="显卡选择"></a>显卡选择</h2><p>如果显卡之间有足够的空间，那么可以使用开放式散热显卡；3 块以上显卡空间有限，需要使用涡轮式散热显卡或水冷显卡。现在各个厂商只发布了开放式散热设计的显卡，下面的型号可能还没有公开发布：</p><ol><li>涡轮式散热显卡：</li></ol><ul><li>GIGABYTE MSI GeForce RTX 3090 TURBO 24G<br><img src="https://mmbiz.qpic.cn/mmbiz_png/OuEFiapfBFkHD2fLPmuYNV1VI4qqLibvOuUu4grYmKoVK8ibGX8uG6vOG9icU4OiakMnMmRY78jDYITHgUdchU7A5qA/0?wx_fmt=png" alt=""></li></ul><ol start="2"><li>水冷散热显卡：</li></ol><ul><li>Colorful iGame Neptune GeForce RTX 30 系列<br><img src="https://mmbiz.qpic.cn/mmbiz_png/OuEFiapfBFkHD2fLPmuYNV1VI4qqLibvOuXWWBtiaBIGyLLLFJOdIgvicfyibzIMdEu4VH4ewV0ZpLtcdyALLdCw7Og/0?wx_fmt=png" alt=""></li><li>EVGA GeForce RTX 3080 10GB HYDRO COPPER<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkHD2fLPmuYNV1VI4qqLibvOuUZibcNM1cWTl5kHDliaQIAGHHJpoqDPIeicd9iboNPjlxBo1HuUCkchqsg/0?wx_fmt=jpeg" alt=""></li><li>EVGA GeForce RTX 3090 KINGPIN Hybrid<br><img src="https://mmbiz.qpic.cn/mmbiz_png/OuEFiapfBFkHD2fLPmuYNV1VI4qqLibvOunK2Bdu2oUgcqrg6UcWnO5vmNg0RDib8OFeBeI7BY3XFx03CE3HiavBrQ/0?wx_fmt=png" alt=""></li></ul><h1 id="四-GPU-工作站"><a href="#四-GPU-工作站" class="headerlink" title="四 GPU 工作站"></a>四 GPU 工作站</h1><h2 id="供电-1"><a href="#供电-1" class="headerlink" title="供电"></a>供电</h2><p>如果使用四张显卡，应该把主机放在专业机房内；在普通民用环境中目前只可能使用四张 3070 显卡，推荐 1500W 电源。</p><p>美国电脑供应商 Puget Systems 近期发表了一篇研究搭建一台<a href="https://www.pugetsystems.com/labs/articles/Quad-GeForce-RTX-3090-in-a-desktop---Does-it-work-1935/" target="_blank" rel="noopener" title="Quad GeForce RTX 3090 in a desktop - Does it work?">拥有 1~4 张 GIGABYTE MSI GeForce RTX 3090 TURBO 24G 显卡的工作站</a>的可能性的博客。当使用 4 块 3090 显卡时，使用了双 1600W 供电。在美国，3 块 3090 已经接近了普通民用电路的供电极限。<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkHu2GuyEJXjiaRyiaVPbc4m1nlvY1aZuSEpILicYCiaqh3HzmiaNxlbqiafmFYgFFwAUEU9Nq9DIJV8LKXQ/0?wx_fmt=jpeg" alt=""></p><h2 id="主板的选择"><a href="#主板的选择" class="headerlink" title="主板的选择"></a>主板的选择</h2><p>如果使用四张显卡，最好每张显卡都有 8 条通道。对于 Threadripper 来说，目前唯一的选择是 Gigabyte TRX40 DESIGNARE Motherboard：<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkHD2fLPmuYNV1VI4qqLibvOuk1REyycb9awpQyFMegvc2rNibHrIEjKwu3UsxeE9hGicdPTt0a3KAbfQ/0?wx_fmt=jpeg" alt=""></p><p>而对于 Core X-Series 来说，可以选择以下主板：</p><ul><li>GIGABYTE X299X AORUS MASTER（8-8-8-8 通道）</li><li>MSI Creator X299 LGA（8-8-16-8 通道）</li><li>MSI MEG X299 CREATION（8-8-16-8 通道）</li><li>EVGA X299 DARK（8 x 3 + 16 x 2 通道）</li></ul><p>还有两张主板有 7 个 PCI-E 插槽，因为有桥接芯片，支持 4 路 x16 PCI-E 通道：</p><ul><li>GIGABYTE X299-WU8<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkHD2fLPmuYNV1VI4qqLibvOuB11KH3PlVCYh5o2FqN3Z4piagwxDAwAhqHAzic7R5SLqpMM8u5YcOIJg/0?wx_fmt=jpeg" alt=""></li><li>ASUS WS X299 SAGE<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkHD2fLPmuYNV1VI4qqLibvOu1ibrHcTfTLw3KJbbmVEUDq6aY2Jg06SRZiaRCyDpC1uI7WCDXAkmRJEA/0?wx_fmt=jpeg" alt=""></li></ul><h2 id="CPU-与内存"><a href="#CPU-与内存" class="headerlink" title="CPU 与内存"></a>CPU 与内存</h2><p>Threadripper 是 24 核起，Core X-Series 是 12 核起，配 4 张 GPU 足够用了。</p><p>内存请参考双 GPU 部分。</p><hr><p>现在是购买 RTX 30 系列显卡的好时候吗？我认为不是。</p><ol><li>现在根本买不到啊；</li><li>深度学习框架对新 CUDA 和 CuDNN 的支持还不够；</li><li>各个厂家的显卡还没有开发完全；</li><li>新显卡的散热效果有待观察。</li></ol><p>NVIDIA 已经说了，目前的缺货会延续到 2021 年。我们还是耐心等待吧。另外也希望 Big Navi 的性能和供货给力，让本来打算买 N 卡的人去买 A 卡，给我们深度学习民工一条生路啊。</p>]]></content>
<summary type="html">
<p>本文接上一篇<a href="https://mp.weixin.qq.com/s?__biz=Mzg3OTIwODUzMQ==&mid=2247485265&idx=1&sn=9da954f0dd5c00884a6b3b92d7f2349a&chksm=cf06b66df8713f7b7ff0200a8e3b7ab324f3e6a72a0355cd5f9d0479cf802e4bb5df29d21ebc&token=1032059941&lang=zh_CN#rd" target="_blank" rel="noopener">《2020 年 10 月的单 GPU 深度学习工作站配置指南》</a>，探讨多 GPU 工作站的搭建。很多在单 GPU 工作站中不甚重要的因素在多 GPU 工作站中变得举足轻重。</p>
</summary>
<category term="深度学习工作站" scheme="https://vincent507cpu.github.io/tags/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%B7%A5%E4%BD%9C%E7%AB%99/"/>
</entry>
<entry>
<title>[工作站] 新 RTX 3090 搭建深度学习工作站的一些思考</title>
<link href="https://vincent507cpu.github.io/2020/10/10/%E5%B7%A5%E4%BD%9C%E7%AB%99-%E6%96%B0-RTX-3090-%E6%90%AD%E5%BB%BA%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%B7%A5%E4%BD%9C%E7%AB%99%E7%9A%84%E4%B8%80%E4%BA%9B%E6%80%9D%E8%80%83/"/>
<id>https://vincent507cpu.github.io/2020/10/10/%E5%B7%A5%E4%BD%9C%E7%AB%99-%E6%96%B0-RTX-3090-%E6%90%AD%E5%BB%BA%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%B7%A5%E4%BD%9C%E7%AB%99%E7%9A%84%E4%B8%80%E4%BA%9B%E6%80%9D%E8%80%83/</id>
<published>2020-10-11T01:58:39.000Z</published>
<updated>2020-10-18T01:33:11.403Z</updated>
<content type="html"><![CDATA[<p>今天在班上看完了 NVIDIA 的 GeForce RTX 30 系列发布会。看完感觉游戏玩家应该做梦都会笑醒：</p><table><thead><tr><th></th><th>RTX 3070</th><th>RTX 3080</th><th>RTX 3090</th><th>RTX 2080 Ti</th></tr></thead><tbody><tr><td>CUDA Core</td><td>5888</td><td>8704</td><td>10496</td><td>4352</td></tr><tr><td>Core Clock</td><td>1500 Mhz</td><td>1440 Mhz</td><td>1400 Mhz</td><td>1350 Mhz</td></tr><tr><td>Boost Clock</td><td>1730 Mhz</td><td>1710 Mhz</td><td>1700 Mhz</td><td>1545 Mhz</td></tr><tr><td>Memory Capacity</td><td>8 GB DDR6</td><td>10 GB DDR6X</td><td>24 GB DDR6X</td><td>11 GB DDR6</td></tr><tr><td>Memory Bus</td><td>256 bit</td><td>320 bit</td><td>384 bit</td><td>352 bit</td></tr><tr><td>Memory Speed</td><td>16 Gbps</td><td>19 Gbps</td><td>19.5 Gbps</td><td>14 Gbps</td></tr><tr><td>Memory Bandwidth</td><td>512 Gbps</td><td>760 Gbps</td><td>936 Gbps</td><td>616 Gbps</td></tr><tr><td>TDP</td><td>220w</td><td>320W</td><td>350W</td><td>275W</td></tr><tr><td>MSRP</td><td>$499 US</td><td>$699 US</td><td>$1499 US</td><td>$999 US</td></tr></tbody></table><a id="more"></a><p>一张中端卡吊打上代卡皇：<br><img src="https://mmbiz.qpic.cn/mmbiz_png/OuEFiapfBFkEPVr1buRIXzb1FIicicJWBcW2ZmupLZREtGLqic6LdgVEh2Jqahxzjotl204JUVptia0HICk0NTVLCzQ/0?wx_fmt=png" alt=""><br>新一代卡皇 RTX 3090 据老黄说可以以 8k 分辨率全开特效无压力玩任何游戏。苏妈现在应该是压力山大吧。</p><p>RTX 30 系列对游戏玩家是一个巨大的提升，但对深度学习研究者呢？假设想组一台 4 块 RTX 3090 的服务器，个人认为现在现有的 3090 显卡是不现实的。原因有三：</p><ol><li>功耗是一个巨大的隐患</li></ol><p>普通居民区、写字楼内的单根电线的最大载荷约为 2000W，超过这个数字会跳闸，用电器必须放在专门设计的机房里。以前 RTX 2080 Ti 在超频时最大功耗为不到 350W，四个 RTX 2080 Ti 加上 CPU 的功耗接近 2000W，还可以放在办公室里，而 RTX 3090 的 TDP 已经是 350W。NVIDIA 专门设计了一个 12 pin 的供电接口，最大载荷暂时未知，但有传言说最大载荷为 600W。而 RTX 2080 Ti 的供电接口为双 8 pin，其最大载荷为 150W （单个 8 pin 接口的供电）* 2 + 75W（PCIE 插槽供电）= 375W。目前已知的 12 pin 接口均为双 8 pin 转接而来，其最大载荷不会超过 375W；一旦电脑电源原生支持 12 pin 接口，功耗就难说了。</p><ol start="2"><li>版型太大，主板上容纳不下</li></ol><p>GeForce RTX 3090 is massive…这可能是第一张单卡三插槽的显卡，PCB 板在中间，前后都有风扇。<br><img src="https://mmbiz.qpic.cn/mmbiz_png/OuEFiapfBFkEeWLg0UUrtsztYiasVDzTXABFqoPQKnazZ2kE8mWQCHoBaRj7kU1hHdJ9uwibp1bK6XlgfDic3fa7zw/0?wx_fmt=png" alt=""><br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkEPVr1buRIXzb1FIicicJWBcWu8DdXpE8v7U7BrHs6S0IicAhWhibA8R43xoBuDrvF0NcVwHzCeosGicbQ/0?wx_fmt=jpeg" alt="RTX 3090 与 RTX 2080 Ti 对比"><br>三插槽就有问题了：主板上没位置插，机箱里也装不下。比如下面的一张主板：<br><img src="https://mmbiz.qpic.cn/mmbiz_png/OuEFiapfBFkEPVr1buRIXzb1FIicicJWBcWk7LslqLFYqsTgNwBG91kKM2ZZeL5HH1ctjF97Via4QpbX070Y9BMpCg/0?wx_fmt=png" alt=""><br>一般的双槽显卡可以插四张，而 RTX 3090 只能插两张（1x 和 3x），因为一张 RTX 3090 需要两边都有空位。就算主板上有充足的空间，一般机箱上只有 8 个 PCI-E 槽位。所以想要 4 x RTX 3090，需要更大的机箱（EATX 或者 WATX）和专用的主板，或者希望以后会出双卡槽的 RTX 3090。</p><ol start="3"><li>散热问题</li></ol><p>如果只有 1 张显卡，那么散热不是主要问题；而如果想放多张显卡，那么显卡一定要是涡轮散热设计。话说回来，显卡的风冷散热主要分涡轮散热和涡扇散热两种，主要区别在于出风的方向。涡扇散热的出风口在显卡背板：<br><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkEeWLg0UUrtsztYiasVDzTXAn2SxuPxZMJJpypazcqu9ClqDg4Mp8C4qS3msZccQ0UToNyUd02djpA/0?wx_fmt=jpeg" alt=""><br>而涡扇散热的出风口在显卡的四周：<br><img src="https://mmbiz.qpic.cn/mmbiz_png/OuEFiapfBFkEeWLg0UUrtsztYiasVDzTXAXdYqgXaCg16lKakuakvSlUqEzenoH5Po4AUKHibzpBbicBqETcMMmwRw/0?wx_fmt=png" alt=""><br>如果机箱里有超过两张显卡，那么一定要选择涡轮散热，否则散热放出的热风会被其他显卡重新使用，造成散热失效。</p><p>目前公布的所有 RTX 3090 都是涡扇设计，所以并不适合多块显卡组装深度学习工作站。而公版 RTX 3090 简直是单卡的福音，多卡的噩梦，看看 founder edition 的风道设计：<br><img src="https://mmbiz.qpic.cn/mmbiz_png/OuEFiapfBFkEeWLg0UUrtsztYiasVDzTXAtF8FicIPSbeQHEzxOV8DNpghgDqD0xy4dUCwNpukaD814HaWJalwfLA/0?wx_fmt=png" alt=""><br>假如几块 RTX 3090 并联排列，头一块显卡的热风直接被第二块显卡利用，以此类推…</p><p>现在已经有板厂（貌似是 EVGA）计划推出双卡槽的水冷散热版 RTX 3090。不过四个水冷风扇也不好摆。</p><p>那两张 RTX 3090 能超过四张 RTX 2080 Ti 吗？Only time will tell.</p><hr><p>有传言说 NVIDIA 其实还有 20GB 显存的 RTX 3080 （个人猜测可能会被叫做 super）和 16GB 显存的 RTX 3070 （super）没有被发布，可能是在观望 AMD 的下一代显卡 Big Navi 的表现。双卡槽的 RTX 3080 如果有 20GB 显存无疑会解决以上的所有问题，成为深度学习 GPU 的理想工具。希望苏妈能够给力一点，让老黄早点发布 RTX 3080 super 和 RTX 3070 super。</p><p>现在没有合适的显卡，不代表以后也没有。反正我准备入一块 RTX 3070 先用着，等 20GB RTX 3080 super 出了直接换 4 卡哈哈哈。</p>]]></content>
<summary type="html">
<p>今天在班上看完了 NVIDIA 的 GeForce RTX 30 系列发布会。看完感觉游戏玩家应该做梦都会笑醒：</p>
<table>
<thead>
<tr>
<th></th>
<th>RTX 3070</th>
<th>RTX 3080</th>
<th>RTX 3090</th>
<th>RTX 2080 Ti</th>
</tr>
</thead>
<tbody><tr>
<td>CUDA Core</td>
<td>5888</td>
<td>8704</td>
<td>10496</td>
<td>4352</td>
</tr>
<tr>
<td>Core Clock</td>
<td>1500 Mhz</td>
<td>1440 Mhz</td>
<td>1400 Mhz</td>
<td>1350 Mhz</td>
</tr>
<tr>
<td>Boost Clock</td>
<td>1730 Mhz</td>
<td>1710 Mhz</td>
<td>1700 Mhz</td>
<td>1545 Mhz</td>
</tr>
<tr>
<td>Memory Capacity</td>
<td>8 GB DDR6</td>
<td>10 GB DDR6X</td>
<td>24 GB DDR6X</td>
<td>11 GB DDR6</td>
</tr>
<tr>
<td>Memory Bus</td>
<td>256 bit</td>
<td>320 bit</td>
<td>384 bit</td>
<td>352 bit</td>
</tr>
<tr>
<td>Memory Speed</td>
<td>16 Gbps</td>
<td>19 Gbps</td>
<td>19.5 Gbps</td>
<td>14 Gbps</td>
</tr>
<tr>
<td>Memory Bandwidth</td>
<td>512 Gbps</td>
<td>760 Gbps</td>
<td>936 Gbps</td>
<td>616 Gbps</td>
</tr>
<tr>
<td>TDP</td>
<td>220w</td>
<td>320W</td>
<td>350W</td>
<td>275W</td>
</tr>
<tr>
<td>MSRP</td>
<td>$499 US</td>
<td>$699 US</td>
<td>$1499 US</td>
<td>$999 US</td>
</tr>
</tbody></table>
</summary>
<category term="深度学习工作站" scheme="https://vincent507cpu.github.io/tags/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%B7%A5%E4%BD%9C%E7%AB%99/"/>
</entry>
<entry>
<title>[工作站] 2020 年 10 月的单 GPU 深度学习工作站配置指南</title>
<link href="https://vincent507cpu.github.io/2020/09/20/%E5%B7%A5%E4%BD%9C%E7%AB%99-2020-%E5%B9%B4-10-%E6%9C%88%E7%9A%84%E5%8D%95-GPU-%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%B7%A5%E4%BD%9C%E7%AB%99%E9%85%8D%E7%BD%AE%E6%8C%87%E5%8D%97/"/>
<id>https://vincent507cpu.github.io/2020/09/20/%E5%B7%A5%E4%BD%9C%E7%AB%99-2020-%E5%B9%B4-10-%E6%9C%88%E7%9A%84%E5%8D%95-GPU-%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%B7%A5%E4%BD%9C%E7%AB%99%E9%85%8D%E7%BD%AE%E6%8C%87%E5%8D%97/</id>
<published>2020-09-20T22:04:45.000Z</published>
<updated>2020-10-18T01:30:48.576Z</updated>
<content type="html"><![CDATA[<p>随着电脑硬件的性能提升、价格下降，搭建个人用深度学习工作站的支出越来越低，需求也会越来越大。因此从今年开始，每年的 5、10 月份均会发布最新的深度学习工作站的配置指南。</p><hr><p>随着 NVIDIA 的新一代 Ampere 架构的 GeFore 30 系列显卡的发布，在算力得到了极大提升的同时价格也大幅下降，花费不到 $1000 搭建一台性能强大的深度学习工作站已经成为了可能。适逢 AMD 的新一代 Ryzen 处理器也在 10 月 8 日发布，硬件性能的提升毫无疑问会再次推动深度学习的热潮。</p><p>我最近准备搭建自己的第一台深度学习工作站，本文（单 GPU 工作站）与下一篇文章（多 GPU 工作站）正是基于本人最近的研究。水平有限，没有实践，欢迎指正。</p><a id="more"></a><p>本文主要参考了以下文章：</p><ul><li><a href="https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning/" target="_blank" rel="noopener" title="Which GPU(s) to Get for Deep Learning: My Experience and Advice for Using GPUs in Deep Learning">Which GPU(s) to Get for Deep Learning: My Experience and Advice for Using GPUs in Deep Learning</a></li><li><a href="https://timdettmers.com/2018/12/16/deep-learning-hardware-guide/" target="_blank" rel="noopener" title="A Full Hardware Guide to Deep Learning">A Full Hardware Guide to Deep Learning</a></li><li><a href="https://lambdalabs.com/blog/deep-learning-hardware-deep-dive-rtx-30xx/#blower-gpus" target="_blank" rel="noopener" title="Deep Learning Hardware Deep Dive – RTX 3090, RTX 3080, and RTX 3070">Deep Learning Hardware Deep Dive – RTX 3090, RTX 3080, and RTX 3070</a></li></ul><hr><p>工作站与个人游戏电脑不同，在配置上有一些需要改变的地方。对于深度学习来说，目前的唯一选择是 NVIDIA 的 GPU 产品；又因为本文的主题是个人深度学习工作站，所以本文仅涉及 NVIDIA 的 GeForce 系列消费级显卡（Tesla 以及 Quadro 系列都已经成为历史，统一到 GeForce 系列下）。本文首先来讨论深度学习工作站 must have 的部分，然后是 nice to have 的部分，再后是 don’t matter much 的部分，最后是 try to avoid 的部分。</p><h1 id="Must-Have"><a href="#Must-Have" class="headerlink" title="Must Have"></a>Must Have</h1><p>这部分不够就不行，但是超过也完全没用。</p><h2 id="显存"><a href="#显存" class="headerlink" title="显存"></a>显存</h2><p>通常来说，对显存的要求如下：</p><ul><li>研究 SOTA 模型：>= 11GB</li><li>一般的研究：8GB</li><li>Kaggle 及其它竞赛：4 - 8GB</li><li>公司业务：8GB 用于部署及原型测试，>= 11GB 用于训练</li></ul><p>对应到 RTX 30xx 系列显卡来说，可将 3060（6GB 显存），3060 Ti/3070 （8GB 显存）/3080（10GB 显存），3070 Super（16GB 显存）/3080 Super（20GB 显存）/3090（24GB 显存）对号入座。</p><p><strong>注</strong>： 3070 显卡将于 10 月 29 日上市，3060 Ti/2070 Super/3080 Super 预计在今年底前会陆续发布，3060 预计在明年年初发布。</p><h2 id="内存"><a href="#内存" class="headerlink" title="内存"></a>内存</h2><p>对于最大需要多少内存难以下定论，而 Tim Dettmers 说“额外的内存对特征工程非常有帮助”。综上，本人的推荐是 <em>内存容量 = 显存容量 + 6 ~ 8GB</em>。</p><h2 id="电源"><a href="#电源" class="headerlink" title="电源"></a>电源</h2><p>没人想在训练一半的时候因为供电不足而电脑重启，因此要预留足够的电源供电。主机内耗电的部分主要为 GPU、CPU 和主板上的其它部件。通过研究 GPU 与 CPU 的耗电数据，我发现 GPU 的峰值功耗要超过 TDP 100w 左右，而 8 核以下的 CPU 的峰值功耗大概可以归纳为</p><table><thead><tr><th>核心数量</th><th>峰值功耗（w）</th></tr></thead><tbody><tr><td>4</td><td>100</td></tr><tr><td>6</td><td>150</td></tr><tr><td>8</td><td>200</td></tr></tbody></table><p>主板的功耗（内存和硬盘之类的总和）大概为 80w，故电源功率的最低要求为：CPU 峰值功耗 + GPU 峰值功耗 + 80。因为 CPU 和 GPU 很少同时满负荷工作，因此不需要考虑冗余电源。比如 RTX 3080 的 TDP 为 320w，故一台 Ryzen 5 3600 与 一张 RTX 3080 的工作站需要一个额定功率最少为 150 + 320 + 100 + 80 = 650w 的电源。</p><p>另外不像游戏主机在不运行的时候关闭，工作站一般是 7 * 24 小时开机的，所以电源的转换效率也很重要。以下为 80 Plus 认证在 115V 电压下 100% 负载时的转换效率表<a href="https://en.wikipedia.org/wiki/80_Plus" target="_blank" rel="noopener" title="80 Plus 介绍"></a>：</p><table><thead><tr><th>认证等级</th><th>利用率</th></tr></thead><tbody><tr><td>White</td><td>80%</td></tr><tr><td>Bronze</td><td>82%</td></tr><tr><td>Silver</td><td>85%</td></tr><tr><td>Gold</td><td>87%</td></tr><tr><td>Platinum</td><td>89%</td></tr><tr><td>Titanium</td><td>90%</td></tr></tbody></table><p>一般来说，功耗在 600w 以下 Bronze 就可以了，600w ~ 1000w 之间推荐 Gold，1000w 以上推荐 Platinum 或 Titanium。</p><h1 id="Nice-to-Have"><a href="#Nice-to-Have" class="headerlink" title="Nice to Have"></a>Nice to Have</h1><p>以上的因素决定了模型能不能训练，下面的因素决定了训练模型的速度和操作者的体验。</p><h2 id="Tensor-Core"><a href="#Tensor-Core" class="headerlink" title="Tensor Core"></a>Tensor Core</h2><p>Tensor Core 可以极大地加快矩阵乘法，深度学习优先使用 Tensor Core 进行训练。由于 RTX 架构的 Tensor Core 可以以半精度（16bit）进行训练，显存需求减半，所以相比 GTX 显卡在同样的显存下可以训练大一倍的模型，因此除非预算极度有限，应该优先考虑 RTX 20/30 系列显卡。一张显卡有多少 Tensor Core 决定了这张显卡的算力，而 Tensor FLOPS 则量化了显卡的算力<a href="https://en.wikipedia.org/wiki/GeForce_20_series" target="_blank" rel="noopener" title="GeForce 20 series 系列介绍"></a><a href="https://www.reddit.com/r/buildapcsales/comments/ikqulm/meta_information_thread_for_the_nvidia_30series/" target="_blank" rel="noopener" title="[Meta] Information Thread for the Nvidia 30-series GPUs launch"></a>。</p><table><thead><tr><th>芯片型号</th><th>Tensor Core</th><th>Tensor FLOPS (万亿)</th><th>显存(GB)</th><th>TDP (W)</th><th>MSRP (USD)</th></tr></thead><tbody><tr><td>2060</td><td>240</td><td>51.6</td><td>6</td><td>160</td><td>349</td></tr><tr><td>2060 super</td><td>272</td><td>57.4</td><td>8</td><td>175</td><td>399</td></tr><tr><td>2070 super</td><td>320</td><td>72.5</td><td>8</td><td>215</td><td>499</td></tr><tr><td>2080 super</td><td>384</td><td>89.2</td><td>8</td><td>250</td><td>699</td></tr><tr><td>2080 Ti</td><td>544</td><td>107.6</td><td>11</td><td>250</td><td>999</td></tr><tr><td>Titan RTX</td><td>576</td><td>130.5</td><td>24</td><td>280</td><td>2499</td></tr><tr><td>3070</td><td>184</td><td>163</td><td>8</td><td>220</td><td>499</td></tr><tr><td>3080</td><td>272</td><td>238</td><td>10</td><td>320</td><td>699</td></tr><tr><td>3090</td><td>328</td><td>285</td><td>24</td><td>350</td><td>1499</td></tr></tbody></table><p>虽然 30 系列的 Tensor Core 数量比 20 系列少， 但官方称 30 系列的 Tensor Core 的性能是 20 系列的 4 倍，所以（如果官方宣传为真的话）3070 的实际算力要强于 2080 Ti。3060 Ti 尚未被官方确认，但估计其算力应该与 2080 Ti 相当。</p><p><strong>购买建议</strong>：消息说 3060 Ti 的官方指导价格是 $349。在 3060 Ti 存在的前提下 3070 是比较尴尬的存在，比上不足，比下有余。而 20 系列显卡毫无性价比，除非预算有限买二手 2060/2060 Super，否则不推荐。我的购买建议是：</p><ul><li>预算严重不足的入门菜鸟：二手 1660 Super</li><li>预算不足的入门菜鸟：二手 2060/2060 Super/2070</li><li>有一点预算的入门菜鸟：3060 Ti</li><li>中阶使用者：3070 Super/3080 Super</li><li>高阶使用者：3090</li></ul><p>现在避免购买任何 2080/2080 Super/2080 Ti 显卡（包括二手显卡）。</p><h2 id="外设"><a href="#外设" class="headerlink" title="外设"></a>外设</h2><p>除了 GPU，1 到 2 台额外的显示器和一把趁手的键盘可能是最有价值的投资。不过这部分比较主观，如何选择由各位读者考虑。</p><p><strong>购买建议</strong>：</p><ul><li>购买带翻转屏功能的显示器；我现在用的两台显示器中有一台 Dell 2718Q。</li><li>购买高分辨率的显示器；我现在用的是两台 27” 4K 显示器。</li><li>选择机械键盘。轴体根据自己的喜好选择，我现在用的是茶轴。</li></ul><p><img src="https://mmbiz.qpic.cn/mmbiz_jpg/OuEFiapfBFkEVjIE1DBlj7MuodfxytVDQ1mM3G5zCqM1rFhuFI3Ex7f3d9hnsva4w7AnxofoGWhhhnN57moeVSQ/0?wx_fmt=jpeg" alt="二次元真香"></p><h1 id="Don’t-Matter-Much"><a href="#Don’t-Matter-Much" class="headerlink" title="Don’t Matter Much"></a>Don’t Matter Much</h1><p>这部分对性能提升非常有限，不如节省下来减少开支。</p><h2 id="CPU-的核心数量和主频"><a href="#CPU-的核心数量和主频" class="headerlink" title="CPU 的核心数量和主频"></a>CPU 的核心数量和主频</h2><p>在深度学习中，CPU 的主要工作是数据预处理。有两种策略：</p><p><em>Loop</em>:</p><ul><li>Load mini-batch</li><li>Preprocessing mini-batch</li><li>Train on mini-batch</li></ul><p>或者</p><ul><li>Preprocess data</li><li>Loop:<ul><li>Load preprocessed mini-batch</li><li>Train on mini-batch</li></ul></li></ul><p>对于第一种策略，一颗强大的 CPU 会显著提高性能，推荐为 GPU 配备至少 4 个 CPU 线程；而第二种策略通常不需要非常好的 CPU，2 个线程足够。所以对于单 GPU 工作站而言，最低端的 Core i3 10100F 或者 Ryzen 3 3100 已经足够（两者都是 4 核心 8 线程），6 核以上完全没有必要。</p><p>而对于 CPU 频率而言，频率的影响非常有限（因为 CPU 在深度学习中不起主导作用），主频从 1.1GHz 提升到 3.6GHz 的综合性能提升在 4% ~ 8% 之间。</p><h2 id="PCI-E-等级-amp-通道数"><a href="#PCI-E-等级-amp-通道数" class="headerlink" title="PCI-E 等级 & 通道数"></a>PCI-E 等级 & 通道数</h2><p>PCI-E（Peripheral Component Interconnect Express）总线在 2003 年推出，取代了曾经的 PCI 和 AGP 总线，目前在使用的标准为 PCI-E 3.0 和 PCI-E 4.0。PCI-E 总线是一种串行总线，单个插槽上可以有 1、2、4、8、16 条通道，带宽如下：</p><table><thead><tr><th>PCI-E 版本</th><th>x1</th><th>x2</th><th>x4</th><th>x8</th><th>x16</th></tr></thead><tbody><tr><td>3.0</td><td>1.97GB/s</td><td>7.88GB/s</td><td>15.75GB/s</td><td>31.5GB/s</td><td></td></tr><tr><td>4.0</td><td>3.98GB/s</td><td>15.75GB/s</td><td>31.51GB/s</td><td>63GB/s</td><td></td></tr></tbody></table><p>可以看到，PCI-E 4.0 的带宽是 PCI-E 3.0 的两倍，因为 AMD 的 X570 和 B550 芯片组支持 PCI-E 4.0，而 Intel 要到明年上半年的 11 代 Core 才支持，所以使用 Ryzen 3 代以后的 CPU 和 500 系列主板会有带宽的优势。</p><p>通常显卡占用 8 或 16 条 PCI-E 通道，而一块 NVME M.2 存储器占用 4 条 PCI-E 通道。虽然显卡接收与传递数据经过 PCI-E 总线，然而在仅有 1 张显卡的时候，PCI-E 总线的带宽与级别对显卡性能的影响并不大，PCI-E 的带宽对文件的读取/写入性能的影响更大一点。不过在一个 pipeline 里面数据一般仅仅读取/写入一次，因此 PCI-E 4.0 或者 3.0 对性能影响有限。</p><h2 id="内存频率与延迟"><a href="#内存频率与延迟" class="headerlink" title="内存频率与延迟"></a>内存频率与延迟</h2><p>同上面一条，因为数据在 GPU 与 CPU 之间的交互次数有限，故速度更快、延迟更低的内存对性能提升有限。</p><h2 id="散热"><a href="#散热" class="headerlink" title="散热"></a>散热</h2><p>对于一台仅有 1 张显卡的工作站而言，散热不是需要考虑的问题。</p><h1 id="Try-to-Avoid"><a href="#Try-to-Avoid" class="headerlink" title="Try to Avoid"></a>Try to Avoid</h1><h2 id="超频"><a href="#超频" class="headerlink" title="超频"></a>超频</h2><p>对于长时间运行的工作站来说，超频会减少原件的寿命，降低系统的稳定性，增加功耗，因此超频是大忌。不要购买任何出厂预超频（ Overclock 或 OC 版）的显卡或自己超频。</p><h2 id="灯光效果"><a href="#灯光效果" class="headerlink" title="灯光效果"></a>灯光效果</h2><p>不是说不能有光效，但是工作站是用来干活的，不是用来欣赏的，而且工作站一般放在不起眼的地方，有光效也看不见。看得见的光效除了分散注意力以外还耗费额外的电能（还费钱），实在没有意义。</p>]]></content>
<summary type="html">
<p>随着电脑硬件的性能提升、价格下降，搭建个人用深度学习工作站的支出越来越低，需求也会越来越大。因此从今年开始，每年的 5、10 月份均会发布最新的深度学习工作站的配置指南。</p>
<hr>
<p>随着 NVIDIA 的新一代 Ampere 架构的 GeFore 30 系列显卡的发布，在算力得到了极大提升的同时价格也大幅下降，花费不到 $1000 搭建一台性能强大的深度学习工作站已经成为了可能。适逢 AMD 的新一代 Ryzen 处理器也在 10 月 8 日发布，硬件性能的提升毫无疑问会再次推动深度学习的热潮。</p>
<p>我最近准备搭建自己的第一台深度学习工作站，本文（单 GPU 工作站）与下一篇文章（多 GPU 工作站）正是基于本人最近的研究。水平有限，没有实践，欢迎指正。</p>
</summary>
<category term="深度学习工作站" scheme="https://vincent507cpu.github.io/tags/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%B7%A5%E4%BD%9C%E7%AB%99/"/>
</entry>
<entry>
<title>[NLP] 新手的第一个 NLP 项目：文本分类（4）</title>
<link href="https://vincent507cpu.github.io/2020/08/23/NLP-%E6%96%B0%E6%89%8B%E7%9A%84%E7%AC%AC%E4%B8%80%E4%B8%AA-NLP-%E9%A1%B9%E7%9B%AE%EF%BC%9A%E6%96%87%E6%9C%AC%E5%88%86%E7%B1%BB%EF%BC%884%EF%BC%89/"/>
<id>https://vincent507cpu.github.io/2020/08/23/NLP-%E6%96%B0%E6%89%8B%E7%9A%84%E7%AC%AC%E4%B8%80%E4%B8%AA-NLP-%E9%A1%B9%E7%9B%AE%EF%BC%9A%E6%96%87%E6%9C%AC%E5%88%86%E7%B1%BB%EF%BC%884%EF%BC%89/</id>
<published>2020-08-23T21:12:39.000Z</published>
<updated>2020-08-23T21:13:58.509Z</updated>
<content type="html"><![CDATA[<p>在之前的文章中，我们使用了 CNN 和 RNN 对 IMDB 数据集进行了分析，10 个 epoch 以后准确率不到 85%。除了使用更复杂的模型以外，我们还可以使用更好的词向量。本文中我们将使用 Bert 词向量和 GRU 层搭建另一个简单的神经网络模型。由于 transformers 涉及到大量计算，本文中将使用 Google Colab 提供的 GPU。</p><p>与前面的数据预处理流程不同，这里我们将使用 <code>torchtext</code> 来封装数据。有关 <code>torchtext</code> 的知识请看 <a href="https://mp.weixin.qq.com/s?__biz=Mzg3OTIwODUzMQ==&mid=2247485080&idx=1&sn=aea4f1162268db972506f9df6046c9c1&chksm=cf06b7a4f8713eb214b2ffe39e3f792e9dc0990ff691a97fafae1f7361e300d5e6188da5fafe&token=1227810784&lang=zh_CN#rd" target="_blank" rel="noopener">PyTorch 折桂 13：TorchText</a>。</p><a id="more"></a><p>安装所需的包：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">!pip install -U torch <span class="comment"># 1.7</span></span><br><span class="line">!pip install -U torchtext <span class="comment"># 0.7</span></span><br><span class="line">!pip install -U transformers <span class="comment"># 3.0.2</span></span><br></pre></td></tr></table></figure><p>设置随机种子：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> torch</span><br><span class="line"><span class="keyword">import</span> random</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"></span><br><span class="line">SEED = <span class="number">1988</span></span><br><span class="line"></span><br><span class="line">random.seed(SEED)</span><br><span class="line">np.random.seed(SEED)</span><br><span class="line">torch.manual_seed(SEED)</span><br><span class="line">torch.cuda.manual_seed(SEED)</span><br><span class="line">torch.backends.cudnn.deterministic = <span class="literal">True</span> <span class="comment"># 这样可以稍微增加训练的速度</span></span><br></pre></td></tr></table></figure><h1 id="数据准备"><a href="#数据准备" class="headerlink" title="数据准备"></a>数据准备</h1><p>之前的文章中，我们仅仅使用了 <code><PAD></code> 来填充不足的空位；而在 Bert 里，除了 <code><PAD></code> 还使用了 <code>BOS</code> 和 <code><EOS></code> 来表示句子的开始和结束以及 <code><UNK></code> 来表示单词表以外的单词。另外 Bert 取每句话前 512 个单词。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">init_token_id = tokenizer.cls_token_id <span class="comment"># BOS</span></span><br><span class="line">eos_token_id = tokenizer.sep_token_id <span class="comment"># EOS</span></span><br><span class="line">pad_token_id = tokenizer.pad_token_id <span class="comment"># PAD</span></span><br><span class="line">unk_token_id = tokenizer.unk_token_id <span class="comment"># UNK</span></span><br><span class="line"></span><br><span class="line">max_length_input = tokenizer.max_model_input_sizes[<span class="string">'bert-base-uncased'</span>]</span><br></pre></td></tr></table></figure><p>我们载入预训练好的 Bert 分词器并以此构建分词函数。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> transformers <span class="keyword">import</span> BertTokenizer</span><br><span class="line">tokenizer = BertTokenizer.from_pretrained(<span class="string">'bert-base-uncased'</span>)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">tokenize_and_cut</span><span class="params">(sentence)</span>:</span></span><br><span class="line"> tokens = tokenizer.tokenize(sentence)</span><br><span class="line"> tokens = tokens[:max_length_input - <span class="number">2</span>]</span><br><span class="line"> <span class="keyword">return</span> tokens</span><br></pre></td></tr></table></figure><p>下一步是构建数据集的域。所谓“域”指的是数据集里对文本与标签的处理方式的声明。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> torchtext.data <span class="keyword">import</span> Field, LabelField</span><br><span class="line"></span><br><span class="line">TEXT = Field(batch_first=<span class="literal">True</span>,</span><br><span class="line"> use_vocab=<span class="literal">False</span>,</span><br><span class="line"> tokenize=tokenize_and_cut,</span><br><span class="line"> preprocessing=tokenizer.convert_tokens_to_ids,</span><br><span class="line"> init_token=init_token_id,</span><br><span class="line"> eos_token=eos_token_id,</span><br><span class="line"> pad_token=pad_token_id,</span><br><span class="line"> unk_token=unk_token_id)</span><br><span class="line"></span><br><span class="line">LABEL = LabelField(dtype=torch.float)</span><br></pre></td></tr></table></figure><p>最后就是读取与封装数据。<code>batch size</code> 设置为 64。因为使用 GPU 训练，数据需要转移到 GPU 上。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> torchtext <span class="keyword">import</span> datasets</span><br><span class="line"></span><br><span class="line">train, test = datasets.IMDB.splits(TEXT, LABEL)</span><br><span class="line">LABEL.build_vocab(train)</span><br><span class="line"><span class="keyword">from</span> torchtext.data <span class="keyword">import</span> BucketIterator</span><br><span class="line"></span><br><span class="line">BATCH_SIZE = <span class="number">64</span></span><br><span class="line"></span><br><span class="line">device = torch.device(<span class="string">'cuda'</span> <span class="keyword">if</span> torch.cuda.is_available() <span class="keyword">else</span> <span class="string">'cpu'</span>)</span><br><span class="line"></span><br><span class="line">train_iter, test_iter = BucketIterator.splits(</span><br><span class="line"> (train, test),</span><br><span class="line"> batch_size=BATCH_SIZE,</span><br><span class="line"> device=device</span><br><span class="line">)</span><br></pre></td></tr></table></figure><h1 id="模型搭建"><a href="#模型搭建" class="headerlink" title="模型搭建"></a>模型搭建</h1><p>载入 Bert 预训练模型：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> transformers <span class="keyword">import</span> BertTokenizer, BertModel</span><br><span class="line">bert = BertModel.from_pretrained(<span class="string">'bert-base-uncased'</span>)</span><br></pre></td></tr></table></figure><p>根据 <a href="https://arxiv.org/pdf/1810.04805.pdf" target="_blank" rel="noopener" title="Bert 论文">Bert 论文</a>，Bert base 模型的超参数有：transformers 层数为 12，隐藏层维度为 768，self-attention head 数量为 12。我们在实际模型中只需要隐藏层维度。现在我们搭建一个在 Bert 后面连接一个双层、双向 GRU 的模型。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> torch <span class="keyword">import</span> nn</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">BertGRU</span><span class="params">(nn.Module)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, bert, hidden_dim, n_layers, bidirectional, dropout)</span>:</span></span><br><span class="line"> super().__init__()</span><br><span class="line"></span><br><span class="line"> self.bert = bert</span><br><span class="line"></span><br><span class="line"> embed_dim = bert.config.to_dict()[<span class="string">'hidden_size'</span>]</span><br><span class="line"></span><br><span class="line"> self.gru = nn.GRU(embed_dim, hidden_dim, num_layers=n_layers, bidirectional=bidirectional, </span><br><span class="line"> batch_first=<span class="literal">True</span>, dropout=<span class="number">0</span> <span class="keyword">if</span> n_layers < <span class="number">2</span> <span class="keyword">else</span> dropout)</span><br><span class="line"> </span><br><span class="line"> self.fc = nn.Linear(hidden_dim * <span class="number">2</span> <span class="keyword">if</span> bidirectional <span class="keyword">else</span> hidden_dim, <span class="number">1</span>)</span><br><span class="line"></span><br><span class="line"> self.dropout = nn.Dropout(dropout)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">forward</span><span class="params">(self, text)</span>:</span> <span class="comment"># text: [BATCH_SIZE, SEQ_LENGTH]</span></span><br><span class="line"> <span class="keyword">with</span> torch.no_grad():</span><br><span class="line"> embedded = self.bert(text)[<span class="number">0</span>] <span class="comment"># embedded: [BATCH_SIZE, SEQ_LENGTH, EMBED_DIM]</span></span><br><span class="line"></span><br><span class="line"> _, hidden = self.gru(embedded) <span class="comment"># hidden: [N_LAYERS * n_driections, BATCH_SIZE, EMBED_DIM]</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> self.gru.bidirectional:</span><br><span class="line"> hidden = self.dropout(torch.cat((hidden[<span class="number">-2</span>, :, :], hidden[<span class="number">-1</span>, :, :]), dim=<span class="number">1</span>))</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> hidden = self.dropout(hidden[<span class="number">-1</span>, :, :])</span><br><span class="line"></span><br><span class="line"> output = self.fc(hidden) <span class="comment"># hidden: [BATCH_SIZE, 1]</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> output</span><br></pre></td></tr></table></figure><p>首先实例化这个模型。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">HIDDEN_DIM = <span class="number">768</span></span><br><span class="line">N_LAYERS = <span class="number">2</span></span><br><span class="line">BIDIRECTIONAL = <span class="literal">True</span></span><br><span class="line">DROPOUT = <span class="number">0.5</span></span><br><span class="line"></span><br><span class="line">model = BertGRU(bert, HIDDEN_DIM, N_LAYERS, BIDIRECTIONAL, DROPOUT)</span><br></pre></td></tr></table></figure><p>因为 Bert 是已经训练好的词向量，我们不希望它被训练，也不希望它的权重被更新，所以模型里有 <code>with torch.no_grad()</code> 代码块。另外我们也手动关闭 Bert 有关的权重更新：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">for</span> name, param <span class="keyword">in</span> model.named_parameters():</span><br><span class="line"> <span class="keyword">if</span> name.startswith(<span class="string">'bert'</span>):</span><br><span class="line"> param.requires_grad = <span class="literal">False</span></span><br></pre></td></tr></table></figure><p>优化器和损失函数和前面一样，使用 Adam 和二分类交叉熵。同样将优化器和损失函数转移到 GPU 上。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> torch <span class="keyword">import</span> optim</span><br><span class="line"></span><br><span class="line">optimizer = optim.Adam(model.parameters())</span><br><span class="line"></span><br><span class="line">criterion = nn.BCEWithLogitsLoss()</span><br><span class="line"></span><br><span class="line">model = model.to(device)</span><br><span class="line">criterion = criterion.to(device)</span><br></pre></td></tr></table></figure><p>后面的训练和预测同以前的文章一样，不再赘述。训练 10 个 epoch 后的表现为：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">Epoch: <span class="number">10</span> | Epoch Time: <span class="number">38</span>m <span class="number">7</span>s</span><br><span class="line">Train Loss: <span class="number">0.094</span> | Train Acc: <span class="number">96.62</span>%</span><br><span class="line"> Val. Loss: <span class="number">0.243</span> | Val. Acc: <span class="number">92.39</span>%</span><br></pre></td></tr></table></figure><p>有了 Bert 的加持，模型的性能提高了约 10%。</p>]]></content>
<summary type="html">
<p>在之前的文章中，我们使用了 CNN 和 RNN 对 IMDB 数据集进行了分析，10 个 epoch 以后准确率不到 85%。除了使用更复杂的模型以外，我们还可以使用更好的词向量。本文中我们将使用 Bert 词向量和 GRU 层搭建另一个简单的神经网络模型。由于 transformers 涉及到大量计算，本文中将使用 Google Colab 提供的 GPU。</p>
<p>与前面的数据预处理流程不同，这里我们将使用 <code>torchtext</code> 来封装数据。有关 <code>torchtext</code> 的知识请看 <a href="https://mp.weixin.qq.com/s?__biz=Mzg3OTIwODUzMQ==&mid=2247485080&idx=1&sn=aea4f1162268db972506f9df6046c9c1&chksm=cf06b7a4f8713eb214b2ffe39e3f792e9dc0990ff691a97fafae1f7361e300d5e6188da5fafe&token=1227810784&lang=zh_CN#rd" target="_blank" rel="noopener">PyTorch 折桂 13：TorchText</a>。</p>
</summary>
<category term="deep learning" scheme="https://vincent507cpu.github.io/tags/deep-learning/"/>
<category term="PyTorch" scheme="https://vincent507cpu.github.io/tags/PyTorch/"/>
<category term="NLP" scheme="https://vincent507cpu.github.io/tags/NLP/"/>
</entry>
<entry>
<title>[NLP] 新手的第一个 NLP 项目：文本分类（3）</title>
<link href="https://vincent507cpu.github.io/2020/08/18/NLP-%E6%96%B0%E6%89%8B%E7%9A%84%E7%AC%AC%E4%B8%80%E4%B8%AA-NLP-%E9%A1%B9%E7%9B%AE%EF%BC%9A%E6%96%87%E6%9C%AC%E5%88%86%E7%B1%BB%EF%BC%883%EF%BC%89/"/>
<id>https://vincent507cpu.github.io/2020/08/18/NLP-%E6%96%B0%E6%89%8B%E7%9A%84%E7%AC%AC%E4%B8%80%E4%B8%AA-NLP-%E9%A1%B9%E7%9B%AE%EF%BC%9A%E6%96%87%E6%9C%AC%E5%88%86%E7%B1%BB%EF%BC%883%EF%BC%89/</id>
<published>2020-08-18T13:00:27.000Z</published>
<updated>2020-08-18T13:32:04.187Z</updated>
<content type="html"><![CDATA[<h1 id="前文回顾"><a href="#前文回顾" class="headerlink" title="前文回顾"></a>前文回顾</h1><p>在前两篇文章<a href="https://mp.weixin.qq.com/s?__biz=Mzg3OTIwODUzMQ==&mid=2247485105&idx=1&sn=0fac0adff0dff8812d73a0e510d65e9a&chksm=cf06b78df8713e9b9a4024c7f09d2b031afce20e43dc696006be9bf9c34b6f4bf25f4e71acae&token=1910822964&lang=zh_CN#rd" target="_blank" rel="noopener">新手的第一个 NLP 任务：文本分类（1）</a>和<a href="https://mp.weixin.qq.com/s?__biz=Mzg3OTIwODUzMQ==&mid=2247485108&idx=1&sn=62ec5a9782c5e7dbecd6cac8f7e0eb58&chksm=cf06b788f8713e9e19c8513a687e92e782720f3e851f1219e8f956e02c8f69c1bc5838606cc1&token=134245917&lang=zh_CN#rd" target="_blank" rel="noopener">新手的第一个 NLP 项目：文本分类（2）</a>中，我们读取了数据、对数据进行了预处理和封装，并搭建了一个 CNN 模型。本文中，我们将 CNN 模型换为 RNN 模型。</p><h1 id="数据的准备"><a href="#数据的准备" class="headerlink" title="数据的准备"></a>数据的准备</h1><p>同<a href="https://mp.weixin.qq.com/s?__biz=Mzg3OTIwODUzMQ==&mid=2247485105&idx=1&sn=0fac0adff0dff8812d73a0e510d65e9a&chksm=cf06b78df8713e9b9a4024c7f09d2b031afce20e43dc696006be9bf9c34b6f4bf25f4e71acae&token=1910822964&lang=zh_CN#rd" target="_blank" rel="noopener">新手的第一个 NLP 任务：文本分类（1）</a>一样，不再赘述。</p><a id="more"></a><h1 id="基础-RNN-模型"><a href="#基础-RNN-模型" class="headerlink" title="基础 RNN 模型"></a>基础 RNN 模型</h1><p>有关 RNN 的知识可以参考我以前写的文章 <a href="https://mp.weixin.qq.com/s?__biz=Mzg3OTIwODUzMQ==&mid=2247485071&idx=1&sn=b9b570591e340c53b8d5cc161ef382cc&chksm=cf06b7b3f8713ea5f4b689a5c018ef21f5f1e42f32d2f4965acfec589d1260c21e9ceb90cee3&token=129126518&lang=zh_CN#rd" target="_blank" rel="noopener">PyTorch 折桂 11：CNN & RNN</a>。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> torch <span class="keyword">import</span> nn, optim</span><br><span class="line"><span class="keyword">from</span> torch.nn <span class="keyword">import</span> functional <span class="keyword">as</span> F</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">RNN</span><span class="params">(nn.Module)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, vocab_size, embed_dim, hidden_dim)</span>:</span></span><br><span class="line"> super(RNN, self).__init__()</span><br><span class="line"> </span><br><span class="line"> self.embedding = nn.Embedding(vocab_size, embed_dim) <span class="comment"># (BATCH_SIZE, SEQ_LEN, EMBED_DIM)</span></span><br><span class="line"> self.rnn = nn.RNN(embed_dim, hidden_dim, batch_first=<span class="literal">True</span>)</span><br><span class="line"> self.fc = nn.Linear(hidden_dim, <span class="number">1</span>)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">forward</span><span class="params">(self, x)</span>:</span></span><br><span class="line"> x = self.embedding(x)</span><br><span class="line"></span><br><span class="line"> output, hidden = self.rnn(x)</span><br><span class="line"> <span class="comment"># output: (BATCH_SIZE, SEQ_LENGTH, HIDDEN_DIM)</span></span><br><span class="line"> <span class="comment"># hidden: (1, BATCH_SIZE, HIDDEN_DIM)</span></span><br><span class="line"> </span><br><span class="line"> <span class="keyword">return</span> self.fc(hidden.squeeze(<span class="number">0</span>))</span><br></pre></td></tr></table></figure><p>我们首先使用一层单向 RNN。RNN 网络生成两个张量：输出层与保存了历史信息的隐藏层。使用哪一个呢？这要具体问题具体分析。对于文本摘要类任务，一般使用保存了历史信息的隐藏层。</p><p>这里要注意隐藏层的维度：当 <code>batch_first=True</code> 时，隐藏层的维度为（<code>(num_layers * directions, BATCH_SIZE, HIDDEN_DIM)</code>）；当 <code>batch_first=False</code> 时，隐藏层的维度为（<code>(num_layers * directions, SEQ_LENGTH, HIDDEN_DIM)</code>）。</p><p>因为这是一个单词单向的 RNN，所以第 0 维为 1；在将隐藏层进行全连接处理以前，先去除无用的第 0 维。</p><p>实例化 RNN 网络：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">EMBED_DIM = <span class="number">128</span></span><br><span class="line">HIDDEN_DIM = <span class="number">256</span></span><br><span class="line">rnn = RNN(len(vocab), EMBED_DIM, HIDDEN_DIM)</span><br></pre></td></tr></table></figure><p>损失函数、优化器、训练过程与前文一致，不再赘述。训练 10 个 epoch 后的结果如下：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">Epoch: <span class="number">10</span> | Epoch Time: <span class="number">1</span>m <span class="number">8</span>s</span><br><span class="line">Train Loss: <span class="number">0.590</span> | Train Acc: <span class="number">68.58</span>%</span><br><span class="line"> Val. Loss: <span class="number">0.682</span> | Val. Acc: <span class="number">61.32</span>%</span><br></pre></td></tr></table></figure><p>可以看到，模型过拟合了。下面我们改进一下这个 RNN 模型。</p><h1 id="改进-RNN-模型"><a href="#改进-RNN-模型" class="headerlink" title="改进 RNN 模型"></a>改进 RNN 模型</h1><p>我们主要从以下两个方面进行改进：</p><ol><li>改进词嵌入；</li><li>增加模型的复杂度（使用两层双向 LSTM）；</li><li>增加正则化。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">LSTM</span><span class="params">(nn.Module)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, vocab_size, embedding_dim, hidden_dim, n_layers, </span></span></span><br><span class="line"><span class="function"><span class="params"> bidirectional, dropout)</span>:</span></span><br><span class="line"> super(LSTM, self).__init__()</span><br><span class="line"> </span><br><span class="line"> self.embed = nn.Embedding(vocab_size, embedding_dim, padding_idx=<span class="number">0</span>)</span><br><span class="line"> </span><br><span class="line"> self.lstm = nn.LSTM(embedding_dim, </span><br><span class="line"> hidden_dim, </span><br><span class="line"> num_layers=n_layers, </span><br><span class="line"> bidirectional=bidirectional, </span><br><span class="line"> dropout=dropout,</span><br><span class="line"> batch_first=<span class="literal">True</span>)</span><br><span class="line"> </span><br><span class="line"> self.dropout = nn.Dropout(dropout)</span><br><span class="line"> self.num_directions = <span class="number">2</span> <span class="keyword">if</span> bidirectional <span class="keyword">else</span> <span class="number">1</span></span><br><span class="line"> self.fc = nn.Linear(hidden_dim * self.num_directions, <span class="number">1</span>)</span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">forward</span><span class="params">(self, x)</span>:</span></span><br><span class="line"> embedded = self.dropout(self.embed(x)) <span class="comment"># (BATCH_SIZE, SEQ_LEN, EMBED_DIM)</span></span><br><span class="line"></span><br><span class="line"> output, (hidden, cell) = self.lstm(embedded)</span><br><span class="line"> <span class="comment"># output: (BATCH_SIZE, SEQ_LENGTH, HIDDEN_DIM)</span></span><br><span class="line"> <span class="comment"># hidden: (n_layers * num_directions, BATCH_SIZE, HIDDEN_DIM)</span></span><br><span class="line"> <span class="comment"># cell: (n_layers * num_directions, BATCH_SIZE, HIDDEN_DIM)</span></span><br><span class="line"> </span><br><span class="line"> hidden = self.dropout(torch.cat((hidden[<span class="number">-2</span>, :, :], hidden[<span class="number">-1</span>, :, :]), dim=<span class="number">1</span>))</span><br><span class="line"> <span class="comment"># hidden: (BATCH_SIZE, HIDDEN_DIM * 2)</span></span><br><span class="line"> <span class="keyword">return</span> self.fc(hidden)</span><br></pre></td></tr></table></figure>首先，填充 <code><PAD></code> 应该恒为 0，所以我们在词嵌入层中加入 <code>padding_idx=0</code> 条件。这里 <code>padding_idx</code> 为 0 是因为我们在数据准备过程中将填充占位设为 0。</li></ol><p>其次，将 RNN 层变成 LSTM 层。LSTM 模型的输出有三个，<code>output, (hidden, cell)</code>，隐藏层与细胞状态在一个元组内。当 <code>batch_first=True</code> 时，隐藏层与细胞状态的维度为（<code>(num_layers * directions, BATCH_SIZE, HIDDEN_DIM)</code>）；当 <code>batch_first=False</code> 时，隐藏层与细胞状态的维度为（<code>(num_layers * directions, SEQ_LENGTH, HIDDEN_DIM)</code>）。当方向为双向且层数多于 1 时，隐藏层与细胞状态的堆叠层次为：$[第一层正向，第一层反向，…，最后一层正向，最后一层反向]$。这里使用了两层双向 LSTM。我们需要最后一层的正向与反向隐藏层，并把它们拼接在一起。</p><p>最后，还加入了 dropout 正则化。LSTM 内部的 dropout 可以使用 <code>dropout</code> 声明，LSTM 与全连接层之间的 dropout 可以使用 <code>nn.Dropout</code> 层。</p><p>实例化这个 LSTM 模型。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">EMBED_DIM = <span class="number">128</span></span><br><span class="line">HIDDEN_DIM = <span class="number">256</span></span><br><span class="line">N_LAYERS = <span class="number">2</span></span><br><span class="line">BIDIRECTIONAL = <span class="literal">True</span></span><br><span class="line">DROPOUT = <span class="number">0.5</span></span><br><span class="line"></span><br><span class="line">lstm = LSTM(len(vocab), EMBED_DIM, HIDDEN_DIM, N_LAYERS, BIDIRECTIONAL, DROPOUT)</span><br></pre></td></tr></table></figure><p>损失函数、优化器、训练过程与前面相同。最终的训练效果为：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">Epoch: <span class="number">10</span> | Epoch Time: <span class="number">7</span>m <span class="number">34</span>s</span><br><span class="line">Train Loss: <span class="number">0.303</span> | Train Acc: <span class="number">87.30</span>%</span><br><span class="line"> Val. Loss: <span class="number">0.412</span> | Val. Acc: <span class="number">83.43</span>%</span><br></pre></td></tr></table></figure><p>比前面的 CNN 效果稍好。下文中我们将使用 SOTA 的预训练模型 - BERT。</p><p>本文的代码可以在 <a href="https://github.com/vincent507cpu/nlp\_project/blob/master/text%20classification/02%20RNN.ipynb" target="_blank" rel="noopener">https://github.com/vincent507cpu/nlp\_project/blob/master/text%20classification/02%20RNN.ipynb</a> 查看。</p>]]></content>
<summary type="html">
<h1 id="前文回顾"><a href="#前文回顾" class="headerlink" title="前文回顾"></a>前文回顾</h1><p>在前两篇文章<a href="https://mp.weixin.qq.com/s?__biz=Mzg3OTIwODUzMQ==&mid=2247485105&idx=1&sn=0fac0adff0dff8812d73a0e510d65e9a&chksm=cf06b78df8713e9b9a4024c7f09d2b031afce20e43dc696006be9bf9c34b6f4bf25f4e71acae&token=1910822964&lang=zh_CN#rd" target="_blank" rel="noopener">新手的第一个 NLP 任务：文本分类（1）</a>和<a href="https://mp.weixin.qq.com/s?__biz=Mzg3OTIwODUzMQ==&mid=2247485108&idx=1&sn=62ec5a9782c5e7dbecd6cac8f7e0eb58&chksm=cf06b788f8713e9e19c8513a687e92e782720f3e851f1219e8f956e02c8f69c1bc5838606cc1&token=134245917&lang=zh_CN#rd" target="_blank" rel="noopener">新手的第一个 NLP 项目：文本分类（2）</a>中，我们读取了数据、对数据进行了预处理和封装，并搭建了一个 CNN 模型。本文中，我们将 CNN 模型换为 RNN 模型。</p>
<h1 id="数据的准备"><a href="#数据的准备" class="headerlink" title="数据的准备"></a>数据的准备</h1><p>同<a href="https://mp.weixin.qq.com/s?__biz=Mzg3OTIwODUzMQ==&mid=2247485105&idx=1&sn=0fac0adff0dff8812d73a0e510d65e9a&chksm=cf06b78df8713e9b9a4024c7f09d2b031afce20e43dc696006be9bf9c34b6f4bf25f4e71acae&token=1910822964&lang=zh_CN#rd" target="_blank" rel="noopener">新手的第一个 NLP 任务：文本分类（1）</a>一样，不再赘述。</p>
</summary>
<category term="deep learning" scheme="https://vincent507cpu.github.io/tags/deep-learning/"/>
<category term="PyTorch" scheme="https://vincent507cpu.github.io/tags/PyTorch/"/>
<category term="NLP" scheme="https://vincent507cpu.github.io/tags/NLP/"/>
</entry>
<entry>
<title>[NLP] 新手的第一个 NLP 项目：文本分类（2）</title>
<link href="https://vincent507cpu.github.io/2020/08/07/NLP-%E6%96%B0%E6%89%8B%E7%9A%84%E7%AC%AC%E4%B8%80%E4%B8%AA-NLP-%E9%A1%B9%E7%9B%AE%EF%BC%9A%E6%96%87%E6%9C%AC%E5%88%86%E7%B1%BB%EF%BC%882%EF%BC%89/"/>
<id>https://vincent507cpu.github.io/2020/08/07/NLP-%E6%96%B0%E6%89%8B%E7%9A%84%E7%AC%AC%E4%B8%80%E4%B8%AA-NLP-%E9%A1%B9%E7%9B%AE%EF%BC%9A%E6%96%87%E6%9C%AC%E5%88%86%E7%B1%BB%EF%BC%882%EF%BC%89/</id>
<published>2020-08-07T18:00:47.000Z</published>
<updated>2020-08-07T18:27:43.547Z</updated>
<content type="html"><![CDATA[<p>现在数据已经准备就绪，可以构建模型了。</p><p>本文的模型参考了论文 <a href="https://arxiv.org/pdf/1408.5882.pdf" target="_blank" rel="noopener" title="Convolutional Neural Networks for Sentence Classification">《Convolutional Neural Networks for Sentence Classification》</a>，原文代码<a href="https://github.com/dennybritz/cnn-text-classification-tf" target="_blank" rel="noopener" title="论文代码">在此</a>。</p><p>论文里使用了两个词嵌入：随模型进行训练的词嵌入和 Google 预训练好的 Word2Vec 词嵌入。本文里为了直观，没有采用预训练的词嵌入。</p><h1 id="构建模型"><a href="#构建模型" class="headerlink" title="构建模型"></a>构建模型</h1><p>论文里使用了三个卷积核分别为 3、4、5 的二维卷积层，拼接后经过一个范围为 4 的池化层。最后经过一个全连接层，经过 sigmoid 函数处理后输出。</p><a id="more"></a><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> torch <span class="keyword">import</span> nn</span><br><span class="line"><span class="keyword">from</span> torch.nn <span class="keyword">import</span> functional <span class="keyword">as</span> F</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">CNN</span><span class="params">(nn.Module)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, vocab_size, embed_size, dropout, batch_size)</span>:</span></span><br><span class="line"> super(CNN, self).__init__()</span><br><span class="line"> self.batch_size = batch_size</span><br><span class="line"> </span><br><span class="line"> self.embedding = nn.Embedding(vocab_size, embed_size) <span class="comment"># (BATCH_SIZE, SEQ_LEN, embed_size)</span></span><br><span class="line"> </span><br><span class="line"> self.conv1 = nn.Conv2d(<span class="number">1</span>, <span class="number">1</span>, <span class="number">3</span>)</span><br><span class="line"> self.conv2 = nn.Conv2d(<span class="number">1</span>, <span class="number">1</span>, <span class="number">4</span>)</span><br><span class="line"> self.conv3 = nn.Conv2d(<span class="number">1</span>, <span class="number">1</span>, <span class="number">5</span>)</span><br><span class="line"></span><br><span class="line"> self.dropout = nn.Dropout(dropout)</span><br><span class="line"> </span><br><span class="line"> self.fc = nn.Linear(<span class="number">2232</span>, <span class="number">1</span>)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">forward</span><span class="params">(self, x)</span>:</span></span><br><span class="line"> x = self.embedding(x)</span><br><span class="line"> x.unsqueeze_(<span class="number">1</span>) <span class="comment"># (BATCH_SIZE, 1, SEQ_LEN, embed_size)</span></span><br><span class="line"> output1 = self.conv1(x)</span><br><span class="line"> output1 = F.max_pool2d(F.relu(output1), <span class="number">4</span>)</span><br><span class="line"> </span><br><span class="line"> output2 = self.conv2(x)</span><br><span class="line"> output2 = F.max_pool2d(F.relu(output2), <span class="number">4</span>)</span><br><span class="line"> </span><br><span class="line"> output3 = self.conv3(x)</span><br><span class="line"> output3 = F.max_pool2d(F.relu(output3), <span class="number">4</span>)</span><br><span class="line"> output = torch.cat([output1, output2, output3], axis=<span class="number">1</span>)</span><br><span class="line"> output = self.dropout(output)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> self.fc(output.view(self.batch_size, <span class="number">-1</span>))</span><br></pre></td></tr></table></figure><h5 id="注意：因为经过词嵌入的张量维度为-BATCH-SIZE-SEQ-LEN-embed-size-，而-nn-Conv2d-的输入张量的维度要求为-BATCH-SIZE-CHANNEL-NONE-NONE-，所以我们需要使用-x-unsqueeze-1-为张量添加一个维度。"><a href="#注意：因为经过词嵌入的张量维度为-BATCH-SIZE-SEQ-LEN-embed-size-，而-nn-Conv2d-的输入张量的维度要求为-BATCH-SIZE-CHANNEL-NONE-NONE-，所以我们需要使用-x-unsqueeze-1-为张量添加一个维度。" class="headerlink" title="注意：因为经过词嵌入的张量维度为 (BATCH_SIZE, SEQ_LEN, embed_size)，而 nn.Conv2d 的输入张量的维度要求为 (BATCH_SIZE, CHANNEL, NONE, NONE)，所以我们需要使用 x.unsqueeze_(1) 为张量添加一个维度。"></a>注意：因为经过词嵌入的张量维度为 <code>(BATCH_SIZE, SEQ_LEN, embed_size)</code>，而 <code>nn.Conv2d</code> 的输入张量的维度要求为 <code>(BATCH_SIZE, CHANNEL, NONE, NONE)</code>，所以我们需要使用 <code>x.unsqueeze_(1)</code> 为张量添加一个维度。</h5><p>我们使用 Adam 为优化器，<code>nn.BCEWithLogitsLoss()</code> 为损失函数。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> torch <span class="keyword">import</span> optim</span><br><span class="line"></span><br><span class="line">optimizer = optim.Adam(model.parameters())</span><br><span class="line">criterion = nn.BCEWithLogitsLoss()</span><br></pre></td></tr></table></figure><h5 id="注意：nn-BCEWithLogitsLoss-是先进行了-sigmoid-运算后再求交叉熵的损失函数，无需额外的-sigmoid-运算。"><a href="#注意：nn-BCEWithLogitsLoss-是先进行了-sigmoid-运算后再求交叉熵的损失函数，无需额外的-sigmoid-运算。" class="headerlink" title="注意：nn.BCEWithLogitsLoss() 是先进行了 sigmoid 运算后再求交叉熵的损失函数，无需额外的 sigmoid 运算。"></a>注意：<code>nn.BCEWithLogitsLoss()</code> 是先进行了 sigmoid 运算后再求交叉熵的损失函数，无需额外的 sigmoid 运算。</h5><p>然后我们再定义一个求准确率的函数：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">binary_accuracy</span><span class="params">(preds, y)</span>:</span></span><br><span class="line"> rounded_preds = torch.round(torch.sigmoid(preds))</span><br><span class="line"> correct = (rounded_preds == y).float()</span><br><span class="line"> acc = correct.sum() / len(correct)</span><br><span class="line"> <span class="keyword">return</span> acc</span><br></pre></td></tr></table></figure><p>紧接着我们开始定义训练和验证的函数：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 训练函数</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">train</span><span class="params">(model, iterator, optimizer, criterion)</span>:</span></span><br><span class="line"> epoch_loss = <span class="number">0</span></span><br><span class="line"> epoch_acc = <span class="number">0</span></span><br><span class="line"> </span><br><span class="line"> model.train() <span class="comment"># 训练模式</span></span><br><span class="line"> </span><br><span class="line"> <span class="keyword">for</span> text, label <span class="keyword">in</span> iterator:</span><br><span class="line"> optimizer.zero_grad()</span><br><span class="line"> preds = model(text)</span><br><span class="line"> loss = criterion(preds.squeeze(), label.float())</span><br><span class="line"> acc = binary_accuracy(preds.squeeze(), label)</span><br><span class="line"> loss.backward()</span><br><span class="line"> optimizer.step()</span><br><span class="line"> </span><br><span class="line"> epoch_loss += loss.item()</span><br><span class="line"> epoch_acc += acc.item()</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">return</span> epoch_loss / len(iterator), epoch_acc / len(iterator)</span><br><span class="line"> </span><br><span class="line"><span class="comment"># 验证函数</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">evaluate</span><span class="params">(model, iterator, criterion)</span>:</span></span><br><span class="line"> epoch_loss = <span class="number">0</span></span><br><span class="line"> epoch_acc = <span class="number">0</span></span><br><span class="line"> </span><br><span class="line"> model.eval() <span class="comment"># 验证模式</span></span><br><span class="line"> </span><br><span class="line"> <span class="keyword">with</span> torch.no_grad():</span><br><span class="line"> <span class="keyword">for</span> text, label <span class="keyword">in</span> iterator:</span><br><span class="line"> preds = model(text)</span><br><span class="line"> loss = criterion(preds.squeeze(), label.float())</span><br><span class="line"> acc = binary_accuracy(preds.squeeze(), label)</span><br><span class="line"></span><br><span class="line"> epoch_loss += loss.item()</span><br><span class="line"> epoch_acc += acc.item()</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> epoch_loss / len(iterator), epoch_acc / len(iterator)</span><br></pre></td></tr></table></figure><p>可以看到，训练函数与验证函数大同小异，主要区别在于：</p><ol><li>训练模式下权重更新，验证模式下权重不更新；</li><li>验证模式没有优化器。<h5 id="注意：在计算损失函数时，真实标签也要转换成-float-格式，否则会报错。"><a href="#注意：在计算损失函数时，真实标签也要转换成-float-格式，否则会报错。" class="headerlink" title="注意：在计算损失函数时，真实标签也要转换成 float 格式，否则会报错。"></a>注意：在计算损失函数时，真实标签也要转换成 float 格式，否则会报错。</h5>下面就可以构建真正的训练、评估循环了：<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> time</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">epoch_time</span><span class="params">(start_time, end_time)</span>:</span> <span class="comment"># 计算每一轮花费的时间</span></span><br><span class="line"> elapsed_time = end_time - start_time</span><br><span class="line"> elapsed_mins = int(elapsed_time / <span class="number">60</span>)</span><br><span class="line"> elapsed_secs = int(elapsed_time - elapsed_mins * <span class="number">60</span>)</span><br><span class="line"> <span class="keyword">return</span> elapsed_mins, elapsed_secs</span><br><span class="line"> </span><br><span class="line">N_EPOCHS = <span class="number">10</span></span><br><span class="line">best_test_loss = float(<span class="string">'inf'</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> epoch <span class="keyword">in</span> range(N_EPOCHS):</span><br><span class="line"> start_time = time.time()</span><br><span class="line"> </span><br><span class="line"> train_loss, train_acc = train(model, train_iter, optimizer, criterion)</span><br><span class="line"> test_loss, test_acc = evaluate(model, test_iter, criterion)</span><br><span class="line"> </span><br><span class="line"> end_time = time.time()</span><br><span class="line"> epoch_mins, epoch_secs = epoch_time(start_time, end_time)</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">if</span> test_loss < best_test_loss:</span><br><span class="line"> best_test_loss = test_loss</span><br><span class="line"> torch.save(model.state_dict(), <span class="string">'model.pt'</span>)</span><br><span class="line"> </span><br><span class="line"> print(<span class="string">f'Epoch: <span class="subst">{epoch+<span class="number">1</span>:<span class="number">02</span>}</span> | Epoch Time: <span class="subst">{epoch_mins}</span>m <span class="subst">{epoch_secs}</span>s'</span>)</span><br><span class="line"> print(<span class="string">f'\tTrain Loss: <span class="subst">{train_loss:<span class="number">.3</span>f}</span> | Train Acc: <span class="subst">{train_acc*<span class="number">100</span>:<span class="number">.2</span>f}</span>%'</span>)</span><br><span class="line"> print(<span class="string">f'\t Val. Loss: <span class="subst">{test_loss:<span class="number">.3</span>f}</span> | Val. Acc: <span class="subst">{test_acc*<span class="number">100</span>:<span class="number">.2</span>f}</span>%'</span>)</span><br></pre></td></tr></table></figure>我们进行 10 轮训练，如果验证集的准确率大于最大准确率，则保存模型。最佳结果为：<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">Epoch: <span class="number">10</span> | Epoch Time: <span class="number">4</span>m <span class="number">5</span>s</span><br><span class="line">Train Loss: <span class="number">0.171</span> | Train Acc: <span class="number">93.41</span>%</span><br><span class="line"> Val. Loss: <span class="number">0.426</span> | Val. Acc: <span class="number">82.85</span>%</span><br></pre></td></tr></table></figure>这个结果马马虎虎，希望在后面将模型改进后，模型的表现会更好。</li></ol><p>可以在 <a href="https://github.com/vincent507cpu/nlp\_project/blob/master/text%20classification/01%20CNN.ipynb" target="_blank" rel="noopener">https://github.com/vincent507cpu/nlp\_project/blob/master/text%20classification/01%20CNN.ipynb</a> 查看全部代码。</p>]]></content>
<summary type="html">
<p>现在数据已经准备就绪，可以构建模型了。</p>
<p>本文的模型参考了论文 <a href="https://arxiv.org/pdf/1408.5882.pdf" target="_blank" rel="noopener" title="Convolutional Neural Networks for Sentence Classification">《Convolutional Neural Networks for Sentence Classification》</a>，原文代码<a href="https://github.com/dennybritz/cnn-text-classification-tf" target="_blank" rel="noopener" title="论文代码">在此</a>。</p>
<p>论文里使用了两个词嵌入：随模型进行训练的词嵌入和 Google 预训练好的 Word2Vec 词嵌入。本文里为了直观，没有采用预训练的词嵌入。</p>
<h1 id="构建模型"><a href="#构建模型" class="headerlink" title="构建模型"></a>构建模型</h1><p>论文里使用了三个卷积核分别为 3、4、5 的二维卷积层，拼接后经过一个范围为 4 的池化层。最后经过一个全连接层，经过 sigmoid 函数处理后输出。</p>
</summary>
<category term="deep learning" scheme="https://vincent507cpu.github.io/tags/deep-learning/"/>
<category term="PyTorch" scheme="https://vincent507cpu.github.io/tags/PyTorch/"/>
<category term="NLP" scheme="https://vincent507cpu.github.io/tags/NLP/"/>
</entry>
<entry>
<title>[NLP] 新手的第一个 NLP 任务：文本分类（1）</title>
<link href="https://vincent507cpu.github.io/2020/08/01/NLP-%E6%96%B0%E6%89%8B%E7%9A%84%E7%AC%AC%E4%B8%80%E4%B8%AA-NLP-%E4%BB%BB%E5%8A%A1%EF%BC%9A%E6%96%87%E6%9C%AC%E5%88%86%E7%B1%BB%EF%BC%881%EF%BC%89/"/>
<id>https://vincent507cpu.github.io/2020/08/01/NLP-%E6%96%B0%E6%89%8B%E7%9A%84%E7%AC%AC%E4%B8%80%E4%B8%AA-NLP-%E4%BB%BB%E5%8A%A1%EF%BC%9A%E6%96%87%E6%9C%AC%E5%88%86%E7%B1%BB%EF%BC%881%EF%BC%89/</id>
<published>2020-08-01T17:57:01.000Z</published>
<updated>2020-08-07T18:28:24.248Z</updated>
<content type="html"><![CDATA[<p>从终端任务来说，NLP 任务有文本分类、文本生成、翻译、文本摘要等等，其中文本分类是一个比较基础的任务。所以让我们从文本分类开始练习，从最简单的模型开始做起，然后尽量一步步提高它的性能。</p><p>文本分类有主题分类和感情分类两种。其中感情分类又比主题分类更加简单一点，因为很多感情分类是二分类任务（主题分类其实也可以，但是一般很少只分两个主题），所以我们将使用 IMDB 电影评论数据集进行一个感情分类任务。</p><a id="more"></a><h1 id="NLP-的-pipeline"><a href="#NLP-的-pipeline" class="headerlink" title="NLP 的 pipeline"></a>NLP 的 pipeline</h1><p>简单来说，NLP 的 pipeline 的主要步骤为：</p><ol><li>载入数据；</li><li>数据探索与分析（EDA）；</li><li>数据预处理；</li><li>数据的封装；</li><li>构建模型；</li><li>训练模型；</li><li>评估模型；</li><li>（可选）模型的推断。</li></ol><p>本文主要关注第 1、3、4 步。数据分析这里就略过了，因为 1）这个数据集是一个很经典的数据集，网上已经有无数人做了 EDA；2）我对 <code>pandas</code> 和 <code>matplotlib</code> 还不熟。因为我们现在要构建一个基线模型，采用的方法也比较原始，后面会介绍更高效、简便的方式。</p><h1 id="准备工作"><a href="#准备工作" class="headerlink" title="准备工作"></a>准备工作</h1><p>首先安装、升级所需的库（代码在 Jupyter Notebook 里运行，在 shell 里运行需要把每个命令前面的 <code>!</code> 去掉）：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">!pip install -U tqdm <span class="comment"># 4.48.0</span></span><br><span class="line">!pip install -U nltk <span class="comment"># 3.5</span></span><br><span class="line">!pip install -U spacy <span class="comment"># 2.3.2</span></span><br><span class="line">!pip install -U numpy <span class="comment"># 1.19.1</span></span><br><span class="line">!pip install -U pandas <span class="comment"># 1.1.0</span></span><br><span class="line">!pip install -U sklearn <span class="comment"># 0.23</span></span><br><span class="line">!pip install -U torch <span class="comment"># 1.6</span></span><br><span class="line">!pip install -U torchtext <span class="comment"># 0.7.0</span></span><br></pre></td></tr></table></figure><p>后续文章中默认使用以上最新的库。然后下载 <code>spacy</code> 和 <code>nltk</code> 的数据：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">!python -m spacy download en_core_web_md</span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> nltk.stem <span class="keyword">import</span> WordNetLemmatizer</span><br><span class="line">nltk.download()</span><br></pre></td></tr></table></figure><h1 id="载入数据"><a href="#载入数据" class="headerlink" title="载入数据"></a>载入数据</h1><p>我们首先使用 <code>pandas</code> 读取 <code>csv</code> 文件。IMDB 电影评论一共有 50000 条，分为 <code>positive</code> 和 <code>negative</code> 两种。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"></span><br><span class="line">data = pd.read_csv(<span class="string">'.../datasets/IMDB Dataset.csv'</span>)</span><br></pre></td></tr></table></figure><h1 id="数据预处理"><a href="#数据预处理" class="headerlink" title="数据预处理"></a>数据预处理</h1><p>对于 NLP 任务来说，数据即文本。文本预处理任务一般有：</p><ol><li>文本清洗（去除乱码、停用词等）；</li><li>分词；</li><li>（仅限英文）将词语进行还原；</li><li>文本的截取与补全；</li><li>构建词汇表；</li><li>创建一个将 token 转换为 id 的映射并将文本转换为 id（有时候还需要创建一个将 id 转换为token 的映射）。</li></ol><p><code>nltk</code> 和 <code>spacy</code> 是处理英文 NLP 任务的两个常用的库。本来我习惯使用 <code>nltk</code> 进行分词，然而发现 <code>nltk</code> 的效果没有 <code>spacy</code> 好。所以我这次使用 <code>spacy</code> 进行分词，使用 <code>nltk</code> 将词语还原成原型。</p><p>首先做一些准备工作：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">from nltk.stem import WordNetLemmatizer</span><br><span class="line">lemmatizer = WordNetLemmatizer() # 初始化 lemmatizer</span><br><span class="line"></span><br><span class="line">import spacy</span><br><span class="line">nlp = spacy.load('en_core_web_md') # 初始化语言处理引擎，用于分词</span><br></pre></td></tr></table></figure><p>因为深度学习模型只能处理数字，我们需要将文本转换为数字。我把所有的事情放在一起做了：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> tqdm <span class="keyword">import</span> tqdm</span><br><span class="line"><span class="keyword">import</span> re</span><br><span class="line"></span><br><span class="line">processed_review = []</span><br><span class="line">sentiment = []</span><br><span class="line"></span><br><span class="line">word2id = {<span class="string">'<PAD>'</span>:<span class="number">0</span>} <span class="comment"># token 到 id 的映射</span></span><br><span class="line"><span class="comment"># id2word = {0:'<PAD>'} # id 到 token 的映射，这个任务用不到</span></span><br><span class="line">vocab = set([<span class="string">'<PAD>'</span>]) <span class="comment"># 词汇表</span></span><br><span class="line">count = <span class="number">1</span></span><br><span class="line">SEQ_LEN = <span class="number">100</span> <span class="comment"># 每条文本的固定长度</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> tqdm(range(len(data))): <span class="comment"># tqdm 显示进度</span></span><br><span class="line"> text = data.review[i].lower() <span class="comment"># 转换为小写</span></span><br><span class="line"> text = re.sub(<span class="string">'<.+?>'</span>, <span class="string">''</span>, text) <span class="comment"># 去掉 HTML 文本</span></span><br><span class="line"> text = re.sub(<span class="string">'[<>]'</span>, <span class="string">''</span>, text) <span class="comment"># 去掉 HTML 文本</span></span><br><span class="line"> text = [lemmatizer.lemmatize(token.text) <span class="keyword">for</span> token <span class="keyword">in</span> nlp.tokenizer(text)][:SEQ_LEN] <span class="comment"># 先分词，再还原，最后截取</span></span><br><span class="line"></span><br><span class="line"> tmp = [<span class="number">0</span>] * (SEQ_LEN - len(text)) <span class="keyword">if</span> len(text) < SEQ_LEN <span class="keyword">else</span> [] <span class="comment"># 用 0 补全短文本</span></span><br><span class="line"> </span><br><span class="line"> <span class="comment"># 构建词汇表以及映射</span></span><br><span class="line"> <span class="keyword">for</span> word <span class="keyword">in</span> text:</span><br><span class="line"> <span class="keyword">if</span> word <span class="keyword">not</span> <span class="keyword">in</span> vocab:</span><br><span class="line"> vocab.add(word)</span><br><span class="line"> word2id[word] = count</span><br><span class="line"> tmp.append(count)</span><br><span class="line"> count += <span class="number">1</span></span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> tmp.append(word2id[word])</span><br><span class="line"></span><br><span class="line"> processed_review.append(tmp)</span><br><span class="line"> </span><br><span class="line"> <span class="comment"># 将 positive 转换 为 1，将 negative 转换为 0</span></span><br><span class="line"> <span class="keyword">if</span> data.sentiment[i] == <span class="string">'positive'</span>:</span><br><span class="line"> sentiment.append(<span class="number">1</span>)</span><br><span class="line"> <span class="keyword">elif</span> data.sentiment[i] == <span class="string">'negative'</span>:</span><br><span class="line"> sentiment.append(<span class="number">0</span>)</span><br></pre></td></tr></table></figure><h1 id="数据封装"><a href="#数据封装" class="headerlink" title="数据封装"></a>数据封装</h1><p>现在数据和标签都变成了数字，然后是划分训练集和测试集（我们暂时不用验证集）。这里使用 <code>sklearn</code> 里的函数实现，产生 40000 条训练集和 10000 条测试集。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> sklearn.model_selection <span class="keyword">import</span> train_test_split</span><br><span class="line"></span><br><span class="line">X_train, X_test, y_train, y_test = train_test_split(processed_review, sentiment, train_size=<span class="number">0.8</span>, random_state=<span class="number">1988</span>)</span><br></pre></td></tr></table></figure><p>在构建模型之前的最后一步是封装数据，以便以 batch 的数量将数据送进网络。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> torch.utils.data <span class="keyword">import</span> TensorDataset, DataLoader</span><br><span class="line"><span class="keyword">import</span> torch</span><br><span class="line"></span><br><span class="line">BATCH_SIZE = <span class="number">64</span></span><br><span class="line"></span><br><span class="line">train_ds = TensorDataset(torch.as_tensor(X_train), torch.as_tensor(y_train))</span><br><span class="line">test_ds = TensorDataset(torch.as_tensor(X_test), torch.as_tensor(y_test))</span><br><span class="line"></span><br><span class="line">train_iter = DataLoader(train_ds, batch_size=BATCH_SIZE, drop_last=<span class="literal">True</span>) <span class="comment"># (BATCH_SIZE, SEQ_LEN)</span></span><br><span class="line">test_iter = DataLoader(test_ds, batch_size=BATCH_SIZE, drop_last=<span class="literal">True</span>) <span class="comment"># (BATCH_SIZE, )</span></span><br></pre></td></tr></table></figure><p>首先使用 <code>TensorDataset</code> 将训练集和测试集转换成 PyTorch 可以识别的格式，然后使用 <code>DataLoader</code> 将数据集进行封装，生成一个以 <code>BATCH_SIZE</code> 为读取批量的生成器。</p><p>下一篇文章将进行建模和训练。可以在 <a href="https://github.com/vincent507cpu/nlp_project/blob/master/text%20classification/01%20CNN.ipynb" target="_blank" rel="noopener">https://github.com/vincent507cpu/nlp_project/blob/master/text%20classification/01%20CNN.ipynb</a> 查看全部代码。</p>]]></content>
<summary type="html">
<p>从终端任务来说，NLP 任务有文本分类、文本生成、翻译、文本摘要等等，其中文本分类是一个比较基础的任务。所以让我们从文本分类开始练习，从最简单的模型开始做起，然后尽量一步步提高它的性能。</p>
<p>文本分类有主题分类和感情分类两种。其中感情分类又比主题分类更加简单一点，因为很多感情分类是二分类任务（主题分类其实也可以，但是一般很少只分两个主题），所以我们将使用 IMDB 电影评论数据集进行一个感情分类任务。</p>
</summary>
<category term="deep learning" scheme="https://vincent507cpu.github.io/tags/deep-learning/"/>
<category term="PyTorch" scheme="https://vincent507cpu.github.io/tags/PyTorch/"/>
<category term="NLP" scheme="https://vincent507cpu.github.io/tags/NLP/"/>
</entry>
<entry>
<title>DL-PyTorch-折桂-18：使用-TorchText-和-transformers-进行情感分类(2)</title>
<link href="https://vincent507cpu.github.io/2020/08/01/DL-PyTorch-%E6%8A%98%E6%A1%82-18%EF%BC%9A%E4%BD%BF%E7%94%A8-TorchText-%E5%92%8C-transformers-%E8%BF%9B%E8%A1%8C%E6%83%85%E6%84%9F%E5%88%86%E7%B1%BB-2/"/>
<id>https://vincent507cpu.github.io/2020/08/01/DL-PyTorch-%E6%8A%98%E6%A1%82-18%EF%BC%9A%E4%BD%BF%E7%94%A8-TorchText-%E5%92%8C-transformers-%E8%BF%9B%E8%A1%8C%E6%83%85%E6%84%9F%E5%88%86%E7%B1%BB-2/</id>
<published>2020-08-01T17:52:36.000Z</published>
<updated>2020-11-01T21:36:56.694Z</updated>
<content type="html"><![CDATA[<p><a href="https://vincent507cpu.github.io/2020/06/10/DL-PyTorch-折桂-17：使用-TorchText-和-transformers-进行情感分类/">接上文</a></p><h1 id="9-搭建模型"><a href="#9-搭建模型" class="headerlink" title="9. 搭建模型"></a>9. 搭建模型</h1><p>首先是载入预训练模型。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> transformers <span class="keyword">import</span> BertTokenizer, BertModel</span><br><span class="line"></span><br><span class="line">bert = BertModel.from_pretrained(<span class="string">'bert-base-uncased'</span>)</span><br></pre></td></tr></table></figure><p>我们使用 Bert 预训练词向量与 GRU 组成模型，然后接一个全连接层。我们需要使用 <code>with torch.no_grad()</code> 避免预训练词向量发生变化。</p><a id="more"></a><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">BERTGRUSentiment</span><span class="params">(nn.Module)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self,</span></span></span><br><span class="line"><span class="function"><span class="params"> bert,</span></span></span><br><span class="line"><span class="function"><span class="params"> hidden_dim,</span></span></span><br><span class="line"><span class="function"><span class="params"> output_dim,</span></span></span><br><span class="line"><span class="function"><span class="params"> n_layers,</span></span></span><br><span class="line"><span class="function"><span class="params"> bidirectional,</span></span></span><br><span class="line"><span class="function"><span class="params"> dropout)</span>:</span></span><br><span class="line"> </span><br><span class="line"> super().__init__()</span><br><span class="line"> self.bert = bert</span><br><span class="line"> embedding_dim = bert.config.to_dict()[<span class="string">'hidden_size'</span>]</span><br><span class="line"> self.rnn = nn.GRU(embedding_dim,</span><br><span class="line"> hidden_dim,</span><br><span class="line"> num_layers = n_layers,</span><br><span class="line"> bidirectional = bidirectional,</span><br><span class="line"> batch_first = <span class="literal">True</span>,</span><br><span class="line"> dropout = <span class="number">0</span> <span class="keyword">if</span> n_layers < <span class="number">2</span> <span class="keyword">else</span> dropout)</span><br><span class="line"> </span><br><span class="line"> self.out = nn.Linear(hidden_dim * <span class="number">2</span> <span class="keyword">if</span> bidirectional <span class="keyword">else</span> hidden_dim, output_dim)</span><br><span class="line"> self.dropout = nn.Dropout(<span class="number">0.5</span>)</span><br><span class="line"> </span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">forward</span><span class="params">(self, text)</span>:</span></span><br><span class="line"> <span class="comment">#text = [batch size, sent len]</span></span><br><span class="line"> <span class="keyword">with</span> torch.no_grad():</span><br><span class="line"> embedded = self.bert(text)[<span class="number">0</span>]</span><br><span class="line"> <span class="comment">#embedded = [batch size, sent len, emb dim]</span></span><br><span class="line"> _, hidden = self.rnn(embedded)</span><br><span class="line"> </span><br><span class="line"> <span class="comment">#hidden = [n layers * n directions, batch size, emb dim]</span></span><br><span class="line"> <span class="keyword">if</span> self.rnn.bidirectional:</span><br><span class="line"> hidden = self.dropout(torch.cat((hidden[<span class="number">-2</span>,:,:], hidden[<span class="number">-1</span>,:,:]), dim = <span class="number">1</span>))</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> hidden = self.dropout(hidden[<span class="number">-1</span>,:,:])</span><br><span class="line"> </span><br><span class="line"> <span class="comment">#hidden = [batch size, hid dim]</span></span><br><span class="line"> output = self.out(hidden)</span><br><span class="line"> </span><br><span class="line"> <span class="comment">#output = [batch size, out dim]</span></span><br><span class="line"> <span class="keyword">return</span> output</span><br></pre></td></tr></table></figure><p>接下来我们使用标准超参数将模型实例化。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">HIDDEN_DIM = <span class="number">256</span></span><br><span class="line">OUTPUT_DIM = <span class="number">1</span></span><br><span class="line">N_LAYERS = <span class="number">2</span></span><br><span class="line">BIDIRECTIONAL = <span class="literal">True</span></span><br><span class="line">DROPOUT = <span class="number">0.25</span></span><br><span class="line"></span><br><span class="line">model = BERTGRUSentiment(bert,</span><br><span class="line"> HIDDEN_DIM,</span><br><span class="line"> OUTPUT_DIM,</span><br><span class="line"> N_LAYERS,</span><br><span class="line"> BIDIRECTIONAL,</span><br><span class="line"> DROPOUT)</span><br></pre></td></tr></table></figure><p>由于 transformer 的训练量实在比较大，我们设置不更新 bert 的权重：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">for</span> name, param <span class="keyword">in</span> model.named_parameters(): </span><br><span class="line"> <span class="keyword">if</span> name.startswith(<span class="string">'bert'</span>):</span><br><span class="line"> param.requires_grad = <span class="literal">False</span></span><br></pre></td></tr></table></figure><h1 id="10-训练模型"><a href="#10-训练模型" class="headerlink" title="10. 训练模型"></a>10. 训练模型</h1><p>优化器使用 Adam，损失函数使用 <code>nn.BCEWithLogitsLoss()</code>。除此以外，我们再定义一个评价准确率的函数：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">binary_accuracy</span><span class="params">(preds, y)</span>:</span></span><br><span class="line"> <span class="string">"""</span></span><br><span class="line"><span class="string"> Returns accuracy per batch, i.e. if you get 8/10 right, this returns 0.8, NOT 8</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> <span class="comment">#round predictions to the closest integer</span></span><br><span class="line"> rounded_preds = torch.round(torch.sigmoid(preds))</span><br><span class="line"> correct = (rounded_preds == y).float() <span class="comment">#convert into float for division </span></span><br><span class="line"> acc = correct.sum() / len(correct)</span><br><span class="line"> <span class="keyword">return</span> acc</span><br></pre></td></tr></table></figure><p>训练函数：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">train</span><span class="params">(model, iterator, optimizer, criterion)</span>:</span></span><br><span class="line"> epoch_loss = <span class="number">0</span></span><br><span class="line"> epoch_acc = <span class="number">0</span></span><br><span class="line"> </span><br><span class="line"> model.train()</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">for</span> batch <span class="keyword">in</span> iterator:</span><br><span class="line"> optimizer.zero_grad()</span><br><span class="line"> predictions = model(batch.text).squeeze(<span class="number">1</span>)</span><br><span class="line"> loss = criterion(predictions, batch.label)</span><br><span class="line"> acc = binary_accuracy(predictions, batch.label)</span><br><span class="line"> loss.backward()</span><br><span class="line"> optimizer.step()</span><br><span class="line"> epoch_loss += loss.item()</span><br><span class="line"> epoch_acc += acc.item()</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">return</span> epoch_loss / len(iterator), epoch_acc / len(iterator)</span><br></pre></td></tr></table></figure><p>验证函数与训练函数类似，区别在于：</p><ol><li>不更新权重；</li><li>没有优化器。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">evaluate</span><span class="params">(model, iterator, criterion)</span>:</span></span><br><span class="line"> epoch_loss = <span class="number">0</span></span><br><span class="line"> epoch_acc = <span class="number">0</span></span><br><span class="line"> </span><br><span class="line"> model.eval()</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">with</span> torch.no_grad():</span><br><span class="line"> <span class="keyword">for</span> batch <span class="keyword">in</span> iterator:</span><br><span class="line"> predictions = model(batch.text).squeeze(<span class="number">1</span>)</span><br><span class="line"> loss = criterion(predictions, batch.label)</span><br><span class="line"> acc = binary_accuracy(predictions, batch.label)</span><br><span class="line"> epoch_loss += loss.item()</span><br><span class="line"> epoch_acc += acc.item()</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">return</span> epoch_loss / len(iterator), epoch_acc / len(iterator)</span><br></pre></td></tr></table></figure>训练函数与验证函数都写好了以后就可以进行真正的训练了。在每一轮里，我们首先更新权重，然后用新的权重去验证。如果验证的损失小于之前的最小值，我们保存当前的模型。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">N_EPOCHS = <span class="number">10</span></span><br><span class="line">best_valid_loss = float(<span class="string">'inf'</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> epoch <span class="keyword">in</span> range(N_EPOCHS):</span><br><span class="line"> train_loss, train_acc = train(model, train_iterator, optimizer, criterion)</span><br><span class="line"> valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">if</span> valid_loss < best_valid_loss:</span><br><span class="line"> best_valid_loss = valid_loss</span><br><span class="line"> torch.save(model.state_dict(), <span class="string">'tut6-model.pt'</span>)</span><br><span class="line"> </span><br><span class="line"> print(<span class="string">f'Epoch: <span class="subst">{epoch+<span class="number">1</span>:<span class="number">02</span>}</span></span></span><br><span class="line"><span class="string"> print(f'</span>\tTrain Loss: {train_loss:<span class="number">.3</span>f} | Train Acc: {train_acc*<span class="number">100</span>:<span class="number">.2</span>f}%<span class="string">')</span></span><br><span class="line"><span class="string"> print(f'</span>\t Val. Loss: {valid_loss:<span class="number">.3</span>f} | Val. Acc: {valid_acc*<span class="number">100</span>:<span class="number">.2</span>f}%<span class="string">')</span></span><br></pre></td></tr></table></figure>训练过程如下：<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line">Epoch: <span class="number">01</span> | Epoch Time: <span class="number">7</span>m <span class="number">5</span>s</span><br><span class="line">Train Loss: <span class="number">0.468</span> | Train Acc: <span class="number">76.80</span>%</span><br><span class="line"> Val. Loss: <span class="number">0.266</span> | Val. Acc: <span class="number">89.47</span>%</span><br><span class="line">Epoch: <span class="number">02</span> | Epoch Time: <span class="number">7</span>m <span class="number">4</span>s</span><br><span class="line">Train Loss: <span class="number">0.280</span> | Train Acc: <span class="number">88.42</span>%</span><br><span class="line"> Val. Loss: <span class="number">0.244</span> | Val. Acc: <span class="number">90.20</span>%</span><br><span class="line">Epoch: <span class="number">03</span> | Epoch Time: <span class="number">7</span>m <span class="number">4</span>s</span><br><span class="line">Train Loss: <span class="number">0.239</span> | Train Acc: <span class="number">90.48</span>%</span><br><span class="line"> Val. Loss: <span class="number">0.220</span> | Val. Acc: <span class="number">91.07</span>%</span><br><span class="line">Epoch: <span class="number">04</span> | Epoch Time: <span class="number">7</span>m <span class="number">4</span>s</span><br><span class="line">Train Loss: <span class="number">0.211</span> | Train Acc: <span class="number">91.66</span>%</span><br><span class="line"> Val. Loss: <span class="number">0.236</span> | Val. Acc: <span class="number">90.85</span>%</span><br><span class="line">Epoch: <span class="number">05</span> | Epoch Time: <span class="number">7</span>m <span class="number">5</span>s</span><br><span class="line">Train Loss: <span class="number">0.187</span> | Train Acc: <span class="number">92.91</span>%</span><br><span class="line"> Val. Loss: <span class="number">0.222</span> | Val. Acc: <span class="number">91.12</span>%</span><br><span class="line">Epoch: <span class="number">06</span> | Epoch Time: <span class="number">7</span>m <span class="number">5</span>s</span><br><span class="line">Train Loss: <span class="number">0.164</span> | Train Acc: <span class="number">93.71</span>%</span><br><span class="line"> Val. Loss: <span class="number">0.251</span> | Val. Acc: <span class="number">91.29</span>%</span><br><span class="line">Epoch: <span class="number">07</span> | Epoch Time: <span class="number">7</span>m <span class="number">4</span>s</span><br><span class="line">Train Loss: <span class="number">0.137</span> | Train Acc: <span class="number">94.94</span>%</span><br><span class="line"> Val. Loss: <span class="number">0.231</span> | Val. Acc: <span class="number">90.73</span>%</span><br><span class="line">Epoch: <span class="number">08</span> | Epoch Time: <span class="number">7</span>m <span class="number">4</span>s</span><br><span class="line">Train Loss: <span class="number">0.115</span> | Train Acc: <span class="number">95.73</span>%</span><br><span class="line"> Val. Loss: <span class="number">0.374</span> | Val. Acc: <span class="number">86.99</span>%</span><br><span class="line">Epoch: <span class="number">09</span> | Epoch Time: <span class="number">7</span>m <span class="number">4</span>s</span><br><span class="line">Train Loss: <span class="number">0.095</span> | Train Acc: <span class="number">96.57</span>%</span><br><span class="line"> Val. Loss: <span class="number">0.259</span> | Val. Acc: <span class="number">91.22</span>%</span><br><span class="line">Epoch: <span class="number">10</span> | Epoch Time: <span class="number">7</span>m <span class="number">5</span>s</span><br><span class="line">Train Loss: <span class="number">0.078</span> | Train Acc: <span class="number">97.30</span>%</span><br><span class="line"> Val. Loss: <span class="number">0.282</span> | Val. Acc: <span class="number">91.77</span>%</span><br></pre></td></tr></table></figure><h1 id="11-模型推断"><a href="#11-模型推断" class="headerlink" title="11. 模型推断"></a>11. 模型推断</h1>训练好模型以后，我们可以用这个模型来做推断。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">predict_sentiment</span><span class="params">(model, tokenizer, sentence)</span>:</span></span><br><span class="line"> model.eval()</span><br><span class="line"> tokens = tokenizer.tokenize(sentence)</span><br><span class="line"> tokens = tokens[:max_input_length<span class="number">-2</span>]</span><br><span class="line"> indexed = [init_token_idx] + tokenizer.convert_tokens_to_ids(tokens) + [eos_token_idx]</span><br><span class="line"> tensor = torch.LongTensor(indexed).to(device)</span><br><span class="line"> tensor = tensor.unsqueeze(<span class="number">0</span>)</span><br><span class="line"> prediction = torch.sigmoid(model(tensor))</span><br><span class="line"> <span class="keyword">return</span> prediction.item()</span><br><span class="line"> </span><br><span class="line">predict_sentiment(model, tokenizer, <span class="string">"This film is terrible"</span>) <span class="comment"># 0.021611081436276436</span></span><br><span class="line">predict_sentiment(model, tokenizer, <span class="string">"This film is great"</span>) <span class="comment"># 0.9428628087043762</span></span><br></pre></td></tr></table></figure>上面直接返回概率，也可以处理一下返回 positive 或 negative。</li></ol>]]></content>
<summary type="html">
<p><a href="https://vincent507cpu.github.io/2020/06/10/DL-PyTorch-折桂-17：使用-TorchText-和-transformers-进行情感分类/">接上文</a></p>
<h1 id="9-搭建模型"><a href="#9-搭建模型" class="headerlink" title="9. 搭建模型"></a>9. 搭建模型</h1><p>首先是载入预训练模型。</p>
<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> transformers <span class="keyword">import</span> BertTokenizer, BertModel</span><br><span class="line"></span><br><span class="line">bert = BertModel.from_pretrained(<span class="string">'bert-base-uncased'</span>)</span><br></pre></td></tr></table></figure>
<p>我们使用 Bert 预训练词向量与 GRU 组成模型，然后接一个全连接层。我们需要使用 <code>with torch.no_grad()</code> 避免预训练词向量发生变化。</p>
</summary>
<category term="deep learning" scheme="https://vincent507cpu.github.io/tags/deep-learning/"/>
<category term="PyTorch" scheme="https://vincent507cpu.github.io/tags/PyTorch/"/>
<category term="NLP" scheme="https://vincent507cpu.github.io/tags/NLP/"/>
</entry>
<entry>
<title>[DL] PyTorch 折桂 17：使用 TorchText 和 transformers 进行情感分类(1)</title>
<link href="https://vincent507cpu.github.io/2020/06/10/DL-PyTorch-%E6%8A%98%E6%A1%82-17%EF%BC%9A%E4%BD%BF%E7%94%A8-TorchText-%E5%92%8C-transformers-%E8%BF%9B%E8%A1%8C%E6%83%85%E6%84%9F%E5%88%86%E7%B1%BB/"/>
<id>https://vincent507cpu.github.io/2020/06/10/DL-PyTorch-%E6%8A%98%E6%A1%82-17%EF%BC%9A%E4%BD%BF%E7%94%A8-TorchText-%E5%92%8C-transformers-%E8%BF%9B%E8%A1%8C%E6%83%85%E6%84%9F%E5%88%86%E7%B1%BB/</id>
<published>2020-06-10T19:51:36.000Z</published>
<updated>2020-08-01T16:10:05.712Z</updated>
<content type="html"><![CDATA[<p> 我们已经了解了 PyTorch 的基本操作和功能，现在让我们实践一下。自从 transformer 横空出世以后，在 NLP 领域有”大一统“ 的趋势。但 transformer 的本质是什么？transformer 的本质是一个能够有效提取语义信息的词嵌入生成器，它比前辈 word2vec、GloVe 等等能够更有效地提取词语的语义信息，所以以 transformer 生成的词嵌入可以有 SOTA（state-of-the-art，最高水平）的性能。这等于电脑可以更好地理解文本中每个词语的意思，理解了每个词语的意思自然就可以更好地理解文本的整体意思。所以 transformer 只是取代了以前用的 Embedding 层，根据具体的任务的不同还可以接上 CNN、RNN 等层。</p><p> 本文及下一篇文章中，我们将使用 PyTorch，TorchText 和 transformers 库里的 Bert 预训练模型来进行一个基本的情感分类任务：IMDB 影片评论的情感分类。</p><a id="more"></a> <p> Bert 之类的 transformer 预训练模型虽然性能强大，它也有一个致命的缺点，即资源的消耗非常巨大。简单的模型跑小数据库还可以在个人 PC 上运行，Bert 之类的模型必须用到 GPU。写这篇文章的时候，我是在 Google Colab 上完成模型的训练的。<br> <figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span> !nvidia-smi</span><br><span class="line"> Wed Jun <span class="number">10</span> <span class="number">18</span>:<span class="number">00</span>:<span class="number">42</span> <span class="number">2020</span> </span><br><span class="line">+-----------------------------------------------------------------------------+</span><br><span class="line">| NVIDIA-SMI <span class="number">440.82</span> Driver Version: <span class="number">418.67</span> CUDA Version: <span class="number">10.1</span> |</span><br><span class="line">|-------------------------------+----------------------+----------------------+</span><br><span class="line">| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |</span><br><span class="line">| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |</span><br><span class="line">|===============================+======================+======================|</span><br><span class="line">| <span class="number">0</span> Tesla P100-PCIE... Off | <span class="number">00000000</span>:<span class="number">00</span>:<span class="number">04.0</span> Off | <span class="number">0</span> |</span><br><span class="line">| N/A <span class="number">69</span>C P0 <span class="number">48</span>W / <span class="number">250</span>W | <span class="number">9149</span>MiB / <span class="number">16280</span>MiB | <span class="number">0</span>% Default |</span><br><span class="line">+-------------------------------+----------------------+----------------------+</span><br><span class="line"> </span><br><span class="line">+-----------------------------------------------------------------------------+</span><br><span class="line">| Processes: GPU Memory |</span><br><span class="line">| GPU PID Type Process name Usage |</span><br><span class="line">|=============================================================================|</span><br><span class="line">+-----------------------------------------------------------------------------+</span><br></pre></td></tr></table></figure><br> 本文的代码来自 <a href="https://colab.research.google.com/github/bentrevett/pytorch-sentiment-analysis/blob/master/6%20-%20Transformers%20for%20Sentiment%20Analysis.ipynb#scrollTo=xZ1S1o1iH5jT" target="_blank" rel="noopener" title="Transformers for Sentiment Analysis">Transformers for Sentiment Analysis</a>。</p><p> *注：本文的很多资源可能需要科学上网，不能科学上网的话我也没办法哈。</p><h1 id="1-必要库的加载"><a href="#1-必要库的加载" class="headerlink" title="1. 必要库的加载"></a>1. 必要库的加载</h1><p> 首先安装最新版 PyTorch，TorchText 和 transformers。<br> <figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">!pip install -U torchtext</span><br><span class="line">!pip install -U torch</span><br><span class="line">!pip install -U transformers</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> torch</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> random</span><br><span class="line"></span><br><span class="line">SEED = <span class="number">1234</span></span><br><span class="line"><span class="comment"># 初始化随机种子</span></span><br><span class="line">random.seed(SEED)</span><br><span class="line">np.random.seed(SEED)</span><br><span class="line">torch.manual_seed(SEED)</span><br><span class="line">torch.backends.cudnn.deterministic = <span class="literal">True</span> <span class="comment"># 可以加快训练速度一点点</span></span><br></pre></td></tr></table></figure><br>虽然每一步都会加载必需的库，我还是会把完整路径写出来，方便确认从属。</p><h1 id="2-获取数据："><a href="#2-获取数据：" class="headerlink" title="2. 获取数据："></a>2. 获取数据：</h1><p>在 Kaggle 上下载<a href="https://www.kaggle.com/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews" target="_blank" rel="noopener" title="IMDB 电影评论数据">数据</a>。</p><h1 id="3-分词器的准备"><a href="#3-分词器的准备" class="headerlink" title="3. 分词器的准备"></a>3. 分词器的准备</h1><p>我们将使用 Bert 预训练模型的分词器。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> transformers <span class="keyword">import</span> BertTokenizer</span><br><span class="line"></span><br><span class="line">tokenizer = transformers.BertTokenizer.from_pretrained(<span class="string">'bert-base-uncased'</span>)</span><br></pre></td></tr></table></figure><p>‘uncased’ 意味着这个分词器是不区分大小写的，意味着仅仅处理小写字母。不用担心，分词器会自动把文本转换成小写形式再处理。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span>tokens = tokenizer.tokenize(<span class="string">'Hello WORLD how ARE yoU?'</span>) <span class="comment"># 大小写不敏感的预训练模型会自动转换大小写</span></span><br><span class="line"></span><br><span class="line"><span class="meta">>>> </span>print(tokens)</span><br><span class="line">[<span class="string">'hello'</span>, <span class="string">'world'</span>, <span class="string">'how'</span>, <span class="string">'are'</span>, <span class="string">'you'</span>, <span class="string">'?'</span>]</span><br><span class="line"></span><br><span class="line"><span class="meta">>>> </span>indexes = tokenizer.convert_tokens_to_ids(tokens) <span class="comment"># 找到 token 对应的 id</span></span><br><span class="line"></span><br><span class="line"><span class="meta">>>> </span>print(indexes)</span><br><span class="line">[<span class="number">7592</span>, <span class="number">2088</span>, <span class="number">2129</span>, <span class="number">2024</span>, <span class="number">2017</span>, <span class="number">1029</span>]</span><br></pre></td></tr></table></figure><p>接下来我们需要指定 4 个特殊的 token：句起始 token，句结束 token，填充 token 和未知词语 token 备用。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span>init_token_idx = tokenizer.cls_token_id <span class="comment"># 起始</span></span><br><span class="line"><span class="meta">>>> </span>eos_token_idx = tokenizer.sep_token_id <span class="comment"># 结束</span></span><br><span class="line"><span class="meta">>>> </span>pad_token_idx = tokenizer.pad_token_id <span class="comment"># 填充</span></span><br><span class="line"><span class="meta">>>> </span>unk_token_idx = tokenizer.unk_token_id <span class="comment"># 未知</span></span><br><span class="line"></span><br><span class="line"><span class="meta">>>> </span>print(init_token_idx, eos_token_idx, pad_token_idx, unk_token_idx)</span><br><span class="line"><span class="number">101</span> <span class="number">102</span> <span class="number">0</span> <span class="number">100</span></span><br></pre></td></tr></table></figure><p>然后我们还需要获得预训练 Bert 模型的序列长度。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span>max_input_length = tokenizer.max_model_input_sizes[<span class="string">'bert-base-uncased'</span>]</span><br><span class="line"></span><br><span class="line"><span class="meta">>>> </span>print(max_input_length)</span><br><span class="line"><span class="number">512</span></span><br></pre></td></tr></table></figure><h1 id="4-构建分词器"><a href="#4-构建分词器" class="headerlink" title="4. 构建分词器"></a>4. 构建分词器</h1><p>上一篇文章里说我们可以使用 spacy 作为分词器，这里我们使用 <code>BertTokenizer</code>。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">tokenize_and_cut</span><span class="params">(sentence)</span>:</span></span><br><span class="line"> tokens = tokenizer.tokenize(sentence) </span><br><span class="line"> tokens = tokens[:max_input_length<span class="number">-2</span>]</span><br><span class="line"> <span class="keyword">return</span> tokens</span><br></pre></td></tr></table></figure><p>因为预训练 Bert 模型的最长序列为 512，为给数据点加上 <code>[CLS]</code> 和 <code>[EOS]</code>，我们需要把分词后的序列长度减 2.</p><h1 id="5-定义-field"><a href="#5-定义-field" class="headerlink" title="5. 定义 field"></a>5. 定义 field</h1><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> torchtext <span class="keyword">import</span> data</span><br><span class="line"></span><br><span class="line">TEXT = torchtext.data.Field(batch_first = <span class="literal">True</span>,</span><br><span class="line"> use_vocab = <span class="literal">False</span>,</span><br><span class="line"> tokenize = tokenize_and_cut,</span><br><span class="line"> preprocessing = tokenizer.convert_tokens_to_ids,</span><br><span class="line"> init_token = init_token_idx,</span><br><span class="line"> eos_token = eos_token_idx,</span><br><span class="line"> pad_token = pad_token_idx,</span><br><span class="line"> unk_token = unk_token_idx)</span><br><span class="line"></span><br><span class="line">LABEL = torchtext.data.LabelField(dtype = torch.float)</span><br></pre></td></tr></table></figure><p>因为 batch 在第一维，所以我们设定 <code>batch_first = True</code>。由于我们已经有了单词表（bert 的 embedding），所以需要设置 <code>use_vocab = False</code>。然后我们在 <code>preprocessing</code> 这里将 token 转换成对应的 id。最后，我需要定义特殊的 token id。</p><h1 id="6-加载数据"><a href="#6-加载数据" class="headerlink" title="6. 加载数据"></a>6. 加载数据</h1><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> torchtext <span class="keyword">import</span> datasets</span><br><span class="line"></span><br><span class="line">train_data, test_data = torchtext.datasets.IMDB.splits(TEXT, LABEL)</span><br><span class="line"></span><br><span class="line">train_data, valid_data = train_data.split(random_state = random.seed(SEED))</span><br></pre></td></tr></table></figure><p><code>IMDB</code> 数据库使用 <code>splts</code> 方法创建训练集和测试集，具体参数如下：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">splits(text_field, label_field, root=<span class="string">'.data'</span>, train=<span class="string">'train'</span>, test=<span class="string">'test'</span>, **kwargs)</span><br></pre></td></tr></table></figure><p>两个必需参数 <code>text_field</code> 和 <code>label_field</code> 分别对应了文本与标签的 field。</p><h1 id="7-建立标签的词汇表"><a href="#7-建立标签的词汇表" class="headerlink" title="7. 建立标签的词汇表"></a>7. 建立标签的词汇表</h1><p>虽然我们已经在定义 field 的时候定义了 TEXT 的词汇表，我们还需要将 LABEL 转换为数字。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">LABEL.build_vocab(train_data)</span><br></pre></td></tr></table></figure><h1 id="8-创建训练集、验证集和测试集"><a href="#8-创建训练集、验证集和测试集" class="headerlink" title="8. 创建训练集、验证集和测试集"></a>8. 创建训练集、验证集和测试集</h1><p>最后我们使用 <code>BucketIterator</code> 创建训练集、验证集和测试集的迭代器。这里我们使用 128 作为 batch size。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">BATCH_SIZE = <span class="number">128</span></span><br><span class="line"></span><br><span class="line">device = torch.device(<span class="string">'cuda'</span> <span class="keyword">if</span> torch.cuda.is_available() <span class="keyword">else</span> <span class="string">'cpu'</span>)</span><br><span class="line"></span><br><span class="line">train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(</span><br><span class="line"> (train_data, valid_data, test_data), </span><br><span class="line"> batch_size = BATCH_SIZE, </span><br><span class="line"> device = device)</span><br></pre></td></tr></table></figure><p>欢迎关注我的微信公众号“花解语 NLP”：<br><img src="https://img-blog.csdnimg.cn/20200514100635366.jpg#pic_center" alt=""></p>]]></content>
<summary type="html">
<p> 我们已经了解了 PyTorch 的基本操作和功能，现在让我们实践一下。自从 transformer 横空出世以后，在 NLP 领域有”大一统“ 的趋势。但 transformer 的本质是什么？transformer 的本质是一个能够有效提取语义信息的词嵌入生成器，它比前辈 word2vec、GloVe 等等能够更有效地提取词语的语义信息，所以以 transformer 生成的词嵌入可以有 SOTA（state-of-the-art，最高水平）的性能。这等于电脑可以更好地理解文本中每个词语的意思，理解了每个词语的意思自然就可以更好地理解文本的整体意思。所以 transformer 只是取代了以前用的 Embedding 层，根据具体的任务的不同还可以接上 CNN、RNN 等层。</p>
<p> 本文及下一篇文章中，我们将使用 PyTorch，TorchText 和 transformers 库里的 Bert 预训练模型来进行一个基本的情感分类任务：IMDB 影片评论的情感分类。</p>
</summary>
<category term="deep learning" scheme="https://vincent507cpu.github.io/tags/deep-learning/"/>
<category term="PyTorch" scheme="https://vincent507cpu.github.io/tags/PyTorch/"/>
<category term="NLP" scheme="https://vincent507cpu.github.io/tags/NLP/"/>
</entry>
<entry>
<title>[DL] PyTorch 折桂 16：transformers</title>
<link href="https://vincent507cpu.github.io/2020/06/05/DL-PyTorch-%E6%8A%98%E6%A1%82-16%EF%BC%9Atransformers/"/>
<id>https://vincent507cpu.github.io/2020/06/05/DL-PyTorch-%E6%8A%98%E6%A1%82-16%EF%BC%9Atransformers/</id>
<published>2020-06-05T16:47:59.000Z</published>
<updated>2020-06-17T11:04:34.173Z</updated>
<content type="html"><![CDATA[<p> 严格意义上讲 transformers 并不是 PyTorch 的一部分，然而 transformers 与 PyTorch 或 TensorFlow 结合的太紧密了，而且可以把 transformers 看成是 PyTorch 或 TensorFlow 的延伸，所以也在这里一并讨论了。<br> <a id="more"></a><br> transformers 内置了 17 种以 transformer 结构为基础的神经网络：</p><ul><li><p>T5 model</p></li><li><p>DistilBERT model</p></li><li><p>ALBERT model</p></li><li><p>CamemBERT model</p></li><li><p>XLM-RoBERTa model</p></li><li><p>Longformer model</p></li><li><p>RoBERTa model</p></li><li><p>Reformer model</p></li><li><p>Bert model</p></li><li><p>OpenAI GPT model</p></li><li><p>OpenAI GPT-2 model</p></li><li><p>Transformer-XL model</p></li><li><p>XLNet model</p></li><li><p>XLM model</p></li><li><p>CTRL model</p></li><li><p>Flaubert model</p></li><li><p>ELECTRA model</p><p>这些模型的参数、用法大同小异。默认框架为 PyTorch，使用 TensorFlow 框架在类的前面加上 ‘TF” 即可。</p><p>每种模型都有至少一个预训练模型，限于篇幅，这里仅仅列举 Bert 的常用预训练模型：</p><table><thead><tr><th align="center">模型</th><th align="center">模型细节</th></tr></thead><tbody><tr><td align="center"><code>bert-base-uncased</code></td><td align="center">12-layer, 768-hidden, 12-heads, 110M parameters. Trained on lower-cased English text.</td></tr><tr><td align="center"><code>bert-large-uncased</code></td><td align="center">24-layer, 1024-hidden, 16-heads, 340M parameters. Trained on lower-cased English text.</td></tr><tr><td align="center"><code>bert-base-cased</code></td><td align="center">12-layer, 768-hidden, 12-heads, 110M parameters. Trained on cased English text.</td></tr><tr><td align="center"><code>bert-large-cased</code></td><td align="center">24-layer, 1024-hidden, 16-heads, 340M parameters. Trained on cased English text.</td></tr><tr><td align="center"><code>bert-base-multilingual-cased</code></td><td align="center">12-layer, 768-hidden, 12-heads, 110M parameters. Trained on cased text in the top 104 languages with the largest Wikipedias</td></tr><tr><td align="center"><code>bert-base-chinese</code></td><td align="center">12-layer, 768-hidden, 12-heads, 110M parameters. Trained on cased Chinese Simplified and Traditional text.</td></tr></tbody></table><p>完整的预训练模型列表可以在 <a href="https://huggingface.co/transformers/pretrained_models.html" target="_blank" rel="noopener" title="Pretrained models">transformers 官网</a>上找到。</p><p>使用 transformers 库有三种方法：</p></li></ul><ol><li><p>使用 <code>pipeline</code>；</p></li><li><p>指定预训练模型；</p></li><li><p>使用 <code>AutoModels</code> 加载预训练模型。</p><h1 id="1-transformers-pipeline"><a href="#1-transformers-pipeline" class="headerlink" title="1. transformers.pipeline"></a>1. <code>transformers.pipeline</code></h1><p>这个管线函数包含三个部分：</p></li><li><p>Tokenizer；</p></li><li><p>一个模型实例；</p></li><li><p>其它增强模型输出的功能。</p><p>它只有一个必需参数 <code>task</code>，接受如下变量之一：</p></li></ol><ul><li>”feature-extraction”</li><li>”sentiment-analysis”</li><li>”ner”</li><li>”question-answering”</li><li>”fill-mask”</li><li>”summarization”</li><li>”translation_xx_to_yy”</li><li>”text-generation”</li></ul><p>这个函数还有其它可选参数，但是我的试用经验是，什么都不要动，使用默认参数即可。</p><p>例子：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span><span class="keyword">from</span> transformers <span class="keyword">import</span> pipeline</span><br><span class="line"></span><br><span class="line"><span class="meta">>>> </span>nlp = pipeline(<span class="string">"sentiment-analysis"</span>)</span><br><span class="line"></span><br><span class="line"><span class="meta">>>> </span>print(nlp(<span class="string">"I hate you"</span>))</span><br><span class="line">[{<span class="string">'label'</span>: <span class="string">'NEGATIVE'</span>, <span class="string">'score'</span>: <span class="number">0.9991129040718079</span>}]</span><br><span class="line"></span><br><span class="line"><span class="meta">>>> </span>print(nlp(<span class="string">"I love you"</span>))</span><br><span class="line">[{<span class="string">'label'</span>: <span class="string">'POSITIVE'</span>, <span class="string">'score'</span>: <span class="number">0.9998656511306763</span>}]</span><br></pre></td></tr></table></figure><h1 id="2-指定预训练模型"><a href="#2-指定预训练模型" class="headerlink" title="2. 指定预训练模型"></a>2. 指定预训练模型</h1><p>这里我们以 Bert 为例。</p><h2 id="2-1-配置-Bert-模型（可选，推荐不使用）transformers-BertConfig"><a href="#2-1-配置-Bert-模型（可选，推荐不使用）transformers-BertConfig" class="headerlink" title="2.1 配置 Bert 模型（可选，推荐不使用）transformers.BertConfig"></a>2.1 配置 Bert 模型（可选，推荐不使用）<code>transformers.BertConfig</code></h2><p><code>transformers.BertConfig</code> 可以自定义 Bert 模型的结构，以下参数都是可选的：</p><ul><li><code>vocab_size</code>：词汇数，默认 30522；</li><li><code>hidden_size</code>：编码器内隐藏层神经元数量，默认 768；</li><li><code>num_hidden_layers</code>：编码器内隐藏层层数，默认 12；</li><li><code>num_attention_heads</code>：编码器内注意力头数，默认 12；</li><li><code>intermediate_size</code>：编码器内全连接层的输入维度，默认 3072；</li><li><code>hidden_act</code>：编码器内激活函数，默认 ‘gelu’，还可为 ‘relu’、’swish’ 或 ‘gelu_new’</li><li><code>hidden_dropout_prob</code>：词嵌入层或编码器的 dropout，默认为 0.1；</li><li><code>attention_probs_dropout_prob</code>：注意力的 dropout，默认为 0.1；</li><li><code>max_position_embeddings</code>：模型使用的最大序列长度，默认为 512；</li><li><code>type_vocab_size</code>：词汇表类别，默认为 2；</li><li><code>initializer_range</code>：神经元权重的标准差，默认为 0.02；</li><li><code>layer_norm_eps</code>：layer normalization 的 epsilon 值，默认为 1e-12.</li></ul><p>使用方法：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">configuration = BertConfig() <span class="comment"># 进行模型的配置，变量为空即使用默认参数</span></span><br><span class="line"></span><br><span class="line">model = BertModel(configuration) <span class="comment"># 使用自定义配置实例化 Bert 模型</span></span><br><span class="line"></span><br><span class="line">configuration = model.config <span class="comment"># 查看模型参数</span></span><br></pre></td></tr></table></figure><h2 id="2-2-分词-transformers-BertTokenizer"><a href="#2-2-分词-transformers-BertTokenizer" class="headerlink" title="2.2 分词 transformers.BertTokenizer"></a>2.2 分词 <code>transformers.BertTokenizer</code></h2><p>所有的 tokenizer 都继承自 <code>transformers.PreTrainedTokenizer</code> 基类，因此有共同的参数和方法实例化的参数有：</p><ul><li><code>model_max_length</code>：可选参数，最大输入长度，默认为 1e30；</li><li><code>padding_side</code>：可选参数，填充的方向，应为 ‘left’ 或 ‘right’；</li><li><code>bos_token</code>：可选参数，每句话的起始标记，默认为 ‘<BOS>‘；</li><li><code>eos_token</code>：可选参数，每句话的结束标记，默认为 ‘<EOS>‘；</li><li><code>unk_token</code>：可选参数，未知的标记，默认为 ‘<UNK>‘；</li><li><code>sep_token</code>：可选参数，分隔标记，默认为 ‘<SEP>‘；</li><li><code>pad_token</code>：可选参数，填充标记，默认为 ‘<PAD>‘；</li><li><code>cls_token</code>：可选参数，分类标记，默认为 ‘<CLS>‘；</li><li><code>mask_token</code>：可选参数，遮盖标记，默认为 ‘<MASK>‘。</li></ul><p>为了演示，我们先实例化一个 <code>BertTokenizer</code>。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">tokenizer = BertTokenizer.from_pretrained(<span class="string">'bert-base-cased'</span>)</span><br></pre></td></tr></table></figure><p>常用的方法有：</p><ul><li><code>from_pretrained(model)</code>：载入预训练词汇表；</li><li><code>tokenizer.tokenize(str)</code>：分词；<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span>tokenizer.tokenize(<span class="string">'Hello word!'</span>)</span><br><span class="line">[<span class="string">'Hello'</span>, <span class="string">'word'</span>, <span class="string">'!'</span>]</span><br></pre></td></tr></table></figure></li><li><code>encode(text, ...)</code>：将文本分词后编码为包含对应 id 的列表；<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span>tokenizer.encode(<span class="string">'Hello word!'</span>)</span><br><span class="line">[<span class="number">101</span>, <span class="number">8667</span>, <span class="number">1937</span>, <span class="number">106</span>, <span class="number">102</span>]</span><br></pre></td></tr></table></figure></li><li><code>encode_plus(text, ...)</code>：将文本分词后创建一个包含对应 id，token 类型及是否遮盖的词典；<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">tokenizer.encode_plus(<span class="string">'Hello world!'</span>)</span><br><span class="line">{<span class="string">'input_ids'</span>: [<span class="number">101</span>, <span class="number">8667</span>, <span class="number">1937</span>, <span class="number">106</span>, <span class="number">102</span>], <span class="string">'token_type_ids'</span>: [<span class="number">0</span>, <span class="number">0</span>, <span class="number">0</span>, <span class="number">0</span>, <span class="number">0</span>], <span class="string">'attention_mask'</span>: [<span class="number">1</span>, <span class="number">1</span>, <span class="number">1</span>, <span class="number">1</span>, <span class="number">1</span>]}</span><br></pre></td></tr></table></figure></li><li><code>convert_ids_to_tokens(ids, skip_special_tokens)</code>：将 id 映射为 token；<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span>tokenizer.convert_ids_to_tokens(tokens)</span><br><span class="line">[<span class="string">'[CLS]'</span>, <span class="string">'Hello'</span>, <span class="string">'word'</span>, <span class="string">'!'</span>, <span class="string">'[SEP]'</span>]</span><br></pre></td></tr></table></figure></li><li><code>decode(token_ids)</code>：将 id 解码；<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span>tokenizer.decode(tokens)</span><br><span class="line"><span class="string">'[CLS] Hello word! [SEP]'</span></span><br></pre></td></tr></table></figure></li><li><code>convert_tokens_to_ids(tokens)</code>：将 token 映射为 id。<figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span>tokenizer.convert_tokens_to_ids([<span class="string">'[CLS]'</span>, <span class="string">'Hello'</span>, <span class="string">'word'</span>, <span class="string">'!'</span>, <span class="string">'[SEP]'</span>])</span><br><span class="line">[<span class="number">101</span>, <span class="number">8667</span>, <span class="number">1937</span>, <span class="number">106</span>, <span class="number">102</span>]</span><br></pre></td></tr></table></figure><h2 id="2-3-使用预训练模型"><a href="#2-3-使用预训练模型" class="headerlink" title="2.3 使用预训练模型"></a>2.3 使用预训练模型</h2>根据任务的需要，既可以选择没有为指定任务 finetune 的模型如 <code>transformers.BertModel</code>，也可以选择为指定任务 finetune 之后的模型如 <code>transformers.BertForSequenceClassification</code>。一共有 6 个指定的任务类型：</li><li><code>transformers.BertForMaskedLM</code>：语言模型；</li><li><code>transformers.BertForNextSentencePrediction</code>：判断下一句话是否与上一句有关；</li><li><code>transformers.BertForSequenceClassification</code>：序列分类如 GLUE；</li><li><code>transformers.BertForMultipleChoice</code>：文本分类；</li><li><code>transformers.BertForTokenClassification</code>：token 分类如 NER，</li><li><code>transformers.BertForQuestionAnswering</code>；问答。</li></ul><h1 id="3-使用-AutoModels"><a href="#3-使用-AutoModels" class="headerlink" title="3. 使用 AutoModels"></a>3. 使用 <code>AutoModels</code></h1><p>使用 <code>AutoModels</code> 与上面的指定模型进行预训练大同小异，只不过是另一种方式加载模型而已。</p><h2 id="3-1-加载自动配置-transformers-AutoConfig"><a href="#3-1-加载自动配置-transformers-AutoConfig" class="headerlink" title="3.1 加载自动配置 transformers.AutoConfig"></a>3.1 加载自动配置 <code>transformers.AutoConfig</code></h2><p>使用类方法 <code>from_pretrained</code> 加载模型配置，参数既可以为模型名称，也可以为具体文件。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">config = AutoConfig.from_pretrained(<span class="string">'bert-base-uncased'</span>)</span><br><span class="line"><span class="comment"># 或者直接加载模型文件</span></span><br><span class="line">config = AutoConfig.from_pretrained(<span class="string">'./test/bert_saved_model/'</span>)</span><br></pre></td></tr></table></figure><h2 id="3-2-加载分词器-transformers-AutoTokenizer"><a href="#3-2-加载分词器-transformers-AutoTokenizer" class="headerlink" title="3.2 加载分词器 transformers.AutoTokenizer"></a>3.2 加载分词器 <code>transformers.AutoTokenizer</code></h2><p>与上面的 <code>BertTokenizer</code> 非常相似，也是使用 <code>from_pretrained</code> 类方法加载预训练模型。</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">tokenizer = AutoTokenizer.from_pretrained(<span class="string">'bert-base-uncased'</span>)</span><br><span class="line"><span class="comment"># 或者直接加载模型文件</span></span><br><span class="line">tokenizer = AutoTokenizer.from_pretrained(<span class="string">'./test/bert_saved_model/'</span>)</span><br></pre></td></tr></table></figure><h2 id="3-3-加载模型-transformers-AutoModel"><a href="#3-3-加载模型-transformers-AutoModel" class="headerlink" title="3.3 加载模型 transformers.AutoModel"></a>3.3 加载模型 <code>transformers.AutoModel</code></h2><p>可以使用 <code>from_pretrained</code> 加载预训练模型：</p><figure class="highlight py"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">model = AutoModel.from_pretrained(<span class="string">'bert-base-uncased'</span>)</span><br><span class="line"><span class="comment"># 或者直接加载模型文件</span></span><br><span class="line">model = AutoModel.from_pretrained(<span class="string">'./test/bert_model/'</span>)</span><br></pre></td></tr></table></figure><p>选好了预训练模型以后，只需要给模型接一个全连接层，这个神经网络就搭好了（当然可以根据需要添加更复杂的结构）。是不是香？</p><p>欢迎关注我的微信公众号“花解语 NLP”：<br><img src="https://img-blog.csdnimg.cn/20200514100635366.jpg#pic_center" alt=""></p>]]></content>
<summary type="html">
<p> 严格意义上讲 transformers 并不是 PyTorch 的一部分，然而 transformers 与 PyTorch 或 TensorFlow 结合的太紧密了，而且可以把 transformers 看成是 PyTorch 或 TensorFlow 的延伸，所以也在这里一并讨论了。<br>
</summary>
<category term="deep learning" scheme="https://vincent507cpu.github.io/tags/deep-learning/"/>
<category term="PyTorch" scheme="https://vincent507cpu.github.io/tags/PyTorch/"/>
</entry>
</feed>