-
Notifications
You must be signed in to change notification settings - Fork 14
/
Copy pathindex.html
197 lines (178 loc) · 14.8 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
<!DOCTYPE html>
<html lang="en-US"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<!-- Begin Jekyll SEO tag v2.7.1 -->
<title>MULTI-TASK DEEP RESIDUAL ECHO SUPPRESSION WITH ECHO-AWARE LOSS</title>
<meta name="generator" content="Jekyll v3.9.0">
<meta property="og:title" content="TODO: title">
<meta property="og:locale" content="en_US">
<link rel="canonical" href="https://echocatzh.github.io/GFTNN/">
<meta property="og:url" content="https://echocatzh.github.io/GFTNN/">
<meta name="twitter:card" content="summary">
<!-- End Jekyll SEO tag -->
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="theme-color" content="#157878">
<link rel="stylesheet" href="./GFTNN_files/style.css">
</head>
<body data-new-gr-c-s-check-loaded="14.1001.0" data-gr-ext-installed="">
<section class="page-header">
</section>
<section class="main-content">
<h1 id=""><center>MULTI-TASK DEEP RESIDUAL ECHO SUPPRESSION WITH ECHO-AWARE LOSS
</center></h1>
<center> <a href="https://github.com/echocatzh">Shimin Zhang</a><sup>1</sup>, Ziteng Wang, Jiayao Sun<sup>1</sup>, Yihui Fu<sup>1</sup>, Biao Tian, Qiang Fu, Lei Xie<sup>1</sup></center>
<center> <sup>1</sup>Audio, Speech and Language Processing Group (ASLP@NPU), Northwestern Polytechnical University, Xi'an, China</center>
<h2>0. Contents</h2>
<ol>
<li><a href="https://echocatzh.github.io/GFTNN/#abstract">Abstract</a></li>
<li><a href="https://echocatzh.github.io/GFTNN/#near-end-single-talk">Demos -- ICASSP 2022 AEC-Challenge blind test (near-end single-talk)</a></li>
<li><a href="https://echocatzh.github.io/GFTNN/#far-end-single-talk">Demos -- ICASSP 2022 AEC-Challenge blind test (far-end single-talk)</a></li>
<li><a href="https://echocatzh.github.io/GFTNN/#double-talk">Demos -- ICASSP 2022 AEC-Challenge blind test (double-talk)</a></li>
<li><a href="https://echocatzh.github.io/GFTNN/#full">Demos -- Full 300 clips of ICASSP 2022 AEC-Challenge blind test far-end single-talk scenario</a></li>
</ol>
<br><br>
<h2 id="abstract">1. Abstract<a name="abstract"></a></h2>
<p>
This paper introduces the NWPU Team's entry to the ICASSP 2022 AEC Challenge. We take a hybrid approach that cascades a linear AEC with a neural post-filter. The former is used to deal with the linear echo components while the latter suppresses the residual non-linear echo components. We use gated convolutional F-T-LSTM neural network (GFTNN) as the backbone and shape the post-filter by a multi-task learning (MTL) framework, where a voice activity detection (VAD) module is adopted as an auxiliary task along with echo suppression, with the aim to avoid over suppression that may cause speech distortion. Moreover, we adopt an echo-aware loss function, where the mean square error (MSE) loss can be optimized particularly for every time-frequency bins (TF-bins) according to the signal-to-echo ratio (SER), leading to further suppression on the echo. Extensive ablation study shows that the time delay estimation (TDE) module in neural post-filter leads to better perceptual quality, and an adaptive filter with better convergence will bring consistent performance gain for the post-filter. Besides, we find that using the linear echo as the input of our neural post-filter is a better choice than using the reference signal directly. In the ICASSP 2022 AEC-Challenge, our approach has ranked the 1st place on word acceptance rate (WAcc) (0.817) and the 3rd place on both mean opinion score (MOS) (4.502) and the final score (0.864).<br>
<center><img src="./GFTNN_files/aec_flow.png" style="height:120px"/></center>
<center><img src="./GFTNN_files/system.png" style="height:360px"/></center>
<br><br>
<h2>2. Demos -- Near-end single-talk<a name="near-end-single-talk"></a></h2>
<table>
<thead>
<tr>
<th style="text-align: center"><strong>Models</strong></th>
<th style="text-align: center"><strong>Sample 1</strong></th>
<th style="text-align: center"><strong>Sample 2</strong></th>
<th style="text-align: center"><strong>Sample 3</strong></th>
<th style="text-align: center"><strong>Sample 4</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left">Microphone</td>
<td style="text-align: left"><audio src="samples/mic/near-end-single-talk/0e9a9c23-4867-41d4-85b3-d3236ad20f6c_nearend-singletalk_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/mic/near-end-single-talk/MXn4F3uO6EaQ0NJtITuYbA_nearend-singletalk_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/mic/near-end-single-talk/1fd9bea8-3f81-45db-8a63-f73b34d0a078_nearend-singletalk_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/mic/near-end-single-talk/zfl8eeOVGUWdMx7jzvNRBg_nearend-singletalk_mic.wav" controls="" preload=""></audio></td>
</tr>
<tr>
<td style="text-align: left">Reference</td>
<td style="text-align: left"><audio src="samples/ref/near-end-single-talk/0e9a9c23-4867-41d4-85b3-d3236ad20f6c_nearend-singletalk_lpb.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/ref/near-end-single-talk/MXn4F3uO6EaQ0NJtITuYbA_nearend-singletalk_lpb.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/ref/near-end-single-talk/1fd9bea8-3f81-45db-8a63-f73b34d0a078_nearend-singletalk_lpb.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/ref/near-end-single-talk/zfl8eeOVGUWdMx7jzvNRBg_nearend-singletalk_lpb.wav" controls="" preload=""></audio></td>
</tr>
<tr>
<td style="text-align: left">Baseline</td>
<td style="text-align: left"><audio src="samples/baseline/near-end-single-talk/0e9a9c23-4867-41d4-85b3-d3236ad20f6c_nearend-singletalk_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/baseline/near-end-single-talk/MXn4F3uO6EaQ0NJtITuYbA_nearend-singletalk_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/baseline/near-end-single-talk/1fd9bea8-3f81-45db-8a63-f73b34d0a078_nearend-singletalk_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/baseline/near-end-single-talk/zfl8eeOVGUWdMx7jzvNRBg_nearend-singletalk_mic.wav" controls="" preload=""></audio></td>
</tr>
<tr>
<td style="text-align: left">GFTNN-VAD-L</td>
<td style="text-align: left"><audio src="samples/gftnn/near-end-single-talk/0e9a9c23-4867-41d4-85b3-d3236ad20f6c_nearend-singletalk_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/gftnn/near-end-single-talk/MXn4F3uO6EaQ0NJtITuYbA_nearend-singletalk_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/gftnn/near-end-single-talk/1fd9bea8-3f81-45db-8a63-f73b34d0a078_nearend-singletalk_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/gftnn/near-end-single-talk/zfl8eeOVGUWdMx7jzvNRBg_nearend-singletalk_mic.wav" controls="" preload=""></audio></td>
</tr>
</tbody>
</table>
<br><br>
<h2>3. Demos -- Far-end single-talk<a name="far-end-single-talk"></a></h2>
<table>
<thead>
<tr>
<th style="text-align: center"><strong>Models</strong></th>
<th style="text-align: center"><strong>Sample 1</strong></th>
<th style="text-align: center"><strong>Sample 2</strong></th>
<th style="text-align: center"><strong>Sample 3</strong></th>
<th style="text-align: center"><strong>Sample 4</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left">Microphone</td>
<td style="text-align: left"><audio src="samples/mic/far-end-single-talk/0sKU1JZ9rEe5nPleS6JiFw_farend-singletalk-with-movement_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/mic/far-end-single-talk/dFw0RXbAc00Royb0kKxm0Q_farend-singletalk-with-movement_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/mic/far-end-single-talk/QFkrZZDXBUKU9Dq67EHlyg_farend-singletalk_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/mic/far-end-single-talk/yciM00EmXkuhWoy7fUbvIg_farend-singletalk-with-movement_mic.wav" controls="" preload=""></audio></td>
</tr>
<tr>
<td style="text-align: left">Reference</td>
<td style="text-align: left"><audio src="samples/ref/far-end-single-talk/0sKU1JZ9rEe5nPleS6JiFw_farend-singletalk-with-movement_lpb.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/ref/far-end-single-talk/dFw0RXbAc00Royb0kKxm0Q_farend-singletalk-with-movement_lpb.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/ref/far-end-single-talk/QFkrZZDXBUKU9Dq67EHlyg_farend-singletalk_lpb.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/ref/far-end-single-talk/yciM00EmXkuhWoy7fUbvIg_farend-singletalk-with-movement_lpb.wav" controls="" preload=""></audio></td>
</tr>
<tr>
<td style="text-align: left">Baseline</td>
<td style="text-align: left"><audio src="samples/baseline/far-end-single-talk/0sKU1JZ9rEe5nPleS6JiFw_farend-singletalk-with-movement_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/baseline/far-end-single-talk/dFw0RXbAc00Royb0kKxm0Q_farend-singletalk-with-movement_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/baseline/far-end-single-talk/QFkrZZDXBUKU9Dq67EHlyg_farend-singletalk_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/baseline/far-end-single-talk/yciM00EmXkuhWoy7fUbvIg_farend-singletalk-with-movement_mic.wav" controls="" preload=""></audio></td>
</tr>
<tr>
<td style="text-align: left">GFTNN-VAD-L</td>
<td style="text-align: left"><audio src="samples/gftnn/far-end-single-talk/0sKU1JZ9rEe5nPleS6JiFw_farend-singletalk-with-movement_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/gftnn/far-end-single-talk/dFw0RXbAc00Royb0kKxm0Q_farend-singletalk-with-movement_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/gftnn/far-end-single-talk/QFkrZZDXBUKU9Dq67EHlyg_farend-singletalk_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/gftnn/far-end-single-talk/yciM00EmXkuhWoy7fUbvIg_farend-singletalk-with-movement_mic.wav" controls="" preload=""></audio></td>
</tr>
</tbody>
</table>
<br><br>
<h2>4. Demos -- Double-talk<a name="double-talk"></a></h2>
<table>
<thead>
<tr>
<th style="text-align: center"><strong>Models</strong></th>
<th style="text-align: center"><strong>Sample 1</strong></th>
<th style="text-align: center"><strong>Sample 2</strong></th>
<th style="text-align: center"><strong>Sample 3</strong></th>
<th style="text-align: center"><strong>Sample 4</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left">Microphone</td>
<td style="text-align: left"><audio src="samples/mic/double-talk/655836a6-d361-488f-aafb-745d8df0e1eb_doubletalk_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/mic/double-talk/ZZaPBCFUG0O4YPSJ3Ra72g_doubletalk-with-movement_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/mic/double-talk/BxSd4oDC4EKg4S5T0E0Zyw_doubletalk_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/mic/double-talk/s7c6fM3Qt0KlZ2cJono40w_doubletalk-with-movement_mic.wav" controls="" preload=""></audio></td>
</tr>
<tr>
<td style="text-align: left">Reference</td>
<td style="text-align: left"><audio src="samples/ref/double-talk/655836a6-d361-488f-aafb-745d8df0e1eb_doubletalk_lpb.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/ref/double-talk/ZZaPBCFUG0O4YPSJ3Ra72g_doubletalk-with-movement_lpb.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/ref/double-talk/BxSd4oDC4EKg4S5T0E0Zyw_doubletalk_lpb.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/ref/double-talk/s7c6fM3Qt0KlZ2cJono40w_doubletalk-with-movement_lpb.wav" controls="" preload=""></audio></td>
</tr>
<tr>
<td style="text-align: left">Baseline</td>
<td style="text-align: left"><audio src="samples/baseline/double-talk/655836a6-d361-488f-aafb-745d8df0e1eb_doubletalk_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/baseline/double-talk/ZZaPBCFUG0O4YPSJ3Ra72g_doubletalk-with-movement_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/baseline/double-talk/BxSd4oDC4EKg4S5T0E0Zyw_doubletalk_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/baseline/double-talk/s7c6fM3Qt0KlZ2cJono40w_doubletalk-with-movement_mic.wav" controls="" preload=""></audio></td>
</tr>
<tr>
<td style="text-align: left">GFTNN-VAD-L</td>
<td style="text-align: left"><audio src="samples/gftnn/double-talk/655836a6-d361-488f-aafb-745d8df0e1eb_doubletalk_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/gftnn/double-talk/ZZaPBCFUG0O4YPSJ3Ra72g_doubletalk-with-movement_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/gftnn/double-talk/BxSd4oDC4EKg4S5T0E0Zyw_doubletalk_mic.wav" controls="" preload=""></audio></td>
<td style="text-align: left"><audio src="samples/gftnn/double-talk/s7c6fM3Qt0KlZ2cJono40w_doubletalk-with-movement_mic.wav" controls="" preload=""></audio></td>
</tr>
</tbody>
</table>
<h2>5. Demos -- Full 300 far-end single-talk clips<a name="full"></a></h2>
<li><a href="samples/farend-singletalk.zip">farend-singletalk.zip(17MB)</a></li>
<br>
It should be noted that <b> 4ecd5889-aa9e-4c02-a81a-ff87ad6e9c38_farend-singletalk_mic.wav</b>, <b> 51bdf2f1-bb37-4eba-a5ee-39102a0fbb9e_farend-singletalk-with-movement_mic.wav </b> and <b> f783b002-4a43-4e89-ad6e-b9f999e8e39f_farend-singletalk_mic.wav </b> belong double-talk scenario, not residual echo.
<br>
The full double-talk clips and the near-end single-talk clips have 800M and 200M respectively, so they are not put here.
</table>
<footer class="site-footer">
<span class="site-footer-credits">This page was generated by <a href="https://pages.github.com/">GitHub Pages</a>.</span>
</footer>
</section>
</body></html>