[HWConfig] narrow_range parameter is introduced in hardware config #3196
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
NNCF PT
daniil-lyakhov
force-pushed
the
dl/narrow_range_to_qconfig
branch
4 times, most recently
from
07a6520 to
a9a0c19
Compare
daniil-lyakhov
force-pushed
the
dl/narrow_range_to_qconfig
branch
from
7134fcb to
c8b1761
Compare
github-actions
bot
added
the
NNCF OpenVINO
daniil-lyakhov
force-pushed
the
dl/narrow_range_to_qconfig
branch
4 times, most recently
from
0b064a7 to
0733ac3
Compare
daniil-lyakhov
force-pushed
the
dl/narrow_range_to_qconfig
branch
from
0733ac3 to
10267b0
Compare
alexsu52
reviewed
Jan 28, 2025
| @@ -288,7 +293,7 @@ | |||
| { | |||
| "type": "Embedding", | |||
| "quantization": { | |||
| "weights": ["q8_w_sym", "q8_w_asym"] | |||
| "weights": ["q8_w_sym", "q8_w_asym", "q8_a", "q8_a_sym", "q8_a_ch"] | |||
There was a problem hiding this comment.
I'm not sure that LPT supports q8_a_ch for embedding. Please, double check.
As far as I understand, this is applicable only for embedding -> depthwise convolution sub-graph. However, I did not meet such sub-graph in the real word.
Suggested change
| "weights": ["q8_w_sym", "q8_w_asym", "q8_a", "q8_a_sym", "q8_a_ch"] | |
| "weights": ["q8_w_sym", "q8_w_asym", "q8_a", "q8_a_ch"] |
There was a problem hiding this comment.
I'm not sure that even a per-tensor quantization scheme is applicable here for the Embedding layer since it contains sensitive weights. We need to verify that this scheme does not introduce performance/accuracy regressions on the known cases.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
versa model validation after the rebase:
| versa INT8 | roc_auc_score | Embedding weights config | narrow range | Throughput, FPS | Gather execType | Convs execType |
|---|---|---|---|---|---|---|
| develop | 91.54 | 'B:8 M:S SGN:S PC:N' | TRUE | 642.18 | jit_avx512_i8 | brgconv_avx512_i8 |
| daniil-lyakhov:dl/narrow_range_to_qconfig | 91.54 | B:8 M:S SGN:S PC:N NR:N' | FALSE | 641.19 | jit_avx512_i8 | brgconv_avx512_i8 |
| /* | ||
| * Narrow range: should NNCF use 2**num_bits quants or 2**num_bits - 1 | ||
| */ | ||
| "narrow_range": false | ||
| }, | ||
| "q8_sym_tnr_-128_127": { // Alias name for set of hyperparameters | ||
| "bits": 8, // Number of quantization bits | ||
| "mode": "symmetric", // Quantization mode | ||
| "granularity": "pertensor", // Granularity: one scale for output tensor | ||
| "level_low": -128, // Low quantization level |
There was a problem hiding this comment.
Does NNCF support for level_low and level_high?
There was a problem hiding this comment.
Isn't it redundant, since other params define how to calculate them?
There was a problem hiding this comment.
KodiaqQ
reviewed
Jan 28, 2025
KodiaqQ
reviewed
Jan 28, 2025
KodiaqQ
reviewed
Jan 28, 2025
daniil-lyakhov
force-pushed
the
dl/narrow_range_to_qconfig
branch
from
a2e42d5 to
7257b2e
Compare
daniil-lyakhov
commented
Jan 30, 2025
daniil-lyakhov
force-pushed
the
dl/narrow_range_to_qconfig
branch
2 times, most recently
from
4b0dc46 to
4d3a892
Compare
daniil-lyakhov
force-pushed
the
dl/narrow_range_to_qconfig
branch
2 times, most recently
from
987c51d to
ea5d0fd
Compare
daniil-lyakhov
commented
Jan 31, 2025
Comment on lines
-185
to
188
| "input_low": -0.9350724220275879, | ||
| "input_low": -0.9424352049827576, | ||
| "input_high": 0.9350724220275879, | ||
| "output_low": -0.9350724220275879, | ||
| "output_low": -0.9424352049827576, | ||
| "output_high": 0.9350724220275879 |
daniil-lyakhov
force-pushed
the
dl/narrow_range_to_qconfig
branch
from
ea5d0fd to
a85f7b9
Compare
daniil-lyakhov
force-pushed
the
dl/narrow_range_to_qconfig
branch
from
e7a1453 to
cfedaf7
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
On top of #3232
Changes
Reason for changes
Related tickets
Tests
post_training_quantization/586/ - Passed
job/weekly/job/ubuntu20_eval/245/ (+ job/ubuntu20_eval/246/) Passed
eval_tf/461/ - Passed
torch_weekly/100/ - Passed
job/nightly/job/torch_nightly/444/ - Passed