LLM optimization documentation fixes and updates. #28212
Merged
+18
−30
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AlexKoff88
requested review from
kblaszczak-intel and
tsavina
and removed request for
a team
|
@kblaszczak-intel, @tsavina, can you take a look? |
kblaszczak-intel
approved these changes
Jan 31, 2025
| For instance the 7 billion parameter Llama 2 model can be reduced | ||
| from about 25GB to 4GB using 4-bit weight compression. | ||
| For instance the 8 billion parameter Llama 3 model can be reduced | ||
| from about 16.1 GB to 4.8 GB using 4-bit weight quantization on top of bfloat16 model. |
There was a problem hiding this comment.
Suggested change
| from about 16.1 GB to 4.8 GB using 4-bit weight quantization on top of bfloat16 model. | |
| from about 16.1 GB to 4.8 GB using 4-bit weight quantization on top of a bfloat16 model. |
| compression may result in more accuracy reduction than with larger models. | ||
| Therefore, weight compression is recommended for use with LLMs only. | ||
|
|
||
| LLMs and other GenAI models that require |
There was a problem hiding this comment.
Suggested change
| LLMs and other GenAI models that require | |
| LLMs and other generative AI models that require |
Comment on lines
+140
to
+141
| NNCF allows stacking the supported optimization methods. For example, AWQ, Scale Estimation | ||
| and GPTQ methods can be enabled all together to achieve better accuracy. |
There was a problem hiding this comment.
Suggested change
| NNCF allows stacking the supported optimization methods. For example, AWQ, Scale Estimation | |
| and GPTQ methods can be enabled all together to achieve better accuracy. | |
| NNCF enables you to stack the supported optimization methods. For example, AWQ, Scale Estimation | |
| and GPTQ methods may be enabled all together to achieve better accuracy. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.