Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

prompt sharing for faster page attn #28

Open
mmoskal opened this issue Dec 22, 2023 · 0 comments
Open

prompt sharing for faster page attn #28

mmoskal opened this issue Dec 22, 2023 · 0 comments
Assignees
@mmoskal
Labels
rLLM

Comments

@mmoskal
Copy link
Member

mmoskal commented Dec 22, 2023

Right now (validate this!) the paged attn kernel doesn't take advantage of the fact that a significant part of the prompt may be shared between many queries - probably the kernel could be modified to only read these prompt entries once.
@mmoskal mmoskal added the rLLM label Dec 22, 2023
@mmoskal mmoskal self-assigned this Dec 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mmoskal mmoskal

Labels
rLLM
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mmoskal