Skip to content

Commit

Permalink
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix(qlinear): speed up weights transfer
Browse files Browse the repository at this point in the history
By instantiating the QLinear weights directly on the device, it saves
two copies: from device to cpu and back.
dacorvo committed Feb 13, 2024
1 parent f9e7779 commit f56d0df
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion bench/generation/benchmark.py
Original file line number Diff line number Diff line change
@@ -173,8 +173,9 @@ def main():
# Very simple calibration to avoid completely off results
with Calibration():
generate(model, tokenizer, device, prompt=CALIBRATION_PROMPT)
print("Freezing")
freeze(model)
print(f"Finished: {time.time()-start}")
print(f"Finished: {time.time()-start:.2f}")

memory = get_device_memory(device)
if memory is not None:

0 comments on commit f56d0df

Please sign in to comment.