Skip to content

Commit

Permalink
Results from self hosted Github actions
Browse files Browse the repository at this point in the history
  • Loading branch information
@arjunsuresh
arjunsuresh committed Jan 16, 2025
1 parent c28e837 commit 9c47c55
Show file tree
Hide file tree
Showing 14 changed files with 428 additions and 428 deletions.
2 changes: 1 addition & 1 deletion ...ns/measurements/gh_action-reference-cpu-pytorch_v2.5.1-default_config/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
| Model | Scenario | Accuracy | Throughput | Latency (in ms) |
|---------------|------------|------------------------------------|--------------|-------------------|
| llama2-70b-99 | offline | (61.7021, 37.9679, 39.3617, 610.0) | 0.358 | - |
| llama2-70b-99 | offline | (61.7021, 37.9679, 39.3617, 610.0) | 0.351 | - |
2 changes: 1 addition & 1 deletion ...ion-reference-cpu-pytorch_v2.5.1-default_config/llama2-70b-99/offline/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -107,4 +107,4 @@ Model Precision: fp32
`TOKENS_PER_SAMPLE`: `610.0`, Required accuracy for closed division `>= 265.005` and `<= 323.895`

### Performance Results
`Samples per second`: `0.35789`
`Samples per second`: `0.351429`
10 changes: 5 additions & 5 deletions ...on-reference-cpu-pytorch_v2.5.1-default_config/llama2-70b-99/offline/accuracy_console.out
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
INFO:datasets:PyTorch version 2.5.1+cpu available.
Loading dataset...
Finished loading dataset.
Loading checkpoint shards: 0%| | 0/15 [00:00<?, ?it/s]Loading checkpoint shards: 7%|▋ | 1/15 [00:02<00:29, 2.10s/it]Loading checkpoint shards: 13%|█▎ | 2/15 [00:04<00:25, 1.99s/it]Loading checkpoint shards: 20%|██ | 3/15 [00:06<00:24, 2.03s/it]Loading checkpoint shards: 27%|██▋ | 4/15 [00:07<00:21, 1.96s/it]Loading checkpoint shards: 33%|███▎ | 5/15 [00:09<00:19, 1.91s/it]Loading checkpoint shards: 40%|████ | 6/15 [00:11<00:16, 1.87s/it]Loading checkpoint shards: 47%|████▋ | 7/15 [00:13<00:14, 1.84s/it]Loading checkpoint shards: 53%|█████▎ | 8/15 [00:15<00:14, 2.05s/it]Loading checkpoint shards: 60%|██████ | 9/15 [00:18<00:12, 2.10s/it]Loading checkpoint shards: 67%|██████▋ | 10/15 [00:20<00:11, 2.21s/it]Loading checkpoint shards: 73%|███████▎ | 11/15 [00:22<00:08, 2.18s/it]Loading checkpoint shards: 80%|████████ | 12/15 [00:24<00:06, 2.07s/it]Loading checkpoint shards: 87%|████████▋ | 13/15 [00:26<00:03, 1.99s/it]Loading checkpoint shards: 93%|█████████▎| 14/15 [00:35<00:04, 4.12s/it]Loading checkpoint shards: 100%|██████████| 15/15 [00:35<00:00, 2.94s/it]Loading checkpoint shards: 100%|██████████| 15/15 [00:35<00:00, 2.37s/it]
Loading checkpoint shards: 0%| | 0/15 [00:00<?, ?it/s]Loading checkpoint shards: 7%|▋ | 1/15 [00:02<00:28, 2.00s/it]Loading checkpoint shards: 13%|█▎ | 2/15 [00:03<00:24, 1.91s/it]Loading checkpoint shards: 20%|██ | 3/15 [00:05<00:23, 1.92s/it]Loading checkpoint shards: 27%|██▋ | 4/15 [00:07<00:20, 1.88s/it]Loading checkpoint shards: 33%|███▎ | 5/15 [00:09<00:18, 1.84s/it]Loading checkpoint shards: 40%|████ | 6/15 [00:11<00:16, 1.81s/it]Loading checkpoint shards: 47%|████▋ | 7/15 [00:12<00:14, 1.80s/it]Loading checkpoint shards: 53%|█████▎ | 8/15 [00:14<00:13, 1.88s/it]Loading checkpoint shards: 60%|██████ | 9/15 [00:17<00:11, 1.98s/it]Loading checkpoint shards: 67%|██████▋ | 10/15 [00:19<00:10, 2.02s/it]Loading checkpoint shards: 73%|███████▎ | 11/15 [00:21<00:08, 2.02s/it]Loading checkpoint shards: 80%|████████ | 12/15 [00:23<00:05, 1.97s/it]Loading checkpoint shards: 87%|████████▋ | 13/15 [00:24<00:03, 1.92s/it]Loading checkpoint shards: 93%|█████████▎| 14/15 [00:30<00:02, 2.95s/it]Loading checkpoint shards: 100%|██████████| 15/15 [00:30<00:00, 2.12s/it]Loading checkpoint shards: 100%|██████████| 15/15 [00:30<00:00, 2.03s/it]
INFO:Llama-70B-MAIN:Starting Benchmark run
/home/cmuser/venv/cm/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:628: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
warnings.warn(
Expand All @@ -22,7 +22,7 @@ IssueQuery started with 1 samples
IssueQuery done
Saving outputs to run_outputs/q0.pkl
Samples run: 1
BatchMaker time: 0.0216982364654541
Inference time: 1001.4157693386078
Postprocess time: 0.07915091514587402
==== Total time: 1001.5166184902191
BatchMaker time: 0.022020578384399414
Inference time: 878.0318548679352
Postprocess time: 0.03908991813659668
==== Total time: 878.0929653644562
Expand Down
52 changes: 26 additions & 26 deletions ...s/gh_action-reference-cpu-pytorch_v2.5.1-default_config/llama2-70b-99/offline/cm-deps.mmd
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,15 +13,15 @@ graph TD
get-mlperf-inference-src,4b57186581024797 --> get,git,repo,_branch.master,_repo.https://github.com/mlcommons/inference
get-mlperf-inference-utils,e341e5f86d8342e5 --> get,mlperf,inference,src
app-mlperf-inference,d775cac873ee4231_(_reference,_llama2-70b-99,_pytorch,_cpu,_test,_r5.0-dev_default,_bfloat16,_offline_) --> get,mlperf,inference,utils
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> detect,os
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> detect,cpu
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,sys-utils-cm
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,python
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> detect,cpu
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,sys-utils-cm
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,python
get-generic-python-lib,94b62a682bc44791_(_torch_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,generic-python-lib,_torch
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,generic-python-lib,_torch
get-generic-python-lib,94b62a682bc44791_(_torchvision_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,generic-python-lib,_torchvision
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,generic-python-lib,_torchvision
get-ml-model-huggingface-zoo,53cf8252a443446a_(_clone-repo,_model-stub.meta-llama/Llama-2-70b-chat-hf_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_huggingface_hub_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
Expand All @@ -32,7 +32,7 @@ graph TD
get-ml-model-huggingface-zoo,53cf8252a443446a_(_clone-repo,_model-stub.meta-llama/Llama-2-70b-chat-hf_) --> get,generic-python-lib,_huggingface_hub
get-ml-model-huggingface-zoo,53cf8252a443446a_(_clone-repo,_model-stub.meta-llama/Llama-2-70b-chat-hf_) --> get,git,repo,_lfs,_repo.https://huggingface.co/meta-llama/Llama-2-70b-chat-hf
get-ml-model-llama2,5db97be9f61244c6_(_pytorch_) --> get,ml-model,huggingface,zoo,_clone-repo,_model-stub.meta-llama/Llama-2-70b-chat-hf
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,ml-model,llama2,raw,_pytorch
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,ml-model,llama2,raw,_pytorch
get-preprocessed-dataset-openorca,5614c39cb1564d72_(_validation,_mlcommons_) --> get,sys-utils-cm
get-preprocessed-dataset-openorca,5614c39cb1564d72_(_validation,_mlcommons_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_package.pyarrow_) --> get,python3
Expand All @@ -42,7 +42,7 @@ graph TD
get-generic-python-lib,94b62a682bc44791_(_package.transformers_) --> get,python3
get-preprocessed-dataset-openorca,5614c39cb1564d72_(_validation,_mlcommons_) --> get,generic-python-lib,_package.transformers
get-preprocessed-dataset-openorca,5614c39cb1564d72_(_validation,_mlcommons_) --> download-and-extract,_rclone,_url.mlc-inference:mlcommons-inference-wg-public/open_orca
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,preprocessed,dataset,openorca,_validation,_mlcommons
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,preprocessed,dataset,openorca,_validation,_mlcommons
generate-mlperf-inference-user-conf,3af4475745964b93 --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
generate-mlperf-inference-user-conf,3af4475745964b93 --> detect,cpu
Expand All @@ -53,40 +53,40 @@ graph TD
generate-mlperf-inference-user-conf,3af4475745964b93 --> get,mlcommons,inference,src
get-mlperf-inference-sut-configs,c2fbf72009e2445b --> get,cache,dir,_name.mlperf-inference-sut-configs
generate-mlperf-inference-user-conf,3af4475745964b93 --> get,sut,configs
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> generate,user-conf,mlperf,inference
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,loadgen
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> generate,user-conf,mlperf,inference
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,loadgen
get-mlperf-inference-src,4b57186581024797 --> detect,os
get-mlperf-inference-src,4b57186581024797 --> get,python3
get-mlperf-inference-src,4b57186581024797 --> get,git,repo,_branch.master,_repo.https://github.com/mlcommons/inference
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,mlcommons,inference,src
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,mlcommons,inference,src
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,mlcommons,inference,src
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,mlcommons,inference,src
get-generic-python-lib,94b62a682bc44791_(_package.psutil_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,generic-python-lib,_package.psutil
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,generic-python-lib,_package.psutil
get-generic-python-lib,94b62a682bc44791_(_package.transformers_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,generic-python-lib,_package.transformers
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,generic-python-lib,_package.transformers
get-generic-python-lib,94b62a682bc44791_(_package.datasets_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,generic-python-lib,_package.datasets
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,generic-python-lib,_package.datasets
get-generic-python-lib,94b62a682bc44791_(_package.sentencepiece_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,generic-python-lib,_package.sentencepiece
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,generic-python-lib,_package.sentencepiece
get-generic-python-lib,94b62a682bc44791_(_package.protobuf_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,generic-python-lib,_package.protobuf
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,generic-python-lib,_package.protobuf
get-generic-python-lib,94b62a682bc44791_(_package.accelerate_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,generic-python-lib,_package.accelerate
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,generic-python-lib,_package.accelerate
get-generic-python-lib,94b62a682bc44791_(_package.absl-py_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,generic-python-lib,_package.absl-py
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,generic-python-lib,_package.absl-py
get-generic-python-lib,94b62a682bc44791_(_package.evaluate_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,generic-python-lib,_package.evaluate
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,generic-python-lib,_package.evaluate
get-generic-python-lib,94b62a682bc44791_(_package.nltk_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,generic-python-lib,_package.nltk
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,generic-python-lib,_package.nltk
get-generic-python-lib,94b62a682bc44791_(_package.numpy_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,generic-python-lib,_package.numpy
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,generic-python-lib,_package.numpy
get-generic-python-lib,94b62a682bc44791_(_package.rouge-score_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,generic-python-lib,_package.rouge-score
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,generic-python-lib,_package.rouge-score
get-generic-python-lib,94b62a682bc44791_(_package.more-itertools_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,generic-python-lib,_package.more-itertools
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,generic-python-lib,_package.more-itertools
get-generic-python-lib,94b62a682bc44791_(_package.compressed_tensors_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> get,generic-python-lib,_package.compressed_tensors
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> get,generic-python-lib,_package.compressed_tensors
detect-cpu,586c8a43320142f7 --> detect,os
benchmark-program,19f369ef47084895 --> detect,cpu
benchmark-program-mlperf,cfff0132a8aa4018 --> benchmark-program,program
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_llama2-70b-99,_pytorch,_cpu,_offline,_bfloat16_) --> benchmark-mlperf
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_pytorch,_llama2-70b-99,_offline,_bfloat16_) --> benchmark-mlperf
Binary file modified ...n-reference-cpu-pytorch_v2.5.1-default_config/llama2-70b-99/offline/cm-deps.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit 9c47c55

Please sign in to comment.