Skip to content

Commit

Permalink
Updated results summary
Browse files Browse the repository at this point in the history
  • Loading branch information
@gateoverflow-admin
gateoverflow-admin committed Jan 17, 2025
1 parent 235aa7b commit b51a059
Show file tree
Hide file tree
Showing 6 changed files with 17 additions and 17 deletions.
26 changes: 13 additions & 13 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
Please download [summary.xlsx](summary.xlsx) to view the most recent results.
```
[2025-01-16 19:39:06,942 submission_checker1.py:3183 INFO] Results=38, NoResults=0, Power Results=0
[2025-01-16 19:39:06,942 submission_checker1.py:3190 INFO] ---
[2025-01-16 19:39:06,942 submission_checker1.py:3191 INFO] Closed Results=2, Closed Power Results=0
[2025-01-17 05:47:21,592 submission_checker1.py:3183 INFO] Results=38, NoResults=0, Power Results=0
[2025-01-17 05:47:21,592 submission_checker1.py:3190 INFO] ---
[2025-01-17 05:47:21,592 submission_checker1.py:3191 INFO] Closed Results=2, Closed Power Results=0
[2025-01-16 19:39:06,942 submission_checker1.py:3196 INFO] Open Results=36, Open Power Results=0
[2025-01-17 05:47:21,592 submission_checker1.py:3196 INFO] Open Results=36, Open Power Results=0
[2025-01-16 19:39:06,942 submission_checker1.py:3201 INFO] Network Results=0, Network Power Results=0
[2025-01-17 05:47:21,592 submission_checker1.py:3201 INFO] Network Results=0, Network Power Results=0
[2025-01-16 19:39:06,942 submission_checker1.py:3206 INFO] ---
[2025-01-16 19:39:06,942 submission_checker1.py:3208 INFO] Systems=28, Power Systems=0
[2025-01-16 19:39:06,942 submission_checker1.py:3212 INFO] Closed Systems=1, Closed Power Systems=0
[2025-01-16 19:39:06,942 submission_checker1.py:3217 INFO] Open Systems=28, Open Power Systems=0
[2025-01-16 19:39:06,942 submission_checker1.py:3222 INFO] Network Systems=0, Network Power Systems=0
[2025-01-16 19:39:06,942 submission_checker1.py:3227 INFO] ---
[2025-01-16 19:39:06,942 submission_checker1.py:3232 INFO] SUMMARY: submission looks OK
[2025-01-17 05:47:21,592 submission_checker1.py:3206 INFO] ---
[2025-01-17 05:47:21,592 submission_checker1.py:3208 INFO] Systems=28, Power Systems=0
[2025-01-17 05:47:21,592 submission_checker1.py:3212 INFO] Closed Systems=1, Closed Power Systems=0
[2025-01-17 05:47:21,592 submission_checker1.py:3217 INFO] Open Systems=28, Open Power Systems=0
[2025-01-17 05:47:21,592 submission_checker1.py:3222 INFO] Network Systems=0, Network Power Systems=0
[2025-01-17 05:47:21,592 submission_checker1.py:3227 INFO] ---
[2025-01-17 05:47:21,592 submission_checker1.py:3232 INFO] SUMMARY: submission looks OK
INFO:root: ! call "postprocess" from /home/runner/CM/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-submission-checker/customize.py
```
Expand All @@ -40,7 +40,7 @@ INFO:root: ! call "postprocess" from /home/runner/CM/repos/mlcommons@mlper
| 16 | MLCommons | available | open | datacenter | gh_action | gh_action-reference-cpu-pytorch-v2.5.1-default_config | llama2-70b-99 | llama2-70b-99 | Offline | 0.387999 | ROUGE1: 61.7021 ROUGE2: 37.9679 ROUGEL: 39.3617 TOKENS_PER_SAMPLE: 610.0 | 1 | Intel(R) Xeon(R) w7-2495X | 1 | 24 | nan | 0 | open/MLCommons/results/gh_action-reference-cpu-pytorch-v2.5.1-default_config/llama2-70b-99/offline | pytorch v2.5.1 | Ubuntu 22.04 (linux-6.2.0-39-generic-glibc2.35) | nan | 1 | 0 | v5.0 | 0 | False | Tokens/s | fp32 |
| 17 | MLCommons | available | open | edge | gh_windows-latest | gh_windows-latest-reference-cpu-onnxruntime_v1.19.2-default_config | retinanet | retinanet | Offline | 0.349311 | mAP: 76.951 | 1 | undefined | 1 | 1 | nan | 0 | open/MLCommons/results/gh_windows-latest-reference-cpu-onnxruntime_v1.19.2-default_config/retinanet/offline | onnxruntime v1.19.2 | Windows-2022Server-10.0.20348-SP0 | Automated by MLCommons CM v3.2.6. | 1 | 0 | v5.0 | 0 | False | Samples/s | fp32 |
| 18 | MLCommons | available | open | edge | gh_macos-latest_x86 | gh_macos-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config | bert-99 | bert-99 | Offline | 4.26497 | F1: 80.0 | 1 | undefined | 1 | 3 | nan | 0 | open/MLCommons/results/gh_macos-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config/bert-99/offline | deepsparse v1.8.0 | (darwin-23.6.0) | Automated by MLCommons CM v3.2.6. | 1 | 0 | v5.0 | 0 | False | Samples/s | fp32 |
| 19 | MLCommons | available | open | datacenter | gh_action | gh_action-reference-cpu-pytorch_v2.5.1-default_config | llama2-70b-99 | llama2-70b-99 | Offline | 0.351429 | ROUGE1: 61.7021 ROUGE2: 37.9679 ROUGEL: 39.3617 TOKENS_PER_SAMPLE: 610.0 | 1 | Intel(R) Xeon(R) w7-2495X | 1 | 24 | nan | 0 | open/MLCommons/results/gh_action-reference-cpu-pytorch_v2.5.1-default_config/llama2-70b-99/offline | pytorch v2.5.1 | Ubuntu 22.04 (linux-6.8.0-49-generic-glibc2.35) | nan | 1 | 0 | v5.0 | 0 | False | Tokens/s | fp32 |
| 19 | MLCommons | available | open | datacenter | gh_action | gh_action-reference-cpu-pytorch_v2.5.1-default_config | llama2-70b-99 | llama2-70b-99 | Offline | 0.344622 | ROUGE1: 61.7021 ROUGE2: 37.9679 ROUGEL: 39.3617 TOKENS_PER_SAMPLE: 610.0 | 1 | Intel(R) Xeon(R) w7-2495X | 1 | 24 | nan | 0 | open/MLCommons/results/gh_action-reference-cpu-pytorch_v2.5.1-default_config/llama2-70b-99/offline | pytorch v2.5.1 | Ubuntu 22.04 (linux-6.8.0-49-generic-glibc2.35) | nan | 1 | 0 | v5.0 | 0 | False | Tokens/s | fp32 |
| 20 | MLCommons | available | open | datacenter | 3b07702db56d | 3b07702db56d-reference-gpu-pytorch_v2.4.1-scc24-base | stable-diffusion-xl | stable-diffusion-xl | Offline | 0.374837 | CLIP_SCORE: 15.18544016778469 FID_SCORE: 235.69504308101006 | 1 | Intel(R) Xeon(R) w7-2495X | 1 | 24 | NVIDIA GeForce RTX 4090 | 1 | open/MLCommons/results/3b07702db56d-reference-gpu-pytorch_v2.4.1-scc24-base/stable-diffusion-xl/offline | pytorch v2.4.1 | Ubuntu 22.04 (linux-6.2.0-39-generic-glibc2.35) | Automated by MLCommons CM v2.3.9. | 1 | 0 | v5.0 | 0 | False | Samples/s | fp32 |
| 21 | MLCommons | available | open | edge | gh_windows | gh_windows-reference-cpu-onnxruntime_v1.19.2-default_config | resnet50 | resnet | Offline | 17.8036 | acc: 76.000 | 1 | undefined | 1 | 1 | nan | 0 | open/MLCommons/results/gh_windows-reference-cpu-onnxruntime_v1.19.2-default_config/resnet50/offline | onnxruntime v1.19.2 | Windows-2022Server-10.0.20348-SP0 | Automated by MLCommons CM v2.4.0. | 1 | 0 | v5.0 | 0 | False | Samples/s | fp32 |
| 22 | MLCommons | available | open | edge | gh_action | gh_action-reference-gpu-pytorch-v2.5.0-cu124 | gptj-99 | gptj-99 | Offline | 47.0771 | ROUGE1: 32.2581 ROUGE2: 6.6667 ROUGEL: 22.5806 GEN_LEN: 264 | 1 | Intel(R) Xeon(R) w7-2495X | 1 | 24 | NVIDIA GeForce RTX 4090 | 2 | open/MLCommons/results/gh_action-reference-gpu-pytorch-v2.5.0-cu124/gptj-99/offline | pytorch v2.5.0 | Ubuntu 22.04 (linux-6.2.0-39-generic-glibc2.35) | Automated by MLCommons CM v3.2.6. | 1 | 0 | v5.0 | 0 | False | Tokens/s | fp32 |
Expand Down
2 changes: 1 addition & 1 deletion dbversion
View file
Original file line number Diff line number Diff line change
@@ -1 +1 @@
334
335
2 changes: 1 addition & 1 deletion summary.csv
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ MLCommons,available,open,"datacenter","gh_action",gh_action-reference-cpu-pytorc
MLCommons,available,open,"datacenter","gh_action",gh_action-reference-cpu-pytorch-v2.5.1-default_config,llama2-70b-99,llama2-70b-99,Offline,0.387999,ROUGE1: 61.7021 ROUGE2: 37.9679 ROUGEL: 39.3617 TOKENS_PER_SAMPLE: 610.0,1,"Intel(R) Xeon(R) w7-2495X",1,24,"N/A","0",open/MLCommons/results/gh_action-reference-cpu-pytorch-v2.5.1-default_config/llama2-70b-99/offline,"pytorch v2.5.1","Ubuntu 22.04 (linux-6.2.0-39-generic-glibc2.35)","",1,0,v5.0,0,False,Tokens/s,"fp32"
MLCommons,available,open,"edge","gh_windows-latest",gh_windows-latest-reference-cpu-onnxruntime_v1.19.2-default_config,retinanet,retinanet,Offline,0.349311,mAP: 76.951,1,"undefined",1,1,"N/A","0",open/MLCommons/results/gh_windows-latest-reference-cpu-onnxruntime_v1.19.2-default_config/retinanet/offline,"onnxruntime v1.19.2","Windows-2022Server-10.0.20348-SP0","Automated by MLCommons CM v3.2.6. ",1,0,v5.0,0,False,Samples/s,"fp32"
MLCommons,available,open,"edge","gh_macos-latest_x86",gh_macos-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config,bert-99,bert-99,Offline,4.26497,F1: 80.0,1,"undefined",1,3,"N/A","0",open/MLCommons/results/gh_macos-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config/bert-99/offline,"deepsparse v1.8.0"," (darwin-23.6.0)","Automated by MLCommons CM v3.2.6. ",1,0,v5.0,0,False,Samples/s,"fp32"
MLCommons,available,open,"datacenter","gh_action",gh_action-reference-cpu-pytorch_v2.5.1-default_config,llama2-70b-99,llama2-70b-99,Offline,0.351429,ROUGE1: 61.7021 ROUGE2: 37.9679 ROUGEL: 39.3617 TOKENS_PER_SAMPLE: 610.0,1,"Intel(R) Xeon(R) w7-2495X",1,24,"N/A","0",open/MLCommons/results/gh_action-reference-cpu-pytorch_v2.5.1-default_config/llama2-70b-99/offline,"pytorch v2.5.1","Ubuntu 22.04 (linux-6.8.0-49-generic-glibc2.35)","",1,0,v5.0,0,False,Tokens/s,"fp32"
MLCommons,available,open,"datacenter","gh_action",gh_action-reference-cpu-pytorch_v2.5.1-default_config,llama2-70b-99,llama2-70b-99,Offline,0.344622,ROUGE1: 61.7021 ROUGE2: 37.9679 ROUGEL: 39.3617 TOKENS_PER_SAMPLE: 610.0,1,"Intel(R) Xeon(R) w7-2495X",1,24,"N/A","0",open/MLCommons/results/gh_action-reference-cpu-pytorch_v2.5.1-default_config/llama2-70b-99/offline,"pytorch v2.5.1","Ubuntu 22.04 (linux-6.8.0-49-generic-glibc2.35)","",1,0,v5.0,0,False,Tokens/s,"fp32"
MLCommons,available,open,"datacenter","3b07702db56d",3b07702db56d-reference-gpu-pytorch_v2.4.1-scc24-base,stable-diffusion-xl,stable-diffusion-xl,Offline,0.374837,CLIP_SCORE: 15.18544016778469 FID_SCORE: 235.69504308101006,1,"Intel(R) Xeon(R) w7-2495X",1,24,"NVIDIA GeForce RTX 4090","1",open/MLCommons/results/3b07702db56d-reference-gpu-pytorch_v2.4.1-scc24-base/stable-diffusion-xl/offline,"pytorch v2.4.1","Ubuntu 22.04 (linux-6.2.0-39-generic-glibc2.35)","Automated by MLCommons CM v2.3.9. ",1,0,v5.0,0,False,Samples/s,"fp32"
MLCommons,available,open,"edge","gh_windows",gh_windows-reference-cpu-onnxruntime_v1.19.2-default_config,resnet50,resnet,Offline,17.8036,acc: 76.000,1,"undefined",1,1,"N/A","0",open/MLCommons/results/gh_windows-reference-cpu-onnxruntime_v1.19.2-default_config/resnet50/offline,"onnxruntime v1.19.2","Windows-2022Server-10.0.20348-SP0","Automated by MLCommons CM v2.4.0. ",1,0,v5.0,0,False,Samples/s,"fp32"
MLCommons,available,open,"edge","gh_action",gh_action-reference-gpu-pytorch-v2.5.0-cu124,gptj-99,gptj-99,Offline,47.0771,ROUGE1: 32.2581 ROUGE2: 6.6667 ROUGEL: 22.5806 GEN_LEN: 264,1,"Intel(R) Xeon(R) w7-2495X",1,24,"NVIDIA GeForce RTX 4090","2",open/MLCommons/results/gh_action-reference-gpu-pytorch-v2.5.0-cu124/gptj-99/offline,"pytorch v2.5.0","Ubuntu 22.04 (linux-6.2.0-39-generic-glibc2.35)","Automated by MLCommons CM v3.2.6. ",1,0,v5.0,0,False,Tokens/s,"fp32"
Expand Down
2 changes: 1 addition & 1 deletion summary.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -579,7 +579,7 @@
"Model":"llama2-70b-99",
"MlperfModel":"llama2-70b-99",
"Scenario":"Offline",
"Result":0.351429,
"Result":0.344622,
"Accuracy":"ROUGE1: 61.7021 ROUGE2: 37.9679 ROUGEL: 39.3617 TOKENS_PER_SAMPLE: 610.0",
"number_of_nodes":1,
"host_processor_model_name":"Intel(R) Xeon(R) w7-2495X",
Expand Down
Binary file modified summary.xlsx
View file
Binary file not shown.
2 changes: 1 addition & 1 deletion summary_results.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@
"Code": "https://github.com/mlcommons/mlperf_inference_test_submissions_v5.0/tree/auto-update/open/MLCommons/code",
"Unique ID (e.g. for Audit)": "datacenter/open/MLCommons/gh_action-reference-cpu-pytorch_v2.5.1-default_config/llama2-70b-99",
"ColorKey": "availableMLCommons",
"Performance_Result": 0.351429,
"Performance_Result": 0.344622,
"Performance_Units": "Tokens/s"
},
{
Expand Down

0 comments on commit b51a059

Please sign in to comment.