Skip to content

Commit

Permalink
Updated results summary [skip ci]
Browse files Browse the repository at this point in the history
  • Loading branch information
@mlcommons-bot
mlcommons-bot committed Jan 30, 2025
1 parent 24d1ac2 commit 24d0058
Show file tree
Hide file tree
Showing 6 changed files with 36 additions and 36 deletions.
30 changes: 15 additions & 15 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
Please download [summary.xlsx](summary.xlsx) to view the most recent results.
```
[2025-01-30 16:27:36,780 submission_checker1.py:3263 INFO] Results=57, NoResults=0, Power Results=0
[2025-01-30 16:27:36,780 submission_checker1.py:3270 INFO] ---
[2025-01-30 16:27:36,780 submission_checker1.py:3271 INFO] Closed Results=2, Closed Power Results=0
[2025-01-30 17:38:54,484 submission_checker1.py:3263 INFO] Results=57, NoResults=0, Power Results=0
[2025-01-30 17:38:54,484 submission_checker1.py:3270 INFO] ---
[2025-01-30 17:38:54,484 submission_checker1.py:3271 INFO] Closed Results=2, Closed Power Results=0
[2025-01-30 16:27:36,780 submission_checker1.py:3276 INFO] Open Results=55, Open Power Results=0
[2025-01-30 17:38:54,484 submission_checker1.py:3276 INFO] Open Results=55, Open Power Results=0
[2025-01-30 16:27:36,780 submission_checker1.py:3281 INFO] Network Results=0, Network Power Results=0
[2025-01-30 17:38:54,484 submission_checker1.py:3281 INFO] Network Results=0, Network Power Results=0
[2025-01-30 16:27:36,780 submission_checker1.py:3286 INFO] ---
[2025-01-30 16:27:36,780 submission_checker1.py:3288 INFO] Systems=43, Power Systems=0
[2025-01-30 16:27:36,780 submission_checker1.py:3292 INFO] Closed Systems=1, Closed Power Systems=0
[2025-01-30 16:27:36,780 submission_checker1.py:3297 INFO] Open Systems=43, Open Power Systems=0
[2025-01-30 16:27:36,780 submission_checker1.py:3302 INFO] Network Systems=0, Network Power Systems=0
[2025-01-30 16:27:36,780 submission_checker1.py:3307 INFO] ---
[2025-01-30 16:27:36,780 submission_checker1.py:3312 INFO] SUMMARY: submission looks OK
[2025-01-30 16:27:37,869 module.py:5492 INFO] - ! call "postprocess" from /home/runner/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-submission-checker/customize.py
[2025-01-30 17:38:54,484 submission_checker1.py:3286 INFO] ---
[2025-01-30 17:38:54,484 submission_checker1.py:3288 INFO] Systems=43, Power Systems=0
[2025-01-30 17:38:54,484 submission_checker1.py:3292 INFO] Closed Systems=1, Closed Power Systems=0
[2025-01-30 17:38:54,484 submission_checker1.py:3297 INFO] Open Systems=43, Open Power Systems=0
[2025-01-30 17:38:54,484 submission_checker1.py:3302 INFO] Network Systems=0, Network Power Systems=0
[2025-01-30 17:38:54,484 submission_checker1.py:3307 INFO] ---
[2025-01-30 17:38:54,484 submission_checker1.py:3312 INFO] SUMMARY: submission looks OK
[2025-01-30 17:38:55,790 module.py:5492 INFO] - ! call "postprocess" from /home/runner/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-submission-checker/customize.py
```

Expand Down Expand Up @@ -69,8 +69,8 @@ Please download [summary.xlsx](summary.xlsx) to view the most recent results.
| 45 | MLCommons | available | open | edge | gh_ubuntu-latest_x86 | gh_ubuntu-latest_x86-reference-cpu-onnxruntime_v1.19.2-default_config | resnet50 | resnet | Offline | 23.227 | acc: 76.000 | 1 | AMD EPYC 7763 64-Core Processor | 1 | 2 | nan | 0 | open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-onnxruntime_v1.19.2-default_config/resnet50/offline | onnxruntime v1.19.2 | Ubuntu 22.04 (linux-6.5.0-1025-azure-glibc2.35) | Automated by MLCommons CM v3.2.6. | 1 | 0 | v5.0 | 0 | False | Samples/s | fp32 |
| 46 | MLCommons | available | open | edge | gh_ubuntu-latest_x86 | gh_ubuntu-latest_x86-mlcommons_cpp-cpu-onnxruntime-default_config | retinanet | retinanet | Offline | 0.432223 | nan | 1 | AMD EPYC 7763 64-Core Processor | 1 | 2 | nan | 0 | open/MLCommons/results/gh_ubuntu-latest_x86-mlcommons_cpp-cpu-onnxruntime-default_config/retinanet/offline | onnxruntime | Ubuntu 24.04 (linux-6.8.0-1020-azure-glibc2.39) | nan | 1 | 0 | v5.0 | 0 | False | Samples/s | fp32 |
| 47 | MLCommons | available | open | edge | gh_ubuntu-latest_x86 | gh_ubuntu-latest_x86-mlcommons_cpp-cpu-onnxruntime-default_config | resnet50 | resnet | Offline | 22.9262 | acc: 76.000 | 1 | AMD EPYC 7763 64-Core Processor | 1 | 2 | nan | 0 | open/MLCommons/results/gh_ubuntu-latest_x86-mlcommons_cpp-cpu-onnxruntime-default_config/resnet50/offline | onnxruntime | Ubuntu 24.04 (linux-6.8.0-1020-azure-glibc2.39) | nan | 1 | 0 | v5.0 | 0 | False | Samples/s | fp32 |
| 48 | MLCommons | available | open | edge | gh_action | gh_action-reference-gpu-pytorch_v2.5.1-cu124 | gptj-99 | gptj-99 | Offline | 48.3215 | GEN_LEN: 264 | 1 | Intel(R) Xeon(R) w7-2495X | 1 | 24 | NVIDIA GeForce RTX 4090 | 2 | open/MLCommons/results/gh_action-reference-gpu-pytorch_v2.5.1-cu124/gptj-99/offline | pytorch v2.5.1 | Ubuntu 22.04 (linux-6.8.0-49-generic-glibc2.35) | nan | 1 | 0 | v5.0 | 0 | False | Tokens/s | fp32 |
| 49 | MLCommons | available | open | edge | gh_action | gh_action-reference-gpu-pytorch_v2.5.1-cu124 | stable-diffusion-xl | stable-diffusion-xl | Offline | 0.358711 | nan | 1 | Intel(R) Xeon(R) w7-2495X | 1 | 24 | NVIDIA GeForce RTX 4090 | 2 | open/MLCommons/results/gh_action-reference-gpu-pytorch_v2.5.1-cu124/stable-diffusion-xl/offline | pytorch v2.5.1 | Ubuntu 22.04 (linux-6.8.0-49-generic-glibc2.35) | nan | 1 | 0 | v5.0 | 0 | False | Samples/s | fp32 |
| 48 | MLCommons | available | open | edge | gh_action | gh_action-reference-gpu-pytorch_v2.5.1-cu124 | gptj-99 | gptj-99 | Offline | 48.3215 | GEN_LEN: 264 | 1 | AMD Ryzen 9 7950X 16-Core Processor | 1 | 16 | NVIDIA GeForce RTX 4090 | 1 | open/MLCommons/results/gh_action-reference-gpu-pytorch_v2.5.1-cu124/gptj-99/offline | pytorch v2.5.1 | Ubuntu 22.04 (linux-6.8.0-51-generic-glibc2.35) | nan | 1 | 0 | v5.0 | 0 | False | Tokens/s | fp32 |
| 49 | MLCommons | available | open | edge | gh_action | gh_action-reference-gpu-pytorch_v2.5.1-cu124 | stable-diffusion-xl | stable-diffusion-xl | Offline | 0.352694 | nan | 1 | AMD Ryzen 9 7950X 16-Core Processor | 1 | 16 | NVIDIA GeForce RTX 4090 | 1 | open/MLCommons/results/gh_action-reference-gpu-pytorch_v2.5.1-cu124/stable-diffusion-xl/offline | pytorch v2.5.1 | Ubuntu 22.04 (linux-6.8.0-51-generic-glibc2.35) | nan | 1 | 0 | v5.0 | 0 | False | Samples/s | fp32 |
| 50 | MLCommons | available | open | edge | gh_windows-latest_x86 | gh_windows-latest_x86-reference-cpu-tf_v2.18.0-default_config | resnet50 | resnet | Offline | 21.5168 | acc: 76.000 | 1 | undefined | 1 | 1 | nan | 0 | open/MLCommons/results/gh_windows-latest_x86-reference-cpu-tf_v2.18.0-default_config/resnet50/offline | tf v2.18.0 | Windows-2022Server-10.0.20348-SP0 | nan | 1 | 0 | v5.0 | 0 | False | Samples/s | fp32 |
| 51 | MLCommons | available | open | edge | gh_ubuntu-latest_x86 | gh_ubuntu-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config | bert-99 | bert-99 | Offline | 6.24578 | F1: 80.0 | 1 | AMD EPYC 7763 64-Core Processor | 1 | 2 | nan | 0 | open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config/bert-99/offline | deepsparse v1.8.0 | Ubuntu 24.04 (linux-6.8.0-1020-azure-glibc2.39) | nan | 1 | 0 | v5.0 | 0 | False | Samples/s | fp32 |
| 52 | MLCommons | available | open | edge | gh_windows-latest_x86 | gh_windows-latest_x86-reference-cpu-tf_v2.17.0-default_config | resnet50 | resnet | Offline | 21.7501 | acc: 76.000 | 1 | undefined | 1 | 1 | nan | 0 | open/MLCommons/results/gh_windows-latest_x86-reference-cpu-tf_v2.17.0-default_config/resnet50/offline | tf v2.17.0 | Windows-2022Server-10.0.20348-SP0 | Automated by MLCommons CM v3.2.6. | 1 | 0 | v5.0 | 0 | False | Samples/s | fp32 |
Expand Down
2 changes: 1 addition & 1 deletion dbversion
View file
Original file line number Diff line number Diff line change
@@ -1 +1 @@
544
545
4 changes: 2 additions & 2 deletions summary.csv
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,8 @@ MLCommons,available,open,"edge","gh_ubuntu-latest_x86",gh_ubuntu-latest_x86-refe
MLCommons,available,open,"edge","gh_ubuntu-latest_x86",gh_ubuntu-latest_x86-reference-cpu-onnxruntime_v1.19.2-default_config,resnet50,resnet,Offline,23.227,acc: 76.000,1,"AMD EPYC 7763 64-Core Processor",1,2,"N/A","0",open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-onnxruntime_v1.19.2-default_config/resnet50/offline,"onnxruntime v1.19.2","Ubuntu 22.04 (linux-6.5.0-1025-azure-glibc2.35)","Automated by MLCommons CM v3.2.6. ",1,0,v5.0,0,False,Samples/s,"fp32"
MLCommons,available,open,"edge","gh_ubuntu-latest_x86",gh_ubuntu-latest_x86-mlcommons_cpp-cpu-onnxruntime-default_config,retinanet,retinanet,Offline,0.432223,,1,"AMD EPYC 7763 64-Core Processor",1,2,"N/A","0",open/MLCommons/results/gh_ubuntu-latest_x86-mlcommons_cpp-cpu-onnxruntime-default_config/retinanet/offline,"onnxruntime","Ubuntu 24.04 (linux-6.8.0-1020-azure-glibc2.39)","",1,0,v5.0,0,False,Samples/s,"fp32"
MLCommons,available,open,"edge","gh_ubuntu-latest_x86",gh_ubuntu-latest_x86-mlcommons_cpp-cpu-onnxruntime-default_config,resnet50,resnet,Offline,22.9262,acc: 76.000,1,"AMD EPYC 7763 64-Core Processor",1,2,"N/A","0",open/MLCommons/results/gh_ubuntu-latest_x86-mlcommons_cpp-cpu-onnxruntime-default_config/resnet50/offline,"onnxruntime","Ubuntu 24.04 (linux-6.8.0-1020-azure-glibc2.39)","",1,0,v5.0,0,False,Samples/s,"fp32"
MLCommons,available,open,"edge","gh_action",gh_action-reference-gpu-pytorch_v2.5.1-cu124,gptj-99,gptj-99,Offline,48.3215,GEN_LEN: 264,1,"Intel(R) Xeon(R) w7-2495X",1,24,"NVIDIA GeForce RTX 4090","2",open/MLCommons/results/gh_action-reference-gpu-pytorch_v2.5.1-cu124/gptj-99/offline,"pytorch v2.5.1","Ubuntu 22.04 (linux-6.8.0-49-generic-glibc2.35)","",1,0,v5.0,0,False,Tokens/s,"fp32"
MLCommons,available,open,"edge","gh_action",gh_action-reference-gpu-pytorch_v2.5.1-cu124,stable-diffusion-xl,stable-diffusion-xl,Offline,0.358711,,1,"Intel(R) Xeon(R) w7-2495X",1,24,"NVIDIA GeForce RTX 4090","2",open/MLCommons/results/gh_action-reference-gpu-pytorch_v2.5.1-cu124/stable-diffusion-xl/offline,"pytorch v2.5.1","Ubuntu 22.04 (linux-6.8.0-49-generic-glibc2.35)","",1,0,v5.0,0,False,Samples/s,"fp32"
MLCommons,available,open,"edge","gh_action",gh_action-reference-gpu-pytorch_v2.5.1-cu124,gptj-99,gptj-99,Offline,48.3215,GEN_LEN: 264,1,"AMD Ryzen 9 7950X 16-Core Processor",1,16,"NVIDIA GeForce RTX 4090","1",open/MLCommons/results/gh_action-reference-gpu-pytorch_v2.5.1-cu124/gptj-99/offline,"pytorch v2.5.1","Ubuntu 22.04 (linux-6.8.0-51-generic-glibc2.35)","",1,0,v5.0,0,False,Tokens/s,"fp32"
MLCommons,available,open,"edge","gh_action",gh_action-reference-gpu-pytorch_v2.5.1-cu124,stable-diffusion-xl,stable-diffusion-xl,Offline,0.352694,,1,"AMD Ryzen 9 7950X 16-Core Processor",1,16,"NVIDIA GeForce RTX 4090","1",open/MLCommons/results/gh_action-reference-gpu-pytorch_v2.5.1-cu124/stable-diffusion-xl/offline,"pytorch v2.5.1","Ubuntu 22.04 (linux-6.8.0-51-generic-glibc2.35)","",1,0,v5.0,0,False,Samples/s,"fp32"
MLCommons,available,open,"edge","gh_windows-latest_x86",gh_windows-latest_x86-reference-cpu-tf_v2.18.0-default_config,resnet50,resnet,Offline,21.5168,acc: 76.000,1,"undefined",1,1,"N/A","0",open/MLCommons/results/gh_windows-latest_x86-reference-cpu-tf_v2.18.0-default_config/resnet50/offline,"tf v2.18.0","Windows-2022Server-10.0.20348-SP0","",1,0,v5.0,0,False,Samples/s,"fp32"
MLCommons,available,open,"edge","gh_ubuntu-latest_x86",gh_ubuntu-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config,bert-99,bert-99,Offline,6.24578,F1: 80.0,1,"AMD EPYC 7763 64-Core Processor",1,2,"N/A","0",open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config/bert-99/offline,"deepsparse v1.8.0","Ubuntu 24.04 (linux-6.8.0-1020-azure-glibc2.39)","",1,0,v5.0,0,False,Samples/s,"fp32"
MLCommons,available,open,"edge","gh_windows-latest_x86",gh_windows-latest_x86-reference-cpu-tf_v2.17.0-default_config,resnet50,resnet,Offline,21.7501,acc: 76.000,1,"undefined",1,1,"N/A","0",open/MLCommons/results/gh_windows-latest_x86-reference-cpu-tf_v2.17.0-default_config/resnet50/offline,"tf v2.17.0","Windows-2022Server-10.0.20348-SP0","Automated by MLCommons CM v3.2.6. ",1,0,v5.0,0,False,Samples/s,"fp32"
Expand Down
18 changes: 9 additions & 9 deletions summary.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1452,14 +1452,14 @@
"Result":48.3215,
"Accuracy":"GEN_LEN: 264",
"number_of_nodes":1,
"host_processor_model_name":"Intel(R) Xeon(R) w7-2495X",
"host_processor_model_name":"AMD Ryzen 9 7950X 16-Core Processor",
"host_processors_per_node":1,
"host_processor_core_count":24,
"host_processor_core_count":16,
"accelerator_model_name":"NVIDIA GeForce RTX 4090",
"accelerators_per_node":2,
"accelerators_per_node":1,
"Location":"open\/MLCommons\/results\/gh_action-reference-gpu-pytorch_v2.5.1-cu124\/gptj-99\/offline",
"framework":"pytorch v2.5.1",
"operating_system":"Ubuntu 22.04 (linux-6.8.0-49-generic-glibc2.35)",
"operating_system":"Ubuntu 22.04 (linux-6.8.0-51-generic-glibc2.35)",
"notes":null,
"compliance":1,
"errors":0,
Expand All @@ -1479,17 +1479,17 @@
"Model":"stable-diffusion-xl",
"MlperfModel":"stable-diffusion-xl",
"Scenario":"Offline",
"Result":0.358711,
"Result":0.352694,
"Accuracy":null,
"number_of_nodes":1,
"host_processor_model_name":"Intel(R) Xeon(R) w7-2495X",
"host_processor_model_name":"AMD Ryzen 9 7950X 16-Core Processor",
"host_processors_per_node":1,
"host_processor_core_count":24,
"host_processor_core_count":16,
"accelerator_model_name":"NVIDIA GeForce RTX 4090",
"accelerators_per_node":2,
"accelerators_per_node":1,
"Location":"open\/MLCommons\/results\/gh_action-reference-gpu-pytorch_v2.5.1-cu124\/stable-diffusion-xl\/offline",
"framework":"pytorch v2.5.1",
"operating_system":"Ubuntu 22.04 (linux-6.8.0-49-generic-glibc2.35)",
"operating_system":"Ubuntu 22.04 (linux-6.8.0-51-generic-glibc2.35)",
"notes":null,
"compliance":1,
"errors":0,
Expand Down
Binary file modified summary.xlsx
View file
Binary file not shown.
18 changes: 9 additions & 9 deletions summary_results.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -732,14 +732,14 @@
"Scenario": "Offline",
"Accuracy": "GEN_LEN: 264",
"Nodes": 1,
"Processor": "Intel(R) Xeon(R) w7-2495X",
"Processor": "AMD Ryzen 9 7950X 16-Core Processor",
"host_processors_per_node": 1,
"host_processor_core_count": 24,
"host_processor_core_count": 16,
"Accelerator": "NVIDIA GeForce RTX 4090",
"a#": 2,
"a#": 1,
"Location": "open/MLCommons/results/gh_action-reference-gpu-pytorch_v2.5.1-cu124/gptj-99/offline",
"Software": "pytorch v2.5.1",
"operating_system": "Ubuntu 22.04 (linux-6.8.0-49-generic-glibc2.35)",
"operating_system": "Ubuntu 22.04 (linux-6.8.0-51-generic-glibc2.35)",
"Notes": "",
"compliance": 1,
"errors": 0,
Expand Down Expand Up @@ -768,14 +768,14 @@
"Scenario": "Offline",
"Accuracy": "",
"Nodes": 1,
"Processor": "Intel(R) Xeon(R) w7-2495X",
"Processor": "AMD Ryzen 9 7950X 16-Core Processor",
"host_processors_per_node": 1,
"host_processor_core_count": 24,
"host_processor_core_count": 16,
"Accelerator": "NVIDIA GeForce RTX 4090",
"a#": 2,
"a#": 1,
"Location": "open/MLCommons/results/gh_action-reference-gpu-pytorch_v2.5.1-cu124/stable-diffusion-xl/offline",
"Software": "pytorch v2.5.1",
"operating_system": "Ubuntu 22.04 (linux-6.8.0-49-generic-glibc2.35)",
"operating_system": "Ubuntu 22.04 (linux-6.8.0-51-generic-glibc2.35)",
"Notes": "",
"compliance": 1,
"errors": 0,
Expand All @@ -788,7 +788,7 @@
"Code": "https://github.com/mlcommons/mlperf_inference_test_submissions_v5.0/tree/main/open/MLCommons/code",
"Unique ID (e.g. for Audit)": "edge/open/MLCommons/gh_action-reference-gpu-pytorch_v2.5.1-cu124/stable-diffusion-xl",
"ColorKey": "availableMLCommons",
"Performance_Result": 0.358711,
"Performance_Result": 0.352694,
"Performance_Units": "Samples/s"
},
{
Expand Down

0 comments on commit 24d0058

Please sign in to comment.