Track E2E gaps for LLVM-path vs SPIRV-path.

[tdeng5]

Issue Description

Try to run E2E test on llvm-path, figure out the gaps for llvm-path vs spirv-path.

Code Example

System Info

• OS:
• PyTorch:
• Intel Extension For Pytorch:
• LLVM Version.
• Triton Version:
• Intel® XPU Backend for Triton*:

[ESI-SYD]

Update 1: E2E test in SPIRV path

Upstream openai/triton run torchinductor E2E test for CUDA device based on torch-inductor-tests.yml like:

          ./.github/workflows/torch-inductor/scripts/install_torchinductor.sh torchbench
      - name: Environment
        run: |
          source /opt/torchinductor_venv/bin/activate
          ./.github/workflows/torch-inductor/scripts/install_triton.sh
      - name: Performance
        run: |
          ./.github/workflows/torch-inductor/scripts/run_torchinductor_perf.sh torchbench

Benchmark commands: run_torchinductor_perf.sh

echo "Running with Triton Nightly"
for model in "${MODELS[@]}"; do
  if [ "$model" != "$MODEL_SPEC" ] && [ "$MODEL_SPEC" != "all" ]; then
    continue
  fi
  echo "Running performance test for $model"
  python3 benchmarks/dynamo/"$model".py --float32 -dcuda --training --inductor --performance \
    --output "$TRITON_TEST_REPORTS_DIR"/"$model".csv
done

Note:
Since pytorch XPU upstream is WIP, and E2E test need pytorch dynamo benchmark scripts, so we need to build public torch and ipex with patches for XPU.

We manage version of Pytorch and IPEX as Pre-Request in ci_pins like:

torch_repo=https://github.com/pytorch/pytorch.git
torch_branch=release/2.1
torch_commit=209f2fa8ff86652f67d75c2f19bf9cb9942fd018
ipex_repo=https://github.com/intel/intel-extension-for-pytorch.git
ipex_branch=xpu-main-pre
ipex_commit=7980a37028023037b4f0b47617c5fc3343a6d09b
oneapi_version=2024.0

torch patches

We have implemented this for huggingface suite, including accuracy check and performance test.
E2E
E2E nightly
doc

[ESI-SYD]

Update 2: Gap to run E2E in llvm-path

1. Support for pytorch 2.4 #151 Dynamo benchmark needed

2. Use oneAPI 2024.0 rather than 2023.2 to align with PyTorch. #173 Also IPEX's requirements

3. apply torch patches

[etiotto]

@ESI-SYD according to your investigation we cannot yet run these benchmarks because because of the gaps highlighted here . Correct ? Any other issues you were able to uncover at this point ?

[ESI-SYD]

@ESI-SYD according to your investigation we cannot yet run these benchmarks because because of the gaps highlighted here . Correct ? Any other issues you were able to uncover at this point ?

yes, we can't run e2e directly for now. Also I added torch patch link.

[ESI-SYD]

Issues from inductor E2E accuracy check on llvm-path summarized in: #412 #423 #438 #439

[vlad-penkin]

llvm branch have reached the parity with the spirv branch this week