Better cuBLAS handle management

[ptrendx]

Description

Create only 1 (per thread) cublasLtHandle in order to not leak memory. We do not want to actually destroy the handle, since it would incur an implicit cudaDeviceSynchronize call (see cuBLAS docs).

Fixes #1372

Type of change

• Documentation change (change only to the documentation, either a fix or a new content)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Infra/Build change
• Code refactor

Changes

• Introduced a singleton manager for cuBLAS handles similar to the one used for cuDNN handles.

Checklist:

• I have read and followed the contributing guidelines
• The functionality is complete
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[ksivaman]

LGTM

[shenzhenghai]

May I have a question about the commit? If there is multiple devices invoke cublas_gemm(), it seems always use the same cublasLt handle?
The document mentioned "one cuBLASLt handle should be created for each device".

[shenzhenghai]

May I have a question about the commit? If there is multiple devices invoke cublas_gemm(), it seems always use the same cublasLt handle? The document mentioned "one cuBLASLt handle should be created for each device".

@ptrendx @ksivaman any comment on my concern? Thanks.

[ptrendx]

@shenzhenghai Yeah, if your usage is to have multiple devices per thread then you are correct - it would use the same handle, which is wrong. I don't necessarily like the fact that fixing this will require calling cudaGetDevice before every cuBLAS call - this adds overhead and the most typical usage by the DL Frameworks nowadays is 1 GPU per process. But I agree that the functional correctness is the most important - will submit a fix (also will fix the cuDNN handle management, since it has similar problem).

[ptrendx]

/te-ci

[ptrendx]

/te-ci

[timmoon10]

LGTM

[ptrendx]

/te-ci