Integrate NCCL

[EricLBuehler]

No description provided.

[github-actions]

Code Metrics Report

  ===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 C Header                2           35           28            0            7
 Dockerfile              1           41           22           10            9
 JSON                   12          105          104            0            1
 Python                 66         2801         2425           72          304
 Shell                   1           57           22           18           17
 Plain Text              3         3723            0         2413         1310
 TOML                   18          619          550            2           67
 YAML                    2           21           19            2            0
-------------------------------------------------------------------------------
 Jupyter Notebooks       4            0            0            0            0
 |- Markdown             2           77           32           31           14
 |- Python               2          205          178            1           26
 (Total)                            282          210           32           40
-------------------------------------------------------------------------------
 Markdown               44         3491            0         2650          841
 |- BASH                 6          103          100            0            3
 |- JSON                 1           12           12            0            0
 |- Python               7          121          109            0           12
 |- Rust                13          440          373            0           67
 |- TOML                 2           75           63            0           12
 (Total)                           4242          657         2650          935
-------------------------------------------------------------------------------
 Rust                  301        96208        86246         1955         8007
 |- Markdown           145         1672           25         1524          123
 (Total)                          97880        86271         3479         8130
===============================================================================
 Total                 454       107101        89416         7122        10563
===============================================================================
  

[guoqingbao]

Hi Eric,
Great work on supporting the DeepSeek R1 model! I’m curious about how mistral.rs is currently handling the loading of the R1 model, given that its size far exceeds the memory capacity of a single GPU. I noticed you’ve introduced a solution called AutoDeviceMapper—does it provide a TensorParallel-like functionality similar to NCCL?

[EricLBuehler]

Hi @guoqingbao!

I noticed you’ve introduced a solution called AutoDeviceMapper—does it provide a TensorParallel-like functionality similar to NCCL?

No, the automatic device mapping system only moves parts of the model to different devices.

The problem with integrating a TensorParallel functionality is that there would be many far-reaching changes that make this difficult to implement, including:

• abstracting the varbuilder (sharded vs unsharded)
• abstracting the retrieval of sharded tensors (we need to make a Shard struct)