Skip to content

fix: use the accelerator API instead of hard-coded cuda/cpu device selection - #6455

Open
li-lizhe wants to merge 1 commit into
hpcaitech:mainfrom
li-lizhe:fix/accelerator-device-selection
Open

li-lizhe wants to merge 1 commit into
hpcaitech:mainfrom
li-lizhe:fix/accelerator-device-selection

Conversation

@li-lizhe

Copy link
Copy Markdown

Fixes #6454

Three call sites still picked the device by hand instead of using ColossalAI's own accelerator abstraction, so on a non-CUDA accelerator they silently fell back to the CPU (fx tracer) or raised (rope cache).

Changes

file before after
colossalai/fx/tracer/experimental.py::default_device() torch.device("cuda:0") if torch.cuda.is_available() else torch.device("cpu") get_accelerator().get_current_device()
colossalai/fx/passes/meta_info_prop.py::metainfo_trace() same expression get_accelerator().get_current_device()
colossalai/inference/utils.py::init_to_get_rotary() torch.cos(freqs).to(self.dtype).cuda() torch.cos(freqs).to(self.dtype).to(device), device = get_accelerator().get_current_device()

CUDA behaviour is unchanged: get_current_device() is cuda:<current device> there (the previous expression hard-coded cuda:0), and the rope cache still lands on the current CUDA device.

Verification — Ascend 910B4, torch 2.15.0.dev20260917+cpu + torch_npu, CANN, single process

Before:

torch.cuda.is_available() = False
get_accelerator().name = npu
get_accelerator().get_current_device() = npu:0
--- default_device() ---
default_device() = cpu
--- init_to_get_rotary ---
  File ".../colossalai/inference/utils.py", line 63, in init_to_get_rotary
    self._cos_cached = torch.cos(freqs).to(self.dtype).cuda()
AssertionError: Torch not compiled with CUDA enabled

After (same script, same box, only the three files swapped):

get_accelerator().name = npu
get_accelerator().get_current_device() = npu:0
--- default_device() ---
default_device() = npu:0
--- metainfo_trace ---
source line: device = get_accelerator().get_current_device()
--- init_to_get_rotary ---
cos device = npu:0 | sin device = npu:0
cos dtype = torch.float16 | shape = (65664, 4)

The before/after files were md5-compared with the files in this PR after upload, so the code exercised on the box is byte-identical to the code being submitted.

Not tested

  • XPU / MPS backends.
  • End-to-end ShardFormer training or inference on NPU — only the three functions above were exercised directly.
  • The test container does not have every optional ColossalAI dependency installed (galore_torch, bitsandbytes, diffusers, PIL are absent), so those imports were stubbed while loading the package. No ColossalAI source was modified for the test.

@li-lizhe
li-lizhe requested a review from a team as a code owner September 26, 2026 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fx tracer / rope init hard-code the device, so NPU runs fall back to CPU (or crash)

1 participant