ROCm on Windows Performance Troubleshooting: Why Is the Speedup Unimpressive When Running PyTorch on an RX 6650 XT?

Preface

Here's the story: I have an RX 6650 XT (gfx1032) and wanted to run PyTorch on Windows for deep learning. AMD's official HIP SDK for Windows does not formally support gfx1032, but I still got the environment working by building ROCm 7.11.0a + PyTorch 2.9.1 from source.

torch.cuda.is_available() returned True, rocm-sdk test also passed, and Qwen3-0.6B could be loaded onto the GPU to generate text. Everything looked fine.

But when I ran a benchmark, the GPU speedup was only 1.7-2.0x, and GPU utilization in Task Manager wasn't high either. What's going on?

Test Environment

Benchmark Results

Ran 3 times for each of 5 prompt types, max_new_tokens=256:

                    CPU vs GPU Comparison
╔════════════════════╤══════════════╤══════════════╤════════════╗
║ Prompt             │    CPU tok/s │    GPU tok/s │    Speedup ║
╟────────────────────┼──────────────┼──────────────┼────────────╢
║ Short Factual      │         16.8 │         30.0 │       1.8x ║
║ Summarization      │         14.8 │         30.3 │       2.0x ║
║ Reasoning          │         18.0 │         30.2 │       1.7x ║
║ Code Generation    │         18.4 │         30.4 │       1.7x ║
║ Long-form          │         18.0 │         30.6 │       1.7x ║
╚════════════════════╧══════════════╧══════════════╧════════════╝

A few straightforward observations:

• The GPU path does work; speedup is about 1.7-2.0x.
• But for a discrete GPU, this speedup really isn't great.
• GPU-side throughput is stable at ~30 tok/s, roughly the same no matter how the prompt type changes—meaning the bottleneck isn't in the prompt content.

First, Let's See What AMD Says

In the official support matrix for AMD HIP SDK for Windows, RDNA2 support by SKU looks like this:

Runtime support means the HIP/OpenCL runtime can run; HIP SDK not supported means the official prebuilt HIP SDK libraries do not cover this hardware, and various issues may appear at runtime.

This is the most critical point: gfx1032 is simply not within the official HIP SDK support scope on Windows. So even if a from-source build can run, you should be mentally prepared for compatibility and performance.

Smoke Test: Which Operations Can Actually Run on the GPU?

To figure out how PyTorch operations actually behave in the current environment, I wrote a smoke-test script (tests/test_operators.py). The idea is simple: create CUDA/HIP tensors, run operations, and check whether the outputs are still on the GPU.

Results:

Summary: 54/55 returned CUDA/HIP tensors | 1 errors | 1 warnings

Of 55 checks, 54 successfully returned CUDA/HIP tensors. The two that failed:

batch_norm errored—when MIOpen compiles kernels at runtime, HIPRTC cannot find the C++ standard library header <type_traits>. This is a build-environment issue, not a lack of GPU support. Installing the C++ toolchain from VS Build Tools afterward should fix it. But note that this does not mean BatchNorm will necessarily have a high-performance GPU kernel after the fix—it still needs to be retested.

sdpa had a warning—memory efficient attention was not enabled when PyTorch was built, so scaled_dot_product_attention takes the math fallback. Functionality is fine, but the performance gap will be more noticeable with long sequences and large batches.

It needs to be emphasized that this test result cannot be directly interpreted as "54/55 operators have GPU kernels, 98% coverage".

Because checking the output device only proves that PyTorch's device semantics hold (the result of a CUDA tensor is still on CUDA); it does not prove which backend was called underneath, which kernel was used, or whether there was implicit synchronization.

What's Really Behind the Low Speedup?

To be honest, I don't have a definitive answer. The most reasonable judgment at this point is that multiple factors are stacking:

Autoregressive decode inherently underutilizes the GPU. Only one token is generated at a time; with batch=1, kernel granularity is tiny, so launch latency and scheduling overhead take a larger share. This is a common problem in LLM inference, not unique to ROCm.

Qwen3-0.6B is too small. 600M parameters, fp16 weights are only 1.2GB. For a small model in the batch=1 decode stage, the bottleneck is often memory access and framework overhead, not GPU compute.

SDPA is not taking the high-performance path. The compile-time warning already indicated that memory efficient attention was not enabled. For Transformer models, this is a non-trivial performance factor.

Windows + gfx1032 itself is not an officially supported scenario. Building from source can get operations to run, but in terms of kernel selection, library integration, and performance tuning, it's hard to expect the same as officially supported hardware.

High CPU utilization ≠ the forward pass is computed on the CPU. The ~20% CPU usage in Task Manager is actually mostly the Python interpreter, the Transformers generation loop, tokenizer processing, kernel launches, and synchronization; you cannot directly infer that "most of the computation fell back to the CPU".

What to Investigate Next

To investigate further, these things may still need to be done:

1. Use torch.profiler to look at the time distribution of CPU and CUDA/HIP activity.
2. Time prefill and decode separately to see where the bottleneck actually is.
3. Benchmark the critical paths—linear, RMSNorm, RoPE, SDPA, KV cache—separately with real LLM tensor shapes.
4. Enable MIOpen/rocBLAS logs to confirm which library the key operations actually go through.
5. Try different batch sizes and see how the tok/s curve changes.
6. If conditions allow, compare against Linux ROCm or an officially supported RDNA3 GPU.

Conclusion

Wrap-up

For friends who also want to run deep learning on RDNA2 + Windows:

1. Treat a from-source ROCm/PyTorch build as an experimental environment, not a production one.
2. Run a smoke test first to confirm that common operations don't error out immediately.
3. Performance issues need to be pinpointed with a profiler and backend logs; looking at the output device alone doesn't tell you much.
4. For LLM inference, separately analyze prefill/decode, batch size, and the attention backend.
5. If you want stable runs and predictable performance, prefer Linux ROCm, or switch to a GPU that is officially supported on Windows.