cross-posted from: https://lemmy.ml/post/37817953
Hi all, when I am using software with high gpu load(in the case AI model). It also happens with game. It just kinda happens after a random amount of with games(I can play for like 30 mins then crash or sometime not at all).
here is my journalctl log:
Oct 20 12:57:18 Linux kernel: amdgpu 0000:10:00.0: amdgpu: Dumping IP State
Oct 20 12:57:18 Linux kernel: amdgpu 0000:10:00.0: amdgpu: Dumping IP State Completed
Oct 20 12:57:18 Linux kernel: amdgpu 0000:10:00.0: amdgpu: [drm] AMDGPU device coredump file has been created
Oct 20 12:57:18 Linux kernel: amdgpu 0000:10:00.0: amdgpu: [drm] Check your /sys/class/drm/card1/device/devcoredump/data
Oct 20 12:57:18 Linux kernel: amdgpu 0000:10:00.0: amdgpu: ring comp_1.1.1 timeout, signaled seq=618, emitted seq=620
Oct 20 12:57:18 Linux kernel: amdgpu 0000:10:00.0: amdgpu: Process python pid 4571 thread python pid 5777
Oct 20 12:57:18 Linux kernel: amdgpu 0000:10:00.0: amdgpu: GPU reset begin!
Oct 20 12:57:18 Linux kernel: amdgpu 0000:10:00.0: amdgpu: device lost from bus!
Oct 20 12:57:18 Linux kernel: amdgpu 0000:10:00.0: [drm] device wedged, but recovered through reset
Oct 20 12:57:18 Linux kernel: amdgpu 0000:10:00.0: [drm] *ERROR* [CRTC:61:crtc-0] flip_done timed out
I tried to check the path /sys/class/drm/card1/device/devcoredump/data after reboot, but there isn’t any thing(in fact, devcoredump folder dont even exist.
My specs: Distro: Arch Kernel: 6.17.3.arch2-1 Driver: Mesa 1:25.2.4-2 Gpu: rx 580 Cpu: r5 5500 PSU: EVGA 650 N1 650w I am on latest version of my bios)
Edit: my
Is there anything I can do to diagnose the issue? Any help is appreciated. Thanks you!
Solved my GPU is dead
Distro? Driver version? Temperature? Is it receiving enough power?
If everything checks out, it might just be defective.
deleted by creator
what/how do you run LLM on a RX 580? I thought ROCM was for RX 6xxx and newer?
deleted by creator
one possibly expensive way to find out is to add an expensive cooling solution to it to see if it stays active.
I have two machines running the latest kernels on EndeavourOS. One with a Radeon RX 7900 XTX has no issues.
The other one has a Radeon 6650 XT, which since a week or two ago starts getting kworker threads stuck while throwing errors about fence queues. Load can go up to the hundreds (while there’s no real load, but just blocked threads), until the machine crashes.
As I recall there was an amdgpu firmware update around the time it started happening, but the changelog on the amdgpu kernel driver hints at solving similar issues.
deleted by creator
When I experienced the same symptoms, i eventually found out if was because ROCm didn’t support having an AMD GPU as well as an AMD iGPU (iGPU is an integrated GPU, on the motherboard). Once i disabled the iGPU, those symptoms stopped.
l don’t remember how i disabled the iGPU. Might have been in the bios settings, might have been a kernel parameteretc in
/default/grub.If it doesn’t fix your issue, you can just re-enable the iGPU.
deleted by creator
This happens to me when I run games sometimes in 4k at max settings, with a 7900XTX. So far I have not found anything that prevents it, and I’m starting to suspect my power supply or my house’s wiring might be the issue. It almost seems like a voltage sag.
deleted by creator
I experience something similar on a vega56, but it doesn’t happen on generic high loads, it happens only
- in some specific loads (for me it was mainly A Plague Tale: Requiem)
- when something touches the grounding of my monitor (the display port of the GPU is faulty, it seems)
See my other comment: https://lemmy.world/comment/20051971
Do you have a powerful/decent/not-too-old enough PSU?.
deleted by creator
Long time ago, after lot of headaches, I discovered the PSU was the culprit. Changing it for another with more power/better quality solved it in a instant. I mentioned it because is something you might not think about initially. Good luck!.
