Kernel Failure Behavior and Service Failure Policy

Rough draft for review, 2026-08-23.

This document records the research spike on what the Linux kernel and the amdgpu driver do when hwmon sysfs reads or writes fail at runtime. It defines the failure policy for the fan-control service.

Motivation

The MI25 is a datacenter GPU whose BIOS cannot control the fan. Without fan control, it overheats and shuts down the server. The service must re-establish fan control if at all possible. Understanding the kernel failure modes determines whether the service retries, exits, or degrades.

Kernel failure taxonomy

Verified 2026-08-23 against the kernel sources and the live system.

-EBUSY: device busy or not ready

The amdgpu driver gates every hwmon access through amdgpu_pm_dev_state_check (amdgpu_pm.c). It returns -EBUSY when:

• the GPU is in reset (amdgpu_in_reset).
• the device is not fully initialized (init level below default).
• the device is in suspend and the access is not a runtime-PM resume.

The sysfs show and store functions propagate this error to the reader or writer. A read or write during GPU reset or suspend fails with -EBUSY.

This is a transient condition. The GPU reset completes and the device resumes.

-ENODEV: device removed

kernfs returns -ENODEV when the sysfs attribute node has been deactivated (fs/kernfs/file.c, kernfs_get_active_of). This happens after device removal: amdgpu_pci_remove calls drm_dev_unplug, which unregisters the DRM device and its sysfs tree. A read or write after removal fails with -ENODEV.

This is a permanent condition for the removed device. The hwmon directory disappears.

-EINVAL: interface misuse

The pwm1 store path rejects writes that are not in manual mode (amdgpu_hwmon_set_pwm1, amdgpu_pm.c). It logs “manual fan speed control should be enabled first” and returns -EINVAL. This is a service sequencing error, not a hardware failure: write pwm1_enable=1 before pwm1.

The pp_power_profile_mode store path rejects writes unless the performance level is manual. This is documented in amdgpu_pm.c (DOC: pp_power_profile_mode): “To select a profile or create a custom profile, first select manual using power_dpm_force_performance_level.” Verified live on beast.lan: writing profile 5 fails with -EINVAL under forced-high and under auto; it succeeds after writing manual.

Silent no-op: bNoFan

The Vega 10 fan-control path checks bNoFan (vega10_thermal.c, vega10_fan_ctrl_set_fan_speed_pwm). If bNoFan is set, the write returns 0 without changing the fan. bNoFan comes from the VBIOS fan table (vega10_processpptables.c, ATOM_VEGA10_PP_FANPARAMETERS_NOFAN).

On the MI25 on beast.lan, the fan attributes exist, so bNoFan is false. But a future driver or a different card could set it. A pwm1 write that returns success without effect is a control failure.

-EOPNOTSUPP: interface not supported

Some amdgpu power interfaces return -EOPNOTSUPP when the ASIC or driver version does not implement them. The fan service does not depend on any of these.

Service failure policy

The policy follows from the taxonomy.

Transient failure (-EBUSY): retry, keep last good PWM

• Log the failure at a bounded rate.
• Keep the last good PWM value on the fan.
• Keep retrying the read or write on the next loop iteration.
• Do not exit.

This covers GPU reset, suspend, and boot-time initialization.

Permanent failure (-ENODEV): exit nonzero

• Log the removal.
• Exit nonzero so systemd records the failure.
• systemd Restart=always will restart the service, which will re-run discovery; if the device is truly gone, discovery fails and the service logs and exits again.

Sequencing error (-EINVAL on pwm1): re-assert manual mode

• Write pwm1_enable=1 again.
• Retry the pwm1 write.
• If the error persists, treat it as a control failure.

Silent no-op (bNoFan): treat as control failure

• The service cannot detect a silent no-op from the write return value.
• The service checks pwm1 read-back after a write when possible.
• If the read-back never changes while the write succeeds, log a control failure and keep retrying.
• This is a safety fallback, not a common path on the current card.

Startup discovery failure: exit nonzero after timeout

• Discovery retries while the hwmon node appears at boot.
• On timeout, log and exit nonzero for systemd to restart.

Interaction with the COMPUTE profile

The service writes power_dpm_force_performance_level = manual and pp_power_profile_mode = 5 (COMPUTE) at startup. Verified live on beast.lan: this sequence succeeds. The COMPUTE profile lowers the SMU busy set point from 70 to 30 (vega10_hwmgr.c profile table), so the SMU boosts clocks eagerly under compute load and idles low. This matches the homelab goal: quiet and low-power at idle, maximum performance when running an LLM. The profile is a runtime setting; the service must re-apply it at every startup, after every GPU reset, and whenever a -EBUSY sequence ends.

References

• ~/sources/linux/drivers/gpu/drm/amd/pm/amdgpu_pm.c
• ~/sources/linux/drivers/gpu/drm/amd/pm/amdgpu_dpm.c
• ~/sources/linux/drivers/gpu/drm/amd/pm/powerplay/hwmgr/vega10_thermal.c
• ~/sources/linux/drivers/gpu/drm/amd/pm/powerplay/hwmgr/vega10_hwmgr.c
• ~/sources/linux/fs/kernfs/file.c
• Live verification on beast.lan: verified 2026-08-23.