Record: Deploy amd-mi25-fanctl on beast.lan — 2026-08-24

Outcome

Deployed the new C++ fan-control service on beast.lan, replacing the old shell service, with staged risk controls.

Phases completed (0-2; reboot verification deferred to a later session):

• Phase 0: hardened startup (profile/level write failures log and continue; only pwm1_enable/pwm1 failures stay fatal). Built on beast.lan. Live –check found the MI25 controllable.
• Phase 1: staged binary, unit, and rollback script; systemd-analyze verify passed; new unit left disabled.
• Phase 2: swapped old service for new (stop old, start new). Verified active, journald reporting, temps stable.

Current state (verified 2026-08-24):

• amd-mi25-fanctl.service: enabled + active.
• mi25-fanctl.service: disabled + inactive (unit file kept for rollback).
• MI25 at equilibrium: edge 45-48 C, junction 48-49 C, mem 47-48 C, pwm1 71, pwm1_enable 1.
• Rollback: /usr/local/bin/amd-mi25-fanctl-rollback.sh re-enables the old service.

Decisions

• Keep the old service files in place until the new service survives a reboot; rollback is pure re-enable, no file surgery.
• Emergency brake: human watching; physical power is the ultimate stop.
• Reboot verification is a separate, agreed session.

Incidents

• /proc/modules read truncated at 4096 bytes, missing the DKMS (OE) flag (amdgpu line sits at byte offset 6064). Fixed with a line-wise read; out-of-tree detection now reports yes. No fan-control impact.

Verification

• make build and fixture tests pass on beast.lan.
• Service active and controlling: PWM ramped to target, temps steady.
• Verified 2026-08-24.

Phase 4 — reboot verification (2026-08-24)

Rebooted beast.lan with the service enabled.

Post-boot state (verified 2026-08-24):

• amd-mi25-fanctl.service: active + enabled, NRestarts=0.
• mi25-fanctl.service: inactive (disabled).
• Service started once on boot and found the hwmon node immediately: “using hwmon=/sys/class/drm/card1/device/hwmon/hwmon1” at 05:11:11.
• No discovery retries, no restarts: the boot-time hwmon race is fixed without the old ExecStartPre sleep hack.
• Temps at boot: edge 44, junction 45, mem 45 C; settled pwm 62, enable 1.

The reboot phase is complete; the old service can be removed at the owner’s discretion (rollback script remains available until then).

Host characteristic (owner note, 2026-08-24): the HP z820 can take quite a while to reboot. Measured on this reboot: roughly 1-1.5 minutes from power-on to kernel start (POST/bootloader), then 1 second from kernel start to fan control (kernel 05:11:10, service active 05:11:11). The long window has no driver and no fan control, but also no compute load; boot temps were 44-45 C. A long reboot is expected behavior, not a hang.

Old service removed (2026-08-24)

After the reboot verification and the LLM load exercise, the owner approved removing the old service files from beast.lan:

• /etc/systemd/system/mi25-fanctl.service — removed.
• /usr/local/bin/mi25-fan-control.sh — removed.
• /usr/local/bin/amd-mi25-fanctl-rollback.sh — removed; its purpose was to stop the new service and start the old one, which no longer exists. Running it after the removal would have left no fan control at all.

The rollback script was also deleted from the repository. The new service is the only fan-control path now. Verified after removal: amd-mi25-fanctl.service active, temps steady.

Commits

• 4bdd454 fix: startup profile and level failures no longer block fan control
• 6e17380 fix: read /proc/modules line-wise so DKMS (OE) flag is detected