Covers Wed 7 Oct 2026 (week 41), 18:15–21:00 PDT: aifoundry1, aifoundry2 and aifoundry3 in the AI Foundry lab at Studio 45. Written Wed 7 Oct 2026 in answer to Roman's question in #community-lab at 18:28. Read-only checks on aifoundry1 and aifoundry3. aifoundry2 is off the network. Sources: the lab repo, the open firmware and the board's manual. The analysis of the runaway itself is on the 6 October page.
Roman's question: no power-management option was changed. On 6 October only Smart Fan 6 was touched on aifoundry2: CPU_FAN to Full Speed, copied to every header with TUNE ALL, then F10. Settings → Platform Power (AC BACK, ErP, Resume by Alarm) was never opened. The machine then booted twice and ran for 25 hours, so the fan change can't block a power-on.
Why aifoundry2 stayed dark tonight, most likely first:
- AC BACK is still at the board's factory default, Always Off. At about 18:16:40 all three machines lost mains power together. aifoundry1 and aifoundry3 came back on their own 40–70 s later, as they have after every cut on record (18 Sep, 30 Sep, 7 Oct). Setting aifoundry2's AC BACK to Always On has been on the visit list since before 2 October, and no visit has done it. Against this: aifoundry2 did come back after the 30 Sep cut, but Roman was on site that day and may have pressed its button.
- A boot stop nobody can see. For example a "Press F1" halt after a CMOS checksum error on a five-year-old coin cell, which would also silently undo the 6 October fan setting.
- The card holding the boot. After the 138 °C runaway it is in an unknown state. Against this: the same host booted twice with the card already bad on 2 and 6 October.
- Running but unreachable: Wi-Fi didn't connect, or the root filesystem dropped to a repair shell.
From outside these all look the same: the wired ports carry no address, and the boards have no remote console. Someone on site has to look first, then press.
The three machines side by side
| aifoundry1 | aifoundry2 | aifoundry3 | Why it matters | |
|---|---|---|---|---|
| Board, BIOS | Z590 AORUS MASTER, F5 (Jun 2021) | same board; BIOS not read from the machine | same board, F6 (Aug 2021) | Nothing from F4 to F10 changes fan control, PCIe or power-loss behaviour; no reason to update now (F7 and later can't be rolled back) |
| CPU | i7-11700K | i5-11600 | i7-11700K | aifoundry2 is the odd one |
| Comes back by itself after a power cut | yes, every time on record | not tonight; AC BACK never set | yes, every time on record | The leading reason aifoundry2 is dark |
| Remote power and console | One remote-controlled outlet switches all three together; no per-machine outlet, no remote console, no Wake-on-LAN (the network chip offers none), no management controller | Any remote power cycle hits all three, and only a visit sees a stuck boot | ||
| Chassis fan setting | stock curve, follows the CPU | every header at Full Speed since 6 Oct | stock curve, follows the CPU | Only aifoundry2 was changed, and the setting lives only in its CMOS |
| Fan-fail warning | off | off | off | No machine reports a stopped header fan |
| Fan speeds visible from Linux | no | no | no | A stopped fan shows only as the die heating at constant power |
| The card's own fan | card 0's was found stuck and replaced on 2 Oct; card 1's original | never inspected: the prime suspect | never inspected | Powered by the card, no speed sensor, invisible to everything |
| Card slot | card 0 in PCIEX16, card 1 in PCIEX8 | PCIEX8 | PCIEX8 | aifoundry2 and aifoundry3 share the slot, so the slot isn't the difference |
| PCIe power saving (ASPM) | off (BIOS) | off (same board, assumed) | off (BIOS) | Neither a cause nor a lever |
| Above 4G decoding | on | not read | on | A CMOS clear turns it off, and then the card looks dead |
| Card firmware | card 0: 1.4.1 (BL2 0.21.2); card 1: 1.2.0 (BL2 0.18.0) | 1.3.1 (BL2 0.20.0) | 1.3.1 (BL2 0.20.0) | Four cards, three builds |
| What its firmware does to a hot idle card | card 0 parks at 300 MHz and 398 mV: 18.6–20.8 W idle. Card 1: nothing | nothing: the 65 °C loop is already at its floor, the safe state is a no-op | nothing: its 0 W power limit wedges the governor, every throttle counter reads 0 | Only aifoundry1's card 0 has a real low-power idle |
| Static power limit (TDP) | 65 W | 65 W | 0 W, set at every boot by et-board-clock-guard | The driver reports 65 W for all four regardless |
| Idle card, 7 Oct | card 0: 49–53 °C, 19–21 W; card 1: 53–59 °C, 30–34 W | 64 → 138 °C, then off the bus | 54–58 °C, 24–26 W | |
| Other work a reboot interrupts | a GitHub Actions runner | the lab dashboard and history jobs | a demo and a chatbot that launch card jobs | The dashboard froze at 17:50 because it runs on the machine that died |
Other ways to stop the runaway, ranked
Three independent designers proposed 66 fixes from three angles: airflow, host software, and firmware. Two skeptics then attacked each one, one on the physics and one on the risk. Most of what survives buys recovery or warning rather than margin. Only three proposals move the tipping point itself, and their size is uncertain:
Only the card's fan, its firmware and better airflow move the tipping point
How much warmer the air at the card could get before the idle card runs away, gained by each fix (°C, the skeptics' corrected ranges)
proposed fix, rangealready done
| # | Fix | What it buys | Who |
|---|---|---|---|
| 1 | Inspect, and if needed replace, aifoundry2's card fan; clean the fins, clear the intake, record the part, stock two spares | +9 to +20 °C of margin if the fan is the fault; otherwise about +5 °C, the 03:55 step back. No cut-off | on site |
| 2 | Photograph every BIOS page first, then set AC BACK = Always On; confirm ErP Disabled and Above 4G decoding Enabled, on all three | No thermal effect. The machine returns by itself after a cut (five lab-wide cuts since July). This must come before any remote power plan | on site |
| 3 | An alarm that doesn't live on the machine it watches: a step in implied cooling resistance, plus a die-temperature ladder, posted to Discord | Warning, not protection. Replayed on 7 Oct, the step rule fires at 03:55–04:40, 12–13 hours before the bus drop, with no false alarms over six healthy card-days. The ladder fires 2 hours ahead at 85 °C and 12 minutes ahead at 95 °C | lab door |
| 4 | One switched outlet per machine, a remote console with power control on aifoundry2, a UPS upstream | Recovery in about 3 minutes instead of a visit. Works only with fix 2. A remote console is what sees a stuck boot | Roman to buy, on site to fit |
| 5 | A fan on a board header, ducted into the card's fin stack, at Full Speed | +4 to +9 °C, about half the first estimate. On a board header, a fan-fail warning can see it | on site, about $30 |
| 6 | Measure the air at each card (the board's spare thermistor in EC_TEMP1, or a logging probe) | No thermal effect, but it settles the biggest unknown: every margin above assumes the inlet temperature | on site |
| 7 | Move aifoundry2 and aifoundry3 to a 1.4.1-class build, the one on aifoundry1's card 0, including its flash settings | +6 to +10 °C, from the 300 MHz / 398 mV idle that 0.20.0's own floors can't reach. Vendor-gated; check read-only first whether it's code or flash settings | Ainekko to supply and sign; lab to flash |
| 8 | Say yes to Roman's per-sensor patch, with amendments: raw 12-bit readings of all 35 sensors, and don't serve the group that resets the peak-hold | No thermal effect: diagnosis, and the easiest first signed build | answered separately |
| 9 | One message to Ainekko: the signed 1.4.1 image and its rollback, the dead PMIC alarm, a real cut-off, the API traps (questions below) | Unblocks fix 7 and any real cut-off | Yaroslav to Roman, Gianluca |
| 10 | Write down the only recovery that works (the full cold reset), and gate any automatic action on a human until fixes 2 and 4 exist | A known path back, without a second outage | lab door |
| 11 | Fan-fail warning on populated headers only, once a remote console exists | Catches a stopped board-header fan; can't see the card's own fan, and may halt boot | on site |
| 12 | Out-of-tree it87 driver, read-only, for board fan speeds and slot temperatures | Attribution (which fan), not protection; risky enough to defer | host root |
| 13 | Lab account in adm so kernel logs are readable | aifoundry2's log of 16:46 is on its disk, but no lab account can read it | Roman |
| 14 | A UPS, and a rule: nobody cuts the shared strip to reset one machine | Protects the disks (all three run the Samsung 980 PRO firmware with the known bug) and the uptime of the other two | Roman |
| 15 | Fixed addresses on the wired ports, as a second diagnostic path | Tells "running but off Wi-Fi" apart, from a sibling | host root |
| 16 | Collector fixes: rename the peak-hold field, fsync each record, show the real TDP | Truer records; nothing may ever trip on the peak-hold field | lab door |
Ideas that don't survive scrutiny
Both skeptics refuted each of these:
- Lower the idle clock to 333–400 MHz when warm. About 1 W, about +1 °C. The tipping point doesn't move when only the fixed power changes; leakage needs a lower voltage.
- Drop the minion rail to 460–480 mV at runtime. On 0.20.0 a failed voltage write retries inside a critical section until the 10 s watchdog resets the card. The fix for that (SW-21990) isn't in any lab card's firmware.
- Write a 300 MHz / 460 mV row into the operating-point table. The 300 MHz safe-state voltages bypass the table, and its 650 mV memory voltage is below the PMIC's own floor.
- Raise aifoundry3's 0 W power limit to restore its governor. That re-enables the climb to 800 MHz its admin blocked on purpose.
- Ask for a 20 W power-limit floor. It still wedges three of the four cards at their measured idle power.
- An idle trip routed into the fixed safe state. On 0.20.0 plus the fix, 300 MHz isn't an exact entry in the clock table, so it logs a line and changes nothing.
- Suspend-and-wake cool-downs (
rtcwake), or cycling the card on a temperature band. The ET driver has no suspend support, and no software path unpowers the card for minutes. - An automatic cold reset when the card goes missing. Replayed over 2–7 Oct it fires 99 times for one useful recovery, and only after the runaway has already happened.
- Swap the two cards between machines, or move one to another slot, to separate card from chassis. The metric drifts 12–17% a day on untouched cards, as much as the effect sought.
- Key the chassis fans to a slot sensor, or leave Full Speed for a thermistor curve. At Full Speed the sensor isn't consulted; any curve gives less air.
- A panel-off box-fan audit. One steady state can't separate the air temperature from the resistance.
- A three-step campaign to measure leakage against voltage. Along the operating-point table voltage and clock move together, so the exponent can't be identified, and no card can run it today.
The next visit, in order
- Before touching aifoundry2: look and listen. Are the case and power-supply fans spinning? What does the board's POST-code display or its CPU, DRAM, VGA and BOOT lights show? That separates off from running-but-unreachable.
- Attach a monitor. A POST code, an F1 prompt or a repair shell answers the question. Photograph it.
- Press the power button, briefly, only if the machine is dark. Never hold it for 4 seconds. If it's running, get the kernel logs off it first.
- Never Clear CMOS or Load Optimized Defaults. Above 4G decoding reverts, and the card looks dead.
- In the BIOS, photograph before changing anything: the version; Platform Power (AC BACK, ErP, Resume by Alarm); IO Ports (Above 4G); every Smart Fan 6 page; PC Health Status with every fan's RPM and both slot temperatures. aifoundry2's current AC BACK value is the only test of the leading hypothesis.
- Then set AC BACK = Always On. Confirm ErP Disabled and Above 4G Enabled. If the fans are back on the stock curve, CMOS was lost: replace the coin cell and re-apply Full Speed. Save both profiles to a USB stick.
- Power down, pull aifoundry2's own plug (never the shared strip) and open the case. Photograph the card, its fan and the air path, and write down the fan's markings.
- Spin the card fan and every chassis fan by hand. Replace anything stiff, keep the old part, clean the fins, clear the intake.
- Fit a thermistor in the air ahead of the card, and a ducted fan on a board header. Set that header to Full Speed.
- Bring the card back with the full cold reset only. Watch for 60–90 minutes. Switch off above 85 °C.
- Test AC BACK on site: clean shutdown, pull the plug for 10 s, plug it back in. The machine should start by itself.
- No BIOS update on this visit. On aifoundry1 and aifoundry3, set AC BACK at the next reboot that was going to happen anyway.
Questions for Ainekko
- aifoundry1 card 0's 300 MHz / 398 mV idle: does it come from BL2 0.21.x code, or from that card's flash (its boot frequency and operating-point table)? If it's the flash, a firmware-only update onto the other cards gains nothing.
- Firmware signing: is the OTP signature "chicken bit" set on these parts? That decides whether the lab can ever boot its own image.
- The 1.4.1 image: please send the signed image that runs on card 0, with its update and rollback procedure. Is the version counter one-way?
- Fixes in 0.21.1 and 0.21.2: neither is in the public tree. Do they contain SW-21990 (the bounded voltage retry) and SW-22053?
- The PMIC alarm has never fired. The throttle counters read 0 through an 87 W pass and a 133 W runaway. Is its interrupt line routed on these boards? Why does the PMIC's temperature read 0?
- Master skips all thermal evaluation on an idle card. Is that intended? What is meant to protect one?
- A last-resort trip: would you accept one that calls
pmic_force_power_off_on()above a hard ceiling, using a live maximum across the 34 sensors, on an idle card? - Specs: is there any ET-SoC-1 airflow, inlet or junction-temperature specification? And what part is the card's own fan?
Sources and method
- Hosts: read-only checks over ssh on aifoundry1 and aifoundry3, 18:35–19:00 PDT: DMI,
/proc,/sys,systemd-analyze,last,tailscale status, the live collectors' records. No device node was opened and nothing was written. aifoundry2 has been unreachable since about 18:20, so its column comes from the lab repo, its collector's records to 16:48 and the 6 October visit record. - Card firmware: et-platform at
ffca4cbb4(the cards' BL2 0.20.0) and836a4ab(newest), et-boards, and the lab repo's findings 14 and 16. - The board: Gigabyte's Z590 AORUS MASTER manual (rev 1002) and BIOS release notes.
- Discord:
#community-lab(Roman's messages of 7 Oct; the shared remote outlet, from earlier messages by Roman and Afonso Oliveira). - Method: 141 agents. Five surveys produced 225 facts; three designers proposed 66 fixes; two skeptics judged each fix, one on the physics, one on the risk; one synthesis. Margins use the model on the 6 October page; they are model estimates that move one-for-one with the assumed inlet temperature.