Spinning Down SAS Disks

I keep adding storage to my R730xd, but I don’t want the power bill to follow. Every extra disk draws power all day, read or not.

The pool is 12 SAS drives with media and backups. They’re used during the nightly backup window and when someone streams a film. The rest of the day they spin for nothing. Parking them when idle drops the server from 168W to 126W — about 1 kWh a day.

Making them stay parked took most of a day. They kept waking up with nothing in the logs to explain it.

What this covers

Getting SAS drives to spin down on Linux and stay down. SAS ignores ATA standby, so hdparm does nothing, and hd-idle stops working after the first wake-up. The fix is one detail plus a script on a timer.

What you’ll need

• Linux host with SAS drives behind an HBA or a controller in JBOD/IT mode
• sg3-utils and smartmontools
• Root access
• Optional: storcli (disables controller patrol read), a BMC (power readings)

Before you start

Parking a disk isn’t destructive — it wakes on the next read. Two things are still worth knowing:

• First access after a park is slow. Spin-up takes 10-20 seconds. Anything with a timeout shorter than that on those disks (a database, a VM image, an iSCSI target) will notice. Keep spin-down to data you’re happy to wait for.
• Some drives need an explicit start. Most wake on their own. A few models park into a state that needs sg_start --start /dev/sgN first. Test one disk and read it back before doing this to a whole array — if a drive doesn’t come back on its own, your array will notice before you do.

Is this worth doing for you?

smartctl -a /dev/sgN | grep -iE "start-stop|Specified cycle count"

Step 1 — Confirm the one detail on your hardware

The standby command has to go to the generic SCSI device, not the block device:

sg_start --pc=3 /dev/sdp    # wrong - kernel spins it straight back up
sg_start --pc=3 /dev/sg15   # right - stays down

Both return success. Both stop the platter. But closing a block device makes the kernel revalidate it, which spins the drive back up. That happens below the block layer, so /proc/diskstats shows nothing — no reads, no writes, no flushes. The disk looks like it woke up on its own.

All 12 of mine, same command: /dev/sdX → 154-170W, awake in seconds. /dev/sgN → 131-136W, stays down.

Check it yourself before going further:

sg=$(basename $(readlink -f /sys/block/sdp/device/generic))   # -> sg15
sg_start --pc=3 "/dev/$sg"
sleep 30
smartctl -i -n standby "/dev/$sg" | tail -1

STANDBY BY COMMAND means it held. Power mode is: ACTIVE means something woke it — which is what you get every time via /dev/sdX.

Step 2 — Make sure nothing is writing

Disks only sleep if nothing touches them. One chatty process is enough to keep a whole pool busy.

grep -E " sd[a-z]+ " /proc/diskstats | awk '{print $3, $4+$8}'; sleep 60; \
grep -E " sd[a-z]+ " /proc/diskstats | awk '{print $3, $4+$8}'   # must match

The counters must match. If they don’t, something is writing.

Mine were invisible to find: Garage’s LMDB heartbeat (moved to SSD) and Jellyfin’s real-time library monitor (turned off, scheduled scan instead). Media servers watching folders and object stores with metadata heartbeats are the usual suspects.

Step 3 — Install the enforcer

A script on a timer, not a daemon. Anything that remembers state gets it wrong — a disk woken by a web UI is invisible to diskstats, so a daemon stops issuing standby and the disk stays up. A script that re-checks the real power state every 5 minutes parks it again regardless.

Decide once for the pool, never per disk

This is the part I got wrong the first time, and it is the whole reason for the update at the top.

My pool is two RAIDZ2 vdevs of six. During a film, ZFS reads records from whichever vdev holds them, so one vdev can go ten minutes without a single request while the other streams. A per-disk rule sees six idle disks and parks them. The next record lands there, and all six spin up at once. Nothing is wrong from the pool’s point of view — but the disks never settle, and each cycle is expensive in a way that has nothing to do with the disks.

Every wake answers SCSI sense 2/04/01 — not ready, becoming ready. megaraid_sas handles that by kicking off a blocking poll of the physical-drive list, and everything else queued on that controller waits behind it. On my H730P that includes the four SSDs holding the Kubernetes control-plane VMs. So a parked media disk waking up stalls etcd’s write-ahead log on all three nodes at the same time.

What that cost, measured over one day of the per-disk rule:

Six days of that went unnoticed. The bundled Prometheus etcd alerts never fired: they want p99 sustained over ten minutes, or four elections inside fifteen, and this was sharp spikes recovering in one to four minutes, spread thin across whole days. Alerts tuned for a cliff miss a slope.

The fix is smaller than the bug. Ask the pool, not the disks: if it moved zero bytes for two consecutive runs, park everything that is awake. No per-disk state, no threshold to calibrate, no /proc/diskstats.

I did try an intermediate version that gated on the pool but needed a 256 MB threshold, because qBittorrent seeding out of the library trickles 5–11 MB a minute forever and that alone held all twelve disks up. I dropped it: a constant tuned to your own library’s bitrates is not something you will still understand in a year, and while seeding runs the pool genuinely is in use. Let the seeding limits bound that, not a magic number in a shell script.

Disk list, derived from the pool so a disk swap needs no edits:

cat > /root/scripts/sas-disks.sh <<'SH'
#!/bin/bash
# Prints "sdX sgN" per line for the pool's spinning members.
set -uo pipefail
POOL="${SAS_POOL:-media}"
zpool status "$POOL" 2>/dev/null | grep -oE "wwn-0x[0-9a-f]+" | sort -u | while read -r wwn; do
  d=$(basename "$(readlink -f "/dev/disk/by-id/$wwn" 2>/dev/null)" 2>/dev/null)
  { [ -z "$d" ] || [ ! -e "/sys/block/$d" ]; } && continue
  [ "$(cat "/sys/block/$d/queue/rotational" 2>/dev/null)" = "1" ] || continue
  sg=$(basename "$(readlink -f "/sys/block/$d/device/generic" 2>/dev/null)" 2>/dev/null)
  { [ -n "$sg" ] && [ -e "/dev/$sg" ]; } && echo "$d $sg"
done
SH

The enforcer. One counter for the pool, two quiet runs, park what’s awake:

cat > /root/scripts/sas-spindown.sh <<'SH'
#!/bin/bash
set -uo pipefail
exec 9>/run/sas-spindown.lock
flock -n 9 || exit 0
STATE=/run/sas-spindown.state   # "<pool_bytes> <idle_runs>" - tmpfs, resets on reboot
POOL="${SAS_POOL:-media}"
if ! mapfile -t DISKS < <(/root/scripts/sas-disks.sh) || [ "${#DISKS[@]}" -eq 0 ]; then
  logger -t sas-spindown "ERROR: no disks returned"; exit 1
fi

# One number for the whole pool. objset kstats count dataset I/O including ARC
# hits, which is what we want: a stream served from cache still means the pool
# is in use, and the next miss would wake the platters.
io=$(awk '/^(nread|nwritten)/ {s+=$3} END {printf "%d", s}' \
     "/proc/spl/kstat/zfs/$POOL"/objset-* 2>/dev/null)
prev=""; idle=0
[ -f "$STATE" ] && read -r prev idle < "$STATE"
# Unreadable counters, first run, or any movement at all resets the count.
# Failing towards "keep spinning" costs watts; the other way costs etcd.
if [ -z "$io" ] || [ -z "$prev" ] || [ "$io" != "$prev" ]; then idle=0; else idle=$((idle+1)); fi
echo "${io:-0} $idle" > "$STATE"

to_park=()
for entry in "${DISKS[@]}"; do
  d=${entry%% *}; sg=${entry##* }
  # Awake only on an explicit ACTIVE - standby, timeout or error means leave it.
  # Capture then match: `cmd | grep -q` under pipefail reports failure even when
  # the match succeeds, because grep exits first and the producer gets SIGPIPE.
  pm=$(smartctl -i -n standby "/dev/$sg" 2>&1 || true)
  case "$pm" in
    *"Power mode is:"*ACTIVE*) [ "$idle" -ge 2 ] && to_park+=("/dev/$sg:$d") ;;
  esac
done

# Parallel: sg_start blocks ~9s per disk; serially the first ones get woken
# again before the last is even asked.
for e in "${to_park[@]}"; do
  ( sg_start --pc=3 "${e%%:*}" >/dev/null 2>&1 \
    && logger -t sas-spindown "parked: ${e##*:}" ) &
done
wait
SH
chmod +x /root/scripts/sas-disks.sh /root/scripts/sas-spindown.sh

Two things about that counter. It comes from ZFS, so on another filesystem you need a different source for “did this pool move any bytes” — summing /proc/diskstats across the pool’s members works and is close enough. And it deliberately counts cache hits as activity: a film playing entirely out of ARC still means someone is watching, and the next miss will wake the platters anyway. Erring towards awake costs a few watts. Erring the other way cost me a week of etcd.

Wire it to a oneshot service on a 5-minute timer.

Step 4 — Stop your monitoring from undoing it

Controller patrol read wakes every disk weekly:

storcli /c0 set patrolread=off

Only do this if your disks are in JBOD/HBA mode. On real RAID volumes, patrol read is what finds bad sectors before a rebuild needs them, and turning it off trades a genuine safety net for some watts. If you’re running RAID on that controller, leave it on and accept the weekly wake.

smartd is worse. Its -n standby skip is ATA-only. On SAS it checks anyway, wakes the drive, and sent me a FailedReadSmartSelfTestLog warning every 30 minutes all night. standby,999,q changes nothing. Drop the SAS disks from smartd.conf and check their health from a nightly job instead, skipping any disk that’s asleep.

If you have custom smartd rules — attribute thresholds, per-device options — edit the file by hand rather than letting the installer rewrite it. It keeps a backup, but it won’t merge your settings.

Step 5 — Watch it without breaking it

Don’t poll with smartctl while waiting. On an awake disk it’s an SG_IO round-trip that resets the idle count. Check every minute and they never sleep — a working setup looks broken.

journalctl -t sas-spindown -f
ipmitool dcmi power reading | grep Instantaneous

ipmitool sensor (what iDRAC shows) is averaged and lags minutes. dcmi responds at once but reads ~9W higher. Pick one and stay on it.

Troubleshooting

• Disks never sleep. Something is writing. Re-run Step 2.
• They sleep, then wake within a minute. You’re sending standby to /dev/sdX. Check Step 1.
• They wake every 30 minutes, with SMART warnings by mail. smartd is still watching them. Step 4.
• They wake once a week. Controller patrol read. Step 4.
• smartctl prints nothing and exits 2. The disk is asleep. That’s the skip working, not an error.
• A disk never comes back. Some models need an explicit start: sg_start --start /dev/sgN.

Troubleshooting, part two

Symptoms that look unrelated to disks, and are:

• A database or etcd reports slow fsyncs, but only sometimes. Check what else is on that controller. Correlate the slow windows against journalctl -t sas-spindown — if they line up with parks, that is your answer.
• Disks park and wake repeatedly while something is streaming. A per-disk rule on a striped pool. Step 3.
• Everything looks healthy and the alerts never fired. Check what the alert actually measures. Percentile-over-ten-minutes rules do not see spikes that recover in two.

Summary

• SAS drives park with sg_start --pc=3 on /dev/sgN, never /dev/sdX
• Decide for the pool, never per disk. On a striped pool a per-disk rule parks half the vdevs mid-read and thrashes them
• Check what shares the controller. A wake answers sense 2/04/01, megaraid_sas turns that into a blocking poll, and anything else on that HBA queues behind it
• A stateless script on a timer beats a daemon, because SG_IO wake-ups are invisible to /proc/diskstats
• smartd and controller patrol read will undo all of it if left alone
• Nothing sleeps until you find whatever is still writing

Where it landed: 86–93% of samples parked over a normal day, 132 W asleep against 174 W awake, and the only etcd noise left is ~30 slow fsyncs during the nightly backup window — which has to happen. Twenty-nine minutes of continuous playback now produces zero parks and zero slow fsyncs.

The measurement that mattered was not watts. It was counting fsyncs over one second per hour, day by day, on a service that has nothing to do with the disks I was trying to save power on.

Full runbook, including the site-specific parts: proxmox/r730xd/spindown-setup.md.