Spinning Down SAS Disks

I keep adding storage to my R730xd, but I don’t want the power bill to follow. Every extra disk draws power all day, read or not.

The pool is 12 SAS drives with media and backups. They’re used during the nightly backup window and when someone streams a film. The rest of the day they spin for nothing. Parking them when idle drops the server from 168W to 126W — about 1 kWh a day.

Making them stay parked took most of a day. They kept waking up with nothing in the logs to explain it.

What this covers

Getting SAS drives to spin down on Linux and stay down. SAS ignores ATA standby, so hdparm does nothing, and hd-idle stops working after the first wake-up. The fix is one detail plus a script on a timer.

What you’ll need

• Linux host with SAS drives behind an HBA or a controller in JBOD/IT mode
• sg3-utils and smartmontools
• Root access
• Optional: storcli (disables controller patrol read), a BMC (power readings)

Before you start

Parking a disk isn’t destructive — it wakes on the next read. Two things are still worth knowing:

• First access after a park is slow. Spin-up takes 10-20 seconds. Anything with a timeout shorter than that on those disks (a database, a VM image, an iSCSI target) will notice. Keep spin-down to data you’re happy to wait for.
• Some drives need an explicit start. Most wake on their own. A few models park into a state that needs sg_start --start /dev/sgN first. Test one disk and read it back before doing this to a whole array — if a drive doesn’t come back on its own, your array will notice before you do.

Is this worth doing for you?

smartctl -a /dev/sgN | grep -iE "start-stop|Specified cycle count"

Step 1 — Confirm the one detail on your hardware

The standby command has to go to the generic SCSI device, not the block device:

sg_start --pc=3 /dev/sdp    # wrong - kernel spins it straight back up
sg_start --pc=3 /dev/sg15   # right - stays down

Both return success. Both stop the platter. But closing a block device makes the kernel revalidate it, which spins the drive back up. That happens below the block layer, so /proc/diskstats shows nothing — no reads, no writes, no flushes. The disk looks like it woke up on its own.

All 12 of mine, same command: /dev/sdX → 154-170W, awake in seconds. /dev/sgN → 131-136W, stays down.

Check it yourself before going further:

sg=$(basename $(readlink -f /sys/block/sdp/device/generic))   # -> sg15
sg_start --pc=3 "/dev/$sg"
sleep 30
smartctl -i -n standby "/dev/$sg" | tail -1

STANDBY BY COMMAND means it held. Power mode is: ACTIVE means something woke it — which is what you get every time via /dev/sdX.

Step 2 — Make sure nothing is writing

Disks only sleep if nothing touches them. One chatty process is enough to keep a whole pool busy.

grep -E " sd[a-z]+ " /proc/diskstats | awk '{print $3, $4+$8}'; sleep 60; \
grep -E " sd[a-z]+ " /proc/diskstats | awk '{print $3, $4+$8}'   # must match

The counters must match. If they don’t, something is writing.

Mine were invisible to find: Garage’s LMDB heartbeat (moved to SSD) and Jellyfin’s real-time library monitor (turned off, scheduled scan instead). Media servers watching folders and object stores with metadata heartbeats are the usual suspects.

Step 3 — Install the enforcer

A script on a timer, not a daemon. Anything that remembers state gets it wrong — a disk woken by a web UI is invisible to diskstats, so a daemon stops issuing standby and the disk stays up. A script that re-checks the real power state every 5 minutes parks it again regardless.

Disk list, derived from the pool so a disk swap needs no edits:

cat > /root/scripts/sas-disks.sh <<'SH'
#!/bin/bash
# Prints "sdX sgN" per line for the pool's spinning members.
set -uo pipefail
POOL="${SAS_POOL:-media}"
zpool status "$POOL" 2>/dev/null | grep -oE "wwn-0x[0-9a-f]+" | sort -u | while read -r wwn; do
  d=$(basename "$(readlink -f "/dev/disk/by-id/$wwn" 2>/dev/null)" 2>/dev/null)
  { [ -z "$d" ] || [ ! -e "/sys/block/$d" ]; } && continue
  [ "$(cat "/sys/block/$d/queue/rotational" 2>/dev/null)" = "1" ] || continue
  sg=$(basename "$(readlink -f "/sys/block/$d/device/generic" 2>/dev/null)" 2>/dev/null)
  { [ -n "$sg" ] && [ -e "/dev/$sg" ]; } && echo "$d $sg"
done
SH

The enforcer. Two runs with no I/O means park it:

cat > /root/scripts/sas-spindown.sh <<'SH'
#!/bin/bash
set -uo pipefail
exec 9>/run/sas-spindown.lock
flock -n 9 || exit 0
STATE=/run/sas-spindown.state   # tmpfs - resets on reboot, by design
declare -A prev idle
if [ -f "$STATE" ]; then
  while read -r d io n; do prev[$d]=$io; idle[$d]=$n; done < "$STATE"
fi
if ! mapfile -t DISKS < <(/root/scripts/sas-disks.sh) || [ "${#DISKS[@]}" -eq 0 ]; then
  logger -t sas-spindown "ERROR: no disks returned"; exit 1
fi
: > "$STATE.new"
to_sleep=()
for entry in "${DISKS[@]}"; do
  d=${entry%% *}; sg=${entry##* }
  io=$(awk -v d="$d" '$3==d {print $4"+"$8}' /proc/diskstats)
  [ -z "$io" ] && continue
  # Awake only on an explicit ACTIVE - a timeout or error means leave it alone.
  if ! smartctl -i -n standby "/dev/$sg" 2>&1 | grep -qi "Power mode is:.*ACTIVE"; then
    echo "$d $io 0" >> "$STATE.new"; continue
  fi
  n=0
  if [ "${prev[$d]:-}" = "$io" ]; then
    n=$(( ${idle[$d]:-0} + 1 ))
    if [ "$n" -ge 2 ]; then to_sleep+=("/dev/$sg:$d"); n=0; fi
  fi
  echo "$d $io $n" >> "$STATE.new"
done
mv "$STATE.new" "$STATE"
# Parallel: sg_start blocks ~9s per disk; serially the first ones get woken
# again before the last is even asked.
for entry in "${to_sleep[@]}"; do
  ( sg_start --pc=3 "${entry%%:*}" >/dev/null 2>&1 \
    && logger -t sas-spindown "standby issued: ${entry##*:}" ) &
done
wait
SH
chmod +x /root/scripts/sas-disks.sh /root/scripts/sas-spindown.sh

Wire it to a oneshot service on a 5-minute timer.

Step 4 — Stop your monitoring from undoing it

Controller patrol read wakes every disk weekly:

storcli /c0 set patrolread=off

Only do this if your disks are in JBOD/HBA mode. On real RAID volumes, patrol read is what finds bad sectors before a rebuild needs them, and turning it off trades a genuine safety net for some watts. If you’re running RAID on that controller, leave it on and accept the weekly wake.

smartd is worse. Its -n standby skip is ATA-only. On SAS it checks anyway, wakes the drive, and sent me a FailedReadSmartSelfTestLog warning every 30 minutes all night. standby,999,q changes nothing. Drop the SAS disks from smartd.conf and check their health from a nightly job instead, skipping any disk that’s asleep.

If you have custom smartd rules — attribute thresholds, per-device options — edit the file by hand rather than letting the installer rewrite it. It keeps a backup, but it won’t merge your settings.

Step 5 — Watch it without breaking it

Don’t poll with smartctl while waiting. On an awake disk it’s an SG_IO round-trip that resets the idle count. Check every minute and they never sleep — a working setup looks broken.

journalctl -t sas-spindown -f
ipmitool dcmi power reading | grep Instantaneous

ipmitool sensor (what iDRAC shows) is averaged and lags minutes. dcmi responds at once but reads ~9W higher. Pick one and stay on it.

Troubleshooting

• Disks never sleep. Something is writing. Re-run Step 2.
• They sleep, then wake within a minute. You’re sending standby to /dev/sdX. Check Step 1.
• They wake every 30 minutes, with SMART warnings by mail. smartd is still watching them. Step 4.
• They wake once a week. Controller patrol read. Step 4.
• smartctl prints nothing and exits 2. The disk is asleep. That’s the skip working, not an error.
• A disk never comes back. Some models need an explicit start: sg_start --start /dev/sgN.

Summary

• SAS drives park with sg_start --pc=3 on /dev/sgN, never /dev/sdX
• A stateless script on a timer beats a daemon, because SG_IO wake-ups are invisible to /proc/diskstats
• smartd and controller patrol read will undo all of it if left alone
• Nothing sleeps until you find whatever is still writing

First full night here: parked in 77% of samples, 126W, backups normal.

Full runbook, including the site-specific parts: proxmox/r730xd/spindown-setup.md.