merox

Commands

The ones worth not having to look up twice. What you want to do on the left, the command that does it on the right — anything in <angle-brackets> is a placeholder.

Triage

First minute on a sick box

  • Uptime and load averagesuptime
  • Kernel errors and warningsdmesg -T --level=err,warn | tail -50
  • Failed systemd unitssystemctl --failed
  • Errors logged since bootjournalctl -p err -b
  • Pressure stall: is CPU, memory or I/O actually contendedgrep . /proc/pressure/*
  • Run queue, swapping, CPU splitvmstat 1 5
  • Disk latency and saturationiostat -xz 1 3
  • Network throughput per interfacesar -n DEV 1 5
  • TCP retransmits per secondsar -n TCP,ETCP 1 5
  • Load per CPU corempstat -P ALL 1 3
  • CPU and disk I/O per processpidstat -u -d 1 3
  • Top memory consumersps aux --sort=-%mem | head
  • Top CPU consumersps aux --sort=-%cpu | head
  • Processes stuck in D state (hung on I/O or NFS)ps -eo stat,pid,wchan:32,cmd | awk '$1 ~ /^D/'
  • Dump stacks of blocked tasks to the kernel logecho w > /proc/sysrq-trigger && dmesg -T | tail -100
  • OOM killer activityjournalctl -k -g 'killed process'
  • Machine check and hardware errorsjournalctl -k -g 'mce|machine check|hardware error'
  • Recent reboots and shutdownslast -x reboot shutdown | head
  • Load earlier today, from sysstat history (Debian path)sar -q -f /var/log/sysstat/sa<day-of-month>

Memory

  • Memory and swapfree -h
  • Available memory, dirty pages, kernel slabgrep -E 'MemAvailable|Dirty|Slab|SUnreclaim' /proc/meminfo
  • Kernel caches by size (memory gone but no process owns it)slabtop -o -s c | head -20
  • Processes using the most swapgrep VmSwap /proc/[0-9]*/status | sort -k2 -n | tail

Limits that fail quietly

  • System-wide file handles: used, free, maxcat /proc/sys/fs/file-nr
  • Open files of a process against its limitls /proc/<pid>/fd | wc -l && grep 'open files' /proc/<pid>/limits
  • Conntrack table full (drops new connections)cat /proc/sys/net/netfilter/nf_conntrack_count /proc/sys/net/netfilter/nf_conntrack_max
  • Threads in use against the PID limitps -eLf | wc -l && cat /proc/sys/kernel/pid_max
  • Clock offset (breaks TLS, Kerberos, etcd)chronyc tracking
  • Time, timezone and NTP sync statetimedatectl

Out of space

  • Usage per real filesystemdf -hT -x tmpfs -x devtmpfs -x overlay
  • Inodes exhausted (df shows space, writes still fail)df -i
  • Largest directories on one filesystemdu -xh --max-depth=1 / | sort -h | tail -20
  • Interactive disk usage browserncdu -x /
  • Files over 1 GBfind / -xdev -type f -size +1G -exec ls -lh {} +
  • Deleted files still held openlsof +L1
  • Free a deleted-but-open file without restarting its process: > /proc/<pid>/fd/<fd>
  • Empty a live log without breaking the writertruncate -s 0 <file>
  • df and du disagree: look under the mountpointsmkdir -p /mnt/rootfs && mount --bind / /mnt/rootfs && du -xh --max-depth=1 /mnt/rootfs | sort -h
  • ext4: shrink the 5% root reserve on a data disktune2fs -m 1 /dev/<device>
  • Shrink the systemd journaljournalctl --vacuum-size=500M

Windows: first look

  • Last boot time(Get-CimInstance Win32_OperatingSystem).LastBootUpTime
  • Why it rebooted: 41 unexpected, 1074 initiated, 6008 dirty shutdownGet-WinEvent -FilterHashtable @{LogName='System'; Id=41,1074,6008} -MaxEvents 10 | Format-List TimeCreated, Id, Message
  • Blue screens: bugchecks on recordGet-WinEvent -FilterHashtable @{LogName='System'; ProviderName='Microsoft-Windows-WER-SystemErrorReporting'; Id=1001} -MaxEvents 5 | Format-List TimeCreated, Message
  • Critical and error events, last 24 hoursGet-WinEvent -FilterHashtable @{LogName='System','Application'; Level=1,2; StartTime=(Get-Date).AddDays(-1)} | Select-Object TimeCreated, ProviderName, Id, Message
  • CPU, free memory and disk latency, live (English counter names)Get-Counter '\Processor(_Total)\% Processor Time', '\Memory\Available MBytes', '\LogicalDisk(_Total)\Avg. Disk sec/Transfer' -SampleInterval 2 -MaxSamples 5
  • Processes with the most CPU timeGet-Process | Sort-Object CPU -Descending | Select-Object -First 10 Name, Id, CPU
  • Processes with the most memoryGet-Process | Sort-Object WorkingSet64 -Descending | Select-Object -First 10 Name, Id, @{n='MB'; e={[int]($_.WorkingSet64 / 1MB)}}
  • Automatic services that are not runningGet-CimInstance Win32_Service -Filter "StartMode='Auto' AND State<>'Running'" | Select-Object Name, DisplayName, ExitCode
  • Free space per volumeGet-Volume | Where-Object DriveLetter | Select-Object DriveLetter, FileSystemLabel, @{n='FreeGB'; e={[int]($_.SizeRemaining / 1GB)}}, @{n='SizeGB'; e={[int]($_.Size / 1GB)}}
  • Reboot pending (servicing or Windows Update)'Component Based Servicing\RebootPending', 'WindowsUpdate\Auto Update\RebootRequired' | ForEach-Object { Test-Path "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\$_" }

System

systemd

  • Status and last log linessystemctl status <unit>
  • Start now and at bootsystemctl enable --now <unit>
  • Reload after editing unit filessystemctl daemon-reload
  • Override a unit without touching the vendor filesystemctl edit <unit>
  • Unit file as systemd sees it, drop-ins includedsystemctl cat <unit>
  • Effective limits and restart policysystemctl show <unit> -p Restart,MemoryMax,LimitNOFILE,TasksMax
  • After “start request repeated too quickly”systemctl reset-failed <unit> && systemctl start <unit>
  • Keep a unit from starting, even as a dependencysystemctl mask <unit>
  • Check a unit file before loading itsystemd-analyze verify <file>.service
  • Run one command with a memory and CPU capsystemd-run --scope -p MemoryMax=2G -p CPUQuota=50% <command>
  • CPU, memory and I/O per cgroup, livesystemd-cgtop
  • Timers with last and next runsystemctl list-timers --all
  • Every user’s crontab (jobs outside systemd timers)for u in $(cut -d: -f1 /etc/passwd); do crontab -l -u "$u" 2>/dev/null | sed "s/^/$u: /"; done
  • What slowed down bootsystemd-analyze blame

Logs

  • Follow one unitjournalctl -u <unit> -f
  • Last hour, all unitsjournalctl --since "1 hour ago"
  • Exact time windowjournalctl --since "2026-01-01 10:00" --until "2026-01-01 10:30"
  • Previous boot (what happened before the crash)journalctl -b -1 -e
  • Boots on record, to pick one for -bjournalctl --list-boots
  • Kernel messages onlyjournalctl -k
  • Everything sudo ranjournalctl _COMM=sudo
  • Keep logs across rebootsmkdir -p /var/log/journal && systemctl restart systemd-journald
  • Follow a plain log file across rotationtail -F <file>

Processes

  • Find a process by namepgrep -a <name>
  • Process treeps -ef --forest
  • Files a process has openlsof -p <pid>
  • Who is using a portss -tlnp 'sport = :<port>'
  • Environment of a running processtr '\0' '\n' < /proc/<pid>/environ
  • Binary and working directory of a processls -l /proc/<pid>/exe /proc/<pid>/cwd
  • Kernel stack of a hung processcat /proc/<pid>/stack
  • Trace system callsstrace -f -p <pid>
  • Only file and network syscalls, timestampedstrace -f -tt -e trace=%file,%network -p <pid>
  • Hottest functions, liveperf top -p <pid>
  • De-prioritise CPU and disk for a noisy jobrenice -n 19 -p <pid> && ionice -c3 -p <pid>
  • Watch a command, highlighting what changedwatch -n1 -d '<command>'
  • Session that survives logouttmux new -s <name>
  • Reattach to ittmux attach -t <name>

Files and text

  • Files changed in the last dayfind <dir> -type f -mtime -1
  • Delete files older than 30 days (run with -print first)find <dir> -type f -mtime +30 -delete
  • Top talkers in a log: count and rank one columnawk '{print $1}' <log> | sort | uniq -c | sort -rn | head
  • “Operation not permitted” as root: immutable bitlsattr <file> && chattr -i <file>
  • Compare one file across two hostsdiff <(ssh <host-a> cat <file>) <(ssh <host-b> cat <file>)
  • Run a command on many hosts in parallelxargs -P 10 -I{} ssh -o BatchMode=yes {} '<command>' < <hosts-file>
  • Pack a directorytar -czf <archive>.tar.gz <dir>
  • Unpack into a directorytar -xzf <archive>.tar.gz -C <dir>
  • List an archive without unpackingtar -tzf <archive>.tar.gz | head

Users and security

  • Change owner recursivelychown -R <user>:<group> <dir>
  • Directories 755, files 644find <dir> -type d -exec chmod 755 {} + && find <dir> -type f -exec chmod 644 {} +
  • Add a user to a group (next login)usermod -aG <group> <user>
  • Show ACLsgetfacl <path>
  • Grant one user access via ACLsetfacl -m u:<user>:rwx <path>
  • Default ACL so new files inherit group accesssetfacl -R -d -m g:<group>:rwX <dir>
  • Password and account expiry (service account locked out)chage -l <user>
  • SELinux denials, recentausearch -m avc -ts recent
  • Reset SELinux labels after moving filesrestorecon -Rv <path>
  • Allow a service on a non-standard port under SELinuxsemanage port -a -t http_port_t -p tcp <port>
  • Failed SSH logins todayjournalctl -u ssh -u sshd -g 'Failed password|Invalid user' --since today

Packages

  • Debian: installed and candidate version, and which repoapt-cache policy <package>
  • Debian: which package owns a filedpkg -S <path>
  • Debian: finish an interrupted upgradedpkg --configure -a && apt -f install
  • Debian: pin a package versionapt-mark hold <package>
  • Debian: reboot required, and by whatcat /var/run/reboot-required.pkgs
  • Debian: services still running old librariesneedrestart -r l
  • RHEL: which package owns a filerpm -qf <path>
  • RHEL: which package provides a binarydnf provides '*/<binary>'
  • RHEL: undo a transactiondnf history && dnf history undo <id>
  • RHEL: reboot requireddnf needs-restarting -r

Boot and rescue

  • Root is read-only in emergency modemount -o remount,rw /
  • Chroot into a broken install from a live ISOmount /dev/<root> /mnt && for d in dev proc sys run; do mount --rbind /$d /mnt/$d; done && chroot /mnt
  • Kernel command line the system booted withcat /proc/cmdline
  • UEFI boot entries and orderefibootmgr -v
  • Debian: rebuild every initramfsupdate-initramfs -u -k all
  • RHEL: rebuild every initramfsdracut -f --regenerate-all
  • Debian: reinstall GRUB on a BIOS diskgrub-install /dev/<disk> && update-grub

Hardware

  • Hardware event log (PSU, DIMM, fan faults)ipmitool sel elist | tail -20
  • Sensors: temperatures, fans, voltagesipmitool sdr elist
  • BMC network settings from the hostipmitool lan print 1
  • BMC unresponsive: cold-reset it from the hostipmitool mc reset cold
  • Serial number / Dell service tagdmidecode -s system-serial-number
  • DIMMs per slot with size and speeddmidecode -t memory | grep -E 'Locator|Size|Speed'
  • CPU model, sockets, cores, flagslscpu
  • ECC errors per memory controllergrep . /sys/devices/system/edac/mc/mc*/[cu]e_count
  • PCIe link trained below its capabilitylspci -vv -s <bus:dev.fn> | grep -E 'LnkCap|LnkSta'
  • NUMA layout (pin VMs and IRQs to one node)numactl --hardware
  • CPU temperatures and fanssensors

NixOS: rebuild

  • Build and switchsudo nixos-rebuild switch --flake .#<host>
  • Activate without a boot entrysudo nixos-rebuild test --flake .#<host>
  • Apply on next boot only (kernel, init changes)sudo nixos-rebuild boot --flake .#<host>
  • Build only, change nothingnixos-rebuild build --flake .#<host>
  • Deploy to a remote machinenixos-rebuild switch --flake .#<host> --target-host <user>@<host> --sudo
  • Back to the previous generationsudo nixos-rebuild switch --rollback
  • What changed since the last bootnix store diff-closures /run/booted-system /run/current-system

NixOS: flakes and store

  • Update all inputsnix flake update
  • Update one inputnix flake update <input>
  • Evaluate every output before deployingnix flake check
  • Evaluate an optionnix eval .#nixosConfigurations.<host>.config.<option>
  • Explore the config in a REPLnix repl --expr 'builtins.getFlake (toString ./.)'
  • System generationsnix-env --list-generations --profile /nix/var/nix/profiles/system
  • Closure size of the running systemnix path-info -Sh /run/current-system
  • GC frees nothing: find what still roots itnix-store --gc --print-roots | grep -v '^/proc'
  • Delete old generations and garbagesudo nix-collect-garbage --delete-older-than 14d
  • Deduplicate the storenix store optimise
  • Store corruption: verify and repairsudo nix-store --verify --check-contents --repair
  • Try a package without installingnix shell nixpkgs#<package>

Windows: repair and updates

  • Repair the component store first (needs Windows Update or a source)DISM /Online /Cleanup-Image /RestoreHealth
  • Then repair system filessfc /scannow
  • What sfc could not repairSelect-String -Path C:\Windows\Logs\CBS\CBS.log -Pattern '\[SR\] Cannot repair'
  • Shrink WinSxS by removing superseded componentsDISM /Online /Cleanup-Image /StartComponentCleanup
  • Recently installed updatesGet-HotFix | Sort-Object InstalledOn -Descending | Select-Object -First 10 HotFixID, Description, InstalledOn
  • Find an update’s package name to remove itDISM /Online /Get-Packages /Format:Table | findstr <kb-number>
  • Remove itDISM /Online /Remove-Package /PackageName:<package-name>
  • Readable Windows Update log (written to the desktop)Get-WindowsUpdateLog
  • Updates fail behind a proxy: the WinHTTP proxy settingnetsh winhttp show proxy
  • Online scan of a volume, no downtimeRepair-Volume -DriveLetter <letter> -Scan

Windows: services and tasks

  • Service stuck in Stopping: its PID (sc.exe, since sc is an alias)sc.exe queryex <service>
  • Then kill ittaskkill /PID <pid> /F
  • Why a service will not start: Service Control Manager eventsGet-WinEvent -FilterHashtable @{LogName='System'; ProviderName='Service Control Manager'} -MaxEvents 20 | Select-Object TimeCreated, Id, Message
  • Scheduled tasks whose last run failed (ignoring running and never-run)Get-ScheduledTask | Where-Object State -ne 'Disabled' | Get-ScheduledTaskInfo | Where-Object LastTaskResult -notin 0, 267009, 267011 | Select-Object TaskName, LastRunTime, LastTaskResult
  • Which process holds a file (Sysinternals Handle)handle.exe -accepteula <name>

Windows: Active Directory

  • Replication summary across all DCsrepadmin /replsummary
  • Replication partners and last result for one DCrepadmin /showrepl <dc>
  • DC health across the forest, errors onlydcdiag /e /q
  • Which DC this machine authenticates againstnltest /dsgetdc:<domain>
  • FSMO role holdersnetdom query fsmo
  • “Trust relationship failed”: repair the secure channel (Windows PowerShell 5.1, local admin)Test-ComputerSecureChannel -Repair -Credential <domain>\<admin>
  • Locked-out accountsSearch-ADAccount -LockedOut | Select-Object Name, SamAccountName, LastLogonDate
  • Which machine keeps locking an account (event 4740 on the PDC)Get-WinEvent -ComputerName (Get-ADDomain).PDCEmulator -FilterHashtable @{LogName='Security'; Id=4740} -MaxEvents 20 | Select-Object TimeCreated, @{n='User'; e={$_.Properties[0].Value}}, @{n='Caller'; e={$_.Properties[1].Value}}
  • Unlock the accountUnlock-ADAccount -Identity <user>
  • Applied Group Policy, as a reportgpresult /h C:\Temp\gpresult.html /f
  • Reapply Group Policy nowgpupdate /force
  • Kerberos tickets: list, then purge after a group changeklist; klist purge

Storage

Disks

  • Block devices with filesystems and UUIDslsblk -f
  • Model, serial, size, rotational, transportlsblk -o NAME,SIZE,MODEL,SERIAL,ROTA,TRAN
  • Logical and physical sector size (pick ashift)lsblk -o NAME,LOG-SEC,PHY-SEC
  • Stable device namesls -l /dev/disk/by-id/
  • Partition tableparted /dev/<disk> print
  • Find a newly hot-added disk (SCSI host rescan)echo "- - -" | tee /sys/class/scsi_host/host*/scan
  • See a disk’s new size after growing it (device rescan)echo 1 > /sys/class/block/<disk>/device/rescan
  • Same for NVMe namespacesnvme ns-rescan /dev/<nvme-controller>
  • Detach a disk cleanly before pulling itecho 1 > /sys/block/<disk>/device/delete
  • Re-read the partition table without rebootingpartprobe /dev/<disk>
  • Grow a partition into the new spacegrowpart /dev/<disk> <partition-number>
  • Find a physical drive: hold its activity LED ondd if=/dev/<disk> of=/dev/null bs=1M status=progress
  • Wipe filesystem, RAID and LVM signatures (destroys data)wipefs -a /dev/<disk>
  • Also clear GPT, including the backup at the endsgdisk --zap-all /dev/<disk>
  • “Device busy” on a wipe: leftover device-mapper entriesdmsetup ls && dmsetup remove <name>

Filesystems and mounts

  • Grow ext4 onlineresize2fs /dev/<device>
  • Grow XFS online (takes the mountpoint, not the device)xfs_growfs <mountpoint>
  • Force a full ext4 check (unmounted)e2fsck -f /dev/<device>
  • Repair XFS (fsck.xfs does nothing)xfs_repair /dev/<device>
  • Mount treefindmnt
  • fstab entry that will not hang boot if the disk is missingUUID=<uuid> /mnt/<name> ext4 defaults,noatime,nofail,x-systemd.device-timeout=10s 0 2
  • Test fstab without rebootingsystemctl daemon-reload && findmnt --verify && mount -a
  • Who keeps a mount busyfuser -vm <mountpoint>
  • Detach a busy mountumount -l <mountpoint>
  • Why a filesystem went read-onlydmesg -T | grep -iE 'remount|i/o error|ext4-fs error|xfs'
  • Trim all mounted SSDsfstrim -av

Disk health and performance

  • SMART verdict and attributessmartctl -H -A /dev/<disk>
  • SMART for a disk behind a PERC / MegaRAIDsmartctl -a -d megaraid,<n> /dev/sda
  • Start a long self-testsmartctl -t long /dev/<disk>
  • Self-test resultssmartctl -l selftest /dev/<disk>
  • NVMe health and wearnvme smart-log /dev/<nvme>
  • Spin down a SATA disk nowhdparm -y /dev/<disk>
  • Spin down a SAS disk nowsdparm --command=stop /dev/<disk>
  • Live I/O per processiotop -o
  • I/O latency of a directory, like pingioping -c 10 <dir>
  • Random 4k read/write benchmarkfio --name=rand --filename=<file> --size=1G --rw=randrw --bs=4k --direct=1 --runtime=30 --time_based --group_reporting

NFS and SMB

  • Server: apply /etc/exports changesexportfs -ra
  • Exports a server offersshowmount -e <server>
  • Mount NFSmount -t nfs4 <server>:/<export> <mountpoint>
  • Mount options actually negotiatednfsstat -m
  • Stale NFS handle: drop the mountumount -f -l <mountpoint>
  • List SMB sharessmbclient -L //<server> -U <user>
  • Mount SMB with a credentials filemount -t cifs //<server>/<share> <mountpoint> -o credentials=/root/.smbcred,uid=<uid>,gid=<gid>,vers=3.0

LVM

  • Physical volumes, groups, logical volumespvs && vgs && lvs
  • Which disks back each LVlvs -a -o +devices
  • Thin pool data and metadata filllvs -o lv_name,data_percent,metadata_percent <vg>
  • Activate a VG after moving disksvgchange -ay <vg>
  • Metadata backups (undo a bad change)vgcfgrestore --list <vg>
  • Initialise a physical volumepvcreate /dev/<partition>
  • Create a volume groupvgcreate <vg> /dev/<partition>
  • Fixed-size logical volumelvcreate -n <lv> -L 50G <vg>
  • Logical volume from all free spacelvcreate -n <lv> -l 100%FREE <vg>
  • Thin poollvcreate --type thin-pool -n <pool> -L 100G <vg>

LVM: grow and shrink

  • Add a disk to a VGvgextend <vg> /dev/<partition>
  • PV after its disk was enlarged (rescan the disk first)pvresize /dev/<partition>
  • Grow LV and filesystem togetherlvextend -r -L +10G /dev/<vg>/<lv>
  • Grow into all free spacelvextend -r -l +100%FREE /dev/<vg>/<lv>
  • Shrink LV and ext4 (unmounted; XFS cannot shrink)lvreduce -r -L 20G /dev/<vg>/<lv>
  • Grow a thin poollvextend -L +50G <vg>/<pool>
  • Thin pool metadata filling up (full means corruption)lvextend --poolmetadatasize +1G <vg>/<pool>
  • Evacuate a diskpvmove /dev/<partition>
  • Then drop it from the VGvgreduce <vg> /dev/<partition>

LVM: snapshots and rescue

  • Snapshot an LVlvcreate -s -n <snap> -L 5G /dev/<vg>/<lv>
  • Roll back to the snapshotlvconvert --merge /dev/<vg>/<snap>
  • Drop the snapshotlvremove /dev/<vg>/<snap>
  • Two VGs with the same name: rename one by UUIDvgs -o vg_name,vg_uuid && vgrename <vg-uuid> <new-name>
  • Map partitions inside an LV (VM disk) to mount themkpartx -av /dev/<vg>/<lv>

ZFS: pools

  • Only pools with problemszpool status -x
  • Files hit by permanent errorszpool status -v <pool>
  • Capacity per vdevzpool list -v
  • Mirror by stable disk IDszpool create -o ashift=12 <pool> mirror /dev/disk/by-id/<disk-a> /dev/disk/by-id/<disk-b>
  • ashift of an existing poolzdb -C <pool> | grep ashift
  • Replace a failed diskzpool replace <pool> <old-disk> /dev/disk/by-id/<new-disk>
  • Use the space after replacing with bigger diskszpool set autoexpand=on <pool> && zpool online -e <pool> <disk>
  • Start a scrubzpool scrub <pool>
  • Reset error counters after fixing the causezpool clear <pool>
  • Latency per vdev, livezpool iostat -vl 2
  • Who ran what against a poolzpool history <pool> | tail -20
  • Old ZFS label blocking reuse of a diskzpool labelclear -f /dev/<disk>
  • Find pools to importzpool import
  • Import by stable IDszpool import -d /dev/disk/by-id <pool>
  • Import read-only for rescuezpool import -o readonly=on -R /mnt <pool>

ZFS: datasets and snapshots

  • Usage per datasetzfs list -o name,used,avail,refer,mountpoint
  • Where the space went: data, snapshots, childrenzfs list -o space -r <pool>
  • Dataset with zstd compressionzfs create -o compression=zstd <pool>/<dataset>
  • Compression ratio achievedzfs get compressratio <pool>/<dataset>
  • Only properties set by handzfs get -s local all <pool>/<dataset>
  • Cap a datasetzfs set quota=100G <pool>/<dataset>
  • Snapshot recursivelyzfs snapshot -r <pool>/<dataset>@<name>
  • Snapshots, oldest firstzfs list -t snapshot -o name,used,creation -s creation
  • Preview destroying a range of snapshotszfs destroy -nv <pool>/<dataset>@<first>%<last>
  • Roll back (destroys later snapshots)zfs rollback -r <pool>/<dataset>@<name>
  • Full send to another hostzfs send <pool>/<dataset>@<name> | ssh <host> zfs receive <pool>/<dataset>
  • Incremental send, one stepzfs send -i @<old> <pool>/<dataset>@<new> | ssh <host> zfs receive <pool>/<dataset>
  • Incremental send with every snapshot in betweenzfs send -I @<old> <pool>/<dataset>@<new> | ssh <host> zfs receive <pool>/<dataset>

ZFS: ARC

  • ARC size and hit ratearc_summary | head -40
  • Live ARC statsarcstat 1
  • Cap ARC at 8 GB nowecho $((8 * 1024**3)) > /sys/module/zfs/parameters/zfs_arc_max
  • Cap ARC at 8 GB across rebootsecho "options zfs zfs_arc_max=8589934592" > /etc/modprobe.d/zfs.conf && update-initramfs -u -k all

Windows: files and storage

  • Mirror a folder with permissions, logged (/MIR deletes extras; add /L to preview)robocopy <src> <dst> /MIR /COPY:DATS /DCOPY:DAT /MT:16 /R:1 /W:1 /LOG:C:\Temp\robocopy.log
  • Largest files on a driveGet-ChildItem <drive>:\ -Recurse -File -ErrorAction SilentlyContinue | Sort-Object Length -Descending | Select-Object -First 20 FullName, @{n='MB'; e={[int]($_.Length / 1MB)}}
  • NTFS permissionsicacls <path>
  • Access denied even as admin: take ownership, reset ACLs to inheritedtakeown /F <path> /R /D Y; icacls <path> /reset /T /C
  • New disk shows Offline (SAN policy): bring it online and writableSet-Disk -Number <n> -IsOffline $false; Set-Disk -Number <n> -IsReadOnly $false
  • Grew a VM disk: rescan, then extend the partition to the endUpdate-HostStorageCache; Resize-Partition -DriveLetter <letter> -Size (Get-PartitionSupportedSize -DriveLetter <letter>).SizeMax
  • Backup fails: VSS writers in a failed statevssadmin list writers
  • Shadow copy storage per volumevssadmin list shadowstorage
  • File server: who has a file openGet-SmbOpenFile | Where-Object Path -like '*<name>*' | Select-Object ClientComputerName, ClientUserName, Path, FileId
  • Close that handleClose-SmbOpenFile -FileId <file-id> -Force
  • Share permissions (NTFS permissions apply on top)Get-SmbShareAccess -Name <share>

Network

Interfaces and routes

  • Addresses, one line per interfaceip -br a
  • Which route and source IP a destination takesip route get <ip>
  • Policy routing (Tailscale, VPNs, multi-WAN add rules)ip rule && ip route show table all
  • ARP / neighbour tableip neigh
  • Add an address until rebootip addr add <ip>/<prefix> dev <iface>
  • RX/TX errors and drops on an interfaceip -s link show <iface>
  • Link speed and driverethtool <iface> && ethtool -i <iface>
  • NIC counters: CRC errors point at cable or opticethtool -S <iface> | grep -iE 'err|drop|crc'
  • Bond state and active slavecat /proc/net/bonding/<bond>
  • Bridge membersbridge link
  • VLANs on a VLAN-aware bridgebridge vlan show
  • Someone else has this IParping -D -I <iface> -c 3 <ip>
  • Reverse-path filter dropping asymmetric trafficsysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.<iface>.rp_filter
  • Apply /etc/network/interfaces (Proxmox, ifupdown2)ifreload -a
  • NetworkManager: static IPv4 on a connectionnmcli con mod <connection> ipv4.method manual ipv4.addresses <ip>/<prefix> ipv4.gateway <gateway> && nmcli con up <connection>

Ports and connections

  • Listening sockets with processesss -tulpn
  • Socket totalsss -s
  • Established connections on a portss -tnp state established '( sport = :<port> or dport = :<port> )'
  • RTT, retransmits and window per connectionss -ti dst <ip>
  • Is a remote port opennc -zv <host> <port>
  • Talk to one backend, bypassing DNS and load balancercurl -v --resolve <host>:443:<ip> https://<host>/
  • HTTP status and timingscurl -o /dev/null -s -w '%{http_code} dns=%{time_namelookup}s connect=%{time_connect}s tls=%{time_appconnect}s total=%{time_total}s\n' <url>
  • Certificate a server presents: subject, SANs, datesopenssl s_client -connect <host>:443 -servername <host> </dev/null 2>/dev/null | openssl x509 -noout -subject -ext subjectAltName -dates
  • Expiry of a certificate fileopenssl x509 -in <cert>.pem -noout -enddate

DNS

  • Just the answerdig +short <name>
  • Ask a specific resolverdig @<server> <name> A
  • What the system resolver returns (/etc/hosts, nsswitch)getent hosts <name>
  • Follow delegation from the rootdig +trace <name>
  • Reverse lookupdig -x <ip>
  • systemd-resolved state per linkresolvectl status
  • Flush the local cacheresolvectl flush-caches

Path and packets

  • Loss and latency per hopmtr -rwc 50 <host>
  • Largest packet that fits: 1472 for 1500 MTU, 1252 for 1280ping -M do -s 1472 <host>
  • Watch traffic on a porttcpdump -ni <iface> port <port>
  • Capture on every interface, bridges includedtcpdump -ni any host <ip>
  • Capture a host to a file for Wiresharktcpdump -ni <iface> -w <file>.pcap host <ip>
  • Throughput test: serveriperf3 -s
  • Throughput test: client, 4 streamsiperf3 -c <server> -P 4
  • TCP retransmits and drops, system-widenstat -az | grep -iE 'retrans|drop'
  • Bandwidth per connectioniftop -i <iface>

Firewall and NAT

  • nftables rulesetnft list ruleset
  • iptables with countersiptables -L -n -v --line-numbers
  • NAT rules (Docker and Kubernetes write here)iptables -t nat -L -n -v
  • Tracked connections to a hostconntrack -L -d <ip>
  • Drop tracked state after changing rulesconntrack -D -d <ip>
  • ufw statusufw status verbose
  • ufw: SSH from one subnetufw allow from <cidr> to any port 22 proto tcp
  • firewalld zone rulesfirewall-cmd --list-all
  • firewalld: open a portfirewall-cmd --permanent --add-port=<port>/tcp && firewall-cmd --reload
  • Enable IPv4 forwarding nowsysctl -w net.ipv4.ip_forward=1
  • IPv4 forwarding across rebootsecho 'net.ipv4.ip_forward=1' > /etc/sysctl.d/99-forward.conf && sysctl --system

Tailscale

  • Peers and connection typetailscale status
  • Direct or relayed through DERPtailscale ping <node>
  • NAT type and DERP latencytailscale netcheck
  • Advertise a subnettailscale set --advertise-routes=<cidr>
  • Accept subnets from other routerstailscale set --accept-routes
  • Route through an exit nodetailscale set --exit-node=<node>

SSH

  • New key pairssh-keygen -t ed25519 -C "<comment>"
  • Install a public key on a hostssh-copy-id -i ~/.ssh/id_ed25519.pub <user>@<host>
  • Key auth refused: fix permissionschmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys
  • Kill a frozen session (press Enter first)~.
  • Verbose handshakessh -vvv <user>@<host>
  • Old switch, iDRAC or iLO rejects the key algorithmssh -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa <host>
  • Forget a changed host keyssh-keygen -R <host>
  • Trust a host key non-interactivelyssh-keyscan -H <host> >> ~/.ssh/known_hosts
  • Fingerprint of a server keyssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
  • Validate sshd_config before restartingsshd -t
  • Effective sshd setting for a given usersshd -T -C user=<user> | grep -i <option>

SSH tunnels

  • Reach a remote-only port locallyssh -N -L 8080:localhost:<remote-port> <host>
  • Reach a web UI behind a bastionssh -N -L 8443:<target>:443 <bastion>
  • Expose a local port on the remotessh -N -R 9000:localhost:<local-port> <host>
  • SOCKS proxy through a hostssh -N -D 1080 <host>
  • Hop through a bastionssh -J <bastion> <host>

Copy and transfer

  • Mirror a directory, preserving everythingrsync -aHAX --info=progress2 --delete <src>/ <dst>/
  • Preview what rsync would changersync -aHAXn --delete --itemize-changes <src>/ <dst>/
  • Verify a copy by checksum: list files that differrsync -rcn --out-format='%n' <src>/ <dst>/
  • Push over SSH, compressedrsync -az --info=progress2 <src>/ <user>@<host>:<dst>/
  • Resumable, throttled copy over SSHrsync -aHAX --partial --bwlimit=50M --info=progress2 <src>/ <user>@<host>:<dst>/
  • Stream a directory over SSHtar -cf - <dir> | ssh <host> 'tar -xf - -C <dst>'
  • Serve the current directory over HTTPpython3 -m http.server 8000

Windows: network

  • Addresses, gateway and DNS per interfaceGet-NetIPConfiguration
  • Is a port reachableTest-NetConnection <host> -Port <port>
  • Trace the routeTest-NetConnection <host> -TraceRoute
  • Resolve against a specific DNS serverResolve-DnsName <name> -Server <server>
  • Flush the DNS client cacheClear-DnsClientCache
  • Listening ports with their processGet-NetTCPConnection -State Listen | Select-Object LocalAddress, LocalPort, @{n='Process'; e={(Get-Process -Id $_.OwningProcess).ProcessName}} | Sort-Object LocalPort
  • Network marked Public blocks sharing and remoting: make it PrivateSet-NetConnectionProfile -InterfaceAlias <interface> -NetworkCategory Private
  • Firewall state per profileGet-NetFirewallProfile | Select-Object Name, Enabled, DefaultInboundAction
  • Allow an inbound portNew-NetFirewallRule -DisplayName "<name>" -Direction Inbound -Protocol TCP -LocalPort <port> -Action Allow
  • Capture full packets with the built-in toolpktmon start --capture --pkt-size 0 --file-name <file>.etl
  • Stop, and convert the capture for Wiresharkpktmon stop; pktmon etl2pcap <file>.etl --out <file>.pcapng
  • Time source and last syncw32tm /query /status
  • Clock offset against a domain controllerw32tm /stripchart /computer:<dc> /samples:5 /dataonly
  • Force a time resyncw32tm /resync /force

Windows: remote management

  • Is WinRM answeringTest-WSMan <host>
  • Run a command on several serversInvoke-Command -ComputerName <host-a>, <host-b> -ScriptBlock { <command> }
  • Interactive remote sessionEnter-PSSession -ComputerName <host>
  • Who is logged on (RDP sessions)quser /server:<host>
  • Log off a stuck sessionlogoff <session-id> /server:<host>
  • Enable RDP and its firewall ruleSet-ItemProperty 'HKLM:\System\CurrentControlSet\Control\Terminal Server' -Name fDenyTSConnections -Value 0; Enable-NetFirewallRule -DisplayGroup 'Remote Desktop'
  • Installed software (not Win32_Product, which triggers MSI repairs)Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\*, HKLM:\Software\WOW6432Node\Microsoft\Windows\CurrentVersion\Uninstall\* | Where-Object DisplayName | Select-Object DisplayName, DisplayVersion | Sort-Object DisplayName
  • Installed roles and featuresGet-WindowsFeature | Where-Object Installed

Virtualization

Proxmox: VMs

  • List VMsqm list
  • Clean shutdownqm shutdown <vmid>
  • Hard stopqm stop <vmid>
  • Stop a VM that holds a lockqm stop <vmid> --skiplock
  • QEMU process ignoring stop: kill itkill -9 "$(cat /var/run/qemu-server/<vmid>.pid)"
  • Clear a stale lockqm unlock <vmid>
  • Show configqm config <vmid>
  • Full KVM command line a VM starts withqm showcmd <vmid> --pretty
  • Serial consoleqm terminal <vmid>
  • Grow a disk (then rescan it inside the guest)qm disk resize <vmid> scsi0 +10G
  • Move a disk to another storage, liveqm disk move <vmid> scsi0 <storage> --delete 1
  • Import a cloud image as a diskqm disk import <vmid> <image>.qcow2 <storage>
  • Import an OVF exportqm importovf <vmid> <file>.ovf <storage>
  • Pick up orphaned disk images on storagesqm disk rescan --vmid <vmid>
  • Snapshotqm snapshot <vmid> <name>
  • Roll back to a snapshotqm rollback <vmid> <name>
  • Full clone of a templateqm clone <template-id> <new-id> --name <name> --full
  • Live migrateqm migrate <vmid> <node> --online
  • IPs via guest agentqm guest cmd <vmid> network-get-interfaces
  • Mount a stopped VM’s partition from a ZFS zvolmount /dev/zvol/<pool>/vm-<vmid>-disk-<n>-part<p> /mnt

Proxmox: containers

  • List containerspct list
  • Shell insidepct enter <ctid>
  • Run one commandpct exec <ctid> -- <command>
  • Grow the root diskpct resize <ctid> rootfs +5G
  • Bind-mount a host directorypct set <ctid> -mp0 <host-path>,mp=<container-path>
  • Unprivileged bind mount: host UID = 100000 + container UIDchown -R 100000:100000 <host-path>
  • Copy a file inpct push <ctid> <file> <container-path>
  • Mount a container’s rootfs on the host (rescue)pct mount <ctid>
  • Check a container’s filesystempct fsck <ctid>
  • Clear a stale lockpct unlock <ctid>

Proxmox: host, storage and backup

  • Versions of every PVE packagepveversion -v
  • Core servicessystemctl status pveproxy pvedaemon pve-cluster
  • Web UI unreachable: restart its servicessystemctl restart pveproxy pvedaemon
  • Cluster quorumpvecm status
  • Lost quorum, /etc/pve read-only (last resort)pvecm expected 1
  • Regenerate node certificatespvecm updatecerts --force
  • ESPs in sync on a ZFS or UEFI rootproxmox-boot-tool status
  • Upgrade (dist-upgrade; plain apt upgrade can break PVE)apt update && apt dist-upgrade
  • Storage pools and usagepvesm status
  • What is on a storagepvesm list <storage>
  • Back up nowvzdump <vmid> --storage <storage> --mode snapshot --compress zstd
  • Restore a VMqmrestore <backup>.vma.zst <new-id> --storage <storage>
  • Restore a containerpct restore <new-id> <backup>.tar.zst --storage <storage>

KVM: passthrough and libvirt

  • CPU virtualization flags presentgrep -Ec '(vmx|svm)' /proc/cpuinfo
  • IOMMU enableddmesg | grep -e DMAR -e IOMMU
  • Devices per IOMMU groupfor d in /sys/kernel/iommu_groups/*/devices/*; do g=${d#*/iommu_groups/}; printf 'group %s\t' "${g%%/*}"; lspci -nns "${d##*/}"; done
  • Driver bound to a device (vfio-pci or not)lspci -nnk -s <bus:dev.fn>
  • Nested virtualization on (Intel)cat /sys/module/kvm_intel/parameters/nested
  • libvirt domainsvirsh list --all
  • libvirt: guest IPsvirsh domifaddr <domain>
  • Disks attached to a domainvirsh domblklist <domain>
  • Image format and sizeqemu-img info <disk>
  • Convert VMDK to qcow2qemu-img convert -p -f vmdk -O qcow2 <in>.vmdk <out>.qcow2

Hyper-V

  • VMs with state, uptime and memoryGet-VM | Select-Object Name, State, Uptime, CPUUsage, MemoryAssigned
  • Force a VM offStop-VM -Name <vm> -TurnOff
  • VM stuck Stopping: kill its worker processGet-CimInstance Win32_Process -Filter "Name='vmwp.exe'" | Where-Object CommandLine -match (Get-VM <vm>).VMId | ForEach-Object { Stop-Process -Id $_.ProcessId -Force }
  • Checkpoints of a VMGet-VMCheckpoint -VMName <vm>
  • Delete all checkpoints, merging them into the diskGet-VMCheckpoint -VMName <vm> | Remove-VMCheckpoint
  • Grow a VHDX (then extend the partition in the guest)Resize-VHD -Path <path>.vhdx -SizeBytes 200GB
  • Virtual switches and the adapters behind themGet-VMSwitch | Select-Object Name, SwitchType, NetAdapterInterfaceDescription

ESXi: host and logs

  • Version and buildvmware -vl
  • Enable and start SSH from the ESXi shellvim-cmd hostsvc/enable_ssh && vim-cmd hostsvc/start_ssh
  • Enter maintenance modeesxcli system maintenanceMode set --enable true
  • Leave maintenance modeesxcli system maintenanceMode set --enable false
  • Reboot (host must be in maintenance mode)esxcli system shutdown reboot --reason "<reason>"
  • Host client or vCenter lost the host: restart management agents/etc/init.d/hostd restart && /etc/init.d/vpxa restart
  • Restart every management service (heavier)services.sh restart
  • Vendor, model and serialesxcli hardware platform get
  • NTP servers and stateesxcli system ntp get
  • Back up the host configuration (prints a download URL)vim-cmd hostsvc/firmware/sync_config && vim-cmd hostsvc/firmware/backup_config
  • Support bundle for VMware / Broadcomvm-support
  • Storage, network and driver eventstail -f /var/log/vmkernel.log
  • Host agent: VM operations, API callstail -f /var/log/hostd.log
  • Hardware and connectivity observationstail -f /var/log/vobd.log
  • Why a VM failed to power ontail -100 /vmfs/volumes/<datastore>/<vm>/vmware.log
  • Send logs to a remote syslogesxcli system syslog config set --loghost='udp://<ip>:514' && esxcli system syslog reload && esxcli network firewall ruleset set -r syslog -e true

ESXi: VMs and snapshots

  • Registered VMs with their IDsvim-cmd vmsvc/getallvms
  • Power statevim-cmd vmsvc/power.getstate <vmid>
  • Power onvim-cmd vmsvc/power.on <vmid>
  • Guest shutdown (needs VMware Tools)vim-cmd vmsvc/power.shutdown <vmid>
  • Hard power offvim-cmd vmsvc/power.off <vmid>
  • VM will not power off: find its world IDesxcli vm process list
  • Then kill it: soft, then hard, then forceesxcli vm process kill --type=soft --world-id=<world-id>
  • Guest IP from VMware Toolsvim-cmd vmsvc/get.guest <vmid> | grep -m1 ipAddress
  • Register a VM from its .vmxvim-cmd solo/registervm /vmfs/volumes/<datastore>/<vm>/<vm>.vmx
  • Unregister (files stay on the datastore)vim-cmd vmsvc/unregister <vmid>
  • Reload a .vmx after editing it by handvim-cmd vmsvc/reload <vmid>
  • Stuck on a question (“moved or copied?”): list itvim-cmd vmsvc/message <vmid>
  • Answer itvim-cmd vmsvc/message <vmid> <message-id> <choice>
  • Snapshot treevim-cmd vmsvc/get.snapshotinfo <vmid>
  • Create a snapshot without memoryvim-cmd vmsvc/snapshot.create <vmid> <name> <description> 0 0
  • Delete all snapshots (consolidates into the base disk)vim-cmd vmsvc/snapshot.removeall <vmid>
  • Delta disks left behindfind /vmfs/volumes/<datastore>/<vm>/ -name '*-0000*.vmdk'

ESXi: storage and virtual disks

  • Datastores with capacity and free spaceesxcli storage filesystem list
  • See a new or grown LUN: rescan every adapteresxcli storage core adapter rescan --all
  • Refresh VMFS volumes after the rescanvmkfstools -V
  • Devices with their naa IDsesxcli storage core device list
  • Paths per device and their stateesxcli storage core path list
  • Multipathing policy per deviceesxcli storage nmp device list
  • SMART data for a local diskesxcli storage core device smart get -d <naa-id>
  • Mount an NFS v3 datastoreesxcli storage nfs add -H <server> -s /<export> -v <datastore-name>
  • Copied or replicated LUN not mounting: VMFS snapshot volumesesxcli storage vmfs snapshot list
  • Mount it keeping its signatureesxcli storage vmfs snapshot mount -l <volume-label>
  • Reclaim freed blocks on thin-provisioned storageesxcli storage vmfs unmap -l <datastore>
  • Grow a VMDK to a new total size (VM off)vmkfstools -X 100G /vmfs/volumes/<datastore>/<vm>/<vm>.vmdk
  • Clone a disk as thinvmkfstools -i <source>.vmdk -d thin <dest>.vmdk
  • Return zeroed blocks of a thin disk (VM off)vmkfstools -K <disk>.vmdk
  • Check a VMDK for consistencyvmkfstools -x check <disk>.vmdk
  • “File is locked”: which host holds the lock (MAC in the output)vmkfstools -D /vmfs/volumes/<datastore>/<vm>/<file>

ESXi: network

  • VMkernel interfaces with IPsesxcli network ip interface ipv4 get
  • Physical NICs: link, speed, driveresxcli network nic list
  • Driver and firmware of one NICesxcli network nic get -n <vmnic>
  • Standard vSwitches and uplinksesxcli network vswitch standard list
  • Port groups and VLAN IDsesxcli network vswitch standard portgroup list
  • Set the VLAN on a port groupesxcli network vswitch standard portgroup set -p "<portgroup>" -v <vlan-id>
  • Ping out a specific vmk, jumbo frames, no fragmentingvmkping -I <vmk> -d -s 8972 <ip>
  • Routes and neighbour tableesxcli network ip route ipv4 list && esxcli network ip neighbor list
  • Which uplink and port a VM uses: find its world IDesxcli network vm list
  • Then its ports, MAC and team uplinkesxcli network vm port list -w <world-id>
  • Capture on a vmk to a filepktcap-uw --vmk <vmk> -o /tmp/<file>.pcap
  • Capture on a physical uplinkpktcap-uw --uplink <vmnic> -o /tmp/<file>.pcap
  • Management IP lost: set vmk0 from the shellesxcli network ip interface ipv4 set -i vmk0 -t static -I <ip> -N <netmask> -g <gateway>
  • Firewall rulesetsesxcli network firewall ruleset list

ESXi: performance and patching

  • Live stats: c CPU, m memory, d/u/v disk, n networkesxtop
  • CPU contention: watch %RDY (>5% per vCPU) and %CSTPesxtop # then press c
  • Disk latency: DAVG is the array, KAVG the hostesxtop # then press u
  • Record 5 minutes for later analysisesxtop -b -d 5 -n 60 > /tmp/esxtop.csv
  • Installed VIBsesxcli software vib list
  • Image profiles inside an offline depotesxcli software sources profile list -d /vmfs/volumes/<datastore>/<depot>.zip
  • Preview an updateesxcli software profile update -d /vmfs/volumes/<datastore>/<depot>.zip -p <profile> --dry-run
  • Apply it (maintenance mode, then reboot)esxcli software profile update -d /vmfs/volumes/<datastore>/<depot>.zip -p <profile>
  • Install a driver or component VIBesxcli software vib install -d /vmfs/volumes/<datastore>/<bundle>.zip
  • Allow community-supported driversesxcli software acceptance set --level=CommunitySupported

ESXi: PowerCLI

  • Connect to vCenter or a hostConnect-VIServer -Server <server>
  • VMs with power state and sizeGet-VM | Select-Object Name, PowerState, NumCpu, MemoryGB | Sort-Object Name
  • Snapshots older than a weekGet-VM | Get-Snapshot | Where-Object { $_.Created -lt (Get-Date).AddDays(-7) } | Select-Object VM, Name, Created, SizeGB
  • VMs whose disks need consolidationGet-VM | Where-Object { $_.ExtensionData.Runtime.ConsolidationNeeded } | Select-Object Name
  • Start SSH on every hostGet-VMHost | Get-VMHostService | Where-Object Key -eq "TSM-SSH" | Start-VMHostService

ESXi to Proxmox

  • Add the ESXi host as an import source on Proxmoxpvesm add esxi <storage-id> --server <esxi-host> --username root --password '<password>' --skip-cert-verification 1
  • Before exporting: remove every snapshotvim-cmd vmsvc/snapshot.removeall <vmid>
  • Import a copied VMDK into a Proxmox VMqm disk import <vmid> <vm>.vmdk <storage> --format raw
  • Guest NIC renamed after the move (ens192 becomes ens18)ip -br link

Containers

Docker

  • Running, with status and portsdocker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'
  • Follow logsdocker logs -f --tail 100 <container>
  • Throwaway containerdocker run --rm -it <image> sh
  • Why it exiteddocker inspect -f '{{.State.ExitCode}} oom={{.State.OOMKilled}} {{.State.Error}}' <container>
  • Restart loops and other events, last 10 minutesdocker events --since 10m
  • Container IPsdocker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}' <container>
  • CPU and memory snapshotdocker stats --no-stream
  • Log file on disk (the one filling /var)docker inspect -f '{{.LogPath}}' <container>
  • Writable layer size per containerdocker ps -s
  • Files changed inside a containerdocker diff <container>
  • Copy a file outdocker cp <container>:<path> <dest>
  • Change restart policydocker update --restart unless-stopped <container>
  • Host PID of every containerdocker ps -q | xargs docker inspect -f '{{.State.Pid}} {{.Name}}'
  • Run host tools inside a container’s network namespacensenter -t "$(docker inspect -f '{{.State.Pid}}' <container>)" -n ss -tlnp

Docker Compose

  • Update images and recreatedocker compose pull && docker compose up -d
  • Rebuild and recreatedocker compose up -d --build
  • Recreate one service even if unchangeddocker compose up -d --force-recreate <service>
  • Follow one servicedocker compose logs -f <service>
  • Final config after env files and mergesdocker compose config
  • Every compose project on the hostdocker compose ls
  • Tear down, volumes too (destroys data)docker compose down -v

Docker: images and cleanup

  • Disk used by images, containers, volumes, build cachedocker system df -v
  • Remove everything unused except volumesdocker system prune -a
  • Build cache (often the biggest)docker builder prune
  • Unused images older than a weekdocker image prune -a --filter "until=168h"
  • Remove unused anonymous volumesdocker volume prune
  • Multi-arch build and pushdocker buildx build --platform linux/amd64,linux/arm64 -t <name>:<tag> --push .
  • Architectures and digest of a remote imagedocker buildx imagetools inspect <image>
  • Layers and their sizesdocker history <image>
  • Export an image for an offline hostdocker save <image> | gzip > <image>.tar.gz
  • Import itdocker load -i <image>.tar.gz

Docker: daemon, networks, volumes

  • Networks and their membersdocker network inspect <network>
  • Debug a container’s network with a full toolboxdocker run --rm -it --network container:<container> nicolaka/netshoot
  • Publish on localhost only (published ports bypass ufw)docker run -d -p 127.0.0.1:<host-port>:<container-port> <image>
  • Back up a named volumedocker run --rm -v <volume>:/data -v "$PWD":/backup alpine tar -czf /backup/<volume>.tar.gz -C /data .
  • Cap container logs (/etc/docker/daemon.json){ "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3" } }
  • Keep Docker networks off your LAN ranges (daemon.json){ "default-address-pools": [{ "base": "10.200.0.0/16", "size": 24 }] }

Kubernetes: inspect

  • Switch clusterkubectl config use-context <context>
  • Default namespace for this contextkubectl config set-context --current --namespace=<ns>
  • Field docs for any resourcekubectl explain <resource>.spec
  • Pods not running, cluster-widekubectl get pods -A --field-selector=status.phase!=Running
  • Why a pod is stuckkubectl describe pod <pod>
  • Events, newest lastkubectl get events -A --sort-by=.metadata.creationTimestamp
  • Warnings onlykubectl get events -A --field-selector type=Warning --sort-by=.metadata.creationTimestamp
  • Events for one objectkubectl get events -n <ns> --field-selector involvedObject.name=<name>
  • Pods by restart countkubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount'
  • Why the last container died (OOMKilled, exit code)kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].lastState}'
  • Node usagekubectl top nodes
  • Heaviest podskubectl top pods -A --sort-by=memory
  • Requests and limits against what a node haskubectl describe node <node> | grep -A 10 'Allocated resources'
  • Service with no endpoints: selector matches nothingkubectl get endpointslices -l kubernetes.io/service-name=<service>
  • Decode a secret keykubectl get secret <name> -o jsonpath='{.data.<key>}' | base64 -d
  • Decode every key of a secretkubectl get secret <name> -o json | jq '.data | map_values(@base64d)'
  • Every image runningkubectl get pods -A -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}' | sort -u
  • API server health, check by checkkubectl get --raw='/readyz?verbose'

Kubernetes: debug

  • Follow a container’s logskubectl logs -f <pod> -c <container>
  • Logs from before the crashkubectl logs --previous <pod>
  • Logs across a labelkubectl logs -l app=<app> --all-containers --prefix --since=1h
  • Shell insidekubectl exec -it <pod> -- sh
  • Debug container in a distroless podkubectl debug -it <pod> --image=busybox --target=<container>
  • Shell on a node, host filesystem at /hostkubectl debug node/<node> -it --image=busybox
  • Forward a service locallykubectl port-forward svc/<service> 8080:<port>
  • Cluster DNS works?kubectl run dns-test --rm -it --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default
  • Copy a file out of a podkubectl cp <ns>/<pod>:<path> <dest>
  • What a service account can dokubectl auth can-i --list --as=system:serviceaccount:<ns>:<sa>
  • Containers on a node without the APIcrictl ps -a

Kubernetes: workloads and nodes

  • Restart pods of a deploymentkubectl rollout restart deploy/<name>
  • Wait for a rolloutkubectl rollout status deploy/<name>
  • Undo the last rolloutkubectl rollout undo deploy/<name>
  • Scale to zerokubectl scale deploy/<name> --replicas=0
  • See what apply would changekubectl diff -f <file>.yaml
  • YAML skeleton without creating anythingkubectl create deploy <name> --image=<image> --dry-run=client -o yaml
  • Drain for maintenancekubectl drain <node> --ignore-daemonsets --delete-emptydir-data
  • Back into servicekubectl uncordon <node>
  • Pods on one nodekubectl get pods -A -o wide --field-selector spec.nodeName=<node>

Kubernetes: stuck resources

  • Namespace stuck Terminating: what is left in itkubectl api-resources --verbs=list --namespaced -o name | xargs -n1 kubectl get -n <ns> --ignore-not-found --show-kind
  • Resource held by finalizerskubectl patch <kind> <name> -n <ns> --type=merge -p '{"metadata":{"finalizers":null}}'
  • Pod stuck Terminating on a dead nodekubectl delete pod <pod> --grace-period=0 --force
  • Volume will not attach: stale attachmentskubectl get volumeattachment
  • Clear out evicted and failed podskubectl delete pods -A --field-selector=status.phase=Failed
  • kubeadm: certificate expirykubeadm certs check-expiration

Helm

  • Add and refresh a repohelm repo add <repo> <url> && helm repo update
  • Chart’s default valueshelm show values <repo>/<chart>
  • Install or upgradehelm upgrade --install <release> <repo>/<chart> -n <ns> --create-namespace -f values.yaml
  • Render manifests locallyhelm template <release> <repo>/<chart> -f values.yaml
  • Releases stuck pending or failedhelm list -A --pending --failed
  • Values a release was deployed withhelm get values <release> -n <ns>
  • Manifests a release actually appliedhelm get manifest <release> -n <ns>
  • Revisions of a releasehelm history <release> -n <ns>
  • Roll back (also unsticks pending-upgrade)helm rollback <release> <revision> -n <ns>

Flux

  • Controllers healthyflux check
  • Everything Flux managesflux get all -A
  • Only what is failingflux get all -A --status-selector ready=false
  • Pull git and apply nowflux reconcile kustomization flux-system --with-source
  • Reconcile one Helm releaseflux reconcile helmrelease <name> -n <ns> --with-source
  • Local changes against the cluster, before pushingflux diff kustomization <name> --path <path>
  • Pause GitOps for manual workflux suspend kustomization <name>
  • Resume itflux resume kustomization <name>
  • Controller errorsflux logs -A --level=error

Talos

  • Cluster health from one nodetalosctl -n <ip> health
  • Live node dashboardtalosctl -n <ip> dashboard
  • Services and their statetalosctl -n <ip> services
  • Kubelet logstalosctl -n <ip> logs kubelet
  • Kernel logtalosctl -n <ip> dmesg
  • Pod containers on a nodetalosctl -n <ip> containers -k
  • Disks Talos seestalosctl -n <ip> get disks
  • etcd memberstalosctl -n <ip> etcd members
  • etcd health and leadertalosctl -n <ip> etcd status
  • Snapshot etcdtalosctl -n <ip> etcd snapshot <file>.snapshot
  • Apply a changed machine configtalosctl -n <ip> apply-config -f <node>.yaml
  • Patch the machine configtalosctl -n <ip> patch mc --patch @<patch>.yaml
  • Upgrade Talos on a nodetalosctl -n <ip> upgrade --image <installer-image>:<version>
  • Preview a Kubernetes upgradetalosctl -n <ip> upgrade-k8s --to <version> --dry-run
  • Upgrade Kubernetestalosctl -n <ip> upgrade-k8s --to <version>
  • Fetch kubeconfigtalosctl -n <ip> kubeconfig