merox

Pagina este disponibilă doar în engleză.

Commands

The ones worth not having to look up twice. What you want to do on the left, the command that does it on the right — anything in <angle-brackets> is a placeholder.

Triage

First minute on a sick box

  • Uptime and load averagesuptime
  • Kernel errors and warningsdmesg -T --level=err,warn | tail -50
  • Failed systemd unitssystemctl --failed
  • Errors logged since bootjournalctl -p err -b
  • Pressure stall: is CPU, memory or I/O actually contendedgrep . /proc/pressure/*
  • Run queue, swapping, CPU splitvmstat 1 5
  • Disk latency and saturationiostat -xz 1 3
  • Network throughput per interfacesar -n DEV 1 5
  • TCP retransmits per secondsar -n TCP,ETCP 1 5
  • Load per CPU corempstat -P ALL 1 3
  • CPU and disk I/O per processpidstat -u -d 1 3
  • Top memory consumersps aux --sort=-%mem | head
  • Top CPU consumersps aux --sort=-%cpu | head
  • Processes stuck in D state (hung on I/O or NFS)ps -eo stat,pid,wchan:32,cmd | awk '$1 ~ /^D/'
  • Dump stacks of blocked tasks to the kernel logecho w > /proc/sysrq-trigger && dmesg -T | tail -100
  • OOM killer activityjournalctl -k -g 'killed process'
  • Machine check and hardware errorsjournalctl -k -g 'mce|machine check|hardware error'
  • Recent reboots and shutdownslast -x reboot shutdown | head
  • Load earlier today, from sysstat history (Debian path)sar -q -f /var/log/sysstat/sa<day-of-month>

Memory

  • Memory and swapfree -h
  • Available memory, dirty pages, kernel slabgrep -E 'MemAvailable|Dirty|Slab|SUnreclaim' /proc/meminfo
  • Kernel caches by size (memory gone but no process owns it)slabtop -o -s c | head -20
  • Processes using the most swapgrep VmSwap /proc/[0-9]*/status | sort -k2 -n | tail

Limits that fail quietly

  • System-wide file handles: used, free, maxcat /proc/sys/fs/file-nr
  • Open files of a process against its limitls /proc/<pid>/fd | wc -l && grep 'open files' /proc/<pid>/limits
  • Conntrack table full (drops new connections)cat /proc/sys/net/netfilter/nf_conntrack_count /proc/sys/net/netfilter/nf_conntrack_max
  • Threads in use against the PID limitps -eLf | wc -l && cat /proc/sys/kernel/pid_max
  • Clock offset (breaks TLS, Kerberos, etcd)chronyc tracking
  • Time, timezone and NTP sync statetimedatectl

Out of space

  • Usage per real filesystemdf -hT -x tmpfs -x devtmpfs -x overlay
  • Inodes exhausted (df shows space, writes still fail)df -i
  • Largest directories on one filesystemdu -xh --max-depth=1 / | sort -h | tail -20
  • Interactive disk usage browserncdu -x /
  • Files over 1 GBfind / -xdev -type f -size +1G -exec ls -lh {} +
  • Deleted files still held openlsof +L1
  • Free a deleted-but-open file without restarting its process: > /proc/<pid>/fd/<fd>
  • Empty a live log without breaking the writertruncate -s 0 <file>
  • df and du disagree: look under the mountpointsmkdir -p /mnt/rootfs && mount --bind / /mnt/rootfs && du -xh --max-depth=1 /mnt/rootfs | sort -h
  • ext4: shrink the 5% root reserve on a data disktune2fs -m 1 /dev/<device>
  • Shrink the systemd journaljournalctl --vacuum-size=500M

Windows: first look

  • Last boot time(Get-CimInstance Win32_OperatingSystem).LastBootUpTime
  • Why it rebooted: 41 unexpected, 1074 initiated, 6008 dirty shutdownGet-WinEvent -FilterHashtable @{LogName='System'; Id=41,1074,6008} -MaxEvents 10 | Format-List TimeCreated, Id, Message
  • Blue screens: bugchecks on recordGet-WinEvent -FilterHashtable @{LogName='System'; ProviderName='Microsoft-Windows-WER-SystemErrorReporting'; Id=1001} -MaxEvents 5 | Format-List TimeCreated, Message
  • Critical and error events, last 24 hoursGet-WinEvent -FilterHashtable @{LogName='System','Application'; Level=1,2; StartTime=(Get-Date).AddDays(-1)} | Select-Object TimeCreated, ProviderName, Id, Message
  • CPU, free memory and disk latency, live (English counter names)Get-Counter '\Processor(_Total)\% Processor Time', '\Memory\Available MBytes', '\LogicalDisk(_Total)\Avg. Disk sec/Transfer' -SampleInterval 2 -MaxSamples 5
  • Processes with the most CPU timeGet-Process | Sort-Object CPU -Descending | Select-Object -First 10 Name, Id, CPU
  • Processes with the most memoryGet-Process | Sort-Object WorkingSet64 -Descending | Select-Object -First 10 Name, Id, @{n='MB'; e={[int]($_.WorkingSet64 / 1MB)}}
  • Automatic services that are not runningGet-CimInstance Win32_Service -Filter "StartMode='Auto' AND State<>'Running'" | Select-Object Name, DisplayName, ExitCode
  • Free space per volumeGet-Volume | Where-Object DriveLetter | Select-Object DriveLetter, FileSystemLabel, @{n='FreeGB'; e={[int]($_.SizeRemaining / 1GB)}}, @{n='SizeGB'; e={[int]($_.Size / 1GB)}}
  • Reboot pending (servicing or Windows Update)'Component Based Servicing\RebootPending', 'WindowsUpdate\Auto Update\RebootRequired' | ForEach-Object { Test-Path "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\$_" }

System

systemd

  • Status and last log linessystemctl status <unit>
  • Start now and at bootsystemctl enable --now <unit>
  • Reload after editing unit filessystemctl daemon-reload
  • Override a unit without touching the vendor filesystemctl edit <unit>
  • Unit file as systemd sees it, drop-ins includedsystemctl cat <unit>
  • Effective limits and restart policysystemctl show <unit> -p Restart,MemoryMax,LimitNOFILE,TasksMax
  • After “start request repeated too quickly”systemctl reset-failed <unit> && systemctl start <unit>
  • Keep a unit from starting, even as a dependencysystemctl mask <unit>
  • Check a unit file before loading itsystemd-analyze verify <file>.service
  • Run one command with a memory and CPU capsystemd-run --scope -p MemoryMax=2G -p CPUQuota=50% <command>
  • CPU, memory and I/O per cgroup, livesystemd-cgtop
  • Timers with last and next runsystemctl list-timers --all
  • Every user’s crontab (jobs outside systemd timers)for u in $(cut -d: -f1 /etc/passwd); do crontab -l -u "$u" 2>/dev/null | sed "s/^/$u: /"; done
  • What slowed down bootsystemd-analyze blame

Logs

  • Follow one unitjournalctl -u <unit> -f
  • Last hour, all unitsjournalctl --since "1 hour ago"
  • Exact time windowjournalctl --since "2026-01-01 10:00" --until "2026-01-01 10:30"
  • Previous boot (what happened before the crash)journalctl -b -1 -e
  • Boots on record, to pick one for -bjournalctl --list-boots
  • Kernel messages onlyjournalctl -k
  • Everything sudo ranjournalctl _COMM=sudo
  • Keep logs across rebootsmkdir -p /var/log/journal && systemctl restart systemd-journald
  • Follow a plain log file across rotationtail -F <file>

Processes

  • Find a process by namepgrep -a <name>
  • Process treeps -ef --forest
  • Files a process has openlsof -p <pid>
  • Who is using a portss -tlnp 'sport = :<port>'
  • Environment of a running processtr '\0' '\n' < /proc/<pid>/environ
  • Binary and working directory of a processls -l /proc/<pid>/exe /proc/<pid>/cwd
  • Kernel stack of a hung processcat /proc/<pid>/stack
  • Trace system callsstrace -f -p <pid>
  • Only file and network syscalls, timestampedstrace -f -tt -e trace=%file,%network -p <pid>
  • Hottest functions, liveperf top -p <pid>
  • De-prioritise CPU and disk for a noisy jobrenice -n 19 -p <pid> && ionice -c3 -p <pid>
  • Watch a command, highlighting what changedwatch -n1 -d '<command>'
  • Session that survives logouttmux new -s <name>
  • Reattach to ittmux attach -t <name>

Files and text

  • Files changed in the last dayfind <dir> -type f -mtime -1
  • Delete files older than 30 days (run with -print first)find <dir> -type f -mtime +30 -delete
  • Top talkers in a log: count and rank one columnawk '{print $1}' <log> | sort | uniq -c | sort -rn | head
  • “Operation not permitted” as root: immutable bitlsattr <file> && chattr -i <file>
  • Compare one file across two hostsdiff <(ssh <host-a> cat <file>) <(ssh <host-b> cat <file>)
  • Run a command on many hosts in parallelxargs -P 10 -I{} ssh -o BatchMode=yes {} '<command>' < <hosts-file>
  • Pack a directorytar -czf <archive>.tar.gz <dir>
  • Unpack into a directorytar -xzf <archive>.tar.gz -C <dir>
  • List an archive without unpackingtar -tzf <archive>.tar.gz | head

Users and security

  • Change owner recursivelychown -R <user>:<group> <dir>
  • Directories 755, files 644find <dir> -type d -exec chmod 755 {} + && find <dir> -type f -exec chmod 644 {} +
  • Add a user to a group (next login)usermod -aG <group> <user>
  • Show ACLsgetfacl <path>
  • Grant one user access via ACLsetfacl -m u:<user>:rwx <path>
  • Default ACL so new files inherit group accesssetfacl -R -d -m g:<group>:rwX <dir>
  • Password and account expiry (service account locked out)chage -l <user>
  • SELinux denials, recentausearch -m avc -ts recent
  • Reset SELinux labels after moving filesrestorecon -Rv <path>
  • Allow a service on a non-standard port under SELinuxsemanage port -a -t http_port_t -p tcp <port>
  • Failed SSH logins todayjournalctl -u ssh -u sshd -g 'Failed password|Invalid user' --since today

Packages

  • Debian: installed and candidate version, and which repoapt-cache policy <package>
  • Debian: which package owns a filedpkg -S <path>
  • Debian: finish an interrupted upgradedpkg --configure -a && apt -f install
  • Debian: pin a package versionapt-mark hold <package>
  • Debian: reboot required, and by whatcat /var/run/reboot-required.pkgs
  • Debian: services still running old librariesneedrestart -r l
  • RHEL: which package owns a filerpm -qf <path>
  • RHEL: which package provides a binarydnf provides '*/<binary>'
  • RHEL: undo a transactiondnf history && dnf history undo <id>
  • RHEL: reboot requireddnf needs-restarting -r

Boot and rescue

  • Root is read-only in emergency modemount -o remount,rw /
  • Chroot into a broken install from a live ISOmount /dev/<root> /mnt && for d in dev proc sys run; do mount --rbind /$d /mnt/$d; done && chroot /mnt
  • Kernel command line the system booted withcat /proc/cmdline
  • UEFI boot entries and orderefibootmgr -v
  • Debian: rebuild every initramfsupdate-initramfs -u -k all
  • RHEL: rebuild every initramfsdracut -f --regenerate-all
  • Debian: reinstall GRUB on a BIOS diskgrub-install /dev/<disk> && update-grub

Hardware

  • Hardware event log (PSU, DIMM, fan faults)ipmitool sel elist | tail -20
  • Sensors: temperatures, fans, voltagesipmitool sdr elist
  • BMC network settings from the hostipmitool lan print 1
  • BMC unresponsive: cold-reset it from the hostipmitool mc reset cold
  • Serial number / Dell service tagdmidecode -s system-serial-number
  • DIMMs per slot with size and speeddmidecode -t memory | grep -E 'Locator|Size|Speed'
  • CPU model, sockets, cores, flagslscpu
  • ECC errors per memory controllergrep . /sys/devices/system/edac/mc/mc*/[cu]e_count
  • PCIe link trained below its capabilitylspci -vv -s <bus:dev.fn> | grep -E 'LnkCap|LnkSta'
  • NUMA layout (pin VMs and IRQs to one node)numactl --hardware
  • CPU temperatures and fanssensors

NixOS: rebuild

  • Build and switchsudo nixos-rebuild switch --flake .#<host>
  • Activate without a boot entrysudo nixos-rebuild test --flake .#<host>
  • Apply on next boot only (kernel, init changes)sudo nixos-rebuild boot --flake .#<host>
  • Build only, change nothingnixos-rebuild build --flake .#<host>
  • Deploy to a remote machinenixos-rebuild switch --flake .#<host> --target-host <user>@<host> --sudo
  • Back to the previous generationsudo nixos-rebuild switch --rollback
  • What changed since the last bootnix store diff-closures /run/booted-system /run/current-system

NixOS: flakes and store

  • Update all inputsnix flake update
  • Update one inputnix flake update <input>
  • Evaluate every output before deployingnix flake check
  • Evaluate an optionnix eval .#nixosConfigurations.<host>.config.<option>
  • Explore the config in a REPLnix repl --expr 'builtins.getFlake (toString ./.)'
  • System generationsnix-env --list-generations --profile /nix/var/nix/profiles/system
  • Closure size of the running systemnix path-info -Sh /run/current-system
  • GC frees nothing: find what still roots itnix-store --gc --print-roots | grep -v '^/proc'
  • Delete old generations and garbagesudo nix-collect-garbage --delete-older-than 14d
  • Deduplicate the storenix store optimise
  • Store corruption: verify and repairsudo nix-store --verify --check-contents --repair
  • Try a package without installingnix shell nixpkgs#<package>

Windows: repair and updates

  • Repair the component store first (needs Windows Update or a source)DISM /Online /Cleanup-Image /RestoreHealth
  • Then repair system filessfc /scannow
  • What sfc could not repairSelect-String -Path C:\Windows\Logs\CBS\CBS.log -Pattern '\[SR\] Cannot repair'
  • Shrink WinSxS by removing superseded componentsDISM /Online /Cleanup-Image /StartComponentCleanup
  • Recently installed updatesGet-HotFix | Sort-Object InstalledOn -Descending | Select-Object -First 10 HotFixID, Description, InstalledOn
  • Find an update’s package name to remove itDISM /Online /Get-Packages /Format:Table | findstr <kb-number>
  • Remove itDISM /Online /Remove-Package /PackageName:<package-name>
  • Readable Windows Update log (written to the desktop)Get-WindowsUpdateLog
  • Updates fail behind a proxy: the WinHTTP proxy settingnetsh winhttp show proxy
  • Online scan of a volume, no downtimeRepair-Volume -DriveLetter <letter> -Scan

Windows: services and tasks

  • Service stuck in Stopping: its PID (sc.exe, since sc is an alias)sc.exe queryex <service>
  • Then kill ittaskkill /PID <pid> /F
  • Why a service will not start: Service Control Manager eventsGet-WinEvent -FilterHashtable @{LogName='System'; ProviderName='Service Control Manager'} -MaxEvents 20 | Select-Object TimeCreated, Id, Message
  • Scheduled tasks whose last run failed (ignoring running and never-run)Get-ScheduledTask | Where-Object State -ne 'Disabled' | Get-ScheduledTaskInfo | Where-Object LastTaskResult -notin 0, 267009, 267011 | Select-Object TaskName, LastRunTime, LastTaskResult
  • Which process holds a file (Sysinternals Handle)handle.exe -accepteula <name>

Windows: Active Directory

  • Replication summary across all DCsrepadmin /replsummary
  • Replication partners and last result for one DCrepadmin /showrepl <dc>
  • DC health across the forest, errors onlydcdiag /e /q
  • Which DC this machine authenticates againstnltest /dsgetdc:<domain>
  • FSMO role holdersnetdom query fsmo
  • “Trust relationship failed”: repair the secure channel (Windows PowerShell 5.1, local admin)Test-ComputerSecureChannel -Repair -Credential <domain>\<admin>
  • Locked-out accountsSearch-ADAccount -LockedOut | Select-Object Name, SamAccountName, LastLogonDate
  • Which machine keeps locking an account (event 4740 on the PDC)Get-WinEvent -ComputerName (Get-ADDomain).PDCEmulator -FilterHashtable @{LogName='Security'; Id=4740} -MaxEvents 20 | Select-Object TimeCreated, @{n='User'; e={$_.Properties[0].Value}}, @{n='Caller'; e={$_.Properties[1].Value}}
  • Unlock the accountUnlock-ADAccount -Identity <user>
  • Applied Group Policy, as a reportgpresult /h C:\Temp\gpresult.html /f
  • Reapply Group Policy nowgpupdate /force
  • Kerberos tickets: list, then purge after a group changeklist; klist purge

Storage

Disks

  • Block devices with filesystems and UUIDslsblk -f
  • Model, serial, size, rotational, transportlsblk -o NAME,SIZE,MODEL,SERIAL,ROTA,TRAN
  • Logical and physical sector size (pick ashift)lsblk -o NAME,LOG-SEC,PHY-SEC
  • Stable device namesls -l /dev/disk/by-id/
  • Partition tableparted /dev/<disk> print
  • Find a newly hot-added disk (SCSI host rescan)echo "- - -" | tee /sys/class/scsi_host/host*/scan
  • See a disk’s new size after growing it (device rescan)echo 1 > /sys/class/block/<disk>/device/rescan
  • Same for NVMe namespacesnvme ns-rescan /dev/<nvme-controller>
  • Detach a disk cleanly before pulling itecho 1 > /sys/block/<disk>/device/delete
  • Re-read the partition table without rebootingpartprobe /dev/<disk>
  • Grow a partition into the new spacegrowpart /dev/<disk> <partition-number>
  • Find a physical drive: hold its activity LED ondd if=/dev/<disk> of=/dev/null bs=1M status=progress
  • Wipe filesystem, RAID and LVM signatures (destroys data)wipefs -a /dev/<disk>
  • Also clear GPT, including the backup at the endsgdisk --zap-all /dev/<disk>
  • “Device busy” on a wipe: leftover device-mapper entriesdmsetup ls && dmsetup remove <name>

Filesystems and mounts

  • Grow ext4 onlineresize2fs /dev/<device>
  • Grow XFS online (takes the mountpoint, not the device)xfs_growfs <mountpoint>
  • Force a full ext4 check (unmounted)e2fsck -f /dev/<device>
  • Repair XFS (fsck.xfs does nothing)xfs_repair /dev/<device>
  • Mount treefindmnt
  • fstab entry that will not hang boot if the disk is missingUUID=<uuid> /mnt/<name> ext4 defaults,noatime,nofail,x-systemd.device-timeout=10s 0 2
  • Test fstab without rebootingsystemctl daemon-reload && findmnt --verify && mount -a
  • Who keeps a mount busyfuser -vm <mountpoint>
  • Detach a busy mountumount -l <mountpoint>
  • Why a filesystem went read-onlydmesg -T | grep -iE 'remount|i/o error|ext4-fs error|xfs'
  • Trim all mounted SSDsfstrim -av

Disk health and performance

  • SMART verdict and attributessmartctl -H -A /dev/<disk>
  • SMART for a disk behind a PERC / MegaRAIDsmartctl -a -d megaraid,<n> /dev/sda
  • Start a long self-testsmartctl -t long /dev/<disk>
  • Self-test resultssmartctl -l selftest /dev/<disk>
  • NVMe health and wearnvme smart-log /dev/<nvme>
  • Spin down a SATA disk nowhdparm -y /dev/<disk>
  • Spin down a SAS disk nowsdparm --command=stop /dev/<disk>
  • Live I/O per processiotop -o
  • I/O latency of a directory, like pingioping -c 10 <dir>
  • Random 4k read/write benchmarkfio --name=rand --filename=<file> --size=1G --rw=randrw --bs=4k --direct=1 --runtime=30 --time_based --group_reporting

NFS and SMB

  • Server: apply /etc/exports changesexportfs -ra
  • Exports a server offersshowmount -e <server>
  • Mount NFSmount -t nfs4 <server>:/<export> <mountpoint>
  • Mount options actually negotiatednfsstat -m
  • Stale NFS handle: drop the mountumount -f -l <mountpoint>
  • List SMB sharessmbclient -L //<server> -U <user>
  • Mount SMB with a credentials filemount -t cifs //<server>/<share> <mountpoint> -o credentials=/root/.smbcred,uid=<uid>,gid=<gid>,vers=3.0

LVM

  • Physical volumes, groups, logical volumespvs && vgs && lvs
  • Which disks back each LVlvs -a -o +devices
  • Thin pool data and metadata filllvs -o lv_name,data_percent,metadata_percent <vg>
  • Activate a VG after moving disksvgchange -ay <vg>
  • Metadata backups (undo a bad change)vgcfgrestore --list <vg>
  • Initialise a physical volumepvcreate /dev/<partition>
  • Create a volume groupvgcreate <vg> /dev/<partition>
  • Fixed-size logical volumelvcreate -n <lv> -L 50G <vg>
  • Logical volume from all free spacelvcreate -n <lv> -l 100%FREE <vg>
  • Thin poollvcreate --type thin-pool -n <pool> -L 100G <vg>

LVM: grow and shrink

  • Add a disk to a VGvgextend <vg> /dev/<partition>
  • PV after its disk was enlarged (rescan the disk first)pvresize /dev/<partition>
  • Grow LV and filesystem togetherlvextend -r -L +10G /dev/<vg>/<lv>
  • Grow into all free spacelvextend -r -l +100%FREE /dev/<vg>/<lv>
  • Shrink LV and ext4 (unmounted; XFS cannot shrink)lvreduce -r -L 20G /dev/<vg>/<lv>
  • Grow a thin poollvextend -L +50G <vg>/<pool>
  • Thin pool metadata filling up (full means corruption)lvextend --poolmetadatasize +1G <vg>/<pool>
  • Evacuate a diskpvmove /dev/<partition>
  • Then drop it from the VGvgreduce <vg> /dev/<partition>

LVM: snapshots and rescue

  • Snapshot an LVlvcreate -s -n <snap> -L 5G /dev/<vg>/<lv>
  • Roll back to the snapshotlvconvert --merge /dev/<vg>/<snap>
  • Drop the snapshotlvremove /dev/<vg>/<snap>
  • Two VGs with the same name: rename one by UUIDvgs -o vg_name,vg_uuid && vgrename <vg-uuid> <new-name>
  • Map partitions inside an LV (VM disk) to mount themkpartx -av /dev/<vg>/<lv>

ZFS: pools

  • Only pools with problemszpool status -x
  • Files hit by permanent errorszpool status -v <pool>
  • Capacity per vdevzpool list -v
  • Mirror by stable disk IDszpool create -o ashift=12 <pool> mirror /dev/disk/by-id/<disk-a> /dev/disk/by-id/<disk-b>
  • ashift of an existing poolzdb -C <pool> | grep ashift
  • Replace a failed diskzpool replace <pool> <old-disk> /dev/disk/by-id/<new-disk>
  • Use the space after replacing with bigger diskszpool set autoexpand=on <pool> && zpool online -e <pool> <disk>
  • Start a scrubzpool scrub <pool>
  • Reset error counters after fixing the causezpool clear <pool>
  • Latency per vdev, livezpool iostat -vl 2
  • Who ran what against a poolzpool history <pool> | tail -20
  • Old ZFS label blocking reuse of a diskzpool labelclear -f /dev/<disk>
  • Find pools to importzpool import
  • Import by stable IDszpool import -d /dev/disk/by-id <pool>
  • Import read-only for rescuezpool import -o readonly=on -R /mnt <pool>

ZFS: datasets and snapshots

  • Usage per datasetzfs list -o name,used,avail,refer,mountpoint
  • Where the space went: data, snapshots, childrenzfs list -o space -r <pool>
  • Dataset with zstd compressionzfs create -o compression=zstd <pool>/<dataset>
  • Compression ratio achievedzfs get compressratio <pool>/<dataset>
  • Only properties set by handzfs get -s local all <pool>/<dataset>
  • Cap a datasetzfs set quota=100G <pool>/<dataset>
  • Snapshot recursivelyzfs snapshot -r <pool>/<dataset>@<name>
  • Snapshots, oldest firstzfs list -t snapshot -o name,used,creation -s creation
  • Preview destroying a range of snapshotszfs destroy -nv <pool>/<dataset>@<first>%<last>
  • Roll back (destroys later snapshots)zfs rollback -r <pool>/<dataset>@<name>
  • Full send to another hostzfs send <pool>/<dataset>@<name> | ssh <host> zfs receive <pool>/<dataset>
  • Incremental send, one stepzfs send -i @<old> <pool>/<dataset>@<new> | ssh <host> zfs receive <pool>/<dataset>
  • Incremental send with every snapshot in betweenzfs send -I @<old> <pool>/<dataset>@<new> | ssh <host> zfs receive <pool>/<dataset>

ZFS: ARC

  • ARC size and hit ratearc_summary | head -40
  • Live ARC statsarcstat 1
  • Cap ARC at 8 GB nowecho $((8 * 1024**3)) > /sys/module/zfs/parameters/zfs_arc_max
  • Cap ARC at 8 GB across rebootsecho "options zfs zfs_arc_max=8589934592" > /etc/modprobe.d/zfs.conf && update-initramfs -u -k all

Windows: files and storage

  • Mirror a folder with permissions, logged (/MIR deletes extras; add /L to preview)robocopy <src> <dst> /MIR /COPY:DATS /DCOPY:DAT /MT:16 /R:1 /W:1 /LOG:C:\Temp\robocopy.log
  • Largest files on a driveGet-ChildItem <drive>:\ -Recurse -File -ErrorAction SilentlyContinue | Sort-Object Length -Descending | Select-Object -First 20 FullName, @{n='MB'; e={[int]($_.Length / 1MB)}}
  • NTFS permissionsicacls <path>
  • Access denied even as admin: take ownership, reset ACLs to inheritedtakeown /F <path> /R /D Y; icacls <path> /reset /T /C
  • New disk shows Offline (SAN policy): bring it online and writableSet-Disk -Number <n> -IsOffline $false; Set-Disk -Number <n> -IsReadOnly $false
  • Grew a VM disk: rescan, then extend the partition to the endUpdate-HostStorageCache; Resize-Partition -DriveLetter <letter> -Size (Get-PartitionSupportedSize -DriveLetter <letter>).SizeMax
  • Backup fails: VSS writers in a failed statevssadmin list writers
  • Shadow copy storage per volumevssadmin list shadowstorage
  • File server: who has a file openGet-SmbOpenFile | Where-Object Path -like '*<name>*' | Select-Object ClientComputerName, ClientUserName, Path, FileId
  • Close that handleClose-SmbOpenFile -FileId <file-id> -Force
  • Share permissions (NTFS permissions apply on top)Get-SmbShareAccess -Name <share>

Network

Interfaces and routes

  • Addresses, one line per interfaceip -br a
  • Which route and source IP a destination takesip route get <ip>
  • Policy routing (Tailscale, VPNs, multi-WAN add rules)ip rule && ip route show table all
  • ARP / neighbour tableip neigh
  • Add an address until rebootip addr add <ip>/<prefix> dev <iface>
  • RX/TX errors and drops on an interfaceip -s link show <iface>
  • Link speed and driverethtool <iface> && ethtool -i <iface>
  • NIC counters: CRC errors point at cable or opticethtool -S <iface> | grep -iE 'err|drop|crc'
  • Bond state and active slavecat /proc/net/bonding/<bond>
  • Bridge membersbridge link
  • VLANs on a VLAN-aware bridgebridge vlan show
  • Someone else has this IParping -D -I <iface> -c 3 <ip>
  • Reverse-path filter dropping asymmetric trafficsysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.<iface>.rp_filter
  • Apply /etc/network/interfaces (Proxmox, ifupdown2)ifreload -a
  • NetworkManager: static IPv4 on a connectionnmcli con mod <connection> ipv4.method manual ipv4.addresses <ip>/<prefix> ipv4.gateway <gateway> && nmcli con up <connection>

Ports and connections

  • Listening sockets with processesss -tulpn
  • Socket totalsss -s
  • Established connections on a portss -tnp state established '( sport = :<port> or dport = :<port> )'
  • RTT, retransmits and window per connectionss -ti dst <ip>
  • Is a remote port opennc -zv <host> <port>
  • Talk to one backend, bypassing DNS and load balancercurl -v --resolve <host>:443:<ip> https://<host>/
  • HTTP status and timingscurl -o /dev/null -s -w '%{http_code} dns=%{time_namelookup}s connect=%{time_connect}s tls=%{time_appconnect}s total=%{time_total}s\n' <url>
  • Certificate a server presents: subject, SANs, datesopenssl s_client -connect <host>:443 -servername <host> </dev/null 2>/dev/null | openssl x509 -noout -subject -ext subjectAltName -dates
  • Expiry of a certificate fileopenssl x509 -in <cert>.pem -noout -enddate

DNS

  • Just the answerdig +short <name>
  • Ask a specific resolverdig @<server> <name> A
  • What the system resolver returns (/etc/hosts, nsswitch)getent hosts <name>
  • Follow delegation from the rootdig +trace <name>
  • Reverse lookupdig -x <ip>
  • systemd-resolved state per linkresolvectl status
  • Flush the local cacheresolvectl flush-caches

Path and packets

  • Loss and latency per hopmtr -rwc 50 <host>
  • Largest packet that fits: 1472 for 1500 MTU, 1252 for 1280ping -M do -s 1472 <host>
  • Watch traffic on a porttcpdump -ni <iface> port <port>
  • Capture on every interface, bridges includedtcpdump -ni any host <ip>
  • Capture a host to a file for Wiresharktcpdump -ni <iface> -w <file>.pcap host <ip>
  • Throughput test: serveriperf3 -s
  • Throughput test: client, 4 streamsiperf3 -c <server> -P 4
  • TCP retransmits and drops, system-widenstat -az | grep -iE 'retrans|drop'
  • Bandwidth per connectioniftop -i <iface>

Firewall and NAT

  • nftables rulesetnft list ruleset
  • iptables with countersiptables -L -n -v --line-numbers
  • NAT rules (Docker and Kubernetes write here)iptables -t nat -L -n -v
  • Tracked connections to a hostconntrack -L -d <ip>
  • Drop tracked state after changing rulesconntrack -D -d <ip>
  • ufw statusufw status verbose
  • ufw: SSH from one subnetufw allow from <cidr> to any port 22 proto tcp
  • firewalld zone rulesfirewall-cmd --list-all
  • firewalld: open a portfirewall-cmd --permanent --add-port=<port>/tcp && firewall-cmd --reload
  • Enable IPv4 forwarding nowsysctl -w net.ipv4.ip_forward=1
  • IPv4 forwarding across rebootsecho 'net.ipv4.ip_forward=1' > /etc/sysctl.d/99-forward.conf && sysctl --system

Tailscale

  • Peers and connection typetailscale status
  • Direct or relayed through DERPtailscale ping <node>
  • NAT type and DERP latencytailscale netcheck
  • Advertise a subnettailscale set --advertise-routes=<cidr>
  • Accept subnets from other routerstailscale set --accept-routes
  • Route through an exit nodetailscale set --exit-node=<node>

SSH

  • New key pairssh-keygen -t ed25519 -C "<comment>"
  • Install a public key on a hostssh-copy-id -i ~/.ssh/id_ed25519.pub <user>@<host>
  • Key auth refused: fix permissionschmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys
  • Kill a frozen session (press Enter first)~.
  • Verbose handshakessh -vvv <user>@<host>
  • Old switch, iDRAC or iLO rejects the key algorithmssh -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa <host>
  • Forget a changed host keyssh-keygen -R <host>
  • Trust a host key non-interactivelyssh-keyscan -H <host> >> ~/.ssh/known_hosts
  • Fingerprint of a server keyssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
  • Validate sshd_config before restartingsshd -t
  • Effective sshd setting for a given usersshd -T -C user=<user> | grep -i <option>

SSH tunnels

  • Reach a remote-only port locallyssh -N -L 8080:localhost:<remote-port> <host>
  • Reach a web UI behind a bastionssh -N -L 8443:<target>:443 <bastion>
  • Expose a local port on the remotessh -N -R 9000:localhost:<local-port> <host>
  • SOCKS proxy through a hostssh -N -D 1080 <host>
  • Hop through a bastionssh -J <bastion> <host>

Copy and transfer

  • Mirror a directory, preserving everythingrsync -aHAX --info=progress2 --delete <src>/ <dst>/
  • Preview what rsync would changersync -aHAXn --delete --itemize-changes <src>/ <dst>/
  • Verify a copy by checksum: list files that differrsync -rcn --out-format='%n' <src>/ <dst>/
  • Push over SSH, compressedrsync -az --info=progress2 <src>/ <user>@<host>:<dst>/
  • Resumable, throttled copy over SSHrsync -aHAX --partial --bwlimit=50M --info=progress2 <src>/ <user>@<host>:<dst>/
  • Stream a directory over SSHtar -cf - <dir> | ssh <host> 'tar -xf - -C <dst>'
  • Serve the current directory over HTTPpython3 -m http.server 8000

Windows: network

  • Addresses, gateway and DNS per interfaceGet-NetIPConfiguration
  • Is a port reachableTest-NetConnection <host> -Port <port>
  • Trace the routeTest-NetConnection <host> -TraceRoute
  • Resolve against a specific DNS serverResolve-DnsName <name> -Server <server>
  • Flush the DNS client cacheClear-DnsClientCache
  • Listening ports with their processGet-NetTCPConnection -State Listen | Select-Object LocalAddress, LocalPort, @{n='Process'; e={(Get-Process -Id $_.OwningProcess).ProcessName}} | Sort-Object LocalPort
  • Network marked Public blocks sharing and remoting: make it PrivateSet-NetConnectionProfile -InterfaceAlias <interface> -NetworkCategory Private
  • Firewall state per profileGet-NetFirewallProfile | Select-Object Name, Enabled, DefaultInboundAction
  • Allow an inbound portNew-NetFirewallRule -DisplayName "<name>" -Direction Inbound -Protocol TCP -LocalPort <port> -Action Allow
  • Capture full packets with the built-in toolpktmon start --capture --pkt-size 0 --file-name <file>.etl
  • Stop, and convert the capture for Wiresharkpktmon stop; pktmon etl2pcap <file>.etl --out <file>.pcapng
  • Time source and last syncw32tm /query /status
  • Clock offset against a domain controllerw32tm /stripchart /computer:<dc> /samples:5 /dataonly
  • Force a time resyncw32tm /resync /force

Windows: remote management

  • Is WinRM answeringTest-WSMan <host>
  • Run a command on several serversInvoke-Command -ComputerName <host-a>, <host-b> -ScriptBlock { <command> }
  • Interactive remote sessionEnter-PSSession -ComputerName <host>
  • Who is logged on (RDP sessions)quser /server:<host>
  • Log off a stuck sessionlogoff <session-id> /server:<host>
  • Enable RDP and its firewall ruleSet-ItemProperty 'HKLM:\System\CurrentControlSet\Control\Terminal Server' -Name fDenyTSConnections -Value 0; Enable-NetFirewallRule -DisplayGroup 'Remote Desktop'
  • Installed software (not Win32_Product, which triggers MSI repairs)Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\*, HKLM:\Software\WOW6432Node\Microsoft\Windows\CurrentVersion\Uninstall\* | Where-Object DisplayName | Select-Object DisplayName, DisplayVersion | Sort-Object DisplayName
  • Installed roles and featuresGet-WindowsFeature | Where-Object Installed

Virtualization

Proxmox: VMs

  • List VMsqm list
  • Clean shutdownqm shutdown <vmid>
  • Hard stopqm stop <vmid>
  • Stop a VM that holds a lockqm stop <vmid> --skiplock
  • QEMU process ignoring stop: kill itkill -9 "$(cat /var/run/qemu-server/<vmid>.pid)"
  • Clear a stale lockqm unlock <vmid>
  • Show configqm config <vmid>
  • Full KVM command line a VM starts withqm showcmd <vmid> --pretty
  • Serial consoleqm terminal <vmid>
  • Grow a disk (then rescan it inside the guest)qm disk resize <vmid> scsi0 +10G
  • Move a disk to another storage, liveqm disk move <vmid> scsi0 <storage> --delete 1
  • Import a cloud image as a diskqm disk import <vmid> <image>.qcow2 <storage>
  • Import an OVF exportqm importovf <vmid> <file>.ovf <storage>
  • Pick up orphaned disk images on storagesqm disk rescan --vmid <vmid>
  • Snapshotqm snapshot <vmid> <name>
  • Roll back to a snapshotqm rollback <vmid> <name>
  • Full clone of a templateqm clone <template-id> <new-id> --name <name> --full
  • Live migrateqm migrate <vmid> <node> --online
  • IPs via guest agentqm guest cmd <vmid> network-get-interfaces
  • Mount a stopped VM’s partition from a ZFS zvolmount /dev/zvol/<pool>/vm-<vmid>-disk-<n>-part<p> /mnt

Proxmox: containers

  • List containerspct list
  • Shell insidepct enter <ctid>
  • Run one commandpct exec <ctid> -- <command>
  • Grow the root diskpct resize <ctid> rootfs +5G
  • Bind-mount a host directorypct set <ctid> -mp0 <host-path>,mp=<container-path>
  • Unprivileged bind mount: host UID = 100000 + container UIDchown -R 100000:100000 <host-path>
  • Copy a file inpct push <ctid> <file> <container-path>
  • Mount a container’s rootfs on the host (rescue)pct mount <ctid>
  • Check a container’s filesystempct fsck <ctid>
  • Clear a stale lockpct unlock <ctid>

Proxmox: host, storage and backup

  • Versions of every PVE packagepveversion -v
  • Core servicessystemctl status pveproxy pvedaemon pve-cluster
  • Web UI unreachable: restart its servicessystemctl restart pveproxy pvedaemon
  • Cluster quorumpvecm status
  • Lost quorum, /etc/pve read-only (last resort)pvecm expected 1
  • Regenerate node certificatespvecm updatecerts --force
  • ESPs in sync on a ZFS or UEFI rootproxmox-boot-tool status
  • Upgrade (dist-upgrade; plain apt upgrade can break PVE)apt update && apt dist-upgrade
  • Storage pools and usagepvesm status
  • What is on a storagepvesm list <storage>
  • Back up nowvzdump <vmid> --storage <storage> --mode snapshot --compress zstd
  • Restore a VMqmrestore <backup>.vma.zst <new-id> --storage <storage>
  • Restore a containerpct restore <new-id> <backup>.tar.zst --storage <storage>

KVM: passthrough and libvirt

  • CPU virtualization flags presentgrep -Ec '(vmx|svm)' /proc/cpuinfo
  • IOMMU enableddmesg | grep -e DMAR -e IOMMU
  • Devices per IOMMU groupfor d in /sys/kernel/iommu_groups/*/devices/*; do g=${d#*/iommu_groups/}; printf 'group %s\t' "${g%%/*}"; lspci -nns "${d##*/}"; done
  • Driver bound to a device (vfio-pci or not)lspci -nnk -s <bus:dev.fn>
  • Nested virtualization on (Intel)cat /sys/module/kvm_intel/parameters/nested
  • libvirt domainsvirsh list --all
  • libvirt: guest IPsvirsh domifaddr <domain>
  • Disks attached to a domainvirsh domblklist <domain>
  • Image format and sizeqemu-img info <disk>
  • Convert VMDK to qcow2qemu-img convert -p -f vmdk -O qcow2 <in>.vmdk <out>.qcow2

Hyper-V

  • VMs with state, uptime and memoryGet-VM | Select-Object Name, State, Uptime, CPUUsage, MemoryAssigned
  • Force a VM offStop-VM -Name <vm> -TurnOff
  • VM stuck Stopping: kill its worker processGet-CimInstance Win32_Process -Filter "Name='vmwp.exe'" | Where-Object CommandLine -match (Get-VM <vm>).VMId | ForEach-Object { Stop-Process -Id $_.ProcessId -Force }
  • Checkpoints of a VMGet-VMCheckpoint -VMName <vm>
  • Delete all checkpoints, merging them into the diskGet-VMCheckpoint -VMName <vm> | Remove-VMCheckpoint
  • Grow a VHDX (then extend the partition in the guest)Resize-VHD -Path <path>.vhdx -SizeBytes 200GB
  • Virtual switches and the adapters behind themGet-VMSwitch | Select-Object Name, SwitchType, NetAdapterInterfaceDescription

ESXi: host and logs

  • Version and buildvmware -vl
  • Enable and start SSH from the ESXi shellvim-cmd hostsvc/enable_ssh && vim-cmd hostsvc/start_ssh
  • Enter maintenance modeesxcli system maintenanceMode set --enable true
  • Leave maintenance modeesxcli system maintenanceMode set --enable false
  • Reboot (host must be in maintenance mode)esxcli system shutdown reboot --reason "<reason>"
  • Host client or vCenter lost the host: restart management agents/etc/init.d/hostd restart && /etc/init.d/vpxa restart
  • Restart every management service (heavier)services.sh restart
  • Vendor, model and serialesxcli hardware platform get
  • NTP servers and stateesxcli system ntp get
  • Back up the host configuration (prints a download URL)vim-cmd hostsvc/firmware/sync_config && vim-cmd hostsvc/firmware/backup_config
  • Support bundle for VMware / Broadcomvm-support
  • Storage, network and driver eventstail -f /var/log/vmkernel.log
  • Host agent: VM operations, API callstail -f /var/log/hostd.log
  • Hardware and connectivity observationstail -f /var/log/vobd.log
  • Why a VM failed to power ontail -100 /vmfs/volumes/<datastore>/<vm>/vmware.log
  • Send logs to a remote syslogesxcli system syslog config set --loghost='udp://<ip>:514' && esxcli system syslog reload && esxcli network firewall ruleset set -r syslog -e true

ESXi: VMs and snapshots

  • Registered VMs with their IDsvim-cmd vmsvc/getallvms
  • Power statevim-cmd vmsvc/power.getstate <vmid>
  • Power onvim-cmd vmsvc/power.on <vmid>
  • Guest shutdown (needs VMware Tools)vim-cmd vmsvc/power.shutdown <vmid>
  • Hard power offvim-cmd vmsvc/power.off <vmid>
  • VM will not power off: find its world IDesxcli vm process list
  • Then kill it: soft, then hard, then forceesxcli vm process kill --type=soft --world-id=<world-id>
  • Guest IP from VMware Toolsvim-cmd vmsvc/get.guest <vmid> | grep -m1 ipAddress
  • Register a VM from its .vmxvim-cmd solo/registervm /vmfs/volumes/<datastore>/<vm>/<vm>.vmx
  • Unregister (files stay on the datastore)vim-cmd vmsvc/unregister <vmid>
  • Reload a .vmx after editing it by handvim-cmd vmsvc/reload <vmid>
  • Stuck on a question (“moved or copied?”): list itvim-cmd vmsvc/message <vmid>
  • Answer itvim-cmd vmsvc/message <vmid> <message-id> <choice>
  • Snapshot treevim-cmd vmsvc/get.snapshotinfo <vmid>
  • Create a snapshot without memoryvim-cmd vmsvc/snapshot.create <vmid> <name> <description> 0 0
  • Delete all snapshots (consolidates into the base disk)vim-cmd vmsvc/snapshot.removeall <vmid>
  • Delta disks left behindfind /vmfs/volumes/<datastore>/<vm>/ -name '*-0000*.vmdk'

ESXi: storage and virtual disks

  • Datastores with capacity and free spaceesxcli storage filesystem list
  • See a new or grown LUN: rescan every adapteresxcli storage core adapter rescan --all
  • Refresh VMFS volumes after the rescanvmkfstools -V
  • Devices with their naa IDsesxcli storage core device list
  • Paths per device and their stateesxcli storage core path list
  • Multipathing policy per deviceesxcli storage nmp device list
  • SMART data for a local diskesxcli storage core device smart get -d <naa-id>
  • Mount an NFS v3 datastoreesxcli storage nfs add -H <server> -s /<export> -v <datastore-name>
  • Copied or replicated LUN not mounting: VMFS snapshot volumesesxcli storage vmfs snapshot list
  • Mount it keeping its signatureesxcli storage vmfs snapshot mount -l <volume-label>
  • Reclaim freed blocks on thin-provisioned storageesxcli storage vmfs unmap -l <datastore>
  • Grow a VMDK to a new total size (VM off)vmkfstools -X 100G /vmfs/volumes/<datastore>/<vm>/<vm>.vmdk
  • Clone a disk as thinvmkfstools -i <source>.vmdk -d thin <dest>.vmdk
  • Return zeroed blocks of a thin disk (VM off)vmkfstools -K <disk>.vmdk
  • Check a VMDK for consistencyvmkfstools -x check <disk>.vmdk
  • “File is locked”: which host holds the lock (MAC in the output)vmkfstools -D /vmfs/volumes/<datastore>/<vm>/<file>

ESXi: network

  • VMkernel interfaces with IPsesxcli network ip interface ipv4 get
  • Physical NICs: link, speed, driveresxcli network nic list
  • Driver and firmware of one NICesxcli network nic get -n <vmnic>
  • Standard vSwitches and uplinksesxcli network vswitch standard list
  • Port groups and VLAN IDsesxcli network vswitch standard portgroup list
  • Set the VLAN on a port groupesxcli network vswitch standard portgroup set -p "<portgroup>" -v <vlan-id>
  • Ping out a specific vmk, jumbo frames, no fragmentingvmkping -I <vmk> -d -s 8972 <ip>
  • Routes and neighbour tableesxcli network ip route ipv4 list && esxcli network ip neighbor list
  • Which uplink and port a VM uses: find its world IDesxcli network vm list
  • Then its ports, MAC and team uplinkesxcli network vm port list -w <world-id>
  • Capture on a vmk to a filepktcap-uw --vmk <vmk> -o /tmp/<file>.pcap
  • Capture on a physical uplinkpktcap-uw --uplink <vmnic> -o /tmp/<file>.pcap
  • Management IP lost: set vmk0 from the shellesxcli network ip interface ipv4 set -i vmk0 -t static -I <ip> -N <netmask> -g <gateway>
  • Firewall rulesetsesxcli network firewall ruleset list

ESXi: performance and patching

  • Live stats: c CPU, m memory, d/u/v disk, n networkesxtop
  • CPU contention: watch %RDY (>5% per vCPU) and %CSTPesxtop # then press c
  • Disk latency: DAVG is the array, KAVG the hostesxtop # then press u
  • Record 5 minutes for later analysisesxtop -b -d 5 -n 60 > /tmp/esxtop.csv
  • Installed VIBsesxcli software vib list
  • Image profiles inside an offline depotesxcli software sources profile list -d /vmfs/volumes/<datastore>/<depot>.zip
  • Preview an updateesxcli software profile update -d /vmfs/volumes/<datastore>/<depot>.zip -p <profile> --dry-run
  • Apply it (maintenance mode, then reboot)esxcli software profile update -d /vmfs/volumes/<datastore>/<depot>.zip -p <profile>
  • Install a driver or component VIBesxcli software vib install -d /vmfs/volumes/<datastore>/<bundle>.zip
  • Allow community-supported driversesxcli software acceptance set --level=CommunitySupported

ESXi: PowerCLI

  • Connect to vCenter or a hostConnect-VIServer -Server <server>
  • VMs with power state and sizeGet-VM | Select-Object Name, PowerState, NumCpu, MemoryGB | Sort-Object Name
  • Snapshots older than a weekGet-VM | Get-Snapshot | Where-Object { $_.Created -lt (Get-Date).AddDays(-7) } | Select-Object VM, Name, Created, SizeGB
  • VMs whose disks need consolidationGet-VM | Where-Object { $_.ExtensionData.Runtime.ConsolidationNeeded } | Select-Object Name
  • Start SSH on every hostGet-VMHost | Get-VMHostService | Where-Object Key -eq "TSM-SSH" | Start-VMHostService

ESXi to Proxmox

  • Add the ESXi host as an import source on Proxmoxpvesm add esxi <storage-id> --server <esxi-host> --username root --password '<password>' --skip-cert-verification 1
  • Before exporting: remove every snapshotvim-cmd vmsvc/snapshot.removeall <vmid>
  • Import a copied VMDK into a Proxmox VMqm disk import <vmid> <vm>.vmdk <storage> --format raw
  • Guest NIC renamed after the move (ens192 becomes ens18)ip -br link

Containers

Docker

  • Running, with status and portsdocker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'
  • Follow logsdocker logs -f --tail 100 <container>
  • Throwaway containerdocker run --rm -it <image> sh
  • Why it exiteddocker inspect -f '{{.State.ExitCode}} oom={{.State.OOMKilled}} {{.State.Error}}' <container>
  • Restart loops and other events, last 10 minutesdocker events --since 10m
  • Container IPsdocker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}' <container>
  • CPU and memory snapshotdocker stats --no-stream
  • Log file on disk (the one filling /var)docker inspect -f '{{.LogPath}}' <container>
  • Writable layer size per containerdocker ps -s
  • Files changed inside a containerdocker diff <container>
  • Copy a file outdocker cp <container>:<path> <dest>
  • Change restart policydocker update --restart unless-stopped <container>
  • Host PID of every containerdocker ps -q | xargs docker inspect -f '{{.State.Pid}} {{.Name}}'
  • Run host tools inside a container’s network namespacensenter -t "$(docker inspect -f '{{.State.Pid}}' <container>)" -n ss -tlnp

Docker Compose

  • Update images and recreatedocker compose pull && docker compose up -d
  • Rebuild and recreatedocker compose up -d --build
  • Recreate one service even if unchangeddocker compose up -d --force-recreate <service>
  • Follow one servicedocker compose logs -f <service>
  • Final config after env files and mergesdocker compose config
  • Every compose project on the hostdocker compose ls
  • Tear down, volumes too (destroys data)docker compose down -v

Docker: images and cleanup

  • Disk used by images, containers, volumes, build cachedocker system df -v
  • Remove everything unused except volumesdocker system prune -a
  • Build cache (often the biggest)docker builder prune
  • Unused images older than a weekdocker image prune -a --filter "until=168h"
  • Remove unused anonymous volumesdocker volume prune
  • Multi-arch build and pushdocker buildx build --platform linux/amd64,linux/arm64 -t <name>:<tag> --push .
  • Architectures and digest of a remote imagedocker buildx imagetools inspect <image>
  • Layers and their sizesdocker history <image>
  • Export an image for an offline hostdocker save <image> | gzip > <image>.tar.gz
  • Import itdocker load -i <image>.tar.gz

Docker: daemon, networks, volumes

  • Networks and their membersdocker network inspect <network>
  • Debug a container’s network with a full toolboxdocker run --rm -it --network container:<container> nicolaka/netshoot
  • Publish on localhost only (published ports bypass ufw)docker run -d -p 127.0.0.1:<host-port>:<container-port> <image>
  • Back up a named volumedocker run --rm -v <volume>:/data -v "$PWD":/backup alpine tar -czf /backup/<volume>.tar.gz -C /data .
  • Cap container logs (/etc/docker/daemon.json){ "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3" } }
  • Keep Docker networks off your LAN ranges (daemon.json){ "default-address-pools": [{ "base": "10.200.0.0/16", "size": 24 }] }

Kubernetes: inspect

  • Switch clusterkubectl config use-context <context>
  • Default namespace for this contextkubectl config set-context --current --namespace=<ns>
  • Field docs for any resourcekubectl explain <resource>.spec
  • Pods not running, cluster-widekubectl get pods -A --field-selector=status.phase!=Running
  • Why a pod is stuckkubectl describe pod <pod>
  • Events, newest lastkubectl get events -A --sort-by=.metadata.creationTimestamp
  • Warnings onlykubectl get events -A --field-selector type=Warning --sort-by=.metadata.creationTimestamp
  • Events for one objectkubectl get events -n <ns> --field-selector involvedObject.name=<name>
  • Pods by restart countkubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount'
  • Why the last container died (OOMKilled, exit code)kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].lastState}'
  • Node usagekubectl top nodes
  • Heaviest podskubectl top pods -A --sort-by=memory
  • Requests and limits against what a node haskubectl describe node <node> | grep -A 10 'Allocated resources'
  • Service with no endpoints: selector matches nothingkubectl get endpointslices -l kubernetes.io/service-name=<service>
  • Decode a secret keykubectl get secret <name> -o jsonpath='{.data.<key>}' | base64 -d
  • Decode every key of a secretkubectl get secret <name> -o json | jq '.data | map_values(@base64d)'
  • Every image runningkubectl get pods -A -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}' | sort -u
  • API server health, check by checkkubectl get --raw='/readyz?verbose'

Kubernetes: debug

  • Follow a container’s logskubectl logs -f <pod> -c <container>
  • Logs from before the crashkubectl logs --previous <pod>
  • Logs across a labelkubectl logs -l app=<app> --all-containers --prefix --since=1h
  • Shell insidekubectl exec -it <pod> -- sh
  • Debug container in a distroless podkubectl debug -it <pod> --image=busybox --target=<container>
  • Shell on a node, host filesystem at /hostkubectl debug node/<node> -it --image=busybox
  • Forward a service locallykubectl port-forward svc/<service> 8080:<port>
  • Cluster DNS works?kubectl run dns-test --rm -it --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default
  • Copy a file out of a podkubectl cp <ns>/<pod>:<path> <dest>
  • What a service account can dokubectl auth can-i --list --as=system:serviceaccount:<ns>:<sa>
  • Containers on a node without the APIcrictl ps -a

Kubernetes: workloads and nodes

  • Restart pods of a deploymentkubectl rollout restart deploy/<name>
  • Wait for a rolloutkubectl rollout status deploy/<name>
  • Undo the last rolloutkubectl rollout undo deploy/<name>
  • Scale to zerokubectl scale deploy/<name> --replicas=0
  • See what apply would changekubectl diff -f <file>.yaml
  • YAML skeleton without creating anythingkubectl create deploy <name> --image=<image> --dry-run=client -o yaml
  • Drain for maintenancekubectl drain <node> --ignore-daemonsets --delete-emptydir-data
  • Back into servicekubectl uncordon <node>
  • Pods on one nodekubectl get pods -A -o wide --field-selector spec.nodeName=<node>

Kubernetes: stuck resources

  • Namespace stuck Terminating: what is left in itkubectl api-resources --verbs=list --namespaced -o name | xargs -n1 kubectl get -n <ns> --ignore-not-found --show-kind
  • Resource held by finalizerskubectl patch <kind> <name> -n <ns> --type=merge -p '{"metadata":{"finalizers":null}}'
  • Pod stuck Terminating on a dead nodekubectl delete pod <pod> --grace-period=0 --force
  • Volume will not attach: stale attachmentskubectl get volumeattachment
  • Clear out evicted and failed podskubectl delete pods -A --field-selector=status.phase=Failed
  • kubeadm: certificate expirykubeadm certs check-expiration

Helm

  • Add and refresh a repohelm repo add <repo> <url> && helm repo update
  • Chart’s default valueshelm show values <repo>/<chart>
  • Install or upgradehelm upgrade --install <release> <repo>/<chart> -n <ns> --create-namespace -f values.yaml
  • Render manifests locallyhelm template <release> <repo>/<chart> -f values.yaml
  • Releases stuck pending or failedhelm list -A --pending --failed
  • Values a release was deployed withhelm get values <release> -n <ns>
  • Manifests a release actually appliedhelm get manifest <release> -n <ns>
  • Revisions of a releasehelm history <release> -n <ns>
  • Roll back (also unsticks pending-upgrade)helm rollback <release> <revision> -n <ns>

Flux

  • Controllers healthyflux check
  • Everything Flux managesflux get all -A
  • Only what is failingflux get all -A --status-selector ready=false
  • Pull git and apply nowflux reconcile kustomization flux-system --with-source
  • Reconcile one Helm releaseflux reconcile helmrelease <name> -n <ns> --with-source
  • Local changes against the cluster, before pushingflux diff kustomization <name> --path <path>
  • Pause GitOps for manual workflux suspend kustomization <name>
  • Resume itflux resume kustomization <name>
  • Controller errorsflux logs -A --level=error

Talos

  • Cluster health from one nodetalosctl -n <ip> health
  • Live node dashboardtalosctl -n <ip> dashboard
  • Services and their statetalosctl -n <ip> services
  • Kubelet logstalosctl -n <ip> logs kubelet
  • Kernel logtalosctl -n <ip> dmesg
  • Pod containers on a nodetalosctl -n <ip> containers -k
  • Disks Talos seestalosctl -n <ip> get disks
  • etcd memberstalosctl -n <ip> etcd members
  • etcd health and leadertalosctl -n <ip> etcd status
  • Snapshot etcdtalosctl -n <ip> etcd snapshot <file>.snapshot
  • Apply a changed machine configtalosctl -n <ip> apply-config -f <node>.yaml
  • Patch the machine configtalosctl -n <ip> patch mc --patch @<patch>.yaml
  • Upgrade Talos on a nodetalosctl -n <ip> upgrade --image <installer-image>:<version>
  • Preview a Kubernetes upgradetalosctl -n <ip> upgrade-k8s --to <version> --dry-run
  • Upgrade Kubernetestalosctl -n <ip> upgrade-k8s --to <version>
  • Fetch kubeconfigtalosctl -n <ip> kubeconfig