Commands

The ones I reach for when a machine misbehaves, and the ones I would otherwise look up for the tenth time. Click any command to copy it.

Triage

First minute on a sick box

Uptime and load averages
Kernel errors and warnings
Failed systemd units
Errors logged since boot
Pressure stall: is CPU, memory or I/O actually contended
Run queue, swapping, CPU split
Disk latency and saturation
Network throughput per interface
TCP retransmits per second
Load per CPU core
CPU and disk I/O per process
Top memory consumers
Top CPU consumers
Processes stuck in D state (hung on I/O or NFS)
Dump stacks of blocked tasks to the kernel log
OOM killer activity
Machine check and hardware errors
Recent reboots and shutdowns
Load earlier today, from sysstat history (Debian path)

Memory

Memory and swap
Available memory, dirty pages, kernel slab
Kernel caches by size (memory gone but no process owns it)
Processes using the most swap

Limits that fail quietly

System-wide file handles: used, free, max
Open files of a process against its limit
Conntrack table full (drops new connections)
Threads in use against the PID limit
Clock offset (breaks TLS, Kerberos, etcd)
Time, timezone and NTP sync state

Out of space

Usage per real filesystem
Inodes exhausted (df shows space, writes still fail)
Largest directories on one filesystem
Interactive disk usage browser
Files over 1 GB
Deleted files still held open
Free a deleted-but-open file without restarting its process
Empty a live log without breaking the writer
df and du disagree: look under the mountpoints
ext4: shrink the 5% root reserve on a data disk
Shrink the systemd journal

Windows: first look

Last boot time
Why it rebooted: 41 unexpected, 1074 initiated, 6008 dirty shutdown
Blue screens: bugchecks on record
Critical and error events, last 24 hours
CPU, free memory and disk latency, live (English counter names)
Processes with the most CPU time
Processes with the most memory
Automatic services that are not running
Free space per volume
Reboot pending (servicing or Windows Update)

System

systemd

Status and last log lines
Start now and at boot
Reload after editing unit files
Override a unit without touching the vendor file
Unit file as systemd sees it, drop-ins included
Effective limits and restart policy
After "start request repeated too quickly"
Keep a unit from starting, even as a dependency
Check a unit file before loading it
Run one command with a memory and CPU cap
CPU, memory and I/O per cgroup, live
Timers with last and next run
Every user's crontab (jobs outside systemd timers)
What slowed down boot

Logs

Follow one unit
Last hour, all units
Exact time window
Previous boot (what happened before the crash)
Boots on record, to pick one for -b
Kernel messages only
Everything sudo ran
Keep logs across reboots
Follow a plain log file across rotation

Processes

Find a process by name
Process tree
Files a process has open
Who is using a port
Environment of a running process
Binary and working directory of a process
Kernel stack of a hung process
Trace system calls
Only file and network syscalls, timestamped
Hottest functions, live
De-prioritise CPU and disk for a noisy job
Watch a command, highlighting what changed
Session that survives logout
Reattach to it

Files and text

Files changed in the last day
Delete files older than 30 days (run with -print first)
Top talkers in a log: count and rank one column
"Operation not permitted" as root: immutable bit
Compare one file across two hosts
Run a command on many hosts in parallel
Pack a directory
Unpack into a directory
List an archive without unpacking

Users and security

Change owner recursively
Directories 755, files 644
Add a user to a group (next login)
Show ACLs
Grant one user access via ACL
Default ACL so new files inherit group access
Password and account expiry (service account locked out)
SELinux denials, recent
Reset SELinux labels after moving files
Allow a service on a non-standard port under SELinux
Failed SSH logins today

Packages

Debian: installed and candidate version, and which repo
Debian: which package owns a file
Debian: finish an interrupted upgrade
Debian: pin a package version
Debian: reboot required, and by what
Debian: services still running old libraries
RHEL: which package owns a file
RHEL: which package provides a binary
RHEL: undo a transaction
RHEL: reboot required

Boot and rescue

Root is read-only in emergency mode
Chroot into a broken install from a live ISO
Kernel command line the system booted with
UEFI boot entries and order
Debian: rebuild every initramfs
RHEL: rebuild every initramfs
Debian: reinstall GRUB on a BIOS disk

Hardware

Hardware event log (PSU, DIMM, fan faults)
Sensors: temperatures, fans, voltages
BMC network settings from the host
BMC unresponsive: cold-reset it from the host
Serial number / Dell service tag
DIMMs per slot with size and speed
CPU model, sockets, cores, flags
ECC errors per memory controller
PCIe link trained below its capability
NUMA layout (pin VMs and IRQs to one node)
CPU temperatures and fans

NixOS: rebuild

Build and switch
Activate without a boot entry
Apply on next boot only (kernel, init changes)
Build only, change nothing
Deploy to a remote machine
Back to the previous generation
What changed since the last boot

NixOS: flakes and store

Update all inputs
Update one input
Evaluate every output before deploying
Evaluate an option
Explore the config in a REPL
System generations
Closure size of the running system
GC frees nothing: find what still roots it
Delete old generations and garbage
Deduplicate the store
Store corruption: verify and repair
Try a package without installing

Windows: repair and updates

Repair the component store first (needs Windows Update or a source)
Then repair system files
What sfc could not repair
Shrink WinSxS by removing superseded components
Recently installed updates
Find an update's package name to remove it
Remove it
Readable Windows Update log (written to the desktop)
Updates fail behind a proxy: the WinHTTP proxy setting
Online scan of a volume, no downtime

Windows: services and tasks

Service stuck in Stopping: its PID (sc.exe, since sc is an alias)
Then kill it
Why a service will not start: Service Control Manager events
Scheduled tasks whose last run failed (ignoring running and never-run)
Which process holds a file (Sysinternals Handle)

Windows: Active Directory

Replication summary across all DCs
Replication partners and last result for one DC
DC health across the forest, errors only
Which DC this machine authenticates against
FSMO role holders
"Trust relationship failed": repair the secure channel (Windows PowerShell 5.1, local admin)
Locked-out accounts
Which machine keeps locking an account (event 4740 on the PDC)
Unlock the account
Applied Group Policy, as a report
Reapply Group Policy now
Kerberos tickets: list, then purge after a group change

Storage

Disks

Block devices with filesystems and UUIDs
Model, serial, size, rotational, transport
Logical and physical sector size (pick ashift)
Stable device names
Partition table
Find a newly hot-added disk (SCSI host rescan)
See a disk's new size after growing it (device rescan)
Same for NVMe namespaces
Detach a disk cleanly before pulling it
Re-read the partition table without rebooting
Grow a partition into the new space
Find a physical drive: hold its activity LED on
Wipe filesystem, RAID and LVM signatures (destroys data)
Also clear GPT, including the backup at the end
"Device busy" on a wipe: leftover device-mapper entries

Filesystems and mounts

Grow ext4 online
Grow XFS online (takes the mountpoint, not the device)
Force a full ext4 check (unmounted)
Repair XFS (fsck.xfs does nothing)
Mount tree
fstab entry that will not hang boot if the disk is missing
Test fstab without rebooting
Who keeps a mount busy
Detach a busy mount
Why a filesystem went read-only
Trim all mounted SSDs

Disk health and performance

SMART verdict and attributes
SMART for a disk behind a PERC / MegaRAID
Start a long self-test
Self-test results
NVMe health and wear
Spin down a SATA disk now
Spin down a SAS disk now
Live I/O per process
I/O latency of a directory, like ping
Random 4k read/write benchmark

NFS and SMB

Server: apply /etc/exports changes
Exports a server offers
Mount NFS
Mount options actually negotiated
Stale NFS handle: drop the mount
List SMB shares
Mount SMB with a credentials file

LVM

Physical volumes, groups, logical volumes
Which disks back each LV
Thin pool data and metadata fill
Activate a VG after moving disks
Metadata backups (undo a bad change)
Initialise a physical volume
Create a volume group
Fixed-size logical volume
Logical volume from all free space
Thin pool

LVM: grow and shrink

Add a disk to a VG
PV after its disk was enlarged (rescan the disk first)
Grow LV and filesystem together
Grow into all free space
Shrink LV and ext4 (unmounted; XFS cannot shrink)
Grow a thin pool
Thin pool metadata filling up (full means corruption)
Evacuate a disk
Then drop it from the VG

LVM: snapshots and rescue

Snapshot an LV
Roll back to the snapshot
Drop the snapshot
Two VGs with the same name: rename one by UUID
Map partitions inside an LV (VM disk) to mount them

ZFS: pools

Only pools with problems
Files hit by permanent errors
Capacity per vdev
Mirror by stable disk IDs
ashift of an existing pool
Replace a failed disk
Use the space after replacing with bigger disks
Start a scrub
Reset error counters after fixing the cause
Latency per vdev, live
Who ran what against a pool
Old ZFS label blocking reuse of a disk
Find pools to import
Import by stable IDs
Import read-only for rescue

ZFS: datasets and snapshots

Usage per dataset
Where the space went: data, snapshots, children
Dataset with zstd compression
Compression ratio achieved
Only properties set by hand
Cap a dataset
Snapshot recursively
Snapshots, oldest first
Preview destroying a range of snapshots
Roll back (destroys later snapshots)
Full send to another host
Incremental send, one step
Incremental send with every snapshot in between

ZFS: ARC

ARC size and hit rate
Live ARC stats
Cap ARC at 8 GB now
Cap ARC at 8 GB across reboots

Windows: files and storage

Mirror a folder with permissions, logged (/MIR deletes extras; add /L to preview)
Largest files on a drive
NTFS permissions
Access denied even as admin: take ownership, reset ACLs to inherited
New disk shows Offline (SAN policy): bring it online and writable
Grew a VM disk: rescan, then extend the partition to the end
Backup fails: VSS writers in a failed state
Shadow copy storage per volume
File server: who has a file open
Close that handle
Share permissions (NTFS permissions apply on top)

Network

Interfaces and routes

Addresses, one line per interface
Which route and source IP a destination takes
Policy routing (Tailscale, VPNs, multi-WAN add rules)
ARP / neighbour table
Add an address until reboot
RX/TX errors and drops on an interface
Link speed and driver
NIC counters: CRC errors point at cable or optic
Bond state and active slave
Bridge members
VLANs on a VLAN-aware bridge
Someone else has this IP
Reverse-path filter dropping asymmetric traffic
Apply /etc/network/interfaces (Proxmox, ifupdown2)
NetworkManager: static IPv4 on a connection

Ports and connections

Listening sockets with processes
Socket totals
Established connections on a port
RTT, retransmits and window per connection
Is a remote port open
Talk to one backend, bypassing DNS and load balancer
HTTP status and timings
Certificate a server presents: subject, SANs, dates
Expiry of a certificate file

DNS

Just the answer
Ask a specific resolver
What the system resolver returns (/etc/hosts, nsswitch)
Follow delegation from the root
Reverse lookup
systemd-resolved state per link
Flush the local cache

Path and packets

Loss and latency per hop
Largest packet that fits: 1472 for 1500 MTU, 1252 for 1280
Watch traffic on a port
Capture on every interface, bridges included
Capture a host to a file for Wireshark
Throughput test: server
Throughput test: client, 4 streams
TCP retransmits and drops, system-wide
Bandwidth per connection

Firewall and NAT

nftables ruleset
iptables with counters
NAT rules (Docker and Kubernetes write here)
Tracked connections to a host
Drop tracked state after changing rules
ufw status
ufw: SSH from one subnet
firewalld zone rules
firewalld: open a port
Enable IPv4 forwarding now
IPv4 forwarding across reboots

Tailscale

Peers and connection type
Direct or relayed through DERP
NAT type and DERP latency
Advertise a subnet
Accept subnets from other routers
Route through an exit node

SSH

New key pair
Install a public key on a host
Key auth refused: fix permissions
Kill a frozen session (press Enter first)
Verbose handshake
Old switch, iDRAC or iLO rejects the key algorithm
Forget a changed host key
Trust a host key non-interactively
Fingerprint of a server key
Validate sshd_config before restarting
Effective sshd setting for a given user

SSH tunnels

Reach a remote-only port locally
Reach a web UI behind a bastion
Expose a local port on the remote
SOCKS proxy through a host
Hop through a bastion

Copy and transfer

Mirror a directory, preserving everything
Preview what rsync would change
Verify a copy by checksum: list files that differ
Push over SSH, compressed
Resumable, throttled copy over SSH
Stream a directory over SSH
Serve the current directory over HTTP

Windows: network

Addresses, gateway and DNS per interface
Is a port reachable
Trace the route
Resolve against a specific DNS server
Flush the DNS client cache
Listening ports with their process
Network marked Public blocks sharing and remoting: make it Private
Firewall state per profile
Allow an inbound port
Capture full packets with the built-in tool
Stop, and convert the capture for Wireshark
Time source and last sync
Clock offset against a domain controller
Force a time resync

Windows: remote management

Is WinRM answering
Run a command on several servers
Interactive remote session
Who is logged on (RDP sessions)
Log off a stuck session
Enable RDP and its firewall rule
Installed software (not Win32_Product, which triggers MSI repairs)
Installed roles and features

Virtualization

Proxmox: VMs

List VMs
Clean shutdown
Hard stop
Stop a VM that holds a lock
QEMU process ignoring stop: kill it
Clear a stale lock
Show config
Full KVM command line a VM starts with
Serial console
Grow a disk (then rescan it inside the guest)
Move a disk to another storage, live
Import a cloud image as a disk
Import an OVF export
Pick up orphaned disk images on storages
Snapshot
Roll back to a snapshot
Full clone of a template
Live migrate
IPs via guest agent
Mount a stopped VM's partition from a ZFS zvol

Proxmox: containers

List containers
Shell inside
Run one command
Grow the root disk
Bind-mount a host directory
Unprivileged bind mount: host UID = 100000 + container UID
Copy a file in
Mount a container's rootfs on the host (rescue)
Check a container's filesystem
Clear a stale lock

Proxmox: host, storage and backup

Versions of every PVE package
Core services
Web UI unreachable: restart its services
Cluster quorum
Lost quorum, /etc/pve read-only (last resort)
Regenerate node certificates
ESPs in sync on a ZFS or UEFI root
Upgrade (dist-upgrade; plain apt upgrade can break PVE)
Storage pools and usage
What is on a storage
Back up now
Restore a VM
Restore a container

KVM: passthrough and libvirt

CPU virtualization flags present
IOMMU enabled
Devices per IOMMU group
Driver bound to a device (vfio-pci or not)
Nested virtualization on (Intel)
libvirt domains
libvirt: guest IPs
Disks attached to a domain
Image format and size
Convert VMDK to qcow2

Hyper-V

VMs with state, uptime and memory
Force a VM off
VM stuck Stopping: kill its worker process
Checkpoints of a VM
Delete all checkpoints, merging them into the disk
Grow a VHDX (then extend the partition in the guest)
Virtual switches and the adapters behind them

ESXi: host and logs

Version and build
Enable and start SSH from the ESXi shell
Enter maintenance mode
Leave maintenance mode
Reboot (host must be in maintenance mode)
Host client or vCenter lost the host: restart management agents
Restart every management service (heavier)
Vendor, model and serial
NTP servers and state
Back up the host configuration (prints a download URL)
Support bundle for VMware / Broadcom
Storage, network and driver events
Host agent: VM operations, API calls
Hardware and connectivity observations
Why a VM failed to power on
Send logs to a remote syslog

ESXi: VMs and snapshots

Registered VMs with their IDs
Power state
Power on
Guest shutdown (needs VMware Tools)
Hard power off
VM will not power off: find its world ID
Then kill it: soft, then hard, then force
Guest IP from VMware Tools
Register a VM from its .vmx
Unregister (files stay on the datastore)
Reload a .vmx after editing it by hand
Stuck on a question ("moved or copied?"): list it
Answer it
Snapshot tree
Create a snapshot without memory
Delete all snapshots (consolidates into the base disk)
Delta disks left behind

ESXi: storage and virtual disks

Datastores with capacity and free space
See a new or grown LUN: rescan every adapter
Refresh VMFS volumes after the rescan
Devices with their naa IDs
Paths per device and their state
Multipathing policy per device
SMART data for a local disk
Mount an NFS v3 datastore
Copied or replicated LUN not mounting: VMFS snapshot volumes
Mount it keeping its signature
Reclaim freed blocks on thin-provisioned storage
Grow a VMDK to a new total size (VM off)
Clone a disk as thin
Return zeroed blocks of a thin disk (VM off)
Check a VMDK for consistency
"File is locked": which host holds the lock (MAC in the output)

ESXi: network

VMkernel interfaces with IPs
Physical NICs: link, speed, driver
Driver and firmware of one NIC
Standard vSwitches and uplinks
Port groups and VLAN IDs
Set the VLAN on a port group
Ping out a specific vmk, jumbo frames, no fragmenting
Routes and neighbour table
Which uplink and port a VM uses: find its world ID
Then its ports, MAC and team uplink
Capture on a vmk to a file
Capture on a physical uplink
Management IP lost: set vmk0 from the shell
Firewall rulesets

ESXi: performance and patching

Live stats: c CPU, m memory, d/u/v disk, n network
CPU contention: watch %RDY (>5% per vCPU) and %CSTP
Disk latency: DAVG is the array, KAVG the host
Record 5 minutes for later analysis
Installed VIBs
Image profiles inside an offline depot
Preview an update
Apply it (maintenance mode, then reboot)
Install a driver or component VIB
Allow community-supported drivers

ESXi: PowerCLI

Connect to vCenter or a host
VMs with power state and size
Snapshots older than a week
VMs whose disks need consolidation
Start SSH on every host

ESXi to Proxmox

Add the ESXi host as an import source on Proxmox
Before exporting: remove every snapshot
Import a copied VMDK into a Proxmox VM
Guest NIC renamed after the move (ens192 becomes ens18)

Containers

Docker

Running, with status and ports
Follow logs
Throwaway container
Why it exited
Restart loops and other events, last 10 minutes
Container IPs
CPU and memory snapshot
Log file on disk (the one filling /var)
Writable layer size per container
Files changed inside a container
Copy a file out
Change restart policy
Host PID of every container
Run host tools inside a container's network namespace

Docker Compose

Update images and recreate
Rebuild and recreate
Recreate one service even if unchanged
Follow one service
Final config after env files and merges
Every compose project on the host
Tear down, volumes too (destroys data)

Docker: images and cleanup

Disk used by images, containers, volumes, build cache
Remove everything unused except volumes
Build cache (often the biggest)
Unused images older than a week
Remove unused anonymous volumes
Multi-arch build and push
Architectures and digest of a remote image
Layers and their sizes
Export an image for an offline host
Import it

Docker: daemon, networks, volumes

Networks and their members
Debug a container's network with a full toolbox
Publish on localhost only (published ports bypass ufw)
Back up a named volume
Cap container logs (/etc/docker/daemon.json)
Keep Docker networks off your LAN ranges (daemon.json)

Kubernetes: inspect

Switch cluster
Default namespace for this context
Field docs for any resource
Pods not running, cluster-wide
Why a pod is stuck
Events, newest last
Warnings only
Events for one object
Pods by restart count
Why the last container died (OOMKilled, exit code)
Node usage
Heaviest pods
Requests and limits against what a node has
Service with no endpoints: selector matches nothing
Decode a secret key
Decode every key of a secret
Every image running
API server health, check by check

Kubernetes: debug

Follow a container's logs
Logs from before the crash
Logs across a label
Shell inside
Debug container in a distroless pod
Shell on a node, host filesystem at /host
Forward a service locally
Cluster DNS works?
Copy a file out of a pod
What a service account can do
Containers on a node without the API

Kubernetes: workloads and nodes

Restart pods of a deployment
Wait for a rollout
Undo the last rollout
Scale to zero
See what apply would change
YAML skeleton without creating anything
Drain for maintenance
Back into service
Pods on one node

Kubernetes: stuck resources

Namespace stuck Terminating: what is left in it
Resource held by finalizers
Pod stuck Terminating on a dead node
Volume will not attach: stale attachments
Clear out evicted and failed pods
kubeadm: certificate expiry

Helm

Add and refresh a repo
Chart's default values
Install or upgrade
Render manifests locally
Releases stuck pending or failed
Values a release was deployed with
Manifests a release actually applied
Revisions of a release
Roll back (also unsticks pending-upgrade)

Flux

Controllers healthy
Everything Flux manages
Only what is failing
Pull git and apply now
Reconcile one Helm release
Local changes against the cluster, before pushing
Pause GitOps for manual work
Resume it
Controller errors

Talos

Cluster health from one node
Live node dashboard
Services and their state
Kubelet logs
Kernel log
Pod containers on a node
Disks Talos sees
etcd members
etcd health and leader
Snapshot etcd
Apply a changed machine config
Patch the machine config
Upgrade Talos on a node
Preview a Kubernetes upgrade
Upgrade Kubernetes
Fetch kubeconfig