Complete Kernel Panics Guide: Linux Crash Analysis & Recovery

1. Understanding Kernel Panics

Kernel panics occur when the Linux kernel detects an unrecoverable error and halts the system to prevent data corruption. Understanding the types and causes is crucial for effective troubleshooting.

What is a Kernel Panic?

Beginner

Definition:

A kernel panic is a safety measure taken by the Linux kernel when it encounters an internal fatal error from which it cannot safely recover. The kernel intentionally crashes the system to prevent data corruption and further damage.

Why it happens:

Better to have a controlled crash than continue with corrupted memory, dangling pointers, or inconsistent state that could lead to silent data corruption or security vulnerabilities.

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.15.0-76-generic #83-Ubuntu

Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.module+el8.8.0+13338+2031a3c7 04/01/2014

Call Trace:

<TASK>

dump_stack_lvl+0x5d/0x70

panic+0x101/0x2e7

mount_block_root+0x23f/0x2e0

---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---

Common Kernel Panic Causes

Intermediate

Software Causes:

• Kernel bugs: Undiscovered issues in kernel code
• Driver failures: Buggy or incompatible hardware drivers
• Module issues: Kernel modules with memory corruption
• Filesystem errors: Corrupted filesystem metadata
• Memory corruption: Buffer overflows, use-after-free
• Initramfs problems: Missing root filesystem drivers

Hardware Causes:

• Memory failures: Bad RAM modules or corruption
• CPU issues: Overheating, overclocking failures
• Storage problems: Disk errors, SSD firmware bugs
• Power issues: Voltage fluctuations, PSU failures
• Motherboard: Chipset bugs, BIOS/UEFI issues
• Peripheral devices: Faulty USB/PCIe devices

Critical Warning Signs

These symptoms often precede kernel panics:

2. Immediate Response & Diagnostics

When a kernel panic occurs, your immediate actions determine how quickly you can diagnose and recover. Follow these systematic steps.

Capturing Panic Information

Critical

What to document:

1. Exact error message: Every word on screen
2. Call trace: Complete stack trace if shown
3. Timing: When did it happen? (During boot, load, idle)
4. Recent changes: Kernel updates, driver installs, hardware changes
5. System state: What was running? Load average?
6. Photo/Video: Take pictures of the screen!

Kernel panic - not syncing: Fatal exception

CPU: 2 PID: 1234 Comm: kworker/2:1 Tainted: P W O 5.15.0-76-generic

RIP: 0010:radeon_fence_wait_timeout+0x12/0x90 [radeon]

RSP: 0018:ffffc900012b7d60 EFLAGS: 00010246

RAX: 0000000000000000 RBX: ffff888107c20000 RCX: 0000000000000000

Call Trace:

<IRQ>

radeon_gem_busy_ioctl+0x1b7/0x240 [radeon]

drm_ioctl_kernel+0xac/0x150 [drm]

drm_ioctl+0x1f2/0x3a0 [drm]

---[ end Kernel panic - not syncing: Fatal exception ]---

Recovery Boot Methods

Intermediate

Safe boot parameters:

At GRUB menu, press 'e' to edit kernel line and add:

# Basic recovery parameters init=/bin/bash # Drop to root shell single # Single user mode 1 # Runlevel 1 (single user) emergency # Emergency mode systemd.unit=rescue.target # Systemd rescue systemd.unit=emergency.target # Systemd emergency # Hardware/driver troubleshooting nomodeset # Disable graphics mode setting noapic # Disable APIC nolapic # Disable local APIC acpi=off # Disable ACPI pci=noacpi # Disable ACPI for PCI irqpoll # Force IRQ polling # Filesystem/mount options rootdelay=10 # Wait 10 seconds for root device rootfstype=ext4 # Specify filesystem type rw # Mount root read-write ro # Mount root read-only

Emergency Recovery Steps

After booting with init=/bin/bash or single:

# 1. Remount root as read-write mount -o remount,rw / # 2. Check filesystem health fsck -y /dev/sda1 # 3. Check kernel logs from previous boot journalctl -b -1 -k # Kernel messages from last boot journalctl -b -1 # All logs from last boot # 4. Check for hardware errors dmesg | grep -i "error\|fail\|corrupt" # 5. Remove problematic kernel modules rmmod radeon # Example: Remove buggy graphics driver rmmod nouveau # 6. Blacklist modules for future boots echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf

3. Kernel Debugging Tools & Techniques

Advanced tools for analyzing kernel crashes, collecting crash dumps, and debugging kernel issues.

dmesg - Kernel Ring Buffer

Beginner

dmesg -T -l emerg,alert,crit,err,warn

dmesg flags explained:

• -T: Human-readable timestamps
• -l: Filter by log level (emerg, alert, crit, err, warn)
• -H: Human-readable output
• -w: Watch/follow mode (like tail -f)
• --follow: Continue printing new messages
• --since: Show messages since time

Critical dmesg patterns:

journalctl for Kernel Messages

Intermediate

journalctl -k -b -1 -p err

Journalctl kernel options:

• -k: Kernel messages only
• -b -1: Previous boot (most recent before current)
• -b -2: Boot before previous
• -p err: Error priority and above
• --list-boots: Show all recorded boots
• --since "2 hours ago": Time-based filtering

Comprehensive Kernel Log Analysis

#!/bin/bash # kernel-log-analyzer.sh - Comprehensive kernel log analysis echo "=== KERNEL LOG ANALYSIS REPORT $(date) ===" echo "" # 1. Kernel version and boot information echo "1. SYSTEM INFORMATION:" uname -a echo "Uptime: $(uptime -p)" journalctl --list-boots | tail -5 echo "" # 2. Recent kernel errors echo "2. RECENT KERNEL ERRORS (last 24h):" journalctl -k --since "24 hours ago" -p err | tail -20 echo "" # 3. Hardware errors (MCE) echo "3. HARDWARE ERRORS:" journalctl -k --since "24 hours ago" | grep -i "mce\|machine check" | tail -10 echo "" # 4. Filesystem issues echo "4. FILESYSTEM ISSUES:" journalctl -k --since "24 hours ago" | grep -i "ext4\|xfs\|btrfs\|filesystem\|I/O error" | tail -10 echo "" # 5. Driver failures echo "5. DRIVER FAILURES:" journalctl -k --since "24 hours ago" | grep -i "driver\|module\|failed to load" | tail -10 echo "" # 6. Memory issues echo "6. MEMORY ISSUES:" journalctl -k --since "24 hours ago" | grep -i "memory\|page\|oom\|out of memory" | tail -10

kdump & crash Utility

Advanced

What is kdump?

kdump is a kernel crash dumping mechanism. When a panic occurs, kdump captures a complete memory dump (vmcore) which can be analyzed offline with the crash utility.

Setting up kdump:

# 1. Install kdump tools sudo apt install kdump-tools crash # Debian/Ubuntu sudo yum install kexec-tools crash # RHEL/CentOS # 2. Configure crashkernel memory # Edit /etc/default/grub and add: GRUB_CMDLINE_LINUX="crashkernel=256M" # 3. Update GRUB and reboot sudo update-grub sudo reboot # 4. Enable kdump service sudo systemctl enable kdump.service sudo systemctl start kdump.service # 5. Test kdump (trigger a panic) echo c > /proc/sysrq-trigger # 6. Analyze crash dump sudo crash /var/crash/$(date +%Y%m%d)/vmcore /usr/lib/debug/boot/vmlinux-$(uname -r)

Triggering Test Panics

WARNING: These commands will crash your system! Only use on test systems:

# Magic SysRq keys (must be enabled) echo 1 > /proc/sys/kernel/sysrq # Trigger kernel panic echo c > /proc/sysrq-trigger # Alternative methods # Load buggy module insmod /lib/modules/$(uname -r)/kernel/drivers/gpu/drm/radeon/radeon.ko # Force NULL pointer dereference echo 1 > /proc/sys/kernel/panic_on_oops echo 0 > /proc/sys/kernel/panic cat /dev/zero > /dev/mem # WARNING: Dangerous!

4. Common Panic Scenarios & Solutions

Real-world kernel panic scenarios with step-by-step diagnosis and resolution procedures.

Scenario: "Unable to mount root fs"

Critical

Error message:

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

Diagnosis steps:

Step 1: Check kernel command line

cat /proc/cmdline

Look for root= parameter. Is it correct? Step 2: Verify root device exists

ls -la /dev/sda* /dev/nvme* /dev/mmcblk*

Check if storage device is detected Step 3: Check initramfs contents

Are filesystem drivers present? Step 4: Verify filesystem

fsck -n /dev/sda1

Check for filesystem corruption

Recovery solutions:

# Solution 1: Boot with correct root parameter # At GRUB, edit kernel line and specify correct root: root=/dev/sda2 # or using UUID: root=UUID=1234-5678 # Solution 2: Rebuild initramfs with missing drivers # Boot from live USB, chroot, and rebuild: chroot /mnt update-initramfs -c -k $(uname -r) # Solution 3: Check/modify /etc/fstab # Ensure root device is correctly specified cat /etc/fstab # Solution 4: Filesystem repair # Boot from live USB and run: fsck -y /dev/sda2 # Solution 5: Check for LVM/RAID # Add appropriate kernel parameters: rd.lvm=1 rd.md=1

Scenario: "Fatal exception" with Call Trace

Advanced

Example error:

[ 1234.567890] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 [ 1234.567891] IP: radeon_fence_wait_timeout+0x12/0x90 [radeon] [ 1234.567892] PGD 0 P4D 0 [ 1234.567893] Oops: 0000 [#1] SMP PTI [ 1234.567894] CPU: 2 PID: 1234 Comm: kworker/2:1 Tainted: P W O 5.15.0-76-generic [ 1234.567895] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996) [ 1234.567896] Workqueue: events_long gpu_reset [radeon] [ 1234.567897] RIP: 0010:radeon_fence_wait_timeout+0x12/0x90 [radeon] [ 1234.567898] RSP: 0018:ffffc900012b7d60 EFLAGS: 00010246 [ 1234.567899] RAX: 0000000000000000 RBX: ffff888107c20000 RCX: 0000000000000000 [ 1234.567900] Call Trace: [ 1234.567901] radeon_gem_busy_ioctl+0x1b7/0x240 [radeon] [ 1234.567902] drm_ioctl_kernel+0xac/0x150 [drm] [ 1234.567903] drm_ioctl+0x1f2/0x3a0 [drm] [ 1234.567904] __x64_sys_ioctl+0x91/0xd0 [ 1234.567905] do_syscall_64+0x5c/0x90 [ 1234.567906] entry_SYSCALL_64_after_hwframe+0x72/0xdc

Analyzing the trace:

• NULL pointer dereference: Code tried to access memory at address 0x10
• Module involved: [radeon] - AMD graphics driver
• Function: radeon_fence_wait_timeout
• Caller: radeon_gem_busy_ioctl (GPU busy check)
• Context: Workqueue gpu_reset - GPU reset handler

Fixing driver-related panics:

# 1. Boot with nomodeset to disable problematic driver # Add to kernel line: nomodeset # 2. Blacklist the problematic driver echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf # 3. Remove/disable kernel module rmmod radeon # Or prevent loading: touch /etc/modprobe.d/disable-radeon.conf echo "install radeon /bin/false" >> /etc/modprobe.d/disable-radeon.conf # 4. Install newer/different driver version sudo apt install xserver-xorg-video-amdgpu # 5. Use basic framebuffer driver echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf echo "vesafb" >> /etc/initramfs-tools/modules

Scenario: Hardware-Related Panics

Hardware

Hardware panic indicators:

Hardware Diagnostic Commands

# 1. Memory testing (requires reboot) sudo apt install memtest86+ # Reboot and select memtest86+ from GRUB # 2. CPU stress test sudo apt install stress-ng stress-ng --cpu 0 --cpu-method matrixprod --timeout 300 # 3. Disk health check sudo apt install smartmontools sudo smartctl -a /dev/sda | grep -i "error\|fail\|reallocated" # 4. Temperature monitoring sudo apt install lm-sensors sudo sensors-detect sudo sensors # 5. PCI device diagnostics lspci -vvv lspci -t # 6. Interrupt monitoring cat /proc/interrupts | sort -rn

5. Prevention & Proactive Measures

Prevent kernel panics through proper configuration, monitoring, and maintenance practices.

Kernel Configuration Best Practices

Intermediate

Critical kernel parameters:

# Edit /etc/default/grub and add to GRUB_CMDLINE_LINUX # Enable early console output (crucial for debugging) earlyprintk=vga,keep # Increase log buffer size log_buf_len=16M # Print all messages regardless of log level ignore_loglevel # Panic on OOPS (makes bugs visible immediately) panic_on_oops=1 # Reboot automatically after panic (seconds) panic=10 # Enable kernel debugging debug # Memory management debugging mminit_loglevel=4 # Disable problematic features if needed nosoftlockup nmi_watchdog=0 nosmap nosmep

Update GRUB after changes:

# Debian/Ubuntu sudo update-grub # RHEL/CentOS/Fedora sudo grub2-mkconfig -o /boot/grub2/grub.cfg # Arch Linux sudo grub-mkconfig -o /boot/grub/grub.cfg # Verify the changes cat /proc/cmdline

Kernel Update Management

Beginner

Safe update strategy:

• Always keep previous kernel: Never auto-remove old kernels
• Test on non-critical systems first: Staging environment
• Check changelogs: Look for bug fixes relevant to your hardware
• Monitor kernel mailing lists: Known issues before updating
• Have rollback plan: Know how to boot old kernel

Kernel Management Commands

# List installed kernels dpkg --list | grep linux-image # Debian/Ubuntu rpm -qa | grep kernel # RHEL/CentOS # Check current kernel uname -r # List available kernels in GRUB grep ^menuentry /boot/grub/grub.cfg | cut -d "'" -f2 # Set default kernel to boot sudo grub-set-default "Ubuntu, with Linux 5.15.0-76-generic" # Remove old kernels (keep last 2) sudo apt autoremove --purge # Debian/Ubuntu sudo package-cleanup --oldkernels --count=2 # RHEL/CentOS # Install specific kernel version sudo apt install linux-image-5.15.0-76-generic sudo apt install linux-headers-5.15.0-76-generic

Monitoring & Alerting

Intermediate

Proactive monitoring setup:

#!/bin/bash # kernel-monitor.sh - Kernel panic early warning system LOG_FILE="/var/log/kernel-monitor.log" ALERT_EMAIL="admin@example.com" CHECK_INTERVAL=300 # 5 minutes while true; do # Check for kernel oops/errors OOPS_COUNT=$(dmesg -T -l emerg,alert,crit,err | tail -100 | grep -c "Oops\|BUG\|WARNING") if [ $OOPS_COUNT -gt 0 ]; then echo "ALERT: $OOPS_COUNT kernel errors detected $(date)" >> $LOG_FILE dmesg -T -l emerg,alert,crit,err | tail -20 >> $LOG_FILE # Send alert echo "Kernel errors detected on $(hostname)" | \ mail -s "KERNEL ALERT: $(hostname)" $ALERT_EMAIL fi # Check for hardware errors HARDWARE_ERRORS=$(dmesg | grep -c "mce\|machine check\|corrected error") if [ $HARDWARE_ERRORS -gt 10 ]; then echo "ALERT: Excessive hardware errors $(date)" >> $LOG_FILE fi sleep $CHECK_INTERVAL done

6. Advanced Debugging Techniques

Advanced techniques for kernel developers and system administrators dealing with complex crash scenarios.

Kernel Symbol Analysis

Advanced

Understanding kernel addresses:

Kernel addresses in stack traces need to be translated to function names using kernel symbols. This requires matching kernel debug symbols.

Installing Debug Symbols

# Debian/Ubuntu sudo apt install linux-image-$(uname -r)-dbgsym # Enable debug symbol repository first: echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" | \ sudo tee -a /etc/apt/sources.list.d/ddebs.list sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C8CAB6595FDFF622 # RHEL/CentOS sudo yum install kernel-debuginfo-$(uname -r) sudo debuginfo-install kernel # Find debug symbols find /usr/lib/debug -name "vmlinux*" # Use addr2line to translate addresses addr2line -e /usr/lib/debug/boot/vmlinux-$(uname -r) 0xffffffffc0123456

Live Kernel Debugging with KGDB

Advanced

KGDB setup:

KGDB allows debugging a live kernel over serial connection or network. Useful for debugging crashes that happen under specific conditions.

# 1. Configure kernel with KGDB support # Recompile kernel with: CONFIG_KGDB=y CONFIG_KGDB_SERIAL_CONSOLE=y CONFIG_KGDB_KDB=y # 2. Add kernel parameters kgdboc=ttyS0,115200 kgdbwait # 3. Connect from another machine # On debug host: sudo apt install gdb gdb /usr/lib/debug/boot/vmlinux-$(uname -r) (gdb) target remote /dev/ttyUSB0 (gdb) continue # 4. Trigger debug break # On target system: echo g > /proc/sysrq-trigger # 5. Common GDB commands for kernel (gdb) bt # Backtrace (gdb) info reg # Register contents (gdb) list # Source code around current location (gdb) x/10i $pc # Disassemble instructions

Kernel Panic Recovery Checklist

Step-by-Step Recovery Procedure:

Phase 1: Immediate Response
1. Document everything: Photo/video of screen
2. Note exact error: Every word of panic message
3. Record call trace: Complete stack trace if available
4. Check recent changes: Updates, new hardware, configuration changes

Phase 2: Safe Boot
5. Boot with recovery parameters: init=/bin/bash or single
6. Remount filesystem: mount -o remount,rw /
7. Check logs: journalctl -b -1 -k (previous boot)
8. Examine dmesg: dmesg | tail -100

Phase 3: Diagnosis
9. Identify pattern: Driver issue? Hardware? Filesystem?
10. Check for known bugs: Search error online
11. Test hardware: Memory, disk, CPU diagnostics
12. Isolate cause: Remove/disable suspected components

Phase 4: Recovery
13. Apply fix: Update, blacklist, patch, or replace
14. Test thoroughly: Stress test the fix
15. Document solution: Create recovery procedure
16. Implement prevention: Monitoring, alerts, backups

Pro Tips for Kernel Stability

• Always keep old kernel: Never auto-remove previous kernel versions
• Test updates first: Use staging environment for kernel updates
• Monitor hardware health: Regular diagnostics for memory, disk, CPU
• Use LTS kernels: Long-Term Support kernels are more stable
• Check vendor drivers: Hardware-specific drivers often better than generic
• Enable kdump: Essential for production systems
• Regular maintenance: Filesystem checks, log rotation, updates
• Document everything: Changes, crashes, solutions

Critical "Never Do This" List

1. Never ignore kernel warnings: They often precede panics
2. Never force unstable hardware: Bad RAM causes random crashes
3. Never use untested kernels: Mainline kernels may be unstable
4. Never disable all safety features: Keep panic_on_oops=1
5. Never skip backups: Always have recovery media ready
6. Never modify /proc/sys/kernel/panic: Set to 0 only for debugging
7. Never use experimental drivers: In production without testing
8. Never forget to test recovery: Practice restores regularly
9. Never assume "it won't happen": Plan for kernel panics
10. Never panic during panic: Stay calm and follow procedure