Security Logs Analysis for DevOps

Master Linux security logs to detect attacks, troubleshoot issues, and maintain system security. This DevOps guide covers /var/log/auth.log, /var/log/secure, practical analysis commands, and automated monitoring patterns.

Why Log Analysis Matters for DevOps

Security logs are your first line of defense. They record every authentication attempt, privilege change, and system event. In DevOps, logs help you:

Essential Security Log Files

Understanding Log Format

Essential Log Analysis Commands

1. Basic Log Viewing

# View entire log file
sudo cat /var/log/auth.log
sudo less /var/log/secure

# View last 100 lines
sudo tail -100 /var/log/auth.log

# Follow log in real-time (CTRL+C to stop)
sudo tail -f /var/log/auth.log

# View with timestamps
sudo grep -E 'Dec  7' /var/log/auth.log

# View specific time range
sudo grep 'Dec  7 10:' /var/log/auth.log

# View compressed logs
sudo zcat /var/log/auth.log.1.gz
sudo zgrep 'Failed' /var/log/auth.log.*.gz

2. SSH Authentication Analysis

# Count failed SSH attempts
sudo grep "Failed password" /var/log/auth.log | wc -l

# Show failed attempts with IPs
sudo grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn

# Show successful logins
sudo grep "Accepted" /var/log/auth.log

# Show invalid users
sudo grep "invalid user" /var/log/auth.log | awk '{print $8,$10,$12}' | sort | uniq -c

# Show root login attempts
sudo grep "root" /var/log/auth.log | grep -E "Failed|Accepted"

# Check for brute force patterns
sudo grep "Failed password" /var/log/auth.log | grep -E "(max [0-9] attempts|Too many)"

# Find IPs with many failures
sudo grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -20

3. Sudo & Privilege Analysis

# Show all sudo usage
sudo grep "sudo:" /var/log/auth.log

# Show specific user's sudo commands
sudo grep "sudo.*alice" /var/log/auth.log

# Show failed sudo attempts
sudo grep "sudo.*FAILED" /var/log/auth.log

# Show commands executed as root
sudo grep "COMMAND" /var/log/auth.log | awk -F 'COMMAND=' '{print $2}'

# Find unusual sudo patterns
sudo grep -E "sudo.*(su|bash|sh|python|perl)" /var/log/auth.log

# Track user privilege escalation
sudo grep -E "(session opened|session closed)" /var/log/auth.log | grep "alice"

4. Advanced Pattern Matching

# Find all authentication failures
sudo grep -i "fail\|invalid\|refused\|denied\|error" /var/log/auth.log

# Find potential intrusion patterns
sudo grep -E "(reverse mapping|bad protocol|possible break-in)" /var/log/auth.log

# Check for port scans
sudo grep -E "connection reset|connection closed|Did not receive" /var/log/auth.log

# Find time gaps in logs (indicating log tampering)
sudo awk -F'[: ]' '{print $1,$2,$3}' /var/log/auth.log | uniq -c | awk '$1<10'

# Monitor for new users
sudo grep "new user" /var/log/auth.log

# Check for password changes
sudo grep "password changed\|password updated" /var/log/auth.log

Key Attack Patterns to Monitor

Automated Log Monitoring Scripts

1. Daily Security Report Script

#!/bin/bash
# daily-security-report.sh
# Generates daily security summary

REPORT_FILE="/var/log/security-report-$(date +%Y%m%d).txt"
LOG_FILE="/var/log/auth.log"

echo "=== SECURITY REPORT $(date) ===" > $REPORT_FILE
echo "" >> $REPORT_FILE

# 1. Failed SSH attempts
echo "1. FAILED SSH ATTEMPTS:" >> $REPORT_FILE
FAILED_COUNT=$(grep "Failed password" $LOG_FILE 2>/dev/null | wc -l)
echo "   Total failed attempts: $FAILED_COUNT" >> $REPORT_FILE

# Top 10 attacking IPs
echo "   Top 10 attacking IPs:" >> $REPORT_FILE
grep "Failed password" $LOG_FILE 2>/dev/null | \
    awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -10 >> $REPORT_FILE

# 2. Successful logins
echo -e "\n2. SUCCESSFUL LOGINS:" >> $REPORT_FILE
SUCCESS_COUNT=$(grep "Accepted" $LOG_FILE 2>/dev/null | wc -l)
echo "   Total successful logins: $SUCCESS_COUNT" >> $REPORT_FILE

# Login summary by user
echo "   Logins by user:" >> $REPORT_FILE
grep "Accepted" $LOG_FILE 2>/dev/null | \
    awk '{print $9}' | sort | uniq -c | sort -rn >> $REPORT_FILE

# 3. Invalid users
echo -e "\n3. INVALID USER ATTEMPTS:" >> $REPORT_FILE
grep "invalid user" $LOG_FILE 2>/dev/null | \
    awk '{print $8,$10,$12}' | sort | uniq -c | sort -rn >> $REPORT_FILE

# 4. Sudo usage
echo -e "\n4. SUDO COMMANDS:" >> $REPORT_FILE
grep "sudo:" $LOG_FILE 2>/dev/null | grep "COMMAND" | \
    awk -F 'COMMAND=' '{print $2}' | sort | uniq -c | sort -rn | head -20 >> $REPORT_FILE

# 5. Root login attempts
echo -e "\n5. ROOT LOGIN ATTEMPTS:" >> $REPORT_FILE
grep "root" $LOG_FILE 2>/dev/null | grep -E "Failed|Accepted" >> $REPORT_FILE

# 6. Suspicious patterns
echo -e "\n6. SUSPICIOUS ACTIVITY:" >> $REPORT_FILE
grep -E "(reverse mapping|possible break-in|POSSIBLE BREAK-IN)" $LOG_FILE 2>/dev/null >> $REPORT_FILE

echo -e "\n=== REPORT COMPLETE ===" >> $REPORT_FILE
echo "Report saved to: $REPORT_FILE"

2. Real-time Attack Detector

#!/bin/bash
# attack-detector.sh
# Monitors auth.log in real-time for attacks

LOG_FILE="/var/log/auth.log"
ALERT_FILE="/var/log/security-alerts.log"
THRESHOLD=5  # Max failed attempts before alert

echo "=== Starting Attack Detector $(date) ===" >> $ALERT_FILE

# Monitor log file
tail -f $LOG_FILE | while read line
do
    # Check for failed SSH attempts
    if echo "$line" | grep -q "Failed password"; then
        IP=$(echo "$line" | awk '{print $(NF-3)}')
        USER=$(echo "$line" | awk '{print $9}')
        
        # Count failures for this IP
        FAIL_COUNT=$(grep "Failed password.*$IP" $LOG_FILE | wc -l)
        
        if [ $FAIL_COUNT -ge $THRESHOLD ]; then
            ALERT="[ALERT] Brute force detected: IP=$IP, User=$USER, Attempts=$FAIL_COUNT"
            echo "$(date): $ALERT" >> $ALERT_FILE
            echo "$ALERT"
            
            # Optional: Block IP with iptables
            # iptables -A INPUT -s $IP -j DROP
        fi
    fi
    
    # Check for invalid users
    if echo "$line" | grep -q "invalid user"; then
        IP=$(echo "$line" | awk '{print $(NF-3)}')
        USER=$(echo "$line" | awk '{print $8}')
        echo "$(date): [WARNING] Invalid user attempt: User=$USER, IP=$IP" >> $ALERT_FILE
    fi
    
    # Check for sudo failures
    if echo "$line" | grep -q "sudo.*FAILED"; then
        USER=$(echo "$line" | awk '{print $6}')
        echo "$(date): [WARNING] Sudo failure: User=$USER" >> $ALERT_FILE
    fi
    
    # Check for root login attempts
    if echo "$line" | grep -q "root.*Failed"; then
        IP=$(echo "$line" | awk '{print $(NF-3)}')
        echo "$(date): [CRITICAL] Failed root login: IP=$IP" >> $ALERT_FILE
    fi
done

3. Log Analysis & Cleanup Script

#!/bin/bash
# log-analyzer.sh
# Comprehensive log analysis and maintenance

LOG_DIR="/var/log"
REPORT_DIR="/root/log-analysis/$(date +%Y%m%d)"
mkdir -p $REPORT_DIR

echo "=== Log Analysis Report $(date) ==="

# 1. Disk usage by log files
echo -e "\n1. LOG FILE SIZES:"
find $LOG_DIR -name "*.log" -type f -exec du -h {} \; | sort -hr | head -20

# 2. Most active IPs (last 7 days)
echo -e "\n2. MOST ACTIVE IPS (last 7 days):"
for i in {0..6}; do
    DATE=$(date -d "$i days ago" +"%b %_d")
    echo "=== $DATE ==="
    grep "Failed password" /var/log/auth.log | grep "$DATE" | \
        awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -10
done > $REPORT_DIR/active-ips.txt

# 3. User login timeline
echo -e "\n3. USER LOGIN TIMELINE:"
grep "Accepted" /var/log/auth.log | \
    awk '{print $1" "$2" "$3" - "$9" from "$(NF-3)}' | \
    tail -50 > $REPORT_DIR/login-timeline.txt

# 4. Suspicious command patterns
echo -e "\n4. SUSPICIOUS COMMANDS IN SUDO:"
grep "sudo.*COMMAND" /var/log/auth.log | \
    grep -E "(wget|curl|nc|nmap|python -c|perl -e|bash -i|sh -i)" | \
    tee $REPORT_DIR/suspicious-commands.txt

# 5. Log rotation status
echo -e "\n5. LOG ROTATION STATUS:"
ls -la /var/log/*.gz | wc -l
echo "Oldest compressed log:"
ls -lt /var/log/*.gz | tail -1

# 6. Log file permissions check
echo -e "\n6. LOG FILE PERMISSIONS:"
find $LOG_DIR -name "*.log" -type f -exec ls -la {} \; | \
    awk '{print $1, $3, $4, $9}' | grep -v "root root" > $REPORT_DIR/permissions-check.txt

# 7. Cleanup old reports (keep 30 days)
find /root/log-analysis -type d -mtime +30 -exec rm -rf {} \; 2>/dev/null

echo -e "\nAnalysis complete. Reports saved to: $REPORT_DIR"

Log Rotation & Management

# View logrotate configuration
sudo cat /etc/logrotate.conf
sudo ls /etc/logrotate.d/

# Check auth.log rotation config
sudo cat /etc/logrotate.d/rsyslog

# Manual log rotation
sudo logrotate -f /etc/logrotate.conf
sudo logrotate -v /etc/logrotate.d/rsyslog

# Configure custom log rotation
sudo nano /etc/logrotate.d/myapp

# Example custom configuration
/var/log/myapp/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 640 root adm
    sharedscripts
    postrotate
        systemctl reload myapp > /dev/null 2>&1 || true
    endscript
}

# Check disk usage
sudo du -sh /var/log/
sudo journalctl --disk-usage

# Clean old journal logs
sudo journalctl --vacuum-time=30d  # Keep last 30 days
sudo journalctl --vacuum-size=500M # Keep under 500MB

Centralized Logging Setup

# On client server (sending logs)
sudo nano /etc/rsyslog.conf

# Add at bottom:
*.* @192.168.1.100:514  # UDP
*.* @@192.168.1.100:514 # TCP

# On log server (receiving logs)
sudo nano /etc/rsyslog.conf

# Enable reception:
module(load="imudp")
input(type="imudp" port="514")
module(load="imtcp")
input(type="imtcp" port="514")

# Create separate directory for each host
$template RemoteLogs,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs

# Restart rsyslog
sudo systemctl restart rsyslog

Monitoring Checklist

Useful One-Liners for DevOps

# Count events per hour
sudo grep "Failed password" /var/log/auth.log | awk '{print $3}' | cut -d: -f1 | sort | uniq -c

# Find IPs with more than 10 failed attempts
sudo grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | awk '$1 > 10'

# Monitor real-time with color coding
sudo tail -f /var/log/auth.log | grep --color=always -E "Failed|Accepted|invalid"

# Generate summary by date
sudo awk '{print $1" "$2}' /var/log/auth.log | uniq -c | tail -30

# Find most active hours for attacks
sudo grep "Failed password" /var/log/auth.log | awk '{print $3}' | cut -d: -f1 | sort | uniq -c | sort -rn

# Check for SSH key issues
sudo grep -E "(Authentication refused|Bad protocol)" /var/log/auth.log

# Find users who logged in from multiple IPs
sudo grep "Accepted" /var/log/auth.log | awk '{print $9,$(NF-3)}' | sort | uniq | awk '{print $1}' | uniq -c | awk '$1 > 1'

# Quick security audit
echo "=== Quick Security Audit ==="; \
echo "Failed attempts:" $(sudo grep -c "Failed" /var/log/auth.log); \
echo "Invalid users:" $(sudo grep -c "invalid user" /var/log/auth.log); \
echo "Root attempts:" $(sudo grep -c "root.*Failed" /var/log/auth.log)