How It Works

Observer classifies log lines through a multi-stage pipeline. Deterministic rules resolve what they can — cheaply, instantly, auditably. The LLM only handles what the rules can't. Evidence from the server's actual response decides whether anything becomes an email.

This page walks through each layer in the order a log line traverses them.

The pipeline at a glance

Log line arrives (Docker container or journald)
  ├─▶ Deterministic suppression (stack traces, failed HTTP probes, SSH brute force)
  │
  ├─▶ Policy Engine (SSH logins, user creation, privilege grants, key changes)
  │
  ├─▶ Seed Patterns (credential dumps, reverse shells, private keys, RCE chains)
  │
  ├─▶ Pattern Store (4-tier cache: exact hash → prefix → regex → substring)
  │     ├─ Hit: resolve silently
  │     └─ Miss: forward to LLM
  │
  └─▶ LLM Classification (novel patterns, intent × outcome reasoning)

If the verdict is alert-worthy:
  ├─▶ Forensic Coordinator holds the alert
  ├─▶ REC captures the HTTP response from inside the container namespace
  ├─▶ Re-classify with evidence
  │     ├─ Downgrade (attack failed): log, don't email
  │     ├─ Confirm (attack succeeded): email immediately
  │     └─ Non-HTTP malicious: email immediately, no evidence needed
  └─▶ Evidence unavailable after reconciler timeout: email as "evidence unavailable"

Every stage is designed to resolve events as early as possible. More than 97% of events in production never reach the LLM — they're handled by deterministic rules or cache hits in nanoseconds.

Layer 0 — Collection

Observer attaches to two sources:

• Docker socket API (/var/run/docker.sock) — streams stdout/stderr from every running container. Auto-discovers new containers; handles start/stop events.
• journald — streams system events via journalctl --follow --output=json. Catches sshd, sudo, systemd, kernel, and any service that logs through the journal.

Every line becomes a structured event with source type, identifier, timestamp, and metadata. No agents, no sidecars, no daemons required on monitored containers.

Observer's own log output is filtered via _SYSTEMD_UNIT=observer.service (kernel-attached, spoof-proof) to prevent self-observation loops. SYSLOG_IDENTIFIER is deliberately not used for this — it can be forged by arbitrary log writers.

Layer 1 — Deterministic Suppression

Before anything else, Observer checks structural facts that could never be an attack:

• Application stack traces (Node.js, Python, Go, Java)
• Failed HTTP probes — 404/403/405/400 with no attack payload
• SSH brute force — thousands per day on every public server, recognized structurally
• nginx file-not-found errors
• Firewall blocks (UFW, iptables)

These are dropped without cost. Zero LLM calls, zero pattern store writes, zero emails. They are not "suppressed alerts" — they never become events.

Layer 2 — Policy Engine

Host identity events never go to the LLM. An SSH login is either from a trusted IP or it isn't — that's a deterministic decision. The Policy Engine runs first for any journald event that matches a known identity rule:

Trusted IPs support exact addresses and CIDR ranges. Matching happens against the incoming connection's source IP — IPv4 and IPv6 both supported.

Why this layer is deterministic: the LLM can't know whether a given IP belongs to you. Policy is identity, not inference. The LLM would be asked to guess; the Policy Engine doesn't guess.

Layer 3 — Seed Patterns

Certain content, if it appears in any log stream, is always exploitation. These are seeded into the pattern store at startup and checked before the cache lookup:

Seed patterns exist because these signals are never ambiguous. No legitimate application prints /etc/passwd contents to stderr. No legitimate service prints private keys. If these appear, exploitation has already occurred — waiting for the LLM to confirm is wasted time.

Seed matches dispatch alerts directly without waiting for HTTP response evidence. Evidence from the server can't change a credential dump that already happened.

Layer 4 — Pattern Store

The pattern store is a four-tier cache:

1. Exact hash — SHA-256 of the normalized log line. O(1) lookup.
2. Prefix — first N tokens of the line. Catches variant paths, differing timestamps, changing IPs.
3. Regex — full regular expression match. Used sparingly, only for patterns the LLM flagged as structurally consistent.
4. Substring — substring presence anywhere in the line. Last-resort tier.

Lookups walk the tiers in order. A hit at any tier short-circuits the rest — the verdict is known, Observer acts on it, the pipeline moves on. Misses fall through to the LLM.

Known-good patterns are allowed silently. Known-noise patterns are suppressed silently. Known-bad patterns trigger the coordinator. The same pattern can have different verdicts on different sources — root:x:0:0:root in nginx output is exploitation; in a container that intentionally logs /etc/passwd for debugging it could be legitimate.

Pattern store hit rate in production: 97.5%.

Layer 5 — LLM Classification

Cache misses go to the LLM with an intent-and-outcome framing:

• Intent — what is the attacker trying to do? SQL injection? Path traversal? RCE?
• Outcome — did the target respond in a way that suggests success?

The LLM returns a structured verdict: allow, malicious, alert, or suppress, plus a confidence score, a human-readable reason, and a pattern recommendation if the line is reusable. The pattern is added to the store — the next time a similar line arrives, it's a cache hit.

Failed probes (404 responses, recon without a real target) are classified as recon_failed and logged to SQLite without an email. Successful attacks (200 responses with sensitive data, RCE output) are classified as malicious and forwarded to the coordinator.

Cost: ~$0.00003 per classification with gpt-5-nano. 517 classifications in 174,000+ events across production servers = approximately $17 lifetime LLM spend.

Layer 6 — Capture Evidence

When a log line suggests an attack on an HTTP service, the verdict is preliminary until Observer knows what the server actually returned. The Response Evidence Capture (REC) pipeline sniffs HTTP responses inside the target container's network namespace using AF_PACKET sockets.

For each alert-worthy request, REC produces:

• Status code — 200, 404, 500, etc.
• Body hash — SHA-256 of the response body (not the body itself — structural redaction)
• Content-Type — text/html, application/json, etc.
• Body size — after normalization

Observer never stores the raw response body. It stores the hash and structural metadata, which is enough to identify "this is the catch-all 404 page" or "this is a unique response not seen before" without retaining sensitive data.

Layer 7 — Verify Outcomes

The Forensic Coordinator holds alert-worthy events for a short window (typically 2–5 seconds) waiting for evidence. When REC delivers the response, the coordinator re-classifies:

• Default page / login redirect / error template → downgrade. Log to SQLite, no email.
• Catch-all fingerprint match (verified via body hash) → suppress. Log to SQLite, no email.
• Unique sensitive response (secrets, database contents, successful RCE output) → escalate. Email immediately.
• Evidence never arrived (reconciler timeout after 15 minutes) → escalate as "evidence unavailable." Honest, not silent.

Non-HTTP malicious findings — credential dumps, private keys, policy engine events — skip evidence waiting and dispatch directly. There's nothing for REC to verify; the log line itself is the evidence.

Layer 8 — Alert or Silence

The four possible terminal states:

Only malicious and confirmed alert produce email. Everything else is persisted to the findings database for trend analysis and dashboard display, without waking anyone up.

Adversarial hardening

Observer is built to stay correct under conditions attackers will actively create — log storms, DDoS traffic, evidence flooding:

• VIP lane for malicious evidence — when the coordinator holds a malicious finding, its evidence request gets priority buffer space that can't be evicted by normal traffic.
• Body hash not body size — catch-all fingerprinting uses SHA-256 of response bodies, not byte counts. An attacker padding their malicious response to match a known catch-all size won't collide.
• Async SQLite writes — all findings, evidence, and pattern updates are written through a batched writer (5000-entry buffer). A log storm can't stall the pipeline.
• 15-minute evidence reconciler — if REC doesn't deliver evidence within the coordinator window, the reconciler sweeps for late arrivals. Terminal states prevent findings from hanging indefinitely.

The pipeline is designed so an attacker can degrade performance but cannot degrade correctness. Every event ends up in a terminal state with an auditable trace.