Automated Threat Detection Pipeline
Build a SIEM-style detection pipeline using Python and the ELK stack to ingest logs, detect anomalies, and trigger automated alerts.
01 Overview
SOC analysts spend most of their time investigating alerts — but the pipeline that generates those alerts is often opaque. In this mission you'll build a full threat detection pipeline from scratch: ingest web server logs and network traffic using Logstash, write detection rules using Sigma, correlate events in Elasticsearch, visualise them in Kibana, and write Python automation to enrich and escalate high-severity events.
02 Environment Setup
**System requirement:** Run `free -h` to confirm you have 8GB+ RAM available. If not, use a cloud VM (t3.large on AWS, or a $12/mo DigitalOcean droplet). **Set vm.max_map_count:** Elasticsearch requires `sudo sysctl -w vm.max_map_count=262144` (add to `/etc/sysctl.conf` to persist). **Clone the starter repo:** `git clone https://github.com/elastic/docker-elk && cd docker-elk && docker-compose up -d`. Kibana will be available at http://localhost:5601 (default creds: elastic / changeme).
03 Mission Tasks
Deploy Elasticsearch, Logstash, and Kibana using Docker Compose. Configure TLS and authentication between components.
Write Logstash pipelines to ingest Apache/Nginx access logs and Suricata alerts. Parse, normalise, and enrich events with GeoIP data.
Convert three real-world Sigma rules (brute-force, port scan, data exfiltration) into Elasticsearch queries. Test against sample log data.
Build a SOC dashboard showing: top source IPs, alert severity timeline, geo-map of sources, and anomaly trends.
Write a Python script that polls Elasticsearch for new high-severity alerts, enriches them via VirusTotal, and sends formatted notifications via webhook.
04 Why This Matters
SOC Analysts and Security Engineers who can build and tune detection pipelines command premium salaries. SIEM expertise is listed in virtually every senior security role.
You'll Need
- Docker and Docker Compose installed
- At least 8GB RAM (ELK is memory-hungry)
- Python 3.10+ with pip
- Basic understanding of log formats (JSON, syslog)
- Familiarity with Linux command line
Tools & Tech Stack
Skills You'll Gain
- SIEM operations
- ELK Stack
- Sigma rules
- Python scripting
- Log analysis
- Threat detection
More Missions
Linux Hardening Fundamentals
Secure a fresh Ubuntu server from scratch — configure SSH, set up firewalls, manage users, and eliminate attack surface.
Docker Security Lab
Scan container images for CVEs, build minimal secure images, and enforce runtime security policies across a multi-container stack.
Cloud Security Audit with AWS
Audit a misconfigured AWS environment — find exposed S3 buckets, overly permissive IAM roles, open security groups, and missing CloudTrail.