TandemLinux web server compromise followed by persistence and credential theft
Public-safe case study Customer names, domains, employee identifiers, exact addresses, hostnames, credentials, evidence locations, and case-specific identifiers have been removed or replaced. Quantitative findings and defensive lessons are retained for educational use.
Executive assessment
An internet-facing Ubuntu application server processed a crafted request that exploited a vulnerable upload component. The web worker spawned a shell, wrote a disguised PHP web shell, installed a systemd user service, read application environment files and SSH material, and sent an encrypted archive to a first-seen destination. The server was removed from the load balancer 8 minutes 37 seconds after confirmed command execution.
95% autonomous investigation. 100% human authority.
TandemTrace AI agents completed 38 of 40 recorded investigation actions autonomously. The incident commander retained final verdict approval and production isolation authorization.
Agents connected web exploitation to host impact. A human controlled production isolation.
Investigation dataset
- Alert
- EDR · web process spawning shell
- First observed
- 2026-07-31 18:42:16 UTC
- Host
- WEB-PRD-07 · Ubuntu 22.04 LTS
- Service
- Customer portal · nginx + PHP-FPM
- Exposure
- Internet-facing · production pool
- Endpoint
- 3,114 process/file events · ±45m
- Web
- 18,906 access/error rows · 2h
- Authentication
- 612 SSH and PAM events · 24h
- Network
- 29 DNS · 14 flow records
- Fleet
- 326 Linux hosts · 30d
| Artifact | Sanitized value | Prevalence | Assessment |
|---|---|---|---|
| Web shell | .cache.php | 1 / 326 hosts | Disguised in upload directory |
| Service | user-cache-sync.service | 1 host | Executes hidden ELF |
| Payload SHA-256 | ca17…44e2 | 1 host | Unsigned first-seen binary |
| Destination | 203.0.113[.]119:443 | 0 prior flows | TEST-NET replacement |
Evidence that changed the verdict
Web-to-shell ancestry
PHP-FPM spawned /bin/sh with a decoded command from the exploited request body; this never occurred in the 30-day service baseline.
Durable persistence
The www-data account created and enabled a user systemd service pointing to a hidden ELF under the application cache.
Credential collection
The process read .env, cloud configuration, deployment tokens, and an SSH private key before creating an encrypted archive.
Confirmed egress
Network flow records show 1.7 MB sent to a first-seen destination immediately after archive creation.
Correlated timeline
A crafted multipart request reaches the vulnerable upload route.
PHP-FPM launches a shell as www-data.
A hidden PHP file is created and invoked by a second request.
A user systemd service is enabled and started.
Four secret-bearing files are read and archived.
1.7 MB leaves the host over TLS.
Node removed from rotation and network quarantine applied.
Competing hypotheses
| Hypothesis | Evidence for | Evidence against | Outcome |
|---|---|---|---|
| Deployment or maintenance | Service account executed shell tools | No pipeline, ticket, package signature, or approved path | Rejected |
| Vulnerability scanner | Internet request targeted upload route | Successful shell, persistence, secret reads, and egress exceed scanning | Rejected |
| External compromise | Exploit-to-shell chain and post-exploitation actions | Exact actor and exploit family unresolved | Confirmed |
Scope and evidence boundary
| Question | Result | Meaning |
|---|---|---|
| Same request elsewhere? | 0 / 11 peer web nodes | Observed exploitation limited to one node |
| Same artifacts fleet-wide? | 0 / 326 hosts | No additional persistence found |
| Stolen SSH key used? | 0 matching successful logins | Reuse not observed through cutoff |
| Archive contents exfiltrated? | Transfer size consistent | Likely, but encrypted payload prevents content proof |
Evidence boundary The outbound transfer is confirmed and temporally linked, but packet contents were unavailable. Credential theft is confirmed by file access and staging; exact transferred content remains an analytical judgment.
Response actions
Completed after incident-commander authorization.
Prioritize deployment and cloud credentials.
Do not return the compromised filesystem to service.
Block executable uploads and restrict the web-worker account.
Technical evidence
| ID | Source | Event UTC | Entity | Verdict contribution |
|---|---|---|---|---|
| L-01 | nginx access | 18:42:16 | upload route | Initial access request |
| L-02 | EDR process | 18:42:17 | php-fpm → sh | Execution confirmed |
| L-03 | auditd | 18:44:08 | systemd user unit | Persistence confirmed |
| L-04 | fanotify | 18:45:21 | secret-bearing files | Credential access confirmed |
| L-05 | network flow | 18:46:02 | external destination | Egress corroborated |
MITRE ATT&CK mapping
| Technique | Observed behavior |
|---|---|
| T1190 · Exploit Public-Facing Application | Crafted request exploited an upload component. |
| T1505.003 · Web Shell | Hidden PHP shell provided repeatable command execution. |
| T1543.002 · Systemd Service | User service established persistence. |
| T1552.001 · Credentials In Files | Environment, cloud, deployment, and SSH secrets accessed. |
| T1560.001 · Archive via Utility | Credential material staged with tar and encryption. |
| T1041 · Exfiltration Over C2 Channel | Outbound transfer followed archive creation. |