Understanding Linux Processes for Blue Team Security

In the world of cybersecurity defense, processes running on a system can be both a sign of normal operations and an early warning indicator of a breach. For the blue team — the defenders in cybersecurity — knowing how to monitor, inspect, and interpret Linux processes is critical for detecting suspicious activity before it becomes a full-blown incident.

This article will break down the fundamentals of Linux processes, explain why they matter in security operations, and explore practical tools and commands the blue team can use to detect possible intrusions.

What is a Linux Process?

A process in Linux is simply an instance of a program in execution. Each process is assigned a unique PID (Process ID) and is associated with specific resources — CPU time, memory, open files, and network connections.

Processes can be:

  • Foreground processes: Programs running interactively, such as commands in a terminal.
  • Background processes: Daemons or services running without user interaction.
  • Zombie processes: Processes that have completed but still occupy a slot in the process table until acknowledged by their parent process.

For security defenders, processes are an active footprint of what's happening in the system right now. If an attacker runs malware, cryptominers, or unauthorized shells, they will manifest as processes — often trying to hide or blend in with legitimate ones.

Why Processes Matter in Threat Detection

From a blue team perspective, processes are real-time indicators of possible compromise. By monitoring process lists and their behavior, defenders can:

  • Spot unfamiliar processes running on production systems.
  • Detect process injection or suspicious parent-child process relationships.
  • Identify rogue network activity tied to specific processes.
  • Catch persistence mechanisms where malicious processes respawn automatically.

For example, if sshd (the secure shell daemon) is running as expected but you see another sshd process running under a non-root user, it might indicate a backdoor or unauthorized remote access tool.

Key Tools for Process Monitoring in Linux

Below are essential tools and commands that the blue team can use to monitor processes and detect anomalies.

ps - The Process Snapshot Tool

ps (process status) gives a snapshot of processes running at a given moment.

ps aux
  • a - show processes for all users
  • u - display the user owning the process
  • x - include processes not attached to a terminal

Example usage for threat hunting:

ps aux | grep sshd

If you see multiple SSH daemons or one running from an unusual directory, investigate further.

top and htop - Real-Time Monitoring

top is the standard Linux tool for viewing real-time process activity:

top

It shows CPU, memory usage, and process details, updating every few seconds.

htop is a more user-friendly version with color coding, process tree view, and interactive controls.

Example blue team check: Look for processes consuming unexpectedly high CPU or memory, which can indicate cryptominers or runaway scripts.

pstree - Process Tree Visualization

Attackers often spawn processes from unusual parents (e.g., a web server process spawning a shell). pstree shows the hierarchy of processes:

pstree -p

Example suspicious scenario: If apache2 or nginx spawns /bin/bash, it may indicate command execution via a web vulnerability.

lsof - List Open Files

Since “everything in Linux is a file,” lsof is a powerful way to link processes to files and network sockets.

lsof -i

This lists processes with open network connections. Combine it with grep to focus on specific ports:

lsof -i :4444

A process listening on an uncommon port like 4444 may be a reverse shell.

netstat and ss - Network Connection Tracking

While netstat is older, ss is faster and more modern:

ss -tulnp
  • t - TCP connections
  • u - UDP connections
  • l - listening sockets
  • n - numeric output
  • p - show processes

If you see a suspicious process listening on an unknown port, trace it back using its PID.

pidstat - Per-Process Statistics Over Time

Part of the sysstat package, pidstat lets you track a process's CPU and memory usage over time:

pidstat -p <PID> 5

If a process is steadily consuming resources in an abnormal way, it could be malicious.

auditd - Process Execution Auditing

auditd is a kernel-level auditing system that can log every command executed on the system:

auditctl -a always,exit -F arch=b64 -S execve

This records all executed commands, making it useful for post-breach investigation.

Real-World Blue Team Scenarios

Scenario 1: Detecting a Reverse Shell

An attacker compromises a web application and spawns a reverse shell back to their server. Using:

ss -tulnp

You notice a process bash owned by www-data connected to an external IP on port 4444. Next, you use:

lsof -p <PID>

It confirms the process is /bin/bash tied to a suspicious IP. Immediate action: kill the process and isolate the server.

Scenario 2: Catching a Cryptominer

A server is running slow. Using top, you see a process named xmrig consuming 98% CPU. You check:

ps aux | grep xmrig

It shows the miner running from /tmp/. Since /tmp/ should not contain executables in production, you terminate it and block the IP it's communicating with.

Scenario 3: Finding Persistence

A malicious process keeps respawning. You run:

pstree -p

You see a parent process /usr/bin/cron launching /tmp/malware.sh every minute. This leads you to inspect crontab entries and remove the malicious one.

Automating Process Monitoring

Blue teams often use SIEM (Security Information and Event Management) systems to automate detection. Logs from auditd, syslog, and custom scripts using tools like ps, ss, and lsof are fed into monitoring dashboards like ELK Stack, Splunk, or Wazuh.

Example automation snippet in bash:

#!/bin/bash
suspicious=$(ps aux | grep -E 'nc|bash|sh|python' | grep -v grep)
if [ ! -z "$suspicious" ]; then
    logger "Suspicious process detected: $suspicious"
fi

Best Practices for Blue Team Process Monitoring

  • Baseline normal activity so you can spot deviations quickly.
  • Enable auditing with auditd for command execution tracking.
  • Use process whitelisting where applicable.
  • Integrate with SIEM for real-time alerts.
  • Investigate anomalies immediately; don't ignore “weird but harmless-looking” processes.

Pro Tips for Blue Team Process Monitoring

If you want to go beyond just listing processes and really dig into deep process analysis on Linux, especially from a Blue Team/defensive perspective, here are some pro tips:

Don't Trust the First Process List

Attackers often hide processes by hooking into ps or top.

  • Use multiple tools to verify process lists:

    ps aux
top
htop

    And compare them with raw /proc scanning:

    ls /proc | grep '^[0-9]' | wc -l

Inspect Suspicious Parent-Child Relationships

  • Check if processes have unusual parents — like a shell (/bin/bash) spawned by nginx or sshd.

  • Use:

    pstree -a

    or:

    ps -eo pid,ppid,cmd --forest

Look for Processes Without an Executable

Sometimes malware deletes its own binary after loading it into memory.

  • Find them:

    lsof | grep deleted

    or:

    find /proc/*/exe -type l -ls | grep deleted

Monitor for Short-Lived Processes

Attackers may run payloads quickly and exit.

  • Use execsnoop from bcc-tools:

    sudo execsnoop

    This catches process execution in real time.

Cross-Check with Network Connections

A process with no business making network calls could be exfiltrating data.

  • Use:

    sudo ss -tulpn
sudo netstat -antp

    and correlate PIDs with ps output.

Inspect Loaded Libraries

Malware may inject malicious .so files.

  • List them:

    lsof -p <PID>
cat /proc/<PID>/maps

Use Memory Analysis for Stealthy Malware

If you suspect something is in memory only (fileless malware):

  • Dump memory with gcore:

    gcore <PID>

  • Analyze with tools like Volatility.

Automate Baseline Comparisons

  • Keep a known-good process baseline and alert when new or unusual processes appear.

  • Example with ps hash:

    ps aux | md5sum

    Compare regularly in a cron job.

Summary

For blue teams, Linux process monitoring is a frontline defense against intrusions. Every running process is a clue — sometimes benign, sometimes a signal of compromise. Mastering tools like ps, top, lsof, ss, and auditd allows defenders to quickly detect and respond to suspicious activity. By combining process inspection with network monitoring and historical logging, defenders gain a powerful edge in detecting breaches before they escalate.

Love it? Share this article: