Malware Analysis Fundamentals

Malware analysis is the process of understanding what a malicious program does, how it does it, and who may have created it. It is the bridge between incident response ("we were compromised") and actionable intelligence ("this is what the malware does, here are the IOCs, and here is how to detect and prevent it").

Every SOC analyst, incident responder, and threat hunter needs malware analysis skills. This page covers the methodology from basic triage to advanced behavioral analysis, the tools used at each stage, and how to write detection rules based on your findings.

Safety Warning

Only analyze malware in isolated environments. Use dedicated VMs with no network access to production systems, no shared folders, and snapshot capability. Even accidental execution can compromise your analysis machine and potentially your entire network.

Malware Analysis Methodology

Analysis Lab Setup

Isolated Analysis Environment

Essential Tool Distributions

Distribution	OS	Purpose	Key Tools
FLARE VM	Windows	Full malware analysis environment	x64dbg, Ghidra, PEStudio, Wireshark, Process Monitor
REMnux	Linux	Network analysis, static analysis	INetSim, fakeDNS, Volatility, YARA, radare2
Tsurugi	Linux	DFIR and malware analysis	Autopsy, Volatility, YARA, Cuckoo

bash

# REMnux setup
# Download REMnux OVA from https://remnux.org
# Import into VirtualBox/VMware
# Configure host-only networking

# Start INetSim (simulates DNS, HTTP, SMTP, etc.)
sudo inetsim

# Start fakeDNS to capture DNS requests
sudo fakedns

# FLARE VM setup (on Windows 10 VM)
# Run in PowerShell as administrator:
# Install Boxstarter
# Run FLARE VM installer from GitHub

Snapshot Before Every Analysis

Always take a VM snapshot before detonating malware. After analysis, revert to the clean snapshot. Never reuse a VM that has been exposed to malware without reverting.

Static Analysis

Static analysis examines the malware without executing it. This is safe and provides valuable intelligence before risking dynamic analysis.

File Identification

bash

# Determine file type (do not trust extensions)
file suspicious.exe
# Output: PE32 executable (GUI) Intel 80386, for MS Windows

# Calculate hashes for lookup
md5sum suspicious.exe
sha256sum suspicious.exe
ssdeep suspicious.exe  # Fuzzy hash for similarity matching

# Check on VirusTotal
# Upload hash (not the file) to https://www.virustotal.com
# Or use VT API:
curl -s "https://www.virustotal.com/api/v3/files/SHA256_HASH" \
    -H "x-apikey: YOUR_API_KEY" | jq '.data.attributes.last_analysis_stats'

# Check MalwareBazaar
curl -s -X POST "https://mb-api.abuse.ch/api/v1/" \
    -d "query=get_info&hash=SHA256_HASH"

PE Header Analysis

The PE (Portable Executable) header contains metadata about a Windows executable: compilation timestamp, imported DLLs, sections, and more.

bash

# PEStudio (Windows GUI) — best PE analysis tool
# Open suspicious.exe in PEStudio

# pestudio checks:
# - Compilation timestamp (is it in the future? Very old?)
# - Imported DLLs and functions (suspicious imports?)
# - Sections (unusual names? High entropy?)
# - Resources (embedded executables? Scripts?)
# - Strings (URLs, IPs, registry keys, file paths?)

# pefile (Python library)
python3 -c "
import pefile
pe = pefile.PE('suspicious.exe')
print(f'Compilation time: {pe.FILE_HEADER.TimeDateStamp}')
print(f'Sections:')
for section in pe.sections:
    print(f'  {section.Name.decode().rstrip(chr(0)):10s} '
          f'Entropy: {section.get_entropy():.2f} '
          f'Size: {section.SizeOfRawData}')
print(f'Imports:')
for entry in pe.DIRECTORY_ENTRY_IMPORT:
    print(f'  {entry.dll.decode()}')
    for imp in entry.imports:
        if imp.name:
            print(f'    {imp.name.decode()}')
"

# CFF Explorer (Windows) — visual PE analysis
# Detect-It-Easy (DiE) — packer/compiler detection

Suspicious Import Functions

Import	DLL	Indicates
`CreateRemoteThread`	kernel32.dll	Process injection
`VirtualAllocEx`	kernel32.dll	Remote memory allocation
`WriteProcessMemory`	kernel32.dll	Writing to another process
`NtUnmapViewOfSection`	ntdll.dll	Process hollowing
`URLDownloadToFile`	urlmon.dll	Download additional payloads
`InternetOpenUrl`	wininet.dll	C2 communication
`RegSetValueEx`	advapi32.dll	Registry persistence
`CryptEncrypt`	advapi32.dll	Encryption (ransomware)
`ShellExecute`	shell32.dll	Execute commands
`IsDebuggerPresent`	kernel32.dll	Anti-debugging
`GetTickCount`	kernel32.dll	Anti-sandbox timing checks

String Extraction

bash

# Extract ASCII and Unicode strings
strings suspicious.exe > strings_ascii.txt
strings -el suspicious.exe > strings_unicode.txt  # Unicode (little-endian)

# FLOSS — automatically deobfuscates strings
floss suspicious.exe > floss_output.txt

# What to look for in strings:
# - URLs and IP addresses (C2 servers)
# - File paths (C:\Windows\Temp\payload.exe)
# - Registry keys (HKCU\Software\Microsoft\Windows\CurrentVersion\Run)
# - Usernames, passwords, API keys
# - Error messages (can identify the language/framework)
# - Mutex names (used to prevent multiple instances)
# - PDB paths (leak developer environment info)
#   Example: C:\Users\APT28\Desktop\malware\Release\dropper.pdb

Packer Detection

Packed malware compresses or encrypts the original code to evade static analysis. The packed binary unpacks itself in memory at runtime.

Indicator	Description
High entropy	Sections with entropy > 7.0 are likely compressed/encrypted
Few strings	Packed binaries have very few readable strings
Few imports	Only basic imports (LoadLibrary, GetProcAddress) for unpacking
Unusual section names	UPX0, .aspack, .petite instead of .text, .data
Small code section	Very small .text section with large data sections

bash

# Detect packing with Detect-It-Easy (DiE)
# Shows: UPX, ASPack, Themida, VMProtect, custom packers

# Unpack UPX (most common packer)
upx -d packed_sample.exe -o unpacked_sample.exe

# For custom packers: dynamic analysis or manual unpacking with x64dbg

Dynamic Analysis

Dynamic analysis executes the malware in a controlled environment and observes its behavior: what processes it creates, what files it writes, what network connections it makes, and what registry keys it modifies.

Sandbox Platforms

Sandbox	Type	Cost	Strengths
Any.Run	Cloud	Free tier + paid	Interactive, real-time browser-based analysis
Joe Sandbox	Cloud	Free community + paid	Deep analysis, MITRE ATT&CK mapping
Cuckoo Sandbox	Self-hosted	Free	Customizable, no upload to third parties
Hybrid Analysis	Cloud	Free	CrowdStrike-powered, good detection
Triage	Cloud	Free	Fast, automated, IOC extraction
CAPE	Self-hosted	Free	Cuckoo fork with advanced unpacking

Manual Dynamic Analysis (Windows VM)

bash

# Before executing: start monitoring tools

# 1. Process Monitor (ProcMon) — file, registry, process, network events
# Set filters:
# - Process Name is suspicious.exe
# - Operation is WriteFile
# - Operation is RegSetValue
# - Operation is TCP Connect

# 2. Process Hacker / Process Explorer — process tree, loaded DLLs, handles
# Watch for: child processes, injected DLLs, hidden processes

# 3. Wireshark / TCPView — network connections
# Capture on the VM's network interface
# Look for: DNS queries, HTTP/HTTPS connections, raw TCP/UDP

# 4. Regshot — registry snapshot comparison
# Take snapshot before and after execution
# Compare: new keys, modified values, deleted keys

# 5. Autoruns — persistence mechanisms
# Check: Run keys, scheduled tasks, services, startup folder

# Execute the malware and observe for 5-15 minutes
# Some malware has delayed execution (sleep timers)

Behavioral Indicators

Behavior	Registry / Artifact	Indicates
Run key persistence	`HKCU\Software\Microsoft\Windows\CurrentVersion\Run`	Auto-start on login
Scheduled task	`schtasks /create /sc minute /tn "Update" /tr malware.exe`	Periodic execution
Service creation	`sc create MalSvc binPath= malware.exe`	System-level persistence
File creation in Temp	`C:\Users\\AppData\Local\Temp\.exe`	Dropped payloads
Modifying hosts file	`C:\Windows\System32\drivers\etc\hosts`	DNS hijacking
Disabling Defender	`Set-MpPreference -DisableRealtimeMonitoring $true`	Defense evasion
Shadow copy deletion	`vssadmin delete shadows /all /quiet`	Ransomware preparation

Unpacking and Deobfuscation

Manual Unpacking with x64dbg

# General unpacking methodology:
# 1. Open packed sample in x64dbg
# 2. Set breakpoint on VirtualAlloc or VirtualProtect
# 3. Run until breakpoint — these allocate memory for unpacked code
# 4. Follow the allocated memory region
# 5. Continue execution until the unpacked code is written
# 6. Find the OEP (Original Entry Point) — where unpacked code starts
# 7. Dump the process memory (use Scylla plugin)
# 8. Fix the Import Address Table (IAT)

# Common API breakpoints for unpacking:
# VirtualAlloc — memory allocation for unpacked code
# VirtualProtect — changing memory permissions (RWX)
# CreateProcessInternalW — process hollowing
# NtWriteVirtualMemory — writing to remote process
# LoadLibraryA — loading DLLs dynamically

String Deobfuscation Patterns

Technique	Description	Deobfuscation Approach
XOR encoding	Strings XORed with single byte or key	Brute force 256 possible single-byte keys
Base64	Standard encoding	Base64 decode
Custom cipher	ROT, substitution, or custom algorithm	Reverse engineer the algorithm
Stack strings	Characters pushed to stack individually	Reconstruct from assembly
API hashing	API names resolved by hash at runtime	Match hashes against known API databases
AES/RC4 encrypted	Strings encrypted with embedded key	Find key in binary, decrypt

python

# XOR brute force — try all single-byte keys
def xor_bruteforce(data):
    for key in range(256):
        decoded = bytes([b ^ key for b in data])
        # Check if result contains readable strings
        try:
            text = decoded.decode('ascii')
            if any(c.isalpha() for c in text):
                print(f"Key 0x{key:02x}: {text}")
        except UnicodeDecodeError:
            pass

# Read suspected encoded data from binary
with open('suspicious.exe', 'rb') as f:
    data = f.read()

# Search for XOR-encoded strings at known offsets
xor_bruteforce(data[0x1000:0x1050])

YARA Rule Writing

YARA rules are the primary method for classifying and detecting malware based on patterns.

YARA Rule Structure

yara

rule Emotet_Dropper {
    meta:
        description = "Detects Emotet dropper samples"
        author = "Malware Analyst"
        date = "2026-03-20"
        severity = "critical"
        hash = "abc123def456..."
        reference = "https://attack.mitre.org/software/S0367/"
        tlp = "WHITE"

    strings:
        // String patterns
        $s1 = "powershell" ascii nocase
        $s2 = "-encodedcommand" ascii nocase
        $s3 = "Invoke-WebRequest" ascii wide nocase

        // URL patterns
        $url1 = /https?:\/\/[a-zA-Z0-9.-]+\/[a-zA-Z0-9_-]+\.php/ ascii

        // Hex patterns (byte sequences from the malware)
        $hex1 = { 4D 5A 90 00 03 00 00 00 }  // MZ header
        $hex2 = { 8B ?? 83 ?? ?? 0F 84 }      // Common code pattern
        // ?? = wildcard byte

        // Registry persistence pattern
        $reg = "CurrentVersion\\Run" ascii wide

    condition:
        uint16(0) == 0x5A4D and           // Must be PE file
        filesize < 5MB and                 // Size constraint
        (2 of ($s*)) and                   // At least 2 string matches
        ($url1 or $reg) and                // URL or registry pattern
        #hex2 > 3                          // hex2 appears more than 3 times
}

YARA Best Practices

yara

// Good rule — specific, low false positives
rule APT28_Dropper_2026 {
    meta:
        description = "APT28 dropper observed in March 2026 campaign"
        author = "Threat Intel Team"
        date = "2026-03-20"
        confidence = "high"

    strings:
        // Unique strings from this specific campaign
        $mutex = "Global\\{8F4E2A1B-3C5D-4E6F-A7B8-9C0D1E2F3A4B}" ascii
        $pdb = "C:\\work\\agent\\Release\\dropper.pdb" ascii
        $c2_pattern = /[a-f0-9]{8}\.(php|aspx)/ ascii

        // Specific code pattern (not generic)
        $code = { 48 8B 05 ?? ?? ?? ?? 48 85 C0 74 ?? 48 8B 48 08 E8 }

    condition:
        uint16(0) == 0x5A4D and
        filesize < 2MB and
        ($mutex or $pdb) and
        ($c2_pattern or $code)
}

// Bad rule — too generic, high false positives
// DO NOT DO THIS:
rule Bad_Rule {
    strings:
        $s1 = "http://" ascii    // Too common
        $s2 = "cmd.exe" ascii    // In many legitimate programs

    condition:
        any of them              // Way too broad
}

bash

# Scan files with YARA rules
yara -r rules/ suspicious.exe
yara -r rules/ /path/to/sample_directory/

# Scan running processes
yara -p 4 rules/ -p  # Use 4 threads

# Generate YARA rules automatically from samples
yarGen -m /path/to/malware/samples/ -o generated_rules.yar

Threat Actor TTPs

Mapping malware behavior to MITRE ATT&CK provides a structured way to communicate findings and compare across campaigns.

TTP Extraction Template

ATT&CK Tactic	Technique	Evidence
Execution	T1059.001 PowerShell	Spawns powershell.exe with encoded command
Persistence	T1547.001 Registry Run Keys	Writes to `HKCU\...\Run\WindowsUpdate`
Defense Evasion	T1027 Obfuscated Files	XOR-encoded strings, packed with custom packer
Defense Evasion	T1055.001 DLL Injection	Injects into explorer.exe via CreateRemoteThread
Credential Access	T1003.001 LSASS Memory	Calls MiniDumpWriteDump on lsass.exe
Discovery	T1082 System Information	Runs `systeminfo`, `whoami`, `ipconfig`
C2	T1071.001 Web Protocols	HTTPS beacons to C2 every 60 seconds
Exfiltration	T1041 Exfil Over C2	Data exfiltrated in POST body to C2

Malware Analysis Report Template

markdown

## Sample Information
- **Filename:** invoice_march_2026.exe
- **MD5:** abc123...
- **SHA256:** def456...
- **File type:** PE32 executable (GUI) Intel 80386
- **File size:** 384 KB
- **First seen:** 2026-03-15
- **VirusTotal detection:** 48/72

## Executive Summary
This sample is a dropper for [MALWARE_FAMILY]. Upon execution,
it decrypts an embedded payload, injects it into explorer.exe,
establishes persistence via registry, and beacons to a C2 server
over HTTPS every 60 seconds.

## IOCs
| Type | Value | Context |
|------|-------|---------|
| Hash (SHA256) | def456... | Dropper |
| Hash (SHA256) | ghi789... | Dropped payload |
| Domain | update-service[.]com | C2 server |
| IP | 185.123.45[.]67 | C2 server |
| URL | hxxps://update-service[.]com/api/check | C2 beacon URL |
| Mutex | Global\{8F4E2A1B...} | Mutex name |
| Registry | HKCU\...\Run\WindowsUpdate | Persistence key |

## MITRE ATT&CK Mapping
[Table of TTPs as shown above]

## YARA Rule
[Custom YARA rule for this sample]

## Recommendations
1. Block IOCs at firewall, proxy, and DNS levels
2. Scan endpoints for mutex and registry persistence
3. Monitor for the specific PowerShell command pattern
4. Update EDR signatures

Malware Types Reference

Type	Behavior	Example	Key Indicators
Dropper	Downloads/extracts second-stage payload	Emotet	URLDownloadToFile, high entropy sections
RAT	Remote access, keylogging, screen capture	njRAT, Quasar	Persistent connection, keystroke logging
Ransomware	Encrypts files, demands payment	LockBit, REvil	File enumeration, CryptEncrypt, shadow deletion
Stealer	Exfiltrates credentials, cookies, wallets	RedLine, Vidar	Browser database access, crypto wallet paths
Worm	Self-replicates across networks	WannaCry	SMB scanning, self-copying, lateral movement
Rootkit	Hides presence at kernel level	TDL4, ZeroAccess	Driver loading, SSDT hooking
Botnet agent	Joins C2 network for DDoS, spam, mining	Mirai, TrickBot	C2 beaconing, command parsing
Loader	Loads and executes other malware	BazarLoader	Memory allocation, reflective loading

Malware Analysis Fundamentals ​

Malware Analysis Methodology ​

Analysis Lab Setup ​

Isolated Analysis Environment ​

Essential Tool Distributions ​

Static Analysis ​

File Identification ​

PE Header Analysis ​

Suspicious Import Functions ​

String Extraction ​

Packer Detection ​

Dynamic Analysis ​

Sandbox Platforms ​

Manual Dynamic Analysis (Windows VM) ​

Behavioral Indicators ​

Unpacking and Deobfuscation ​

Manual Unpacking with x64dbg ​

String Deobfuscation Patterns ​

YARA Rule Writing ​

YARA Rule Structure ​

YARA Best Practices ​

Threat Actor TTPs ​

TTP Extraction Template ​

Malware Analysis Report Template ​

Malware Types Reference ​

Further Reading ​

Malware Analysis Fundamentals

Malware Analysis Methodology

Analysis Lab Setup

Isolated Analysis Environment

Essential Tool Distributions

Static Analysis

File Identification

PE Header Analysis

Suspicious Import Functions

String Extraction

Packer Detection

Dynamic Analysis

Sandbox Platforms

Manual Dynamic Analysis (Windows VM)

Behavioral Indicators

Unpacking and Deobfuscation

Manual Unpacking with x64dbg

String Deobfuscation Patterns

YARA Rule Writing

YARA Rule Structure

YARA Best Practices

Threat Actor TTPs

TTP Extraction Template

Malware Analysis Report Template

Malware Types Reference

Further Reading