CAP4136: Malware Analysis

Course Description

CAP4136 Malware Analysis is the course in reverse engineering hostile software — taking a compiled executable apart to determine what it does, how it does it, and how to detect and defeat it.

The course is offered at a small number of Florida institutions, including the University of Florida, the University of South Florida and the University of West Florida. ⚠ Institution count is low, so expect more variation than for a widely offered course — particularly in how much time goes to static versus dynamic analysis and in which tooling is taught.

The University of West Florida places it in the College of Science and Engineering, Department of Cybersecurity and Information Technology at 3 semester hours, requires CDA 3101, and describes "software reverse engineering of executable code (or malware) to determine its function and affects or to recover the source code implementation." The University of Florida carries it as Malware Reverse Engineering — a title that names the method rather than the object, and describes the same course.

⚠⚠ Read the legal and ethical section of this guide before you read anything else about this course. Handling live malware carries real risk to you, to your institution and to third parties, and the legal boundaries around this work are strict. Nothing else on this page matters as much.

Why the prerequisite is CDA3101, and why it is exactly right. Malware analysis is the practical application of computer organisation. You are reading assembly — the compiled output of a program whose source you do not have — and reasoning about registers, the stack, calling conventions, memory layout and the instruction set. ⚠ A student who found computer organisation difficult will find this course very difficult, because it is that material used in anger, on code deliberately written to be hard to read. This repository publishes a guide for CDA3101; it is the foundation this course stands on.

The discipline divides into two complementary approaches, and the course teaches both because neither is sufficient.

• Static analysis — examining the file without running it. Hashing and identification, string extraction, examining the PE or ELF headers and imports, and disassembly. It is safe and complete in principle: every path through the code is visible. ⚠ It is also defeated by packing, encryption and obfuscation, which most real malware uses precisely to defeat it.
• Dynamic analysis — running the sample in an isolated environment and observing what it does: files created, registry keys written, processes spawned, network connections attempted. It cuts straight through obfuscation, because the code must eventually do its actual work. ⚠ But you only see the paths that execute, and modern malware detects analysis environments and behaves differently or not at all when it thinks it is being watched.

The adversarial character is what makes the subject genuinely interesting. ⚠ Malware is written by people who know it will be analysed and who are actively trying to prevent it. Packing and runtime unpacking, anti-debugging checks, virtual machine and sandbox detection, control flow flattening, encrypted strings decoded only at use, and timing checks. Every technique in the course has a counter-technique, and the course is in part a history of that arms race.

The tooling is specific and learnable. Ghidra — the NSA's reverse engineering suite, released free and open-source, and now the standard teaching tool — and IDA Pro, the long-standing commercial equivalent; x64dbg and OllyDbg for debugging; Wireshark for network behaviour; Process Monitor and the rest of the Sysinternals suite for host behaviour; and YARA for writing the detection rules that turn an analysis into something operationally useful.

⚠ Writing a YARA rule is frequently the course's culminating skill, and it is the right one: it converts private understanding of a sample into a shareable, deployable detection artefact — which is what the job actually produces.

Learning Outcomes

Required Outcomes

• Explain the malware taxonomy — virus, worm, trojan, rootkit, ransomware, botnet, information stealer, loader, wiper — and classify a sample by behaviour rather than by label.
• Establish and operate a safe, isolated analysis environment, and explain each isolation control and its failure mode.
• Apply basic static analysis — hashing, file type identification, string extraction, import and export analysis, section examination.
• Interpret the PE file format (and ELF where covered) — headers, sections, the import address table, entry point.
• Recognise packing and obfuscation from entropy, section characteristics and import anomalies.
• Perform disassembly and static code analysis using Ghidra or IDA; navigate a binary, rename and annotate, and follow control flow.
• Read x86/x64 assembly fluently enough to determine a function's purpose, including calling conventions and stack frame structure.
• Identify common code constructs in disassembly — loops, conditionals, switch tables, structures, string operations.
• Perform dynamic analysis — execute a sample under monitoring and record host and network behaviour.
• Use a debugger — breakpoints, stepping, register and memory inspection, patching — to analyse execution.
• Analyse network behaviour — command and control traffic, beaconing, exfiltration — and identify indicators.
• Recognise and defeat anti-analysis techniques — anti-debugging, VM and sandbox detection, timing checks, obfuscation.
• Unpack a packed sample, manually or with tooling.
• Explain persistence mechanisms — registry, scheduled tasks, services, startup locations — and identify them in a sample.
• Explain process injection and other stealth techniques.
• Extract indicators of compromise and write YARA rules for detection.
• Map observed behaviour to the MITRE ATT&CK framework.
• Write a professional malware analysis report — summary, capabilities, indicators, recommendations — for a technical and a management audience.
• ⚠ Explain and observe the legal and ethical constraints on obtaining, handling, analysing and disposing of malware.

Optional Outcomes

• Analyse document-based and script malware — macros, JavaScript, PowerShell, LNK.
• Analyse mobile malware — Android APK reverse engineering.
• Analyse Linux and macOS malware.
• Apply memory forensics — Volatility, memory image analysis.
• Explain rootkits and kernel-mode techniques.
• Explain ransomware mechanics, key management and recovery considerations.
• Apply automation and scripting — Python for analysis, Ghidra scripting.
• Explain threat intelligence — attribution, campaign tracking, intelligence sharing and its limits.
• Apply machine learning approaches to malware classification and their evasion.
• Explain incident response integration — how analysis feeds containment and remediation.

Major Topics

Required Topics

• Malware types, capabilities and the threat landscape.
• ⚠ Laboratory safety, isolation and legal/ethical constraints.
• Basic static analysis — hashing, strings, imports, file identification.
• Executable file formats — PE, and ELF where covered.
• x86/x64 assembly for reverse engineering.
• Disassembly and static code analysis; Ghidra/IDA workflow.
• Recognising compiled constructs in disassembly.
• Basic dynamic analysis — sandboxing, behavioural monitoring.
• Debugging — breakpoints, stepping, memory and register inspection.
• Network analysis and command-and-control traffic.
• Packing, obfuscation and unpacking.
• Anti-analysis and anti-debugging.
• Persistence and privilege mechanisms.
• Process injection and stealth.
• Indicators of compromise and YARA.
• MITRE ATT&CK mapping.
• Reporting.

Optional Topics

• Script and document malware.
• Mobile malware.
• Linux and macOS malware.
• Memory forensics.
• Rootkits and kernel techniques.
• Ransomware in depth.
• Automation and scripting.
• Threat intelligence and attribution.
• ML-based detection and adversarial evasion.
• Incident response integration.

Resources & Tools

• The standard text is Sikorski and Honig, Practical Malware Analysis — ⚠ near-universal in this course, and its accompanying labs are the de facto curriculum. Also used: Monnappa K A, Learning Malware Analysis; Ligh et al., Malware Analyst's Cookbook; Eagle and Nance, The Ghidra Book; Yurichev's free Reverse Engineering for Beginners.
• Tools, nearly all free: Ghidra (NSA, free and open source — the standard teaching disassembler); IDA Free; x64dbg; Cutter/radare2; PE-bear and CFF Explorer; the Sysinternals suite (Process Monitor, Process Explorer, Autoruns); Wireshark; YARA; Volatility for memory analysis; and FLARE-VM plus REMnux — prebuilt analysis distributions that assemble most of the above.
• Analysis services: VirusTotal, Hybrid Analysis, Any.Run, Joe Sandbox. ⚠⚠ Understand before using them that public submission is publication — an uploaded sample and its metadata become visible to the security industry and, in some cases, to the malware's author. Never upload anything containing organisational or personal data.
• Reference frameworks: MITRE ATT&CK — free, and the shared vocabulary of the field; the Microsoft PE format specification; Intel and AMD architecture manuals for instruction reference.
• Practice material obtained legitimately: the labs shipped with Sikorski and Honig; MalwareBazaar and VirusShare (⚠ only with your instructor's direction and inside the course environment); Flare-On, the annual free reverse engineering challenge, whose past years remain available and are excellent; crackmes.one; and OALabs and MalwareUnicorn's free workshops.
• ⚠ UWF context: UWF is an NSA/DHS-designated National Center of Academic Excellence in Cyber Defense and operates the Florida Cyber Range — which matters here specifically, because a purpose-built isolated range is the correct place to do this work and not every institution has one.
• Certifications adjacent to this course: GIAC GREM (Reverse Engineering Malware) is the specialist credential; CompTIA Security+ remains the baseline entry certification and a DoD 8140 requirement for many defence roles.

Career Pathways

⚠ This is a specialist skill with a small candidate pool and persistent demand — one of the few undergraduate courses whose specific content is directly hired for.

• Malware analysts and reverse engineers (SOC 15-1212) — the direct destination; antivirus and endpoint vendors, threat intelligence firms, and large enterprise security teams.
• Incident responders and digital forensics analysts (SOC 15-1212, 33-9021) — ⚠ the most common realistic entry point, and where the analysis skill is used in anger.
• Threat intelligence analysts (SOC 15-1212, 33-3021).
• Security operations centre analysts (SOC 15-1212) — ⚠ a standard first role; the ability to actually analyse a sample rather than escalate it distinguishes a candidate immediately.
• Detection engineers (SOC 15-1212) — writing the YARA and behavioural rules this course culminates in.
• Vulnerability researchers and exploit developers (SOC 15-1212) — the same reverse engineering skill applied offensively; ⚠ almost entirely government, defence contractor or specialist firm work.
• Federal law enforcement and intelligence (SOC 33-3021, 15-1212) — FBI, HSI, NSA, Cyber Command and the service cyber components. ⚠ Requires US citizenship and a security clearance.
• Defence contractors — ⚠ a substantial Florida pathway: MacDill Air Force Base (Tampa) hosts CENTCOM and SOCOM, and L3Harris (Melbourne, Palm Bay), Lockheed Martin (Orlando), Northrop Grumman and the Space Coast base all employ cyber staff. Contractors sponsor clearances, which is frequently how a first clearance is obtained.
• Financial services and healthcare security — ⚠ Florida's banking sector (Jacksonville, Miami, Tampa) and its large health systems both maintain security teams.

⚠ Practical advice for entering this field: take Security+ early; complete Flare-On challenges and publish your write-ups; keep a public portfolio of analyses of samples you obtained legitimately; and apply for the CyberCorps: Scholarship for Service programme, which funds study at CAE institutions in exchange for federal service and is consistently under-applied for.

Special Information

⚠⚠⚠ Legal and ethical constraints — read this before the course starts

This course teaches you to analyse working malicious software. The constraints are not classroom etiquette; they are law, and violations end careers before they begin.

• Handle samples only in the environment your course provides or explicitly authorises, and only samples obtained through authorised channels. ⚠ Do not collect malware casually, do not analyse a sample from your own infected machine outside the lab, and do not carry samples on a personal device.
• ⚠⚠ Never execute malware on a network you do not fully control. Real malware spreads. Running a worm on a campus or home network can cause genuine damage to third parties, and you would be responsible for it regardless of intent.
• Unauthorised access is a crime under the federal Computer Fraud and Abuse Act and Florida's Computer Crimes Act (Chapter 815, Florida Statutes). ⚠ Intent to learn is not a defence, and causing no damage is not a defence.
• ⚠ Writing malware is a separate question from analysing it. Some courses include controlled exercises in writing benign samples for analysis; doing so outside an authorised assignment — and above all distributing anything — is a serious matter. If in doubt, ask before you write, not after.
• Your institution's acceptable use policy applies to campus systems and networks, independently of criminal law, and violating it is a disciplinary matter.
• ⚠⚠ Public sandbox submission is publication. Uploading a sample to VirusTotal or a similar service makes it and its metadata available to the security industry — and potentially alerts its author that it has been caught. In a professional incident this can be a serious operational error, and samples containing organisational or personal data must never be uploaded.
• ⚠ Responsible disclosure. If you discover a vulnerability while analysing something, report it through the proper channel and stop investigating.
• Why this matters more here than in most courses: this field's employment depends on background investigation and, for the best work, a security clearance. An academic misconduct finding, a policy violation, or anything worse is reportable and frequently disqualifying. ⚠ The professional stakes of a lapse are higher in this discipline than in almost any other.

⚠ Prerequisites — and why CDA3101 is not negotiable

UWF requires CDA3101 (introduction to computer organization). ⚠ This is the most genuinely load-bearing prerequisite in this batch of guides. The course consists of reading assembly, and that means you must already understand:

• The instruction set and registers — what the machine actually does.
• The stack, calling conventions and stack frames — how function calls are implemented, which is how you identify functions and their arguments in a disassembly with no symbols.
• Memory layout and addressing.
• Data representation — because you will be looking at bytes and deciding what they mean.

⚠ A student without solid assembly will spend the course decoding the language rather than analysing the behaviour. If your computer organisation is shaky, revisit it before the term starts — Compiler Explorer (godbolt.org) is free and is the fastest way to rebuild the C-to-assembly intuition this course runs on.

⚠ Other unlisted but genuinely useful preparation: C programming (you are recovering C-like code, so knowing what compiles to what is decisive); operating systems — particularly Windows internals, processes, threads and the registry; networking for the traffic analysis; and Python for automation.

The course sits in the senior year of a cybersecurity or computer science programme, typically after computer organisation, operating systems and an introductory security course.

Title variation

UWF and the statewide inventory: Malware Analysis. UF: Malware Reverse Engineering. ⚠ The same course under two names — one naming the object of study, the other the method. The number is CAP4136 at all three institutions, so articulation is clean. Benign title drift.

Course format and workload

3 credits, 45 contact hours — lecture with heavy hands-on work. ⚠ No C suffix at UWF, so laboratory work is assigned rather than timetabled — the practical hours come out of your own week.

⚠ Expect 10–12 hours per week outside class, and expect it to be lumpy. Analysis is not steadily paced: a sample can yield in an hour or resist for eight, and you cannot reliably predict which. Starting an analysis assignment the night before it is due is the single most common failure mode in this course.

Assessment typically includes analysis assignments with written reports, practical examinations, and often a substantial final analysis project.

⚠ Set up your analysis environment in week one. Virtual machine, snapshots, host-only networking, FLARE-VM or REMnux. Environment problems block assignments rather than merely delaying them, and a misconfigured environment is also the scenario in which something escapes.

⚠ What students find hardest

• Assembly at volume. Reading a few instructions is manageable; reading a few thousand and building a mental model of the program is a different skill, and it comes only with hours.
• Knowing when to stop. ⚠ Complete understanding of a sample is frequently impossible and usually unnecessary. The professional question is "what does it do, how do I detect it, how do I remove it" — and learning to answer that without fully reverse engineering every function is a judgement the course is trying to build.
• Anti-analysis frustration. Discovering that a sample detected your VM and did nothing is normal, not failure. It is the adversary doing their job.
• Report writing. ⚠ Technically strong students routinely underperform on the written deliverable. The report is the product — in practice nobody sees your Ghidra database, they read what you wrote.

Articulation and transfer

CAP4136 is a 4000-level upper-division course, not offered at Florida College System institutions, and taken after transfer. The number is consistent at the three Florida institutions that carry it, so SCNS articulation is clean.

⚠ Prefix note. CAP is computer applications (AI, graphics, and specialised applied topics); CIS computer information systems; COP programming; CDA computer organisation; CNT networking; CEN software engineering; CET computer engineering technology. ⚠ Cybersecurity content is distributed across CAP, CIS, CNT and CET depending on the institution — search by subject rather than by prefix when checking whether a receiving programme's requirement is met. Related: CDA3101 (the prerequisite — a guide exists in this repository), CIS4368 (database security), and various numbers for network security and digital forensics.

AI Integration

⚠ Malware analysis sits directly on the front line of AI's security consequences, in both directions, and a current course should say so.

Where AI is used defensively — and it is used heavily: malware classification at scale is machine learning work, because no human workforce can triage the volume; behavioural anomaly detection on endpoints; automated triage to route the small fraction of samples worth human attention; and similarity clustering to link samples to known families.

⚠ The critical limitation, and it is this course's argument for its own existence: classifiers are trained on known malware and are systematically weakest on the novel. A genuinely new technique is precisely what the model has not seen — and the novel case is the one that matters most. Human analysis is what handles it, and human analysis is what generates the labels the next model trains on.

Where AI is used offensively: adversarial evasion — modifying a sample to defeat a classifier while preserving function — is an active research and criminal practice area; automated obfuscation and polymorphism; and AI-assisted development of malicious code, which lowers the skill floor for attackers even where guardrails prevent the most direct requests. ⚠ The most consequential near-term effect is arguably not on malware at all but on social engineering, where convincing phishing at scale is now trivially cheap — and credential theft remains one of the main routes to deploying malware in the first place.

Using AI tools in this course. Models are genuinely useful for explaining an unfamiliar assembly instruction or Windows API call, for hypothesising what a decompiled function might be doing, for writing Python automation and Ghidra scripts, and for drafting the narrative sections of a report. ⚠ These are legitimate and are becoming normal professional practice.

⚠ Where they fail, specifically:

• Confident wrong interpretations of disassembly. A model will describe what a function "does" fluently and be wrong, and in this course a wrong interpretation propagates through the entire analysis. Verify by debugging — set a breakpoint and watch what actually happens.
• Fabricated API behaviour and structure definitions. Check Microsoft's documentation.
• Missing the adversarial layer entirely. Models do not naturally reason about code written to deceive an analyst, which is the entire subject.

⚠⚠ Two rules that are not negotiable. First: never paste malware samples, or code extracted from them, into a public AI service. You are transmitting potentially malicious code to a third party, you may be disclosing information about a live incident, and in a professional setting it is an operational security failure. Second: AI assistance does not extend authorisation. A model will explain a technique without any knowledge of whether you are permitted to apply it. The legal boundary is yours alone, and it is the one thing on this page with consequences that outlast the course.

Academic integrity. Read the syllabus; computing departments have generally written specific policies. ⚠ As noted above, an academic integrity finding in this discipline is reportable in background investigations and can be disqualifying for a clearance — which makes the stakes here materially higher than a grade.