This course provides a systems-oriented introduction to computer security, focusing on the mechanisms used to protect modern operating systems, applications, and computing platforms. The course examines how security policies are translated into concrete system mechanisms, how these mechanisms are implemented in modern systems, and how failures in software and hardware abstractions can undermine their security guarantees.
The course begins with memory safety, exploitation, and modern exploit mitigations before moving into kernel security and classical protection mechanisms such as access control, mandatory access control, and SELinux. We then study virtualization and confidential computing, followed by the Linux isolation primitives underlying modern containers and sandboxes, including namespaces, cgroups, capabilities, seccomp, and filesystem isolation. Students will use these mechanisms to understand how systems such as Docker and Bubblewrap actually construct security boundaries.
The course also covers compartmentalization, accelerator and heterogeneous-system security, confidential accelerators and machine-learning workloads, and emerging security challenges involving agentic systems. Throughout the course, the emphasis is on understanding the underlying mechanisms rather than treating modern security systems as black boxes.
Course Materials
GitHub: TBD
Discord: TBD
Additional course material, code, examples, and assignments will be made available through GitHub.
Location (Fall 26)
Leonhard Bldg 203
TuTh 10:35AM - 11:50AM
Schedule
The following schedule is tentative and subject to change throughout the semester. Topics, ordering, quiz dates, and the amount of time devoted to individual modules may be adjusted based on class progress, student background, and emerging systems-security developments.
All code used in class will be made available on GitHub when possible.
Short quizzes will be given after major course modules to assess understanding of systems-security concepts, mechanisms, attacks, and design tradeoffs.
Click on any week to expand or collapse topics and readings.
Week 1 — August 25 & 27: Systems Security Foundations & Readings
Readings: Foundations of Systems Security: Least Privilege and Protection
Required Reading
Nick Roessler et al. "μSCOPE: A Methodology for Analyzing Least-Privilege Compartmentalization in Large Software Artifacts." RAID, 2021. Presents a methodology for reasoning about and quantifying least privilege in complex software systems. Using the Linux kernel as a case study, the paper examines overprivilege, protection domains, compartmentalization, and the tradeoff between stronger privilege separation and enforcement overhead. [Paper]
Optional Readings
Adam Barth, Collin Jackson, Charles Reis, and the Google Chrome Team. "The Security Architecture of the Chromium Browser." 2008. Demonstrates how a real-world system applies protection domains, privilege separation, sandboxing, and attack-surface reduction. It also provides a concrete example of reasoning from an explicit threat model to a security architecture. [Paper]
Tianneng Shi, Jingxuan He, Zhun Wang, Linyu Wu, Hongwei Li, Wenbo Guo, and Dawn Song. "Progent: Programmable Privilege Control for LLM Agents." 2025. Applies the principle of least privilege to agentic systems by enforcing fine-grained policies over tool calls and their arguments. It explores how an agent's privileges can be dynamically restricted according to the task being performed. [Paper]
Jinhao Zhu, Kevin Tseng, Gil Vernik, Xiao Huang, Shishir G. Patil, Vivian Fang, and Raluca Ada Popa. "MiniScope: A Least Privilege Framework for Authorizing Tool Calling Agents." 2025. Develops a least-privilege authorization model for tool-calling agents by reconstructing permission hierarchies and granting agents only the permissions necessary to perform a task. [Paper]
Week 2 — September 1 & 3: Memory Safety and Exploitation & Readings
Adriaan Jacobs, Mahmoud Ammar, and Stijn Volckaert. "SoK: On the Fragility of Memory Error Exploit Mitigations." 35th USENIX Security Symposium (USENIX Security '26), 2026. [Paper]
Optional Readings
Aleph One. "Smashing the Stack for Fun and Profit." Phrack Magazine, Vol. 7, Issue 49, 1996. [Paper]
Jonathan Pincus and Brandon Baker. "Beyond Stack Smashing: Recent Advances in Exploiting Buffer Overruns." IEEE Security & Privacy, 2004. [Paper]
Hovav Shacham, Matthew Page, Ben Pfaff, Eu-Jin Goh, Nagendra Modadugu, and Dan Boneh. "On the Effectiveness of Address-Space Randomization." ACM Conference on Computer and Communications Security (CCS), 2004. [Paper]
Hovav Shacham. "The Geometry of Innocent Flesh on the Bone: Return-into-libc without Function Calls (on the x86)." ACM Conference on Computer and Communications Security (CCS), 2007. (Canonical ROP paper). [Paper]
László Szekeres, Mathias Payer, Tao Wei, and Dawn Song. "SoK: Eternal War in Memory." IEEE Symposium on Security and Privacy (S&P), 2013. [Paper]
Kevin Z. Snow, Fabien Monrose, Lucas Davi, Alexandra Dmitrienko, Christopher Liebchen, and Ahmad-Reza Sadeghi. "Just-In-Time Code Reuse: Bypassing Exploit Mitigation on Browsers." IEEE Symposium on Security and Privacy (S&P), 2013. [Paper]
Alexander Sotirov. "Heap Feng Shui in JavaScript." Black Hat USA, 2007. [Paper]
Andrea Bittau, Adam Belay, Ali Mashtizadeh, David Mazières, and Dan Boneh. "Hacking Blind." IEEE Symposium on Security and Privacy (S&P), 2014. (BROP). [Paper]
Felix Schuster, Thomas Tendyck, Christopher Liebchen, Lucas Davi, Ahmad-Reza Sadeghi, and Thorsten Holz. "Counterfeit Object-Oriented Programming: On the Difficulty of Preventing Code Reuse Attacks in C++ Applications (COOP)." IEEE Symposium on Security and Privacy (S&P), 2015. [Paper]
Nicholas Carlini, Antonio Barresi, Mathias Payer, David Wagner, and Thomas R. Gross. "Control-Flow Bending: On the Effectiveness of Control-Flow Integrity." USENIX Security Symposium, 2015. [Paper]
Hong Hu, Shweta Shinde, Sendroiu Adrian, Zheng Leong Chua, Prateek Saxena, and Zhenkai Liang. "Data-Oriented Programming: On the Expressiveness of Non-Control Data Attacks." IEEE Symposium on Security and Privacy (S&P), 2016. [Paper]
László Szekeres, Mathias Payer, Tao Wei, and Dawn Song. "SoK: Eternal War in Memory." IEEE Symposium on Security and Privacy (S&P), 2013. A broad overview of memory corruption, spatial and temporal memory safety, exploitation techniques, and major classes of defenses. [Paper]
Optional Readings
Crispin Cowan et al. "StackGuard: Automatic Adaptive Detection and Prevention of Buffer-Overflow Attacks." USENIX Security, 1998. Classic stack-canary work. Useful for distinguishing exploit mitigation from complete memory safety. [Paper]
Santosh Nagarakatte, Jianzhou Zhao, Milo M. K. Martin, and Steve Zdancewic. "SoftBound: Highly Compatible and Complete Spatial Memory Safety for C." PLDI, 2009. A canonical metadata-based approach for enforcing spatial memory safety. [Paper]
Santosh Nagarakatte, Jianzhou Zhao, Milo M. K. Martin, and Steve Zdancewic. "CETS: Compiler Enforced Temporal Safety for C." ISMM, 2010. A companion to SoftBound focused on temporal memory safety, including dangling pointers and use-after-free. [Paper]
Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitry Vyukov. "AddressSanitizer: A Fast Address Sanity Checker." USENIX ATC, 2012. The foundational paper behind AddressSanitizer, covering shadow memory and practical detection of memory errors. [Paper]
Jonathan Woodruff et al. "The CHERI Capability Model: Revisiting RISC in an Age of Risk." ISCA, 2014. Introduces the CHERI capability architecture and hardware-supported pointer bounds and authority. [Paper]
Ralf Jung et al. "RustBelt: Securing the Foundations of the Rust Programming Language." POPL, 2018. Provides a formal foundation for Rust's ownership-based memory-safety guarantees, including interaction with unsafe code. [Paper]
Module Quiz follows this topic.
Week 4 — September 15 & 17: Kernel Security
Kernel attack surface
System calls and the user/kernel boundary
Kernel memory corruption
Kernel privilege escalation
Kernel drivers
Driver security
Kernel exploitation
Kernel hardening
Kernel attack-surface reduction
Modern kernel defense mechanisms
Module Quiz follows this topic.
Week 5 — September 22 & 24: Access Control & Readings
Hao Chen, David Wagner, and Drew Dean. "Setuid Demystified." 11th USENIX Security Symposium, 2002. Analyzes Unix user-ID management semantics, formulates formal state-machine models for privilege transitions, and highlights subtle vulnerabilities that lead to privilege escalation. [Paper]
Optional Readings
Jerome H. Saltzer and Michael D. Schroeder. "The Protection of Information in Computer Systems." Proceedings of the IEEE, 1975. Foundational survey outlining design principles for protection mechanisms including least privilege, complete mediation, and fail-safe defaults. [Paper]
Norm Hardy. "The Confused Deputy: (or why I know that the 'substitute Programmer' is not a bug)." ACM SIGOPS Operating Systems Review, 1988. The classic paper identifying the confused deputy vulnerability and explaining how capability systems solve ambient-authority pitfalls in access control. [Paper]
Butler W. Lampson. "Protection." ACM SIGOPS Operating Systems Review, 1974. Introduced the foundational Access Control Matrix model formalizing subjects, objects, access rights, and domain switching. [Paper]
Important Milestone: Research-based semester project proposals are due Thursday, September 24. Students without an approved research proposal will complete the instructor-defined project.
Week 6 — September 29 & October 1: Mandatory Access Control and SELinux & Readings
Readings: Mandatory Access Control and Kernel Security Modules
Required Reading
Peter Loscocco and Stephen Smalley. "Integrating Flexible Support for Security Policies into the Linux Operating System." USENIX Annual Technical Conference (FREENIX Track), 2001. Introduces NSA's Security-Enhanced Linux (SELinux) and the Flask architecture, separating policy enforcement mechanisms in the OS kernel from decision logic to provide fine-grained Mandatory Access Control. [Paper]
Chris Wright, Crispin Cowan, Stephen Smalley, James Morris, and Greg Kroah-Hartman. "Linux Security Modules: General Security Support for the Linux Kernel." 11th USENIX Security Symposium, 2002. Describes the design and implementation of the Linux Security Modules (LSM) framework providing mediation hooks across kernel operations. [Paper]
Optional Readings
Xiaolan Zhang, Antony Edwards, and Trent Jaeger. "Using Cqual for Static Analysis of Authorization Hook Placement." 11th USENIX Security Symposium, 2002. Uses type qualifier analysis via Cqual to verify authorization hook placement in the Linux kernel and prove the absence of unmediated access paths in LSM. [Paper]
David E. Bell and Leonard J. LaPadula. "Secure Computer Systems: Unified Exposition and Multics Interpretation." MITRE Technical Report, 1976. The foundational mathematical model for mandatory confidentiality security policies (no read-up, no write-down). [Paper]
Kenneth J. Biba. "Integrity Considerations for Secure Computer Systems." MITRE Technical Report, 1977. The canonical integrity reference model establishing strict integrity access rules (no read-down, no write-up). [Paper]
Stephen Smalley and Robert Craig. "Security Enhanced (SE) Android: Bringing Flexible MAC to Android." Network and Distributed System Security Symposium (NDSS), 2013. Demonstrates the adaptation of SELinux and Type Enforcement to mobile computing to sandbox third-party apps and contain privilege escalation attacks. [Paper]
Mickaël Salaün. "Landlock LSM: Unprivileged Access Control." Linux Kernel Documentation & LPC, 2021. Overview of the Landlock security module, allowing unprivileged processes to create scoped sandbox restrictions for filesystems and networking. [Documentation]
Module Quiz follows this topic.
Week 7 — October 6 & 8: Virtualization
Virtual machines and hypervisors
Type-1 and Type-2 hypervisors
Privileged and unprivileged execution
Hardware-assisted virtualization
VM exits
CPU virtualization
Memory virtualization
Extended/Nested Page Tables
Device virtualization
Emulated and paravirtualized devices
KVM and QEMU
VirtIO
VM isolation
VM escape attacks
Virtual machines as security boundaries
Week 8 — October 13 & 15: Confidential Computing
Threat models in cloud computing
Trusted Execution Environments
Intel SGX
Intel TDX
AMD SEV
AMD SEV-SNP
Memory encryption
Confidential virtual machines
Protecting workloads from the hypervisor
Remote attestation
Measurement and trust establishment
I/O challenges in confidential systems
Limitations of confidential computing
Module Quiz follows this topic.
Week 9 — October 20 & 22: Linux Isolation Primitives
This module examines the mechanisms from which modern Linux containers and application sandboxes are constructed.
chroot
Filesystem isolation
Mounts and bind mounts
pivot_root
Linux namespaces (User, PID, Mount, Network, IPC, UTS)
cgroups
Linux capabilities
seccomp and seccomp-BPF
no_new_privs
unshare and nsenter
Rootless isolation
Students leave this module understanding the individual mechanisms behind a container rather than viewing containers as a single monolithic primitive.
Week 10 — October 27 & 29: Containers and Sandboxing
What a container actually is
Constructing containers from Linux isolation primitives
OCI container model
runc and containerd
Docker architecture
Container root filesystems and OverlayFS
Rootless vs. Privileged containers
Container capabilities & escape vulnerabilities
Dangerous mounts and Docker socket exposure
Bubblewrap (bwrap), Firejail, systemd-nspawn
Building application sandboxes from first principles
Containers versus virtual machines
Module Quiz follows this topic.
Week 11 — November 3 & 5: Compartmentalization
Privilege separation and least-privilege decomposition
Process-based compartmentalization
Capability-oriented systems (Capsicum, pledge, unveil)
Intel Memory Protection Keys (MPK) & ERIM
RLBox and WebAssembly sandboxing
Automatic compartmentalization
Compartment interfaces and cross-compartment communication
Attack-surface reduction and performance tradeoffs
Module Quiz follows this topic.
Week 12 — November 10 & 12: Accelerators and Heterogeneous-System Security
Modern applications, particularly machine-learning systems, increasingly depend on GPUs and other accelerators.
CPU-accelerator architecture & GPU execution model
Host and device memory & GPU virtual memory
GPU runtimes, drivers, and PCIe communication
Direct Memory Access (DMA) & IOMMUs
Device assignment and passthrough
Accelerator multi-tenancy & GPU process isolation
SR-IOV, MIG, and hardware partitioning
Accelerator drivers as part of the Trusted Computing Base
CPU-GPU trust boundaries & isolation failures
Week 13 — November 17 & 19: Confidential Accelerators and Agentic Systems
The final two instructional weeks are reserved entirely for semester projects. Students will present their systems, demonstrate technical artifacts, explain their design decisions, discuss security models and evaluation, and answer technical questions.
Semester project presentations and demonstrations continue.
Semester Project
The semester project is the central component of the course and accounts for 70% of the final grade.
Students have two options for completing the semester project.
Option 1: Research-Based Semester Project
Students interested in conducting systems-security research may propose their own research project.
You will have approximately one month from the beginning of the semester to develop and submit a project proposal. Research-based project proposals are due Thursday, September 24.
During this period, you should investigate the problem, understand existing work, and determine whether the proposed idea represents a meaningful technical contribution.
The proposal should clearly describe:
the systems-security problem being addressed;
why the problem is important;
the relevant threat model;
limitations of existing approaches;
the proposed technical idea;
how the system will be implemented;
how the system will be evaluated; and
why the proposed work is sufficiently different from existing work.
Students choosing the research-project track are expected to conduct enough background investigation to establish the novelty of their idea. Discovering late in the semester that the proposed idea has already been extensively explored will negatively affect the project evaluation.
Research projects must include a substantive technical artifact. A Work-In-Progress result is acceptable when appropriately justified, but an idea or literature survey without a corresponding implementation and evaluation is not sufficient.
Research projects will be evaluated based on technical difficulty, novelty, quality of the artifact, quality of the evaluation, and demonstrated understanding of the security problem.
Option 2: Instructor-Defined Semester Project
Students who do not wish to conduct a research-based project may complete the semester project provided by the instructor.
Students who do not submit an approved research proposal by September 24 will follow this project track.
The instructor-defined project will involve designing, implementing, analyzing, attacking, or defending a real system using techniques covered throughout the course. Detailed specifications, milestones, and evaluation criteria will be provided during the semester.
The instructor-defined project is not a lesser project option. Both project tracks are expected to require substantial systems-security engineering, experimentation, and technical understanding.
Grading
There are no traditional midterm or final exams for this course.
Your final grade will be determined as follows:
Semester Project: 70%
Module Quizzes: 30%
Quizzes will be given after major course modules and will focus on understanding the concepts, mechanisms, attacks, and design tradeoffs discussed in class.
There will be no required paper presentations or paper-reading component this semester. Relevant research papers and systems may still be discussed during lectures when they are useful for explaining important concepts or the development of modern systems-security mechanisms.
AI Usage Policy
Students are permitted to use Generative AI (GenAI) tools as part of their coursework. However, if a student chooses to do so, it is their responsibility to verify the accuracy of any information, code, analysis, or claims produced by the AI. Any errors, hallucinations, insecure code, incorrect analyses, or misleading outputs generated by such tools remain the responsibility of the student.
Students must be able to explain and defend any work they submit, including code, vulnerability analyses, system designs, experimental results, and technical decisions. The instructor may ask students to explain any portion of submitted work in order to verify their understanding.
Students should not treat GenAI systems as an “answering oracle.” These tools should instead be used as assistants for learning, research, debugging, analysis, and engineering. They are not a substitute for understanding the underlying systems-security concepts.
For assignments where requested by the instructor, students may also be required to submit relevant GenAI interaction logs or otherwise document how GenAI tools were used.
Disability Accommodation Statement
Penn State welcomes students with disabilities into the University’s educational programs. Every Penn State campus has an office for students with disabilities. The Student Disability Resources website provides contact information for every Penn State campus. For further information, please visit the Student Disability Resources website.
In order to receive consideration for reasonable accommodations, you must contact the appropriate disability services office at the campus where you are officially enrolled, participate in an intake interview, and provide documentation. If the documentation supports your request for reasonable accommodations, your campus’s disability services office will provide you with an accommodation letter. Please share this letter with your instructors and discuss the accommodations with them as early in your courses as possible. You must follow this process for every semester that you request accommodations.
Counseling and Psychological Services (CAPS) Statement
Many students at Penn State face personal challenges or have psychological needs that may interfere with their academic progress, social development, or emotional well-being. The university offers a variety of confidential services to help you through difficult times, including individual and group counseling, crisis intervention, consultations, online chats, and mental health screenings. These services are provided by staff who welcome all students and embrace a philosophy respectful of clients’ cultural and religious backgrounds, and sensitive to differences in race, ability, gender identity, and sexual orientation.
Penn State Crisis Line (24 hours/7 days/week): 877-229-6400
Crisis Text Line (24 hours/7 days/week): Text LIONS to 741741
Education Equity and Reporting Bias
Penn State takes great pride in fostering a diverse and inclusive environment for students, faculty, and staff. Acts of intolerance, discrimination, or harassment due to age, ancestry, color, disability, gender, gender identity, national origin, race, religious belief, sexual orientation, or veteran status are not tolerated and can be reported through Educational Equity via the Report Bias webpage.