Blue Team: SysAdmin
Blue Team work is defensive system administration:
Note
The attacker only needs one working path. The defender has to remove, monitor, or contain as many paths as possible.
| Phase | Question | Course block |
|---|---|---|
| Harden | What can we remove or restrict before an attack (S.O related)? | Linux hardening |
| Monitor | What is happening now? | Grafana, Prometheus, Loki |
| Detect | Which activity looks malicious? | IDS |
| Deceive | Can we make attacks observable? | Honeypots |
| Maintain | Could there be an exploitable vulnerability in our app that isn’t directly related to our code? | Best practices for our app’s |
| Recover | If we’ve an attack, what happened? | Forensics |
To decide which OPSEC practices to adopt, you first need to understand what you’re defending against. A health app storing sensitive user data, a rock-paper-scissors app, and your own personal server all face very different risks, and therefore have very different threat models.
A threat model is a description of your adversaries with:
The more powerful and well-funded the adversary, the more dangerous it is (States being at the top of the food chain).
Note
Think carefully about whether you really need to apply every measure we cover here. Security always involves trade-offs, so weigh the time and effort each measure requires against the actual protection it gives you.
How can we make it more difficult for an attacker to compromise our operating system?
The Linux hardening guide used for this section defines a core idea, BUT:
Source: https://madaidans-insecurities.github.io/guides/linux-hardening.html
Note
BUT: defensive configuration matters, and it is typically not included out of the box because the distribution usually has no way of knowing your specific threat model.
Note
Implementation Grade: Easy (depending on your threat model). Be aware of the differences between a rolling release distro or a stable relaseone.
Depending on the linux distribution you choose, you’ll probably have different kernels available to choose from your package manager (see the arch linux ones, nixos, gentoo (where you’re compiling it), etc.
Note
Implementation Grade: Medium. Can be hard to change the original kernel from your linux distro.
Note
Implementation Grade: Hard. You’ve a lot of time if you do that on your own server… But big companies usually do this.
The Linux Security Module (LSM) framework provides a mechanism for various security checks to be hooked by new kernel extensions.
— Linux Kernel Documentation, Linux Security Module (LSM)
In practice, this means that the LSM kernel modules can extend the security model with additional security features and enforcement mechanisms. These modules enable a wide range of capabilities, but in this section we will focus specifically on Mandatory Access Control (MAC) modules and modules used for resource and system-call restriction.
Let’s go to amsa.lol for a brief reminder of linux permissions.
Note
Okey, right now amsa.lol is dead :( the new domain name is called lleida.amsa.udl.cat
Mandatory access control (MAC) is a type of security policy that differs significantly from the discretionary access control (DAC) used by default in Arch and most Linux distributions. MAC essentially means that every action a program could perform that affects the system in any way is checked against a security ruleset. This ruleset, in contrast to DAC methods, cannot be modified by users. Using virtually any mandatory access control system will significantly improve the security of your computer, although there are differences in how it can be implemented. — Arch Linux Wiki, SELinux
Note
Fun fact: Android relies heavily on SELinux to enforce security boundaries and access-control policies.
On most systems, the policy files are stored in /usr/share/selinux/mcs. A SELinux policy usually looks like (example with chronyd, NTP client):
Usually, the policies look like:
Note
SELinux can be difficult to understand and configure. However, if you’re using a Red Hat–based distribution, it’s probably enabled by default. I recommend reading the Gentoo SELinux documentation for a good overview.
SELinux/Apparmor are not magic, it just saves context of the domain inside the “metadata” of the file, and then uses the rules to analyze if we can do the action we want with it.
Note
It is only supported on filesystems that have the extended attributes support (this means, that on a file, we can add “metadata” attributes, LSM’s use this). You can see if your filesystem supports it via:
$ touch file.txt
$ setfattr --name=user.checksum --value="3baf9ebce4c664ca8d9e5f6314fb47fb" file.txt
$ getfattr file.txt
# file: file.txt
user.checksum
Note
Implementation Grade: Easy to enable but a pain to get “non-trivial-things” working.
Landlock is a Linux API that lets applications explicitly declare which resources they are allowed to access. Its philosophy is similar to OpenBSD’s unveil() and (less so) pledge(): programs can make a contract with the kernel stating, “I only need these files or resources — deny me everything else if I’m compromised.” — prizrak.me blog, Landlock-ing Linux
Note
Implementation Grade: Easy to enable but Hard to use (since you’ve to adapt userspace applications to use the LSM SDK).
The kernel allows unprivileged users to indirectly cause certain modules to be loaded via module auto-loading. This allows an attacker to auto-load a vulnerable module which is then exploited. Specific kernel modules can be blacklisted by inserting files into /etc/modprobe.d with instructions on which kernel modules to blacklist.
Note
Implementation Grade: Easy (if you know which protocols / modules aren’t really needed).
/proc is a pseudo-filesystem that contains information about all processes currently running on the system. By default, this is accessible to all users, which can allow an attacker to spy on other processes. To permit users to only see their own processes and not those of other users, you must mount /proc with the hidepid=2,gid=proc mount options (change it on fstab).sysfs is a pseudo-filesystem which provides large quantities of kernel and hw information. sysfs has been the cause of numerous information leaks, particularly of kernel pointers. Whonix’s security-misc package includes the hide-hardware-info script, which restricts access to this directory as well as a few in /proc in an attempt to hide potential hardware identifiers and prevent kernel pointer leaks. This script is configurable and allows whitelisting specific applications based on groups.Note
Implementation Grade: Very Hard (not trivial to create systemd services with this, a lot of programs really use procfs and sysfs…
Systemd can sandbox services so they can only access what they need. Here is an example of a sandboxed systemd service:
[Service]
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectHostname=true
ProtectClock=true
ProtectProc=invisible
MemoryDenyWriteExecute=true
NoNewPrivileges=true
RestrictSUIDSGID=true
SystemCallFilter=write read openat close brk fstat lseek mmap mprotect munmap rt_sigaction rt_sigprocmask ioctl nanosleep select access execve getuid arch_prctl set_tid_address set_robust_list prlimit64 pread64 getrandom
SystemCallArchitectures=native
IPAddressDeny=anyNote
Implementation Grade: Medium (requires some familiarity with systemd options, though LLMs can help with that nowadays).
TODO
Usually, there are CVE’s related with SUID programs (see pkexec, sudo baron, sudo chroot privilege escalation, etc.
Try to avoid them, allways. Find them with (prepend a fourth leading digit (4 for SUID, 2 for SGID) to the standard three-digit permission mode):
Note
Implementation Grade: Easy, although your linux distro should already do this to you.
You can use doas instead of sudo (which has had multiple vulnerabilities in the past, see all the CVE’s here…
It has too much features (even an insult one…), and the 99% of them are useless for the 99% of the users.
Note
Implementation Grade: Medium. Some programs expect sudo to be available, but adding a shell alias should do the trick, as long as they don’t rely on any of sudo’s more advanced features.
Note
Implementation Grade: Easy
Note
Implementation Grade: Easy
Note
Implementation Grade: Medium. Once configured, it works well, but getting there requires more than a simple “install and play” setup.
Note
Implementation Grade: Easy.
ss, netstat, or lsof.Note
Implementation Grade: Easy
Port knocking is a stealth method to externally open ports that, by default, the firewall keeps closed. It works by requiring connection attempts to a series of predefined closed ports. With a simple port knocking method, when the correct sequence of port “knocks” (connection attempts) is received, the firewall opens certain port(s) to allow a connection.
You can see how to enable it and more information about it here.
Be aware that it can infere with your firewall.
Note
Implementation Grade: Hard. It can also be a pain to use in practice. For example, before connecting via SSH, you first need to knock on the required ports in the correct sequence, and only then can you attempt to connect to the machine.
Note
Implementation Grade: Medium (you’ve to configure the jails for each service you want to have fail2ban enabled. see a video here
Physical access changes the threat model.
/boot, firmware, and bootloader configuration still need integrity controls. (put a password on your bootloader, bios, etc).This section was heavily inspired by the madaidans blogpost. It’s a great read, as is his blog post explaining why he considers Linux insecure, although we disagree on some of the points he makes.
You can even do more things on your own email accounts, phone, etc. see this youtube video by wolfgang’s that talks about it
The monitoring stack has three main applications:
%%| echo: false
flowchart LR
hosts[Linux hosts and services]
apps[Applications]
prom[Prometheus]
loki[Loki]
grafana[Grafana]
alerts[Grafana alert rules]
discord[Discord notification channel]
hosts -->|metrics| prom
apps -->|logs| loki
prom --> grafana
loki --> grafana
grafana --> alerts
alerts --> discordThe monitoring stack makes those signals visible before and during an incident.
Is the metrics side of the stack.
Examples of metric-style questions:
Is the log side of the stack.
Example log questions:
Is the UI for monitoring.
TODO: Add a github repo with an example for using it, configuring it, etc.
Note
Be aware that this kind of deployment it’s hard to mantain and setup. It’s usually usefull when you’ve a lot of machines.
You can see this FOSDEM video talking about their big infrastructure, and how grafana/loki/prometheus make sense there.
An IDS is an Intrusion Detection System:
An IPS is an Intrusion Prevention System:
Many devices combine IDS, IPS, and firewall features.
Note
It is different from the architecture showed before (grafana) in the sense that it’s just plug and play, it just works. no need to configure a lot of stuff. Also, it analyzes the content of network packages, etc.
An IDS can combine several monitoring functions:
| Input | Legacy slide label |
|---|---|
| Application logs | Log monitors |
| System state | Integrity checkers |
| Behavior patterns | Armoring |
| Network traffic | Sniffers |
| Model | How it works |
|---|---|
| Distributed | Installed on hosts, analyzes local information, detects problems, reports to a coordinator |
| Centralized | Hosts send information to a global coordinator, which analyzes the data and detects problems |
| Type | Where it looks | What it monitors |
|---|---|---|
| HIDS | Host | Local user activity, local program activity, system logs |
| NIDS | Network | Anomalous network activity, often using deep packet inspection |
Network IDS can also be host focused.
| Firewall | IDS |
|---|---|
| Prevents intent | Reacts to intent |
| Static control point | Dynamic monitoring point |
| Door, lock, fence, gate | Alarm or motion detector |
| Stops traffic by policy | Raises evidence and alerts |
| Model | How it works | Strength | Weakness |
|---|---|---|---|
| Signature based | Compares activity to known bad signatures | Fast, low false positives | Polymorphic payloads, 0-days |
| Anomaly based | Learns normal behavior and alerts on deviation | Better fit for unknown attacks | High false positives, needs labeled data |
Anti-virus software is a signature-based HIDS example.
Traditional IDS evaluation uses the same confusion matrix terms as machine learning:
| Term | Meaning |
|---|---|
| True Positive | A real attack is detected |
| True Negative | Good traffic is detected as good |
| False Positive | Regular traffic is tagged as an attack |
| False Negative | An attack is not detected |
TP / (TP + FN)TP + FN is the total number of attacks.FP / (FP + TN)FP + TN is the total size of good traffic.Operationally:
Suppose:
For 1000 events:
P(attack | alarm) = 9 / (9 + 99) = 8.3%
An alarm is false about 92% of the time in this example.
Techniques from the legacy slides:
IDS work is heavier than firewall work, and it needs access to useful traffic.
Deployment options:
| Tool | Notes from legacy slides |
|---|---|
| Snort | Signature based IDS, can act as IPS, large community |
| Suricata | Signature based IDS, can act as IPS, modern architecture |
| Security Onion | Linux distribution with IDS and analytics tools |
We only need the operational view:
-v-l-cImportant paths:
/etc/snort//etc/snort/snort.conf/etc/snort/rules//var/log/snortTODO: Put video: https://www.youtube.com/watch?v=z1hX2PT3Cac
A honeypot is a trap set to detect, deflect, or counteract malicious attempts.
Its value comes from being used wrongly:
Honeypots can be used:
A useful honeypot:
Honeypots can be classified by:
| Low interaction | High interaction |
|---|---|
| Emulates only services or operating-system behavior | Exposes real services and operating systems |
| Cannot usually be fully exploited | Can be fully compromised |
| Easier to constrain | More realistic and more risky |
| Attack data is limited to the emulated level | Attackers may use it for further penetration attempts |
| Physical honeypot | Virtual honeypot |
|---|---|
| Real machine deployed as the honeypot | Simulated by another machine |
| Own IP addresses and services | Many honeypots can run on one server |
| Usually high interaction | “Virtual” does not necessarily mean VirtualBox |
| Production | Research |
|---|---|
| Deployed in a real production network | Usually high interaction and complex |
| Usually low interaction | Used to discover methods, tools, tactics, and motives |
| Deflection or early warning | Forensics and analytics, not deflection |
| Should be easy to deploy and pull offline | Captures extensive data |
The legacy slides note legal uncertainty:
For course purposes, treat honeypots as monitored defensive and research systems, not as legal traps.
A honeynet is a network of cooperating honeypots.
Client honeypots reverse the usual model:
HoneyMonkey was a Microsoft research project using reversed honeypots.
| Resource | Use |
|---|---|
| Honeyd | Monitors unused IP access, takes over the IP, creates a virtual honeypot |
| Project Honeypot | Crowdsourced effort to identify spam generators |
| Low-cost deployments | SBC-based honeypots or honeynets |
| Awesome honeypots | Curated list of honeypot tools |
Links:
Repositories age after every commit:
Dependabot or a similar service makes dependency drift visible in GitHub repositories.
Manual dependency review does not scale well.
| Without automation | With Dependabot or similar |
|---|---|
| Vulnerable versions may stay unnoticed | Pull requests or alerts make updates visible |
| Teams rely on memory | The repository records pending maintenance |
| Security updates compete with feature work | Dependency work becomes part of normal review |
The same defensive pattern applies to AWS, GCloud, or similar architectures:
The tool names change; the Blue Team loop stays the same.
See: Kics: https://www.youtube.com/watch?v=oozrW1tiV8w
Deleting a file usually removes filesystem metadata before it destroys every byte of file content.
File carving tries to recover useful data from the underlying storage bytes after the filesystem no longer shows the file normally.
Important idea:
The source asks for this demo sequence:
sync.Use placeholders. Confirm the device with lsblk before running destructive commands.
Warning
mkfs.ext4 destroys the filesystem on the selected device. Never run it until the target is confirmed.
At this point the directory entry is gone, but the raw device may still contain the marker.
This is a minimal carving demonstration: search the raw device for content that the filesystem no longer lists.
shred overwrites file contents before removing the file.
Why it is useful:
The point of the demo is the contrast with normal deletion: after overwrite, the simple raw-byte search should no longer find the marker.
A Linux operating system is files on disk.
If we copy the contents of a disk to another disk or image file, we can analyze it later:
Encrypt it with LUKS. Be aware that then you must be able to type the password to enter. Or, if not, use a TPM.
The source leaves this as an open question.
Use it to connect the course blocks:
A SIEM is the natural place to ask how these signals could be correlated, but product-specific detail is outside the current source.
src/resources/past_courses/slides/ids-trans.pdfsrc/resources/past_courses/slides/honeypots-trans.pdf
Cybersecurity - 🏠Home