Under the Hood: A System Investigator's Journey Through the Linux File System

What is Linux?

Linux is an open-source operating system modeled on UNIX. It powers servers, cloud infrastructure, containers, and much of the modern internet.

At its core:

• Kernel — interacts with hardware

• Shell — interface for user interaction

The key idea:

Linux exposes system state and control through the filesystem.

---

File System Hierarchy

Everything in Linux starts from the root /.

Common directories:

/bin   # essential binaries
/etc   # configuration files
/home  # user directories
/usr   # installed software
/var   # logs and variable data
/tmp   # temporary files
/dev   # devices
/proc  # process and kernel info

This hierarchy is not just structure—it is the operating system’s interface.

Finding 1: Routing Table — /proc/net/route

cat /proc/net/route

Raw output:

Iface        Destination  Gateway
57a5849767-v BA000415     00000000

These values are little-endian hex IPs.

Example decoding:

import socket, struct
socket.inet_ntoa(struct.pack('<I', int("BA000415", 16)))

Insight

• Tools like ip route read this file

• Interface names reveal environment (e.g., containers)

---

Finding 2: Process Capabilities — /proc/1/status

cat /proc/1/status

Key field:

CapEff: 00000000a82c35fb

Linux uses capabilities instead of all-or-nothing root:

• CAP_NET_ADMIN — network control

• CAP_SYS_ADMIN — broad system access

• Missing capabilities restrict power

Insight

Containers work by limiting capabilities, not removing root entirely.

---

Finding 3: Cgroups — Resource Limits

cat /proc/self/cgroup

Check memory:

cat /sys/fs/cgroup/memory/memory.limit_in_bytes
cat /sys/fs/cgroup/memory/memory.usage_in_bytes

Insight

• Cgroups enforce CPU and memory limits

• Container runtimes are just interfaces over this

---

Finding 4: Permissions — SUID, SGID, Sticky Bit

find /usr/bin -perm /4000

Examples:

• /usr/bin/passwd

• /usr/bin/su

Check sticky bit:

ls -ld /tmp
# drwxrwxrwt

Insight

• SUID enables privilege escalation by design

• Sticky bit prevents cross-user file deletion

• Filesystem permissions are the primary security layer

---

Finding 5: /etc/shadow — Access Control

ls -l /etc/passwd /etc/shadow

/etc/passwd → readable
/etc/shadow → restricted

Insight

Security relies on file permissions rather than encryption.

---

Finding 6: Devices — /dev

ls /dev

Examples:

Device	Purpose
/dev/null	discard output
/dev/zero	infinite zeros
/dev/random	secure randomness
/dev/full	always fails

Test:

echo test > /dev/full

Insight

Devices and behaviors are abstracted as files.

---

Finding 7: Boot Process

ls /boot
cat /proc/cmdline

In containers:

• /boot may be empty

• Kernel is shared

Real sequence:

1. BIOS/UEFI

2. Bootloader (GRUB)

3. Kernel

4. Init (PID 1)

Insight

Containers skip boot entirely.

---

Finding 8: Systemd — Service Management via Files

ls /etc/systemd/system/

Enable a service:

systemctl enable nginx

Creates a symlink inside *.wants/.

Insight

Service state is controlled through filesystem structure.

---

Finding 9: Mounts — Hidden Access Control Layer

cat /proc/mounts

Example:

none /dev/shm tmpfs rw,noexec,nosuid

Useful commands:

mount | column -t
findmnt
df -h

Advanced:

cat /proc/self/mountinfo

Mount flags:

Flag	Meaning
ro	read-only
noexec	cannot execute
nosuid	ignore SUID

Insight

Mount options override file permissions and are critical for isolation.

---

Finding 10: Memory Layout — /proc/self/maps

cat /proc/self/maps

Observations:

• r-xp → executable

• rw-p → writable

• No region is both writable and executable

Insight

W^X policy prevents direct code injection.

---

Finding 11: TCP Connections — /proc/net/tcp

cat /proc/net/tcp

States:

• 01 → ESTABLISHED

• 0A → LISTEN

Better view:

ss -tulnp

Insight

Networking state is exposed as readable data.

---

Finding 12: Environment Configuration

cat /etc/environment
cat /proc/self/environ

Insight

Environment is layered across system and user configs.

---

Finding 13: Drop-in Configuration Pattern

ls /etc/profile.d/
ls /etc/cron.d/
ls /etc/systemd/system/

Insight

Linux avoids monolithic configs in favor of modular directories.

---

Finding 14: Network Debugging Tools

ping — Connectivity

ping google.com

Checks:

• reachability

• latency

• packet loss

---

dig — DNS Resolution

dig google.com

Query specific DNS:

dig @8.8.8.8 google.com

Insight

DNS is a simple query-response system.

---

ssh — Remote Access

ssh user@host

Insight

SSH provides encrypted remote shell access and underpins most cloud operations.

---

Useful but Less Common Commands

lsof — Open Files

lsof -i :3000

---

strace — System Calls

strace ls

---

netcat (nc) — Raw Networking

nc -l 8080

---

watch — Repeated Execution

watch -n 1 "cat /proc/meminfo"

---

findmnt — Better Mount View

findmnt

---

ss — Socket Statistics

ss -tulnp

---

which / type — Command Resolution

which python
type ls

---

Final Insight

Across all observations:

• Processes → /proc

• Network → /proc/net

• Devices → /dev

• Configuration → /etc

• Resources → /sys

The filesystem is Linux’s universal interface.

Commands are abstractions. The files are the source of truth.

---

Conclusion

Understanding Linux at this level changes how you debug and reason about systems:

• You rely less on tools

• You inspect the system directly

• You understand why things work (or fail)

Nothing in Linux is hidden. It is all exposed—you just need to read it.