✧ storage & filesystems ── 67 terms in plain english ── ♡ no jargon, no hype ♡ ── キラ✧キラ ── ✧ storage & filesystems ── 67 terms in plain english ── ♡ no jargon, no hype ♡ ── キラ✧キラ ──
spinning rust, flash cells, journals, and the filesystems built on top. why ext2 still has a job.
$ grep -i ""
67 terms
01 Spinning Rust
A hard drive is a record player that never stops, and almost every filesystem idea below exists because of how it physically moves.
Platter
aka disk
The actual spinning magnetic disc. A drive has several stacked on one spindle, usually spinning at 5400 or 7200 RPM (10k and 15k enterprise drives existed and have largely been killed off by SSDs). Both surfaces of each platter store data.
Track
One concentric ring of data on a platter surface. Not a spiral — unlike a vinyl record, tracks are separate closed circles, so the head can jump between them without following a groove.
Sector
aka block
The smallest chunk a drive will read or write, historically 512 bytes, now almost always 4096. This is the atom of storage: you cannot read half a sector, and the drive rewrites a whole one even if you changed a single byte.
Cylinder
The same track number across every platter, stacked vertically. It matters because the heads all move together on one arm — everything in a cylinder is reachable without seeking, which is why old filesystems tried hard to keep related data in one.
Seek time
How long the arm takes to move to the right track — typically 4–12 ms. It is a physical motion, and it is the reason random I/O on a hard drive is catastrophically slower than sequential I/O. Not a little slower. Two orders of magnitude.
🔮Seeking is walking to a different shelf in the library. Sequential reading is taking the next book along from the one you just finished.
Rotational latency
After the head reaches the track, the wait for the data to come around under it. At 7200 RPM a full turn is ~8.3 ms, so on average you wait about 4.2 ms — half a rotation — for nothing but physics.
Old drives were addressed by their geometry: which cylinder, which head, which sector. LBA replaced that with a flat numbered list of blocks and let the drive keep its real layout private. Every modern drive lies about its geometry — the number you see is a fiction the firmware maintains.
Zone bit recording
aka ZBR
Outer tracks are physically longer, so they hold more sectors than inner ones. This is why a disk benchmark shows higher throughput at the start of a drive than at the end — a fact that used to drive short-stroking, where you deliberately used only the fast outer portion.
Advanced Format
aka 512e / 4Kn
Drives moved to 4096-byte physical sectors for efficiency, but many old tools assumed 512. 512e drives emulate the old size: a misaligned 512-byte write forces the drive to read a 4K sector, modify it, and write it back. Get your partition alignment wrong and you pay that penalty on every single write.
SMR vs CMR
aka Shingled / Conventional Magnetic Recording
SMR overlaps tracks like roof shingles to cram more in. Reads are fine; rewriting a track means rewriting everything overlapping it, so sustained random writes fall off a cliff — sometimes to single-digit MB/s. Several vendors shipped SMR drives into NAS and RAID products without saying so, and rebuild times exploded. Check before you buy for an array.
🔮CMR is writing on ruled paper. SMR is writing on overlapping sticky notes — fine until you need to change one in the middle of the stack.
02 Flash & SSDs
No moving parts, so no seek time — but flash has a genuinely strange constraint: it cannot overwrite anything in place.
NAND cell
The physical thing that stores your data: a transistor holding a trapped electric charge. How much charge it is holding is read back as bits. Charge leaks slowly, which is why an unpowered SSD eventually forgets — years when new, potentially months when heavily worn.
SLC / MLC / TLC / QLC
aka bits per cell
How many bits are squeezed into one cell. SLC stores 1 (2 charge levels), MLC 2 (4 levels), TLC 3 (8 levels), QLC 4 (16 levels). Same physical cell, subdivided more finely each time — so the voltage windows get narrower, reads get slower and need more error correction, and the cell wears out faster. Rough endurance: SLC tens of thousands of write cycles, MLC a few thousand, TLC ~1000, QLC in the hundreds.
🔮Reading a cell is judging how full a glass is. SLC asks "empty or full?" QLC asks you to tell sixteen levels apart — in a glass that is slowly evaporating.
Where you meet themQLC: cheap bulk storageTLC: sensible defaultMLC/SLC: endurance
Page & block
aka the asymmetry
Flash is read and written in pages (a few KB) but erased only in blocks (many pages, often megabytes). You cannot overwrite a page in place — the controller must write elsewhere and erase the old block later, in bulk. Every strange SSD behaviour below follows from this one fact.
P/E cycle
aka program/erase
One write-then-erase of a block. Each one physically degrades the insulating layer, and a cell has a finite budget. This is what "wearing out" an SSD means, and it is why write-heavy workloads are the ones that kill drives.
Wear levelling
The controller spreads writes across all cells so no block dies while others sit unused. It is why you cannot meaningfully "overwrite" a file on an SSD — your new data lands on different silicon, and the old copy sits there until garbage collection gets to it.
Write amplification
aka WAF
The ratio between what you asked to write and what the drive actually wrote. Change 4 KB, and the controller may have to relocate a whole block to do it — so 4 KB in, megabytes written. This is the number that decides how long your drive lives, and it is why a journaling filesystem that writes everything twice is not free.
TRIM
aka discard
Deleting a file only removes it from the filesystem's records — the drive still thinks those blocks matter and keeps dutifully preserving them. TRIM tells the drive "these are garbage now." Without it, the drive slowly fills with data nobody wants, garbage collection thrashes, and write performance decays.
Over-provisioning
aka OP
Spare capacity the drive keeps for itself and never advertises — a 512 GB drive may hold 20% more flash than it sells you. It gives garbage collection room to work. Enterprise drives reserve far more, which is much of why they cost more and last longer at the same capacity.
SLC cache
aka pseudo-SLC, turbo write
QLC and TLC drives run part of their flash in fast 1-bit mode as a write buffer, then fold it down into dense storage later. It is why a cheap drive benchmarks brilliantly for the first few gigabytes and then collapses — sometimes below hard-drive speeds — when the cache fills. Benchmark with a file bigger than the cache or you are measuring a lie.
🔮A shop assistant piling deliveries by the door because it is quick, then having to shelve it all properly once the doorway is full.
DRAM cache & HMB
aka DRAM-less
The drive's own memory, used to hold the map between logical blocks and where they physically live. DRAM-less drives borrow a slice of your system RAM (Host Memory Buffer) or just do without, and they fall apart under random I/O. This is the spec most cheap drives quietly omit.
Endurance rating
aka TBW, DWPD
TBW is total terabytes writable over the warranty; DWPD is how many times you could rewrite the whole drive daily. Consumer drives are rated around 0.3 DWPD, write-intensive enterprise drives 3–10. For most desktops this is a non-issue; for a database or a Ceph journal it is the whole purchasing decision.
03 Filesystem Fundamentals
Everything above is just numbered blocks. A filesystem is the bookkeeping that turns them into files with names.
Inode
aka index node
The record holding everything about a file except its name: size, owner, permissions, timestamps, and where the data blocks live. The name lives in the directory, which is why one file can have several names (hard links) and why deleting a name does not delete a file that another name still points at.
Superblock
The filesystem's own header — size, block count, where everything begins. Lose it and the volume is unreadable, which is why filesystems keep backup superblocks scattered across the disk. fsck -b exists to reach for one.
Block
aka cluster, allocation unit
The filesystem's own smallest unit, usually 4 KB, built on top of the drive's sectors. A 1-byte file still consumes a whole block — the reason a folder of tiny files uses far more disk than the sum of its file sizes.
Extent
A run of consecutive blocks recorded as "start here, continue for N blocks" instead of listing every block individually. A 1 GB file might need one extent rather than a quarter-million block pointers. XFS was built on extents from day one; ext4 added them, and it is the single biggest reason ext4 beats ext3 on large files.
Metadata vs data
Data is your file contents. Metadata is everything describing it — inodes, directories, free-space maps. The distinction matters enormously below: most journals protect only metadata, which is a much weaker promise than most people assume they are getting.
Fragmentation
When a file's blocks end up scattered instead of consecutive. On a hard drive this is a real performance problem because every fragment is another seek. On an SSD, with no seek penalty, it barely matters — which is why defragmenting an SSD is pointless and actively harmful, since it burns P/E cycles to fix a problem you no longer have.
04 Journals: What They Actually Promise
The most misunderstood feature in storage. A journal is not a backup, and by default it is not protecting your file contents.
The problem
aka why journals exist
Writing a file touches several structures — the inode, the free-space map, the directory, the data itself. A crash halfway through leaves them disagreeing: blocks marked used that no file claims, or worse, claimed by two files at once. The filesystem is not corrupt in a dramatic sense. It is merely lying, and it will keep lying until something checks.
Journal
aka write-ahead log
A reserved area where the filesystem writes what it is about to do before doing it. After a crash it replays or discards the log, and the structures are consistent again — in seconds, regardless of how big the volume is. That last part is the whole point.
🔮Writing your intended move on a notepad before touching the chessboard. Knocked table, no argument: the notepad says where the pieces were going.
fsck
aka file system check
The alternative to a journal: walk every inode and every block and rebuild the truth by inspection. It works, and on a large volume it takes hours — during which your server is down. Journals did not make filesystems safer so much as they made recovery fast. That is the actual innovation.
data=ordered
aka the default
ext3/ext4 default. Metadata is journalled, and data blocks are forced to disk before the metadata that references them. So you never see a file pointing at someone else's old data — the classic security hole — but a partly-written file can still end up partly written. Sensible balance, and what almost everyone runs.
data=journal
aka full data journalling
Everything, contents included, goes through the journal. Genuinely safer, and it means every byte is written twice. Slower, and on flash it doubles your write amplification. Reach for it when correctness beats throughput and you have thought about the wear.
data=writeback
Metadata journalled, data written whenever. Fastest and the loosest: after a crash a file can contain stale blocks from a deleted file — someone else's data. Rarely worth it.
What a journal does not do
It does not protect against bit rot, a dying disk, an accidental rm, or a lying drive that claims a flush completed when it has not. It ensures the filesystem's bookkeeping is self-consistent after a crash. That is all it was ever for, and it is not a backup.
Why no journal is sometimes right
aka ext2 lives
A journal costs extra writes on every operation. Without one you get less write amplification, less wear on cheap flash, and a simpler on-disk format that more tools can repair. That is why ext2 still turns up on small removable media and old /boot partitions: they are small enough that a full fsck takes seconds, and they are written to rarely. The tradeoff is genuinely reasonable — it is just narrow. (You can also get the same effect on modern kernels with tune2fs -O ^has_journal.)
🔮A journal is insurance with a monthly premium. On a volume you almost never write to, and can rebuild in ten seconds, you may rationally choose to self-insure.
05 The ext Family
Linux's default for three decades, and a clean illustration of how filesystems grow one problem at a time.
ext2
aka 1993
No journal. Fast, simple, extremely well understood, and after an unclean shutdown it needs a full fsck whose duration scales with the number of inodes — the flaw that made everything after it necessary. Still perfectly good for small, rarely-written, easily-rechecked volumes.
ext3
aka 1998
ext2 plus a journal, and — the clever part — the same on-disk layout, so you could convert in place without reformatting and mount it as ext2 if you needed to. That upgrade path is most of why it took over. It inherited ext2's block-list allocation, so large files stayed slow.
ext4
aka 2008
The modernisation: extents instead of block lists, delayed allocation, multi-block allocation, 64-bit sizes, faster fsck, and optional metadata checksums. Still the sensible default for a single-disk Linux system, and deliberately conservative — it is the filesystem chosen by people who would rather be bored.
What it addedextentsdelayed allocmetadata_csum
Delayed allocation
aka allocate-on-flush
Do not decide where a file goes until you are actually flushing it — by then you know the final size and can place it in one contiguous run. Great for fragmentation. It also caused ext4's most infamous bug class: applications that wrote a file and renamed it over the original without fsync could find a zero-length file after a crash. ext4 added heuristics to paper over it; the applications were wrong, but the filesystem got the complaints.
06 XFS, and What SGI Got Right
Designed in 1993 for machines most people would not see for twenty years — and it aged better than almost anything of its generation.
XFS
aka SGI, 1993
Built at Silicon Graphics for IRIX, where the workload was uncompressed video and scientific data on expensive multi-processor machines. Open-sourced in 2000 and merged into Linux in 2001. It is now the default in RHEL and a very common choice for large volumes.
What SGI got right
aka the design bets
Three calls that looked like overkill in 1993 and turned out to be exactly right: • 64-bit from the start — no painful capacity ceiling to retrofit later • Extents and B+ trees everywhere — free space, extents and inodes all live in balanced trees, so performance degrades gracefully instead of falling off a cliff as the volume grows • Allocation groups — the filesystem is split into near-independent regions that can be allocated into in parallel, because SGI was building for many-CPU machines They designed for a scale that did not exist yet, and the industry grew into it.
🔮Building a motorway with eight lanes in a town of a thousand people. Ridiculous — right up until the town becomes a city, and everyone else is widening roads while yours already works.
Allocation group
aka AG
XFS divides a volume into large regions, each with its own free-space and inode structures. Two CPUs writing different files can allocate at the same time without contending for one global lock. This is why XFS scales with cores and threads where older designs serialise.
Dynamic inode allocation
ext creates all its inodes at mkfs time — so you can run out of inodes with free space remaining, and the only fix is a reformat. XFS creates them as needed, which is why that particular humiliation does not happen there.
xfs_repair
aka not fsck
XFS's repair tool, and the philosophy differs: it expects the journal to have handled crashes and is meant for genuine damage. It is fast and memory-hungry. Notably XFS cannot be shrunk — grow only. If your volumes ever need to get smaller, that alone may decide the question for you.
ext4 vs XFS
aka the honest comparison
For a laptop or a single disk they are close enough that it rarely matters. XFS pulls ahead on large files, parallel I/O and very large volumes; ext4 has historically been better with huge numbers of small files and metadata-heavy workloads, and it can shrink. Pick XFS for the media server and the database volume, ext4 for the root filesystem and anything you might resize.
Rule of thumbXFS: big files, many coresext4: small files, resizeable
07 ZFS: The Filesystem That Does Not Trust Anything
It merges the volume manager, the RAID layer and the filesystem — which sounds like a layering violation, and is exactly what makes it work.
Copy-on-write
aka CoW
Never overwrite live data. Write the new version elsewhere, then atomically switch the pointer. The on-disk state is always valid — there is no window in which a half-updated structure exists, so ZFS needs no journal for consistency and has no fsck at all.
Checksums
aka the actual point
ZFS checksums every block, data included, and verifies on every read. ext4 and XFS checksum only metadata — they will hand you silently corrupted file contents without a word. With redundancy, ZFS does not just detect the damage, it repairs it from the good copy and rewrites the bad one. This is the feature; everything else is scaffolding.
🔮Every page of the book has a checksum in the margin, and a second copy sits on the next shelf. Notice a smudged page, fetch the twin, fix the original.
Pool & vdev
aka zpool
A pool is the storage you use. It is built from vdevs, and the vdev — not the pool — is where redundancy lives. This is the single most consequential thing to understand: losing one vdev loses the entire pool, regardless of how healthy the others are. A pool is striped across its vdevs with no redundancy between them.
RAIDZ
aka RAIDZ1/2/3
ZFS's parity RAID, tolerating one, two or three failed disks per vdev. Because of CoW and variable-width stripes it avoids the RAID write hole that can corrupt traditional RAID5 on a power cut mid-write. Rebuilds (resilvers) copy only live data, so a half-empty array resilvers in half the time.
ARC & L2ARC
aka Adaptive Replacement Cache
ZFS's read cache in RAM, and it is aggressive — it will use most of your free memory, which alarms people the first time they see it. It gives it back under pressure. L2ARC extends it onto an SSD, and is far less often worth it than people assume: it costs RAM to index, so a small-memory machine can be made slower by adding one.
ZIL & SLOG
aka the most misunderstood pair
The ZIL is where synchronous writes are logged so they can be honoured after a crash. A SLOG moves that log to a fast dedicated device. The near-universal misconception: this is not a write cache. Asynchronous writes — most of what a desktop does — never touch it. If your workload is NFS or a database issuing syncs, a SLOG transforms it. Otherwise it does nothing at all.
Scrub
Read every block, verify every checksum, repair what redundancy allows — while the pool stays online. Run it monthly. This is how you find bit rot before it finds you, and it is the routine that turns ZFS's checksums from a nice property into an actual guarantee.
Snapshots & datasets
CoW makes a snapshot nearly free — you simply keep the old pointers. A dataset is a filesystem inside the pool with its own settings (compression, quotas, snapshot schedule), and they cost nothing, so make lots. lz4 compression is on by default in most builds and is usually a speed win: less data to move beats the CPU cost.
Dedup
aka usually a trap
Deduplication sounds wonderful and demands roughly 1–5 GB of RAM per TB of pool, permanently, whether it saves you anything or not. Undoing it means rewriting everything. Compression gets you most of the benefit for almost none of the cost. Say no unless you have measured.
08 Ceph: Storage as a Distributed System
Not a filesystem on a disk — a cluster that survives whole machines dying, and asks a real operational price for it.
RADOS
aka the foundation
The object store everything else is built on: Reliable Autonomic Distributed Object Store. Block devices, a POSIX filesystem and an S3 endpoint are all thin layers over the same pile of objects.
OSD
aka Object Storage Daemon
One process per disk, responsible for actually storing objects, and for replicating, checking and healing them. A hundred-disk cluster runs a hundred OSDs. They talk to each other constantly, which is why Ceph is as much a network product as a storage one.
MON
aka Monitor
Keeps the authoritative cluster map and the membership consensus. Runs in an odd number — three or five — because it needs a quorum. Lose quorum and the cluster stops serving rather than risk splitting in two and diverging.
Placement Group
aka PG
A shard sitting between objects and OSDs. Objects hash into PGs, PGs map onto OSDs. Without them, tracking placement per object across billions of objects would be impossible. PG count is the classic Ceph tuning knob and the classic way to get it wrong.
CRUSH
aka the clever bit
The algorithm that decides where data lives — and it is computed, not looked up. Any client can work out which OSDs hold an object from the cluster map alone, so there is no metadata server in the data path and no central bottleneck. You also describe your physical layout (racks, hosts, rooms) and CRUSH will refuse to put all replicas in one rack.
🔮Instead of a librarian who knows where every book is — and who becomes a queue, and eventually dies — everyone is handed the same shelving rules and derives the location themselves.
Replication vs erasure coding
Replication (typically 3×) is simple, fast to recover, and costs you two thirds of your raw capacity. Erasure coding is RAID-like parity across nodes — far better space efficiency, at the cost of CPU and latency, and much heavier recovery traffic. Replicate the hot and small; erasure-code the big and cold.
RBD, CephFS, RGW
aka the three faces
RBD gives virtual machines block devices. CephFS is a POSIX filesystem (and the one component that does need metadata servers). RGW speaks the S3 API. Same objects underneath, three different shapes.
When Ceph is wrong
aka the honest bit
It wants multiple machines, fast networking and an operator who understands it. Below roughly three nodes it is worse than a single ZFS box in every respect: slower, more fragile, and vastly more to go wrong. Ceph buys you survival of whole-machine failure and growth past one chassis. If you do not need that, you are paying a large complexity bill for nothing.
09 Failure, Rot & Lies
The concepts that explain why the paranoid designs above exist at all.
Bit rot
aka silent data corruption
A stored bit changes without anyone writing to it — cosmic rays, charge leakage, a marginal cell, a flaky cable. Nothing reports an error. The read succeeds and returns the wrong bytes. This is what checksumming filesystems exist to catch, and what everything else silently hands you.
URE
aka Unrecoverable Read Error
The rate at which a drive fails to read a sector it should have been able to, quoted as roughly 1 in 10¹⁴ bits for consumer disks. Once drives got large, a full-array read during a RAID5 rebuild became statistically likely to hit one — while already degraded. That arithmetic is why RAID5 fell out of favour for large disks and why RAID6/RAIDZ2 became the sane default.
RAID write hole
Traditional parity RAID updates data and parity separately. Lose power in between and they disagree — with no record of which is right. The array is now quietly wrong, and a later rebuild will faithfully reconstruct garbage. ZFS closes this with CoW and variable stripes; hardware controllers used a battery.
Barriers, flush & fsync
aka the trust problem
A filesystem's crash guarantees depend entirely on the drive telling the truth when it says data has hit stable storage. Some consumer drives have historically lied — reporting a flush complete while the data sat in volatile cache — for benchmark numbers. Every journal, every CoW transaction, every database durability promise rests on that one honest answer.
🔮Every safety mechanism above is a contract. The disk is the counterparty, and a few of them have been known to forge the signature.
SMART
The drive's self-reported health telemetry. Useful, but a famous Backblaze study found a large fraction of failed drives showed no warning at all. Watch reallocated sector count, pending sectors and wear levelling — and never mistake a clean SMART report for a working backup.