Layman's Guide to Computing - Season 09

Issue 117: Swap space

2021-04-24T08:00:00+08:00

Previously: Hibernation mode causes the computer to store the data configuration into a hibernation file on disk. When powered up, the OS reads the data configuration from the file back into memory. This lets the system avoid having to do a full shutdown and bootup; it performs a shorter version of these two sequences instead.

In Issue 57), I laid out the transfer speeds and latencies for a few places in the computer where data can be stored:

Hard disk drive (HDD): ≈5 ms response latency, 100 MB/s transfer speed
Solid state disk (SSD): up to 0.1 ms response latency, 0.5–1+ GB/s transfer speed
Physical memory (RAM): 0.1 µs (0.0001 ms) response latency, 20GB/s transfer speed
CPU register: <1 ns (<0.000001 ms) response latency

Apart from the CPU itself, physical memory is one of the fastest places to store and retrieve data. It’s just such a pity that there’s a limited amount of it, and you really can’t get more than 32GB of it in a single computer (for future readers, this is written in early 2021). If you bought a cheaper computer that only has 8GB or only 4GB, this quickly limits the number of apps you can have open at any single moment.

What happens when you run out of memory?

Running out of memory

When a program needs more memory, it requests it from the operating system (OS), and waits for the OS to give that memory. After all, it can’t do anything else until the memory is available.

The OS, on the other hand, will give that memory if it is available. If not, it will wait for other programs to free up memory before passing that memory to the waiting program. If no program is willing to release memory …

I guess the computer just hangs 🤷

Mitigating out-of-memory problems

If memory is limited, what can we do to increase it?

As it turns out, not all parts of memory are constantly being written to or read from. Lots of it is just sitting there, waiting for that one moment when the data is needed. Kind of wasteful if out of 8GB of memory, only 2GB is actively changing in any hour.

The OS could write that mostly-static 6GB to disk and free it up for other programs, maybe? Then when the data is needed, the OS reads it back from disk before passing it to the program. Yes, this would mean a slower response time for programs that have been idle for a while, but surely better than hanging because it ran out of memory?

This is what all modern OSes do. They write unused in-memory data to a file known as the page file. In Windows, the page file is C:\pagefile.sys. When data is moved from physical memory to the page file, it is said to be paged out, and when moved from page file to physical memory, it is paged in.

In older systems or software, you may also see this page file referred to as a swap file, or swap space. On Linux, a subregion of storage space can be set aside as a swap partition.

Why is it called the page file?

Computer memory is organised into pages, each page typically being 4 KiB (notice that disk sectors are also typically 4 KiB (Issue 106)) …). The computer may have only 8GB of physical memory, but present 16GB of virtual memory (Issue 56)) to programs. Kind of like how banks only hold some liquidity but present their assets as being much more …

So the 8GB “shortfall” is actually in the pagefile, not in memory. The pagefile essentially acts like (much slower) memory! The OS reads pages from it and writes pages to it, through virtual memory accesses.

This sounds like a waste of space, can I choose not to use it?

Though most OSes are configured to use a page file by default, you can configure it to not do so, although this is ill-advised (and I won’t explain why here; this is a newsletter aimed at layfolks). Modern OSes are pretty smart at managing page files, understanding that it is much slower than physical computer memory. They have their own algorithms to decide when to move data to a pagefile, and these days they even forecast when data that was paged out will soon be needed by the program.

OSes need page space for all kinds of reasons: as a kind of “emergency space” when physical memory is full, as working space for optimising data layout in memory, etc. So that disk space is definitely not going to waste!

Issue summary: Operating systems use a page file on the storage disk as a complement to physical memory. This allows OSes to behave more performantly than they would if they did not have a page file. Data that is rarely accessed is moved to the pagefile (“paged out”), and can be paged in when it is needed later, albeit with a performance hit.

Pagefiles are pretty amazing, even though they used to be a huge pain in older systems, when they were stored on slowwww hard disks. These days, page files stored on solid state disks are pretty fast! While the performance hit of a page miss is still noticeable (like when you switch to a Chrome tab that you haven’t touched in a while), it is far from the groan-inducing wait it used to be.

What I’ll be covering next

Next issue: [LMG S10] Issue 118: When I run two file-copy processes at the same time, why are they much slower?

Next week, a new season! With operating systems much more fleshed out for you (I hope), I can finally delve into the nuts and bolts of the machine itself: the hardware.

Since this is a layman’s guide, I’m going to focus on the interesting bits, the parts that actually answer long-standing questions :)

Let’s start with a bang: sometimes you are copying a file from one flash drive to your external hard disk. And you need to copy another file from your external hard disk to the same flash drive (or even a different one). You’re scared; can the computer handle it? But this is taking so long; let’s just try it anyway.

And you notice the speed suddenly plummets. Why does that happen? Even with 117 issues of Layman’s Guide written, I don’t have enough background laid down to explain this. So let’s delve into the hard disk in the first issue of Season 10.

Sometime in the future: What is:

XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
OpenType? And what are fonts anyway? [Issue 42]
a driver file and why do I need one? [Issue 98]
a video card? [Issue 113]

Issue 116: Hibernation

2021-04-17T08:00:00+08:00

Previously: When you shut a computer down, it sends an exit signal to all running programs to get them to do their exit routine. This process can sometimes take a long time. To preserve the data configuration in memory while minimising power draw, a computer can go into standby mode: all hardware except the memory gets powered down, until the computer is woken up from standby.

Okay, so standby/sleep mode lets you keep the data configuration in memory by putting the computer into a sort of low-power mode, where only the memory still has power while the rest of the computer is shut down.

If you’re not just going for a short jaunt; you’re maybe getting on a flight and don’t have the laptop charger with you and want your laptop to still be in the same configuration (windows open, Spotify playing, etc) without drawing any power … is that possible?

By now, you know the answer is yes! You just put the computer in hibernation mode!

Hibernation

How is it possible to preserve data configuration in memory without any power at all? Any time you notice magic like this happening, you should immediately suspect that data storage is involved somehow.

When you put a computer into hibernate, Windows dumps the contents of computer memory into a hibernation file on your storage disk (check if you have C:\hiberfil.sys on your computer disk; that’s the one), then shuts down your computer. Yup, it’s a shutdown, except the programs don’t have to exit (because remember, the data configuration is already preserved). So this is much faster than a normal shutdown, but not as fast as standby which involves no shutdown at all.

When you press the power button again to bring the system back, the computer still has to go through the bootup sequence (memory was completely wiped, remember). But at the point where control is handed over to the OS (Issue 112)), the OS does not carry out its usual preparations. Instead, it reads the contents of the hibernation file into memory, then acts as though the system is already booted up! You just log in, and everything is the way it was just before hibernation (keeping fingers crossed …)

Issue summary: Hibernation mode causes the computer to store the data configuration into a hibernation file on disk. When powered up, the OS reads the data configuration from the file back into memory. This lets the system avoid having to do a full shutdown and bootup; it performs a shorter version of these two sequences instead.

Okay, so that’s one more mystery explained.

One thing I didn’t think was critical to explain above, but which eventually crops up for most people, is that the hibernation file takes up a lot of space! It needs to contain all the data in memory, so if you have a laptop with 16GB of memory, your hibernation file will be around that size too. If you’re wondering why Windows is taking up so much space, this is usually one of those reasons.

If you found C:\hiberfil.sys, you might have also found C:\pagefile.sys, another huge file, though not as huge as the hibernation file. What is this one used for? I’ll explain that in the season finale :)

What I’ll be covering next

Next issue: [LMG S9] Issue 117: Swap space

Sometimes, you get a laptop with 2 or 4 GB of memory, and you wonder how it’s able to run Chrome with 46 tabs open; shouldn’t you have run out of memory by now? In the next issue, I’ll explain this trick that lets computers pretend they have more memory!

I wanted to write this issue earlier in the season, but it just never presented an opportune moment. So let’s wrap up with it.

Sometime in the future: What is:

XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
OpenType? And what are fonts anyway? [Issue 42]
a driver file and why do I need one? [Issue 98]
a video card? [Issue 113]

Issue 115: Shutdown & standby

2021-04-10T08:00:00+08:00

Previously: Embedded operating systems are unlike user operating systems. They are designed to run the software needed for an appliance’s operation, and are not meant to be used by users directly. Since they are considered somewhere between software and hardware, they are usually referred to as firmware.

In Issue 112), I described the bootup process, and what a process it is! We have sayings in the English language that talk about things taking years to build up, but only seconds to destroy, and that’s certainly the case here. It takes so many steps to get data into computer memory in a way that makes it useable for us humans, and the mere press of a power switch wipes that data configuration once computer memory loses power.

This is obviously undesirable; your work in progress might not have been saved, data that is being transferred to disks might not be written properly and end up getting corrupted (Issue 109)), and you should never switch off a computer at the power socket this way!

Forced shutdowns

But sometimes, the computer gets itself so stuck that there’s just no other way to do it. On a laptop or smartphone, which has a built-in battery and cant be shut off immediately at the power socket, this is usually done by holding the power button for more than 10 seconds. This will literally cut off power to the mainboard immediately.

What does a proper shutdown do differently?

Shutdown

To avoid the Problems described above, we need to give the computer time to wind down. Most programs have a proper exit routine (y’know, the red dot on macOS or the red cross in Windows). This gets the program to write any last bits of data, release any file or database locks (Issue 82)), quit, and then release its resources back to the OS. When shutting down, all running programs and services have this exit routine invoked, and they usually quit pretty quickly.

The exceptions to this usually have to do with disk operations, the slowest of operations on a computer. If a program has lots of data to save to disk, or is waiting for a disk operation to complete, it will remain open while attempting to exit, and prevent the computer from shutting down. If you’re wondering why a folder seems to be stopping the computer from shutting down, this is usually the reason!

“Wait, I’m not copying anything …” Yep, but File Explorer in Windows does more than just let you copy, paste, and delete. It also generates image and video thumbnails for you, create file indexes to speed up searching, and so on. And these are disk operations too!

On rare occasions, these disk operations can mean your computer takes up to an hour to shutdown! My best advice, if you’re not in a hurry, is to just leave it alone and let it do its thing. Much safer than having to scan for errors later.

Standby/Sleep

On older computers, shutting down and booting up took more than 10 seconds, which is quite a killjoy when you are getting ready for a presentation and just need to stuff your laptop into a bag, get up to the conference room, and switch it on again. That feeling when you’re looking at a room full of people looking back at you, waiting for your laptop to boot up … awkwardddddd

Some of us just keep our laptops open (because closing the lid causes it to shut down, but you can change that in the settings by the way), and hold it that way while running around. And the purpose of that is just to keep the computer memory powered so we don’t lose the precious data configuration of the computer memory!

What if there was a way to keep power going to computer memory, but have the rest of the computer system powered off, to save on power? That would be perfect for hopping from coffee joint to coffee joint!

This is exactly what happens when you put a computer into standby (some systems also refer to this as sleep). The computer processor goes into a power state where only the computer memory remains powered (Issue 112)). Everything else is powered off—screen, network hardware, storage disks, … until you power the system on again. The memory configuration remains intact, and everything is exactly the way you left it at the moment you put the laptop into standby. Best of all, you didn’t have to go through the long and tedious shutdown-bootup sequence again!

Issue summary: When you shut a computer down, it sends an exit signal to all running programs to get them to do their exit routine. This process can sometimes take a long time. To preserve the data configuration in memory while minimising power draw, a computer can go into standby mode: all hardware except the memory gets powered down, until the computer is woken up from standby.

Lots of people don’t know about computer sleep, and resort to workarounds to keep their computer running while they have to move around. But now you know.

What I’ll be covering next

Next issue: [LMG S9] Issue 116: Hibernation

Sleep and hibernate; what’s the difference? As it turns out, a lot!

Sometime in the future: What is:

XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
OpenType? And what are fonts anyway? [Issue 42]
a driver file and why do I need one? [Issue 98]
a video card? [Issue 113]

Issue 114: In the beginning (firmware)

2021-04-03T08:00:00+08:00

Previously: If you can’t get to a BIOS screen, it is likely a hardware problem and has to be solved by a technician. If you can’t get the the OS loading screen, it’s a bootloader problem and needs to be solved with more geekery. If something goes wrong with OS loading, and fails to fix itself on subsequent reboots, it’s probably time for a system refresh or reinstall.

In Issue 111), I described the bootup process. These days, anything more complex than a calculator typically has an operating system (OS) running it. Your TV set-top box, your router, even your (high-end) printer and your car have an operating system running them these days!

Embedded operating systems

That OS is quite different from the one running on your laptop. It’s not meant for users to interact with directly. It primarily runs services which the appliance provides; you’re not allowed to install your own apps on it unless it has a manufacturer-approved app store or you are hacking it.

We call these embedded OSes.

Hardware and Software

We often see the term “hardware” used to refer to physical tools and implements that are needed for a job; they’re “hard” in the sense that they often come fixed into a certain configuration, are not easily changeable, and remain that way for the rest of their lifetime.

“Software”, on the other hand, is … fuzzier, more malleable, less clearly defined. Almost anything goes for software! This is usually the user-facing side of a system.

Firmware

We usually consider laptop OSes to be software, because it has an installation process, you get small regular updates on it, you can add and remove features … if it looks like software, talks like software, and behaves like software, you might as well call it software.

Embedded OSes are kinda different. They come preinstalled; if they don’t, you won’t know how to install it yourself on a brand new, unprogrammed appliance. You don’t get small hotfixes and updates, only big version changes. And you can’t control what features it has.

If software sits at the layer nearest the user, and hardware sits at the layer nearest the machine, then I guess we’ll call the middle layer firmware.

So when you see the word firmware used in an interface, I guess you can just think of it as the “OS of the device” :)

Issue summary: Embedded operating systems are unlike user operating systems. They are designed to run the software needed for an appliance’s operation, and are not meant to be used by users directly. Since they are considered somewhere between software and hardware, they are usually referred to as firmware.

Okay, that’s one thing out of the way. Now we can move on to shutdowns, sleeps, and hibernates!

What I’ll be covering next

Next issue: [LMG S9] Issue 115: Shutdown & standby

I often hear a lot of confusion about this in my workplaces, and from folks too. This is often the cause of many problems-that-happen-when-I-press-the-power-button. Let’s see what we can uncover.

Sometime in the future: What is:

XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
~~firmware? [Issue 34]~~
OpenType? And what are fonts anyway? [Issue 42]
a driver file and why do I need one? [Issue 98]
a video card? [Issue 113]

Issue 113: A computer’s existential crisis (boot failure)

2021-03-27T08:00:00+08:00

Previously: When a computer is booted up, it runs the BIOS from a chip on the motherboard. The chip checks that core parts are present, checks for a storage disk containing a bootloader, loads it into memory, and hands over control. The bootloader loads the operating system kernel. The operating system kernel then does whatever it needs to do to get the system ready for use.

In this issue, doom and gloom.

Nothing happens

Sometimes, you press the power button and nothing happens. Absolute stillness.

This is not a software problem; something is wrong with the computer’s power source (the power supply unit of a desktop, or the battery/charging system of a laptop), and you have to bring it to a technician.

Something happens, and then nothing happens

If you hear fans spinning, the power source works. But if nothing appears on screen after that, and then the system seems to reboot again, it may mean the BIOS has failed; if you attempted to upgrade the BIOS recently, that might be a possible cause. A friendly geek might be able to fix this for you.

If you hear fans spinning for a long time, and things seem to be happening, but nothing is showing on screen, your video card (future issue, I promise) may be borked. Can’t do anything other than sending it in to a technician.

If you have attempted a driver upgrade recently (again, future issue), that might be a possible cause.

When the BIOS works, you should see the startup screen—always a comforting sight.

Bootloader issues

Bootloaders are reliable as anything, so seldom anything goes wrong here.

The most common bootloader problem is that it does not detect any OSes to boot—may be the case the first time you attempt dual-boot, or get advanced enough to start playing with bootloaders (Windows and Mac don’t let you do that, so these are almost always Linux users). This is … something that can only be solved with more geekery, and is beyond the scope of this newsletter. Sorry 😬

OS booting problems

The OS, being the most complex part of the bootup process, is often where things go wrong.

If nothing has changed in the system since the last bootup, usually things go smoothly. So it’s almost always changes to the system that cause this.

Sometimes you installed a program that dabbles in the system innards—a system-optimising program maybe—and it attempts to change some settings but without propagating them to all the required places. This leaves the system in an inconsistent state.

Sometimes you attempted a Windows Upgrade¹, possibly a major one, and it did not go as hoped. Windows usually will attempt to undo the damage it wrought, and may or may not succeed in leaving your system in a bootable state.

Sometimes some system files get corrupted—remember that this can happen if you force a computer to power off before it has finished its shutdown properly. If the OS can’t find that file, it sometimes initiates a search for it in the system disk, which can take … really long.

In my opinion, the best way to resolve these issues, especially if they are recurring, is with a system refresh or reinstall.

Issue summary: If you can’t get to a BIOS screen, it is likely a hardware problem and has to be solved by a technician. If you can’t get the OS loading screen, it’s a bootloader problem and needs to be solved with more geekery. If something goes wrong with OS loading, and fails to fix itself on subsequent reboots, it’s probably time for a system refresh or reinstall.

I told you, gloom and doom. Modern OSes are getting more sensible at not requiring human intervention, so when they fail to resolve their own issues, there is often little a layperson can do on their own.

The best thing you can do for yourself and your data is to ensure you have a copy of your critical data elsewhere. Always have backups of things you are working on.

What I’ll be covering next

Next issue: [LMG S9] Issue 114: In the beginning (firmware)

I tried to find a place to talk about firmware in this and the previous issue, but couldn’t do so without taking the story in weird directions. So we’ll take a short detour to do that next issue, before I talk about shutdowns, sleeps, and hibernates.

Sometime in the future: What is:

XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
a driver file and why do I need one? [Issue 98]
a video card? [Issue 113]

These days it’s more like you attempted to stop a Windows Upgrade, but did not manage to do so or it is unskippable. ↩

Issue 112: Bootstrapping into existence (bootup)

2021-03-20T08:00:00+08:00

Previously: Moving a file (within the same disk region) merely updates its file table record, and this happens really quickly. Copying a file, or moving it to a different disk/region, involves copying the contents and then updating the file table record, and is considerably slower. Deleting a file only requires that its file table record be removed, and is a very fast operation (if it does not involve the Recycle Bin).

We humans wake up in the morning and magically there is information in our heads. We open our eyes, trust that all our limbs and body parts are there, and we just get up and make coffee.

Computers, they are different in that regard. After they light up the power LED, there is just … nothing. Computer memory needs power to hold its information¹, so once you really power down your computer, all data in memory is gone. Your operating system (OS), your open programs, your file data … all that now resides only in your hard disk. Whatever was not written to hard disk would have been lost by now (Issue 108)).

Before anything can be done, the computer needs to read all that information back in to memory. But there’s nothing in memory … you need a program loaded in memory to do that, but that program is still residing in file storage—it’s a Catch-22! The OS has to bootstrap itself into existence first, in a process known as the bootup.

Bootup

The way to sidestep the bootup paradox is to have a very tiny chip perform the bootstrap. This tiny chip is already wired up in a particular way from birth; it contains a program called the BIOS (short for Basic Input/Output System)², which does only three things:

Check that the core parts are present: CPU, memory, video card, keyboard, storage disk.
At the BIOS screen³, you can press a button to configure BIOS settings.
The BIOS checks the available disks to see if they are “bootable”, i.e. contain information in a specific part of the disk that contains a bootloader. In modern BIOSes, you can override this process by specifying a valid disk containing the bootloader.
The BIOS loads the bootloader into memory, and runs it.

The bootloader

The BIOS is not optimised—it’s only a basic system after all—and would take forever to try to get the OS running on its own. So, with its limited resources, the BIOS calls in bigger guns: the bootloader.

The bootloader is not the OS! It has only one job: to load the OS during the boot process, and carry out whatever is necessary before that.

Usually, the first thing that needs to be loaded is the filesystem (Issue 106)). Without that, no program will know how to read in data from the storage disks! At this point, if the bootloader detects disk errors or uncompleted operations in the journal (Issue 110)), it may attempt to scan for errors or complete those operations before proceeding with the rest of the bootup process.

Some systems may contain multiple OSes: Mac users may want to run Windows using Parallels Desktop, Windows users may want to dabble in Linux, and many Linux users dual-boot Windows as well. The bootloader, with the help of the filesystem, detects other operating systems on the storage disks, and offers the user a choice of which one to boot. If there is only one OS found, this step might be skipped.

Once an operating system is selected, the bootloader loads the OS kernel, which contains its core instructions, and hands control over. The bootup sequence is not yet complete, but the bootloader has completed its job.

The operating system startup

The operating system is not ready to accept user input at this point yet. It still has to mount other storage disks and their subregions, check that other hardware is available and working properly (such as hardware timers, which are very important for an operating system), start up various necessary services, start up programs which asked to be started up when the system boots (like the annoying Adobe Updater, or your sound card drivers and utility), etc.

Finally, when it is ready, it displays the login screen.

Issue summary: When a computer is booted up, it runs the BIOS from a chip on the motherboard. The chip checks that core parts are present, checks for a storage disk containing a bootloader, loads it into memory, and hands over control. The bootloader loads the operating system kernel. The operating system kernel then does whatever it needs to do to get the system ready for use.

This a prelude to the next issue, which is where I attempt to explain, as simply as possible, the usual things that go wrong in the bootup process.

What I’ll be covering next

Next issue: [LMG S9] Issue 113: A computer’s existential crisis (boot failure)

We’ve all been there. The computer doesn’t get to the login screen. We are shocked that our daily driver, our churner of spreadsheets, can do this to us. Worse, it has left us in an environment we are wholly unfamiliar with: a blinking cursor in a sea of black, cryptic text explaining nothing, and the unbearable pressure of a system silently begging us: fix me.

Sometime in the future: What is:

~~booting up? [Issue 15]~~
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
a driver file and why do I need one? [Issue 98]

Memory comprises lots of capacitors wired together, and capacitors slowly leak charge when they are not powered. Memory chips need periodic “refreshing” every few milliseconds to avoid losing data due to this charge leakage. ↩
Modern BIOSes are no longer stored permanently on a chip, but written to flash storage (similar to a thumbdrive) that is soldered directly to the motherboard. ↩
Today, this process is hidden by a branded startup screen on machines that come with Windows pre-installed. You can usually disable that screen if you like to see this bootstrap process happening. ↩

Issue 111: Copying, moving, and deleting files

2021-03-13T08:00:00+08:00

Previously: Filesystem journals are a record of changes made to the disk, so as to enable those changes to be rolled back, or to be completed properly in case of sudden interruption.

Okay, that’s enough of a look at the nitty-gritty of moving data around. In this issue, let’s turn our attention to the file table, where interesting things happen as well.

What happens when we copy a file?

A copy of the data is made, and moved to an available location (series of sectors) on the disk, and a new file table record is created. No magic yet.

What happens when we move a file?

Okay, this is slightly more complex, but not difficult.

First of all, if we are moving the data within the same disk … do we really need to move the data? In layman terms, when we “move” a file, we are not actually changing its physical location in the disk. We are basically updating a file so that its stated location, which used to be something like C:\Users\myusername\Desktop\video.mp4, will become C:\Users\myusername\Videos\video.mp4. This is information about the data, and is not stored with the data—it is actually stored in the file table!

So when we “move” a file within the same disk region (i.e. within the warehouse, in our earlier analogy), we are actually just updating the file table record, without copying or writing any of its contents. This is why it can seem so amazingly fast to “move” a multi-gigabyte file.

But if we are “moving” the data to a different disk, or to a different region on the same disk (e.g. moving something from the C:\ region to the D:\ region), it is like moving cargo from one warehouse to another. Each warehouse keeps its own file table, so we can’t merely update the file table and keep the contents on the same pallet on the rack. The cargo needs to be moved to a different warehouse, and the file table record copied over as well. For data, this means the contents are copied to the new location, the file table is updated, and if both happen without error, the original contents and file table record are deleted.

What happens when we delete a file?

Ah, fun times. Okay, so I suppose most of you keep your computers on the default setting, where pressing the Delete key doesn’t actually delete a file irretrievably, but simply puts it in the Recycle Bin.

And when you are deleting a lot of files … you notice this can take a long time. Obviously the contents are still around (so that you can still restore them), but the file table records are now hidden from the user, so that they appear to be deleted. If you are deleting 100,000 files, that is 100,000 different file table records to update.

You can, of course, change this setting (and I won’t teach you how to do that here, but it’s just a Google search away). If you disable Recycle Bin, or use the Shift+Delete hotkey combination to force a permanent deletion, something different happens instead. Intuitively, it seems like the OS should actually clear out the (digital) space previously occupied by the file contents. But this offers no advantage; to change a 1 to a 0, you have to write a 0 to it; there is no erasing in the world of storage!

The faster way is to simply tell the filesystem that this file no longer exists, and the space it used to take up can now be overwritten. And that is done … by simply deleting the file table record. This is a simpler operation than updating, which is why permanent deletes are so much faster than Recycle-Bin deletes.

This is also why, if you accidentally permanently deleted a file, you still have a chance of recovering it with file recovery software! The recovery software ignores the file table, and instead scans the entire disk for its actual contents, trying to see if there are any valid fragments of files left. If the space has not been overwritten by anything else, you just might be lucky enough to recover its contents. The chances of this decrease as you use the disk and write over parts of the disk which contained the file data, so this has to be done as soon as possible.

Issue summary: Moving a file (within the same disk region) merely updates its file table record, and this happens really quickly. Copying a file, or moving it to a different disk/region, involves copying the contents and then updating the file table record, and is considerably slower. Deleting a file only requires that its file table record be removed, and is a very fast operation (if it does not involve the Recycle Bin).

I’ve been waiting a long time to explain this, and can finally get it out on screen. Phew!

What I’ll be covering next

Next issue: [LMG S9] Issue 112: Bootstrapping into existence

Now I can move on to something closer to the hardware, and talk about what happens when the power button is pressed.

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
a driver file and why do I need one? [Issue 98]

Issue 110: Safeguarding against data corruption with a journal

2021-03-06T08:00:00+08:00

Previously: Fast writes dump the data to a write cache (in computer memory), then update the file table to look like the file is already written to disk. However, if power is cut before all data is properly moved from the write cache to disk, the data in memory is lost, and file corruption usually results.

Last issue, I showed how a write cache can “speed up” file write operations by allowing data to be dumped to a write cache in memory first. The OS “completes” the operation, making it look the file has been successfully written to disk, when in actuality parts of it are still in memory waiting to be written. This works fine as long as we don’t suffer catastrophic power loss before everything gets put on disk; once power is cut, all the contents of the write cache in memory are lost!

All right, we are into worst-case scenarios here. How do we recover from something like this?

The first thing we’ll need to know is what had happened; what data made it through and was written, and what data didn’t make it. We need a record of changes to the disk.

File journals

This record, in a filesystem, is called a journal. Keeping a journal requires more CPU cycles and can slow down file write operations somewhat, but as disk performance increases, this is increasingly considered a worthwhile tradeoff for data reliability.

The default filesystems on major OSes today are all journalled, and NTFS is no exception. I’m not going into details of how this is done, and instead will list the things that a journal enables Windows 10 to do.

Disk recovery with a journal

So what happens when power is cut and a computer reboots? It goes into recovery mode, where it checks the log for incomplete operations, and attempts to complete them. Any files that were “lost” (i.e. file table records that have been separated or desynced from the actual data) get moved to a separate folder, and the disk is considered okay for use again.

Disk transaction rollbacks

When installing a new application, sometimes you encounter that odd screen where the installation fails because something that needed to happen couldn’t happen. Oh great, the application could not complete installing, and now we have to remove it … how do we undo all the things that were done?

The journal lets you see a list of all the changes that were made to the disk from the start of the installation, so that you can perform the equivalent steps to reverse their effect.

System restore with journals

The same thing happens with some Windows updates. Windows stores a lot of the old data as backup, in case you ever want to roll back some system changes, or return to an older version of Windows (minus some breaking updates, for example). And again, the filesystem journal lets you do that, by identifying which changes were responsible for the update.

Issue summary: Filesystem journals are a record of changes made to the disk, so as to enable those changes to be rolled back, or to be completed properly in case of sudden interruption.

Journals are serious magic, and are what enable OSes to recover gracefully from a crash. They are kind of divorced from everyday experience, because it is usually unwise to mess with them directly, but I hope the past few issues explain why it is important to not get impatient with your computer when it seems a little slow moving data round.

What I’ll be covering next

Next issue: [LMG S9] Issue 111: Copying, moving, and deleting files

I am done with the more technical stuff! Now we can really move on to more everyday experiences, like copying, deleting, and moving files :)

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
a driver file and why do I need one? [Issue 98]

Issue 109: Speeding up data operations

2021-02-27T08:00:00+08:00

Previously: Safe writes ensure that all the data is written to disk sectors properly first before updating the file table. The result is that write operations take a longer time to complete.

If there’s anything to take away from the previous issue, it’s that doing things the right way takes time. And sometimes we are okay with taking shortcuts to get something done faster, the left way.

The write cache

If you have a shipment coming in that urgently needs to be dumped into the warehouse quickly (e.g. because the ship needs to leave quickly for another delivery, or to free up space on the docks), how do we speed this up?

What usually happens is the goods will be offloaded onto some empty space outside the warehouse. The ship will leave first, and assume that the warehouse will be able to get the goods in just fine.

What is the OS equivalent of this? Hard disk writes are actually pretty slow; they can get up to 100 MB/s, but usually hit a sustained speed of 60 MB/s for system disks, and about 20 MB/s over USB. The operating system therefore sets up a “dumping space” in memory; files can be read from other disks at up to 200 MB/s, and written to memory at a few GB/s!

This space is known as the write cache. (I’ve previously covered the idea of caches in Issues 39) and 57), and the write cache works on a similar idea.)

Speeding up file access … and its drawbacks

So how do we speed up the process of copying data to a disk? We could dump the data into the write cache (assuming there is enough space), then update the file table first, and let the OS slowly copy the data from the write cache into the disk. Anyone who needs access to the file records will then be able to go about their merry way, and if they need data from the file, they can just copy it from the disk cache instead of the disk. Easy peasy!

This was what happened in Windows XP. File copying appears to be speeded up, and users are happy.

But remember what happens when power to the computer is accidentally cut, or if a program suddenly hangs, or the computer sometimes runs out of memory.

Yup … it shuts down (or reboots), and the contents of memory are wiped clean. And any data that has not been written to disk is lost forever. And … the file table has already been updated to look like there is valid data!

This is when things look really bad.

Issue summary: Fast writes dump the data to a write cache (in computer memory), then update the file table to look like the file is already written to disk. However, if power is cut before all data is properly moved from the write cache to disk, the data in memory is lost, and file corruption usually results.

Well okay … it’s not a lost cause right? Yup, we do have ways to mitigate this kind of damage, and one such way is called a journal.

What I’ll be covering next

Next issue: [LMG S9] Issue 110: Safeguarding against data corruption with a journal

Woot, answered another sometime-in-the-future question, albeit in a slightly different fashion from how I thought I would do it.

Next issue, I explain how we can still do things the fast way, carefully.

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
a driver file and why do I need one? [Issue 98]
~~why does computer memory exist when apps can read directly from the hard disk? [Issue 105]~~

Issue 108: Safeguarding data operations

2021-02-20T08:00:00+08:00

Previously: When write operations are interrupted prematurely, filesystem corruption often results.

When a batch of data (shipment of cargo)) arrives at our warehouse, there are two things that need doing:

1) The data needs to be written into sectors (the smallest unit of storage that the disk handles), 2) The file table needs to be updated.

Writing data the safe way

If the write operation gets interrupted, it is preferable that our file table is not updated; this way, we will not find any reference to the data in the file table, and we can attempt the write operation again.

This will appear strange in a warehouse analogy; the cargo, damaged or intact, is still occupying space on the racks! But remember that when we are talking about a hard disk storing data, there is no physical cargo; the data exists as a specific arrangement of electrons/atoms. We can override the existing arrangement of electrons/atoms without having to reset it first.

Drawbacks of safe writing

This way of storing data first before updating the file table is advantageous in its security; if the write operation is interrupted halfway, we are less likely to suffer filesystem corruption.

But for large batches of data (or large shipments of cargo), this means a long wait … and in the meantime, nothing else can happen! Hard drives only have one writing needle, which is like a warehouse only having one forklift. If you have other applications waiting for that file (similar to other employees waiting for the file record to appear in the file table), they will be twiddling their thumbs until the last sector of data is written, the last pallet of cargo loaded onto the racks.

But this is the right thing to do, isn’t it? It doesn’t matter; Windows Vista did it this way, waiting for write operations to complete before updating the file metadata in the file table, and the result is that people complained that it was slower. People do not like things slow!

Why did Windows XP feel faster, then? Because it did it the other way round!

Issue summary: Safe writes ensure that all the data is written to disk sectors properly first before updating the file table. The result is that write operations take a longer time to complete.

Actually, Windows 10 uses a system similar to this for mounting portable devices (USB hard drives, flash drives, etc) by default. They sped it up in other ways. So if you accidentally unplug a drive before it is completely done writing … usually it won’t completely screw things up.

What I’ll be covering next

Next issue: [LMG S9] Issue 109: Speeding up data operations

If you want, there are settings you can adjust to make Windows 10 access a portable device the fast way. I don’t teach it here, because this is not the newsletter for it. But we’ll see how the fast way works in the next issue.

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
a driver file and why do I need one? [Issue 98]
why does computer memory exist when apps can read directly from the hard disk? [Issue 105]

Issue 107: The challenges of storage

2021-02-13T08:00:00+08:00

Previously: A hard disk is organised into sectors, which are the smallest unit of storage. The OS’s filesystem determines how and where to store each file on the hard disk. The filesystem manages the file metadata in a file table, separate from the actual contents of the file.

Last issue I painted the picture of a warehouse, where cargo gets stored in racks, their contents and details are stored in the file table (in the warehouse office), and we can easily look up the details of each shipment of cargo without having to inspect rack by rack.

Let’s screw things up here, shall we?

Read/write failure

As mechanical devices, hard disks are prone to failure; I’ll go into detail why in a future season on hardware. Remember that when this hard drive is housed in a computer, that computer can get disturbed by shocks or jolts. If it is housed in an external USB enclosure, sometimes we do terrible things to that enclosure … like when we drop them (usually accidentally 😬).

This is like the warehouse experiencing an earthquake. Sometimes, a small earthquake may be uneventful … other times, we might even get away with a moderate earthquake, if nothing is happening in the warehouse. But if it happens while a forklift is carrying out a precarious loading operation … 🙈

Well, that’s terrible. We’ll have to write off that shipment. Luckily, we can analogy here; data can always be re-copied again. The problem is … when a shipment comes in, we have to load the shipment onto the rack, and update the file table, so that its contents match whatever is on the racks. Which should we do first? If the loading is uneventful, it doesn’t really matter; after 5 minutes or so, both the file table and the racks will be in sync. But if the earthquake happens during this loading, they won’t be!

We don’t need an earthquake to screw things up. Remember that a hard drive, or even a flash drive, needs power to run. When we unplug a hard disk or flash drive while it is still operating, it’s like cutting power to the entire warehouse (including its forklifts) at the same time. If nothing is happening in the warehouse, it is safe to do so. But if there are ongoing operations, well you don’t need me to tell you that’s not a good idea.

Reads are usually safe

Okay, we’ll have to step away from the warehouse analogy for a bit when we talk about reading data from a hard drive; this is usually safe, because unlike cargo, reading data does not destroy the existing copy. We can mount a filesystem (i.e. gain access to it) for reading only, which will protect it from accidentally having its data overwritten.

It is the writes that always get us. When a write operation does not go smoothly, the result is usually filesystem corruption; the file table is no longer up to date, or worse, some parts of it might be improperly written, resulting in the company no longer knowing what is on the warehouse racks.

Issue summary: When write operations are interrupted prematurely, filesystem corruption often results.

There are two ways we can try to get data onto the disk (get cargo into the warehouse): Write the data first, then update the file table, or vice-versa. Which way is better? We’ll compare the pros and cons in the next issue.

What I’ll be covering next

Next issue: [LMG S9] Issue 108: Safeguarding data operations

That remove-safely thing we all hate to do when we unplug our hard drives? Yeah I’m getting to that part.

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
a driver file and why do I need one? [Issue 98]
why does computer memory exist when apps can read directly from the hard disk? [Issue 105]

Issue 106: Organising storage

2021-02-06T08:00:00+08:00

Previously: The OS takes care of booting up, login and user management, window management, memory allocation, storage interfaces, background services, peripheral management, and much more. Access to these services, where allowed, is provided in the form of software libraries that developers can use.

So far, what we understand about storage is that apps send data to an operating system (OS), which then stores it on storage devices (such as hard disks, solid state disks, etc) for later retrieval. And in Issues 99) & 100), I explained that eventually everything gets stored as a file.

What exactly is a file?

Files in operating systems

Let’s start from what we know about files. When we open File Explorer in Windows and open any folder (aka directory), we see a bunch of things with colourful icons, that we know of as files. And we also see things with folder icons that we know as folders (or directories).

They also have additional details displayed alongside them, such as the date/time (henceforth referred to as datetime) it was created, the last datetime it was modified, and the last datetime it was accessed. You might also get additional information, such as what type of file it is, its size, the user who owns that file, etc—in other words, the metadata of the file. That’s a fancy word to refer to data (datetime, filetype, size, …) about the data (the files).

Now, a brief introduction to the hardware, where the actual files get stored.

Introducing storage

When you buy a hard drive (or solid state drive) and connect it up to your computer (or laptop), the computer does not have fine-grained control over every bit of storage. Instead, the hard drive gets presented as a huge volume of space, like a warehouse.

This volume is organised into sectors (these days, a modern hard drive sector is 4KiB), which are the smallest unit of storage on a hard disk. If you have to store 1KiB of data, it will still take up an entire sector; the rest of the space is filled with zeroes. This is similar to the way a warehouse is managed by pallets, and not by single cardboard boxes. If your shipment does not fill the entire pallet, the rest of the space is “wasted”.

Each sector has an address (yes, just like memory!). To store data on the hard disk, the OS has to “tell” the hard disk a) what data to store, and b) the address at which to store the data.

The hard drive itself does not inherently have any system for managing your files or folders; it can’t tell you where draft1.docx is stored, nor tell you what the size is. It only takes care of storing data at addresses, and retrieving data from addresses!

Managing storage

And this is where the operating system comes in. An operating system installing itself is like a company occupying a warehouse. You’ve got to impose some kind of order on the space!

In an operating system, this job is delegated to the file system, a sort of facility director who manages the storage space. Windows uses NT File System (NTFS) for its own space, MacOS uses High Performance File System (HPFS), while portable storage devices (e.g. USB drives) often use File Allocation Table (FAT).

An operating system that is not “aware” of other filesystems will not be able to read storage devices formatted with those filesystems. This is why you can’t just take out a disk from a Mac system and expect it to open in Windows when plugged in. Windows can read HPFS disks, but cannot write data to them. Both OSes can handle FAT (phew!), which is why it is possible to pass files from a Windows user to a Mac user with a portable flash disk.

How filesystems work

Intuitively, we might imagine that the metadata for each file gets stored with its data. This is like storing the shipment details of each package with the package itself. But when you need to find a particular shipment in the warehouse, you can’t be checking every single rack!

Instead, you would store shipment records in a master file, usually in some kind of separate office, which would list the shipment details alongside the location of the cargo (e.g. by rack number and level). The equivalent of this master file on a filesystem is the file table.

In NTFS, this is the Master File Table (MFT). The MFT lists all the files on the disk (by their full path, e.g. C:\Windows\system32\notepad.exe), along with its metadata. This makes it easy for Windows to show you the data quickly when you open any folder in File Explorer; it can get all this data from the MFT easily!

Sounds peachy. What can go wrong with this picture?

Issue summary: A hard disk is organised into sectors, which are the smallest unit of storage. The OS’s filesystem determines how and where to store each file on the hard disk. The filesystem manages the file metadata in a file table, separate from the actual contents of the file.

What I’ll be covering next

Next issue: [LMG S9] Issue 107: The challenges of storage

This is where I talk about all the ways we make life difficult for computers, but easier for ourselves. Like unplugging USB drives before removing them safely …

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
~~How do apps know where a file starts and ends? [Issue 49]~~
a driver file and why do I need one? [Issue 98]
why does computer memory exist when apps can read directly from the hard disk? [Issue 105]

Issue 105: Operating Systems

2021-01-30T08:00:00+08:00

Previously: Shared secrets allow secured access to resources, such as databases or other services. These shared secrets are typically kept on a server controlled by the app developer. For mobile apps, they are usually stored with the operating system, inaccessible to other apps.

Last season, I explained the differences between web apps, mobile apps, and laptop apps. Web apps operate in a browser environment, mobile apps operate in a mobile operating system (OS) environment, and laptop apps operate in a desktop OS environment.

The OS is that complex piece of software between the apps and the hardware that runs it all. The hardware speaks its own special code, while apps are run in machine code written for the CPU. The OS makes sure that everything runs smoothly.

What does everything entail?

What an operating system does

1. Boot-up

When the machine is powered on, how does it know what hardware is available? This hardware discovery is done by the bootloader (which for the purposes of explanation we can consider as part of the OS). It checks for the available hardware resources, and compiles a list of them for the operating system. Added some memory? Added a new hard drive? The bootloader should detect it.

2. Login and user management

While a personal computer (PC) might have only one user, many PCs are used by more than one person. Multiple login accounts allow each user to have their own space on the computer for their files.

In addition to the user accounts, an operating system also creates multiple system accounts to manage services on the OS.

3. Window management

Ever wondered how apps on a PC can have their windows styled so similarly? That’s because the OS provides a standardised window style for apps, with standardised actions that can be performed (maximise, minimise, close, resize, …). Apps can choose not to use this standardised window style, but then they are on their own when it comes to window styling, and they will have to write their own code for all of these actions.

4. Memory allocation and deallocation

Applications primarily run off computer memory, which is a hundred times faster than the hard disk (I’ll explain more in a future season). When an application is first launched, it is allocated a small amount of memory space for its data. If it wants more, it can’t just reach for the memory space and grab more chunks as needed; other apps are using memory as well, and it might inadvertently overwrite data stored by other apps.

So instead, access to memory is mediated by the OS. All requests for more memory are sent to the OS, which will return addresses for available chunks of memory. And any memory that the app frees up is released back into the common pool, for use by other apps.

5. Storage interfaces

Applications do not need to know how the user has their computer set up: what kind of disks they have, or which disk they put the OS on. But what if they need to allow the user to save a file, or load an existing file? The OS provides standard interfaces for doing so—you would likely have seen the “Select A File” interface on any OS. The OS takes care of file reading and writing, sparing the application developers from having to worry about the details.

6. Background services

Applications run by the user typically expect input only from one user; if you are running Microsoft Word at the same time as your sister, each instance of Microsoft Word expects to interact with either you or sister. However, there are special system applications that often have to deal with input from multiple users. For example, antivirus software or Windows Printer Management need to run for all users at the same time, and should not be terminated like a typical program.

These applications thus run differently: they run as background services, which are special applications which do not interact with the desktop. Background services are managed differently from applications, and typically can only be terminated by the system administrator.

7. Peripheral management

Do you use Bluetooth wireless headphones? USB devices of any sort? Did you just buy a new gamepad? The OS detects them and takes care of driver installation (usually). You can see all detected devices and installed drivers in the Device Manager, under Control Panel, in Windows 10.

I’ve just covered the main ones that most users would use on a regular basis. The OS manages much more stuff, including OS updates, per-user settings, etc—remember the Registry Editor? (Issue 100)) For a look at what else the OS manages, take a peek in the Control Panel or System Preferences of your OS.

These form a supporting network of services that applications can draw on. Instead of having to work out the full details of implementation for each of these features, they can use the OS’s provided functions to do the work for them. These functions are typically bundled in a standard library (Issue 17)) provided by the OS manufacturer. For example, .NET is a collection of libraries for Windows applications that developers can use to simplify their work.

Issue summary: The OS takes care of booting up, login and user management, window management, memory allocation, storage interfaces, background services, peripheral management, and much more. Access to these services, where allowed, is provided in the form of software libraries that developers can use.

I’ve touched on per-user data storage in Season 8, and on memory allocation in Season 5, so I will skip those for the rest of this season. Window management and background services isn’t going to be interesting for a layman’s guide, so let’s skip those too. Instead, I will try to set the stage for a next season about hardware, by explaining more about how storage and computer peripherals work, and how the operating system simplifies access to them. and then I’ll get to booting up—how a computer gets ready from the moment you press the power button.

If I still have issues left after that, I might geek out a bit and talk about fonts ;)

What I’ll be covering next

Next issue: [LMG S9] Issue 106: Organising storage

First up: How exactly is data organised in the hard disk? How does an operating system manage it all?

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
How do apps know where a file starts and ends? [Issue 49]
a driver file and why do I need one? [Issue 98]
why does computer memory exist when apps can read directly from the hard disk? [Issue 105]