Layman's Guide to Computing - Season 13

Issue 169: Search engine optimisation

2022-04-23T08:00:00+08:00

Previously: A search engine uses bots to build up a database of URLs and their contents. The search engine uses various algorithms to determine the most relevant results for a search request.

Let’s get to it: why are search results so bad so often?

PageRank

While PageRank is no longer the only or even the dominant algorithm for ranking search results, it is probably the most familiar one to most people and is easy to understand.

PageRank ranks your page higher if there are many other pages (on other domains) linking to it, so one way to raise its rankings is to make other websites with links that point to your page.

Advertising

Naturally, Google runs an online advertising business and it puts advertised results first, so this obviously skews the rankings.

Issue summary: By better understanding how search bots categorise pages, a website owner can use keywords and other techniques to optimise the ranking of their page for specific search terms.

Not very detailed, because I am not an SEO master! This issue is probably the least helpful for more savvy readers, but everyone finds out about page keywords at some point and I hope this issue is that first time for some readers :)

What I’ll be covering next

I am … done! With the main part of Layman’s Guide to Computing at least.

Going even slower

What this means is I still have bits of content to post, but they are no longer substantial enough to form seasons of 13 issues. Instead, I’m going to be posting in dribs and drabs, guided by current affairs and any new technologies that have gone mainstream. This also means my posting frequency is going to be going way down; expect 1–2 issues a month, instead of the current 4–5. And I’m going to be taking the next 2 weekends off for a little celebration of my own, plus some thinking and reflecting.

Better archives

I also realised that Buttondown’s archive page no longer provides a browseable listing of all issues. I am looking into setting up my own issue index instead through Github Pages, on https://ngjunsiang.github.io/laymansguide/. Fortunately I have been using Github to archive past issues, so now it’s simply a matter of editing links for past issues and providing some kind of navigation. More updates on this when I make some kind of progress.

Refactoring

Code refactoring is when developers reorganise and rewrite their code in a way that makes it easier for them moving forward.

You might have noticed how often I am linking to past issues, because so many concepts in computing build on each other. I knew this was going to be a problem when trying to explain computing to anyone, and I had initially envisioned some kind of wiki where pages will link to each other so you can theoretically start exploring from any topic and unpack technical terms as you go.

The main bottleneck for this had been content: it’s rather overwhelming to think about where and even how to start. And that was how this newsletter was born. Thanks to Layman’s Guide, I now have a steady base of content for publishing Layman’s Wiki, and work on it has been slow but progressing. I’ll put up a link through this newsletter once it is ready for more eyes :)

What I’ll be covering next

Next issue: Issue 170: Machine learning models

After a long hiatus, Layman’s Guide returns with Season 14! But first, let’s understand what models are and how they work.

As always, thank you for reading!

Issue 168: Search engines

2022-04-16T08:00:00+08:00

Previously: Fragmentation is likely a contributor of system slowdown, particularly for mobile devices: the database used by most mobile apps tend to store data in many small chunks rather than fewer big chunks, which slows down data search operations. The most effective measure for improving device responsiveness is usually to clear the app cache, so the app does not attempt to read previous data from storage.

Last issue, we shed a little light on the mystery of why phone and laptop systems slow down over time—apparently the way a file database works is to blame?

This week, we switch topics, to look at something we definitely take for granted: search engines!

What is a search engine?

Forgive me if you think I am belabouring the obvious. There is the user definition, which is something like “a search box that answers my questions”. Then there is the developer definition which is more like “an indexed database of URLs, descriptions, and ranks”. Let’s unpack that.

Indexing and arachnids

Another obvious point: at the point when you submit your search query, there is no way the search engine could have trawled the entire internet so quickly to give you the results. It must have know about these pages beforehand … but how?

Search engines run “bots”, also know as web spiders, or web crawlers. These are programs that retrieve pages, makes a note of the content (text, images, links, …) on that page, determine keywords for that page, and then follows the links to other pages and repeats the process. Like following the links of a (very messy) web!¹

When we “search the internet”, we are really searching the database that has been built up by these bots.

Ranking results

So there’s a huge database … how does the search engine determine which are the most relevant results?

Google’s PageRank algorithm is by now well known for disrupting the old directory-based method of organising information, and built for Larry Page and Sergey Brin a sizable empire. This is one way of determining how important/useful a page is: by seeing how many other pages link to it. There are other ways, but this is a layman’s newsletter and I don’t want to dive into a technical analysis and comparison of different ranking algorithms 😬

Instead, consider that there are by now many different algorithms for determining the relevance of database entries for each search, even within Google itself. A search engine is essentially a way to access this database, use one or more appropriate algorithms to determine the most relevant results, and return them to the user.

Issue summary: A search engine uses bots to build up a database of URLs and their contents. The search engine uses various algorithms to determine the most relevant results for a search request.

I know … nobody really cares how a search engine works. We just want to know why it’s not giving us the results we want!

What I’ll be covering next

Next issue: [LMG S13] Issue 169: Search engine optimisation

I am not a consultant for search engine optimisation (SEO), but we can look at some of the ways people attempt to “game the system” and exploit features of the algorithms that search engines use. More next week!

Yep, we all know spiders don’t actually do this. ↩

Issue 167: Database fragmentation

2022-04-09T08:00:00+08:00

Previously: There are easy and quick ways to check the validity of the most common advice for resolving system slowdown. But it still seems to happen even after these tips have been tried.

Last issue, we talked about caches and why they are no longer as effective as a performance-boosting measure.

This issue, let’s look into a solved problem that is not-as-solved on Android: file fragmentation.

Storage fragmentation on mobile devices

I mentioned in Issue 119) that fragmentation is not an issue for faster SSDs; the much lower latency of SSDs makes the retrieval of multiple file chunks from multiple locations trivially easy. But for slower storage devices, whether they are hard disk drives or SSDs over a UFS interface, fragmentation is very real. Unfortunately, its not just a simple matter of defragmenting a disk regularly¹. the source of this fragmentation comes from the way apps themselves store data.

SQLite, a blessing and a curse

SQLite is a popular file-based relational database (Issue 87)) used by many Android apps (including WhatsApp!). This means that unlike other databases, which require a separate database program to run, SQLite just requires the programmer to bundle an appropriate library for their programming language instead of running a separate program. It makes management of data much easier for the app developers, and it is also fast—usually.

However, a 2016 study found that it contributes to fragmentation on the Android filesystem. Even after clearing the app’s data, once SQLite starts storing data again, it tends to store them in many small chunks instead of fewer but larger chunks. On Android devices, which use the slower UFS interface, searching through the database involves reading each database chunk to see if the requested data is there; this gets slower as the database grows, and SQLite adds more chunks to the storage device.

The study ends on a pretty pessimistic note: there doesn’t seem to be any feasible mitigation other than making upstream code changes to SQLite, or designing a different filesystem that overcomes this limitation. But that was five years ago; with any luck some positive developments might have been made into this area. I’m still looking into it!

Issue summary: Fragmentation is likely a contributor of system slowdown, particularly for mobile devices: the databases used by most mobile apps tend to store data in many small chunks rather than fewer big chunks, which slows down data search operations. The most effective measure for improving device responsiveness is usually to clear the app cache, so the app does not attempt to read previous data from storage.

That’s all I’ve managed to find out from reading; further research will either involve detailed tinkering and experimentation, or deeper reading, both of which are time-consuming.

Short issue this time, because if I made it any longer I think it would be more technical than I would really like. Enjoy the brain-break!

What I’ll be covering next

Next issue: [LMG S13] Issue 168: Search engines

How do search engines … know everything?

Defragmentation is discouraged on solid-state devices, as each read/write operation causes the storage medium to degrade. Under typical usage, a solid-state device can be expected to last many years. Defragmentation involves lots of read/write (as you are reading files and storing them elsewhere on the disk), hastens the degradation, and will shorten the lifespan of solid-state devices unnecessarily. ↩

Issue 166: A cause of system slowdown: caches

2022-04-02T08:00:00+08:00

Previously: There are easy and quick ways to check the validity of the most common advice for resolving system slowdown. But it still seems to happen even after these tips have been tried.

Last issue, I walked through common causes of system slowdown suggested by generic tech websites, and explained simple ways of checking if these are really the cause. Quite often, they are not, especially if you are the kind who is careful about internet usage and does regular system maintenance.

So what is going on?

Caches, caches, and more caches

In Issue 39), I explained what caches are: places where you (temporarily) store the result of lookups, so you don’t have to keep performing the lookup again. In context, this referred to DNS lookups: operations that translate a domain name (such as google.com) to an IP address (such as 173.194.217.100).

But caches are everywhere, not just in DNS.

Examples

When you open any Microsoft Office application, it shows you your most recently accessed documents: that’s a cache! (The info has to be stored somewhere, right?).

Windows caches your old installation files “just in case”.

Your browser definitely caches your browsing data (they are called “Temporary internet files”).

And most apps have a cache of some kind or other to hold data which they think you will want to access again soon.

These are generally harmless uses of caches, albeit possibly annoying (when Adobe Premiere caches video files that take up gigabytes of space …). Caches are predicated on the notion that access from a storage disk (on the same device), while slow, is still faster than its alternative (fetching the data from its source, over the internet or through a computationally expensive calculation).

App caches: a short history

Before the internet went mainstream, a lot of software came on compact discs or diskettes. These were slow to spin up, slow to read, and slow to swap out. They were nonetheless necessary in the days when hard disks were still low on storage, and expensive.

As hard disks increased in capacity and decreased in price, it made sense to copy the information from these diskettes or compact discs into the hard disk, and subsequently access the data from disk. This process is what we know as software installation (bet you saw that one coming!).

These days, the disk-is-faster assumption is less true across a spectrum of uses. Especially when it comes to mobile devices.

Mobile storage and internet

A quick note on mobile storage, which I unfortunately missed out on Issue 119) on laptop and desktop solid-state disks (SSDs).

Mobile devices generally do not use the same kinds of SSDs that laptops use. Those are bulkier, use more power (not good for mobile battery life), and run hotter. Laptop SSDs use an interface called NVMe (which iPhones use as well), while most Android devices’ storage use an interface called UFS which is slower (but uses less energy).

At the same time, internet access on phones is speeding up. 4G/5G technology has increased throughput, while maintaining more or less the same latency. On the other hand, with the cloud becoming a staple of everyday life, tech companies have poured immense resources into increasing their servers’ responsiveness, resulting in lower latency for internet access.

Disk caching is no longer king.

Caches on mobile devices

On Android, a tip not mentioned in the previous issue is to clear the app cache. For many apps, the app cache can grow to tens or hundreds of megabytes. As Android device storage slows down over time, the app cache gets slower and no longer serves to speed up the app. So clearing the cache regularly can help keep performance from degrading too much.

However, Android itself uses caches for system processes as well, and these are usually not accessible to the user. That means the only practical way for most users to clear this cache is usually to perform a factory reset.

Issue summary: Caches speed up app operations by storing temporary data on the device’s storage. This assumes that access to storage is much faster than access to the file’s original source. On Android, users can clear an app’s cache, but not the system cache.

What I’ll be covering next

Next issue: [LMG S13] Issue 167: Database fragmentation

This issue, I explained why caches no longer work as well as expected.

Next issue, let’s poke a little deeper: why does clearing the cache work?

Issue 165: The myths of system slowdown

2022-03-26T08:00:00+08:00

Previously: Linux software is distributed through Linux distros. The maintainers of distros maintain repositories of software that have been tested with the distro. Most users will access software in the distro’s repositories through a program called a package manager. So users have full control over when updates and new software should be installed.

Once your laptop hits the magical 1-year window, it somehow seems to … get slower. And slower. Everything takes just a fraction longer. What used to happen near-instantaneously now seems to take a split-second pause. The loading spinner animation feels like it plays just a little longer. And it just gets worse from there with age.

Google search results have a number of things to say about why it happens:

Programs starting up when booting

This is primarily an issue when you are booting up your laptop (Issue 112)) and logging in. Once you log in to your operating system (OS), your OS will run the startup programs (which you can disable), so if you are trying to use your laptop right after logging in, this may cause some slowdown.

If your laptop is still slow about 10 minutes after OS login, this is probably not the cause.

Programs running in background

You can check this easily: open Task Manager (in Windows), and see if CPU, Memory, or Disk are significantly high. You can click on those columns to put the highest-usage processes at the top. Often it is some kind of antivirus or malware scanner that is hitting the disk and causing things to be slow. Wait for these programs to finish (if legitimate), then see if your computer still feels slow.

Insufficient memory

This is easy to check in Task Manager too. On the performance charts on the right, see if memory usage is near 100%. If it is, try closing some applications until usage drops below 100%, and see if system performance improves after a few minutes.

Malware

Malware that slows down your laptop usually does so by taking up a lot of CPU time, or using a lot of memory. You can test this using the above tip too.

If your computer fan is spinning noisily even though CPU/Memory/Disk are low, it’s either malware that is smart enough to hide itself, or blocked computer vent (which is causing your CPU fan to be ineffective). These are easy checks to make too.

OS updates

Just update your OS. Most of the time this does not fix system slowness.

Disk full

Open My Computer (in Windows), which should show how full your system partition is. If it is below 80% full, this probably isn’t the issue, regardless of whether your system is on a hard disk or solid state disk (SSD, see Issue 119)).

File fragmentation

You can run Drive Optimizer on your system partition. Yes, there is a lot of discussion about how defragmenting an SSD is bad; fortunately Drive Optimizer is pretty smart about this and will only carry out defragmentation on hard disks; the option is called Optimize if your system partition is on a solid state disk.

This usually helps a little, especially if the hard disk is quite badly fragmented. If it is only lightly fragmented, or if you are using a solid state disk, this usually won’t resolve any slowdown.

These are some quick ways to check on the validity of these myths about the causes of system slowdown. In many cases they do help! But there seems to be this general malaise that affects even the most cared-for and most well-maintained laptop, after an operating system has been used for years without a reinstall.

Issue summary: There are easy and quick ways to check the validity of the most common advice for resolving system slowdown. But it still seems to happen even after these tips have been tried.

Knowing what we know, this far into this layman’s guide, I’ll look into some other possible causes next issue!

What I’ll be covering next

Next issue: [LMG S13] Issue 166: A cause of system slowdown: caches

Issue 164: Linux, the universal operating system

2022-03-19T08:00:00+08:00

Previously: Software that we use usually comes from the OS makers, or from third-party developers. These two groups of developers are not the same, and might even have conflicting intentions and goals.

Last issue, we looked at the following categories of software that an end-user might need:

System updates
Software by the OS maker (first-party software)
Software from other developers (third-party software)

In general, “trusted” software comes from other sources: compact discs or the internet. While “trusted” software comes from a central, authorised source: usually some kind of app store.

This leads to a lot of conflict over who gatekeeps the list of “trusted” software. This privilege gives app store owners a lot of power, which they claim to earn by investing capital into making the OS possible, and implementing screening and filtering processes to ensure only secure software makes its way into the list.

I will not contest those claims here, but instead invite you to consider: what if it were possible for other app store makers to join in curating lists of software for that operating system?

What is Linux?

Today the term “Linux” refers to a lot of things, which is why we need to clarify here: the term originally referred to the Linux kernel, which is the core of the operating system.

The kernel by itself doesn’t do anything without all the other programs that make it actually useable by an informed user: to create, edit, and browse files and folders, run service programs, etc. This collection of programs, along with the kernel itself, is officially referred to as GNU/Linux (but s’okay, you can just say “Linux” and we understand you mean GNU/Linux).

So, GNU/Linux is similar to Windows and MacOS: they are base operating systems, capable of doing stuff but not actually useful yet.

Where all our software at?

Software for Linux

In the very early days, because there were a number of different hardware configurations that weren’t as compatible as they are now, Linux software was distributed as source code. You downloaded a zip file containing programming code, you ran a compiler program to compile the code (Issue 54)) into an executable program. Then there is usually an installer, a shell script that puts the compiled program in the correct place so the OS can find it, and creates other helper files (like configuration information).

And then you had to check their website (or even email them, in the days before the mainstream Internet) regularly to see if there are any bugfixes, and if yes, you downloaded the new source code and repeated the process …

This was obviously tedious, so people started to compile the useful programs into their own list. This effort expanded, and became automated, so that it was not only a list, it was a collection of different versions of compiled programs for different CPUs (each version of the program is called a package). If you were a developer for one of the programs in that list, once you made a new version of your package you could submit it to these guys, they would incorporate the required changes and then add a listing for the newest version.

These projects came to be called Linux distributions, or Linux distros in short. Distros maintained repositories of software for their specific distribution—the people involved are called maintainers. Maintainers check new versions of software to ensure that it will work as expected on their distro. Another category of software, called package managers, were created for users to be able to use these repositories easily: to check for updates, install them, and perform any other required maintenance.

More and more distros started in the 80s and 90s, as groups of Linux users and developers decided to branch off based on differing principles and philosophies for managing a Linux computer and its software. Today, you have distros focused on reliability and stability, distros focused on simplicity, distros focused on user-friendliness, distros focused on scientific computing, distros focused on hackability, …

It is important to note that the Linux kernel development team itself does not maintain any distros. Any updates to the kernel are for distro maintainers to incorporate into their respective repositories.

Linux software distribution

This means that for the bulk of users, software distribution on Linux is centred around the distro’s repository. Through the package manager, users can search for software, install it, and update it. They can also add the URLs of external repositories to access software from them.

At the same time, if users wish, they are still able to download compiled executables from the internet and run them (with the usual caveats, of course). They can also download source code, compile it themselves, and then run it. These options do not offer the same ease of maintainability as software installed through a package manager, since there is no repository to check for updates.

This is what many install pages look like on Linux, for software with Linux versions.
The Install button usually downloads a compiled executable, which can be run on the computer.
But installing through the distro’s package manager is recommended.

No system updates sneaking up on you from the OS maker; updates and new software all come from the repository through the package manager. Unless you decide otherwise.

Issue summary: Linux software is distributed through Linux distros. The maintainers of distros maintain repositories of software that have been tested with the distro. Most users will access software in the distro’s repositories through a program called a package manager. So users have full control over when updates and new software should be installed.

This is as far as I’ll go for technical detail on Linux. I meant this to pick up from Season 8) on apps. It’s easier to go into the bird’s-eye view of how this works on Linux, because I’m more familiar with it; on the Windows and MacOS side of things it tends to be more esoteric and proprietary.

And it’s instructive to know this because … so far this season, we have been talking about distribution—content, code, and software. Notice how all of them involve infrastructure: worldwide clusters of servers for content distribution, a history of code changes for code distribution (and collaboration), and now we have repositories and package managers for software distribution.

What happens as this infrastructure ages?

What I’ll be covering next

Next issue: [LMG S13] Issue 165: The myths of system slowdown

We won’t dive into topics as broad as digital infrastructure aging; that’s beyond the scope for a layperson’s newsletter I think!

Let’s bring it back to personal scale: what happens as your system ages? The predominant symptom that manifests itself is general slowdown: your computer takes longer to switch on, open any app, save any file, and even to shut down. For years I have been googling for reasons why this happens, and satisfying/useful/sensible answers are almost non-existent. The pithy, vague answers about background services (“bloatware”), outdated apps, malware etc are almost insulting; I have relatives who use the same handful of apps, and even after clearing old/large files their phone is still slow!

Next issue, I attempt to look into these stated reasons and see if they make sense.

Issue 163: System & software ecosystems

2022-03-12T08:00:00+08:00

Previously: Typeface families consist of multiple fonts for each style in the typeface. Each font consists of glyphs, which are mathematical shapes described by curves joining points. These shapes need to be rasterised for display on a computer screen, or for printing on paper. Font files usually come in .ttf, .otf, or .woff formats.

Brief recap of the past few issues:

Content distribution: Images and other media are distributed with the help of content distribution networks (CDNs, Issue 160)), which have regional servers closer to users.
Code distribution: Webpage documents and web scripts (in Javascript) are distributed from the host server (which may comprise more than one computer).

And all of these takes place over the World Wide Web, often through the HTTP protocol (Issue 7)). That is how data gets to us when we use the internet.

What about the software we use, and the oft-dreaded Windows Updates? How does that get to us?

Software distribution on Windows and MacOS

Okay there’s not actually anything new to say here, just checking if we have the same understanding of how to find new software:

You’ve got your system updates and whatnot, inconveniences that sometimes force themselves on you. These generally come from a secure server from the operating system (OS) maker, which is Microsoft or Apple.

You’ve got software made by Microsoft/Apple, which you either download from their website or install through your browser.

Then there’s the software in their app stores. These app stores are listings of software which developers pay to have their software listed in. A big selling point of app stores is their supposed security: app stores usually have a screening process to ensure that submitted apps are not doing Evil Things™ which harm their users. So when you download an app it is assumed that this app has passed some kind of rigorous screening process.

Developers often pay a proportion of their revenue to the app stores for this “privilege”.

And then there is … all the other software you can download from the internet.

I’m guessing the internet is where most of the desktop/laptop software you use comes from.

Software distribution on Android and iPhone

On the mobile side of things, it looks remarkably similar, but with the weightage somewhat different.

System updates, coming from the OS maker — check.

Software made by the OS maker — check, but coming through app store instead of internet.

App stores — check, still the same.

Software from the internet — Android allows installing software from “unknown sources”, but you’ll have to enable a system setting to allow that. It is off by default. On iPhone, this is just not possible.¹

I’m guessing the app store is where most of the mobile apps you use come from.

Software as an ecosystem

Notice that in neither case is finding software like foraging for berries: there is an entire ecosystem that goes into making these actions possible!

The OS makers obviously have to distribute their OS, supporting software, and updates reliably and securely, so that other software can rely on its continued existence. An OS by itself doesn’t usually do much for users; they need software to create and manage their files, and access the internet. And much of this software is going to be created by other developers, not the OS makers.

The developers need documentation and sample code to understand how to write the software, and these usually come from the OS makers. But it can also come from a thriving community of other developers who are writing software for the same system.

And then there are the discovery mechanisms that users need to find useful software; Google is the fallback when this doesn’t exist, but you usually want users to have a better experience than googling for installers and potentially installing malware.

Even this understanding is incomplete, but it’s the beginning of a more nuanced model of software development that will help you understand why software often does not do what it should. We are talking about factions in software development.

Far too often I see users who have no awareness of this divide, and seem to operate on an assumption that software comes from “programmers”, a hallowed, unreachable group of entities that blesses users with features or curses them with bugs at their whim.

Issue summary: Software that we use usually comes from the OS makers, or from third-party developers. These two groups of developers are not the same, and might even have conflicting intentions and goals.

If I keep going on in this vein I’m going to bring in politics, and I don’t want to do that. Instead, I’ll introduce a slightly different kind of software ecosystem, which non-Linux users are likely not familiar with. My hope is for you to see that *handwaves at above paragraphs* this isn’t the only way for software distribution to work!

What I’ll be covering next

Next issue: [LMG S13] Issue 164: Linux, the universal operating system

What does a model look like when there is no central app store, controlled by the OS makers? That is how the Linux distribution system works!

Okay not true, you can replace the OS on your phone through a process called jailbreaking, but this is a layperson’s newsletter and I do not recommend this without much more extensive reading and careful consideration. ↩

Issue 162: Fonts

2022-03-05T08:00:00+08:00

Previously: Cross-site scripting attacks occur when a webpage loads malicious code from a third-party, usually carried out by a script in the page. Today, websites are protected from loading unauthorised scripts through cross-origin resource sharing (CORS) policy implemented in browsers, which only allows a website to load scripts from authorised domains.

This is the issue that doesn’t really fit anywhere, but this season is about lots of things we take for granted and fonts are one of them.

I could probably fill at least half a season talking about fonts and typesetting, but let’s stick to the basics here.

What is a font?

I’m going with Source Sans, an open-source typeface designed in-house by Adobe. Let’s open one up one of its fonts¹, Source Sans Pro Regular, in a font editor² and see:

Source Sans Pro Regular, in Fontforge.

You might already be aware that fonts contain glyphs, which are the shapes of each separate character. What makes up those glyphs?

Uppercase Q from Source Sans Pro Regular.

These glyphs are mathematical shapes. They are stored as a series of points, joined by curves³ or straight lines⁴, which are stored as numbers. The diagrams we draw in Microsoft Word are much the same!

Font variants

When you bold or italicise text, the operating system swaps in the appropriate glyphs from the bold or italic fonts from the same typeface family. A lot of careful work goes into ensuring that these fonts are recognised as belonging to the same typeface, or this feature would not work.

This means that when you install fonts, remember to grab the bold and italic fonts as well!

Many applications, if they are unable to find the bold/italic fonts, will artificially “bold” or “italicise” the regular font by thickening the glyph, or slanting it. Any graphic designer worth their salt will notice this immediately; even without scrutinising the font, it will feel “off” in some vaguely inscrutable way until you take a closer look and notice the proportions are wrong.

Font display

For an application to be able to support text formatting, it must have a program called a text rendering engine. This program takes a single long string of text and determines the appropriate places to chop it up with line breaks. To do that, it first needs to convert the glyph shapes from mathematical formulas into actual real lit pixels or real inky droplets (in a process called rasterisation, Issue 122)).

Then their dimensions have to be considered in the line of text, to know where the line breaks should be placed. Because glyphs are not actually rectangular boxes and they protrude in different ways, optically they need some horizontal adjustment (called kerning) to look evenly spaced optically, so that needs to be done too⁵.

Kerning for some common glyph pairs in Source Sans Pro Regular.

A kern value of 0 or blank means no kerning is required. A negative value means the letters need to be brought closer, and a positive value means they need to be spaced further (quite rare).

Advanced renderers might even do other things, like avoiding too many terminal hyphens on consecutive lines (looks ugly), or making microadjustments to letter spacing. But, oops! This changes the line length, so the engine needs to go back to re-check the line breaks. This is an iterative process.

Text rendering engines are an art in themselves, and we are not going to go in depth here.

Font formats

The classic file extension is .ttf, which stands for Truetype font, a font format created by Apple in the late 1980s and subsequently adopted by other systems.

These days, you might also see .otf, which stands for Opentype font. This is a more modern font format, co-developed in the mid-90s by Microsoft and Adobe. This adds much more functionality and new features, which after some deliberation I have decided not to write about—it is simply not a layperson topic!

If you do web development, you might also see .woff, the Web Open Font Format, co-developed by Mozilla and other type organisations. It shares some features in common with TTF and OTF, but adds other features for licensing information, which is usually more important for the web, where these font files need to be downloaded to the users’ computers.

Issue summary: Typeface families consist of multiple fonts for each style in the typeface. Each font consists of glyphs, which are mathematical shapes described by curves joining points. These shapes need to be rasterised for display on a computer screen, or for printing on paper. Font files usually come in .ttf, .otf, or .woff formats.

This difference in representation vs display, fonts-as-mathematical-shapes vs fonts-as-pixels-or-dots, has been and continues to be the cause of much weeping and gnashing of teeth. But I’ve decided it’s not worth delving into that for a layman’s guide to computing—perhaps in a separate publication!

What I’ll be covering next

Next issue: [LMG S13] Issue 163: System & software ecosystems

With this diversion over, let’s return to talking about distribution. Content distribution, code distribution, and next issue I’ll move on to software distribution!

With this I have also cleared my backlog of questions, and will be closing the below section as well.

Sometime in the future: What is:

~~OpenType? And what are fonts anyway? [Issue 42]~~

A font is a single style in a typeface family. The full family will usually have regular/bold/italic styles. More advanced typefaces may have small caps, display, and caption fonts. ↩
The one I use is called Fontforge, and it is open-source. ↩
For the curious, the specific type of curve used in fonts are Bézier curves. ↩
Don’t be alarmed if the mathematicians among us casually remark that a straight line is a trivial example of a curve. ↩
The kerning information is created by the font designer and embedded in the font file. Applications usually pass this information to the text rendering engine, although some might not actually use it. ↩

Issue 161: Security and XSS

2022-02-26T08:59:00+08:00

Previously: A content delivery network comprises multiple servers around the world that are able to quickly distribute static content (typically images and video) to viewers that request it. This avoids overloading the hosting server, which would otherwise have to serve data over the network, possibly through many intermediary hops.

When you load a modern webpage with all its bells and whistles, it is usually loading its content from a content delivery network (CDN; see previous issue). At the same time, it is running scripts that came with the webpage. These scripts may load other scripts on the same server (first-party scripts), or scripts on other servers (third-party scripts).

What could go wrong?

First-party scripts

These are scripts you trust, because you host them on your own server (or a server you administer). Ideally you also have network security measures in place and other ways to ensure those scripts are not modified by malicious actors.

It’s usually safe to load them in the webpage because they are from the same site (i.e. same server).

Third-party scripts

These are scripts that are loaded from a remote server. You’d usually do this to load scripts from service providers: for analytics, to serve online ads, or to use libraries and frameworks (Issue 17), Issue 18)). This is mightily convenient: as a third-party service provider, you have the flexibility of updating this script and immediately benefiting your client without them having to do anything. Can’t beat that for convenience!

But once you open the door to third-party scripts, they could be loaded from anywhere. And without some mechanism for verification, the client won’t actually know if they are loading your script, or someone else’s.

Cross-site scripting (XSS)

If a webpage is insecurely scripted, e.g. by inserting data directly from a request without verification, a malicious actor might be able to subvert the source of the request and inject malicious code into the page. This malicious code, though sandboxed by the browser, still has access to data that is on the page.

This is called a cross-site scripting attack.

The weakest link

The chain of security is only as strong as its weakest link, so even if your own security is tight, a malicious actor would look at your tech stack (the set of hardware/software/services your company uses), notice that your webpage is loading scripts from a third party, and attempt to hack the third-party’s servers (which might be less secure). When you draw on features from multiple libraries, you are in effect setting up a circle of trust that is only as secure as the least secure library/third-party in your web.

Protections

This is why all browsers today have cross-origin resource sharing (CORS) active by default. This prevents scripts from loading other third-party scripts; only loading of first-party scripts are supported by default. To enable loading of scripts from other sources, your server must include special data in the HTTP response header (Issue 8)) that specify these sources explicitly. It’s tedious, but it is much more secure, and it is also why learning web programming is much harder today than it was a few years ago.

Issue summary: Cross-site scripting attacks occur when a webpage loads malicious code from a third-party, usually carried out by a script in the page. Today, websites are protected from loading unauthorised scripts through cross-origin resource sharing (CORS) policy implemented in browsers, which only allows a website to load scripts from authorised domains.

Ughh, this is already getting more tedious and bureaucratic to write about. As you can see, even in the world of programming there is a load of red-tape to cut through, all to protect our works from malicious actors and ensure there is a chain of authorisation running through everything. I guess there’s a life lesson in here somewhere.

What I’ll be covering next

Next issue: [LMG S13] Issue 162: Fonts

I’ve covered content distribution, code distribution (for the web), and now I think we can go a little wider: let’s talk about software distribution!

But before that I want to slot in a little issue about fonts: just what are they? And how do they work?

// Hopefully all my readers have had the experience of searching for and actually downloading software. What kind of system is behind this? And how do system and software updates actually get to our devices?

Get ready for a deeper dive!

Sometime in the future: What is:

~~XSS? [Issue 8]~~
OpenType? And what are fonts anyway? [Issue 42]

Issue 160: CDNs and content distribution

2022-02-19T08:00:00+08:00

Previously: Instead of GPS satellites, smartphones can also use wifi points and cell towers to determine their position (if enabled in the OS).

All businessmen know that distribution is everything. How good your product is, is secondary to how you get your product to the customer. This act of getting things to your customer—it’s called distribution, and entire businesses have been built around excellent distribution.

In Issue 157), I described how time is synchronised from time source to server and on to other servers, down the strata of the hierarchy tree of time servers. Whereas GPS/wifi location (Issue 158)) has a much shallower distribution system: everybody gets their location directly from a GPS satellite if there’s nothing else available, otherwise they get it from the nearest wifi point or cell tower.

What about content?

The difficulties of content

You make a website, type in the headers and body text, upload the images and videos … and it just works right?

Let’s think through the distribution of that content. Text is generally small in size and easy to pass around, even through multiple hops (Issue 36)) from server to client.

Server load

What about the heavy stuff, like hi-res images and videos? Thousands or millions of clients all requesting the same large video file from your hosting server. That server is going to be spending many CPU cycles (Issue 58)) receiving requests, retrieving the data, splitting and encapsulating it into data packets to be sent out. All that processing adds to the server load. If there are too many clients waiting for the same data … they gonna wait. And that adds to latency (Issue 36)); those viewers are going to be seeing loading spinners for a while.

Some of that processing can be mitigated with techniques such as caching (Issue 39)), but not enough; you will eventually need to add more servers.

Bandwidth and transfer fees

Your hosting provider is going to be paying lots of egress fees to transfer your data out of their servers (imagine sending the same 4GB video to a few thousand Youtube viewers), and they’ll likely pass on the fees to you as well.

Latency again

If the client is geographically far away from the server, possibly even on the other side of the world, the data is going to go through a lot of hops from server to server. And if any of the servers along the way drop the packet, it is going to need to be resent.

Improving content distribution

So how do we lighten the server load on the hosting company’s servers, reduce the amount of data to transfer from that server, and improve latency for the clients?

You place distribution servers as close as possible to the clients, wherever they may be. This usually means you have your servers globally distributed, with regional clusters.

You place the most often requested files from that region in its distribution server, so that it can serve those files without the request hitting the hosting server. Because the distribution server is so near the client, the data goes through fewer hops to get to the client.

The main document data is still served from the hosting provider, so that any changes you make to the page get served to clients almost immediately. Otherwise every little change you make has to be reflected in every distribution server that has a copy of that data. For this reason, distribution servers are typically used for static data: data that doesn’t change frequently, or at all—images and videos especially.

A content delivery network (CDN)

These distribution servers, along with their supporting infrastructure, are collectively known as a content delivery network (CDN). Sometimes, when you are grabbing the URL of an image, you might see “cdn” in the URL domain—now you know what it means!

Issue summary: A content delivery network comprises multiple servers around the world that are able to quickly distribute static content (typically images and video) to viewers that request it. This avoids overloading the hosting server, which would otherwise have to serve data over the network, possibly through many intermediary hops.

What I’ll be covering next

Next issue: [LMG S13] Issue 161: Security and XSS

With this piece of info, our mental picture of the loading of a webpage is getting more and more complex. No longer can we assume that the web document itself, its images, videos, and other content are all being loaded from the same server: static content might be coming from a CDN, and other content (e.g. ads) might be loaded from elsewhere.

Let’s talk about this from a security standpoint next issue.

Sometime in the future: What is:

XSS? [Issue 8]
OpenType? And what are fonts anyway? [Issue 42]

Issue 159: Wifi & cell tower location tracking

2022-02-12T08:00:00+08:00

Previously: To get your location using GPS, your phone requests information from four overhead GPS satellites: their location, and the distance between them and your phone. With this information, your phone can calculate its location.

Okay, so what happens when you are in a tunnel or building and can’t get GPS? How are you still able to use Google Maps to navigate that new sprawl of a mall?

Wifi Positioning System (WPS)

The principles of triangulation still work within a building, thank math 🙏 but now we need other landmarks to replace GPS satellites.

What is something with a known (and ideally fixed) location, is electrically powered to receive and respond to signals, and there are enough of them to provide a sufficient number of landmarks for triangulation? If you are in a building with wifi, the wifi access points scattered throughout the building can probably provide this.

No protocol is involved in wifi positioning, largely because most routers do not carry a precise hardware clock and do not have any way to know their location precisely, and therefore cannot communicate this information meaningfully. Instead, wifi positioning is a collection of techniques for guessing your location. Your smartphone uses these techniques (usually through its operating system) in conjunction with available wifi networks around you to determine its own location.

Wifi positioning techniques

One way to figure out your location to proximal wifi points is to use the signal strength as a weak analogue for your distance from them. You can do a very rough position estimate with this.

Another common technique is to look up the hardware address, or even IP address of the wifi point you are connected to, and just use it directly (with the assumption that wifi signals get too weak outside of a 10 m radius, so you have your location accurate to within ±10 m).

Wifi location databases

One way to keep track of wifi access points and their locations is through a global, public database. A number of these are available, such as the Mozilla Location Service.

Cell tower triangulation

What happens when you are outdoors, far from any wifi point? As long as you have mobile data enabled and are not in airplane mode, you are still going to be getting your cell signal from a cell tower … which also meet the three basic criteria for device-based triangulation 😉

Your smartphone can thus triangulate its location from cell towers that it is able to reach. Again, there is no protocol for this, since your smartphone does not communicate with the towers for the express purpose of obtaining location; it is a set of similar techniques, often implemented in the operating system.

Issue summary: Instead of GPS satellites, smartphones can also use wifi points and cell towers to determine their position (if enabled in the OS).

What I’ll be covering next

Next issue: [LMG S13] Issue 160: CDNs and content distribution

Coincidentally, starting with time turned out to be a good idea: time information is sort of like content. It has an origin, and it gets distributed to “consumers” who want that information. As with all distribution systems, you have cascades of product that flow outward from this origin.

We have covered time and space (I mean, location). Let’s move on to data: how does data get around the world from a few central sources?

Back in Issue 73), when I explained how online advertising works, I mentioned that advertising content is served from a content delivery network (CDN). What is this and how does it work?

Sometime in the future: What is:

XSS? [Issue 8]
OpenType? And what are fonts anyway? [Issue 42]

Issue 158: GPS

2022-02-05T08:00:00+08:00

Previously: Time is synchronised from higher-precision sources through a protocol called Network Time Protocol (NTP). A public pool of time servers is available for synchronisation at pool.ntp.org.

Ah, GPS. The only topic that actually has almost nothing to do with computing … and yet the mobile computers in our pocket rely on it so much.

A short history

The Global Positioning System (GPS) was born of the space age, in 1973, before computers even went mainstream. It was originally used for military applications, particularly for navigation. It was first widely used in a political conflict in the Gulf War (1990–1991). The public finally had access to it in 1996, after US President Bill Clinton issued a policy directive for it to be dual-use (used for both military and civilian purposes).

Principles

The principle of triangulation far predates GPS. Triangles have been used to estimate distance since antiquity; there is evidence of such techniques being used from sources as far back as 6th century BC.

In essence, if you know the location of two reference points, then with those two reference points and your own location as the third point, you can draw a triangle and solve a little geometry puzzle to figure out your own location.

If you were hiking or just taking a walk, you need to have at least two landmarks with locations marked on a map. As long as you stop somewhere with good visibility, you can get your bearings to those landmarks on the map, draw a line backward from each landmark following your bearing, and the intersection of the lines from each landmark will show your location.

But digital compasses are not ubiquitous in all devices yet; typically they are only included in high-end phones. A simpler way for devices to get their location is to estimate their distance to the two landmarks. This they can do using radio waves, which travel at (close to) the speed of light: 300 million metres per second. Provided the landmarks contain devices that can receive this signal and send it back, the time delay can be used to calculate the distance between the device and landmark.

The GPS network

For this to work globally, you are going to need such landmarks positioned all over the world, within receiving range of any device. These landmarks need to be:

electrically powered, so they can broadcast signals
tall, very tall, since radio waves do not follow Earth’s curvature. A short tower would not be able to receive signals for devices that are too far away: the signal would be blocked by the Earths curved surface!
aware of their own position, and synchronised at regular intervals

Let’s solve problem 2 first: instead of building millions of towers worldwide (and how would we do that on the oceans?), we can just launch satellites into space to serve as landmarks for triangulating position. GPS satellites orbit at an altitude of 20,200 km, almost twice of Earth’s diameter, allowing any of them to be reached from almost half of the Earth’s surface.

Solving problem 3: if we launch enough satellites, they can continually synchronise their clocks with each other, and triangulate their own position relative to other satellites.

Solving problem 1: satellites all face this common problem of needing electrical power. They have largely resolved it with the use of solar panels.

Getting your location using GPS

When your smartphone tries to triangulate its location using GPS, it gets its distance from four overhead satellites, along with their location (remember that there is an altitude component!). It then determines its position from this information using triangulation.

The triangulation calculation here is rather more complicated, given that the satellites are in constant motion and not geostationary (above the same spot on earth all the time). At an orbit altitude of 20,200 km, even radio waves, travelling at lightspeed, still take about 0.07 seconds to reach your phone from a GPS satellite. But it is possible, and your phone does it each time you get your location from GPS—to a precision of about 10m!

Issue summary: To get your location using GPS, your phone receives information from four overhead GPS satellites: their location, and the distance between them and your phone. With this information, your phone can calculate its location.

What I’ll be covering next

Next issue: [LMG S13] Issue 159: Wifi & cell tower location tracking

Receiving radio signals all the time requires the receiving unit to be on all the time; if you use GPS heavily you will find your battery draining quickly!

The reason our smartphones can maintain such good battery life is that they don’t use GPS most of the time. After all, it is not possible to get GPS in a lot of places: in basements, tunnels, anywhere you can’t get a solid signal from four overhead satellites. There are less energy-costly ways to get your location these days, especially if high precision isn’t necessary.

Next issue, let’s look at how wifi and cell towers come in!

Sometime in the future: What is:

XSS? [Issue 8]
OpenType? And what are fonts anyway? [Issue 42]

Issue 157: NTP and time-syncing

2022-01-29T08:00:00+08:00

Previously: To speed up execution and avoid translation overhead, some systems employ ahead-of-time translation, storing the translated instructions to be executed in future. But many systems employ a mix of just-in-time (JIT) and ahead-of-time (AOT) techniques.

This season, I’ll attempt to plug the gaps in the layperson’s working knowledge of Internet-related services. Time, location, wifi and mobile data … almost all will be covered this season!

Global time information

Frequent fliers would no doubt be familiar with the existence of timezones: geographical bands stretching from the North to South pole, within which all locations are assumed to be running on the same regional time. These timezones used to be manually synchronised, by phone or telegram, via operators all over the globe.

Today, timezone information and other time information required for global coordination are maintained by the Internet Corporation for Assigned Names and Numbers (ICANN), which also maintains other lists of names and numbers owned by the Internet Assigned Numbers Authority, which we first met back in Issue 27). These lists are used by programmers worldwide when writing programs that require time coordination.

Time synchronisation

What about syncing time? Intuitively this process involves communication between computers, and anytime computers need to communicate, you can be sure a protocol is involved. We have seen a few protocols mentioned so far:

HTTP, the Hypertext Transfer Protocol, for sending web documents (aka webpages) and requests
DNS, the Domain Name Service protocol, to translate domain names (like google.com) to IP addresses (like 142.250.64.110)
DHCP, the Dynamic Host Configuration Protocol, used by routers to assign IP addresses to its client devices
USB, the Universal Serial Bus set of protocols, used for data transfer between a host computer and another device

… I hope that’s enough for an idea of where protocols are involved.

The protocol for time synchronisation is called the Network Time Protocol (NTP), in use since 1985—that makes it as old as me!

Time sources

In the past, people would look at a common time source—the town square clock tower, Big Ben, church bells, and so on—to get the time. When watches were invented, people who had one would synchronise their watches to these common time sources.

But watches, clock towers, and other sources of time can get out of sync: one second as measured by each device does not accurately match the scientific definition of a second:

The second is defined as being equal to the time duration of 9,192,631,770 periods of the radiation corresponding to the transition between the two hyperfine levels of the fundamental unperturbed ground-state of the caesium-133 atom.

That’s from The International System of Units¹, which I am not inclined to argue with in a layman’s newsletter. Needless to say, synchronising to a universal standard is not something the average layperson does. Only select organisations have the need to keep such accurate time on their own, typically using such precise instruments as atomic clocks.

Trickle-down time

How does the rest of the world get its time from these high-precision devices? It doesn’t; most of the world has no need for the sixteen zeroes of precision provided by an atomic clock. Instead, another cluster of servers synchronise their time to within microseconds of precision of these devices. And another cluster of servers synchronise their time to these microsecond-precision servers.

Each “layer of precision” is called a stratum in NTP. Time “trickles down” from higher-precision sources to lower-precision sources, down the stratum. As we descend the stratum, there are more and more devices providing time at that precision. If a time server synchronises to a stratum 1 server, it becomes a stratum 2 server; if it synchronises to a stratum 7 server, it becomes a stratum 8 server. The upper limit for stratum numbers is 15; a stratum 16 device is considered unsynchronised.

The stratum number is not an indication of quality or reliability, it only indicates distance from the reference time source.

Operating a time server

NTP is an open protocol, which means the protocol is readily available online, and anyone can run their own server implementing this protocol. If you don’t want to write your own software, you can also use the open-source ntp distribution, and compile it to make your own time server. Some large companies do this for their own large network, to improve the response time from time servers—public time servers, you can imagine, are under pretty heavy load!

So is this how our smartphones synchronise their time?

More or less, yes. (They actually use a slightly simpler protocol, in the interest of preserving battery life.)

Hypothetically, if you are designing your own operating system and having to provide a “set time automatically” feature, you can let your user connect to a public pool of time servers, pool.ntp.org (yes that is the actual name, and also the web address). This pool is further subdivided by continent and region, down to individual countries, since you will probably need specific time for your location. For instance, the specific time server pool for Singapore is sg.pool.ntp.org.

Issue summary: Time is synchronised from higher-precision sources through a protocol called Network Time Protocol (NTP). A public pool of time servers is available for synchronisation at pool.ntp.org.

I have been waiting ten seasons to write this, and it is finally out of my system!

What I’ll be covering next

Next issue: [LMG S13] Issue 158: GPS

This issue is about time. Next issue is about location. With these two issues I would have explained time and space! 🤭

Sometime in the future: What is:

XSS? [Issue 8]
OpenType? And what are fonts anyway? [Issue 42]

If you need a term to google for more of this kind of geekery, it is called metrology, the scientific study of measurement. ↩

Layman's Guide to Computing - Season 13

Issue 169: Search engine optimisation

PageRank

tags

Advertising

What I’ll be covering next

Going even slower

Better archives

Refactoring

What I’ll be covering next

Issue 168: Search engines

What is a search engine?

Indexing and arachnids

Ranking results

What I’ll be covering next

Issue 167: Database fragmentation

Storage fragmentation on mobile devices

SQLite, a blessing and a curse

What I’ll be covering next

Issue 166: A cause of system slowdown: caches

Caches, caches, and more caches

Examples

App caches: a short history

Mobile storage and internet

Caches on mobile devices

What I’ll be covering next

Issue 165: The myths of system slowdown

Programs starting up when booting

Programs running in background

Insufficient memory

Malware

OS updates

Disk full

File fragmentation

What I’ll be covering next

Issue 164: Linux, the universal operating system

What is Linux?

Software for Linux

Linux software distribution

What I’ll be covering next

Issue 163: System & software ecosystems

Software distribution on Windows and MacOS

Software distribution on Android and iPhone

Software as an ecosystem

What I’ll be covering next

Issue 162: Fonts

What is a font?

Font variants

Font display

Font formats

What I’ll be covering next

Issue 161: Security and XSS

First-party scripts

Third-party scripts

Cross-site scripting (XSS)

The weakest link

Protections

What I’ll be covering next

Issue 160: CDNs and content distribution

The difficulties of content

Server load

Bandwidth and transfer fees

Latency again

Improving content distribution

A content delivery network (CDN)

What I’ll be covering next

Issue 159: Wifi & cell tower location tracking

Wifi Positioning System (WPS)

Wifi positioning techniques

Wifi location databases

Cell tower triangulation

What I’ll be covering next

Issue 158: GPS

A short history

Principles

The GPS network

Getting your location using GPS

What I’ll be covering next

Issue 157: NTP and time-syncing

Global time information