Layman's Guide to Computing - Season 03

Issue 39: Caches and caching

2019-09-21T08:00:00+08:00

Previously: When a webpage document loads (Stage 1), it is processed by the web browser, which then loads other requested resources, such as stylesheets, images, and scripts (Stage 2). Scripts and other interactive code may then request more resources (data fetches, images, icons, data, etc) which are then loaded subsequently (Stage 3, 4, 5, …).

Last issue, I showed you using Developer Tools how a webpage is loaded in stages, and how that contributes to latency alongside DNS querying and data routing. We saw that it could add up to a few seconds of latency, which would be unbearable for a lot of people (first-world problems!).

Searching for anything takes time. Need to fill out a form? You need to search for a pen first. Need to call someone? Before speed dial and contacts apps existed, You used to need to look up a number in order to dial it. If you do it often enough, you would make sure you always had a pen with you, or you would write the number somewhere convenient for you to see so you don’t need to hunt for it.

Computers use the same trick, and it is called caching. Any information it needs repeatedly which is unchanging is stored in a cache. What kind of caches does a computer use to reduce network latency?

DNS cache

When you load a webpage or service and send requests to it, the first thing that happens is the DNS query. Once your OS knows the IP address to send requests to, it doesn’t make sense for it to keep querying the hostname, does it? IP addresses don’t change that quickly! The computer stores the hostnames and associated IP addresses in the DNS cache. You can view the DNS cache on a Windows PC by opening Command Prompt and typing ipconfig /displaydns.

The output of ipconfig /displaydns
(I used Powershell instead of Command Prompt, but it will look the same in Command Prompt.)

The computer always goes to the DNS cache first. If it can’t find the hostname (e.g. facebook.com) in the DNS cache, it will perform a lookup, then store the hostname and associated IP address in DNS cache. This information is stored for a day, then discarded, just in case the information has been refreshed.

Sometimes, this causes problems. A company or service may be in the process of moving servers and thus changing IP addresses. If the move didn’t go smoothly, your computer may be stuck with the wrong DNS information for some hosts. Or sometimes something just goes wrong with the DNS query and you are stuck with bad information.

When this happens, tech support will usually just flush the DNS cache to remove all information from it (yes, the bad with the good). On Windows, you can do so by opening Command Prompt and typing ipconfig /flushdns.

Browser cache

Notice that the first time your web browser loads any page you haven’t visited before, it often takes quite a while, but subsequent loads are really fast? That’s because we now skip a DNS query (grabbing the IP address from the DNS cache instead). Each time we do a lookup to retrieve a file or piece of information from the server, we can skip the DNS query!

The caching trick isn’t applied only to the IP address; many elements you see on the page have been cached: the document itself, images, scripts, stylesheets, … most of the elements from Issue 38) are cacheable, and the browser will cache it.

How long does the browser cache these files? It depends … I know it’s not an answer you like since it means more things to learn about, but I’ll keep it short.

Remember this? It’s the response header we saw from Issue 8) on HTTP error codes:

A response header from Hypothes.is

See the line that says Cache-Control: no-cache? That is the server, hypothes.is, asking my browser not to cache this response (because the next time it makes the same API request, the response might be different).

The server can also set a different Cache-Control time, especially for resources that are used repeatedly on pages (such as logos and headers). On heavily accessed sites, these resources may have a Cache-Control time of up to a year!

If you want to bypass the cached version of the page and force a full reload, you can do so on most browsers using the hotkey Ctrl-F5 instead of F5.

Flushing the browser cache

The browser typically stores cached files and data until it exceeds the storage limit set in the browser, at which point it will begin ejecting the oldest files. You can force the browser to remove these files through a menu setting usually named something like “Clear browsing data” or just “Clear data”.

On some sites, especially internet banking sites, you may be asked to flush your cache after you log out; they are asking you to clear cached files and cookies stored in the browser, especially if you are on a public computer or some device that is not your own. the hotkey for doing so (if you are on a laptop) is usually Ctrl-Shift-Del.

Issue summary: Your computer and browser speed up a lot of lookups by caching information that is unlikely to change from the last view. When the same information is requested, your computer or browser will first look in the cache to find that information, and retrieve it from cache if it is there, otherwise it will load the information (and store it in cache if allowed to). There are usually ways to bypass a cache if the information is stale or no longer correct.

And Season 3’s a wrap! I know I may have mentioned that I don’t intend to write howtos in this newsletter, but clearing the cache is something I google for so often, and I see many others googling for it too, that I figured it might help to include a bit of info for those who want to know. At worst, many more people now know how to force-reload a page … 😅

What I’ll be covering next

Next season: I’ve talked so much about networks: how they work and what they do, and then used that to build up to an explanation for “why is the internet so slow?”. It’s not a complete explanation yet—that will still take awhile—but I think we’re off to a good start.

As I wrote Season 3, it gradually became clear to me that before I start writing about cloud computing, I’m going to need to talk about data first. Computers ultimately store everything in binary, like 010101011100101000100101, so how do they use something so basic to represent everything from text to images to audio to videos and more? and how do they set up a system for storing and retrieving this information easily?

I’m not going to talk about binary and hex numbers much if at all; I don’t think it’s relevant for a newsletter like this. But since so many things are measured in bits and bytes, it’s impossible to escape that discussion. I want to build the issues up to answer questions I’ve been getting, like “why do my JPG files have this weird fuzz”, and “why are my audio files so large”, and “can I make this zip file smaller by putting it in another zip file”, and “why do I get these weird rectangles or question marks in my web browser”, and many more :)

Sometime in the future: What is:

booting up? [Issue 15]
a cookie? [Issue 8]
~~a cache? [Issue 8]~~
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
those ‘\r\n’s in the HTTP request packet [Issue 12,17]?
a good reason developers write code and give it away for free online? [Issue 21]
ASCII? [Issue 23]
compiling code into an application [Issue 26]?
firmware? [Issue 34]
What is HTML [Issue 38]

Issue 38: Loading a web page

2019-09-14T08:00:00+08:00

Previously: Data packets hop from server to server. The more hops a packet must undergo, the longer the latency. The slower the servers along the route, the longer the latency as well.

Last issue, I showed the route taken by data packets being sent to google.com, and to baidu.com. We don’t know what is in those data packets (yet), so in this issue I want to show you just how many requests/responses are involved in loading a webpage. I’ll be doing so using a tool that is available in Chrome and Firefox, which you might have accidentally opened before when you pressed some unknown hotkey combination, and wondered if you broke your browser. This tool is called Developer Tools, and it opens any time you right-click something in a webpage and then click ‘Inspect Element’, or if you press the Ctrl-Alt-I hotkey. It shows you what the backend of a webpage looks like, and today we’ll look at one of its features.

Loading a Github webpage

The Github repository where I keep my laymansguide files can be viewed on a webpage, and it looks like this:

Github page for laymansguide

Loading this page with the Network tab of Developer Tools open produces this report:

Developer Tools, Network view

This is a summary of the network activity that is happening while the page is loading. The top ribbon, showing time in milliseconds (ms) and a series of horizontal lines, is what is known as a waterfall chart. (You can also see this chart in the rightmost column below the ribbon.)

Each line represents the loading of a resource—anything that the page requests while it is loading. Lines stacked on top of each other represent resources that are being loaded simultaneously (ideal), while lines joined end-to-end represent resources that are being loaded in tandem, one after the other (not ideal).

The resource list below it shows the resources that were loaded. You see that all kinds of files are needed: documents, stylesheets, scripts, image files (svg, gif, jpeg, png, webp), icons (.ico), and more. The vertical blue line represents the point where the web browser has loaded and processed the main document (first item in the resource list). The vertical red line represents the point where the webpage is considered to be loaded (able to display its main content), while secondary resources are still loading in the background.

Why can’t they all be loaded simultaneously to save time? Sit down and I’ll tell you a story …

Timeline of a loading webpage

In the beginning was The Void. And then:

laymansguide loads (0–673 ms), because a request was sent for it when I asked my web browser to load https://github.com/ngjunsiang/laymansguide. This is the first resource in the list.
laymansguide is a webpage, an HTML (HyperText Markup Language) file, which I will explain in a future issue. Inside, it contains lines like these, isolated here for your reading pleasure:

<link rel="stylesheet" href="https://github.githubassets.com/assets/frameworks-2e9090135c22aad5f56c2f72dcba7880.css" />
<link rel="stylesheet" href="https://github.githubassets.com/assets/github-cbb49d8cd46cbc8c522a95d52b21ab53.css" />
<img src="https://github.githubassets.com/images/search-key-slash.svg" alt="" class="mr-2 header-search-key-slash">
<include-fragment class="my-2"><img alt="Loading" src="https://github.githubassets.com/images/spinners/octocat-spinner-32.gif" width="32" height="32" /></include-fragment>
<img class="avatar" src="https://avatars3.githubusercontent.com/u/45561895?s=40&amp;v=4">
[...]

Lines like these in the document are interpreted by the web browser. They tell the browser that more resources are needed, and where they need to be loaded from (either through the "href=" addresses or the "src=" addresses). The web browser then sends more requests for these files. That is why you only see them loading after the document has loaded: my web browser has to receive and process the document first to know where to retrieve these files.

The stylesheets (files that end in .css) tell the document how to style elements in the page. I won’t elaborate on that in detail here, perhaps in a different season of laymansguide.

The images are … well, images. More on them in a future season about data types.

The scripts, ah, that’s something to go into. While stylesheets and images are just information to be inserted into the page, scripts are actually code, usually Javascript code.

This code does animations, calculation of time conversions, and many other things, including loading more resources. I’m not going to paste the whole script here, I don’t want to chase my readers away … okay, maybe just a couple of lines. The last script file that is loaded, github-bootstrap-747cdfeb.js, is a companion script file for the Bootstrap¹ framework that Github uses to simplify their webpage code. It has the following lines:

[...]
const e = function(e) {const t = document.createElement("img");
                       return t.className = "emoji"}(this);
e.src = this.getAttribute("fallback-src") || "",this.appendChild(e)
[...]

This creates an img element named e, styles it according to the style class emoji, then associates it with the address retrieved from fallback-src, and finally it gets appended to another element in the page through the function appendChild. That’s right, this code inserts emojis, so the laymansguide document has to retrieve those emoji image files first. Before this script plays, the page won’t even know that those emojis are necessary.

And now we come to the last part of the waterfall, where additional resources requested by scripts and other things are loaded. It’s not worth going through them in detail, we won’t learn much more from that. So let’s summarise.

Issue summary: When a webpage document loads (Stage 1), it is processed by the web browser, which then loads other requested resources, such as stylesheets, images, and scripts (Stage 2). Scripts and other interactive code may then request more resources (data fetches, images, icons, data, etc) which are then loaded subsequently (Stage 3, 4, 5, …).

Bonus content: I tried this with the Baidu homepage, which looks like this:

Baidu homepage

and the network activity from loading it:

Developer Tools, Network view

It takes slightly longer (about 400 ms longer) to load, but most of that time is from loading a single gif. I wont examine the elements here, in the season on the internet cloud, I’ll explain more what some of these elements do.

So far, we have identified 3 sources of latency:

DNS resolving (Issue 29)), which translates the domain names in the requests to IP addresses that we can request data from,
Data packet routing (Issue 37)), which adds to latency with each hop through yet another gateway,
Webpage loading (this issue), where documents or scripts that are loaded may request yet more resources.

All of these layers of information gathering can add up to a few seconds of latency—a big turnoff for folks who have come to expect near-instantaneous response from apps. And often, our pages don’t appear to take that long to load, do they?

In the next issue, the season finale, I’ll explain one common trick for making anything appear faster: it is called caching.

What I’ll be covering next

Next issue: What are caches? What is caching?

Sometime in the future: What is:

booting up? [Issue 15]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
those ‘\r\n’s in the HTTP request packet [Issue 12,17]?
a good reason developers write code and give it away for free online? [Issue 21]
ASCII? [Issue 23]
compiling code into an application [Issue 26]?
firmware? [Issue 34]
What is HTML [Issue 38]

Bootstrap is a front-end) framework) that makes it easy to create webpages. By loading a standard Bootstrap stylesheet and (optionally) a Bootstrap script, any front-end developer can add common elements (e.g. popovers, navigation bars, tooltips, cards, …) with fewer lines of code than if they wrote it from scratch. ↩

Issue 37: Traceroute–Google Maps for data packets

2019-09-07T08:00:00+08:00

Previously: Latency is the time duration between a ping packet being sent out and its response being received. It is an indication of how far away a target server is.

Last issue, we saw that the latency for google.com’s servers was almost 8× shorter than that for baidu.com’s servers. Let’s see why.

traceroute

I use the commandline application (Issue 15)) traceroute to display the entire route taken by the data packet. The commandline prompt in Linux usually starts with a $, so any text you see after a beginning $ in the same line is the command I am using. Everything that follows is the output from the traceroute application. the -T option tells traceroute to use TCP packets to trace the path of our data packets.

$ traceroute -T google.com
traceroute to google.com (74.125.24.138), 30 hops max, 60 byte packets
 1  _gateway (192.168.1.1)  0.971 ms  0.959 ms  0.951 ms
 2  1.128.104.27.unknown.m1.com.sg (27.104.128.1)  4.577 ms  4.579 ms  4.572 ms
 3  43.245.104.65 (43.245.104.65)  3.868 ms  3.865 ms  3.854 ms
 4  5.246.65.202.unknown.m1.com.sg (202.65.246.5)  6.396 ms  3.829 ms  3.799 ms
 5  37.246.65.202.unknown.m1.com.sg (202.65.246.37)  18.766 ms  18.761 ms  5.129 ms
 6  134.246.65.202.unknown.m1.com.sg (202.65.246.134)  8.222 ms 221.246.65.202.unknown.m1.com.sg (202.65.246.221)  3.928 ms  3.901 ms
 7  72.14.222.102 (72.14.222.102)  3.882 ms  3.705 ms  3.661 ms
 8  209.85.243.27 (209.85.243.27)  3.621 ms 74.125.252.107 (74.125.252.107)  4.638 ms 72.14.238.42 (72.14.238.42)  4.621 ms
 9  108.170.254.227 (108.170.254.227)  4.180 ms 108.170.240.241 (108.170.240.241)  4.071 ms 108.170.240.242 (108.170.240.242)  4.059 ms
10  72.14.236.242 (72.14.236.242)  4.882 ms 216.239.57.50 (216.239.57.50)  4.771 ms 216.239.50.192 (216.239.50.192)  5.476 ms
11  72.14.233.43 (72.14.233.43)  9.760 ms 72.14.233.27 (72.14.233.27)  4.700 ms  4.665 ms
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * 74.125.24.138 (74.125.24.138)  4.572 ms *

Wall of numbers and text! Whoa, scary! It’s all right, I’ll throw that output into Google Sheets and prettify it a bit:

The traceroute output for google.com, prettified

traceroute sends 3 TCP packets, tracing the path they take to the destination IP address. Each time the data packet gets forwarded to another gateway, it is considered a ‘hop’.

traceroute also helps us do a reverse DNS lookup; a DNS query is when you want to find out the IP address associated with a hostname, a reverse DNS lookup helps you find out which hostname is associated with an IP address. IP addresses with successful reverse DNS lookups have the hostname shown for easier reading; those that didn’t will have only the IP address shown. Some servers are configured to block traceroute packets (done by the firewall)), and thus return no information; these are represented by asterisks (*) in the output.

As the data packets are sent from my router (_gateway in the output) to the first ISP gateway (27.104.128.1), they don’t always take the same path to the destination. Some of the internet servers along the path are programmed with algorithms that will send the data packets to a group of servers. This group of servers are configured to work together to share the packet-routing load. Two data packets sent through the same internet server may end up getting routed to different places along the route.

Notice that each hop has a latency associated with it. This is the time taken for the server to decode the packet, figure out what is the next destination, and send it forward. This will often not happen immediately. The data packet joins a queue of other data packets waiting to be dispatched; when the server is under a heavy load, with many data packets waiting to be dispatched (perhaps a whole deluge of Google searches are happening?), the waiting time can rise to hundreds of milliseconds or even a few seconds! (Simply unbearable …)

Let’s look the traceroute output for baidu.com:

The traceroute output for baidu.com, prettified

Notice that:

More of the servers along the way are traceroute-friendly
The data packets take fewer hops to reach baidu.com …
But they take longer, because some of the servers along the way have really high latency (almost 300 milliseconds; that’s slower than human reaction time)
My ISP is M1, but sometimes the packets can go through other ISP servers as well.¹ Hop 7 goes through Starhub’s internet servers.

And that, in two images, is why Baidu’s latency is so much higher: the latency for some of their servers is much higher.

Issue summary: The process of forwarding data packets from server to server takes time. Each hop a data packet takes adds to the latency. The more hops a packet must undergo, the longer the latency. The slower the servers along the route, the longer the latency as well.

I didn’t have much writing space (or readers’ mind space, I wager) to show what the entire route of a data packet looks like when I was describing how the Internet (and packet routing) works. I hope this big-picture post puts everything into clearer perspective now :)

We’ve looked at how data packets weave their way through the Internet, and next issue I want to shed some light on when/how data packets are sent.

What I’ll be covering next

Next issue: Loading a web page

Sometime in the future: What is:

booting up? [Issue 15]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
those ‘\r\n’s in the HTTP request packet [Issue 12,17]?
a good reason developers write code and give it away for free online? [Issue 21]
ASCII? [Issue 23]
compiling code into an application [Issue 26]?
firmware? [Issue 34]

Remember that internet servers are supposed to cooperate with each other and forward data packets properly according to Internet Protocol (Issue 27)). That is how the Internet can continue running smoothly. If servers stop obeying this protocol, the Internet will no longer be a connected space, but will become segregated depending on which servers are willing to forward data packets to each other.
For instance, if China blocks some web services (Facebook, Twitter, et al), then packets from those services cannot be routed through China’s servers. ↩

Issue 36: Latency

2019-08-31T08:00:00+08:00

Previously: VPNs link devices that are not within the same network using an encrypted tunnel that prevents gateways from snooping on the data packet as it passes en-route.

In the past two issues, I looked at the basics of web filtering: how data packets are detected (by snoopers or gateways), and how they are protected (by encryption).

In the next few issues, I’ll look at speed: what causes internet speed to be slow or fast. This issue, we look at latency.

Latency

In business, the idea of “turnover time” refers to the amount of time taken to complete a process or fulfill a request. With so many businesses relying on the Internet these days, latency is a big part of this turnover time. Simply put, latency is the time duration between a device sending out a request and the same device receiving a response.

You can see how a long latency will result in long waiting time in an app; if the app has to complete many requests to obtain all the data it needs, it can take a long while before the app is able to even start processing the data, let alone displaying it.

Internet latency: an illustration

ping (n.)

1835, imitative of the sound of a bullet striking something sharply. Meaning “short, high-pitched electronic pulse” is attested from 1943. As a verb from 1855; in computer sense is from at least 1981. Related: Pinged; pinging.

In WWII, a ping is a sound pulse emitted by submarines to detect other submarines and undersea objects. The sound pulse bounces off hard surfaces, such as the ocean floor, or other submarines, and returns to the submarine as an echo, giving it an audio report of what is around it. The longer the time gap between the ping and its echo, the further away the object is. It is called a ping because, well, it sounds like a ping.

An internet ping does not sound like a ping, but it serves a different function through a similar idea. By sending out a data packet to a target server, and measuring the time it takes to get a response, it can “gauge the distance” that the device is away from the target server.

On any laptop, you can measure the latency of any server using a commandline application (Issue 15)) called ping. Here, I will show you the output from ping while it pings two different servers.

The commandline prompt in Linux usually starts with a $, so any text you see after a beginning $ in the same line is the command I am using. Everything that follows is the output from the ping application. the -c 20 option tells ping to measure the latency statistics from sending 20 packets of data to the specified server.

Compare this:

$ ping -c 20 google.com
PING google.com (74.125.24.100) 56(84) bytes of data.
64 bytes from 74.125.24.100 (74.125.24.100): icmp_seq=1 ttl=46 time=19.5 ms
64 bytes from 74.125.24.100 (74.125.24.100): icmp_seq=2 ttl=46 time=37.9 ms
64 bytes from 74.125.24.100 (74.125.24.100): icmp_seq=3 ttl=46 time=40.0 ms
[… TRUNCATED …]
--- google.com ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 48ms
rtt min/avg/max/mdev = 19.484/33.354/50.181/7.413 ms

with this:

$ ping -c 20 baidu.com
PING baidu.com (39.156.69.79) 56(84) bytes of data.
64 bytes from 39.156.69.79 (39.156.69.79): icmp_seq=1 ttl=41 time=329 ms
64 bytes from 39.156.69.79 (39.156.69.79): icmp_seq=2 ttl=41 time=355 ms
64 bytes from 39.156.69.79 (39.156.69.79): icmp_seq=3 ttl=41 time=276 ms
[… TRUNCATED …]

--- baidu.com ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 24ms
rtt min/avg/max/mdev = 275.903/340.161/391.440/30.893 ms

What we are interested in is the last line. Notice that the average time between a ping packet to google.com and its response packet is 50 milliseconds, while that for baidu.com is 391 milliseconds. That is almost an 8× difference!

What accounts for this difference? Stay tuned for the next issue of Layman’s Guide to Computing! ;)

Issue summary: Latency is the time duration between a ping packet being sent out and its response being received. It is an indication of how far away a target server is.

I promised one idea a week, and I am sticking to that promise. Remember how the Internet works: packets being forwarded from gateway to gateway. You can probably guess what might account for the latency difference from this, but I am still saving the juicy details for next issue.

What I’ll be covering next

Next issue: Traceroute – Google Maps for data packets

Sometime in the future: What is:

booting up? [Issue 15]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
those ‘\r\n’s in the HTTP request packet [Issue 12,17]?
a good reason developers write code and give it away for free online? [Issue 21]
ASCII? [Issue 23]
compiling code into an application [Issue 26]?
firmware? [Issue 34]

Issue 35: Virtual Private Networks (VPNs)

2019-08-24T08:00:00+08:00

Previously: Firewalls block data packets that match certain rules. They decrypt the data packet layer by layer, dropping those that match its programmed rules without allowing them to be forwarded to the next point in its journey. The type of filtering that can be applied depends on the processing power available to the router, since some information is hidden more deeply in the data packet than others. Such filtering is typically circumvented by the use of VPNs, or other means of encrypting the data that is required.

Thus far, our picture of networks divides them into public networks (consisting of devices with public IP addresses) and private networks (devices with private IP addresses). Data traffic between devices in the same private network goes through their common gateway without getting forwarded to the rest of the internet. This data traffic cannot be snooped by outsiders who are not in that network. Theoretically, data within this private network is secure; you can do network printing or share files with other computers within your private home network and know that nobody outside of that network will inadvertently receive a leaked data packet.

What if you’re out but desperately need to access a file on your home shared network? Or you’re out of the office but need to send a print job to the office printer? (I leave it to you to imagine more plausible scenarios; I’m just trying to explain when you might want a VPN :P)

You would want some way for your computer to be able to access that private home/office network, but you can’t do it through a public gateway: that would compromise data security, since packets travelling through the Internet from the private network to your computer can be intercepted by other devices along the way.

You need a …

Virtual Private Network (VPN)

A Virtual Private Network (VPN) is a way for devices that are not in the same private network to behave as though they are. This network consists of a VPN server, and one or more VPN clients. The VPN server acts as a DHCP server for the clients, assigning them a private IP address for the VPN. This allows devices within the VPN to communicate with each other, regardless of location.

How is data prevented from leaking outside of the VPN then? By encryption (which we won’t go into the technical details of in this issue). Data travelling between VPN clients is encrypted before being sent out, and decrypted when it arrives. Anyone snooping on the packet contents as it travels gateway to gateway will just see encrypted jumble. That means firewalls won’t be able to identify the information it needs to block data packets, and the data packets thus usually get through unmolested.

Issue summary: VPNs link devices that are not within the same network, such that they can behave as though they are. By encrypting the packet data before it is sent between devices, the VPN software hides these packets from being snooped (i.e. spied upon), effectively forming an encrypted tunnel for information to travel between devices. This enables devices to circumvent firewalls and protect the privacy of information in the data packets.

Last issue was firewalls, this issue is VPNs. You’ll notice that the more advanced the technology, the shorter my posts. That’s because I have already laid out most of the framework for understanding the Internet infrastructure in the issues leading up to these two.

That’s part of the beauty of the Internet: understanding its basic pieces gives you a pretty good picture that helps you understand most of what’s happening in Internet-related news today. But that’s also the difficulty: finding all the information put together in a coherent, easy-to-read way that actually lays out the picture helpfully. I hope this newsletter has largely succeeded in doing that. If it has not, please let me know where it fell short, and how I might improve it :)

What I’ll be covering next

Next issue: Latency

Sometime in the future: What is:

booting up? [Issue 15]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
those ‘\r\n’s in the HTTP request packet [Issue 12,17]?
a good reason developers write code and give it away for free online? [Issue 21]
ASCII? [Issue 23]
compiling code into an application [Issue 26]?
firmware? [Issue 34]

Issue 34: Firewalls

2019-08-17T08:00:00+08:00

Previously: Data is encapsulated when it goes out from an app onto the internet as a request or response. First, in the TCP layer, the OS tags the request with the pre-assigned port number so that it can forward the response to the correct app later. Next, in the IP layer, the network card adds source and destination IP address information so that the data packet arrives at the correct destination server, and the destination server can send the response back to the correct computer. As the packet goes through the router, the router replaces the (source) private IP address assigned to the device by the router with the public IP address assigned to the router by the ISP.

If the above summary of Issues 30–33 makes sense to you, you are probably ready to proceed :) If not, you might want to review those issues quickly through the links below:

Issue 30): Private IP addresses
Issue 32): Sharing a public IP address: Network Address Traversal
Issue 33): Port numbers
Read the rest of the archives here.

Okay, now we understand how the open, unfettered internet of the 1990s worked. But we are still nowhere near understand the modern Internet until we understand how security is applied. Let’s look at one of the earliest features first.

Firewall: a wall to prevent the spread of fire

A software firewall performs an equivalent function: to prevent the transmission of an identified packet through it.

Of course, that means it must be able to be configured with rules that enable it to check the data packets to see if they should be allowed through. What kind of rules can be written for a firewall? That would depend entirely on what information is available in the data packet.

IP and port number filtering

Thus far, we know that the outermost layer of the data packet is the IP layer, which contains IP address information. A firewall can simply block out data packets coming from a certain source IP address, or being sent to a certain destination IP address. This lets a router administrator block certain devices on the network, by blocking data packets with source IP address matching the private IP address of the device. It also allows an ISP to filter out certain servers from receiving data packets from its customers, by blocking data packets with destination IP address matching the public IP address of the server.

Do this with a sufficiently comprehensive list of IP addresses, and you can completely block out an entire region’s or even country’s servers. And it would take a VPN to circumvent this, if the VPN server’s IP address is not in the firewall’s block list.

At the next level, the TCP layer contains port information. A firewall can block out all web browser traffic by restricting data packets with port number 80, or prevent the use of HTTPS by block port 443. It could also theoretically restrict WhatsApp messages, or Apple Messages, just about anything with a well-known or registered port number. This involves more processing power, since the router would have to process the IP layer first to get to the TCP layer.

Hostname filtering

At higher levels of filtering (involving yet more processing power), the router might even block certain hostnames. Remember that the public IP address of the server is needed before a request can be sent, and this IP address is obtained through a DNS query (Issue 29)). If a firewall intercepts and processes the DNS query and blocks the DNS query to resolve that hostname, the device never obtains the public IP address of the server and won’t be able to send the request.

Advanced filtering

Advanced routers might also come with built-in patterns that detect different types of traffic, e.g. streaming video, videoconferencing, bittorrent, and more. Anything with a predictable pattern can be blocked by a firewall that the data packet passes through, some requiring more processing power than others.

Issue summary: Firewalls block data packets that match certain rules. They decrypt the data packet layer by layer, dropping those that match its programmed rules without allowing them to be forwarded to the next point in its journey. The type of filtering that can be applied depends on the processing power available to the router, since some information is hidden more deeply in the data packet than others. Such filtering is typically circumvented by the use of VPNs, or other means of encrypting the data that is required.

Short issue, entirely by design. Firewall blocking and circumvention methods are an arms race between two sides, one side building better software and hardware that detects packets more quickly and accurately (requiring massive computational power), and the other side seeking ways to hide such metadata or enable it to get to its destination via other paths, often with the use of encryption (also requiring computational power). The details of how each method works can fill tomes. So I’ve decided to just introduce the basic idea, which is really quite simple: detect and block.

Today, most basic routers have ways to block data packets using hardware (MAC) address, IP address and port number information. Hostname blocking may be enabled on more advanced (i.e. expensive) routers, or on custom router firmware (I’ll unpack this many issues in the future). You can use this to block ads from certain providers, to restrict access to certain sites (whether to protect your kids or to improve your productivity, I won’t guess), or even to only allow known devices to use your network (in case your neighbour is really good at hacking wifi networks).

I mentioned VPNs a lot, and didn’t elaborate on that. That’s because I’ll be covering that in the next issue :)

What I’ll be covering next

Next issue: VPNs: Virtual Private Networks

Sometime in the future: What is:

booting up? [Issue 15]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
those ‘\r\n’s in the HTTP request packet [Issue 12,17]?
a good reason developers write code and give it away for free online? [Issue 21]
ASCII? [Issue 23]
compiling code into an application [Issue 26]?
firmware? [Issue 34]

Issue 33: Port numbers

2019-08-10T08:00:00+08:00

Previously: The devices on your home network share the single ISP-assigned IP address through your router. The router rewrites the source IP and port number on outgoing data packets, and rewrites the destination IP and port number on incoming data packets, acting as a middleman for your devices so that they can access the Internet.

In Issue 32, when I was explaining Network Address Traversal (NAT), I mentioned this:

Another of those things is that TCP will give the request a port number. Yes, you might have heard this term when configuring routers. We won’t need to discuss port numbers yet, so let’s put that aside.

And then I went on the explain how the source and destination IP addresses are tagged on through the Internet Protocol (IP).

Well, now is the time to talk about what’s going on through the Transmission Control Protocol (TCP). Within a single device, you often have different software services running:

DNS querying (Issue 29))
Internet time synchronisation (over the Network Time Protocol)
Software updaters checking for updates online
Skype/Zoom/videoconferencing
Instant messaging
Web browsers
Others …

When a data packet comes in through the network chip and arrives at the operating system (OS), how does it know which software app to forward the data packet to?

Port numbers

When a software app needs to send a request or some data through the Internet, it has to go through the OS. The OS decides which software gets to use the Internet, and which ones don’t (that is what your firewall is for). If a software app has permission to access the internet, it requests a port number from the OS. The OS assigns the app a port number, and stores information about which application is mapped to which port number.

When the software app sends data to the OS to be sent out via the network, the OS adds identifying information, including the source and destination port number, as part of TCP. Then the OS sends the resulting data packet to the network card to be forwarded to the gateway (via your LAN cable or wifi connection). The network card then adds the source and destination IP address information (and other identifying information) as part of IP, creating a larger data packet (which at this point is app data inside a TCP shell inside an IP shell … like a Matryoshka doll).

Request #4676 to host
Source IP:        *27.104.229.65*
Source Port:      *45784*
Destination IP:   172.217.26.78
Destination Port: 80

When response data is returned, the OS checks the destination port number, looks up its list of assigned ports, and forwards the data to the correct app. Cycle complete!

Request #4676 from host
Source IP:        172.217.26.78
Source Port:      54674 (random port from host)
Destination IP:   27.104.229.65
Destination Port: 45784

Just like there is a source and destination IP address, there is also a source and destination port number. The source port number is the assigned port number from the OS. The destination port number, well … how are we to know that?

Reserved port numbers

With so many types of data being transferred over the Internet, how do things not get mixed up?

Reserved port numbers is how. The well-known internet traffic data types have standardised protocols: web traffic follows HyperText Transfer Protocol (HTTP - Issue 7)), some types of file transfer occur in accordance with the File Transfer Protocol (FTP), DNS queries obey the DNS protocol, etc.

These well-known protocols are assigned standard port numbers by IANA, the same authority that manages known IP addresses. HTTP uses port 80. FTP uses port 20 (data transfer) and 21 (control). DNS uses port 53. You can find the full list of well-known port numbers on Wikipedia. Port numbers 1 to 1023 are reserved for this purpose and may not be used publicly for other purposes.

If your app is trying to query a hostname to get the associated IP address, the destination port number is 53. The app simply uses the standard port number depending on what service it is trying to use, and which protocol it is following. If your app is trying to request data from a web server, the destination port number is 80; your web browser actually requests https://google.com:80 but it will leave out the port number if it is a standard port number. You can also try requesting https://google.com:8000 but you won’t get any data in return since nothing is listening on that port number at Google’s end.

Web servers will (by default) listen on port 80, FTP servers on port 21, DNS servers on port 53, and so on. Any traffic arriving with those port numbers will be directed to these (software) servers for processing.

Registered port numbers

Port numbers 1024 to 49151 are available for organisations, corporations, and other corporate bodies to request if they are setting up widespread area services. Whatsapp uses ports 4244, 5222, 5223, 5228, 5242. Apple uses port 5223 for push notifications. Google Play uses port 5228 for messaging. This makes it possible for these services to work without knowing what the port numbers of the software apps on the receiving end are mapped to.

Private port numbers

Port numbers 49152 to 65535 are available for anyone to use. Tinkerers, hackers, anyone who just needs a number to send data from.

… and that’s a wrap. Source and destination IP addresses get data to the right computers, source and destination port numbers get data to the right apps in those computers.

Issue summary: When an app makes a network request through the OS, the OS adds the source and destination port number to the query in accordance with TCP. When the OS receives the response, it forwards the data to the app which is mapped to the destination port number. Port numbers 1-1023 are registered to standard Internet services, port numbers 1024 to 49151 may be registered to other services, and port numbers 49152 to 65535 may be used by anyone.

Short and sweet! Issues 30-33 are my summary of how data packets find their way across the internet (via IP addresses) and to the right apps (via port numbers). It’s amazing that this is basically how almost all Internet traffic gets routed; the rules are simple and hence very scalable, you don’t need massive computation to get packets to the right places (but you do need massive hardware to process lots of packets quickly).

In the next few issues, I’ll go into some security and performance considerations for internet traffic: firewalls and traffic blocking, VPNs, and internet speeds.

What I’ll be covering next

Next issue: Firewalls and traffic blocking

Sometime in the future: What is:

booting up? [Issue 15]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
those ‘\r\n’s in the HTTP request packet [Issue 12,17]?
a good reason developers write code and give it away for free online? [Issue 21]
ASCII? [Issue 23]
compiling code into an application [Issue 26]?

Issue 32: Sharing a public IP address: Network Address Traversal

2019-08-03T08:00:00+08:00

Previously: A router assigns IP addresses automatically using DHCP. It reserves any registered static IP addresses for devices identified by their MAC address, and assigns the remaining private IP addresses in the pool to any devices that request one. Each IP address has a lease period, after which the device must request an IP address again.

In Issue 30, I tried to answer the question “how do all my devices manage to share the one precious IP address assigned to me by my ISP?”, and in the process introduced two more acronyms: DHCP and NAT. I covered DHCP last issue, and today I’ll explain NAT. It is one of those technologies that work silently in the background, doing its own thing merrily until something bad happens.

Requests and Responses

Remember that when your phone/laptop/device first connects to the router, it doesn’t have a private IP Address yet, and it uses its MAC Address (a hardware ID key) to ask the router for one. Once it gets a private IP address, it can finally start to send and receive requests.

In Issue 8, I gave an example of the content of a response:

The response header from Hypothes.is

This is like the header of a letter that Hypothes.is writes to my web browser (with a company letterhead and all that), saying that it received my request (HTTP 200 OK), and the content of my request (not shown) follows after this header.

In Issue 30, I described an analogy for how packets are requested and received:

When someone sends a physical packet to you, they’ll write a home address and a name (To: ). The mailman doesn’t care that much about the name; they just send the packet to the home address. Once it reaches your home, someone in the house (the early riser, or the stay-home one) looks at the name and figures out where the packet should go. If the packet is addressed to “Peter”, it’s natural to assume that it’s meant for Peter-in-this-house and not Peter-5-blocks-down-the-street.

When I click a link or load an image, my web browser sends another request for the information. This request may include some data (e.g. if I’m sending my credit card details), or it may just consist of a header (e.g. if I’m going to a new URL). But how do I send it out?

When we write letters, we never drop them into the mailbox directly; without To or From address information, the mailman would have absolutely no idea what to do with our letter!

This is where Transmission Control Protocol (TCP) and Internet Protocol (IP) comes in.

Transmission Control Protocol: not going into it for now

It is not the time to delve into what TCP does and how it works, but it is part of the sequence of events that lead to requests being sent. For now, just know that when my web browser sends the request, it has to go through my network card which will encode the request into an electromagnetic signal to be sent over the airwaves as a wifi signal. As it does so, a number of things happen.

One of those things is that TCP will stamp the request with some identifying information, so that when a response is received, it can be matched to the request.

Another of those things is that TCP will give the request a port number. Yes, you might have heard this term when configuring routers. We won’t need to discuss port numbers yet, so let’s put that aside.

Internet Protocol: To and From

As I type this section, it dawned on me that there is one acronym I didn’t introduce properly. All this while I’ve been using “IP address” repeatedly and never even mentioned what “IP” stood for!

Well, now we know: it stands for Internet Protocol (IP). IP is the next protocol layer in the sequence (that ensures our request gets through, and a response gets back). It is a big part of the backbone that forms the internet. At this point, our request has some identifying info and a port number, but no To or From information yet.

Where does the To: information come from? From the URL I am trying to access (or more specifically, the domain name of the URL). My web browser first sends a DNS query (Issue 28)) to resolve the domain name to an IP address, and now we have the IP address to put in the To: field. In Internet Protocol, this is the Destination address.

What about the From: information?

At this point, my network card only knows a few IP addresses:

Gateway: this is the IP address of the router, where all requests are sent
DNS server: this is the IP address where all DNS queries are sent
Private IP address: This is the private IP address the router gave to my device when it first connected to the wifi

The only IP address there that is viable is the private IP address. So that is the IP address to put in the From: field. In Internet Protocol. this is the Source address.

Sending and receiving

Now our letter is packaged in a TCP/IP envelope, with a port number, source address, and destination address. It is ready to go out from the network card! It is broadcast!

Request #4676
Source IP:        192.168.1.10
Source Port:      8976
Destination IP:   172.217.26.78
Destination Port: 80

My smart TV, Playstation, iPad, and other wifi-connected devices around the house are blind as to what it contains. They all receive this request, decode the address information, realise it is not for them (the Destination address is not their private IP), and promptly discard the request.

Only the router receives this request (because it is The Gateway Through Which All Things Pass), and has to figure out what to do with it. It sees that the Destination address is not in its forwarding tables, and knows it must send this packet to another gateway (i.e. your ISP). But that can’t happen yet: the Source address is a private IP address! Those can never go out to the ISP gateway, because the ISP gateway would get confused. It would think “private IP addresses are reserved for internal use and can only come from devices within my own network, so this packet must have come from another ISP computer and not my customers”.

So your router does a clever thing: it rewrites your request.

Network Address Traversal

The router replaces the Source address with its public IP address, the one issued by the ISP. At the same time, it also replaces the port number (e.g. 80 for web traffic) with another random one (e.g. 45784, some large number between 1024 and 65535).

Request #4676
Source IP:        *27.104.229.65*
Source Port:      *45784*
Destination IP:   172.217.26.78
Destination Port: 80

The server responds:

Request #4676
Source IP:        172.217.26.78
Source Port:      54674 (random)
Destination IP:   27.104.229.65
Destination Port: 45784

The router sees the destination port, 45784, and realises that the outgoing packet with the same port number originated from me. So it rewrites this response to send it back to me:

Request #4676
Source IP:        172.217.26.78
Source Port:      54674
Destination IP:   *192.168.1.10*
Destination Port: *8976*

And that’s how it’s done.

Issue summary: When a request from a device on the network is to be forwarded to the gateway, it has to traverse different networks. The router helps it by rewriting the source IP and port number, keeping track of the originating IP and port. When a response is received, it rewrites the destination IP and port so that the response will reach the originating device.

This issue is much, much lengthier than expected; I totally underestimated the prior-knowledge-bridging content that’s required. You notice that the NAT portion of this mailer is pretty short, but there’s a lengthy backstory to pull together disparate pieces of information from past issues.

You intuitively know that you need a router to access the internet, but might not know why. I hope this post illustrates sufficiently one of the key functions of a router :) the magic of NAT is what allows multiple devices to share one public IP address. Keep in mind that when you’re on public wifi, you may be sharing this digital infrastructure with hundreds if not thousands of other users! The hardware needed is immense; certainly far more expansive than the impression given by a home router.

As the picture gets clearer and more detailed it’s also getting more complex. My personal philosophy is “there are no dumb questions, only dumb answers”. I bet there’re tons of things I overlooked. If there’re any lingering questions, please drop me an email and I’ll try not to send a dumb answer :)

What I’ll be covering next

Next issue: All about port numbers

This next issue is fun! As you are receiving packets for your web browser, skype call, network file transfers, DNS queries, etc, how does your computer get the right packets to the right software services?

Sometime in the future: What is:

booting up? [Issue 15]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
those ‘\r\n’s in the HTTP request packet [Issue 12,17]?
a good reason developers write code and give it away for free online? [Issue 21]
ASCII? [Issue 23]
compiling code into an application [Issue 26]?

Issue 31: Getting a private IP address: DHCP (and DDNS)

2019-07-27T08:00:00+08:00

Previously: Private IP addresses are special IP addresses that routers will treat as belonging to devices within the private network, and not outside it. Data packets sent to private IP addresses will never make it past the gateway into the internet. This system allows multiple devices within a private network to share a public IP address.

Last week, I tried to answer the question “how do all my devices manage to share the one precious IP address assigned to me by my ISP?”, and in the process introduced two more acronyms: DHCP and NAT. I don’t intend to spam this newsletter with acronyms, but to use them as neat little terms that conveniently capture ideas about how different parts of the complete Internet experience works. So if you never remember what DHCP is, don’t fret over it; you won’t see it often unless you configure routers or servers often.

In a nutshell, DHCP is how you get a (private) IP address when you connect to your router. Before that, here’s one of those niggly little problems that the early Internet pioneers ran into in trying to write awesome services:

MAC address: Media Access Control address

When your phone/laptop/device first connects to the router, it needs to know what its private IP address is, so it can begin stamping data packets to be sent. It doesn’t have a private IP address yet, so it has to ask the router for one. But if it sends the IP address request, how would the router know who to reply to?

The answer is that each device (or more specifically, the networking chip of the device) has a unique hardware code, known as the MAC address. It will stamp the request with its MAC address first, so the router knows who to offer the IP address to.

The MAC address can also be used to do many other things, like block devices on a network; we’ll get to those other things in future issues ;)

DHCP: Dynamic Host Configuration Protocol

So what happens after that? your device asks for an IP address, and the router reserves one from its pool of private IP addresses and offers it to the device. the device accepts the request, and the router adds the device IP address to its forwarding table. Done.

It’s called a dynamic process because the router assigns private IP addresses only upon request, so a device may not always end up with the same IP address.

Static vs Dynamic IP addresses

One potential drawback of this dynamic process is that if you are trying to set up a network printer the Old Way, it will ask you to input the address of the printer. And unless you have figured out internal DNS on your router and properly hostnamed all your printers, you will be providing this address not as a hostname (like printer.home.com) but as an IP address (192.168.1.5).

An old printer management dialog window for adding printers. Source: UC Davis Coastal And Marine Science Institute

But when you reboot your network printer and it requests an IP address from the router again, will it get a different one? Will you have to re-add the printer again?

Not if you get the router to assign it as a static IP address. The router can store a list of MAC addresses and their associated static (private) IP addresses, so that the device will always get the same IP address.

DHCP and your ISP

One more thing you need to know about DHCP: when a device requests an IP address, it’s not going to get that IP address forever. IP addresses have a lease (5 days by default), after which the IP address will expire and the device will have to request a new IP address.

It’s the same thing with your ISP. Your home router, even if it is left on 24/7, will have to renew its (public) IP address with your ISP every few days, as and when it expires. So your ISP also (most likely) assigns you IP addresses by DHCP; you are not going to enjoy the privilege of a static IP address unless you pay gobs more money!

Wait: why are static IP addresses such a privilege?

Remember what a public IP address is? It’s a way for data packets to get to you wherever they are sent from. And that is possible because of the forwarding tables stored in Internet registries everywhere (including your ISP). All over the world, anyone who knows your IP address can send your router data.

But often they won’t do so. They will use a URL instead, because they are easier to remember. The sender finds out what IP address the domain name is assigned to by querying its DNS servers (Issue 28)).

If you want your own domain name, you have to buy one at a domain name provider. This lets you add your unique domain name to the WHOIS database (Issue 28)) … but you need an IP address to map to your domain name. And you can’t use your ISP-assigned IP address, because it’s dynamic and changes when the lease expires … can you?

Dynamic DNS

Some domain name providers offer a service known as Dynamic DNS (DDNS). This service lets you run software on your computer or a sufficiently advanced router that will update the provider whenever your public IP address has changed. So if you want to be able to stream videos from your own home server for your own viewing, without going through a cloud service, that is possible! But we won’t go into the details here ;)

Issue summary: DHCP is a protocol by which a router assigns IP addresses to devices that connect to it. Static IP addresses are IP addresses that are reserved for a device, so that the device always gets the same IP address when it connects.

I had almost forgotten about DDNS, and decided to include it at the last minute, making this a bit longer than I had aimed for. But I hope this fleshes out your model of the Internet a little more, so that you understand how it is possible for you to have your own domain name too.

I usually introduce new terms and acronyms in bold, but I didn’t bold DHCP here. You won’t ever need that acronym again and you will probably forget it if you don’t do router configuration, and that’s just how things should be.

What I’ll be covering next

Next issue: How do packets from my device get out onto the internet, and how do they come back to me? Network Address Traversal (NAT)

Sometime in the future: What is:

booting up? [Issue 15]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
those ‘\r\n’s in the HTTP request packet [Issue 12,17]?
a good reason developers write code and give it away for free online? [Issue 21]
ASCII? [Issue 23]
compiling code into an application [Issue 26]?

Issue 30: Private IP Addresses

2019-07-20T08:00:00+08:00

Previously: Your web browser resolves a hostname (finds out which IP address a hostname points to) by sending a DNS query to its gateway.

Back in Issue 27), I said this:

If you are good at math, you can work out the total number of available [IP address] combinations: it’s 4,294,967,296 (256^4). It seems like a lot, but we actually have more humans on this planet than we have IP addresses. And if we assign a unique IP address to each device on the Internet, many of us will need multiple IP addresses for our smartphone, laptop, Amazon Echo, smart TV, router, …

IP addresses are a limited resource! They are really precious!

IP addresses are so precious that your ISP only has a handful, and they will only give you one per subscriber line. Which of your devices are going to receive it?

Fortunately, there is a way for all of them to share this one IP address.

Private IP addresses

When you send a data packet through the internet, the data packet gets “stamped” with the destination IP address. The first router it reaches checks its internal forwarding tables (covered in Issue 27) to see if the IP address is there; if it is not, the router forwards it to the router’s gateway, and the process repeats.

Some IP addresses don’t get treated this way. Data packets sent to the following IP address ranges will never be forwarded past a gateway: - 10.0.0.0 to 10.255.255.255 - 172.16.0.0 to 172.31.255.255 - 192.168.0.0 to 192.168.255.255

In effect, packets sent to these addresses will never make it out beyond your router, onto the (public) internet. These addresses are known as private IP addresses.

Not too long ago, you configured your router through a webpage. That means you need to enter an IP address into your browser to access the router settings, and there’s no way for you to know that IP address in advance. Most routers are configured to have a default gateway IP address of 192.168.1.1 or 192.168.0.1 (check your router’s manual for the correct one), so you set that as your gateway address in network settings. the private IP address system prevents commands you send to your router from making their way out to the Internet, and potentially leaking your information.

Imagine how many devices at home are sending data packets to 192.168.0.1 (or 192.168.1.1), to be forwarded to their destinations through the gateway … private IP addresses allow private networks to be separated from the internet. Every device in the private network has an IP address that only the router has a complete list of. You can’t send a data packet to a private address (say, 192.168.1.3) on a different private network, since that data packet will never make it past the gateway. The router will simply attempt to send it to 192.68.1.3 within the same private network.

(A router is a device that forwards data packets it receives; a gateway is a destination point that a data packet gets sent to as it makes its way across the internet to its destination. I’ll try not to get too philosophical about the difference between them, but I’ll be using them interchangably depending on whether I’m emphasising the device or the network layout.)

Handling the complexity: DHCP and NAT

It all sounds terribly simple when described this way. And then somebody at the back of the room raises their hand and asks, “so when we want to send a packet to IP address 45.63.153.22, should I send it to 45.63.153.22 (destination) or 192.168.0.1 (gateway)?”

No spoilers, we’ll cover this a couple issues later. The router juggles between the various IP addresses through a process called Network Address Traversal (NAT). NAT goes hand-in-hand with IP addresses to enable this IP-sharing magic. But first it has to have a public IP address …

How do our devices get an IP address from the router? How does our router get an IP address from the ISP? This usually occurs according to a set of rules known as Dynamic Host Configuration Protocol (DHCP)—yep, it’s another protocol.

An analogy: home addresses and names

When someone sends a physical packet to you, they’ll write a home address and a name (To: ). The mailman doesn’t care that much about the name; they just send the packet to the home address. Once it reaches your home, someone in the house (the early riser, or the stay-home one) looks at the name and figures out where the packet should go. If the packet is addressed to “Peter”, it’s natural to assume that it’s meant for Peter-in-this-house and not Peter-5-blocks-down-the-street.

This system allows multiple people to share one address. That address is like your ISP-given public IP address. The names are like private IP addresses; they are only meaningful within the house, and ambiguous outside of that context.

Issue summary: Private IP addresses are special IP addresses that routers will treat as belonging to devices within the private network, and not outside it. Data packets sent to private IP addresses will never make it past the gateway into the internet. This system allows multiple devices within a private network to share a public IP address.

This was a little longer than I expected, but I’m pretty pleased with it. I hope you found the analogy helpful; it just popped to mind as I was nearing the end of the issue.

Private IP addresses were something I started learning when I started playing with my home router to try to optimise its performance (this was back in the slow days of 56k ADSL), and it took me weeks to understand all the different knobs and dials in the web interface. If you ever come across them, now you know what’s going on :)

What I’ll be covering next

Next issue: How do my devices get an IP address? Dynamic IP addresses and DHCP

Sometime in the future: What is:

booting up? [Issue 15]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
those ‘\r\n’s in the HTTP request packet [Issue 12,17]?
a good reason developers write code and give it away for free online? [Issue 21]
ASCII? [Issue 23]
compiling code into an application [Issue 26]?

Issue 29: How to resolve a hostname

2019-07-13T08:00:00+08:00

Previously: An IP address is a string of four numbers that looks like 255.255.128.1. IP addresses are a list managed by the IANA, and all Internet registries agree to route data packets through their servers according to that list, so that you can send your data to anyone whose IP address you know. If you don’t know their IP address, but you know their domain name, you can look up the domain name in a WHOIS database through a Domain Name Server to get their IP address.

A really old screenshot of Chrome, showing the resolving host message. Source: Webnots

I think it’s basically impossible to be living in the age of the Internet and not have encountered one of those absolutely annoying times when your web browser just seems to be stuck on “resolving host …” for what seems like forever. What is it doing? And what does resolving host mean?!

Let’s address the second part first:

What is a host?

In the early days of the internet, before blogging happened, if you wanted your own webpage you had to write the content and code yourself and put it up somewhere that people can access. the one place almost everybody in my generation knew could do this was Yahoo! GeoCities, a web hosting service. They are called hosting services because they act like a host to your content, serving them biscuits and water, making sure they know the wifi password to the place, and the way to the bathroom.

This is why the part of the domain name after the subdomain, i.e. facebook.com or google.com without the www in front, is also known as the hostname; on the internet, that is literally the name of the host(ing server) that holds your content.

What does resolving mean?

Merriam-Webster online:

resolve verb

re·solve | ri-ˈzälv

4 b : to find an answer to

To resolve a hostname is to seek an answer to the question “which IP address does this hostname point to?”

Your web browser, bless its poor soul, has to ask this question every time you type a URL into the address bar. It does not keep a list of domain names (and the addresses they map to), so someone else has to seek the answer for it.

This someone, or rather something, is a Domain Name Service.

DNS: Domain Name Service

Network connection properties, a window that one used to see very often when configuring a router. Source: Help Desk Geek

In the days when you had to set up your router through a LAN cable connected to your internet-less laptop, this was a screen you would see regularly. You had to do this because you had to set your gateway address to point to the router’s default IP address in order to access its internal admin page.

But I talked about gateways back in (Issue 27)), so we’re here to talk about DNS servers instead.

Just like other data request packets, the request to resolve a host (known as a DNS query) first goes to the gateway (usually your router). If it can’t be resolved there, it gets forwarded to the next gateway (your ISP), and it keeps getting forwarded until it reaches a server that is able to answer that query and return the long-desired IP Address (hey, a few seconds is already lifetime for an HTTP session!)

Usually, this request would be resolved by your ISP’s DNS servers. But sometimes the ISP gets a little swamped, or their DNS server decided to call in sick, or didn’t update its WHOIS databases properly and gave you a wrong answer, or … anyway, if you wanted to bypass that DNS server and try another one, you could input its IP address in the Preferred DNS Server field above.

Short question for you, dear reader: Why does the Preferred DNS Server field require an IP address and not a hostname?

Advice on the Internet these days suggests that if you find your browser is taking a little too long to resolve a hostname, you can try configuring your network interface to use Google Public DNS instead. Since this is a newsletter and not a helpdesk, I won’t go into the details here but you can always drop me a message if you need help.

Issue summary: Resolving a hostname means answering the question “which IP address does this hostname point to?”. Your web browser seeks this answer by sending a DNS query to the gateway. If the gateway is unable to provide a satisfactory answer, you can configure your network interface to send the DNS query to a different DNS server.

This a is a nice, sweet, and short issue, though it took almost as long to write as Issue 28. Crazy, I know.

I have more to say about DNS, but I don’t think it belongs in this issue. I’ll figure out where to slot that info, whenever it becomes more relevant :)

I hope a picture of how your browser gets its data is gradually forming in your head: registries looking up databases of addresses, requests getting forwarded to higher and higher gateways, and now hostnames resolving into IP addresses.

Yeah I really should have introduced hostnames last issue, but couldn’t find a place to put them without making the issue more unwieldy than it needs to be. I think it made perfect sense here.

Did you answer the question above? It’s meant to make you think a little; that helps a lot for remembering what you read. You have to input the DNS Server as an IP address because if you used a hostname, your laptop would have to resolve that hostname before knowing where to send the DNS query. For this reason, Google Public DNS makes its DNS Server IP addresses easy to remember: 8.8.8.8 and 8.8.4.4.

What I’ll be covering next

Next issue: Private IP addresses – how do I use one IP address for all my devices?

Sometime in the future: What is:

booting up? [Issue 15]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
those ‘\r\n’s in the HTTP request packet [Issue 12,17]?
a good reason developers write code and give it away for free online? [Issue 21]
ASCII? [Issue 23]
compiling code into an application [Issue 26]?

Issue 28: Domain Names and DNS

2019-07-06T08:00:00+08:00

Last issue, I introduced IP addresses: a string of four numbers that tells routers where to send the data packet. These IP addresses are managed by the Internet Assigned Numbers Authority (IANA), and Internet registries everywhere agree to configure their routers to abide by this agreement. Without this coordination, our data won’t be able to make it across the Internet so easily.

Domain names

When was the last time you remembered entering an IP address into the address bar in your browser? (If you’re a tinkerer, maybe the last time you tried to configure your router through a web browser.) Almost all the time, we enter web addresses that use domain names, not IP addresses. Here are some domain names you probably know: www.facebook.com, www.google.com, www.instagram.com.

The https part in front of addresses like https://www.facebook.com is not part of the domain name; it is the protocol: a web browser would use that to determine which set of rules to use in processing the data received. I first introduced the idea of a protocol in Issue 7).

Here’s something that might be new to some of you: www.facebook.com is still not the domain name. facebook.com is the domain name. That’s how it was conceptualised in the 1980s, where people already realised that memorising names is much easier than memorising IP addresses. The original specifications for the Domain Name System were published in 1983, making this system less than 40 years old.

The right to use a domain name is governed by domain name registrars (basically any service you see selling and managing domain names), which have to be accredited by the Internet Corporation for Assigned Names and Numbers (ICANN).

Subdomains

Some domains are a huge place. Just look at the number of services Google offers. Each of these services might be residing on a different server, under a different IP address. It would be mightily inconvenient if each domain name could only be mapped to a single IP address: we would have to use multiple domain names to access them, perhaps gmail.com, googledrive.com, googleslides.com, and so on.

Subdomains are a way for a router to consolidate multiple locations under a single domain name. A data packet with the address www.google.com would be routed to the www server, drive.google.com would be routed to the drive server, mail.google.com would be routed to the mail server … and Google can strengthen its identity under this single domain name, and save on registering additional domain names.

Subdomains make it easier for a business or organisation to organise their web presence and identity. Once you have registered myorganisation.com as your domain name, you can configure your hosting server (the server that holds your webpages and files) with multiple subdomains (up to 100 per domain).

… and Top-level domains

What about international organisations? Some of them may have different branch offices, with webpages hosted on different servers with different IP addresses. Subdomains are a clunky way to deal with this: what if each office needed those subdomains to further divide sections of their website? You would have sgsales.acmecorp.com, sgsupport.acmecorp.com, sgadmin.acmecorp.com, ussales.acmecorp.com, … what a mess! Reserving a part of the full domain just for countries seemed like an obvious idea at the time, and so it happened.

The Domain Name System originally specified two main groups of top-level domains (TLD):

The country code top-level domain, based on two-character territory codes. Example: .sg, .my, .us, .ru
The seven generic top-level domains – .gov, .edu, .com, .mil, .org, .net, .int

A web address must have a generic top-level domain describing its category, and optionally may also have a country code top-level domain to signify its associated territory. This helps people to identify the category/purpose and origin of the website.

Each top-level domain is managed by an administrative registry which takes care of the database of names under that TLD. Registration requests from the domain name registrars (mentioned above under domain names) are forwarded to the respective registries. This enables some TLDs to be protected. For instance, .edu domains may only be registered by accredited organisations on a the U.S. Department of Education’s list (Wikipedia link).

WHOIS

What do these organisations do with their huge list of domain names? They put them in databases, of course. If you want to find out who registered mydomainname.net, you would send a request to a WHOIS server (that is not an acronym, it just means “who is”!) and it would give you the registrant’s information. Here, give it a try. Just key in any domain name you know. I’ll wait.

When you register a domain name with a domain name registrar, you are putting your name on the WHOIS database, so that people know who the domain belongs to when they query the WHOIS database. The domain name registrar lets you configure the domain name to associate it with your IP address. This information then propagates through the registrar’s Domain Name System (DNS) servers, so that within a day or two, data packets will start to arrive at your server.

How does that happen? That’s a story for the next issue.

Issue summary: Domain names consist of an optional subdomain, the domain name, and the top-level domain. The top-level domains are managed by a registry, which receives registration requests from domain name registrars, and maintains registrant information for each domain under their TLD in a WHOIS database. The domain name registrars let you configure which IP address to forward data packets to, and propagate that information through their DNS servers so that data packets will be routed accordingly.

This was wayyy longer than I thought it would be—there are a lot of things I needed to look up and confirm. That’s part of the reason I started this newsletter: to help me clarify what I know and ensure it’s not just “something I heard”. Domain names, something that looks so simple, has a really huge backend system maintaining its existence!

If you really try to imagine how all this happens, one thing starts to become clear: the backbone of the Internet is lists and lists and lists and lists … and the hardware that looks up these lists for you are really carrying a heavy load! There are numerous organisations responsible for ensuring that the Internet keeps on running. The day they are no longer sustainable, the Internet is going to be reconfigured into something a lot less interconnected.

I’m sorry if this issue was a little too heavy on the terms. I bolded the terms that I’m likely to use again, if it helps to focus your attention. I’ll reintroduce the terms again if I use them in future issues.

What I’ll be covering next

Next issue: DNS lookup and resolving (how do I know which IP address to send this packet to?)

Sometime in the future: What is:

booting up? [Issue 15]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
those ‘\r\n’s in the HTTP request packet [Issue 12,17]?
a good reason developers write code and give it away for free online? [Issue 21]
ASCII? [Issue 23]
compiling code into an application [Issue 26]?

Issue 27: What is an IP address?

2019-06-29T08:00:00+08:00

In Season 2, I took a detour to introduce some cool things that developers typically work with, and ideas they implement to make their work as smooth as possible.

And now we’re back on track, where I last stopped in Season 1. There, I did a little dive into my app to introduce HTTP), the protocol that forms the de facto means of communication for the internet. And then I showed some HTTP requests), the basic means of requesting data from servers, served over unsecured HTTP and secured HTTPS, and I hope that illustrated sufficiently why HTTPS is really important.

In Season 3, I’ll continue where I left off. We’ll see where that HTTP request packet goes, and what this Internet thing looks like on the outside.

The HTTP packet, revisited

Remember this from Issue 12)?

The two columns (Source and Destination) that I censored out contain my IP address, and Hypothes.is’s IP address. Why would I want to censor that? I hope this and the next few issues will make it clear.

When my laptop sends out the HTTP packet, the radio waves containing the data do not have enough energy to reach all the way to wherever Hypothes.is’s servers are based. Instead, before they completely dissipate to undetectable levels, my router has to receive them (just like a radio receives FM waves with its antenna), decode the information, and pass it forward to my ISP’s server, which then forwards it to another server … until it finally reaches the Hypothes.is server.

How does it do that?

The IP address

Each HTTP packet contains information about the source of the packet (i.e. my router), and the destination of the packet (i.e. Hypothes.is). This information is encoded as a string of four numbers, ranging from 0 to 255 (the smallest possible combination is 0.0.0.0, the largest is 255.255.255.255).

This string of four numbers is known as an IP address.

If you are good at math, you can work out the total number of available combinations: it’s 4,294,967,296 (256^4). It seems like a lot, but we actually have more humans on this planet than we have IP addresses. And if we assign a unique IP address to each device on the Internet, many of us will need multiple IP addresses for our smartphone, laptop, Amazon Echo, smart TV, router, …

IP addresses are a limited resource! They are really precious! If you don’t have one, nobody would be able to send you anything. The HTTP packets that other people send will never reach you. The HTTP packets you send will not have an IP address in the Source column above. You would be cut off from the internet. (Less dramatically, you would be offline.)

When you sign a contract with your ISP for an internet connection, they might give you a free router and other niceties or freebies. But the most important thing they give you is the privilege of using one of the IP addresses they own.

Wait what?! Why do the ISPs get to hog all the IP addresses?

A long long time ago … well, actually just back in 1987, a bunch of internet pioneers came together and published a list of reserved IP addresses in RFC1010 (starting on page 16). The RFCs (Request For Comments) are documents published by the Internet Engineering Task Force (IETF) for interested folks to review. Considering the magnitude of responsibility involved, a committee was drafted … and the Internet Assigned Numbers Authority (IANA) was born in 1988.

IANA parcelled out IP addresses to regional Internet registries, which further subassigned them to smaller Internet registries, such as your ISP. Lots of money was involved, unsurprisingly. To own an IP address means to have every registry out there agree that all data packets intended for that IP address should be forwarded to your server. That’s a lot of obedience you own, and it will cost you.

Okay, so how do those packets actually make their way to you if they don’t originate anywhere near you?

Forwarding Tables and The Gateway

Your smartphone, or laptop, first sends the packet to its gateway (which is typically your router). Your router differs from your handy gadgets in one key way: it is a server, while your gadgets are clients (I covered clients and servers in Issue 7.)). The router contains a forwarding table, which is a table telling it where to forward data meant for various IP addresses. For example, if you are trying to send a document to your home network printer, that data packet is not meant to go out onto the Internet—it’s meant to be forwarded to the printer! The information in the forwarding table ensures that this happens. Your laptop has no idea what the IP address of the printer is; all it knows is your router’s IP address, and it will forward everything to your router for it to figure out.

Network connection properties, a window that one used to see very often when configuring a router. Source: Help Desk Geek

If the destination IP address is not in the forwarding table, the data packet gets forwarded to your router’s gateway, which is typically your ISP’s router. The ISP router then checks its forwarding tables to see if the destination address is in there (maybe the data packet is meant for one of your ISP’s customers?), if not it gets forwarded to the next gateway … a few hops and bounces later, it finally reaches its destination. Phew!

Issue summary: IP addresses are a string of four numbers. A list of reserved IP addresses is managed by IANA, and all Internet registries agree to forward data packets according to that list. A data packet sent from a client goes to its gateway. At the gateway, the destination IP address is checked against the gateway’s forwarding tables. If the IP address is found in the forwarding table, it gets sent along that route, otherwise it gets forwarded to the next gateway, … until it reaches its destination.

This looked like a good place to stop before I get any deeper. I’ve left out a lot of the history of development of the internet and Internet Protocol (which is where the acronym IP comes from), because it is necessarily messy and not really relevant here. But one important result of these developments is that there isn’t a single master server out there to which all data packets must go to be sorted. This makes the Internet more robust (if one gateway fails, your packet can still reach its destination via other routes), but more importantly it prevents server owners from dominating the entire Internet and its development (because whoever owns that master gateway will hold great sway over the flow of information).

What I’ll be covering next

Next season: Domain Names (because who memorises IP addresses anyway?)

Sometime in the future: What is:

booting up? [Issue 15]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
those ‘\r\n’s in the HTTP request packet [Issue 12,17]?
a good reason developers write code and give it away for free online? [Issue 21]
ASCII? [Issue 23]
compiling code into an application [Issue 26]?