Layman's Guide to Computing - Season 01

Issue 13: How do I use HTTPS?

2019-03-09T08:00:00+08:00

Last issue I compared HTTPS and HTTP, and explained why you should use HTTPS as far as possible to keep your data secure.

How would I know if I’m using HTTP or HTTPS?

Most web browsers now indicate whether you are using HTTP or HTTPS in the address bar.

Since version 68, Google Chrome marks HTTP sites as insecure in the address bar

Firefox uses a green lock icon in the address bar to indicate that you are using HTTPS to access the website

I can’t possibly keep you updated on how every single browser informs you that you are using HTTPS instead of HTTP, so here’s a guideline instead: look for a lock icon, a green instead of red icon, or something that says “secure” and not “insecure”.

If you are still not sure, click the green/lock icon for more information:

Firefox displays additional information when you click on the lock icon. Other browsers do this as well.

How do I actually use HTTPS?

Short answer: When you type a URL in the web browser, make sure you specify the HTTPS method (https://) in front.

Long answer: In 2018, when you type a URL in a web browser and go, most websites will direct you to their HTTPS site. Most sites won’t even let you access their HTTP webpage anymore; going to their HTTP webpage returns a 301 error (see Issue 8) forcing you to be redirected to their HTTPS webpage.

So as long as you see a green and/or lock icon in the address bar, you are already using HTTPS.

Issue summary: Most web browsers use a visual indicator, often a green or lock icon, to let you know that you are using an HTTPS connection. HTTPS is the default connection method for webpages these days and it is quite unlikely you will be able to access an HTTP site.

End of file

I just finished another short issue arc about authentication and encryption! Phew, I hope it wasn’t too complicated.

Congrats: you have been reading this newsletter for a quarter of a year! I’m ending this issue set here, to give it a nice theme. Maybe I’ll name it “Ask and You Shall Receive”. Or maybe not. I’m not too good with names, reply and let me know if you have a cool one :)

Of course, this isn’t all there is to know about HTTP and HTTPS and APIs, but my purpose here is to help layfolks know the things they should know while getting around the internet, and I hope I’ve convinced more people that

APIs are really cool (I’ll likely touch on this again in future issue sets)
Wifi is really unsafe if you are using HTTP
You really should be using HTTPS as far as possible. You don’t need to worry about it too much, since you are likely already using it.

In the next issue set, I’ll talk a bit about how software developers do their jobs. Again, the issue set will only cover the basic ideas. There are some really cool ideas that I want you to know already exist and are being used, even if I won’t be getting into the technical detail.

If you have ideas for things you would like to have a basic explanation of, let me know in an email reply!

What I’ll be covering next

Next issue set: How do developers do their job?

Sometime in the future: What is:

a specification? [Issue 6,8]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
What are those ‘\r\n’s in the HTTP request packet [Issue 12]?

Issue 12: What is HTTPS? How is it different from HTTP?

2019-03-01T08:00:00+08:00

Was the last issue too scary? I hope you haven’t completely lost faith in protecting your privacy on the internet! If you’re close to it, let this issue bring you some hope.

We are going to get a little technical here, because I want to give you a view of the internet that most layfolks won’t get a chance to see. I just want you to remember something: you don’t need to understand every single bit of it. The things I point out will help you come to useful realisations that help you make good decisions about what you do on the internet.

What did the laptop say to the access point?

[Insert screenshot of random letters and numbers here … :D]

Nah, I won’t do that. It would be merely decorative and not at all illustrative. Instead let’s focus your attention on one particular packet:

An HTTP request captured in Wireshark. Notice the line Authorization: Bearer [CENSORED]. That’s my developer API key!

This screenshot comes from an app called Wireshark, used to “capture” packets received by my laptop. This includes packets that my laptop sends to and receives from the access point, but it also includes packets from surrounding devices, such as my robot vacuum, smartphone, home electricity monitor … let’s see what information is visible from these captured packets. (I say “capture” because I haven’t trapped the packet at all, and other devices connecting to the same access point can read the packet as well.)

It took some filtering and digging, but I’ve finally found the HTTP request data packet which my laptop sent to the server. It’s the right packet; the request is GET /api/profile HTTP/1.1 like we saw in Issue 7. What are those \r\n’s? Not important for now, we’ll eventually get to those sometime.

Look what else is visible there: my developer API key! Look for the line that starts with Authorization: Bearer [CENSORED]. Even though the API key appears to be hidden in the request header, any device that receives the packet and decodes the header can actually receive my API key, and the user can use it to retrieve my profile data if they know how to :( Or worse, they can intercept the packet, modify the header, and send it to the Hypothes.is server, which will think it is coming from me!

HTTPS to the rescue

Since wifi works by broadcasting the data packets, I can’t possibly stop other devices receiving my data. How do I protect my identity and prevent other people stealing my API key then?

The same way we prevent people who can overhear us from understanding what we are saying: by speaking in code.

How will the server know what I am saying then? The server and my laptop will need to coordinate a way of speaking in code so that any information that is intercepted will make no sense to the interceptor, and if the data is modified by them, my laptop or the server would know when trying to decode the data. These are called cryptography methods, and we will not go into more detail than that for this issue.

That means we need a slightly different set of rules, that enable us to coordinate such cryptography. This is where HTTPS comes in.

What is HTTPS?

HTTPS stands for HTTP Secure. Why is it secure? I think a screenshot will make it clear. This is a screenshot from Wireshark again, but this time capturing an HTTPS request packet:

An HTTPS request captured in Wireshark. Notice that the packet header data is now encrypted, and an app would need to know the prearranged encryption code to be able to decode the data.

The Authorization: Bearer line is no longer visible; in fact, all the information we saw in the HTTP packet is no longer visible. It has all been encrypted! Any third party intercepting this packet will not be able to decode or modify it without knowing the encryption code that was prearranged between my laptop and the Hypothes.is server.

HTTP considered harmful

In a more innocent time, it was perfectly all right for passwords to just be sent unencrypted, and for anything and everything to use HTTP. Online shopping wasn’t even a thing then, and there was no financial value to anything that happened on the internet. Hard to believe, I know.

It is now the year 2018. That is no longer the case. We transmit all kinds of valuable data over the internet: financial transactions and credit card details, home addresses and email addresses and passwords, even pictures. If you are using HTTP, any data you send or receive is being broadcast unencrypted. If you are just browsing and not transmitting critical data, that is fine. But you are really much, much safer using HTTPS.

Issue summary: HTTPS encrypts the request or response header and body, ensuring that anyone trying to intercept it will not be able to decode the data without knowing the encryption code. It is much safer to use HTTPS in general to protect your personal data from being snooped.

What I’ll be covering next

Next issue: How do I use HTTPS?

Sometime in the future: What is:

a specification? [Issue 6,8]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
What are those ‘\r\n’s in the HTTP request packet [Issue 12]?

Issue 11: How does wifi work?

2019-02-23T08:00:00+08:00

In the last issue, I showed you how I create HTTP requests with custom headers. I put my developer API key in the request header and Hypothes.is gave me a response containing my profile data.

Let’s do a perspective change. Zoom out, out of the computer. Where does my request go? Obviously it has to get out of my computer and get to the server. My laptop’s wifi chip turns the request from an electrical signal into electromagnetic waves, which are broadcasted to the nearest wifi access point, which converts it back to an electrical signal and sends it on its way to the rest of the internet.

Pause that picture for a while. How does my laptop know which access point to send the request to? How does the access point know, from among a bunch of laptops in the vicinity, which laptop to return the response to? (This is becoming quite a disturbing pattern of questioning isn’t it.)

Short answer: my laptop doesn’t know! Neither does the access point!

Long answer: When you tune in to the radio and get an FM signal, do you ever wonder how the radio broadcast station knows which radios to send the signal to?

Silly question isn’t it? It doesn’t send the signal, it broadcasts the signal. That means the signal is simply sent out in all directions from the wifi antenna, like someone shouting aloud or like a lighthouse spreading light all around.

That is how wifi works too. More shocking news: unlike radio, wifi works on the same two frequencies: 2.4GHz and 5GHz. That means that everything you are sending via wifi can actually be picked up by other wireless devices in the vicinity.

Your signals not only can be picked up, they are picked up by other devices. Remember: those other devices have no way of telling beforehand which signals are from you, and which ones are from the access point! They would have to receive the signal, decode the request (or response), look at the metadata information that is provided about the data packet, and then determine if it is actually for them. Think of it as a shared data mailbox where everyone can see everyone else’s data packets.

Okay, take a deep breath, control your breathing first … feeling better? Good. So you heard the bad news: all your data is being broadcasted in the open, for anybody with a wifi antenna to receive. Now for some good news.

Wifi was introduced in 1998 and has been around for 20 years. In those 20 years, someone must have thought about this, realised “oh man, this is actually a really bad idea”, and did something about it, right?

Yep! The answer to this problem is HTTPS.

Issue summary: wifi broadcasts your data in the open. Any device with a wifi antenna receives your data packets, but normally does not decode it unless it is meant for them. This may seem to be a security risk, but measures have been developed and implemented to keep your critical information secure. HTTPS is one such measure.

What I’ll be covering next

Next few issues: What’s the difference between HTTP and HTTPS?

Sometime in the future: What is:

a specification? [Issue 6,8]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]

Issue 10: How do websites actually know if you are really you?

2019-02-16T08:00:00+08:00

Short answer: They actually don’t. This is not a snarky answer; it is true! Think about it. When the server receives a request, it has no idea if it’s coming from real-you, or someone pretending to be you. All it knows is your IP address, your user agent, and your request. And all three can actually be spoofed (but we won’t go into detail here).

Long answer: We have a long and complicated system of verifying that the person making the request really owns the account and it is called … the login.

Logins depend on what is known in cryptography as a shared secret. A shared secret is a piece information that only you and the server know, and no one else is supposed to know it. If you are able to provide that shared secret, it is probably safe to assume that you are … really you (hopefully?).

When I first access the Hypothes.is API, the Hypothes.is server has no way to know that I am the user whose username is “kureshii”. Even if I declare that I am kureshii, the server has no way to know I really am kureshii—anybody could say that! So when I request a profile, it returns me the default public profile:

https://hypothes.is/api/profile (without custom header)
{
  "userid": null,
  "groups": [
    {
      "public": true,
      "name": "Public",
      "id": "__world__"
    }
  ],
  "authority": "hypothes.is",
}

I need to submit a shared secret in my request to let the server know, “hey, I really am kureshii! Look, this is the shared secret which you gave to me when I set up my account”.

The shared secret I share with the Hypothes.is server: my developer API key

How do I put that shared secret in the header? I could write a few lines of code in Python, a programming language … but I found an online API tester, which makes my life a bit easier. I just need to fill in the appropriate text fields:

The API tester makes it really easy for me to create and send HTTP requests with customised headers

When the server receives the header with my developer API key, it can verify that the key is correct, and hence give me my profile data.

https://hypothes.is/api/profile (with custom header)
{
  "user_info": {
    "display_name": "JS Ng"
        },
    "preferences": {},
    "groups": [
      {
          "public": true,
            "name": "Public",
            "id": "__world__"
        }
    ],
    "userid": "acct:kureshii@hypothes.is",
    "authority": "hypothes.is"
}

That’s pretty cool, right? Only people who know my developer API key and provide it in the header are able to access my profile data. And thankfully I’m not some kind of bigshot whose shared secrets are coveted by many … but does that mean this shared secret is safe? As my web browser, the client, sends this request, it passes through many digital hands. The first step is making it out of my laptop to the nearest wifi access point.

In the next issue, I’ll tell you some things about how wifi works which I hope will make you pause and think about this more carefully.

What I’ll be covering next

Next few issues: How does wifi work? What’s the difference between HTTP and HTTPS?

Sometime in the future: What is:

a specification? [Issue 6,8]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]

Issue 9: How do I make an HTTP request?

2019-02-09T08:00:00+08:00

Last issue, I talked about HTTP response headers, and how they inform the client about the request status using a status code. Let’s talk about the HTTP request today. An HTTP request header looks like this:

A request header

Really simple. In Issue 6, I said that to use an API we need to send a request using HTTP methods such as GET, POST, DELETE. You can see the GET method in use here. It’s literally just a word in the request header! Followed by the URL which we are requesting, in his case I’m searching for annotations made by me on Hypothes.is.

The rest of the first line in the request header is my request URL. The full URL I am using is actually https://hypothes.is/api/search?user=kureshii. Notice that the request header URL does not have the front bit. This request header is already being crafted according to HTTP (which is a protocol, a set of rules) so we don‘t need the https part. “hypothes.is” is known as the host because it is the server hosting the API.

This means that when I enter the full URL in the address bar, my client takes the full URL, noticing that I want to use the HTTPS method (future issue!) to request the data from the host hypothes.is, and then sends the request (which is actually the api/search?user=kureshii bit). The full URL is not just one long string, but multiple bits containing information for the client and server.

The third line, Accept */*, is the client’s way of saying “send me anything”. A more limited client, for example image-browsing software, might send an HTTP header requesting only images.

The last line is the interesting one.

How do I make an HTTP request?

As humans, we can’t speak computerese. If I asked the server for information, it would not understand what I said. I need software to help me translate my request in a way the server can understand. This software acts as an agent for me, the user. So any software that helps to form an HTTP request, send it to a server, and receive the response, is known as a User Agent. The user agent needs to identify itself in the HTTP header (because the HTTP, a set of rules for transferring webpages, says so).

What is my User Agent?

You can find out what information your user agent is sending in its HTTP header on websites like whatismybrowser.com. Go ahead, it’s safe.

It is the year 2018, and there are all kinds of user agents out there. Some are highly advanced—think about your laptop web browser. Some are really simple and barebones and can’t handle very much—think of the web browser on your Kindle Paperwhite. A server can actually serve different user agents differently, but first it must be able to identify them. The user agent string, the fourth line in the request header, is how it identifies the user agent and delivers a response, hopefully an appropriate one with 200 in the header.

Ever wondered how some websites or services seem to know what kind of web browser you are using, perhaps even what operating system you are using? If you have ever needed to install drivers or software, some websites figure out whether to give you the Linux version or the Windows version or the Mac version, and they do so through your user agent string.

Yep, that means that on less scrupulous websites, the server can actually know something about you from your user agent declaration. It can, for example, know that you are using an iPhone and hence serve you a different webpage format—or different ads. In the past, when the old version of Internet Explorer failed to render some pages correctly, some websites took to blocking it from visiting their site entirely, or serving a different, simpler site. They did so by detecting the user agent.

Why does my user agent in the request look so weird? That’s because I’m using an API tester, apitester.com, to check if I am using Hypothes.is’s API correctly (by looking at the response status code, introduced in Issue 8). I need to do that because I am accessing some information that is not available publicly. The Hypothes.is API needs to know who I am before it will serve that information, so I need to create a customised request header to include that identity. I can’t do that easily with a regular web browser, so I need specialised software tools.

And that will bring us to authentication and HTTPS in the next few issues.

Issue summary: A user agent communicates with the server on your behalf. It takes your request (a URL), breaks it down into parts (the protocol, the host, and the request string), puts this information and its user agent declaration into the header, and sends the request for you. To take greater control over what gets sent in the request header, you need specialised software tools.

What I’ll be covering next

Next few issues: What’s the difference between HTTP and HTTPS? How do websites actually know if you are really you and not someone pretending to be you?

Sometime in the future: What is:

a specification? [Issue 6,8]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]

We spent three bite-sized issues just talking about HTTP requests, responses, and their headers. Was it too much info? Too dry? Hit ‘Reply’ and let me know what you liked and what you didn’t :)

Issue 8: HTTP error codes—How does a server let the client know if there’s something wrong with their HTTP request?

2019-02-02T08:00:00+08:00

Last issue, I said that HTTP is a set of rules for sending and receiving documents that link to other documents. According to those rules, if you want a webpage, you (the client) must send a (HTTP) request to a server, which will return a response.

I almost forgot one of my rules here, that it is just as important to know why you should do something as to know how to do it. All I need right now is to craft an HTTP request to get my data from Hypothes.is. In fact, HTTP requests and responses form the backbone of everything we do on the internet. Knowing what I know about them, I want to go a little in depth (just one issue, I promise), so that later some of the things I say about internet security will make a little more sense.

Last thing before the dive: A request or response typically has two parts: a header and a body. Think of the body as a letter you receive; the header is the envelope that the letter came in. It tells you something about where or who this letter is from, whether it was returned or successfully delivered, maybe even a date stamp.

What does an HTTP Response header look like?

We’ll get to looking at what kind of data goes out in the HTTP request next issue. Right now, the HTTP response is simpler and easier to talk about. This is what an HTTP response from the Hypothes.is API (Issue 4) looks like:

The response header from Hypothes.is

Whoa there, I thought this was supposed to be simple to talk about?? It really is! We’re not going to go through all of that, I just wanted to show you an HTTP response first.

The key parts of the response for us are all in the first three lines:

HTTP/1.1 200 OK
Date: Sat, 17 Nov 2018 04:01:02 GMT
Content-Type: application/json; charset=UTF-8
[...]

In the first line, HTTP/1.1 tells us that this is version 1.1 of the HTTP specification (in use since 1990), and the response from the server is 200 which means OK (more on this below).

The second line is just the date and time that the response was sent. Nothing new.

The third line tells is what kind of data was sent with the HTTP response. We’ll have a look at the data after the response and request, but for now we can see that the data is sent in JSON format which I introduced in Issue 5.

UTF-8 stands for “Unicode Transformation Format - 8 bit”, which means it is a format for transform letters and symbols into 1s and 0s. charset=UTF-8 tells us that the characters in the data come from this set of tables known as Unicode. The tables matter, because characters not in the table cannot be used. You sometimes see this in webpages as � or □. We’ll talk about Unicode when we get to emojis :)

HTTP status codes

Phew, that was some unpacking. Now let’s get to the core of this issue: from this response, how would my client (i.e. your web browser) know if there was something wrong with the request?

It is all in the first line: The 200 part is what we call a status code, since it is a number code that tells us the status of the request. Like error codes that you get from your electronic devices or appliances, there is a whole list of error codes defined by the HTTP/1.1 specification to tell you the exact status of your request. It looks long and scary, so here’s the gist.

Status codes are always three digits. The first digit tells you the category of status:

1XX status codes are technical, and you should never see them as they are supposed to be invisibly handled by your client (i.e. your web browser).

2XX status codes are good news: it means everything is okay! HTTP 200 means OK (as you saw above) and is the usual response your client gets when browsing the internet. If you are using an API, you may also get status codes 201, 202, 203, and 204. We do not need to go into those today but they are generally good news.

3XX status codes are response headers that basically say “your princess is in another castle!” In general, they are usually redirects. If you have noticed your web browser going to a URL and then jumping to another one by itself, you’ve just experienced a redirect. These codes are usually handled invisibly by your client so you shouldn’t run into them often.

4XX status codes are bad news, and generally the server’s way of blaming the client.
400 means the server didn’t understand what you meant (I didn’t understand your English).
401 means you don’t have permission (this place is members only; do I know you?).
403 means you don’t have permission and may not ask for permission (These are not the droids you are looking for).
404, the often-parodied status code, means what you are looking for at that URL didn’t exist.

5XX status codes are also bad news, but this time round the server is blaming itself.

500 means “something is wrong with me; I can’t fulfill your request” (be right back; learning how to make avocado brioche!).
501 means “I don’t know how to do that yet” (but coming soon to a browser near you!).
502 means “I was trying to help you but when I asked Server C it gave me nonsense so now I can’t help” (please complain to your Member of Parliament on Facebook).
503 usually means “all our customer service officers are currently busy, please try again later.”
504 is the worst: you’ve waited until the request timed out and didn’t get a response. (the lights are on but nobody’s home …)

A 404 response header from Hypothes.is

Issue summary: A request or a response consists of a header and a body. The response header contains information about the response. The status code in the response header determines if the request was successful or unsuccessful.

Bonus: What is a URI?

If you scrutinised the rest of the header, you’ll notice terms like “cookie”, “cache”, “XSS Protection”, and acronyms in the URL such as “cdn” and “uri”. I will get to those other terms in a future issue, but let’s zap one of those now: A URI is a Uniform Resource Identifier. It is a string of text that helps to uniquely identify the thing that was tagged with the URI. The URLs (Uniform Resource Locators) we have all come to know and use are a special type of URI that uniquely identifies the page we are visiting.

What I’ll be covering next

Next few issues: What does an HTTP request look like? How do I make an HTTP request? What’s the difference between HTTP and HTTPS?

Sometime in the future: What is:

a specification? [Issue 6,8]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]

Issue 7: What is HTTP?

2019-01-26T08:00:00+08:00

Before I jump into talking about requests, maybe we should start with a more fundamental question: what is HTTP?

HTTP: HyperText Transfer Protocol

Let’s break that term down:

Protocol

A protocol is “a procedure or set of rules”. So HTTP is a set of rules. For doing what?

Transfer Protocol

Okay, this tells us that HTTP is a set of rules for transferring something. Most likely referring to data, right?

HyperText

“HyperText” is a term coined by author Ted Nelson in 1963 to describe documents that can reference each other. Today, we know these references as “hyperlinks” or “URLs” (Uniform Resource Locators. Not important.) So hypertext is what we simply know today as “webpage”.

HTTP = HyperText Transfer Protocol

So, putting all that together, HTTP is a set of rules for transferring webpage data from one computer to another. Not too difficult, was it?

Who sends and who receives?

Oh man, you got me there. Just kidding. Obviously, before the sending and receiving happens, someone has to request, and someone has to respond.

Usually, you’re the one requesting. Okay, you’re not actually requesting directly; your web browser (Safari, or Chrome, or Firefox, or Edge) does so on your behalf. So it (the browser) is known as the client. Your request is sent to the computer that has the data. The computer then serves you the data, so it (the computer with the data) is known as the server. When you read about people talking about client-side blahblah or server-side blahblah, they are talking about the requesting computer or the responding computer.

To add just a bit more jargon (you can take a little bit more, right? It’s not difficult, I promise): the data your browser sends to ask for a webpage is known as the request. And the data the server sends back to you is known as the response. There, I promised you it wasn’t so difficult.

Summary: The client (that’s you!) sends a request for data to the server. The server returns a response with data to the client.

Huh, that sounds pretty nice on paper. But in real life, with humans, we know that’s not how it works. The client asks for something, the server says “nope, we’re all out of avocado toast”, the server sometimes forgets your order, sometimes the client forgets to request data and waits for the server to take their order which never happens …

All kidding aside, computers aren’t that complex, but obviously that’s not all there is to HTTP. There’s a client who sends a request, and a server who returns a response, and a million and one ways to miscommunicate. How does the server let the client know that what they’re asking for is out of stock or not on the menu?

Issue summary: HTTP is a set of rules for sending and receiving webpages that link to other webpages. According to the rules, if you want a webpages, you (the client) must send a (HTTP) request to a server, which will return a response.

What I’ll be covering next

Next few issues: How does a server let the client know if there’s something wrong with their HTTP request? What does an HTTP request look like? How do I make an HTTP request? What’s the difference between HTTP and HTTPS?

The past two issues have been pretty dry and text-only. Definitely pictures to come in the next issue, I promise.

—JS

Issue 6: How do you use an API?

2019-01-19T08:00:00+08:00

You ask it nicely, of course.

Sorry, I didn’t mean to be snarky. But that really is what you do. The question is, “what is a nice request?”

These days, the answer is “a request that follows REST principles”.

Nope, I’m not going to tell you what REST (REpresentational State Transfer) is. It is not something you must know to get around. But there are some things about it you should know:

It is simpler and easier to use than what came before it (SOAP).
It uses existing features of the internet as we know it, such as HTTP/HTTPS (okay, okay … I will get to this at some point), rather than trying to come up with yet more systems to build.
Because of (2), it uses familiar methods from HTTP to send/receive data, such as GET, POST, DELETE. These were originally proposed by Tim Berners-Lee in 1996 (with PUT and DELETE in 1999) in the HTTP specification (what’s that? Another issue on this sometime in the future …)
It uses error status codes (such as the all-too-familiar 404 error) to tell you if you succeeded or failed, and why. So you aren’t left guessing. Unlike some people you know.
These days, everyone uses it. Just like they do with JSON.

To receive data, you need to craft (and send) a GET request. To send data, you need to craft (and send) a PUT/POST/PATCH request. To remove data, you need to craft (and send) a DELETE request. Don’t know what these are? Don’t sweat it. We’ll get to them … sometime.

Issue summary: There are different kinds of HTTP requests-GET, PUT, DELETE, for example. To use an API, you need something that can craft an HTTP request.

What I’ll be covering next

Next few issues: What is HTTP? What does an HTTP request look like? What’s the difference between HTTP and HTTPS? How do I make an HTTP request? And what are error codes?

Sometime in the future: What is a specification?

Okay, I am really starting to lay on the terminology thick in this issue. I did that consciously because I thought the HTTP request types (GET, PUT, DELETE, but there are others) would be clear enough even for non-techies. REST and SOAP are there on the off-chance that someone might want to Google it. You are also going to see REST commonly enough elsewhere on the internet once you start googling about APIs, and at least now you know it means it is easy to ask it nicely for data. HTTP requests are what I’m going into next, and while it isn’t critical understanding for everyday digital life, it gives background understanding for the meat of Season 1: why HTTPS is important and how it protects you.

Am I going too fast? Any suggestions for clarity or ease of reading? What do you like/dislike? I would love to hear your comments.

Issue 5: What is JSON?

2019-01-12T08:00:00+08:00

Say you want to send some data to another app. Let’s say it’s an annotation from Hypothes.is:

A highlight in Hypothes.is, to illustrate the various bits of info that must be sent.

What data is included in this annotation?

Date (and time) that the annotation was created
user who created the annotation (that’s me!)
The URL of the page that the annotation was made on (that’s on NYT)
A link to the URL of the annotation (the permalink you can use to bring someone straight to your annotation)
Permissions: is this annotation visible to public or only to certain groups?

Besides the above, there is also some information my app may need to make things easier: - A unique id for the annotation so that I can refer to it again. That’s the jumbled text you see in the URL, 4mN0juVuEeipmD-559uHoQ - Whether there were any replies to the annotation (the picture above shows an annotation and a reply to it) - Document title - Tags, usually used for organising lots of annotations

How is the data from the API going to tell you which bits are which?

It could do it with a bulleted list like I used above. But what if there were multiple replies and I need sub-bullets? How do I differentiate data that is a simple list [a,b,c] vs a mapping of values {"name": a, "date": b, "user": c}?

XML, which you saw in the previous issue, is one way to do it. JSON (https://www.json.org/) is another way. The JSON way minimises the number of extra text you need to structure the data nicely. This is a big deal when you are sending data over the internet, for billions of users.

XML, with its XML tags, often balloons in data size (see this example where an Excel spreadsheet was exported to different file formats. The XML file was 840MB, while the same data in comma-separated value (CSV) format was only 34MB.

Each bit of data that is sent has a cost with it; no doubt if you have a mobile data plan you will know what I mean. In software engineering, we have a term to describe this, borrowed from business: we say that it has immense overhead.

What? You mean processing XML requires money? Of course! Think about it: - Visually, it takes longer to read and skim through XML, which takes up precious time for software engineers - More data needs to be sent for an XML packet than for, say, a JSON packet, which either takes more time on a connection or costs more for services that charge per unit of data - It requires more computing resources to process more data: all that data needs to be copied to the processor for processing, which means that it takes marginally longer to process the same data in XML vs other formats, all else being equal. And processing time is precious, even if hardware is cheap; there is just so much other data to process!

Amid other competing standards, JSON won out and became a de facto standard for applications to exchange data. Which makes things a little easier for everyone: when we can agree on one de facto standard, everyone knows they gotta make things that support it. So it’s really easy to find something that can help me process JSON.

The same annotation, in JSON format

Issue summary: JSON is the de facto data format for sending data over the web. Almost everything you use that was updated significantly in the last 5 years is probably using JSON-formatted data in some way.

Next issue: How do you use an API (to send/receive JSON)?

Issue 4: What is an API?

2019-01-05T08:00:00+08:00

Short version: An Application Programming Interface (API) is a data browser for apps, a way for apps to talk to each other.

Long version: Imagine you’re an app. You would rather have the data raw, without all the formatting that humans consider vital to their understanding.

If a human wanted to access my Hypothes.is profile page, they would access https://hypothes.is/users/kureshii and see a nicely formatted profile page.

My Hypothe.is profile page. Yeah, you already saw this in Issue 2.

If an app wanted a data-only view, it could access https://hypothes.is/api/search?user=kureshii and get the same data:

Raw data from the Hypothes.is API
(that’s Application Programming Interface)

Cool, isn’t it? Looks very different from HTML, but still looks kinda … structured. Those {curly brackets} and [square brackets] look annoying though. How’re we ever going to work through those?

Luckily for us, this way of using curly and square brackets is pretty standardised these days. It is called JavaScript Object Notation (JSON for short). It was specified by Douglas Crockford in the early 2000s, presumably to make things easier for all of us, because it was meant to replace some competing formats, such as XML.

The same data, in XML format.
(Those look like HTML tags, but they are XML tags. Don’t worry about the difference for now, I’ll write an issue about it when it matters.
Maybe when I move on to ebooks.)

How do you pronounce JSON? Here’s a Youtube video of Douglas telling you he doesn’t care how you pronounce it (he pronounces it “jay-sun” like Jason, I pronounce it “jay-sawn”. Whatever.)

By the way, I believe API is pronounced “ay-pee-aye”. Some people may pronounce it “ah-pee”, “ay-pee” or something. Try not to laugh.

Oops, I got sidetracked. So, Hypothes.is is really cool because they got an API that makes it easier for my app to get raw data. How do we use this API thing? And what is JSON anyway?

Next two issues: What is JSON? How do you use an API?

Four issues in, and I’ve just introduced the first technical term: API. This is going to be a pattern for this newsletter: I’ll illustrate some examples that highlight a problem or issue, before I introduce the incumbent solution and the term people use to refer to it. One drawback to this approach is that the introduction can sometimes seem obtuse or confusing: “where is he going with this example?” If this should occur, the fault is mine for the poor writing.

How are the first four issues for you? Was it difficult grasping the examples and what they illustrate? Hit reply and let me know 📧 Even if you don’t think there’s anything to complain about, I would find it reassuring to know that the current format is okay.

—Jun Siang

Issue 3: What is all this clutter?

2018-12-29T08:00:00+08:00

You would think that the first challenge in writing an app is learning the programming language. In my case, there was an even more fundamental challenge to overcome first: how to get my data.

We don’t think about who actually owns the data you upload onto the internet until it is too late. You access your photos by logging in to Instagram, your tweets by logging in to Twitter, and everything else by logging in to something else. Until one of more of these services break down or you get tired of them and you want out, you would never even think about the fact that you often can’t get all this data out except by manually opening each and everyone one of them and saving them to your disk. But I will need some way to get my data out from Hypothes.is, or my app is not going to happen …

“Well, maybe if I learn programming and figure out a way for an app to load these pages and save them automatically, it could go a lot faster.” It certainly can; this method is called page scraping (more on this possibly in a future issue; but not now). But it means your app will have to deal with HTML data, like this:

Instapaper page source. That’s HTML.

Whoa. I may have intentionally formatted it to make it look more dense, but I mean no deceit; in fact, this is closer to what your app has to deal with than nicely formatted code (which will have you tearing your hair out when you have to strip out all the newlines and spaces and tabs from it).

Instapaper’s Notes page is actually not complex; visually it looks like this (I have cropped the sidebar since that didn’t appear in the code chunk above):

Instapaper notes page. Sidebars have been cropped off.

What are all those things (they’re called HTML tags) doing in the page source? They tell the browser how to format the text, usually to appeal to internet users. But when all you want is the raw data, these tags are annoying and tedious to sort through and remove.

Isn’t there an easier way to just get only the data? Why yes; it is known as an Application Programming Interface, or API.

Next issue: What is an API?

Issue 2: Don’t reinvent the wheel

2018-12-22T08:00:00+08:00

Just thinking about recreating the Readmill reading experience in an app was mind-boggling, especially for someone like me with minimal programming experience. I decided to poke around looking for ideas, projects, things I could hop on to.

As a way to have highlight permalinks, Hypothes.is looks really promising. I can link people to highlights I have made, and they can reply inline, and the whole conversation can be referenced via a convenient URL (such as this). It even appears in context if you load the Hypothes.is extension while on the page.

One of my highlights in Hypothes.is. It has its own permalink!

The best part? I already have an account, and quite a few highlights in there for use. No need to write the whole annotation system myself! Now I just need to figure out how to get my annotations from there, and put new annotations into the Hypothes.is service.

My Hypothes.is profile and annotations

First, I will need to figure out how to read and write the data from the website, and how to interpret it.

Next issue: Reading and writing web data-what is all this clutter?

Issue 1: Why make an app?

2018-12-15T07:00:00+08:00

These holidays, I started making an app. It wasn’t a spontaneous decision. I had been thinking about it for years.

It started when I started saving articles from the internet to read in my free time. First on diigo, then Pinboard, then Pocket, then Instapaper, and a few other little experimental apps I’ve forgotten (and which probably aren’t accessible now). I realised, hundreds of articles in, that I didn’t really have a way to revisit what I had read. Worse still, I didn’t even have an easy way to retrieve all the thoughts I had when I read what I read. And I was far from having a consistent way of linking what I read.

The current best services for highlighting and annotating, in my experience, are: 1) Pocket, which was great for reading but didn’t have highlights (back then). 2) Instapaper, which was great for highlighting, but not very good for getting highlights out of the service into anything else. 3) Readmill, which was bought by Dropbox in 2014 and scuttled. This was, arguably, the first social reading app that made an impact. And I had been hoping for someone to recreate the experience.

If your impression of “social reading” brings to mind Facebook, Twitter and other services, you are way off. That already exists, and it is closer to “social sharing”. Facebook and Twitter don’t make it easy to have meaningful conversations around discrete bits of reading which are easily referenced later.

Readmill was shaped primarily around that. The first thing it brought was a permalink to each highlight you made. A permalink is short for a “permanent link”, a bit of text that you can send to someone, paste into the address bar of a web browser, and be taken to a webpage that you want them to see.

A Readmill highlight on the Readmill webpage (with a permalink)

These highlights (optionally) appeared in-location in the mobile apps. You could choose to have them off, for focused reading, or to have them on, for the serendipitous experience of meeting someone else who has something to say about what you are reading.

A Readmill highlight in the Readmill app

The social reading experience seemed gimmicky at first, but the turning point came for me when Clive Thompson, who had just published Smarter Than You Think, started replying to notes and comments in his book. Being able to speak with the books author on specific highlights was world-changing. You could ask questions about points in the book that weren’t clear to you, and have the author clarify it right there. You could mention something it reminds you of, and have the author point you to an article they read or wrote about the same idea. And these were available to other users of the service. Better yet, their correspondence with the author was available to you too! I regret not taking screenshots of some of the exchanges I had; I never expected I would need them at some point.

Even without the social reading experience, simply having a permalink to each highlight, and your notes on it, is immensely helpful for having conversations online, with people. The web has similar approaches, such as Medium’s highlights feature. But none of them enable a deeper conversation around that discrete bit of highlighting; Medium does not even provide a permalink.

I want permalinks for all my highlights, and I am going to have to make them myself if no one will provide them.

Issue 0: Why write a newsletter?

2018-12-08T08:00:00+08:00

Where do I start?

Someone I know just took up a (sponsored) course to learn HTML, CSS, and eventually maybe Javascript and maybe even Python!

What does it mean to have been taught HTML? What does it mean to receive a certificate stating that you have learnt HTML? Why would anyone need to learn HTML? Will you be able to make a webpage, knowing some HTML? Will you be able to make a nice website with CSS? Many of these courses claim you can. That’s why you’re paying for them, right?

These courses often guarantee that in the course of 2 hours, you will learn how to write HTML. You will learn what CSS is. But here is what these courses cannot guarantee:

whether you are able to link HTML and CSS to everything else you know about computers, webpages, and the internet.
Whether you know how to google for help when you want to try to do something else with HTML, CSS, and whatnot.
Whether your big-picture view of computers has improved in any way.

They cannot guarantee that because it takes more than just knowing programming. It’s a kind of generalised knowledge that lately is becoming the kind of common sense that any internet denizen must know to keep themselves safe and avoid trouble on the internet.

I’ve spent more than 10 years using computers: toying with MS-DOS, escaping lab staff while trying to complete an accelerated download in Go!zilla, assembling computers by hand, googling every question in my head, processing data in MATLAB for undergraduate research, mucking around in MPI and FORTRAN for my final year project, setting up a home server, crashing the disks and losing all the data, figuring out how to restore a disk array, crashing the disks and losing all the data again, writing scripts to do Rube-Goldberg backups, giving up and simplifying my life, doing it all over again, and finally deciding to start writing useful things.

This is what I learnt:

It is better to know a bit of everything than to know everything about only one thing.
You can learn one thing a lot faster if you know something about everything related to it. And you can remember it a lot better too.
Having examples really helps. Real-life examples are a million times more helpful than contrived examples if you’re not sure what’s going on.
Knowing why you should do something is as important as knowing how to do it.
You don’t need to be an expert to make something. You only need a reason to make it. And lots and lots of time. Having a laptop with an internet connection would really help.

This is what I think:

Software engineers are responsible for more and more of our digital infrastructure today. They do need to be held accountable for their decisions.
That means everybody needs to understand a little about computing to be able to grasp what it is they are doing, and the implications of the decisions they make.
Knowing some of the jargon of software engineering greatly expands your library of experiences. It helps build up a vocabulary of what it means to collaborate and to build. It is inevitable that every programmer builds from the tools and pieces created by other programmers. Their work is indirectly coordinated through some really cool systems. Knowing that these tools actually exist(!) and are actually possible(!) is immensely inspiring.
Knowing some of the common practices of software engineering can help you recognise when something is broken and can be improved.
Knowing computing at a very broad level helps you realise that many aspects of computing are really human decisions. Understanding a computer may require logical thinking, but understanding software ultimately requires understanding the engineer’s perspective.

I want it in newsletter form and not as a blog because:

Maintaining a blog is tiring. Everybody can read a blog—actual software engineers looking to nitpick, a random troll looking for a target, spammers spamming the comments, …. and writing for a blog becomes a kind of defensive act of trying to say things that are correct for as many people as possible.
Email is more personal—I’m writing only to people who actually subscribe! Not to a nebulous, amorphous audience.
I can drop the email from my mind once it’s sent out. With blog posts, I am tempted to go back and edit it again for correctness, grammar, tone, etc. Too emotionally draining. With email, what I have written, I have written.

In these newsletter issues, I will be telling stories. Stories about what things were like in the past. Stories about what I’m doing now. Stories that hopefully resonate with something in you, inspire you, or intrigue you enough to put a search term into Google. Stories that I write within an hour, to stop me from being bogged down by the inertia of trying to write a masterpiece.

Thank you for coming on board. It makes it less lonely :)

—JS

This is the zeroth issue. Subsequently, all issues will be available online at https://buttondown.email/laymansguide. If you have any comments, requests, or corrections, just reply to this email! Please feel free to forward the online issues to anyone you think will benefit from it. And encourage them to sign up too, at https://buttondown.email/laymansguide, if they like it.