Layman's Guide to Computing - Season 02

Issue 26: Software distribution

2019-06-15T08:00:00+08:00

In the past 12 issues I’ve talked a bit about different kinds of programs (web, desktop, mobile, …)), different kinds of code (frontend, backend)), tools that programmers use while programming (command line), libraries) and frameworks), version control), bug/issue trackers), text editors and integrated development environments)), and ideas they use to work more efficiently (specifications), testing), continuous integration)).

To wrap this up, let’s answer one final question: how do they actually get their code out to the rest of the world?

Mobile apps

Let’s start with the easiest: mobile apps, of course, get published to the app store (Apple App Store, or Google Play, for most phones). Programmers could, of course, encourage people to enable installation of third-party apps and ask them to download and open an installation file … but how many people are going to do that?

The process of putting an app onto the app store is known as publishing. This often involves lots of steps—see Google Play’s publishing process—including adding images and writing for the app’s page.

Before a program can be uploaded, a lot of extra information is needed by the app store: the app author’s name and particulars, the version number, minimum OS requirements (e.g. some apps won’t work on Android versions older than 4.4), and so on. The app store may also need a manifest, a list of files in the app. The process of adding all this information and organising the files into something that can be uploaded to an app store is known as packaging.

How is code for other platforms packaged and released? That depends, really, on the community.

Linux: software repositories

On Linux, in the old days, you had to download the source code, compile it into an application (self-note: this needs explaining in a future issue), and install updates for each application individually … this soon became a gargantuan effort for a user. Then some smart people got together and thought: why don’t we make a list of applications, and where they can be found? And then automate it so that a computer can download the source code and compile it automatically? Or, even better, have a central place containing compiled applications that can be downloaded, ready to use? And thus the software repository was born. (don’t confuse this with a code repository, which I introduced in Issue 19).)

Each Linux distribution (Ubuntu, GNOME, Linux Mint are some of the more popular ones) has its own software repositories (also known as repos). These repositories contain the most important or popular applications that their users need, and are updated and maintained regularly by maintainers, developers who take on the role of testing code for compatibility. The maintainers often have to make changes, some little and some big, for the applications to run well with their distribution (also known as distro).

Linux distros will often have multiple repositories to organise the huge list of applications available, and users can configure which repositories they want to use. They can edit the list of repositories directly, but most users will use a package manager, a program that helps users search for, install, and update the applications on their computer. Many of these package managers are command line applications, but there are graphical package managers available as well.

Plugins: it depends …

Really popular software that allows users to write and release their own plugins will often also have a way for users to search for them. For instance, Wordpress has a Plugins page for Wordpress users to add plugins to their Wordpress blog. Depending on what kind of plugins you are writing, the release process will vary widely. Sometimes this plugin repository is run by a community, sometimes it is run by the application developer, and sometimes it is run by a heroic volunteer, so … it depends :)

Libraries: package repositories

If you are learning a programming language, chances are you will have to use at least one third-party library. The easiest way to find them is with a package repository. These are the most popular package/gem repositories for the most popular programming languages:

Python: PyPI
Ruby: RubyGem
NodeJS: npm
Javascript: uhh, wow … there’s just so many specialised ones and no single central repository. But yarn is an up-and-rising one

Issue summary: A developer can simply put up a download on a webpage and hope people download it and figure out how to get the app on their devices. But often, the more popular way is to publish the app to a repository (also known as an app store, for mobile devices). Before this can be done, the code or application needs to be packaged according to the requirements of the repository.

It’s the end of Season 2 of Layman’s Guide. I hope the programmer’s world looks a little less murky now, even if it looks a lot more complex!

Many programming courses tell their students that they can just jump into programming without worrying about all these. And they often set up their courses minimally, introducing Github or a package repository on occasion when they can’t avoid it. But for the most part, they don’t explain what these parts of the programmer’s world are and how to use them.

To me, that’s like bringing a friend from overseas to your country and not explaining how the transportation system works, how to buy things, what the most popular methods of payment are, how to cross the road … you can kind of hole yourself up at home and maybe venture out to the convenience store down the street, but what fun is that?

These are tools and ideas you can learn and use separately from programming (I’m using Github to get version control on Layman’s Guide, in fact), and you can get really good with them while knowing very minimal programming. And if you learned anything from Season 2 that really rocked your world, gave you an idea for how to improve things, or changed your view of the way things are done, please drop me an email and let me know :)

What I’ll be covering next

Next season: Data? Cloud?

Pick your own adventure! Let me know what you’d like me to write about next:

Data: different types of data (text, images, video, code). Why JPGs are so small and BMPs are so huge and PNGs are kinda in-between. Why video files are so huge and so difficult to compress. How do cloud companies prevent losing data when a hard disk goes kaput? Things like that.

Cloud: What happens to all the images I upload on Instagram? How does data actually travel from wherever it is to my laptop or smartphone? How do people actually start making their programs use the cloud? Things like that.

I’ll take another break next weekend to get Season 3 hashed out. Expect the first issue end-June!

Sometime in the future: What is:

booting up? [Issue 15]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
those ‘\r\n’s in the HTTP request packet [Issue 12,17]?
a good reason developers write code and give it away for free online? [Issue 21]
ASCII? [Issue 23]
compiling code into an application [Issue 26]?

Issue 25: Text Editors and Integrated Development Environments

2019-06-01T08:00:00+08:00

Last week, I introduced issue trackers, which are software tools that developers use to keep track of problems and issues with their software. Issue trackers are usually places that users can submit bug reports to help out the developers in tracking down a bug.

In the second-last issue of this season, I’m going to do a little detour to talk about what developers use to edit their code. Yeah, the actual software they type their code into.

Editor: something used to edit text

Newspaper editors, book editors, photo editors … nope we’re not talking about those. Text editors are software that is used to edit text. The simplest example you know of is probably Notepad. On the other end of the scale you can have really fancy editors like Sublime Text or Notepad++, or Atom.

Sublime Text

Every programmer has their own favourite. They can get prickly and defensive about their text editors of choice, so be careful what you say about their preferred text editors.

Notepad++

Github Atom

Text editors do what is written on the box: they let you edit text. Often (but not always), that means code. It may have nice features that help you autocomplete words, or close your brackets for you, or highlight keywords to make the code easier to read, but it doesn’t help you think about the flow of logic in your code. For that you need …

IDE: an Integrated Development Environment

An IDE is rather different. Yes, it probably has a big window for you to edit your code. But it has to do much more. It is called a development environment because it lets developers explore their code. An IDE can run your code, show you the values of the variables within your code in real-time, or let you pause the code at a point so you can see what the program is doing before it exits or crashes. This helps greatly with the debugging process.

Spyder IDE, showing its object inspector (for checking variables) and built-in console (where the code is run)

An IDE can test the performance of your code to let you know where it is running slowly (a process known as profiling).

Netbeans IDE showing its profiler at work

An IDE integrates many tools into one software package so you can do your work mostly without having to switch around windows.

Issue summary: A text editor helps programmers to edit their code. An IDE (integrated development environment) helps programmers to see what’s going on in their code, test their code’s performance, and provide almost all the necessary tools in one package.

This is a short issue that I wanted to write to answer questions I get on “where to start learning programming”? The usual answer they get is books and websites teaching you how to code, but I contend that the first piece of software you use is critically more important.

In every hobby or pursuit, the tools, gear, and equipment determine the pleasure you get out of it, and likely determine whether you continue to stick with it. Anyone beginning to learn programming should start with a good text editor, and gradually move on to an IDE if they’re working with complex code.

I will save the recommendations for other places; I don’t think a newsletter is appropriate for that. The best text editor for you is going to depend on your use case, your OS, your machine (laptop or workstation), and potentially many other factors.

What I’ll be covering next

Next issue: ???

I haven’t quite decided what I’ll wrap up this season with. I said in Issue 14 that I started this season to help people understand what developers do, to build up their picture of Computing before they start trying to learn programming. Maybe I’ll do a short introduction to operating systems, or maybe we’ll talk about package distribution :) We’ll see next week!

Sometime in the future: What is:

booting up? [Issue 15]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
those ‘\r\n’s in the HTTP request packet [Issue 12,17]?
a good reason developers write code and give it away for free online? [Issue 21]
ASCII? [Issue 23]

Issue 24: Issue trackers, Bug trackers

2019-05-25T13:18:00+08:00

Last week, we talked about how developers plan out a project and describe specifically what needs to be done: they write out a specification, and then they write code to make their app meet the specification. Specifications are also used to standardise the way apps communicate, such as they way to report time.

So what happens when apps and programs do not follow the specification?

Bugs, bugs, and more bugs

There’s a reason for it, and that reason is usually a bug. A bug is a mistake in the software that causes it to not follow the specification. Sometimes it’s a little typo, sometimes it’s a logic error, sometimes it’s running code causing a computer to run out of resources (usually memory), sometimes a library (Issue 17)) you rely on has changed, sometimes it’s just way-too-complex code that nobody can fully figure out … and often, you won’t know until you have spent enough time investigating it.

Software everywhere is full of bugs. They are like bacteria; you can kill them but there are always more!

Why are they called bugs anyway?

In the early days of electromechanical computers, which carried out computation through a mix of electrical and mechanical parts, problems that arose during the computer’s operation were not always caused by wrong instructions sent to the computer. In 1946, operators of the Harvard Mark II encountered an error that was eventually traced to a moth trapped in a relay. Bug hunting then was literal; today it is figurative, but the term has stuck.

A page from the Harvard Mark II electromechanical computer’s log, featuring a dead moth that was removed from the device. Source: Wikipedia

How do developers keep track of so many bugs?

This refrain should start to sound familiar … they do so with the help of machines :)

Software that helps to keep track of bugs are known as bug trackers. Today, when problems with software can sometimes go beyond bugs, such software may also be known as issue trackers.

Such features may also come with other software or services; Github has a built-in issue tracker.

Github’s issue tracker. From Kenneth Reitz’s requests package.

This is where developers working on a piece of software can report bugs or issues they discovered. Often, these reports also come from users of the software.

Through the issue created in the issue tracker, developers can communicate with the user who reported the bug, ask for more information, clarify uses of the software, assign the problem to other developers, or close the issue if it is resolved.

What an issue in Github looks like. From Kenneth Reitz’s requests package.

Projects will often have a contributing guideline to help users understand how best to write a good bug report. Reading and understanding a bug report can be really trying, especially if it is unclear or does not provide enough information, and this can make the debugging work of developers an emotional drain.

It is considered polite to read the project intro and contributing guidelines to understand how the project is being managed and how to write a helpful bug report.

Contributing guidelines for Kenneth Reitz’s requests package.

Issue trackers are never empty. Popular software gets lots of attention from users, who will use it in all kinds of ways not considered or intended by the developers, so there is always something to be worked on!

If you found a bug in a piece of software you use, and don’t know where or how to inform the developers, try searching to see if it has a place to submit bug reports or issues. Most active projects do, and their issue trackers are often easier to find than their email addresses :)

Issue summary: A bug tracker, or issue tracker, is where users can submit problems they encounter with the software. To submit a helpful issue, users should understand the project’s philosophy and purpose, and read the contributing guidelines.

This is a short one, and one I’ve been waiting to write. So many people don’t know that some of the software they use every day have issue trackers, and they can contribute to the healthy development of their favourite software by helping to submit issues they encounter.

How is your workplace keeping track of issues raised by staff? Are they doing it manually in a document? Would an issue tracker help make this job easier?

Next week, I’ll try to answer a question I sometimes get: why do developers like Notepad so much? (They don’t! Just because it is all text does not make it the same as Notepad!) We’ll talk about the kind of editing tools that developers use.

What I’ll be covering next

Next issue: Editors and IDEs: what’s the difference?

Sometime in the future: What is:

booting up? [Issue 15]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
those ‘\r\n’s in the HTTP request packet [Issue 12,17]?
a good reason developers write code and give it away for free online? [Issue 21]
ASCII? [Issue 23]

Issue 23: Specifications in software

2019-05-18T08:00:00+08:00

Last week, we took a look at part of the (heavily simplified) software development pipeline:

fork a repository → make changes → commit changes (and do automated testing) → merge changes when the branch is ready and all tests pass.

At this point, it is time to ask a stupid question: how do developers know what they need to do?

Software requirements

When a project is first accepted, the customer will usually have at least a vague idea of what they want the software to do. Hopefully, they will also have a clearer idea of what they want. For example, if they want an online shopping website, they might have some idea of they want shopping categories, a shopping cart, a search bar (with or without advanced search?), the ability to add or customise promotions, guest checkout, and so on.

Fine decisions, such as the checkout page—the order in which the text fields are arranged, the kind of date/time picker to use for the delivery schedule page, how users should be alerted if there’s an error in the input (alert below the text field or in a warning box at the top), also need to be made. And very often, the customer doesn’t want to make these decisions (which is why they hired a firm like yours to make the website), and so it falls to you (or your designers) to make these decisions instead.

What your customer wants—those are requirements. How the designers (or project lead, or whoever makes the decisions) decide to meet those requirements—those are called specifications.

Specifications

There are more kinds of specifications than I can cover in detail, but let’s talk about some of the ones you are already familiar with.

USB Specification (Hardware)

Did you know that USB was a specification? A group of engineers known as the USB Implementers Forum decides the specifications that USB devices must meet before they can be certified as USB-compatible.

The specification covers a few categories:

Hardware requirements: Shape and dimensions of connectors, which pins (those gold contacts on the USB connector) are for power and which are for data, how to implement backward compatibility with older USB sockets, etc.
Performance requirements: Maximum data transfer rates, maximum charging power, etc.
Operational requirements: How transfers are to be implemented and carried out, supported types of devices and the type of transfer modes to use, how different features are to be implemented (e.g. audio streaming), etc.

HTTP Specification

The HTTP specifications are maintained by the Internet Engineering Task Force (IETF), an open community of people who collectively make the technical decisions about what the Internet should be able to do, and how it is going to do them.

Take a look at some well-known specifications that they have produced (click the links that say “ASCII”—yeah we’ll talk about ASCII sometime). It’s going to be way over your head, but don’t worry about that. Here’s an example from TCP, the Transmission Control Protocol that describes how packets should be sent over the internet:

3.1.  Header Format

  TCP segments are sent as internet datagrams.  The Internet Protocol
  header carries several information fields, including the source and
  destination host addresses [2].  A TCP header follows the internet
  header, supplying information specific to the TCP protocol.  This
  division allows for the existence of host level protocols other than
  TCP.

  TCP Header Format

    0                   1                   2                   3   
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Source Port          |       Destination Port        |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Sequence Number                        |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Acknowledgment Number                      |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Data |           |U|A|P|R|S|F|                               |
   | Offset| Reserved  |R|C|S|S|Y|I|            Window             |
   |       |           |G|K|H|T|N|N|                               |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Checksum            |         Urgent Pointer        |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Options                    |    Padding    |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                             data                              |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                            TCP Header Format

          Note that one tick mark represents one bit position.

                               Figure 3.

  Source Port:  16 bits
    The source port number.

  Destination Port:  16 bits
    The destination port number.

Look at how specific these documents are about how data should be transmitted and received! These documents are how developers and programmers resolve arguments about who is writing their code correctly and who is responsible for a bug. The requirements have to be specific enough that the expected behaviour for a scenario is clear, yet leave enough room for better implementations to replace old ones.

API Specifications

We discussed APIs in an earlier issue: an API describes how apps/programms should send and request information from each other. They are how web developers of online services know how to get you to use your Facebook account to log in, or how to access your phone contacts. APIs are another type of software specification: your app must communicate in a manner that fits the API specification, or it would not receive the desired response from the server.

Time Specifications

We don’t think about it very often, but when someone writes 02/03/19, how do we know if they mean March second or February third of 2019? Such ambiguity is often disastrous in software applications, so there are clear specifications about how apps should standardise the commmunication of time between apps and across the Internet (for internet applications, the standard is RFC3339). Because computers and humans do not process time the same way (see Wikipedia’s System time page), there are specifications for how to do the conversion of machine representations to human-readable representations as well. And because this is applicable across many areas and not just in software, some of these specifications are called standards: specifications which should be followed by everyone who wishes to communicate clearly.

Think there are only 24 time zones? Or that time zones are always 1 hour apart? Or that the current timezones are exactly the same as they were 20 years ago? Think a leap second is always one second? That calendar time always moves forwards? That February never has more than 29 days?

I used to think these things too, until I read Zach Holman’s detailed tract on UTC. Fascinating, and ever so slightly disturbing.

Why are there so many specifications?!

Because they are useful!

When a developer first learns about specifications, it can be really alarming to find out that they haven’t been following many of them. They may even start feeling oppressed by the number of specifications that must be followed.

And then the developer advances. They tackle more finicky problems. They tackle problems that require more precision: time to the nearest nanosecond, synchronisation within a fraction of a second, lag time of no more than 200 milliseconds, etc. When they run into sufficiently difficult problems and find that they need help (such as the aforementioned time representations), that is where they will welcome the specification overlords into their life.

Many specifications were written because similarly talented developers ran into the same issue. And if there is no need to reinvent the wheel, those specifications save you many hours hashing out a suitable specification and implementation (since other developers would have offered sample implementations online), and enable your app to communicate with others who follow the same specification.

Specifications are part of an engineer’s toolbox in the struggle to create order from chaos, and a well-written specification is a thing of beauty.

Issue summary: Specifications describe the details of how a piece of hardware/software should work in order to meet a set of requirements. When well written and well implemented, they aid the coordination of the multitude of devices across the world, enabling them to communicate seamlessly, unambiguously, and unnoticeably with each other. Your devices work because they follow specifications.

Yay, I finally cleared one item from the “sometime in the future” list! When I first started reading implementations, I found them really intimidating and impossible to get into. It was only when I started programming and having to do things like getting my data from an online service through an API, that specifications started making sense. Hardware specifications were useful too, in trying to understand whether a USB charger will be able to do quick charging or support higher rates of data transfer before I buy it.

Because making an app from scratch can be so stressful and annoying, developers usually separate the design work from the programming. Creating a specification and design for an app is both a science and an art, and when done well, can make programming a breeze. A developer just needs to read the relevant part of the specification, think about how to implement that section, and get it done.

Next week, we’ll look at something else that is part of a developer’s life as well: debugging.

What I’ll be covering next

Next issue: Debugging

Sometime in the future: What is:

booting up? [Issue 15]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
those ‘\r\n’s in the HTTP request packet [Issue 12,17]?
a good reason developers write code and give it away for free online? [Issue 21]
ASCII? [Issue 23]

Issue 22: Continuous Integration in software

2019-05-11T08:00:00+08:00

Last week I introduced forking and merging, which are how developers ensure that they don’t override other peoples’ work. They fork a repository to create their own copy, make the changes required to introduce the feature they want, and then merge it back with the main repository when it is ready, for the repository owner to review.

Two weeks ago I introduced the idea of testing, which in programming means writing more code that checks your code so that it is producing the correct output when you give it a certain input.

By now your idea of a developers’ workflow might look something like this:

Fork a repository (which creates a branch).
Make changes to code on that branch.
Test the code on that branch.
If the tests pass, send a pull request to the repository owner to merge this branch with the main branch.

Git branching. From Atlassian’s git branch tutorial.

You can imagine that steps 2 and 3, the bread and butter of developers everywhere, is really tedious. Write code, test fails. Edit code, test fails again. On and on and on.

So engineers automate this part. Git (from Issue 19) can run little bits of code (called a hook) when a commit is made to carry out this testing. So this lets developers set up repositories in a way that tests every single commit (or series of commits)). The above process now looks like:

Fork a repository (which creates a branch). 2a. Make changes to code on that branch. 2b. Git tests the code on that branch and reports the result.
If the tests pass, send a pull request to the repository owner to merge this branch with the main branch.

One step less! But the part that developers dread is usually merging, which is where the code conflicts occur …

Merge conflicts

Git merging. From Atlassian’s git merge tutorial.

Most development teams organise their work into (at least) 3 types of branches:

master, which is usually production code that is actually being run on the servers.
dev, for code in development that the team is working on.
feature, for code forked from development which a developer or smaller team is editing

While Alice is working on a feature branch, Bob is working on his feature branch as well, and Charmaine is merging her change to the dev branch. Charmaine’s changes might cause problems with Alice’s code. Alice is not going to know about this until she tries to merge her branchc with dev! So her work process now looks like:

Fork a repository (which creates a branch). 2a. Make changes to code on that branch. 2b. A git hook tests the code on that branch and reports the result. 2c. Pull any changes in the parent branch (automatically or manually) to keep the code current.
If the tests pass, send a pull request to the repository owner to merge this branch with the main branch.

This means that on top of just testing her code, Alice (and Bob and Charmaine) need to pull in changes from dev whenever it is updated, to ensure that her own changes still work with the latest copy of dev branch. This can be automated with git hooks (those little bits of code mentioned earlier), but the mental stress from having the code change under you can be pretty draining.

Big features, in particular, suffer from problems like this because they involve many code changes that accumulate and affect many parts of dev. So some teams have taken to using a process called Continuous Integration.

Continuous Integration

In a nutshell, continuous integration means to keep your code changes as small as possible, and merge your changes into the parent branch as often as possible, instead of letting them pile up to be released in one day.

Its main advantage is that it allows the process of making a change to be really short. Which then allows these changes to be merged as soon as possible, and be updated to other developers’ branches as soon as possible, so that the latest updates are synchronised to the team, and nobody is working in the dark with an unsynced copy of the base code.

Issue summary: Continuous Integration means merging changes back to the main branch as often as possible. This means keeping code changes as small as possible, and using automated testing to speed up the development process.

When I first learnt about continuous integration, I thought this was something every organisation should know about! Have you had colleagues who tend to do things in the dark, and nobody really knew what they were doing and they seemed to be out of sync with the team? Yeah, continuous integration and help with that.

Understandably, nobody wants to spend more time in meetings and on communication that is otherwise necessary. But in workplaces where the team shares a common repository (like a shared network folder, or Dropbox folder, or Google Drive folder, …), why not try to automate some of the updates which are done by hand?

One example: I don’t like it when events are added to a calendar and I only find out much later. So I used IFTTT, a web service that lets me set up automated processes, to send myself a Telegram message whenever an event is added to a calendar.

In a team, automating such updates means somebody is not wasting time typing out these updates to the team. What can your team automate today so that it is easier for everyone to be in sync?

What I’ll be covering next

Next issue: Specifications: how developers know what to work on (finally!)

Sometime in the future: What is:

booting up? [Issue 15] ~~- a specification? [Issue 6,8]~~
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
those ‘\r\n’s in the HTTP request packet [Issue 12,17]?
a good reason developers write code and give it away for free online? [Issue 21]

Issue 21: Forking and merging

2019-05-04T08:00:00+08:00

So far I have been describing things as though only one developer was working on a project. What happens when two developers are involved? Any kind of collaboration between people working on one thing usually leads to issues coordinating their efforts. How do developers manage this?

Usually, by not working on the same part of the code at the same time. But in addition to that, developers have also come up with some good practices and work philosophies to minimise the friction involved in working together.

In this issue, I want to introduce forking and merging, key ideas behind version control systems and how they keep developers sane.

How developers suggest changes

One really cool thing about developers is how much they love to share their code online. When you see someone else’s code online and you can see improvements you’d love to make to the code, what can you do?

The typical non-developer way is to email them politely, spend a long hour thinking about what to write to explain your point of view, and ask them if they can consider incorporating your proposed change to the code which looks like [attachment].

The typical developer way is to fork their repository and make a pull request (usually after dropping them a quick message to ask first). The repository is where they store their code, and this repository is managed by a version control system (VCS), typically git.

Forking: making your own (copy of the) world

Forking, simply speaking, is making a copy of their code. You need a complete copy of their code so that you can run the app and test the changes you make. It’s called a fork because it’s like coming to a fork in the path of coding. Now there are two paths: the original path, and the path you are about to create.

You could do it stealthily (just copy and paste!), but it’s best not to. Instead, you should do it through git. This informs the code owner that someone has forked their repository and is possibly working on changes to it. It also makes it easy for you to pull any subsequent changes the code owner has made to the code and merge it with your code automatically, so you can test your changes with the latest copy of the code. And more importantly, it allows git to help you with merging your code back into the main codepath (more on this later).

Forking somebody’s project is usually a compliment. It means you find their work interesting enough to want to fork it and work on it!

Pull requests: joining the main codepath again

It’s no fun just working on your own copy of the code. The best part is incorporating your changes back into the main codepath so others can enjoy your work too.

In git, this means merging. When two codepaths are merged, git checks the commits of each codepath (see why version control is so important?), and automagically merges the commits into one path. In the process, it highlights any conflicts (when two commits change the same part of the code) which require human intervention—usually by the owner of the main codepath.

You know how much it drives you nuts when you arrange your stuff a certain way, and the person sharing your table rearranges it. Or when you are working on something and you have your tools set up a certain way, and someone else working on it with you changes the way things are done.

Git forking and merging allows contributions to take place in a saner and more controlled manner. But it does not eliminate the need for discussion and coordination; it just makes the collaboration much more sane and less tedious.

Github: the social network + collaboration platform of developers

Coordinating all this development work offline involves lots of emails and messages, and an online platform helps to ease that problem. Out of many such competing platforms, one particular platform has become the most talked-about: Github, introduced in Issue 19.

Github is like multiplayer git. It lets you create your own account, create your own projects (each one with its own git repository running on Github’s machines), lets you browse other people’s projects, comment on them, report issues and request features, and also fork their repository. Many developers use it to show the code they have been working on, to coordinate efforts for online projects, and to build code that is used around the world.

Why would developers want to do this, to make things for free and let others benefit from it and no cost? The history of open source development is interesting and important to the future of the Internet, but it’s far more than I can cover in one issue—perhaps I’ll do that in a future season.

Issue summary: In git, forking a repository creates a copy of it for you to work on. Merging a repository with the original combines the commits from both so that they become one repository again. Conflicts arising from commits from both codepaths that affect the same part of the code will need to be resolved manually. Developers do most of this forking and merging in Github, an online platform for working on and talking about code.

I’ve simplified a huge portion of how git works, which I think isn’t important to know about for laypeople, so as to focus on the key features of git and how it makes collaboration much saner for developers.

Do you use any online platforms for collaborating with others? If your work is primarily text-based and you would like a way to work on it more easily with people, Creating a git repository on Github is something you might want to try. It is a little less automated than Dropbox, but it gives you more control, especially when you need to roll back changes.

Sometimes, forking and merging can go bad. For example, two codepaths could change parts of the same code so much that it is more effort than it is worth to combine the changes back again. Or sometime, legal issues stand in the way. In the next issue, I want to talk about one way that developers usually set things up to make it as easy as possible to edit code without breaking anything. It is called Continuous Integration.

What I’ll be covering next

Next issue: Continuous Integration: how to not break software while working on it

Sometime in the future: What is/are:

booting up? [Issue 15]
a specification? [Issue 6,8]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
those ‘\r\n’s in the HTTP request packet [Issue 12,17]?
a good reason developers write code and give it away for free online? [Issue 21]

Issue 20: Testing

2019-04-27T08:00:00+08:00

Last issue, I introduced version control, the idea that you can set things up in a way that lets you see the change history of a document. This lets you undo changes which you later come to regret, and compare the document to a previous point in history. If you have a version control system (VCS) set up, you can commit changes to it and view a list of those changes later.

When one is just starting out in software development, one thing that often happens is this. You start writing code for an app-say, a todo list. You write code to set up a database, display a login and other stuff. You commit the changes to the code. You add core features: add/remove/update a todo item; commit. You come up with more ideas: subtasks, recurring items, tags, filters, and all that jazz. Commit, commit, commit. You now have a complete app with bells and whistles, but the code is a sprawling mess: when you finally test the app, there are strange bugs everywhere, the lists don’t filter properly, and the database gets messier and messier.

Proper planning can make this more orderly to some extent, but it cannot eliminate one fact of programming: that there will always be bugs. And the only way to catch them is to test the app.

Manual testing

The most straightforward way is, of course, to get a human to test the app. This gets tiring quickly, for obvious reasons. Asking a human to repeatedly carry out a set of procedures to make sure the app returns the correct data is tedious … which makes it perfect for automating!

If you are testing an app manually, it is human nature to not want to run 200 tests by hand. This leads to people trying to keep the number of tests small, which leads to some bugs not being caught, or they try to go as long as possible without testing, which leads to bugs piling up. Furthermore, manual testing is just slow.

Being able to speed up the testing rate, and have something else do the testing for you, eliminates these problems and lets you focus on writing code rather than typing inputs and checking outputs.

Web testing

How exactly do you test a website? While there is no replacement (yet) for a human when it comes to judging look and feel, you can rely on software to get the functional parts tested. Software such as Selenium have a built-in browser that is able to request webpages and check the webpage code to see if what is supposed to be there is actually there.

For example, after I perform a login, I should be in my Home screen, the main section should display a welcome, my avatar should be in the top right hand corner, etc. All of this can be tested for in the code.

Test suites and automated testing

So after we have a bunch of tests, what happens? Write code, run the test suite (a collection of tests), check the result, and commit the code if the test passes. What happens when we have over 200 tests? Won’t that take a while?

Yeah it would, but why do you need to test parts of the app that haven’t changed? You can set up your testing so that the basic bits of code (units conversion, time conversion, data cleaning, …) have their own tests. Then the next layer, the main code (data management, login, setup, etc) each have their own tests. The complete app then has more tests. Each layer tests for bugs specific to its level.

Now you can just run the relevant tests when you change one small part of your code. Or not; you could set your tests up so that once you try to commit, the testing software detects which parts of the code has changed, runs tests for those parts, and only lets your commit through if it passes all the tests.

Issue summary: Testing is the only way to know your app really works. Tests can be set up for the different parts of your app, from the basic building blocks to the main code and finally even the interface, including web pages.

Yeah, almost nobody liked tests and exams in school, I know. But in software, testing is the only way to know your app really works, and not just because you say/think it does.

Tests are one of those things that beginning programmers tend to neglect until they learn their first hard lessons. I can see there are many ways to apply it outside of programming, particularly in areas where lots of vetting is needed—vetting really is just human testing, isn’t it? When vetting relies on clear rules and has an outcome that you know in advance, you can think of it as a test. Wouldn’t it be nice if parts of that could be automated, so that you can focus on applying human judgement?

Last issue, I realised that the issues were taking me 1.5 hrs to write, which I think means I am cramming too much content. I promised to keep each issue short and digestible; they are now back to being 1 hour of writing. Again, if you do actually prefer longer issues, drop me an email and let me know!

What I’ll be covering next

Next issue: Multiplayer git, and how git stops people from wanting to kill each other

Sometime in the future: What is:

booting up? [Issue 15]
a specification? [Issue 6,8]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
What are those ‘\r\n’s in the HTTP request packet [Issue 12,17]?

Issue 19: Version control and git

2019-04-20T08:00:00+08:00

Have you ever worked on a large document with a team of people before? If you have, did it go well? Did people get along and not get into conflict with each other over conflicting ideas or changes? If it did, please drop me an email and let me know! In my experience it is so frustrating and I have difficulty seeing how it can be done without some kind of standardised, automated process for keeping everyone updated.

In the past, when documents were still mostly managed manually (before the Internet happened), important documents used to be in a file for everyone to reference. Maybe there were multiple copies for VIPs. Whenever that document got updated, it had to have a change log attached to say what had changed and when. You still see this sometimes today, in some reference documents, and in software updates. The idea behind being able to see the history of a document—a record of how it has changed over time—is an idea that has survived and been passed down to each generation of makers. We often refer to such inherited ideas as wisdom, and the modern form of this idea is known as version control.

Version control in software today

Some examples you may or may not be familiar with:

Google Docs/Sheets/Slides has a line beside the document title, “All changes saved in Drive”. That is a link that shows you the version history (also known as change history) of the document.
Microsoft Word has something similar.
Dropbox keeps a file version history (for 30 days on the free plan) so that you can recover deleted files, or restore a file to an earlier state. Each time you change a file inside Dropbox but keep the filename the same, the file version history is updated.
Wikipedia keeps a change history of every article, as well as the discussion page for that article. Here’s the change history of the Wikipedia page on Git.

This is version control simplified for the everyday person, and if you were already aware of these features, you might already have benefited from being able to roll back a document you accidentally overwrote, or undeleting something you thought you no longer needed.

But what about software, which often consists of more than one file? The addition of a significant feature may involve changes in more than one file, and those changes must be considered together to make any sense of it. How do programmers do that?

Git: modern version control

The modern-day tool which (almost) all developers have standardised on is called git, and it was made in 2005. Its name is Git, and it was written by Linus Torvalds, the same man who first began working on the Linux kernel.

Git tracks changes across multiple files. For example, if you added a new About page to your project, it may involve adding a new file about.html (the actual content), updating app.py (to create a new route so people can access the URL), and adding a new line to the changelog (so other members of your team have an easy-to-read changelog). You inform the git server (we covered clients and servers in Issue 7) that you have changed those 3 files by creating a commit. Git can prepare those commits for you, and they look like this:

A git commit on Github. Yup, this newsletter is also a project on Github.

The screenshot above makes it look fancy, but a commit is really just a simple text file that records the changes of the commit.

    Issue 014 updates

--- a/season02/issue014/issue014.md
+++ b/season02/issue014/issue014.md
@@ -57,7 +57,7 @@ Beyond writing apps, code is also written for many other purposes. I outline som
 Yes, code can be written to process data! This is one of the coolest uses of software, because manually searching through data and performing calculations is tedious, and computers can do it so much faster.

 **System administration**
-A sysadmin’s job is to make sure a server, or group of servers, remain up and running. I might detail the kinds of trouble a server can get up to once I have enough stories to tell, but I think [Rachel](https://rachelbythebay.com/w/) tells much more interesting stories about those, albeit with lots of technical detail.
+A sysadmin’s job is to make sure a server, or group of servers, remain up and running. I might detail the kinds of trouble a server can get up to once I have enough stories to tell, but I think [Rachel tells much more interesting stories](https://rachelbythebay.com/w/) about those, albeit with lots of technical detail.

 A sysadmin might have to write code to automate processes, to troubleshoot, to create new users, to manage files, and for many other reasons. They work primarily in the command line, because most system administration tools are written as commandline tools, although sometimes a user interface might be created to make a popular tool easier to use.

This is a short snippet of a commit. It records some text before and after the change so it’s easy to place the change, and then records both the old version (prefixed with a -) and the new version (prefixed with a +).

Version control systems

So … if I want to see how the code looked at a particular point in time, I have to go through all the commits and mentally imagine the old version in my head?

Well, no. That’s what we have computers for :) Git (and any decent version control system really) can handle that for you to show you the code before/after the commit(s). It can also merge commits, roll back the code to a particular point in time (i.e. undo commits), and more.

If you’ve ever thought “hmm, when did this part change? I don’t remember it looking like this …”, a change history, coupled with the right tools, makes it easy for you to know when that happened. But that means setting up a VCS (such as git), which requires maintenance.

If you don’t want your code being online, git can be installed on your own computer, but the far more popular way that developers use it is in the cloud. And the most popular platform for code today is Github. It’s so popular that Microsoft, realising just how much of their code is on Github, decided to buy Github.

When you create a Github account, you get a repository that you can put your code in. This repository is managed by a git server that you can push commits to, and pull changes from. You get complete change history for your code that is uploaded (here’s the change history for this newsletter). You don’t have the same level of control you would over a git server that you run yourself, but it saves you much more time and hassle.

Using a VCS for non-code purposes

Your eyes are probably glazing over at this point—just another techie gushing about one particular technology, right? Git may seem really cool for code, but it can’t be used for anything else … well actually it can!

People have tried to use git for writing academic papers. Here’s a workflow for writing papers in LaTeX. Here’s another one. Writers use it too. So can designers.

The cool part is not that they are all using git. Git works, but primarily for text-based mediums. But increasingly, people are seeing the value of a change history, and are trying to apply it in ways that make sense. Designers certainly are catching on. Versions and Abstract are two platforms that try to apply the concept in a way that makes sense for designers.

Issue summary: A version control system (VCS) tracks changes to documents. Git is a version control system for source code. Keeping a change history enables a VCS to roll back code in a previous point in time. A change history can also be used in areas other than coding.

Phew! I thought this was going to be short, but I was sorely wrong. In fact, I’m not completely done with the issue on version control yet—I realised as I was writing that it would make more sense to talk about testing first.

This is the first big idea I’ve introduced this season: version control. Knowing about version control and not being able to introduce it into workflows that would really benefit from it is frustrating. If you use Google Docs, Dropbox, or anything that has simplified version control, and you’ve asked yourself questions that can actually be resolved by checking a change history, do try it out and get familiar with the idea!

How can version control improve the way you work? If you’ve got an idea for applying version control where no version control has boldly gone before, hit reply and let me know :)

To wrap up, I realised that this issue took me 1.5 hrs to write. That’s not what I promised when I started this newsletter, that each issue should be short and digestible and possible for me to write in an hour. I’ll look at how to further split up future issues to keep to this promise. If you do actually prefer longer issues, drop me an email and let me know!

What I’ll be covering next

Next few issues: Testing, and how git stops people from wanting to kill each other

Sometime in the future: What is:

booting up? [Issue 15]
a specification? [Issue 6,8]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
What are those ‘\r\n’s in the HTTP request packet [Issue 12,17]?

Issue 18: Frameworks

2019-04-13T08:00:00+08:00

Last issue, I introduced libraries, which are basically magic spells. You call them, they do advanced magic in the background, and return magical results to your code. Or they simplify complicated code, wrapping them up into functions that are easy to call.

All that is cool, but even with that magical aid, working on some projects can be complicated in different ways. For example, if you are working on a web app (or website), you can end up with folders everywhere, and all kinds of HTML and CSS and javascript files.

Web app folder structure example

In a simple app, you might start off with files like this:

project
|   app.py
|   config.py
|   error.log
|   README.md
|
└───templates
|   |   error.html
|   |   main.html
|   |   settings.html
|   |   index.html
|   |   login.html
|
└───static
|   |   styles.css
|   |   app.js

You have app.py which is the web app’s main logic, config.py which contains its settings, an error log you can check to figure out what’s wrong, and a README file for any new members of the team.

Then you have the template files, which contain the code that determines how the page looks. You have your basic splash page (index.html), the login page, the main app view, the user settings view, and a simple error view. These templates have sections that the app can fill in with the relevant information once it is retrieved from a backend.

Finally, you have files which are used in exactly the same way by each page. styles.css describes the colours, margins and styling of all the page elements to make them look consistent, while app.js contains javascript code that is meant to be run in the user’s browser (for animations, simple utilities, and so on).

Web app features

Of course, the above files do not just sit around looking pretty. They contain code that does things. app.py will need code that takes a request and figures out what do do with it. If a client sends a request for /api/annotations/bjPc2xyLTS6Gcu2WW2xtlg, app.py needs to know how to digest that request, pull the relevant information from the backend of database, filter it into a presentable format, and then send it back as a response to the client.

A professional web app developer is likely going to be writing similar code, over and over again, different content but similar logic for different customers … surely there’s a more efficient way to do this?

Web frameworks

If you are at all concerned with efficiency and productivity at work, you would no doubt have templates prepared for all kinds of situations: boilerplate email replies, forms, documents, all prepared and just waiting for some information to be filled in before they are ready.

It’s the same with app development! If a developer specialises in, say, retail apps, it takes some work but they can definitely put together a customisable set of files for any client that needs the usual online commerce setup.

A web framework makes all this easier by providing many of these standard pieces of code, and allowing a developer to just focus on the core parts of the app that change from project to project.

Web frameworks tend to be language-specific

Since web frameworks involve code, it should surprise you little that they tend to be written for a specific programming language. Some examples:

Python web frameworks: Django, Flask, Bottle, …
Ruby web frameworks: Hanami, Sinatra, Scorched, …
NodeJS web frameworks: Express.js, Sails.js, Meteor, …

Whichever one you pick, it should make your job easier than doing everything from scratch.

Issue summary: While libraries make it easier to do the same thing in a programming language, a framework makes it easier to make a particular kind of app. Like libraries, frameworks are usually specific to a particular programming language, and can’t be used in another programming language.

I have greatly simplified the idea of frameworks here, likely to the consternation of developers everywhere. The only thing I will say in my defense is: at least your potential customers now have a better idea of what you are doing for them!

Speaking of developers, we’re now getting deeper into the kind of work that developers do. Some work alone, but they usually work in teams. How does a development team build projects like this without getting in each others’ way, overwriting each others’ changes, and drawing boundary lines on the floor with marker? What do they use to ensure that everyone knows what is going on?

Next issue is going to be really interesting!

What I’ll be covering next

Next issue: How do developers avoid code conflict? On version control and more

Sometime in the future: What is:

booting up? [Issue 15]
a specification? [Issue 6,8]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
What are those ‘\r\n’s in the HTTP request packet [Issue 12,17]?

Issue 17: Libraries

2019-04-06T08:00:00+08:00

Welcome back! Last issue must have been exhausting to read, and I want this issue to be more readily digestible. So there’re only going to be two examples. Promise.

A shell script works really well when you just need to do things with folders, and run programs to work on files. But anything more advanced, such as making network requests, quickly gets tricky. Usually this is where the programmers start to whip out their favourite programming language. But …

Low-level Sockets vs High-level Requests

In python, crafting an HTTP request from scratch looks like:

import socket

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)                 
s.connect(("www.hypothes.is" , 80))
s.sendall("GET /api/annotations/GqDxiuhpEeiMa6f5UN7ttA HTTP/1.1\r\n\r\n")
print s.recv(4096)
s.close

5 lines of code every time I want to get an annotation from Hypothes.is? That’s going to add up fast … I could wrap that code up into a function, and then it would be easier to just call that function in one line instead of 5.

Carry that thought to its logical conclusion, and you get a library.

Creating an HTTP request with the Requests library be like:

import requests

r = requests.get('https://www.hypothes.is/api/annotations/GqDxiuhpEeiMa6f5UN7ttA')

A good library makes complicated things simpler, so you can focus on what you need to do.

There’s a library for it

Apple’s App Store slogan, “There’s an app for that”, is so famous that they actually trademarked it. So it is with programmers. For almost everything you want to do in the most commonly used programming languages, someone has created a library for it. If they haven’t, then it’s up to you to make one.

But sometimes, some projects are just way beyond your current level of knowledge. If you wanted to dabble in advanced machine learning(i.e. AI), one option is to go for a PhD … or tinker with Google’s TensorFlow platform.

How do you do that in Python? You need the TensorFlow Python library. And then you probably need some tutorials to get started.

If there isn’t a library available … you might as well get started on that PhD.

But TensorFlow only comes with a Python library. What if you don’t use Python, but use Ruby instead? Then your best hope is to petition Google for a Ruby library, or hope someone else ports it for you.

Either way, it’s plain to see that without these advanced libraries, it would take much longer to get started.

Issue summary: Libraries make it easier to do the same thing in a programming language, or enable advanced functionality that wasn’t available previously. Libraries are usually specific to a particular programming language, and can’t be used in another programming language.

Okay, so it turned out I only wrote one example, and linked extensively to another one. This didn’t need to be complicated, and aside from the code I hope I didn’t lose you. Was this issue too difficult to grasp, or am I moving too fast? Drop me an email, I can adjust :)

What I’ll be covering next

Next issue: Frameworks

Sometime in the future: What is:

booting up? [Issue 15]
a specification? [Issue 6,8]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
What are those ‘\r\n’s in the HTTP request packet [Issue 12,17]?

Yep, if you’re sharp-eyed you might have noticed the ‘\r\n’s in the socket example ;) All in good time.

Issue 16: Shell scripts and automation

2019-03-30T08:00:00+08:00

So you just got back from a trip with lots of photos in your camera. You take the SD card out from the camera, put it in your laptop, open up the photos … and realise they all have the wrong white balance set.

Or if you’re like me, you spend the better part of an afternoon using your camera to ‘scan’ a textbook for later reference. That means you now have over 300 photos to crop separately (left and right pages), adjust levels (to make the image more like it was done with a scanner), and do other minor corrections.

You know your way around Photoshop (or GIMP), but no way you are going to do that by hand for a week or so—we’ve all got lives to get on with! But you spend half an hour googling, and can’t find any tools that can help you to get the job done quickly. (Okay, there’s XnView, but let’s just pretend that doesn’t exist for the purpose of this issue.)

You have just discovered the need for computerised batch processing.

Converting an image in the command line

Once you have found a command line program to help you edit an image (such as imagemagick), you can do that scanning adjustment with a command like:

convert DCIM0100.JPG -colorspace LinearGray -crop 1260x1800+160+240 -level 64,160 page001.png

We just covered command lines in the previous issue, and you don’t need to be able to reproduce this command, but I hope you are seeing the patterns mentioned in the last issue here. The program convert is being called with 3 options:

-colorspace LinearGray converts the image to grayscale (you can probably just set this in the camera beforehand, but no harm being sure)
-crop 1260x1800+160+240 crops the image to a size of 1260x1800 starting 160 pixels from the left and 240 pixels from the bottom.
level 64,160 basically increases the contrast of the image, making the text blacker, and the page whiter.

The documentation for Imagemagick’s convert command is available online, and I will get round to discussing image formats in a future season. This is as far as we’ll talk about images for now.

Great. Here’s a command that can open DCIM0100.JPG, do those operations to the image, and save it to the file page001.png. (I pre-pad numbers with 0 because many file browsers are still terrible at sorting 2 before 10, but can sort 02 before 10 just fine.)

Do I have to copy-paste this command 300 times into the terminal while editing the filename each time? Of course not, that’s why we have computers in the first place.

Making a batch file/shell script

A batch file (Windows) or shell script (Linux/Mac) is simply a file with a list of commands that the terminal will execute when the file/script is invoked. They can be one line, ten lines, a hundred lines, or more. They can be simple, or incredibly complex. They can simply start an application with a different set of options, or help you to update all the packages on your computer periodically and handle simple errors. If you opened the three links from the last issue, you would have seen some examples of what these scripts can do.

So how would I put that command into a shell script so that it can do the above for all the images in a folder?

There are a number of things we need to figure out first:

How do we want to number the final output files? I’m just going to go with page001.png, page002.png, etc to keep things simple, but you can customise the filename in all kinds of ways.
The script needs a way to get the names of the original images as well.

My go-to terminal is bash, one of the default terminals in Linux. You may sometimes see terminals referred to as CLIs (command line interfaces), as opposed to GUIs (graphical user interfaces), but this can be done in Windows Command Prompt, Powershell, zsh, Mac Terminal, and just about any CLI that supports scripts. Each CLI has its own syntax and format of commands that will need to be learnt, but in this day and age it is not difficult to find help for that online.

In bash, this script would look like:

n=1
for file in *.JPG
do
  outputname=page$(printf %03d $n).png
  convert $file -colorspace LinearGray -crop 1260x1800+160+240 -level 64,160 $outputname.png
  n = n+1
done

I promised not to make this newsletter technical, and I said this wouldn’t be a howto, but this is as far as I can go without programming. I’ll step you through what each part does:

Simple Bash programming

for file in *.JPG
do
  [command]
done

Bash checks the current folder for files ending with .JPG. (Why is it written as *.JPG? We’ll revisit that in a future issue.) For each file found, it passes the filename to [command] and carries out the command. This command is able to access the filename using the name $file.

n=1
...
n = n+1

This is how I am creating a number counter that I can use to number the final output files sequentially. Mathematically this makes no sense (how can n be equal to n+1?), but in programming, n = n+1 means “add 1 to the value of n, and assign the new value to n.”

outputname=page$(printf %03d $n).png

printf is a program that creates lines of text, and it can help format text in all kinds of useful ways. %03d describes the format I want: a 3-digit decimal number, pre-padded with 0s. printf %03d $n changes 1 to 001, and will do this for each value of n. So page$(printf %03d $n).png helps me craft the filename page001.png and I can refer to it using the name $outputname.

Putting everything together …

n=1
for file in *.JPG
do
  outputname=page$(printf %03d $n).png
  convert $file -colorspace LinearGray -crop 1260x1800+160+240 -level 64,160 $outputname.png
  n = n+1
done

Name a counter, $n, and set its value as 1. For each $file ending with .JPG, calculate the result of printf %03d $n, insert it into page$(printf %03d $n).png, and name the result as $outputname. Carry out the conversion on $file, and save the result to $outputname. Increase the value of $n by 1.

Scripting: talking to a really really really dumb thing

Was that exhausting to read? I bet. If you had to ask a simpleton to do something, you know you need to be explicit: “Go to the market, stall number #01-23, and buy two packs of chili paste, the orange kind with the circular symbol, 200g. Also buy 100g of dried shrimp. Put the chili paste packs into one plastic bag, and the shrimp in another plastic bag. If they can’t do that for any reason, write down what the stall owner says, and call me. If the stall is closed, come home.”

Programming is like that. A good programmer can take a difficult task, break it down into steps so simple that even a computer can understand, and figure out how to write the repetitive parts to make them easier to say.

Issue summary: Everybody has a simpleton in their pocket, and maybe one at home on the desk. These simpletons are able to run sets of instructions that they are carrying. You can give them more sets of instructions, often obtained through a Store. Some of the really good instruction sets will cost some money, though. And almost none of them will get the simpleton to do exactly what you want.

The only way to do that is to learn how to write the instructions yourself. The ability to do so is what we call Programming.

A shell script is just a set of commands that a simpleton can carry out.

Phew, this issue sure took a while! I knew I was going to have to write some code, and explain it for laypeople, and it wasn’t going to be enjoyable. I’m glad it’s over and I hope I won’t have to do that again in the later issues of this season. I’m still figuring out how to do it more simply.

Do let me know how you find this format? I don’t explain things line by line, but rather part by part, as that makes more sense to me. I also try to start from simple commands, and build them up into more complex cases. But doing that will result in a tutorial-length issue, so I compress where I can. If there’s anything I can do to make it easier for you, drop me an email :)

We took a short detour from looking at what sysadmins do, to see what a shell script is like. Lots of things that happen in a computer can be automated this way. But it has its limitations. Next issue, we’ll look at how programmers help each other make their jobs easier by writing programming libraries.

What I’ll be covering next

Next two issues: Libraries, Frameworks

Sometime in the future: What is:

booting up? [Issue 15]
a specification? [Issue 6,8]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
What are those ‘\r\n’s in the HTTP request packet [Issue 12]?

Issue 15: Sysadmins and the command line

2019-03-23T08:00:00+08:00

If something’s not working correctly in your operating system, someone has to dig into the settings and configuration, figure out what’s wrong, and resolve the issue. If you have ever been in those shoes, you have been basically doing a sysadmin’s job. (Okay, put away those pitchforks you sysadmin purists and hear me out …)

Every system you have ever interacted with needs to run on a computer, usually hidden away somewhere: point-of-sale systems, advertising screens, email servers, print servers, web servers, … and these systems can start going awry too. Sysadmins, experienced or otherwise, are the ones who fix them. If this system has a screen and a place to plug in a keyboard, fine. But what if it doesn’t?

Before graphical interfaces and the mouse cursor came about, there was a primitive (but no less powerful) interface: the command line.

The language of the command line

A computer is just a box of chips waiting to do your bidding, and the command line is one way to give it commands (get the name now?) But of course, that means you’ll need to learn its language.

First, some names/terms. You may see the command line also referred to as a terminal (Mac/Linux), command prompt (Windows), or shell (also Windows). The differences are technical, they all refer to that window with the pointer waiting for you to enter your commands.

What does a command look like? It can look as simple as:

touch image.iso

or as complex as:

ffmpeg -i input -c:v libx264 -preset slow -crf 22 -x264-params keyint=123:min-keyint=20 -c:a copy output.mkv

If for whatever reason you have to try to interpret what a command is doing, here’s the one thing all commands have in common: the first word is always the name of a program. The first line is calling the touch program, the second line is calling the ffmpeg program.

The second thing is that any word starting with -, --, or / (used by Windows) denote options. Like, you know when you go into an app’s settings and change stuff? This is how command lines do the same thing. If you don’t give any options to the command line program, it will use its defaults, just like any other app.

All other words which come after the first word are just options. So let’s try to interpret those two command lines:

touch image.iso → Tell the program touch to do its thing to the file image.iso

ffmpeg → Tell the program ffmpeg …
-i input → to use the input file input …
-c:v libx264 → and use the video codec library called x264 …
-preset slow → and the “slow” preset …
-crf 22 → and use constant rate factor 22 …
-x264-params keyint=123:min-keyint=20 -c:a copy output.mkv → okay nevermind, I think you get the idea 😅

It took me years of gradual exposure to various command line programs and their options to know what they can do, and I don’t expect (m)any of you to be heading down that path, but if you ever encounter a command line instruction, I hope it strikes less fear in you than it used to, because now you know how to read it!

Why do people use command lines?

Fallback mechanisms

Let me use an analogy:

When you go camping, you sleep in/on a sleeping bag (or air mattress, or hammock, or …) and not a bed, right? When the car breaks down and you don’t have a spare, you cycle or walk. When your school computer blocks you from using filesharing apps, you resort to emailing yourself.

When your usual powerful methods don’t work, you fall back to more primitive methods.

The command line is a fallback for graphical interfaces when they crash. If Windows encounters a problem while starting up (this process is known as booting up, we’ll get to this in a future series on hardware), it usually tells you to get a technician to look at it. But in Linux, when the computer encounters a bootup problem, it drops you to a command line so you can at least try to fix the problem. Any time the graphical interface in Linux starts getting glitchy, you can always drop to a terminal (another name for command line, remember?) to figure out what’s wrong. If even the terminal is glitching up, something really bad is happening … (probably a video driver issue, but we haven’t gotten to troubleshooting yet).

Remote work

The really cool thing about a lot of developer work (including sysadmin work) is that you’re just manipulating text on a screen (after manipulating a lot of ideas in your head first). This is something you can do remotely, over the Internet! There is no reason you need to be physically present where the actual machine is.

Sysadmins of large systems often have to manage multiple servers that may be in another part of the building, or even spread across the world: remote management. A manager works with remote workers by telling them what to do over email/video call, a sysadmin tells her remote servers what to by sending them command line instructions, usually over an encrypted line, because you wouldn’t want these instructions intercepted by a malicious person.

Why don’t they do it through a graphical interface?

I don’t have any single primary reason, but if you ask developers why they prefer command lines, they’ll probably tell you something along the lines of:

“Who’s going to make that graphical interface? No one, because it’s more work and we’d prefer command line anyway.”
“I can type commands faster than I can click a mouse.”
“Sending graphics over the internet takes a few thousand times the bandwidth it takes to send text. The lag is so bad, I’d rather use text commands.”
“What’s wrong with command lines?”
“You can save commands into a script file and run them. That makes our work insanely easier. No graphical interface needed!”

I guess it’s something that just has to be discovered through experience ;)

Issue summary: A command line is a way of giving commands to the computer in the form of text. An instruction consists of the name of the program to be run, and the options that it needs to use. Command lines provide a fallback mechanism when graphical interfaces break down, and are a much more remote-friendly interface.

Phew, slightly over 1100 words! So this issue is about 40% shorter than the previous :) I know movies and dramas like to make hacker work look all mysterious-like and one popular way is to randomly show some hoodie guy hunched over a command line. May this issue make such things less scary and encourage more people to dabble in it, even if only a little bit.

I intentionally left the juciest bit for the last: “You can save commands into a script file and run them.” Amazingly few people know this! I suspect a big reason people get so jazzed up over AI is an ignorance that many of the use cases they imagine for AI can already be handled by sufficiently complex script files.

I’ll save my fingers the typing and let the Internet provide the stories of automation:

A programmer wrote scripts to secretly automate a lot of his job — and email his wife and make a latte (Business Insider)
Here’s an extreme (and not very nice) example, but look at all the things you can do with just the command line! (But do nicer things of course.)
What are some of the most common things you can automate with python? Looking for ideas (Reddit)
More of what you might expect to be able to do in daily life. If you’ve been procrastinating on picking up Python because you can’t think of a project to do, try this thread for ideas.
Automation is fun. Do you have any stories? (Reddit)
A bragging-rights thread, I put this here because the original poster’s story really illustrates why sysadmins love command lines. Which is what this issue is about, right?

I want to talk a little about how this automation is done in the next issue. Again, not a howto, just an explanation of what goes on. And hopefully it helps you see that you don’t actually need AI for many of the problems you face.

What I’ll be covering next

Next few issues: Shell scripts and automation, Libraries, Frameworks

Sometime in the future: What is:

booting up? [Issue 15]
a specification? [Issue 6,8]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
What are those ‘\r\n’s in the HTTP request packet [Issue 12]?

(I know I’m taking a while to get to these. I’ll touch on specifications later in this series, but the rest will have to wait until we return to networking and the Internet again, probably.)

Issue 14: What do developers do?

2019-03-16T08:00:00+08:00

Last season, I used an app I was making to get into the differences between HTTP and HTTPS. It wrapped things up at a point I’m pretty happy with, left some questions unanswered (for now), and got me thinking what to cover next. I’ve thought about how best to cover the remaining topics, and I could well continue in the same thread, going on to say more (relevant and untechnical) things about networking and its technicalities, but from experience it gets dry and dreary very quickly.

One question I get pretty often is: “I’m interested in Computing, but where should I start? What should I do?” The first thing I tell them is not to learn programming, but to understand what developers do, because this is not obvious knowledge. So I’m going to try to build up your picture of computing first, before we get back to networking.

Season 2 is going to be about the work of developers: what they do, a bit of how they do it, and the cool things they get to play with. Yup, there are 13 issues worth of things to talk about!

Let’s start with the obvious: developers write code. They write code that runs all the services you use. The not-so-obvious part: not all code is the same. Just as the same letters can be used in different languages, the same keyboard can type code in different languages. The code serves different purposes.

Because code is so complex, and very very few people can really be proficient at every aspect of it, developers tend to specialise. And when they specialise, they label their specialties to make it easier for employers to understand what they do. I’d like to start us all off on the same page and get these terms out in this issue.

A primer to applications

When you open an app on your phone or laptop, what you see is not all there is. It’s like a customer service desk: there’s the front desk, the back-office, the storeroom, and behind all of that is the sprawling mass of how the company delivers value to you, its customer.

Likewise, what you see when the app opens is only the frontend. You type data into the interface. The application processes some of that data, and sends some of it to the backend, where it is processed further, or used to look up information, which gets returned to the application.

Frontend, backend, and application

The meat of frontend development is closely linked to user design. Frontend developers work closely with designers to understand how they are supposed to make the app look and behave. While default templates can make a webpage look slick and professional, it does not help an app differentiate itself. So frontend developers often code custom animations, button behaviour, and figure out ways to make an app do things out of the ordinary, among other things.

Frontend developers write code in: HTML, CSS, Javascript, PHP, ASP.net, and more

Backend development is hidden from the user, and so is primarily concerned with speed and stability. Many backend applications-server, database, custom software-run concurrently in the same system and must communicate to each other. Figuring out how to make everything work together harmoniously and bug-free is a full-time job, and the work of the backend developer.

Backend developers write code in: Java, C/C++, Ruby, Perl, Python, PHP, Javascript, Go, Rust, and more

Anytime there are two (or more) kinds of people, there is the potential for them to split into two tribes, each blaming the other for some way the system has gone wrong. This is where you need developers who are entrenched in both worlds, and are able to understand how the frontend and backend interact with each other to produce the final outcome. Developers that work in both frontend and backend in the course of their work are called full-stack developers.

A “stack” is a way of thinking about applications that work together; as you go from frontend deeper into the backend, you are going down the stack. Complex applications can have more than 20 different services running concurrently, and a full-stack developer needs the skills to diagnose what is happening when something goes wrong somewhere in the system.

A web development stack, just one of many. From Eden Vital on Medium.

Frontend, backend, and full-stack describe which part of the code a developer is working on. But the kind of application a development team is working on can vary widely depending on where it is being used.

Applications

Web applications are primarily run in a browser, even if that browser is embedded in a mobile app or installed device. These applications primarily interface with a web browser and the internet. They are device-agnostic, which means they can technically run in any device that is able to connect to the internet and run a browser.

Laptop/desktop applications are run in an operating system (Windows, MacOS, Linux). They are typically OS-specific, although with the right tools a developemnt team can develop versions for multiple operating systems without adding too much complexity to the code.

Mobile applications have a bit less leeway as compared to laptop applications, due to the sandboxed nature of smartphones and other mobile platforms. That means mobile apps have fewer privileges and do not have access to many parts of your smartphone (which is why they keep asking you for permissions).

Embedded applications run on low-power devices, typically routers, home gadgets, and other devices that require access to the internet. Because of their limited speed, embedded developers have to be skilled at optimisation and low-level coding (explained further down in this issue).

Game engines run on … well, any of the first three really. A game engine helps game developers to not have to worry about rendering 3D graphics, setting up a quest system, and inventory system, character behaviour, scripted events, cutscenes, …

Enterprise applications are applications which are developed primarily for use by large corporations. It may be adapted from consumer apps (e.g. GMail), or from scratch (e.g. SAP).

Specialties

Beyond writing apps, code is also written for many other purposes. I outline some of them below, although people who write code for these purposes might not actually have “developer” in their job title.

Data Science Yes, code can be written to process data! This is one of the coolest uses of software, because manually searching through data and performing calculations is tedious, and computers can do it so much faster.

System administration A sysadmin’s job is to make sure a server, or group of servers, remain up and running. I might detail the kinds of trouble a server can get up to once I have enough stories to tell, but I think Rachel tells much more interesting stories about those, albeit with lots of technical detail.

A sysadmin might have to write code to automate processes, to troubleshoot, to create new users, to manage files, and for many other reasons. They work primarily in the command line, because most system administration tools are written as commandline tools, although sometimes a user interface might be created to make a popular tool easier to use.

Graphics Graphics engineers specialise in the art of rendering: turning data about a 3D object into an image. This work is very much mathematical, and needs to be performed quickly, because any lag in visual output is easily noticed.

Machine Learning What is currently also referred to as “AI”. Machine learning researchers try to come up with ways to spot patterns in data, and use those patterns for various purposes, such as classifying data, generating realistic images, predicting behaviour, or just for fun, really.

Cybersecurity A server is designed to give administrators and managers more privileges, and give users fewer privileges. Cybersecurity researchers figure out ways to make a server reveal more data than it should, to enable a user to do things they shouldn’t, or to downright gain access to a server they shouldn’t have access to. These are known as exploits. They publish information about these exploits, in the hope that the software developers responsible for those software will fix those bugs and keep people safe.

Levels of programming

Some developers like to work “close to the metal”; they write low-level code. They often have a deep understanding of the hardware they are working on and the algorithms that are being used, and are able to write code that is lean and efficient. They may be able to speed up a function by a few thousand times by rewriting it in low-level code! But this is tedious work to maintain, and can’t be done for an entire software project, so such code is often employed in strategic areas only.

To manage low-level code, developers write compilers, interpreters, and debuggers that compile or interpret high-level instructions and break them down into lower-level instructions that get run instead. This makes it possible to write high-level code like open('file.txt','r') and be able to open a text file without having to know all the low-level code required to make it happen. High-level code is what most people learn when they learn programming, because it enables them to do cool things quickly without needing to understand all the intricate details; high-level code abstracts away the detail of low-level code and handle it for the developer.

Low-level code and high-level code are not a classification; they are a sliding scale. You can place different programming languages on this scale to indicate what level of abstraction they work at.

Some programming languages, arranged from low-level (left) to high-level (right). From codecommit.com

Issue summary: Developers can specialise in frontend, backend, or full-stack development. They write applications for more purposes than I can really count. Not all code is written for an application; some code is written for specialised purposes. And code can be written at different levels of abstraction. High-level code is easier to understand and write because it hides the details of how a computer operates, while low-level code is detailed and able to squeeze the most performance from a computer, but difficult to understand and write.

Phew! This was a long issue, and I apologise for that. Subsequently I am not going to be covering such a broad scope, although the tools I talk about could potentially be used in any of the above areas.

In the next issue, I want to talk about a tool that laypeople almost never encounter: the command line. Because humans have a tendency to believe that what they see is what others see, I think it is important for people to understand that most of computing doesn’t happen with a graphical interface. And although laymansguide will not teach you how to program, it will hopefully help you to stop fearing the command line.

What I’ll be covering next

Next issue: System administration and the command line.

Sometime in the future: What is:

a specification? [Issue 6,8]
a cookie? [Issue 8]
a cache? [Issue 8]
XSS? [Issue 8]
a CDN? [Issue 8]
Unicode? And what does it have to do with emoji? [Issue 8]
What are those ‘\r\n’s in the HTTP request packet [Issue 12]?