Layman's Guide to Computing - Season 08

Issue 104: Storing sensitive data

2021-01-23T08:00:00+08:00

Previously: A race condition happens when threads depend on instructions happening with coincidental timing for success. When instructions are not executed with appropriate timing, one or more threads can get stuck waiting on a response that never comes.

To wrap up this season on how apps work, I’m going to try answering a question I had on my mind as I was still new to computers: where do my secrets get stored? If I don’t want them to be stored, what are my options?

I will answer that from an app’s perspective in this issue.

Why would I want to keep secrets from my users?

You just wrote an app. Your app syncs data to a cloud database (Issue 90)). But the cloud database has many other app developers using it as well—how would it know it is you and not some other malicious hacker? It recognises you via a shared secret: a token or an API key that you can see after you log in to your dashboard on their website.

After your users install the app, every request sent by your app to the cloud database has to be authenticated using this shared secret. That means you are going to have to get this shared secret onto the app somehow. But this has to happen without the user being able to see it or access it, otherwise a savvy user could use that key to gain access to your cloud database.

Storing secrets on the web

The code that is loaded by the user’s browser runs under their control, so putting the shared secret anywhere in that code is a bad idea. Any savvy user who knows how to view the script’s source can potentially find it!

A much safer option is to store the secret with the code that runs on your server. But not in the server’s source code! If you are most developers, you would be using some kind of version control system (Issue 19)) that maintains a copy of your source code and all its changes. If you are using Github or some other public platform for this, you have to be very careful that the shared secret is not visible (or otherwise guessable) just by reading the source code.

For a simple shared secret, such as a short string of characters, app developers usually use environment variables. These are pieces of information that are kept in memory only, accessible by the app, and are set by the operating system whenever the app starts up. The server where you run your code will let you configure the environment variables that your app needs, keeping them out of sight of the users.

Storing secrets in a mobile app

Mobile apps are supported by a host of services provided by the operating system (OS), typically managed by Google or Apple. They each offer a way for you to store a shared secret with the OS. Your app can use this shared secret from the OS to encrypt information so that other apps are not able to access it. When your app starts up, it requests the secret from the OS, and uses it to decrypt the secret again.

Storing secrets in a laptop app

If you are developing a laptop app that does not rely on a connection to your server (i.e. a “standalone” app), your options are somewhat more limited. Since all your app code and resources will be in the user’s machine and thus accessible to the user, your best bet is to find some way to obfuscate it and hope no one finds it easily.

This is one reason why so many apps require an online connection: it is much easier to hide secrets on a machine you own and control! With a server connection, you can require the app to retrieve the secret from the server, and delete the temporary copy of the secret after use.

Storing passwords

If your users log in with an email and password (which is almost every online service ever), you don‘t actually store their passwords; that is terrible security practice, even if you do it in a database! A nifty piece of software technology, known as a hash function, takes that password (regardless of length) and turns it into a unique¹ hash with a fixed number of characters.

Examples of hashes:

661c425549bc70b98e908325b8c64f82
056cd6eb540ace37e64572c64c778d45
239b1ddbb45caf82408cb89f13816185

What you do, then, is to store the password hash instead of the password. When a user sends a username and password, you hash their attempted password, compare it with the stored hash, and see if they are the same.

The hashes are designed to be difficult to reverse. The state-of-the-art algorithm used today can generate hashes that would take millions of years to reverse using hardware currently available. But there are techniques that can reverse hashes of some older algorithms in as little as 30 minutes, so if you are a developer, please find out which one to use!

Issue summary:

Shared secrets allow secured access to resources, such as databases or other services. These shared secrets are typically kept on a server controlled by the app developer. For mobile apps, they are usually stored with the operating system, inaccessible to other apps.

Phew, we had enough issues here to cover the main parts. And I managed to answer one of the sometime-in-the-future questions! Actually, I also answered another one on software installation earlier, in issues 99) and 100), so I’m going to go ahead and strike it off.

What I’ll be covering next

Next issue: [LMG S9] Issue 105: Operating Systems

This wraps up another season of Layman’s Guide on how apps work. Next season, I am going to zoom out and look at the environment that apps operate in: the operating system. Yep, I’m going to tackle the most complex pieces of software ever to be written, and try to explain them in terms that laypeople can understand 😅

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
~~involved in installing a piece of software? [Issue 48]~~
How do apps know where a file starts and ends? [Issue 49]
~~a password hash? [Issue 63]~~
a driver file and why do I need one? [Issue 98]

In practice, there is a very low chance that two different passwords may end up giving the same hash. This technology is still being improved! ↩

Issue 103: Why apps hang even with multiple threads

2021-01-16T08:00:00+08:00

Previously: Applications are assigned a thread by the OS for running a sequence of instructions. The instructions are executed sequentially, and the app cannot proceed if it gets stuck on any instruction.

Multithreading

An app can hang if its sole thread gets stuck. In some cases, an app can be written to make use of multiple threads. This is possible when a computer has more than one processing core, or if an operating system is designed to divide computing time among multiple threads.

Race conditions

Trying to design apps to use multiple threads is hard! Apps running into an error is one thing; a more subtle form of failure is known as a race condition. This happens when the success of two or more tasks depend on near-perfect timing which the threads have little control over.

Main and secondary threads

A common pattern is to have the app’s graphical interface and main code run in its own thread, with any subsidiary tasks (such as opening files) running in a secondary thread. If the task in the secondary thread is taking too long, the main thread can still issue instructions to terminate the secondary thread’s task, and thereby restore order and control.

Sounds fair enough. How might this fail?

Lets take an example: the main thread has a task that involves sending a signal to secondary thread, and then waiting for a response from it. Secondary thread has a task that involves sending a signal to main thread, and then waiting for a response from it. Both tasks complete successfully when they are carried out independently. But what if the main and secondary threads both run those tasks near-simultaneously, before the other thread has a chance to respond? They both get stuck waiting for a response. The app has just hung!

Multiple worker threads

Another pattern is to split the job up into multiple parts, and have multiple threads each take a part of the job. When they have all completed, the completed parts are then stitched back together into the finished result.

But this has its own ways of failing too.

The threads have to coordinate their job status, and often do so by updating a common set of data. Thread 1 might request access to that data to update it. To ensure that the data doesn’t change before it is done, it will usually request a lock (Issue 82)) on it, to prevent other threads from modifying it at the same time.

One way this can fail in practice is if two or more worker threads request a lock simultaneously. They both get a lock, because at the moment their requests are processed, nothing else has locked the resource. But now they can’t proceed to modify the data because it has been locked by another thread that isn’t them.

This situation is known as a deadlock. This and similar situations are just one out of many ways that apps can hang.

Issue summary: A race condition happens when threads depend on instructions happening with coincidental timing for success. When instructions are not executed with appropriate timing, one or more threads can get stuck waiting on a response that never comes.

Yep, multithreaded programming is hard.

What I’ll be covering next

Next issue: [LMG S8] Issue 104: Storing sensitive data

To wrap up this season on apps, I’ll look at one last question: how do apps keep our data secure

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
involved in installing a piece of software? [Issue 48]
How do apps know where a file starts and ends? [Issue 49]
a password hash? [Issue 63]
a driver file and why do I need one? [Issue 98]

Issue 102: Threading

2021-01-09T08:00:00+08:00

Previously: An app crashes when it encounters a situation it can’t handle, or when it attempts to perform an operation that is disallowed by the operating system.

This post is a prelude to talking about app hangs. Hangs are both simple yet complicated to talk about, but there’s a piece of the puzzle that has to come into the picture first. That piece is about how apps work.

In a computer, the operating system (OS) has to coordinate the requested actions of so many different apps. How does it know which action came from which app?

Through a mechanism known as threads.

Threading

When you run an app, the OS creates a separate thread. A thread is a sequence of programmed instructions, like a thread of thought. Or a thread of bureaucracy. The OS completes each instruction in the thread, and if it gets stuck on any single task, it cannot move on.

Sometimes, this is good and necessary, like when you need input from the user (don’t be doing anything else until I tell you what I need!). Other times, it is unnecessary waiting.

What causes threads to get stuck?

Some of us really hate math, but not computers! The math is hardly ever what causes threads to stop.

Like in the workplace, it is often other ~~people~~ devices.

When an app (running in a thread) tries to open a file to read data from it, the operating system has to look up the virtual memory address (Issue 55)), follow it to the hard disk or solid state disk, and then wait for the disk to respond with the data.

And in that moment, lots of things can go wrong.

If the disk is failing, and unable to read the sector where the data resides, it will usually keep attempting to do so. Meanwhile, back in the operating system, the thread is stuck. It cannot move on, because the previous instruction to open the file has not completed. It can’t even decide to abort the currently-running instruction—telling the app to stop is already another instruction which has to wait!

The only thing to do now is wait for the OS to realise that this thread is taking too long to do its thing, and forcibly terminate the thread. This is known as a thread timeout.

Is there any way to work around this? Yep! The termination instruction has to come from a separate thread. This means the app has to run multiple threads.

Issue summary: Applications are assigned a thread by the OS for running a sequence of instructions. The instructions are executed sequentially, and the app cannot proceed if it gets stuck on any instruction.

What I’ll be covering next

Next issue: [LMG S8] Issue 103: Why apps hang even with multiple threads

Processors today already have multiple cores, and many apps can already run on multiple threads. Why do they still hang? I’ll answer this in the next issue :)

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
involved in installing a piece of software? [Issue 48]
How do apps know where a file starts and ends? [Issue 49]
a password hash? [Issue 63]
a driver file and why do I need one? [Issue 98]

Issue 101: Why apps crash

2021-01-02T08:00:00+08:00

Previously: Windows systems categorise data into two types: files, and settings. Files are stored under an appropriate subfolder in C:\, while other storage devices and network locations are stored elsewhere or given their own drive letters. Settings are managed through the Windows Registry, which is stored in C:\Windows\System32\Config\ and C:\Windows\Users\Name\.

Besides general slowness, two of the most frustrating experiences we have with computers is when they crash, and when they hang.

You mean there’s a difference? Sometimes we use the two terms interchangeably, but they are really not the same.

Remember: applications are just a list of computer instructions telling the computer what to do: where to get the data, how to process it, and what to return. When the instructions make perfect sense, everything goes well. But sometimes they don’t.

Crashing

A crash happens when the app receives (a) a response that it does not know how to handle, or (b) is not allowed to carry out.

Unhandled responses

A common error made by many programming newbies (including me) is failing to account for all the ways that things can go wrong. For example, if I am writing a simple app to read a text file and perform some calculations, an obvious step in the app is sending a request to the operating system (OS) to open the text file.

Even that simple step is fraught with many possible failures! The text file may have been locked by another app (which is writing data to the file), or the user running the app might not have permission to open the file (especially if it is in another user’s home directory), or …

Well, a whole bunch of things can go wrong. And when they do, the OS throws an error. If the app does not have any code to handle that error … game over, it cannot proceed and it crashes abruptly.

This is a lot more common than you think, even for experienced programmers, especially when a process that isn’t expected to throw an error actually does it. And sometimes it just can’t be helped: when your computer runs out of memory, and an app requests for more memory but doesn’t get it, and it just cant go on without that memory … it crashes.

Illegal instructions

Memory in the computer is managed by the OS (Issue 65)), which partitions it into different zones. The memory used by OS processes is protected from access by other apps (for your privacy and protection), and memory used by an app cannot be used by another app, unless it is shared memory space.

So when an app sends an instruction requesting to access memory space it does not have authorisation to, or when the OS itself attempts to access an address that it can’t (especially addresses that point to hardware devices) … it crashes. An app crash just brings you back to your desktop, but an OS crash usually leads to the famous Blue Screen Of Death (BSOD).

These days, OSes are better at handling crashes. If the crash occurs in the window management system (the part that lets apps create windows on screen and icons in the taskbar), Windows can often just restart it without restarting or touching the rest of the OS. But if it happens in a critical part that can’t be restarted by itself, then … BSOD :)

Issue summary: An app crashes when it encounters a situation it can’t handle, or when it attempts to perform an operation that is disallowed by the operating system.

Definitely oversimplified for ease of understanding, but I see no point going into the technical details unless a future issue calls for it.

Before going into app hangs, I’ll need to talk about threads first. If you have heard of multithreading before, yep I am going to talk about that next issue!

What I’ll be covering next

Next issue: [LMG S8] Issue 102: Threading

“Many hands make light work” is true for computers as well, and I’ll go into more detail about how a computer uses its many hands to speed up the work it does :) Before that, let’s examine the simple case of an app doing only one thing at a time: the single-threaded app.

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
involved in installing a piece of software? [Issue 48]
How do apps know where a file starts and ends? [Issue 49]
a password hash? [Issue 63]
a driver file and why do I need one? [Issue 98]

Issue 100: Where does all the app data go? A look at Windows systems

2020-12-26T08:00:00+08:00

Previously: MacOS, Linux, and other similar systems treat everything as a file, organised into appropriate subfolders.

Previous issue: Mac- and Linux-like systems. Since the key points are so short, let’s summarise:

On Mac-like systems, the top-level folders are

an /Applications folder for apps
a /Library folder for shared files (see Issue 17), but on Mac this extends to things like sounds, profile pics, colors, …)
a /System folder for, well, you know what.
a /Network folder for accessing resources on the network (such as shared folders)
a /Users folder for accessing user folders and files
a /Drives folder for accessing other storage devices (e.g. USB drives)

Linux systems are similarly divided, but into differently named folders.

This issue: (A rant on) Windows.

Windows-like systems (who am I kidding, there’s only Windows)

Unlike Mac-like systems, where all data comes in the form of a file, Windows systems recognise two types of data: settings, and files.

Files in Windows systems

Apps get put in C:\Program Files or C:\Program Files (x86), for 64-bit and 32-bit programs respectively¹
Library? Shared files? Ha! (I’ll talk about this further down)
System files go into C:\Windows
Network resources, well … don’t really have a … well they are a different category of location that does not start with a drive letter and instead starts with \\, unless you assign these locations to a drive letter, then they have a drive letter. Sorta.
User files go into C:\Users
Other storage devices are auto-detected and assigned a drive letter, though not always consistently.

And now we talk about settings.

Settings in Windows systems

Settings are stored in the C:\Windows\System32\Config\ and C:\Windows\Users\Name\ folder, which technically makes them system files, which … wait, how are apps supposed to access them then?

App developers are supposed to do it through a system library, which provides variables named like ApplicationData.LocalSettings, ApplicationDataCompositeValue, and RoamingSettings. These variables let developers store and retrieve settings, which all end up stored in a system known as the Windows Registry. And Administrators can edit them using something known as the Registry Editor.

The Windows Registry consists of 5 top-level areas (known as hives), each one beginning with the word HKEY_:

HKEY_CLASSES_ROOT is for storing application settings, and file extension information (e.g. which app to use to open each type of file extension)
HKEY_CURRENT_USER is for storing settings and configuration specific to the current (logged in) user
HKEY_LOCAL_MACHINE is for storing settings and configuration common to all users (e.g. default settings)
HKEY_USERS is for storing settings and configuration of each user. The HKEY_CURRENT_USER data for all users is stored here, and copied to HKEY_CURRENT_USER when they log in.
HKEY_CURRENT_CONFIG is for storing information about the computer’s configuration and resources

The usual way of finding out how to modify a particular setting for X is to google “registry setting for X” and proceed from there.

Logos, backgrounds, buttons, and other application data? They go into C:\Program Files (or C:\Program Files (x86) if still 32-bit) for traditional Windows apps, or into C:\Program Files\WindowsApps for Windows App Store apps. What if other apps also need to use them? Then they go into ~~C:\Library, just kidding, if only it were so easy~~ C:\Program Files\Common Files, but you’ll notice it’s pretty empty. Usually, they’ll be stored within the app’s folder, and you have to find out where to edit the Windows Registry so other programs know where to find them (apparently you can look in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\KnownDLLs). Whoa, wait, what happened to apps not messing around in each other’s folders—

OKAY MOVING ON—What if a user installs a program that they don’t want other users using? It goes into C:\Users\username\AppData\. User settings? They go into—nah, they don’t go into a file, they’re supposed to be settings so they go into the Windows Registry somewhere under HKEY_CURRENT_USER. Temporary files? They go into C:\Windows\Temp; isn’t that a system folder? Well yes, but if you put it in C:\Temp folks will complain and Disk Cleanup will not find it.

So to uninstall a Windows app, you run its uninstaller. Which may or may not work perfectly. Or it might remove its files but still appear in the Program List because it did a terrible job cleaning up its settings in Windows Registry. So you reinstall the program, this time using a third-party app that helps you track app installations and registry changes, so that it detects what new files/settings it creates, and then when you uninstall the program you do it through the third-party app so that it hopefully removes all traces once and for all.

Phew. And that’s all I hope.

Issue summary: Windows systems categorise data into two types: files, and settings. Files are stored under an appropriate subfolder in C:\, while other storage devices and network locations are stored elsewhere or given their own drive letters. Settings are managed through the Windows Registry, which is stored in C:\Windows\System32\Config\ and C:\Windows\Users\Name\.

Okay so this ran much longer than I expected. In fact, it ran so long that I split it into two issues. I promised to explain computers as simply and jargon-free as possible, and I hope I have managed to do that. I am definitely biased, and that I do not apologise for, because this newsletter issue would be half its original length if *muttering* some operating systems would just follow sensible principles that other operating systems have no problem following …

What I’ll be covering next

Next issue: [LMG S8] Issue 101: Why do apps crash?

Moving on from app files and settings, the next few issues will explore common app problems. Coming up next issue: why do apps crash?

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
involved in installing a piece of software? [Issue 48]
How do apps know where a file starts and ends? [Issue 49]
a password hash? [Issue 63]
a driver file and why do I need one? [Issue 98]

I really was hoping not to have to explain 32-bit vs 64-bit programs ever since Issue 55), so for now let’s just say 32-bit programs are for 32-bit CPUs and 64-bit programs are for 64-bit CPUs. Unfortunately many old 32-bit apps have not caught up with the times and converted themselves to 64-bit apps, so Windows has to do hacky stuff to make old 32-bit apps work on modern 64-bit CPUs. ↩

Issue 99: Where does all the app data go? A look at Mac-like systems

2020-12-19T08:00:00+08:00

Previously: Apps generally handle three categories of files: its own (permanent) app files, (shared) user files, and (ephemeral) temporary files.

What we are here to find out is: where do these apps keep their data, and how can we get rid of them (if we really want to)?

Web apps and mobile apps will not be discussed here, because they are much more heavily sandboxed, everything gets confined into the app’s little prison, and we generally don’t have these concerns when it comes to them.

Just one little niggle …

I am so sorry to burden you with this otherwise unrelated information, but since there are a significant number of Windows users and a significant number of MacOS users, I had to bring this up at some point.

Windows and Mac manage this differently, so we are going to have to talk about two different kinds of systems. I will spend more time on the Windows system, because it needs more time.

Let’s get the easy one out of the way first: this issue deals with …

Mac-like systems¹

On MacOS (and Linux) systems, everything is a file. All files get stashed into some kind of folder.

I personally prefer this because you have everything sorted into sensible top-level folders². Mac has:

an /Applications folder for apps
a /Library folder for shared files (see Issue 17), but on Mac this extends to things like sounds, profile pics, colors, …)
a /System folder for, well, you know what.
a /Network folder for accessing resources on the network (such as shared folders)
a /Users folder for accessing user folders and files
a /Drives folder for accessing other storage devices (e.g. USB drives)

and then, similar to Linux, it has /bin, /etc and other weird-looking folders that we don’t need to worry about at this point. Just treat them similar to system files and try not to touch them. The apps that we install generally do not clutter up these folders unnecessarily.

Logos, backgrounds, buttons, and other application data? They go into /Applications. What if other apps also need to use them? Then they go into /Library. What if a user installs a program that they don’t want other users using? It goes into /Users/username/Applications. User settings? They go into /Users/username/Library (under a subfolder for the app). Temporary files? They go into /Library/Caches.

Linux systems are similarly divided, but into differently named folders. Everything is still a file, and belongs in some folder somewhere.

So to uninstall an app, you remove its files from /Applications or /Users/username/Applications and from /Library, and that’s usually it. Apps are usually quite good at doing that themselves, so you don’t need to worry.

And then we deal with Windows systems in the next issue.

Issue summary: MacOS, Linux, and other similar systems treat everything as a file, organised into appropriate subfolders.

I’m keeping this issue short because the next issue will be much longer. *Ominous music plays*

What I’ll be covering next

Next issue: [LMG S8] Issue 100: Where does all the app data go? A look at Windows systems

The reasons for the difference between Mac-like systems and Windows systems is, again, historical, but I better prepare you because you are not going to like the next issue.

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
involved in installing a piece of software? [Issue 48]
How do apps know where a file starts and ends? [Issue 49]
a password hash? [Issue 63]
a driver file and why do I need one? [Issue 98]

The technical term is “Unix-like systems”, but we don’t need to know that, even if all the Unix fanfolks are pointing pitchforks at me now. ↩
There’s lots of things that make less sense once we get into more detail, but fortunately we don’t do that here. ↩

Issue 98: Temporary files

2020-12-12T08:00:00+08:00

Previously: A laptop app can do practically anything, if it is running through the Administrator/root account. Sandboxing is carried out through permission control.

Thus far, I’ve summarised the salient differences between web apps, mobile apps, and laptop apps (my own terminology). I think we can move on to talking about their similarities.

There is a category of app that needs access to your device storage. It might be Youtube trying to download (part of) a stream onto your device for playback. Or it might be Tiktok trying to help you record a video for uploading. These apps need storage access so they can stash all the data into files, rather than hogging device memory with it. The same way we stash things into drawers and cabinets when we don’t need them, so they don’t clutter the space around us. And then we forget about them until we run out of space 😅

You almost never see where those files appear. They get hidden … somewhere *gesticulates around*.

Temporary files

These files are short-lived; they typically don’t stick around for more than a few days. For that reason, they are known as temporary files.

In a webapp, the browser generally stores these files into one of its allocated folders, somewhere in C:\Users\[USERNAME]\AppData\Local\Google\Chrome\User Data\Default\Cache or the like. They get cleared when you clear your browser cache. On a mobile or laptop app, the operating system designates a space for temporary files, in C:\Windows\TEMP or AppData\Local\Temp for Windows, or /tmp on Linux. You can clear those files through Disk Cleanup in Windows.

In general, temporary files are things you are not supposed to think about too much. The operating system has ways to clear them regularly. Apps are supposed to use these standardised locations to stash temporary files, and attempting to place them anywhere else is considered impolite, like leaving your stuff lying around in an office or otherwise public space.

User files

But then, sometimes the app produces useful data that you want to keep around! Your journal which you keep in a Word document, photos of your cat or dog, and the copious, copious video files …

Webapps have no space for you to do that. You are just supposed to save them onto your phone or laptop; the browser has no way for different users to stash their own files.

Most smartphones assume they are going to be used by a single user, and you just stash those files directly into phone storage. Not the best system, but it is what it is.

Laptops are where it gets a bit more interesting. Most laptop operating systems (OSes) assume they might be used by multiple users, each on their own account (hence the login screen), and therefore allocate separate spaces where each user may keep their stuff, inaccessible to other users except Administrators (Issue 97)).

On Windows, these users each have their own folder in C:\Users\; on MacOS, that’s in /Users/; on Linux, it’s typically /home/. (Don’t ask about the \s vs /s; its one of those things that’s just the way history happened and has no real technical reason behind it.)

App files

Of course, each app needs to have its own space to keep its own files, which allow it to do what it does.

Webapps get their own folder somewhere in C:\Users\[USERNAME]\AppData\Local\Google\Chrome\..., they can only see what is in that folder, and they cannot see what is in the parent folder, or sibling folders. It’s sandboxing, again (Issue 92))!

Mobile apps get stored into /data/app or some similar folder, and you’re not supposed to think too hard about where, because of *handwaving* sandboxing. The same idea applies: The app is not supposed to know, or be able to see, where other apps store their data! Eyes on the app’s own data only, and the user’s data (you did give it permission to access storage, right?), and any temporary data which it has created.

Laptop apps get stored in C:\Program Files, and interestingly enough have some kind of civil arrangement where they agree not to delete each others’ files, although antivirus programs have this passive-aggressive low-key thing where they like to mark each others’ program files as potential malware *shrug*.

System files

These files were around ~~in the beginning of time~~ when the OS was installed; that means when you bought your laptop, they were already there, and any sensible system would prevent non-Administrators from mucking around with them. Windows stashes them in C:\Windows, while MacOS and Linux store them in /bin/, /lib, and various similarly opaque folders.

I might go into more detail about these, possibly in a future season when I talk about operating systems, but for now we are done talking about categories of files. Phew!

Issue summary: Apps generally handle three categories of files: its own (permanent) app files, (shared) user files, and (ephemeral) temporary files.

In reality, there are a whole bunch of different filetypes and other little details that apps need to worry about, but this is a newsletter for layfellas so let’s start simple.

What I’ll be covering next

Next issue: [LMG S8] Issue 99: Where does all the app data go? A look at Mac-like systems

I separated temporary files and user files in this issue. Temp files are files that come and go, like stray cats, while user files are shared with other apps as well and there’s really not very much that you can predict about them except hope users don’t do anything too crazy. System files are strictly off-limits so don’t even think about that.

But meanwhile, as an app developer, even after you exclude the above categories of files, there is still a whole bunch of questions you kinda need to worry about at some point:

where do I store my logos and backgrounds and buttons and other data?
where do I keep my settings?
where do I keep user settings?
if my program is meant to make a USB device useable, where do I drop the driver files? (Yep, that’s gonna need its own issue)
what if someone uninstalls my program but I want them to be able to keep their settings around in case they decide to reinstall and then it can feel exactly the way they left off?

Yeah I have no idea who actually thinks about that last question either, but it’s something that seems to get asked in every software uninstallation *shrug*.

So let’s get into that in the next issue.

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
involved in installing a piece of software? [Issue 48]
How do apps know where a file starts and ends? [Issue 49]
a password hash? [Issue 63]
a driver file and why do I need one? [Issue 98]

Issue 97: Laptop apps

2020-12-05T08:00:00+08:00

Previously: Mobile apps are sandboxed by the operating system. As a result, they have to bundle all the libraries they need, and are not allowed to share libraries with other apps. This results in mobile apps with huge filesizes.

This issue is going to be short, because laptop apps are … well, almost unlimited in what they can do.

Web apps are sandboxed by the web browser. Mobile apps are sandboxed by the mobile operating system (OS). Laptop apps are sandboxed by hte desktop OS (yup, Windows on laptop and desktop is practically the same).

The main difference here lies in the difference between a mobile and desktop OS. Mobile OSes do not allow mobile apps to share libraries with other apps, and restrict their privileges (Issue 96)). Desktop OSes, on the other hand, let you do anything that is computationally possible … if you have permission.

The Admin account

A desktop OS often has an all-powerful user, known as the Administrator (Windows/MacOS), or root user (Linux). This user does not need permission to do anything. But with great power comes great responsibility, and with an admin account it is all too easy to do something that renders the computer unuseable.

So lower-privilege accounts exist—these are the user accounts. Logging in as a user gives you limited privileges: often you cannot change OS files, install or remove apps, or do anything risky. This is, for the most part, how desktop OSes sandbox the computer environment from damage by other apps.

What an admin can do

So what happens when you run an app on an admin account?

This app can:

Edit, delete, rename OS-related files
Create new “virtual” (emulated) hardware devices, and manage drivers for it
Send data to any device, or receive data from any device
Make changes to storage devices, including the disk where the OS itself is installed (but not the partition¹ where the OS is installed)
Run programs in the background
Send data over the network to any IP address, over any port (Issue 33))
Prevent other programs from doing so (Issue 34))
Install libraries that can be used by other programs
Access OS settings and make changes that affect OS operation
… and many more things!

User Account Control

What if a user needs access to some of these permissions (but not all)? Does that mean they need to become an Admin?

Windows, and other OSes as well, usually have some way to give users limited permissions for some tasks. Windows uses User Account Control, which pops up a dialog box to alert the user. If the user gives permission for the app to proceed, then it is able to do so. If it is running on a user account, it can only perform tasks that the user account is allowed to perform. Linux uses the concept of groups; for a user to have permission to access bluetooth, for example, the linux OS often requires the user to be added to the bluetooth group in the OS.

Issue summary: A laptop app can do practically anything, if it is running through the Administrator/root account. Sandboxing is carried out through permission control.

This is the reason why you should still buy a machine with a desktop operating system if you plan to be doing anything really productive; the sandboxing systems of web and mobile apps ultimately still impose a significant limit what you can do with the device. This is intentional; it is done for your safety! But if you want your device to do more, you’ll often need to override these “safety limits”, and that is where desktop operating systems come in.

What I’ll be covering next

Next issue: [LMG S8] Issue 98: Temporary files

Next issue is going to round off this mini-arc on how different kinds of apps operate. In the process of doing whatever it is they do, apps often generate temporary files that can be safely removed. How does this work for web apps and mobile apps? And for laptop apps?

From there, I’ll expand to talking about how apps store their data on and retrieve their data from the operating system.

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
What is involved in installing a piece of software? [Issue 48]
How do apps know where a file starts and ends? [Issue 49]
What is a password hash? [Issue 63]

I’ll talk about partitions in a future issue, when I move on to hardware devices ↩

Issue 96: Why are mobile apps so large in size?

2020-11-28T08:00:00+08:00

Previously: Mobile apps, unlike web apps, can bundle resources and libraries to be installed to a mobile device. They can also request access to storage, and typically have a higher memory limit than web apps.

In the process of creating an app, a developer often needs to use libraries (Issue 17)), which are ready-packaged chunks of code she can run without having to write the code herself. Some provide core functions (e.g. sending information via the Internet, or checking if a data connection is available), while some provide optional features (e.g. mobile payments, or user feedback systems).

Libraries that provide core functions are typically provided by the operating system (OS), which ultimately controls the device’s resources, but all else have to be pulled in by the app, somehow.

Web apps vs mobile apps

Web apps, which I covered in Issue 94), pull in any resources they need through web requests. This includes any libraries that they need. The browser allows it to do this, but prevents access to most parts of the operating system, and allows limited access to camera, sound, storage, etc. The sandboxing features of the browser make web apps generally safer to access.

A mobile app, on the other hand, is sandboxed by the operating system. Most of the resources it needs have to be present at the time of running the app, and that includes libraries. In an unsandboxed environment, commonly used libraries (e.g. mobile payment libraries) could be installed in the OS and shared by the apps. But this opens up a means of unauthorised access to multiple apps: hack this library successfully, and all other apps on the OS are also affected!

The sandboxing system in a mobile device does not allow this. Each app must bundle all the libraries it requires, to be installed into storage after downloading. This way, if an app has one or more libraries compromised, it would at least not expose the user’s data in other apps.

Libraries in a mobile app

The tradeoff to separating all these mobile apps and preventing sharing, is that each app now comes with its own copy of all the libraries it needs. And the file size can really add up—you already see it in the huge app sizes. I unbundled the installation package of a popular shopping app, Lazada, just to see what is inside it.

The list is way too long to post as an image, or even as text; it has over 300,000 code functions bundled inside! I should note that I am not an Android developer and can’t tell you very much about whether these libraries are absolutely necessary, but here are some noteworthy libraries included that I can make an educated guess about:

Libraries to translate code from one programming language to another (often to translate an easier language into a faster or better-supported language)
Compatibility fallback & device evaluation libraries (probably for devices on older Android versions)
Layout libraries (for calculating placement of window frames in devices with different screen sizes)
Graphics, media, augmented reality (AR) libraries (for graphics rendering, video playback, capturing images from camera, etc)
System, network, version detection and updating, etc
Animation libraries (I see one from AirBnB)
Analytics libraries (to track user and ad engagement, and do A/B testing)
Debug, crash reporting, logging libraries (for troubleshooting app crashes)
Integration libraries (for login using FB and other accounts)
Maps, location
Search, image search, QR code, user feed, and related services
Database access (it seems to use Google Firebase, in addition to others)
Mobile payments

and then there is the app itself, which contains code for:

Address validation, checkout, delivery
User feed and homepage, login management, recommendations
Search

and many others which I don’t know about.

Why aren’t web apps so huge then?

They are! But they don’t need many of these (e.g. code translation and compatibility), and most of the libraries in 1–13 would have been loaded separately from the main page (see Issue 78) for a visual example). Much of the functionality would not need to be loaded or installed upfront, only when it is required (e.g. map display).

For a web app, many more functions would also have been offloaded to Lazada’s servers, such as address validation. On a mobile app, this code is included upon installation to reduce data usage.

More importantly, you have little idea how large a web app really is, since you are never shown its filesize anywhere ;)

Issue summary: Mobile apps are sandboxed by the operating system. As a result, they have to bundle all the libraries they need, and are not allowed to share libraries with other apps. This results in mobile apps with huge filesizes.

It’s worth thinking about what this says whenever we hear about so much data being transmitted over the internet. Much of this data is actually duplicated data (for security of inefficiency reasons), or metadata (for data management), or overhead data (because of the way the data is packaged). Just like Amazon packaging!

What I’ll be covering next

Next issue: [LMG S8] Issue 97: Laptop apps

Finally we can move on to more fully explore the complexity of apps that integrate more closely with the operating system: laptop apps!

This should be enough of a primer before I go on to talk about where all this app data goes, and then about app installation and uninstallation (and hence strike out another “sometime in the future” question, woohoo! 🙌)

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
What is involved in installing a piece of software? [Issue 48]
How do apps know where a file starts and ends? [Issue 49]
What is a password hash? [Issue 63]

Issue 95: What’s in a mobile app?

2020-11-21T08:00:00+08:00

Previously: Web apps require the browser to request memory on their behalf, and thus their memory usage shows up under the browser process in the OS Task Manager. Web apps use this data to store a more convenient (but larger) representation of the webpage document, and to store the data needed by the app.

“Why use the mobile app when there’s already a website?”

“Why even have a mobile app that looks almost exactly like the website?”

I’m not going to answer from an aesthetic or user experience point of view, you’re all experts in your own preferences :) Instead, I’ll focus on whats actually under the hood in this newsletter issue.

If you haven’t read Issue 93) on web apps (a.k.a. websites), I’d recommend you do that for the full context, because in this issue I’ll highlight some key differences between web apps and mobile apps.

Resources

Web apps have to request every single image, video, non-text object on the page via a web request. Caching (Issue 39))—storing these resources for offline use—can reduce subsequent load times, but the first load will still be the most painful.

A mobile app can package the most common, unchanging resources (logos, button images, backgrounds, etc) into the mobile app itself, so they can be loaded directly in the app, without having to make a web request and wait for the response. This lets it load faster (theoretically … in practice, many apps still have to retrieve other data from the server, so the loading speed improvement is marginal)

Flexibility

The document object model, or DOM (Issue 94)) is how web apps keep track of all the elements and their contents on a page, but it is not the most efficient way to do so. A mobile app has more choice in deciding which user interface library (Issue 17)) it wants to use.

Storage access

A mobile app can request permission to access storage on the mobile device, allowing it to store files (images, data, …) on the device without having to interrupt the user each time. It is not limited only to browser storage interfaces (localstorage, sessionstorage) and browser databases (IndexedDB)—see Issue 93)—but can use other kinds of interfaces and databases, if they are available on the mobile device, or bundled into the mobile app.

Memory use

Both Android and iOS impose a memory limit on each app that is running. And they treat a mobile browser as a single app, despite all the web apps running inside it. So a web app has to share that limit with all the other web apps running in the mobile browser (which is why your tabs have to reload so often—they are also cleared often!).

On the other hand, a mobile app can have that per-app limit all to itself.

All told, a mobile app has more resources, which it requests directly from the OS instead of via the web browser, and it has more freedom in using those resources. How so? And more importantly, why are some mobile apps just so darn huge?

This and mobile app sandboxing explained next issue.

Issue summary: Mobile apps, unlike web apps, can bundle resources and libraries to be installed to a mobile device. They can also request access to storage, and typically have a higher memory limit than web apps.

Mobile apps are a bit of a weak spot for me since I haven’t had as much experience here as I had in other areas, but nonetheless the limits of sandboxing are pretty visible. For the most part, we have accepted this tradeoff between size and security since storage space became much cheaper. But this tradeoff is also apparent not only in software, but also in business management and other areas: to increase security, we often also have to increase bloat.

What I’ll be covering next

Next issue: [LMG S8] Issue 96: Why are mobile apps so large in size?

Remember the days when most apps we downloaded and installed on a laptop were 2MB or less? Today, mobile apps are many times that size. This is partly because of the way sandboxing is done for mobile apps. How so? I’ll go into more detail next week ;)

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
What is involved in installing a piece of software? [Issue 48]
How do apps know where a file starts and ends? [Issue 49]
What is a password hash? [Issue 63]

Issue 94: Why do web browsers take up so much memory?

2020-11-14T08:00:00+08:00

Previously: Web apps have limited access to the device’ storage, and can only store data in browser-managed databases. Progressive Web Apps (PWAs) can additionally register service workers that run in the background. Because they are so cleanly sandboxed, they can be easily removed by clearing the browser cache and storage, and deregistering any service workers manually.

Apps once lived on a computer. You double-clicked them or pressed Enter or right-clicked Run, a window pops up and a rectangle appears on your taskbar. If Task Manager is showing you that it is using up gobs of memory, you just End Task and the rectangle disappears. Later in this season I’ll say more about what it is like to live in a computer, but we are on web apps now.

Web apps are somewhat more complicated. Because they are so tidily sandboxed (Issue 92)), they cannot actually live on your computer. Instead, they live in your browser.

Living in a web browser

Living in a computer means that the operating system (OS) takes care of you; it gives you the memory and disk space you need, gives you CPU time to run your processes, and gives you access to devices (such as the screen and keyboard).

When you live in the browser, the browser takes care of you. Everything you need is requested from the operating system (OS) by the browser. The space that the web app uses in localstorage, sessionstorage, IndexedDB, and the cache, is space that the browser requested. The CPU cycles that the web app needs are cycles that the browser has requested.

When you open the OS Task Manager, where do these requests show up? Under (one of) the browser processes, naturally. If you have more than 20 browser tabs open for more than 5 apps, it shouldn’t be surprising that they are using a lot of memory; I’ll go into why shortly. More worryingly, that’s not helpful if you’re trying to figure out which browser tab to close so you can play your memory-consuming video game.

Only the browser has that information; you will have to open the browser’s Task Manager (another google away) to see that information.

A web app’s needs

I have a browser tab open now, with a Google Sheet loaded. What is the Google Sheet app doing on that page? Let’s open DevTools and find out.

DevTools has a really cool tab labelled “Memory”, and it has a nice visual depiction of what the app is doing with all that memory:

DevTools in Firefox lets you inspect the memory that web apps use.
The Memory tab shows what is stored in memory.
Most of the memory here is being taken up by javascript objects.

Javascript objects here are Javascript’s own internal representation of data, which is quite similar to a document database’s format. Altogether, they take up 32 MiB of memory space (difference between MB and MiB is covered in Issue 40)). Google Sheets is juggling a lot of data internally, data which is not stored in IndexedDB or localstorage!

other, taking up 15 MiB, seems to be pointing to a javascript library that Google Sheets is using to render the spreadsheet.

strings are simpler than objects, each one representing a snippet of text, or possibly even a number. They only take up 5MiB. scripts are the internally stored scripts that the page is executing; they take up 11 MiB.

domNode is where it gets interesting. We are used to seeing HTML documents (Issue 50)) as a plain text document with lots of formatting, but in a browser it becomes more than just text. Each part of the page, an HTML element, can have its properties changed by Javascript as the page reacts to new data, or to user input.

The HTML Document Object Model

It would be too computationally taxing to keep scanning through the text document to figure out which part of the page is meant to be changed. Instead, the browser has its own way of storing the hierarchy of elements: each menu option falls under a menu heading, which falls under the navigation bar, which falls under the header, which falls under the main document, and so on. If each browser had its own way of doing that, a web developer would have to learn all of them to make a webpage that worked across all browsers; that’s terrible!

Instead, the web standardised on one way of doing so: the HTML Document Object Model (DOM).

Internally, a web browser converts the HTML page into a DOM—a data structure that makes it easy to find the specific HTML element (or elements) that need to be modified by each function. The HTML DOM for the page I’m on takes up 14 MiB, which may sound like a lot, until you remember that each element also has associated metadata stored along with its content. And Google Sheets has lots of elements!

Issue summary: Web apps require the browser to request memory on their behalf, and thus their memory usage shows up under the browser process in the OS Task Manager. Web apps use this data to store a more convenient (but larger) representation of the webpage document, and to store the data needed by the app.

And that is how a web app uses up 84MiB of memory space. If you have multiple tabs running the same app (e.g. multiple Google Sheets open), some of the memory can be shared (Issue 84)) by these tabs (e.g. scripts), but otherwise each tab is going to have its own memory needs.

In the earlier days of the internet, when spreadsheets were still a separate app, this memory usage would have showed up in the OS Task Manager under Lotus 1-2-3, Microsoft Excel, or some other spreadsheet program. Today, it shows up under Chrome or Firefox, and the details are only inspectable through the browser’s Task Manager.

No wonder browsers get all the blame these days.

What I’ll be covering next

Next issue: [LMG S8] Issue 95: What’s in a mobile app?

This issue felt like a data dump; I know it’s a lot all to take in 😅 In my childhood, I had access to lots of books with these cutaways showing the inner mechanisms of devices of all sorts, and I loved those books. It’s rather harder to do the same with software, since there’s nothing to physically slice through (even if only in the imagination!) I hope that the screenshots in this season of LMG will help you imagine the inner mechanisms of apps. Let me know if it’s working for you, and if there’s anything you’d like to see :)

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
What is involved in installing a piece of software? [Issue 48]
How do apps know where a file starts and ends? [Issue 49]
What is a password hash? [Issue 63]

Issue 93: What’s in a web app?

2020-11-07T08:00:00+08:00

Previously: Sandboxing is a catch-all term for the concept of ensuring apps don’t have access to resources outside of their privileges. Sandboxed apps are generally safer than non-sandboxed apps in terms of security, and easier to manage, terminate, and uninstall.

The simplest apps we use do not generate data; think about your calculator, which simply crunches calculations and displays the result (conveniently ignoring the ones with memory slots for storing calculated values …). Or currency converters, or timezone converters … no storage needed.

So how do web apps store data?

Data storage in web apps

As a web-first programming language, Javascript programs were not expected to have to access, open, or create files on the device. That would make it really easy for a malicious script to download malware to a folder, where it could be accidentally invoked! Instead, it relies on other features to store and retrieve data for specific purposes:

To remember user logins (the “Remember me” feature you see on almost every login screen), web apps can set/unset cookies (Issue 69)) in the browser.
To obtain files for use, the web app can invoke a File Select dialog for the user to choose a file, such as for uploading to the server. The web app is not allowed to access arbitrary files this way.
If data needs to be provided to the user in the form of a file, it can be stored on the disk with the user’s permission through a download dialog.
The web app can store data through a browser feature called localstorage. This is a key-value database, managed by the browser, that allows you to store data (the value) tagged to a key. The same way a hotel lobby holds your luggage for you and lets you access it through a luggage tag, or the way you can rent a locker for storing your stuff (value) and access it through the locker key.
For data that is only needed in that tab (e.g. partially filled form data), and can be safely deleted when the tab is closed, the browser provides sessionstorage. This works similarly to localstorage.
For more significant amounts of data, web apps can use IndexedDB, a more advanced database also managed in the browser. It is a document database (Issue 88)), with each document tagged to a key in a key-object system.

DevTools in Firefox lets you inspect the data that web apps keep.
The Storage tab shows what is stored in cache, cookies, IndexedDB, localstorage, and sessionstorage.
IndexedDB, localstorage, and sessionstorage are key-value databases that store the data (value) tagged to a key.

Requesting and receiving data on a server

So a web app doesn’t much in the way of storage access, but they were not designed for that at all. Most of the heavy lifting is not meant to be done in the browser, but elsewhere, on a server. A web app would send heavy workloads to a server (typically owned by the same company) through an API (Issue 4)) through a web request (Issue 9)), and receive the results through a server response (Issue 8)).

A web app would also need resources for display: images, videos, PDFs, … these are requested and received via web requests as well.

Cool, so a web developer can just write Javascript code to get the data and resources it needs, display stuff to the user, wait for the user to interact, and then make more requests to the server to calculate stuff, or send the app more data (such as the user’s tweets or posts or other stuff).

Just one problem with this: if internet connectivity is intermittent or laggy, none of this is going to work! Even when the internet is fine, it makes for a very slow experience. How do we improve this?

One way is to cache (Issue 39)) as many things as possible: header images, logos, emojis, icons, … these can all be stored in localstorage and accessed even when the app is offline.

Some apps, such as Google Docs, will also store user data in IndexedDB for a smoother experience—imagine having to wait for a request-response round-trip to the Google servers for every word you type. The data gets modified in IndexedDB first, and then synced to the servers. If the device gets disconnected from the internet, at least you will still be able to read whatever is in IndexedDB (and if you have enabled offline access, you can even edit the data in IndexedDB, and the Docs app will attempt to sync it to the server once connectivity is restored).

Running background processes

Then what’s with these popups on some websites asking you to install them? And how are some websites actually able to send us notifications? Something has got to be running in the browser background for these to happen, and none of what we have learned so far explains that … what gives?

Enter Progressive Web Apps (PWAs). With some Googling and lots of reading on StackOverflow and other web documents, a web developer can get started meeting the various requirements needed to create a PWA.

And in exchange for that inconvenience, she can use service workers: javascript scripts that run on their own and are not dependent on the browser tab staying open. These service workers can listen for messages from the server, carry out some processing, make requests and receive responses, all independently from the app running in the tab. On mobile devices, they are gradually gaining more features as well, such as access to the Share feature (enabling users to share content with the PWA), and being able to access cameras, microphones, location services, and other things (provided the user grants permission).

Installing web apps

Okay, wait. WHAT?!

I imagine most folks would be okay with cookies, with localstorage and even IndexedDB. You want things from an app, you gotta give it space to work, right? That’s fair.

But service workers, what?! You mean if I click Install, these apps get to run stuff in the background in my browser, even after the tab is closed? If I’m not cool with that, I can just choose not to install and then these service workers won’t get installed, right?

Umm, I don’t know how to break this to you gently, but nope.

Once you visit a site (URL) with a registered service worker, your browser automatically registers it. The only thing the Install button does is to add a shortcut on your Desktop/Home Screen, and perhaps enable some features (such as mobile Share). But the service workers are already there.

Uninstalling web apps

And now the good news.

Because web apps are so cleanly sandboxed, they don’t stick tendrils into your operating system or device storage (beyond the space reserved by the browser, anyway). Removing apps and their files just involves clearing your browser cache and website storage. You will, however, have to deregister the service workers manually; please google for instructions.

Issue summary: Web apps have limited access to the device’ storage, and can only store data in browser-managed databases. Progressive Web Apps (PWAs) can additionally register service workers that run in the background. Because they are so cleanly sandboxed, they can be easily removed by clearing the browser cache and storage, and deregistering any service workers manually.

What I’ll be covering next

Next issue: [LMG S8] Issue 94: Why do web browsers take up so much memory?

I’m finally starting to answer one of the sometime-in-the-future questions below, and can’t wait to get to the meat of “What is involved in installing a piece of software?”; it’ll be a ride! :)

Before I move on to compare web apps with mobile apps, I’m going to take a short detour next issue and answer a question I hear all too often: “Why do web browsers take up so much memory?”

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
What is involved in installing a piece of software? [Issue 48]
How do apps know where a file starts and ends? [Issue 49]
What is a password hash? [Issue 63]

Issue 92: All about apps

2020-10-31T08:00:00+08:00

Previously: Depending on what you need a database for, there may be online database platforms that can manage and automate much of the work for you. Airtable, Smartsheet, Knack, and Zoho Creator are just 4 of many options that offer an easier way to set up and input your data, then access them through apps or other means.

There was a time when apps, short for applications, were these files that came on a CD or diskette, usually ending in .exe (if your settings enabled you to see file extensions). You double-clicked them or pressed ‘Enter’, and things happened.

Today … the idea of an app is more nebulous. The .exe files are still there, but now there are also apps that you install from the app store. And what’s up with webpages that display an app installation pop-up and create an icon on your home screen or desktop? Are they the same kind of app? If not, what’s the difference between them?

Before I answer that question, we need to talk about an important concept called sandboxing.

Why do apps need to be sandboxed?

An unrestrained app running in your operating system would have access to any and all resources on that machine. It could potentially modify or remove system files, halt running programs or accidentally overwrite their memory contents, and so on.

The operating system) takes some measures to minimise or prevent this. But even within these limits, a app developed with good intentions could still potentially cause trouble. It might accidentally hang on to audio access when it no longer needs it, make a mess of the user’s home folder, leave files in limbo, etc. Programs can leave a mess, and a way to limit this mess is very welcome.

This is why sandbox systems were already being researched as early as the 1970s, and are still an ongoing research interest at many institutions.

What is sandboxing?

If you are old enough you might remember playing in a sandbox. While there are usually no explicit rules about how to play in a sandbox, there is usually one unspoken rule:

What was in the sandbox, stays in the sandbox.

A sandbox limits the mess, yet gives you unrestrained freedom within that box.

In a computer, a sandbox system imposes restrictions on running applications. Some common restrictions include:

only being able to access/change/delete files within a particular subfolder
only having access to some OS resources (e.g. not being able to access USB devices, or audio, or webcam)
having limited privileges while the application is not active (e.g. no internet access when not being directly interacted with, and just running in the background)

These restrictions are intended to limit any damage and keep the system more stable than it might otherwise have been.

So one way to categorise laptop apps, mobile apps, and web apps is in the amount of sandboxing they are subject to.

Issue summary: Sandboxing is a catch-all term for the concept of ensuring apps don’t have access to resources outside of their privileges. Sandboxed apps are generally safer than non-sandboxed apps in terms of security, and easier to manage, terminate, and uninstall.

Keeping it vague in this issue because the details really differ between operating systems, kinds of apps, and even the way they are distributed. We’ll be digging into details next issue, starting with the most heavily sandboxed app: the web app.

What I’ll be covering next

Next issue: [LMG S8] Issue 93: What’s in a web app?

From just little snippets of script that animated buttons and counted visitors back in the 90s, Javascript now powers a huge portion of the Internet, processing payments, serving ads, and much more besides. We’ll look at how this gets packaged into a web app next issue.

Sometime in the future: What is:

booting up? [Issue 15]
XSS? [Issue 8]
a good reason developers write code and give it away for free online? [Issue 21]
firmware? [Issue 34]
OpenType? And what are fonts anyway? [Issue 42]
What is involved in installing a piece of software? [Issue 48]
How do apps know where a file starts and ends? [Issue 49]
What is a password hash? [Issue 63]

Layman's Guide to Computing - Season 08

Issue 104: Storing sensitive data

Why would I want to keep secrets from my users?

Storing secrets on the web

Storing secrets in a mobile app

Storing secrets in a laptop app

Storing passwords

What I’ll be covering next

Issue 103: Why apps hang even with multiple threads

Multithreading

Race conditions

Main and secondary threads

Multiple worker threads

What I’ll be covering next

Issue 102: Threading

Threading

What causes threads to get stuck?

What I’ll be covering next

Issue 101: Why apps crash

Crashing

Unhandled responses

Illegal instructions

What I’ll be covering next

Issue 100: Where does all the app data go? A look at Windows systems

Windows-like systems (who am I kidding, there’s only Windows)

Files in Windows systems

Settings in Windows systems

What I’ll be covering next

Issue 99: Where does all the app data go? A look at Mac-like systems

Just one little niggle …

Mac-like systems1

What I’ll be covering next

Issue 98: Temporary files

Temporary files

User files

App files

System files

What I’ll be covering next

Issue 97: Laptop apps

The Admin account

What an admin can do

User Account Control

What I’ll be covering next

Issue 96: Why are mobile apps so large in size?

Web apps vs mobile apps

Libraries in a mobile app

Why aren’t web apps so huge then?

What I’ll be covering next

Issue 95: What’s in a mobile app?

Resources

Flexibility

Storage access

Memory use

What I’ll be covering next

Issue 94: Why do web browsers take up so much memory?

Living in a web browser

A web app’s needs

The HTML Document Object Model

What I’ll be covering next

Issue 93: What’s in a web app?

Data storage in web apps

Requesting and receiving data on a server

Running background processes

Installing web apps

Uninstalling web apps

What I’ll be covering next

Issue 92: All about apps

Why do apps need to be sandboxed?

What is sandboxing?

What I’ll be covering next

Mac-like systems¹