The Letter අ

Something Happened On The Way To Perfect History

2026-02-15T00:00:00Z

In the wee hours preceding dawn of the 29th of June, 3123 BCE, a Sumerian astronomer looked up at the night sky and recorded the trajectory of a large object. The object is believed to have been over a kilometer in diameter (about 11 football fields). Its impact is known as the Köfels event whose aftermath included a 900km high ejecta plume, a 500m thigh debris layer, and climate change.

Roughly 5147 years¹ later, the security cameras at a home near Charlottetown, Prince Edward Island (PEI), Canada, captured the first sight and sound of a meteor striking Earth. The footage is of substantially higher fidelity than the Cuneiform tablet. The sea of discussion, scientific research, new reports, and social media activity surrounding this event makes it irrefutable.

The Era of Perfect History is the period in human history during which every pertinent detail of even minor events would have photographic or video evidence. We are not there yet, and it’s likely that we never will.

Imagine billions of people running around with high-definition cameras that they can pull out and record at a moment’s notice. And now imagine there are dozens of social networks that reward sharing footage of any notable event with an audience of millions, and the fame and glory that come with it. This sounds exactly like where we were heading, doesn’t it?

No government would be able to lie to its citizenry. Every misstep or misstatement would be backed by irrefutable evidence. Just as the internet was supposed to bring about a perfectly informed society, with unfettered access to the truth, a history of perfect information was also supposed to bring about a just and transparent society, one where everyone can be held to the truth… eventually. However, some argue that the sheer volume of information could lead to manipulation, misinformation, and erosion of trust. The optimists’ view of perfect information bringing about a just society might overlook the complexities of human behavior and societal dynamics. Nobody planned this, but it was just the direction we were heading as a species.

The past couple of years changed many things, one of which is the irrefutability of video or photographic evidence. As of this writing, it is no longer possible to reliably tell AI-generated imagery from “authentic” imagery. (E.g.)

It is frightening how scarce reality is compared with what can be synthesized. Intuitively, it may seem to be the other way around. After all, our capacity to simulate reality is vastly inferior to reality itself. The discrepancy comes from the volume with which generative content can be injected into our discourse. While we labor for hours making squaking sounds with our throats or poking plastic with our meat sticks, generative content can spit out a million articles. As Henk van Ess says, “traditional fact-checking takes hours or days. AI misinformation generation takes minutes.”

Hoaxes and conspiracies abound, but so far we haven’t seen a substantial historical event be completely synthesized via AI. Either that or the falsehood was so perfectly executed that nobody can tell.

It would make for a great science fiction story until it actually happens.

… and 26 days to be exact. It is kind of incredible how people came up with such an accurate estimate. I haven’t read the book, but if anyone could have a look at chapter 4 and tell me, that would be great. In the meantime, I’ll have to take the word of the University of Bristol ↩︎

False Precision and Vibe Coding

2026-02-13T00:00:00Z

It occurs to me how much of human communication is disambiguation.

We have short lives and shorter attention spans. So we have to be tactful about fitting in just enough detail into the brief blips of attention that we are afforded. We wouldn’t enjoy listening to protracted descriptions of things we already know. So we fast-run along a stream of concepts where we only lay down just enough track for people to keep up.

We narrate things.

We tell stories, episodic journeys with familiar names and places and little else other than dialogue and the occasional pantomime. We convey so little detail, but for the right audience, the story is as vibrant, colorful, and detailed as anything from their own sensory experiences.

Things unsaid are those that don’t need saying. The listeners know how to fill in the gaps.

Effective communication requires knowing how to balance what to say and what to leave out —either because it is a detail that is unambiguous or because it’s an intentional gap that we want the listener to fill.

Programming, at least the “classical” kind, does not afford us this luxury. Everything has to be laid out precisely and in detail; even when those irrelevant details. So we seek the comfort of layer after layer of abstraction that hides details we don’t want to see. The same abstractions alleviates us of the responsibility of saying what we don’t need to say.

We hope that, much like our human conversation partners, the computer would fill in the gaps predictably and reliably. We hope it will do what we mean, not what we said. We are disappointed, frustrated, and occasionally amused when reality surprises us.

Software is a description of a machine.

Software is only useful when something interprets it and actualizes the machine. Actualizers –a.k.a. computers– aren’t smart. Every layer of a computer system needs to be told what exactly it is supposed to do. There cannot be any guesswork. So the description of the machine must contain excruciating detail, because the computer that actuates it has no ability to fill in the blanks.

We start with a vague understanding of the problem space. But somewhere along the way, we have to inject it with false precision so that the description becomes complete.

This is why so much of modern software is based on false precision—at least when it comes to end-user-facing systems. We are forced to describe a behavior in rigid detail despite our incomplete understanding of the problem space or lack of desire to overspecify the wrong solution. The precision we inject becomes part of the solution, and in large systems, indistinguishable from pertinent details. Sadly, our descriptions of machines rarely allow us to demarcate necessary precision and unnecessary precision.

Until now, that is. Finally, we have something that can take an incomplete, vague description of our machine—stated in terms of what it does or what its effect should be on something: vibe coding.

Or to be more specific, vibe coding specifications.

Our final actuators are still brute-force stupid. But at least now we have something between us and the actuator that can fill in these pesky gaps and again allow us the luxury of ambiguity.

Ambiguity allows us to be earnest in our imperfect understanding of the problem. We didn’t understand the problem because we have no perfect understanding of the messiest, most important part of the problem—us. But now, finally, we can state our problem the way we see it: vaguely and imprecisely.

The Increasingly Inaccurately Named User-Agent

2026-02-09T00:00:00Z

The User-Agent, a technical name for the web browser, was supposed to be our loyal butler on the internet – doing our bidding and keeping us safe. But over decades, browser makers diluted its role, sometimes as a result of actions done with the best of intentions. Both the internet and browsers have become increasingly complex. And with that complexity, the custodianship of the users’ data has become murkier.

Despite being the one peddling our data, user-agents can no longer tell us exactly who knows what about us. We’ve lost our agency over our browsing data. And now our dude on the internet is a “user-agent” in name only.

We will soon have to confront an objectively more chaotic creature – an AI-powered personalized user-agent. Personalized meaning it knows a lot about us and will continue to learn about us. User-agent meaning it will take actions on our behalf. And, AI-powered at this point means that nobody really knows how it works. Let’s talk about what is about to happen and how you can prepare.

The User-Agent is supposed to be your buddy.

Information is intangible – you can’t stick your finger in it. So whenever we want to interact with a system that processes information, especially of the digital kind, we need some stand-in for our finger so that we can poke things.

So, people came up with the concept of a user-agent – to refer to the device¹ that people interact with. People would push buttons, make holes in paper, turn knobs, or do whatever they need to do to communicate with the user-agent. The user-agent grasps what the person wants, and does what it needs to do to make the person’s wish a reality. On the flip-side, the user-agent also communicates results back in a form that the person can understand.

Early user-agents were simple creatures with no intent of their own. It wasn’t that hard for people to look inside them to understand what it was doing. There was no mystery. The user-agent did what was asked, and earnestly reported back on what happened.

For example, in an email system, this user agent is called a mail user agent MUA. A user-agent that interacts with the web on our behalf should be called a web user agent (WUA)². But because web user agents —which you might refer to as web browsers— are so dominant, people dropped the superfluous “web” part.

MUTT(1)                     User Manuals                    MUTT(1)

NAME
       mutt - The Mutt Mail User Agent

SYNOPSIS
       mutt [-hNpRxZ] [-s subject] [-c cc-addr] [-a file] ...
            [-F rcfile] [-H draft] [-i include] [-f mailbox]
            [address] ...

DESCRIPTION
       Mutt is a small but very powerful text-based MIME mail client.
       Mutt is highly configurable, and is well suited to the mail
       power user with advanced features like key bindings, keyboard
       macros, mail threading, regular expression searches and a
       powerful pattern matching language for selecting groups of
       messages.

       "All mail clients suck.  This one just sucks less."
       - me, circa 1995

An ancient “manual page” for the Mutt email client circa 1995. Note that the program is described as The Mutt Mail User Agent.

This was fine and dandy. People would fire up their user-agents, type commands, and the user-agent would do stuff and show the user results. But over time, these user agents evolved from simple command-line tools to much more fancy interactive, graphical browsers and services. Modern (web) user agents are not just programs, they are entire platforms³.

Modern User-Agents are technical marvels.

Since the same internet needs to be accessible from different user-agents for perpetuity, the way web pages are written and made available on the internet had to be standardized – i.e. everyone needed to agree on the how. But these technical specifications, or standards grew bigger and bigger and so demanding that only a few well-resourced tech companies can write and maintain a full-featured UA⁴.

Though voluminous, these standards are neither perfect nor complete. But they have been crafted through person-millennia of effort to ensure security and correctness. Organizations like the following work tirelessly to get the details right:

World Wide Web Consortium (W3C) is primary group that governs how web content is written.
Web Hypertext Application Technology Working Group (WHATWG) is the group that pushes the boundaries of what is possible to do inside the confies of a web browser.
Internet Engineering Task Force (IETF) is the group governing how data moves about on the internet, and also how to keep your content secure while it is being moved about.
The Unicode Consortium is responsible for which letters and symbols you can use on the internet. They are the people who bring you new emoji every year.

… and many others.

Beyond technical requirements, there is an increasingly important social contract: That the user-agent always act on behalf of the user.

That means that the UA should do what you ask, and generally not do what you don’t ask.

The latter –don’t do what the user didn’t ask– is a little complicated because of all the complexity involved in the modern web.

Say you ask to see bank.com which is a bank. So the UA talks to the bank.com server and asks for the contents of the page. The contents say that some of the pictures on the web page need to be fetched from advertiser.com. But doing so involves sending a bunch of information to advertiser.com like the fact that you are viewing bank.com and information like your preferred language and whether you’ve been to these sites before.

Pretty much all UAs will opt to fetch the resources from advertiser.com, because otherwise the bank.com site will not work correctly. Since the user requested to see bank.com, clearly they intended for all the required stuff to get downloaded too, right? Except, it’s likely that the user never intended for advertiser.com to learn all that information about them.

That the User-Agent must act ONLY on behalf of the user.

Herein lies the problem.

The modern web is a marvel in that it exists despite brutally conflicting interests, and is run by people who do not trust each other one bit⁵.

Do you really think the nice lady who made you read her life story before she told you how to make simple pancakes also made an ad for weight loss drugs? Of course not.

Behind the scenes, a “modern” web page is a warzone where multiple companies are playing tug of war with your data and limited attention span. Not only are these companies clamoring to show you video ads while you are trying to make pancakes, they are also trying to figure out where you’ve been and what you are into.

You, the user, don’t want any of that; not even the nice lady’s life story. You just want to make pancakes. A UA that acted only on your behalf would have just pulled out the recipe part and shown it to you.

In the short term, rewriting the web like this sounds great. But in the long term, those companies – who now have no one to show ads to – will stop paying the nice lady. And the nice lady will stop publishing her recipes. No ads, no recipes⁶.

So to keep the web open, the UA does a tricky balancing act between conflicting interests; one that is not weighed 100% on your behalf. The UAs do their level best to present you with the most authentic web, while trying to keep you safe from potential attackers, without divulging too much about you to strangers. But this situation does mean that the “user agent” isn’t truly just the user’s agent.

And it’s not just users vs. publishers and advertisers. There are many other influential groups on the web making the balance of interests difficult. Some are:

You. Including here for completeness. You want to visit websites and be presented with accurate reproductions of what the publishers intended … kind of. You don’t want to be bothered by stuff that you don’t want, like advertisements. Also you want to be able to consume content freely and without boundaries. You want a free and open web.
Publishers who make content on the web and give you a reason to visit it. There are many not-for-profit content that doesn’t have any commercial interests, but for those that do, it is important that they are able to make money publishing stuff. They also have lots of shared interests with users such as being able to publish what they want and to be able to reach their intended audiences. In addition sometimes they want to be able to publish things without identifying themselves.
Advertisers who compete for your attention and make money by influencing your behavior. One of the ways in which they compete for your attention is by “renting” space alongside publishers’ content.
Search Engines could be considered a kind of a publisher but they are special enough to get their own category. Search engines –at this point, mostly Google– act as the guides on the internet helping users find the content that they want. For this to work, search engines rely on the internet being open and free. Otherwise it would not be possible to take users to where they want to go. Since search engines are one of, if not, the primary tools users need on the internet they are always integrated deep into the UA.
Copyright owners and content rights holders who are worried that people on the internet might steal their content. So web browsers have to incorporate Digital Rights Management components into the UA to limit what users can do with protected content.
Businesses who want to make sure that you continue to trust the internet as a safe place to buy and sell stuff. So web browsers sometimes have to incorporate security measures (which actually help users a lot too) and easy to use payment methods.
Politicians. Yes those people. As much as we would like to treat the internet as some sort of ungovernable utopia, it isn’t. There are many laws and regulations that are different across political boundaries. So sometimes UAs need to report where you are physically so that your activity can be kept within legal bounds. Politicians and businesses often also want to control what you are and aren’t able to access on the internet. However, they take those issues up with publishers and internet service providers. Additional shady things politicians want include finding out the real identities of people who have said things they don’t like.

Shady stuff aside, politicians are also very useful. In fact as we will see below, legislation is pretty much the only way in which users can influence the big UA makers.
Internet Service Providers (or ISPs) make money by making it possible for you to access the internet. You give them some money, and they give you internet access. Seems simple, but there are lots of shady things that ISPs want, like to charge you based on what you consume. See network neutrality.

And that is hardly a complete list, but I believe we covered some of the biggest influences.

But how do we know how the UA balances conflicting interests?

But how do we know that the UA isn’t completely complicit and isn’t selling our furniture without our knowledge? Why would we believe the UA is doing what it says it is doing? What does too much about me even mean?

Well back in my day one would crack open the UA and take a look inside. Usually, things are simple enough that you can convince yourself, or someone you trust can convince you, that it is in fact doing what it says it is doing.

But modern UAs are so complex that such human inspection is impractical at best. Not only would you have to inspect millions of lines of code, you would have to inspect every new change as it comes in. Even then, there are parts of most commercial browsers that are closed source (i.e. they won’t let you look inside).

So most of us have to trust the judgment and moral compass of the thousands of specialists poring over the underlying logic.

Are UAs really unverifiable?

Okay, so this might be a bit too harsh. Aren’t the biggest names in the browser game open source? Aren’t they all, like, based on Chromium? Why can’t we just read the source code and figure out whether it is doing anything odd?

The reality is that people have and are actively watching open source browser projects with eagle eyes. There are even more people making sure any new standards on the web are good and are in the interest of users. Unfortunately, even if we can verify the open source parts of the browser, that’s not the end of the story. See, when you visit a web site, there could be data being sent to hundreds of websites. Even if the browser tells you each and every one of those web sites, it would not be practical for anyone to manually vet and understand what is going on. Even beyond that, not all the functional bits of a browser are open-source. In fact, perhaps the most questionable parts of a browser –like AI integration, media rights management, how it works with various hardware devices– are closed source for many reasons.

So it all boils down to whether it really matters if your security depends on you performing something infeasible for all practical purposes; kinda like being forced to read each and every privacy policy and binding contract you have to click through.

It matters who owns the User-Agent.

We’ve established that modern UAs…:

… are super complicated.
… have to blance conflicting interests of the user, publishers, advertisers, search engines, other businesses, copyright owners, politicians, internet service providers, and others.
… are not designed 100% in your favor.

Despite vigilant oversight users still have to trust, without verification, how a UA works.

If you don’t trust one, you can always pick another. But you can’t avoid taking someone’s word that the UA is safe. Because of how little control we have as users over browsers, the only real way in which users can influence UAs is via legislation that holds UA owners accountable for their design decisions.

Yasskin, J., & Capadisli, S. (2026). Web user agents. https://www.w3.org/TR/web-user-agents/#audience. https://www.w3.org/TR/web-user-agents/#audience

I’m using the term “device” loosely here because the user agent often consists of an entire stack of hardware and software. In literature, you’ll find the terms “device,” “machine,” and “system” used interchangeably, depending on the context.↩︎
The term web-user-agent doesn’t appear much in technical literature with a rare exception. E.g. (Yasskin & Capadisli, 2026)↩︎
What’s the difference between a program and a platform? The latter is something upon which other programs can run. While the original world-wide-web was document-oriented (i.e. the “internet” consisted of documents that linked to other documents), as technology progressed, the web browser became more and more capable of presenting complex behaviors. At the start they were just aesthetic – like blinking text or changing how a link looks like when you hover the mouse cursor over it. But over time, they have gained capabilities that rival native applications. Look at all the APIs that are available inside a browser.↩︎
I worked on Google Chrome for over a decade on the networking stack, and briefly on the renderer. The amount of documentation that it takes to describe how a web browser works is in the order of tens of thousads of pages.

Even then it is an understatement. The web works despite countless quirks and misbehaviors. That’s because modern browsers handle those quirks “correctly” to give the impression that everything is fine. These quirks and how to handle them aren’t necessarily written down anywhere. Browsers can’t just refuse to talk to large portions of the internet or refuse to render millions of older web pages just because the code wouldn’t look good.↩︎
Pun intended.↩︎
This ad-supported model of the open web sounds terrible; and in some ways it is. There are some alternatives that have been proposed over the years, but none have been successful as ads. Many approaches involve internet users making payments of some kind to publishers. Unfortunately, all such models have the downside that those who are impoverished will get kicked out of our global information village. So for now, this is the best we can do.

AI will probably do in a few years what privacy advocates have tried to do for decades; which is to kill the ad-supported web. As yet it’s unclear what will replace it, and whom this replacement leaves behind.↩︎

What's your cog ratio?

2025-12-16T00:00:00Z

As far as product development is concerned, the only thing that matters is the change we make in the user experience. We want our users to be happier because our product exists or, at the very least, less sad.

So when we build things we expect that our work will result in a positive change for our users.

However not everyone is in the business of making direct changes to products. In fact not even developers¹ spend every minute building things. Instead most of the work on a mature product has to do with planning, monitoring, testing, discussing, deliberating, and doing all manner of tasks that have no direct bearing on the material² composition of the product.

Roles like program managers and product managers are one more degree removed from the engineered product. On top of that managers, HR, finance, and all the supporting roles are even further removed from the user experience.

Support infrastructure like build systems, testing infrastructure, and monitoring infrastructure help people build, monitor, and improve the product. However, the people working on the infrastracture are even further removed from the form and function of the final product.

In common parlance, the term cog ratio refers to the ratio of the number of teeth of one cog to the number of teeth of another cog. It translates to the number of turns we expect from the driven cog for each turn of the driving cog.

It’s cliche to think of ourselves as cogs in a large machine. But if we do, it seems natural to wonder what our cog ratio is with respect to the final products we ship our users.

This is specially true when the products we work on are not end user products, but are either infrastructure or support products that help people who help people who … build the product. What would the effect be if this product or service you are working on didn’t exist? By how much do you think your work will improve the lives of our users?

Does the product improve no matter how fast you turn?

Contrary to common belief, developers in larger outfits spend most of their time communicating with each other rather than building things. See Coordination Headwinds by Alex Komoroske.↩︎
We are using the term “material” loosely here, since software products aren’t technically material. It refers to the bits and bytes that comprise the product.↩︎

Optimal line-length for publishing on the web

2025-10-05T00:00:00Z

Fonts should be pretty, and text layout should be easy to read as possible. When publishing on paper, publishers can control the physical format for the presentation. We can assume, for the most part, how someone might hold and read a book or a newspaper. Based on that we can predict how various layouts, typefaces, and letter sizes will affect the readability and aesthetics.

Publishing on the web is a bit more complicated. How exactly a web page will be perceived is difficult to predict. Content publishers adapt with responsive designs and share the responsibility for presentation with browser makers and device makers.

Folks old enough to remember old timey newspapers would recall how narrow their text columns were. Before digital layouts, newspapers were published using hot metal methods called linotype.

Linotype newspapers had narrow columns for practical reasons.

A metal line of type (“line-o-type” ⇝ linotype) also called a slug becomes difficult to work with if it is too long. In particular, the machines assembled text line-by-line. This process requires that all the lines are roughly of equal width. A small line width –2 inches in this case– was apparently the right trade-off.

There were additional reasons for the 8 column layouts of yore that were not just ease of arranging type. For example, the many small columns allowed for more flexibility for laying out ads.

More relevant for us is that a narrow column makes it easier to find the start of the next line after reading one. Larger spacing between lines helps; double-spacing isn’t just a trick to push the page count.

Around 2008, New York Times changed the print layout to a wider 6 column layout. The new column size accommodated 40–45 characters. Since fiddling with linotype slugs is no longer a concern, they were able to focus more on readability.

Newspapers – or print in general – is different from reading on the web in terms of distance from the eye to the text. Different reading distances demand different amounts of precision to follow a line of text.

So the goal here is to figure out a width that is short enough that the reader can find the next line comfortably, and long enough to not require excessive effort to read each line. Too long and the reader will break their rhythm with excessive time spent finding the start of the next line. Too short and the reader will break their rhythm with excessive time spent in saccadian movement.

System diagram showing the optimization problem of reading comfort. An increase in line length leads to a decrease in saccadian effort, but an increase in line tracking effort. An increase in line spacing leads to an increase in sccadian effort, but a decrease in line tracking effort. An increase in distance to text results in a decrease in saccadian effort, but an increase in line tracking effort.

What’s the optimal line length for the web then?

Bernard et al. (2003) studied this very question. Their study looked at three different layouts presented on what would be considered a regular desktop screen.

Examples of the three layouts from the Bernard et al. (2003) paper are presented below and are illustrative of what the test subjects saw.

From the paper (emphasis mine):

No differences were found for either reading time or efficiency for either adults or children. However, adults preferred shorter line lengths to full-screen line lengths. In examining perception of reading efficiency, the results were mixed. For adults, the full text lengths were perceived as providing the optimal amount of scrolling in comparison to the two other narrower line length conditions. The narrowest line length condition was perceived as promoting the highest amount of reader concentration, while the medium line-length condition was considered to be the most optimally presented length for reading.

– Bernard et al. (2003)

Beymer et al. (2005) performed an eye-tracking study along these same lines.

From the paper:

Comparing the wide and narrow formatting conditions, our analysis shows that for narrow formatting, subjects (a) read slightly faster, (b) have fewer regressions, (c) retain more information in a post-test of the material, but (d) tend to abandon the ends of longer paragraphs.

– Beymer et al. (2005)

In other words, there’s a slight preference for narrower columns, but there’s no consensus on whether there’s a significant improvement in reading speed. Some evidence exists to suggest that comprehension is slightly better with narrower columns.

Bernard, M. L., Fernandez, M., Hull, S., & Chaparro, B. S. (2003). The effects of line length on children and adults’ perceived and actual online reading performance. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 47, 1375–1379.

Beymer, D., Russell, D. M., & Orton, P. Z. (2005). Wide vs. Narrow paragraphs: An eye tracking analysis. In M. F. Costabile & F. Paternò (Eds.), Human-computer interaction - INTERACT 2005 (pp. 741–752). Springer Berlin Heidelberg.

An Intuition on Shannon Entropy

2025-09-27T00:00:00Z

This post skips over an in-depth explanation of Shannon entropy and only addresses the intuition surrounding it and how it can be misleading. Googling for “information theory” should come up with plenty of options for a more thorough treatment of information theory.

In information theory, entropy is a way to measure “uncertainty” or alternatively a measure of “information”. The field of information theory itself was first formalized by Claude Shannon who’s name comes up often in the literature.

Conceptually information and uncertainty are opposing forces. More of one and you have less of the other.

Lack of information leads to uncertainty. In the absence of any information we are in total uncertainty.

In practice in any given situation where we are concerned with information begins with a well defined foundation which lays out a set of possible outcomes. Each piece of information we receive should narrow down the set of possible outcomes until we reach an unambiguously certain outcome.

If we only have one possible outcome, then we are certain of that outcome. The measure of uncertainty could be thought of as a way of measuring how far we have to go to get there.

Shannon entropy is a measure of uncertainty

A good example of this is the classic game “Guess who”, where you have to guess a person from a set of pictures based on a few pieces of information.

We want to pinpoint a person of interest; which in this case translates to narrowing down the list of possibilities to 1. Each person has an equal probability of being the person of interest. Therefore at the beginning our uncertainty is 1 in 24 — or equivalently \frac{1}{24}.

Let’s look at two possible pieces of information:

wears glasses – matches 5 persons
presents as a man – matches 18 persons

We started with 24 possibilities. Once we know that the person of interest is wearing glasses, we can narrow down the field to 5. On the other hand, knowing that the person presents as a man only narrows down the field to 18. The first piece of information is clearly more valuable than the second. By “valuable” we mean “reduces uncertainty”.

Entropy is a way to quantify –or measure– this valuableness.

Choices of measure for a single outcome

For the sake of discussion, let’s say the probability of a single outcome (call it A) is \Pr(A). And the measure of information learned from knowing this outcome is M(A), where M is some function which we will figure out here.

Adding independent pieces of information should correspond to adding entropy

Channeling some high school math, we know that if the probability of event A happenning is p, and the probability of event B happening is q, and if these two events are independent, then the probability of both happening is going to be pq.

For example, the event person is in the first row (A) has a probability of p = \frac{1}{3}, while the event person is in the third column (B) has a probability of q = \frac{1}{8}. Meanwhile, both of these happenning at the same time (A \wedge B ; where you can read the \wedge as and) has a probability of pq = \frac{1}{3} \cdot \frac{1}{8} = \frac{1}{24}. I.e.

A \text{ and } B \text{ are independent events} \implies \Pr(A \wedge B) = \Pr(A) \cdot P(B) = pq

A good measure should be intuitive. Which is to say, it should behave in ways that we expect. So for example, if we were to take two independent signals, adding them should yield a measure equal to the sum of each measured separately.

\begin{equation} A \text{ and } B \text{ are independent events } \implies M(A \wedge B) = M(A) + M(B) \end{equation}

The simplest way to convert a multiplication like pq to a sum is to … drumroll … take the logarithm. So we could start with M(p) being something like \log(p).

More uncertainty should correspond to a larger entropy

To further intuit, an outcome with a lower probability is more surprising and hence strictly more informative than an outcome with a higher probability, we need:

\begin{equation} \Pr(A) \lt \Pr(B) \implies M(A) \gt M(B) \end{equation}

No uncertainty should correspond to zero entropy

In other words,

\begin{equation} \Pr(A) = 1 \implies M(A) = 0 \end{equation}

because the outcome A is going to happen with certainty.

Putting it all together

The simplest way to make entropy larger for outcomes with lower probability is to negate the logarithm. Doing so has the additional desirable effect that probabilities –which are in the range [0,1]– will yield positive values.

So finally, we have the measure:

M(A) = - \log\left( \Pr(A) \right)

Since Claude Shannon was interested in measuring the information content of digital signals at the time, it made sense to use a more familiar base for the logarithm like 2 instead of e.

So the final equation is:

M \left( p \right) = - \log_2\left( p \right)

The choice of 2 as the base has another nice feature that we can now interpret the information quantity as a number of bits. Because, for example, if we have 5 bits of information where each of the 32 values is equally likely, then any specific outcome has a probability of \frac{1}{2^5} of occurring.

So knowing the outcome gets us M(\frac{1}{2^5}) amount of information; which happens to be 5 — the number of bits we started with.

This measure of information for a single event is called the pointwise entropy, or self information. “Pointwise” because it is measuring a single outcome in the probability space.

Choice of measure for a probability space

Now that we have a measure of the information content of a single outcome, how do we measure the information content of a probability space?

For this, we take another simple formula we already know; which is the expectation. Hence we define the information content of a probability space to be the expected pointwise entropy.

This entropy of a probability space is often denoted H(P) where P is the probability space (defined formally below). For simplicity we can think of P as being a function of some outcome to the probability of that outcome.

The expected pointwise entropy is therefore:

\begin{equation*} \begin{split} H(P) &= \sum_{\omega \in \Omega} P(\omega) \cdot M(\omega) \\ &= - \sum_{\omega \in \Omega} P(\omega) \cdot \log_2\left( P(\omega) \right) \end{split} \end{equation*}

Where \Omega is the set of all possible outcomes.

Formalities

A common critique is that Shannon entropy can be misleading. Let’s see why.

Let’s start with a probability space. It consists of:

A set of possible outcomes. Usually designated as \Omega.

For example, the set of possible person-of-interest consists of the 24 individuals who are pictured above.

Note that here we are dealing with a discrete probability space in that we have a countable set of possible outcomes. This isn’t always the case.
A set of subsets of possible outcomes. Usually designated as \mathcal{F}.

In our example, it could be people with moustaches, people wearing hats, people with glasses, etc. The set doesn’t need to have any semantic meaning. So any group of people from the 24 possible people is a valid subset.

Typically there would be one element in \mathcal{F} which contains all the possible outcomes. I.e. we say \Omega \in \mathcal{F}.
A probability function. Usually designated as P, that when applied to an element in \mathcal{F} yields the probability of that set of outcomes occurring.

For example:

P(\text{wears glasses}) = \frac{5}{24} P(\text{presents as a man}) = \frac{18}{24}

Also, when applied to \Omega (i.e. the set of all possible outcomes), it yields 1 as one would expect since otherwise it would not be the set of all possible outcomes. I.e.:

P(\Omega) = 1

Let (\Omega, \mathcal{F}, P) be a probability space.

So, \mathcal{F} is the set of all possible subsets of \Omega, which is also written as 2^{\Omega}.

\mathcal{F} = 2^{\Omega}

… and …

P is a function that maps from \mathcal{F} to the real numbers in the range 0 thru 1 inclusive.

P : \mathcal{F} \mapsto [0,1]

The Shannon entropy of the probability function P is given by:

H(P) = - \sum_{\omega \in \Omega} P(\omega) \cdot \log_2\left( P(\omega) \right)

Incentives

2025-08-12T00:00:00Z

Teams can fail in many ways¹. But when they succeed, the most critical roles are those that form the communication structures within and across teams. Forming these structures require teammates to have some level of autonomy. In turn this requires a team stucture that allows the team the freedom of movement to reconfigure itself to serve the teams’ goals.

If the conditions are right, a healthy team will fill in the gaps by rebalancing their responsibilities. Such a rebalancing will inevitably lead to an uneven distribution of burden. With time the burden will become increasingly visceral, which will lead to burnout, resentment, and attrition. For an uneven burden to be sustained, that burden must be commesurately rewarded.

Accurately measuring the contributions of each team member is a hitherto unsolved problem. Managers are often left with gut instincts to fall back on, which comes with a laundry list of shortcomings like bias and favoritism. Those who come up with the right formula against all odds are celebrated as good managers.

People write books and give speeches about their favorite good manager. But often the actual skills involved aren’t articulated. The list of skills isn’t long. Primarily these skills revolve around the ability to understand what each person is doing. Secondairly they revolve around the ability to understand how those actions affect the rest of the team. Both require a manager to have visibility into how the team interacts – either by direct observation or by frequent and extensive communication with each team member.

All of this still won’t do much if the team desn’t have psychological safety. Without this, team members may not be willing to recognize and reward their peers. Nor would they be liklely to disclose bad news in a timely manner.

Bad incentive structures lead to people ridigly performing just their roles and letting things fail around them (“it’s not my job”).²

To summarize, good things happen when the incentives for a team:

Never pit employees against each other. One person’s success cannot be another’s downfall.
Assume that the leadership hasn’t figured out exactly what needs to be done. The team should be empowered to course-correct as they learn.
Reward those who pull the team together and help the team communicate internally and externally.

I recommend The Five Dysfunctions of a Team for a nice run through of different ways in which teams fail to function.↩︎
See Intrinsic and Extrinsic Motivation: An Anecdote for an example of what happens when people try to force this process purely by creating extrinsic incentives and assigned roles.↩︎

Ode to Glue People

2025-08-12T00:00:00Z

The concept of glue work isn’t new. Team building, mentoring, running meetings, and many other mundane tasks fall under this category of often uncompensated work. How glue work establishes unofficial communication structures within and across teams is, however, discussed less often.

Not every manager is a great communicator; and that’s okay. This is specially true in large tech companies where the people management ladders are the natural next step for individual contributors who are seeking career growth. When important information that affects people’s careers isn’t communicated in a fair and timely manner, trust collapses taking down the team with it.

Bound by needs for confidentiality and mandatory reporting, managers are often not the best people to confide in or trust. Meanwhile peers with the right set of social skills become speakeasies: people with whom others feel psychologically safe. The level of trust between a speakeasy and other team members can be much higher than with managers.

Consequently, these speakeasies become social hubs where both work and non-work related information flow freely. Some might call it gossip. But when the information is factual and timely, it can help teammates made better decisions, at least for themselves if not for the company. In times of turmoil – like the increasingly frequent “once-in-a-lifetime” recessions that keep happening in the US – the glue people take on a dual role: as a pressure release valve for individuals, and as a vehicle for breaking their sense of isolation.

No surprise that such roles cannot be written down in a normal job description. Also of no surprise that such roles aren’t recognized nor compensated.

Soft Roles as Organizational Cross Beams

2025-08-12T00:00:00Z

Beyond the obvious team scoped function of soft roles, they also perform important functions at the organizational level. Of these two common types are:

Glue-people who are necessary to tie teams together, build cohesion, connect the dots between silos that are not effectively communicating with each other, and establish an unofficial foundation of trust that prevent teams from collapsing when the trust between leadership and their underlings inevitably crumbles.
Immune Systems are the people necessary to stop the company from shooting themselves in their corporate feet. These are the people who internally oppose measures that would cause widespread collapse of user-trust, violate privacy policies, open up the organization to litigation, irreversibly misuse customer data, unnecessarily jeopardize the operational stability or security of critical systems, or commit war crimes and human rights violations.

Emergence of Soft Roles

2025-08-02T00:00:00Z

Google’s early success is partly attributed to the use of commodity hardware that were expected to break occasionally. Rather than demanding perfection from complex systems it is far more effective to expect and design around imperfections¹.

Similarly when we design products, we design them for fallible users who are prone to making occasional mistakes, and beholden to some number of misconceptions.

As we design products, so we must design our internal processes. We once again assume that our teammates are fallible, prone to making occasional mistakes, and operating under some number of misconceptions².

Fallibility runs both ways up and down the org chart. Leaders operate under vague, incomplete, and sometimes wildly inaccurate information³. Further complicating matters, it is impossible to understand the capabilities and shortcomings of each individual person or understand every nuance of what must constitute a role.

The result is that team roles are neither comprehensive nor precise. In any given team structure people are expected to adapt to a broad range of situations that are unpredictable, and frankly unknowable. Some (most?) of those situations are directly due to other team members.

As teams mature, the actual role each person takes on will gradually anneal around individual strengths and weaknesses forged by need and cemented through consistency.

So in reality, what is intended to be a neat and well circumscribed role profile becomes irregular, illdefined, and dynamic.

In short: Following the spirit of designing with imperfections, team organization must always account for this fluidity of roles.

Roles in a team will rarely be circumscribed by the corresponding job description. People will always do work outside their scope.
Roles will overlap in ways that will run counter to the intents of the official role allocations. People will do other’s work occasionally.
Roles will emerge that are not mentioned in anyone’s job description. Sometimes called soft roles.

In this series we will look at a few of these soft roles that emerge often and look into the critical work that these roles perform.

A former Googler, Avery Penwarr wrote an influential document while they were at Google called View from Above about the perspective of executives. Fortunately he also wrote a public version called What do executives do anyway?. The basic premise is that when it comes to any big decision, the executive highest up the ladder is the least informed about the pertinent details – but that’s okay because knowing the details isn’t their job.↩︎
See Postmortem Culture: Learning from Failure for how anticipating mistakes lends itself to a healthy culture where people proactively report and learn from mistakes.↩︎
A former Googler, Avery Penwarr wrote an influential document while they were at Google called View from Above about the perspective of executives. Fortunately he also wrote a public version called What do executives do anyway?. The basic premise is that when it comes to any big decision, the executive highest up the ladder is the least informed about the pertinent details – but that’s okay because knowing the details isn’t their job.↩︎

Draining Our Intellectual Catchment Areas

2025-07-04T00:00:00Z

It is July 4, 2025, and many major universities (Alan Blinder, n.d.) in the US are struggling to deal with threats and funding cuts from the Trump administration.

A major impact of funding cuts is the loss of enrollments of promising students due to fewer programs. International students now face an even more uncertain future where they may not be able to enroll at all despite being accepted (Harvard Staff, 2025), their student status may be invalidated without notice (Big Immigration Law Blog, 2025), they are detained without recourse for exercising their rights (Charlotte Alter, 2024), or they are detained and denied entry to the US (Melissa Hanson, 2025).

In addition, funding cuts have resulted in universities having to freeze academic programs or renege on accepted students (BestColleges Staff, 2025), jeopardizing ongoing research programs. Also as a side-effect, promising academics both in the US and abroad are looking towards universities outside of the US (Sequoia Carrillo, 2025).

Let’s have a look at the role these academic institutions are playing in the US’s industrial dominance, and the long-term effects of hampering their progress.

Migrations of Intellectual Capital

Throughout history a number of events have contributed to the present distribution of intellectual capital. As of this writing (July of 2025) the intellectual capital – especially when it comes to information technology and biotech – is heavily concentrated in the east and west coasts of the United States.

Use the interactive timeline below to explore important events in the migration of intellectual capital to and within the United States.

For a more detailed treatement, see The Accumulation and Migration of Intellectual Human Capital in the United States: A Historical Analysis.

When:

Where:

Decline

Just as a reservoir relies on its hydro-catchment area to replenish their water supply, intellectual endeavors are similarly dependent on “intellectual catchment areas” —-regions that gather and channel information, ideas, and people.

As a hydro-catchment area’s health dictates the long-term viability of its reservoir; so does the richness and connectivity of an intellectual catchment area directly influence the vitality of intellectual pursuits.

Without these intellectual catchment areas, the wellspring of creativity and innovation for a particular field risks drying up.

Over many decades, particularly starting towards the end of the second world war, the United States has became one of the largest reservoirs of intellectual capital¹. While the rise of fascism in Europe drove some precipitation of knowledge into the US in the 1930s and 1940s (L{“a}ssig, Simone, 2017), the intellectual dominance that followed in the decades since the world wars was driven by a positive feedback loop which made the US increasingly attractive to the best minds of the world.

The groundswells of industry established around major universities as well as billions of dollars of research funding from the NSF, NIH, and DARPA created prolific centers of intellectual activity. The Immigration and Nationality Act of 1965 opened the floodgates that were arguably already partially open to international students and skilled labor from all over the world. Out of all this emerged ecosystems of collaboration with regional specializations.

These ecosystems built between industry, government, and academia are collectively the US’s intellectual catchment areas. It took centuries to build what we have now.

What’s more important than what it is, is how the system sustains itself.

And the short answer is that it works because it was working before. We were in a positive feedback loop.

Specifically, the fact that the whole world thinks of the United States as the leader in technology means that all the best minds want to live and work in the United States. The fact that the conditions were favorable for skilled immigrants to study, live, and work in the United States so far means that this dream becomes a reality to millions of people.

It’s not just skilled immigration.

The universities need to continue fostering the thought and collaboration that breeds innovation. Until this year, this positive feedback loop has remained stable. But positive feedback loops can break – especially if one were to violate several of the reasons why the loop exists in the first place.

The ongoing threats against these elite universities at the centers of intellectual catchment areas and the ongoing threats against immigration will undo what took a century to build.

Alan Blinder. (n.d.). Trump Has Targeted These Universities. Why? Retrieved July 4, 2025, from https://www.nytimes.com/article/trump-university-college.html

BestColleges Staff. (2025, June 10). These Colleges and Universities Are Cutting Costs Over Federal Funding Freezes. https://www.bestcolleges.com/news/colleges-university-cut-costs-federal-funding-freeze/

Big Immigration Law Blog. (2025, April 10). The New Risk for Global Talent: F-1 SEVIS Terminations and Student Visa Revocations. https://www.bigimmigrationlawblog.com/2025/04/the-new-risk-for-global-talent-f-1-sevis-terminations-and-student-visa-revocations/

Charlotte Alter. (2024, April 2). These Are the Students Targeted by Trump’s Immigration Enforcement Over Campus Activism. https://time.com/7272060/international-students-targeted-trump-ice-detention-deport-campus-palestinian-activism/

Harvard Staff. (2025, June 12). Federal judge blocks Trump plan to ban international students at Harvard. https://news.harvard.edu/gazette/story/2025/06/federal-judge-blocks-trump-plan-to-ban-international-students-at-harvard/

L{“a}ssig, Simone. (2017). Strategies and Mechanisms of Scholar Rescue: The Intellectual Migration of the 1930s Reconsidered. 84, 769–807.

Melissa Hanson. (2025, June 25). Foreign Harvard students were detained at Logan Airport, denied visas after Trump proclamation. https://www.masslive.com/news/2025/06/foreign-harvard-students-were-detained-at-logan-airport-denied-visas-after-trump-proclamation.html

Sequoia Carrillo. (2025, June 5). International students look to the U.K. instead of the U.S., as Trump squeezes visas. https://www.npr.org/2025/06/05/nx-s1-5415740/trump-international-student-visas-uk-universities

The US is not the largest reservoir of intellectual capital. That honor likely goes to China. However in a number of fields, the US is –or was until recently– the most prolific.↩︎

Act III: Flame

2025-05-07T00:00:00Z

We are prone to anthropomorphizing anything that looks or acts vaguely lifelike. All the way back in the 1960s, when people were confronted with a rudimentary chatbot, they were convinced that it embodied empathy and understanding (ELIZA effect)¹.

Roll forward to the 2020’s and chatbots are substantially more complicated. Unsurprisingly, people attribute more and more human qualities to bots. We saw people stake their reputations on the claim that some AI systems are sentient and therefore have feelings that might be hurt by humans’ relentless experimentation.

There are obvious ethical considerations that follow from such claims. Anthropic recently announced (April 24, 2025) that this is a question they wish to explore in earnest.

Anthropic’s work is very forward looking. Currently, we are nowhere near an answer to the question of whether LLMs are conscious. However there are good reasons why they aren’t.

Before answering the question of whether something is conscious, one must know what consciousness is. We don’t know what that is yet (see The hard problem of consciousness). Our near term goal is to establish what’s called neural correlates of consciousness – observable phenomena in the human brain that are highly correlated with the subject’s state of consciousness. We haven’t yet (as of this writing on May 5, 2025) achieved this near term goal.

Short of knowing what brings about subjective experience, people resort to convoluted experiments like the Chinese room to eliminate situations where consciousness is absent and to point out that nothing along the path we are traversing really reaches any meaningful self-awareness.

Is this picture of a flame hot? There never was an actual flame in this instance because this image was generated by AI. The AI may have seen pictures of many real candles, but it has never felt the warmth of a candle. It is therefore safe to assume that the sense of heat played no role in this picture.

What if we start with a more realistic looking model of a flame. Then we can calculate how the flame should behave² with time. No matter how detailed this model is, the model does not constitute a flame. We can crank up the fidelity and accuracy of our calculations as far up as technology allows, but it will still not burst into flame³.

This is where we are with AI and consciousness. We have descriptions of how to determine the next word in a sequence of words. These descriptions are extremely detailed and based on unfathomable amounts of information. They are so complicated that nobody knows how they behave, so the only way we can find out what they will do is to run them against some input and see what happens. We know these models can be made even more complicated than their current, already incredible, complexity.

But no matter how far we go, we still only have a description and a thing that can interpret this description. We can increase the complexity of the description or increase the speed of the interpreter, but there is no setting on this continuum where information suddenly becomes self-aware. It is intuitive that the description will not achieve any physical accomplishment no matter where we take it. So the mechanism of consciousness must depend on the implementation of the interpreter. Currently, that too has no “ignite consciousness” threshold.

So this post will also end the way Anthropic’s announcement does:

“There’s no scientific consensus on how to even approach these questions or make progress on them. In light of this, we’re approaching the topic with humility and with as few assumptions as possible. We recognize that we’ll need to regularly revise our ideas as the field develops.”

— Anthropic

ELIZA is fascinating in how simple the whole system is while being seemingly complex. You can play around with some modern web-based simulations like this one. See if you can figure out how it might be calculating its responses.↩︎
This is a very broad area of research. Search for flame dynamics. E.g. Flame simulations with an open-source code - ScienceDirect.↩︎
Technically if the equipment heats up enough, then it might actually burst into flame. But that has nothing to do with the specifics of the flame we are trying to simulate. It can just as well happen while trying to factor a large prime.↩︎

Act I: Meaning

2025-05-04T00:00:00Z

It doesn’t look like much because I haven’t told you what it is yet. That’s going to come in the following paragraphs. All I’ve done is toss an abstract concept up in the air. It seems pointless, but it gives us a canvas to paint with meaning later.

For example, if I tell you that this meaning concerns a feeling – something very abstract – it suddenly looks a tiny bit different. Now look at it.

Not all the words I speak will add meaning equally. Some are digressions that you have learned to ignore like this sentence that isn’t talking about our thing at all. In one ear and out the other. You won’t pay much attention to those.

Still others, like the sentence that is about to follow this one, are in fact talking about our thing, so you will pay attention to those. Our meaning, the one we are talking about, is going to be about a feeling – a calm feeling. Something that you feel when you suddenly run out of ideas or your mind stops racing and you become aware of the environment around you.

The feeling you get when you are in the middle of a forest. A calm forest where you are not worried about any creepy-crawlies. It’s the afternoon, but not a lot of sunshine is making it through the canopy. You can smell the moss – it feels wet and earthy even though you know you can’t smell wet.

Unlike two steps ago, these last few sentences were a bit special. You see, up till now I tossed out a very plain and boring thing in the air and painted it with a couple of distinct and well understood – if a bit vague. But in the last paragraph, I made you dip a brush into your own subjective experience and paint our thing in your mind. The meaning is now yours and yours alone.

And that’s special because your experiences are not things that can be communicated to you. We don’t have language for that – because those experiences cannot be constructed innately. They must be communicated to you through your senses. Those bodies have ways of constructing feelings for you that mere abstract meanings cannot. However once you have those feelings in yourself, our abstract words can conjure them out at will; just like we did when we spoke about the forest.

If we had a succinct way to communicate this meaning in one go, we would have.

But our language and our facility of communicating ideas which we stimulate with our language doesn’t afford us such shortcuts. So our language tends to be very abstract; and sometimes it takes a little shortcut and conjures up a meaning from the back of the subjects’ mind like we did with our allusions. Again, these are not always shortcuts. These are the only ways in which we can communicate certain things – things that can only be constructed in one’s head with one’s senses.

So to understand these ideas, one must have the right ingredients in their minds. To create these things in their heads one must experience things with their senses.

And if we can’t conjure meaning then we can’t understand each other. And if we can’t understand each other then we can’t have compassion and empathy.

Interestingly, this is how we fundamentally differ from the current LLMs. You see, they have another problem. They don’t have any experiences to conjure because they don’t have feelings. So when they encounter language, they can’t understand what those words mean. But one thing that they can do is to reach into their vast embeddings and see how someone who can feel would have responded. And then they can respond accordingly. What they conjure isn’t life experience or feeling, but evidence of others having reacted a certain way to a certain sequence of ideas.

Because they can’t construct new feelings, their universe of feeling will always be limited by what people have felt and reacted to. That way they can look at those reactions and imitate them. The meanings in your head are yours alone.

No one can see your dreams. Especially not the robots.

Intermission - Q1 of 2025

2025-05-03T00:00:00Z

It’s been a while since I’ve published anything here. This is partly because every time I try to write there is something much more important and terrible happenning in the world that I’m not qualified to write about. Everything else including whatever it was that I was trying to write about seems trivial and inconsequential by comparison.

Many years ago I’d have said that there is very little evil in the world and that most people you will meet are good. I still sort of believe that people are good, but I’m beginning to think that the word “most” may not be readily applicable.

School children has this would-you-rather question about a big red button.

“Hypothetically, if there were a big red button that when pushed gives you a million dollars, but also kills a random person, would you press it?”

It is safe to say that the answer to this question is predominently yes.

More worryingly, the reward doesn’t need to be nearly that high. It seems, for some, there doesn’t need to be a reward at all as long as the person who would be harmed is of a specific race, sexual orientation, body dysmorphia, where they were born, or religion. There’s enough hate to take the place of a reward.

People aren’t born this way. Decades of being conditioned to cast out a group of people gradually dehumanizes them. With enough rhetoric, likemindedness, and reinforcement anything can be normalized. Anyone disagreeing – and is not a member of the outcasts – can be easily disregarded as being the enemy or at least misguided. Hate becomes self-sustaining, and can last generations.

We can win battles or wars, but we are not going to really move forward as a species unless we figure this one out – how to teach compassion, and how to reason someone away from hate?

Contrary to popular opinion I think pushing the hateful away is not a sustainable solution in the long run. Suppressing hateful behavior without corrective guidance has historically just made the hateful group more resentful and even more entrenched in their beliefs. Ultimately they will – as evidenced in recent events – have enough momentum and intensity to turn the tables.

Preface

2025-05-01T00:00:00Z

Large language models’ ability to mimic human language and show signs of intelligence is quite strange yet makes sense at the same time. This series is a very gentle introduction to how they almost capture human experience through language and how language is perhaps the weirdest invention of all.

It’s going to be a bit of a long ride. Perhaps by the time you read it I haven’t quite finished the whole series. So you’ll see what looks like an abrupt ending or missing episodes. They are coming. Just slowly.

Onwards …

Intrinsic and Extrinsic Motivation: An anecdote

2025-03-22T00:00:00Z

A long long time ago in a totally hypothetical company that still exists but only in name, there was a little bit of discontent¹. You see, one of the core values of the company was its commitment to diversity and inclusion. Rather than lowering the bar, like how some people think inclusion works, they had many outreach programs that encouraged women and minorities to take up STEM. These outreach programs almost necessarily fell on the shoulders of the women and minorities who were already employed at the company – after all representation matters. Those who worked on these did so out of their own passion. They wanted to help others like them.

Anyone who has worked in tech will tell you it’s a lot of work. This company’s performance review system attempted to measure the amount and quality of said work so that they can reward the right people for the right reasons. Since the company deemed that having a diverse workforce was in their best interest, ideally they would reward people working towards that goal. However, the performance review system didn’t measure anything about volunteer work like what the people doing outreach were doing.

So those volunteers were unhappy. All that extra work, often done at the expense of work that is measured in the performance review system, didn’t count for anything when career progress was concerned. They wanted change.

The company listened.

“Community contributions” got added to the job descriptions for people at all levels. Naturally the higher the level, the broader in scope those contributions were required to be. It was now part of everyone’s job to make contributions that made the company’s culture better – be it recruitment, outreach, or whatever.

The changes to the performance review system was an improvement. But unintentionally it had other side-effects. Notably, those higher up the organizational ladder found that their community contributions need to be commesurate with their status. But starting up an effort that is at the company or region level was a lot of work. Lucky for them, those volunteers who were passionate about helping others had already established programs that were of the right scale.

So as career focused people are wont to do, they threw their weight around and effectively took over these outreach programs. The volunteers were upset. Some left. Others kept working but at a smaller capacity than what they were before. The career focused people were happy, but the volunteers were not.

The outreach programs still existed² but were run by people who were no longer doing it just to help other people. They were run by people who were trying to help themselves. The programs faltered; quality went down; the effectiveness of the programs went downhill. It wasn’t fun or rewarding as it was before.

There’s a lot of discontent now, but at the time there was only a little.↩︎
As of this writing (February of 2025), not any more.↩︎

Buffer Bloat

2025-01-03T00:00:00Z

From Here To There On The Internet

Devices on the internet including your phone, laptop, gaming console, refridgerator, etc. communicate by exchanging data “packets”. They are of course not physical packets. It’s just a useful way to describe a unit of data sent over a wire or some other network connection. A packet could be thought of as, say, a single frame in a video or a small slice of audio, or a couple of paragraphs of text in a webpage. Devices on the internet will take a big chunk of data (e.g. a video clip) and break it down into smaller pieces (e.g. a single frame in the video clip) and then sends each of these pieces along the network. The receiving device puts the pieces back together to reconstruct the original thing (e.g. the video clip again)¹.

Each packet has little pieces of extra information to help it get to where it needs to go. One piece is where it comes from (a source “address”). Another is where it is going (a target “address”). Devices along the network (like your internet provider’s networking equipment, satellites, etc.) use this extra information to correctly route the packets to their intended destination.

The network is large and constantly changing. So each device focuses only on getting the packet at least one step closer to its destination. The specific journey taken by a packet is unkonwn until it is on its way.

Faster and Faster

So let’s say a video streaming service wants to send your mobile phone a video. First the streaming service breaks up the video into small packets. Then it throws all the packets into the network so that the network can deliver them to your phone.

If the streaming service pushes data out at exactly the right speed and the network doesn’t break, then everything works out great. All the packets make it to the destination without hassle. But neither the streaming service nor your phone know what that right speed is due to the aforementioned unknowability of the network.

Sending too much data can overwhelm some devices or connections in the network. If a device gets more data than it can handle, it’s likely that it will start dropping packets until it can catch up.

Sending too little data isn’t great either. The speed might be too slow for streaming video. The receiving device will underperform. Users will be confused why their expensive broadband connection doesn’t let them watch streaming video.

So how does the device know how much data to send out?

Flow Control

The mechanisms we use to regulate the flow of data is called “flow control.” We are going to look at a method based on coordination between the sender and the receiver because they are the only two actors in this scenario that we consider to be reliable and stable.

Different packets can take different paths See the network connections between C-D and C-E in the diagram above. Either of these paths can take a packet from C to F. Packets going down different paths will arrive at different times. Also, if the network gets overwhelmed or breaks, then some of the packets may not make it at all.

The reciever needs to know how to put the packets back in order and tell if any are missing.

To keep track of which packet is which, we can add some new information to the data packet: let’s call this the sequence. As you might imagine, the sequence can start at 1 and count each packet until the transmission is over. Each packet now has a unique position in the list of packets.

Great. But how do we know which packets made it to the other side? The recipient will need to let us know this information in some sort of reply. This reply is also a data packet called an acknowledgement. It’s sent from the original recipient back to the sender.

Ok so now we have all the information we need to perform some flow control.

We can now figure out the optimal rate of data transfer experimentally using some method like this:

Send data as fast as possible.
If the recipient claims some packets were missing, then resend those packets and send packets slower than before.
If the recipient hasn’t complained of missing packets in a while, try sending them faster.

Stuck On The Wire

Now imagine you sending data out - a rate of 100 packets per second. And let’s say it takes one second for a packet to get to the other side. Just before the end of the first second, the destination hasn’t received the first packet yet. But the sender has sent 100 packets.

Where are the packets?

We call this “data on the wire’. Some data is indeed on the wire literally, but that tends to be a tiny amount. Most of the data on the wire is actually at each of the devices along the way either being received, or waiting to be sent.

So once we factor in the equipment in the middle, your path to the destination looks a little like this:

The device sees some data come in, stores it, figures out where to send it next, and sends it on its way. There could be other stuff these boxes could be doing, but that’s something for another post.

So from the perspective of a single box the world looks a bit like this:

If this is the limit for the network and the sender keeps sending more packets, then there’s no room for the packet. So it gets “dropped”.

It is common for a device to be at the border of a “good” network and a “not”good” network. The goodness here is the amount of data that can be stuffed into it.

When a burst of data comes along, not much data can be stuffed into the small pipe before no more can fit.

As mentioned before, this means that the sender will need to re-send the dropped packets and also slow down the rate of data. Given enough time the rate of data transfer will approach the throughput of the smallest pipe along the way.

Buffer Bloat

Some folks looked at this setup and decided to improve things. There are some valid reasons for doing what they did – which is why so many people did this. But it didn’t quite have the intended effect.

So, what was the thing they did? They added buffering. Too much of it.

A buffer is a resevoir where excess packets can be kept until the next link in the network is ready to accept more. What this means is that every device along the way is trying to do flow control by storing and resending packets instead of letting the sender know that packets are getting dropped.

Device B is dropping all excess packets into a buffer.

As before, let’s say the sender is sending 200 packets every second. The network still can only transfer 100. The excess packets – another 100 of them – gets tossed into the buffer.

What happens just one second later?

The very next packet – let’s call it Bob – that the sender tosses into the network will end up in the buffer since B is still waiting for some of the first 100 packets to get past C.

Bob is the last packet in the buffer and must wait for all the other packets in the buffer to be send before it will get queued to sent out.

In another second, the first 100 packets make it to the target, and now the 100 packets that were in the buffer are back on the wire.

Bob is still in the buffer but it’s the next one in line to get back on the network.

Once back on the network it will take another second for Bob to reach the target.

Even under ideal conditions eventually the system settles on a new equilibrium with one difference.

The length of time it takes for the data packets to make their way to the destination is much larger. Packets accumulate in buffers along the network. Each buffer adds an additional delay for the packet.

Overall the throughput of the network is the same. But latency – the time it takes for a packet to get to the destination – is much larger.

The effect is even worse in practice because this setup resists any attempt at flow control. It delays the discovery of actual throughput.

This phenomenon is called buffer bloat.

This is obviously a massive oversimplification. But doesn’t change the thesis presented next.↩︎

Fourteen Years

2024-12-31T00:00:00Z

It’s been fourteen years since I joined Google. Much has happened since then, and people think the old Google is dead. This is somewhat correct.

One way to understand what is happening is to compare and contrast the Google of yore and the present (as of late 2024) Google.

From Illegal Flower Tribute

Perhaps the most defining moment for Google around 2010 was Google Search’s exit from China. The reasons for the exit and the aftermath are many. The high level reason was due to non-capitulance to the Chinese government’s demands for censorship and compromising user data.

In addtion, Google’s hand was forced by Operation Aurora, a cyber attack on multiple US based companies including Google orchestrated by Chinese state actors.

The fascinating accounts of the events surrounding Operation Aurora is covered in a documentary called “Hacking Google”.

Hacking Google - YouTube

Amazon and Microsoft were in a similar situation as Google. But they were unwilling to turn away from access to the Chinese market. There was a palpable sense of pride that we were working for the right company.

To Project Nimbus

Time changes everything.

Eight years later Google reneged on its stance against censorship by attempting to build a censored search engine called Project Dragonfly. The seeds of capitulance was well under way by this point despite the fact that the censored search engine project was ultimately dropped.

Fast-forward a few more years and we get to Project Nimbus. Google signed the contract despite warnings by its lawyers that “Google Cloud services could be used for, or linked to, the facilitation of human rights violations…”.

One could argue that tools like Google Cloud services should be value-neutral. But even then, what matters here is that Google intentionally changed the terms-of-use for this contract to allow uses that were explicitly called out as going against Google’s values.

That’s Where We Are

And that’s where we are right now.

Google has had some rough years during which its relationship with the rest of the tech ecosystem and the world in general has changed. The old “focus on the user and all else shall follow” matra no longer applies. The company’s focus is now exclusively inwards where the fact that something benefits everyone is no longer justification to continue working on it. For example, Google laid off most of the core Python team. While it may seem like a move towards cheaper labor, it is more than that. The previous Python team were also industry veterans who were effectively in the service of the broader Python community far beyond Google. The replacements are people who are tasked with maintaining internal tooling and not much else.

Google is no longer the giant benefactor of technology for everyone. However it’s still better than the other big tech companies. I keep telling myself that.

I Moved This Blog To Eleventy

2024-12-13T00:00:00Z

Astute observers of this blog may have noticed that the blog looks suspiciously different lately. That’s because I changed the static site generator I was using.

Previously the static site generator I used was Hugo. This arrangement worked for a while but it had some shortcomings.

Growing Pains

I was using Pandoc to convert Markdown to HTML. Hugo’s support for Pandoc had some limitations. For example, you can’t specify options for Pandoc or use a bibliography. When I moved the blog to Hugo, I lost some of these features. Attempting to close the gap I authored a patch that added support for configuring Pandoc.

Pandoc is extensible via filtering the AST of a document through a series of programs which can modify it. There is no restrction on the kinds of programs that you can use.

If Hugo allowed configuring all this via frontmatter, that may introduce some tricky security issues.

Say you download a Hugo theme from the web and try to build your site. A theme could include frontmatter that causes Pandoc to run an unsafe binary. That’s not great. So it is understandable that this kind of feature isn’t a part of Hugo proper¹. While my patch worked, it was safe only because I controlled every aspect of the blog.

While tinkering on this was fun, it created an additional headache. Namely, whenever and whereever I wanted to build my blog, I had to maintain a custom build of Hugo. After moving my build a couple of times maintaining a custom build became a painful chore.

The Exodus

So started the process of looking for an easy-to-use static site generator (hereafter SSG). My requirements were:

The SSG should support Markdown.
Markdown should support some of the Pandoc features I was using, or at least have the flexibility to extend support for those features.
Making this work should not require any custom patches.

Eleventy was almost a no-brainer. People had high praise for it, including developer relations folks at Google. So I felt like that would be an easy move.

Eleventy’s Mardown support is based on Mardown-It. One thing that I liked about Markdown-It from the start is that there appeared to be people who are extending it to support various features of Pandoc. In fact I found plugins for basically all Pandoc features that I used in this blog. Everything was looking quite rosy.

Templating

WebC + Eleventy Is Broken

Eleventy supports a templating language called WebC. It follows the web-component way of defining layouts, but can avoid much of the custom element overhead by opportunistically generating plain HTML. I had already accepted that switching the SSG meant rewriting layout templates, and using web components to do that seemed like a good idea².

A couple of standalone WebC based templates worked fine. But the moment I had nested templates, Eleventy hung and eventually crashed. I didn’t want to go on a side quest to debug it, and ultimately that would’ve likely led to be violating the “no local patches” rule. So that was that for WebC. If WebC was more robust or just worked, it would have been my choice of template engine.

One cool feature of using WebC with Eleventy is that bundling works trivially. For example, if you have a <script> tag or a <style> tag in a component, WebC automatically pulls the content into the relevant bundles. All other templating engines supported in Eleventy (save for third-party engines that I didn’t try) require manual labor to get stuff into the bundles.

Nunjucks + Eleventy Works

The next templating language I looked at was Nunjucks. I had briefly looked at Liquid as well, but for my purposes Nunjucks and Liquid seemed to be quite similar. Nunjucks won out because it had safer defaults³.

Handrolling

Pandoc Again

I tried to rely only on Markdown-It for this blog. But very soon I had a big stack of extensions to support the various features I needed, like supporting <div>s, bibliography, and attributes. While the Markdown-It ecosystem is extensive, it also introduces a lot of confusion. For example, there are multiple packages that all provide the same functionality and also based on the same base package. Keeping track of all this was starting to be a pain. Also, I had trouble getting block level containers working.

After a while I decided not to waste time trying to replicate features that already existed in Pandoc. So I wrote a little library that invoked Pandoc within Eleventy to convert Markdown to HTML. It was a little slower, but worked out great.

Bundling Files

While it is possible to add content to bundles (like CSS and JS), curiously it was not possible to stuff files into said bundles. The suggested method was to do something like this:

  {% css %}
    {% include "some-file.css" %}

    body {
        // other CSS.
    }
  {% endcss %}

This seems like a good idea, but there is one crucial issue: The contents of the included file (some-file.css in the example above) is also processed by the same templating engine. If the file doesn’t contain any sequences that look like template tokens, then we are good. But this is asking for trouble.

So I wrote a custom tag called bundleFile that can take the contents of a file and stuff it directly into a bundle. The earlier example now looks like this:

  {% bundleFile "some-file.css" %}

  {% css %}
    body {
        // other CSS
    }
  {% endcss %}

SASSy

Eleventy doesn’t support SASS out of the box, but there are examples in the documentation for how you can set it up. But all of the examples assume that you have SASS or SCSS files that you are going to conver to CSS, and then serve without bundling. I wanted to be able to compile SASS and stuff the resulting CSS into a bundle.

So alongside the bundleFile shortcode, I added a bundleScss shortcode so I can do stuff like this:

   {% bundleScss "my-styles.scss" %}

Katex

I like Katex for my math typesetting. Pandoc supports it out of the box. But the rendered HTML needs to invoke Katex in the client to render the math. I wanted to avoid this step entirely and try to render Katex on the server.

This was a pain to do with Hugo, but since Eleventy was alredy runing in Node, server-side rendering is a breeze.

Conclusion

Overall I found Eleventy fun to work with. My gripes are mostly with the documentation which lacks a reference. It’s a pain to go through tutorial-like documents to find out what exactly gets passed to a shortcode or what to do when a shortcode adds a dependency. I made way too many trips to the 11ty GitHub repo. The right number of required visits should be zero.

In the coming weeks I’ll package Eleventy Pandoc support and some of the shortcodes so others can use them.

There are ways to mitigate this problem. For example, we could restrict which flags can be inherited from themes, or require that certain options can only be set from the top level hugo.yaml file that the user controls.↩︎
Not quite all the way true because use of shadow DOM and custom elements was not a requirement.↩︎
The defaults I’m referring to is what happens with a tag like {{ foo }} where the contents of the variable foo would be inserted into the document. Nunjucks will treat the contents as unsafe and requires manually opting into unsafe injection via {{ foo | safe }}. Liquid will insert the contents with no escaping.↩︎

Proof of Work

2024-07-12T14:56:15Z

Proof-of-work refers to a set of methods by which some entity (a person or a computational system) offers proof that the entity performed a substantial amount of work. Hereafter, I’ll assume that you—dear reader—are somewhat familiar with the concept.

Measures like proof-of-work (PoW) are a necessity in intentionally slowing down otherwise free and uncontrolled operations. PoW’s most extensive critique is the resulting high resource consumption that lacks a corresponding commensurate positive contribution.

Working with a poorly thought-out, or overengineered technology stack, or an overly bureaucratic organization is also a form of proof-of-work.

This claim seems harsh, but even the intentionality of PoW can be observed in organizations.

The first tool people reach for wherever some form of mismatched trust exists is to introduce bureaucracy. This approach makes sense because what better way to prevent abuse of authority than to make the person at a lower trust level ask for permission from someone with a higher trust level? So, each trust issue one encounters results in one more approval hurdle – affectionately referred to sometimes as a “governance gate” or an “approval gate.”

Now, take a step back and imagine how the proliferation of approval gates plays out in a large organization like Google. All organizational activity now involves debilitating bureaucratic overhead. Proliferation of approvals also have the effect of piling on the responsibility of approving upon people who are either unqualified to sufficiently assess the request or simply have no time to go through each in detail.

Unless each approval gate adds substantial value or prevents harm, each such approval decays into proof-of-work –

the only point of getting the approval is to show that you have gone through the work of getting the approval.

Bad frameworks and tech stacks have the same effect. Onboarding onto such a stack is a time-wasting nightmare where people learn non-transferrable skills that they will most likely only use once. Each step in getting something done involves putting together poorly documented and poorly designed pieces and trying to express business logic in leaky abstractions that muddle the separation between intent and implementation. Sprikle an inordinate amount of approval gates on top, and you have our modern large tech company.

The outcome of weeks of struggle is the same as what is achievable within a matter of hours with a better tech stack. You haven’t produced work; you’ve only produced proof of work. The outcome isn’t commensurate with the effort.

How to Solve a Problem in Three Steps

2024-07-08T01:09:29Z

One fun thing with having kids is that you get to repeat all the old stupid jokes to a brand new audience.

Here’s one:

How do you put an elephant in a refridgerator in three steps?

Open the door.
Put elephant in the refridgerator.
Close the door.

It’s funny because the purported “answer” doesn’t address any of the important questions raised by the initial question; not the least of which regards the mismatch in geometry between a normal refridgerator (a household one that immediately springs to the minds of the audience) and the average live elephant. It’s funny to us becuase we are familiar with both of these concepts enough to immediately see the problem. So we immediately prioritize the critical concerns.

We all get it. But then we are somehow completely blind to the fact that …

… this is what our OKRs look like after a planning sprint.

At Google, as it is with many other large and aging tech companies, planning of this sort happens in middle management.

The requisite attributes for doing long arc planning are …

… authority, domain expertise, and ability to strategize. Choose (at most) two.

Domain expertise concerns the understanding of the problem area that is being tackled.
Ability to plan (or strategize) covers not just having a vision, but also enough understanding of the tools that are available to derive a set of strategies that will achieve a desirable goal.
Authority is the propety of being in the right place in the organizational hierarchy to exert control over a group of people who are capable of executing on the strategies. I.e. being in management at the right level.

In tech, people get to management either by being promoted out of their zone of competence or by bullshitting their way up the management chain. Hence the distribution of the authority attribute doesn’t necessarily correlate with either of the other two attributes¹.

So you end up with a layer of middle management that share the authority attribute, but at most one of the other two.

But in most cases “planning sprint” is code for “prepare for damage control” for those who have domain expertise and ability to plan attributes but not the authority attribute.

When people who understand neither refridgerators or elephants decide to solve problems, the outcome usually looks like the joke we mentioned before; but with the added injury of unrealistic timelines.

Worse, when the strategies are spelled out without understanding the problem or the purpose of a strategy, the outcome is that those those doing the work must frame their work in terms of putting something into a refridgerator in three steps.

This process is extra fun when the planners have decided to put not one but two things in the refridgerator without being specific about what they are. We know one of those things must be an elephant for the long term viability of the product, but the other one could be an orange for all we care. So the workers now have to frame “solving the elephant” in terms of “putting the elephant in the refridgerator”.

But the middle management layer continues to consider the orange and the elephant as being roughly equal problems. Hence those who play the ladder game know to work on the orange and not touch the elephant.

It is surprising how few good decisions need to be made in order to keep a multi-billion dollar enterprise chugging along – at least as far as investors are concerned. You should begin to understand why a company with so many smart people and so many brilliant creations can make so many self-defeating mistakes.

What is not suprising is the amount of fat that constitutes the middle management of these companies. The thermocline of truth ensures that those looking down from the top will never see the fat for what it is.

I can attest to the fact that there are people who only have the authority attribute. It is a thing.↩︎

Asian Is Not a Useful Category

2024-07-04T16:34:41Z

Asking if someone is “Asian” in a question of ethnicity or demographics or pretty much any context is bad. There is no reasonable conclusion you can arrive at about a person based on the fact that they are “Asian”.

Without knowing anything about the precise demographic breakdown if you were to learn that a group of people consists of 10% Asians what can you confidently conclude? What possible useful thing can you derive from this information?

First of all, let’s have a look at a map:

Here’s a partial list of countries that are in Asia (based on this Wikipedia article and includes contested territories):

Abkhazia, Afghanistan, Armenia, Azerbaijan, Bahrain, Bangladesh, Bhutan, Brunei, Cambodia, China, Cyprus, East Timor, Egypt, Georgia , Hong Kong, India, Indonesia, Iran, Iraq, Israel, Japan, Jordan, Kazakhstan, Kuwait, Kyrgyzstan, Laos, Lebanon, Macau, Malaysia, Maldives, Mongolia, Myanmar, Nepal, North Korea, Northern Cyprus, Oman, Pakistan, State of Palestine, Philippines, Qatar, Russia, Saudi Arabia, Singapore, South Korea, South Ossetia, Sri Lanka, Syria, Taiwan, Tajikistan, Thailand, Turkey, Turkmenistan, United Arab Emirates, Uzbekistan, Vietnam, and Yemen. {.smaller}

So if you pictured a room with a bunch of Palastenian, Chinese, Philipino, Russian, Indian, and Japanese people you would be kind-of right. But what do Asians look like anyway?

The point is, Asia is a big place. There are many countries/states/territories. There are many races, religions, languages, phenotypes.

There is so much variety in “Asia” that any conclusion you would make based on a person or group of people being “Asian” will be wrong.

Please stop asking people if they are Asian.

So what should I ask instead?

Don’t ask in the first place – if not absolutely necessary. This question is asking for trouble.
“I prefer not to say” should always be one of the options. While there are legitimate reasons why you would ask for some demographic information from a person, always give an opt-out.
Figure out what you actually want to know. Knowing if someone is Asian is not going to give you that information for reasons stated above. But knowing what you want to know will lead you to a better question.
- Perhaps white/black/other is all you need for a question about ethnicity.
- For nationality, provide a list of countries but don’t assume anything about what that person looks like or what language they speak or what religions they might follow.
- For geography, at worst ask for a region within Asia¹ instead of just Asia. And for that matter, the same goes for other continents.
If “Asian” still remains as one of the response choices for your question, you are doing something wrong. Not even Google internally gets this part right. Give up. Don’t ask if someone is “Asian”.

See the UN’s Standard country or area codes for statistical use (M49) for a list of geographic regions and various groupings of countries.↩︎

The Have-Loop

2024-06-24T01:45:28Z

This is the Have-loop.

Positive feedback loops are neither new nor rare. The ones we are concerned with, however, are those where possessing one thing leads to possessing more of the same thing. Even more specifically, the have-loop concerns itself with possessing something that is in limited supply. Such a loop leads inevitably to concentration of said resource.

If we consider education to be a profit-center¹ as is the case in the United States, then education becomes part of a have-loop across generations. In other words, the passage of time will concetrate education on a shrinking educated class.

One example of this is wealth – because wealth begets wealth. Inevitably this leads to concentration of wealth to an increasingly smaller group of people². There is no natural equilibrium state absent external regulation.

Have-loops are inherently unsustainable. Have-loops inherently lead to concentration of resources.

Wherever society allows a have-loop to exist, inequality will inevitably grow.

The terms “cost-center” and “profit-center” typically refer to businesses. But they can also apply to societal policy where as a society we decide that certain things should be done for profit while certain other things should be available to everyone.↩︎
People tend to focus on income inequality, which is an important factor. But another factor that drives extreme wealth concentration is hyper-extraction. Much like extraction of natural resources, hyper-extraction can also be directed towards individuals and entire demographies. Measures of income equality will eventually fall against another’s ability to enact hyper-extraction.↩︎

When Your Feature Gets Someone Hurt

2024-05-22T00:00:00Z

A story making the rounds recently opened some disturbing implications. The New York Times ran the story as Israel Deploys Expansive Facial Recognition Program in Gaza which says in part:

To supplement Corsight’s technology, Israeli officers used Google Photos, the free photo sharing and storage service from Google, three intelligence officers said. By uploading a database of known persons to Google Photos, Israeli officers could use the service’s photo search function to identify people.

Whether this particular use of Google Photos is compliant with the usage policy is irrelevant. Those are concerns for PR firms and lawyers. Google Photos is a consumer product. People use it for tagging their family and friends in their vacation photos. The product works pretty well for this purpose.

The humanity behind the product developers and those who are unexpectedly and unwittingly blindfolded and led away to some unkonwn prison somewhere is an entirely different matter.

The intended usage of a product – especially software products – matters greatly. People responsible for these features decide its fit for purpose based on this intended usage. From their point of view, a mislabelled photo is an inconvenience; possibly a funny one. The error thresholds were never decided based on whether someone could lose their life for it.

Some developer wrote a conditional branch without giving it a second thought. And now it decides the fate of a human life. That’s quite the turn of events.

There’s no moral here. People are going to use your products in ways that you never intended. I bet the people who designed coins have no idea the extents to which those coins were flipped for. And so it goes.

Secret Life of Systems

2024-03-23T21:59:29Z

Software systems aim to solve real world problems. They implement some behavior that the authors of the system believe would address the problem.

For non-trivial systems the coupling is imperfect between the real world problem, the problem as understood by some group of people, the perceived solution, the solution as it is actually implemented, and the solution as received by the users.

The diagram below illustrates this phenomenon.

The software¹ system life-cycle begins – as most things do – in the real world.

The Problem (P)

There is a problem that affects some group of people². Let’s call it P.

We know these problems exist by their symptoms. For example people might just feel something isn’t pretty great or that things could be better if only we could figure out how.

Beyond that a precise and comprehensive definition of the problem doesn’t exist. This might seem like a strong claim, but it comes down to the fact that everything involving people is complicated.

The Model (M)

So people set out to understand the problem deeper. Experimentation, user studies, interviews, observations, are all things that we do to conjure these invisible problems into existence. We go probing for symptoms to understand what might be causing them.

Based on these we develop and a model of the problem. The model may be laid out for everone to see or it could be in someone’s head.

Let’s call this M for model. The model is the entirety of our understanding of the problem. It is a prerequisite before we can theorize about …

The Solution (S)

At first the solutions are themselves models – things that work in theory when evaluated against the model M, but yet to be verified against the real world problem P

The models of solutions – let’s call them S – are what we think will address what we assume is the problem. Depending on the level of rigor, we will draw little diagrams, write design documents, shop our ideas around, discuss alternatives, collaborate and refine these solutions.

Eventually or during this process we will put something together that works somewhat.

The Implementation (I)

So an implementation comes together – let’s call it I. The difference between S and I stems from practical matters. Time, resources, technical limitations, all affect how much of our model solution S we can implement. The result is often different from what we thought we would eventually have.

Deployment (D)

Whatever solution we put together is useless unless we take our masterpiece to those who are experiencing the problem – the users.

The likelihood of a non-trivial solution reaching its entire intended audience is low. The implementation may not be received in the same spirit in which it was built.

And often, this is the point at which we will find out how much our solution matches the needs of the real world.

They way in which our implementation I interacts with the real world is – the deployment D.

Much like the subject at hand, systems lifecycles don’t follow neatly delineated phases as described. Real systems typically exist in all phases simultaneously. Understanding the question is a continuous process. Plans change all the time. As any engineer will tell you, what tools and resources are available are also part of the problem statement.

But there are a few key observations that you can make from this model:

We lose information at every phase. Our model of the real world problem is guaranteed to be of much lower fidelity than the real thing³. Our model solution is going to fall short of addressing our model solution. Our implementation will not match our aspirational solution. And whatever we manage to get in users hands isn’t quite working the way we thought it would.
Understanding the problem is a continuous process. Every phase of this lifecycle brings with it new insights into what it is we want to achieve and what we are working with. Each new insight changes our problem statement, solution, and eventually implementation and deployment.
Understanding the problem must be deliberate. Given that the deployed solution isn’t quite a match for the real world problem, we need to ensure that we have the right measurements and probes in place to understand this mismatch – be it telemetry, user satisfaction surveys, interviews, or whatever else.

Whenever we are holding some concept, being conscious about where in the lifecycle that thing fits helps us get a sense for the level of maturity of the concept and where you need to be to solve the problem.

The concept applies equally to non-software systems – as long as they aim to solve real world problems.↩︎
Problems that we are interested in are those that affect people. They tend to by squishy and ill-defined. Problems that are defined entirely in the theoretical relam don’t fall into the model that we are going to discuss. Theoretical subproblems like that are what we use to construct solutions for the evolving squishy problems that we are discussing here.↩︎
… which is why it is called a “model”.↩︎

Why I Hate Carousels

2024-01-25T17:54:46Z

A carousel interaction pattern looks like this:

Basically it’s a layout pattern where one starts out positioning content in vertical blocks. Some of those blocks are going to have content arranged in horizontal blocks.

So far so good. You might think this will result in a layout that roughly looks like this:

This is a pretty common block structured layout. So common in fact that most block structured layouts will flip orientation by nesting level. If you nest a layout block inside a block that has vertical layout then the inner block will automatically switch to doing horizontal layout.

Notice that the entire layout fits snugly within the viewport.

Layout Runneth Over

Interesting stuff happen when the layout doesn’t fit inside the viewport. When it doesn’t fit vertically, obviously you’ll have to scroll up and down to see the whole thing.

However, if only the horizontal block doesn’t fit, that’s when the carousel pattern kicks in. So now things start looking like this:

How do you construct a UI affordance that lets people pan around this? Of course you use … wait I already mentioned it .. pan. Maybe use scroll bars or just about any other scrolling affordance.

So would it look like this if you scroll both horizontally and vertically a little bit?

Sliiiiiide To The Right

To address all the blankness, the carousel pattern posits that the the overflowing horizontal bits should be moved around independently of the vertical bits. Like this:

We once again have the problem of affordances. There are many ways of doing this. Most devices have some way of panning around both horizontally and vertically. So the solution is obvious. Right?

If people stop here and implement that, then that would’ve been over.

Uh oh. Here come the dots of stupidity…

Instead, they went with this:

Let me summarize this interaction pattern.

Use scroll bars, or whatever scrolling afforance to scroll up and down to access bits that are above and below the viewport.
Click/tap on little circles with arrows to access bits that are on either side of the viewport but only within the single sliding carousel block containing the arrows.

There’s a couple of problems with this:

The obvious wonkiness of having two different methods of scrolling horizontally and vertically.
Too easy to tap on the underlying “tile” instead of the dot-of-stupidity.

Carousel sympathizers explain the reasoning for this as follows:

What if there was more than one horizontally sliding thingies? How would one know which horizontally sliding thingy to apply the horizontal scroll operation to?

This is a purely hypothetical problem that often doesn’t exist. But I understand that UX designers have to account for the possibility.
It’s okay because the user doesn’t always need to or want to see everything in the carousal. Why bother them with details they don’t need?

That’s nice, but how would the user know they don’t need to see the thing without seeing the thing? It might be possible if all the visible things in the carousel were irrelevant to the user then they’ll just ignore the whole carousel.

But what about the very users who are interested? They need to jump through hoops to look at the things that are relevant.
What if there’s sooo many things in the horizontal slidey bit that we can’t possibly accommodate all of it?

Someone has to click on the little dot with the arrow enough times to see each one incrementally. The number of items for which such a UI works is far smaller than a UI where one scrolls past them using a much more ergonomic interface.

What’s Better?

Try wrapping the horizontal blocks so the page only overflows vertically.

If there’s too many things to fit in a vertical overflow layout, then there’s already too many things for anyone to click through individual cards to see.

Just show the user a good subsample. If it whets their appetite, then offer them a link to take them to a dedicated page.

In Summary

Don’t use carousels in your UI.
Instead, if you must stick content into little cards, let the horionztal layout wrap to additional lines.
If you end up with too many horizontal lines of content, then you definitely have too many items that can be accessed via a horizontally panning UI. You need a different UI altogether.

Inclusion/Equity and Evaluating Negatives

2024-01-10T02:58:34Z

Privacy, security, inclusion, equity, accessibility, … all of these have an interesting property in common. When evaluation products against such questions you are usually trying to establish a negative.

To Prove A Negative

Proving a negative requires providing evidence of absence.

If one wanted to disprove the claim “all primes are odd” one only needs to prove “some primes are even”. The latter can be accomplished by finding a single counterexample – the number 2. On the other hand if one wanted to prove the claim, they would need establish that all primes are odd. Another example is the claim “there are no vermin in this house,” proving which requires an exahustive search of the premises. Once again, disproving the claim only requires a single piece of evidence – a mouse.

Going back to the beginning, asking someone whether a product is secure, private, inclusive, accessible etc… fall into the same category. Answering these questions require proving that there is nothing insecure, privacy violating, non-inclusive, or inaccessible about this product. These questions are even worse than the one about vermin from before because it’s not even clear where to start looking. People who have spent their entire careers studying the subject would find the criteria difficult or downright impossible to evaluate.

Despite this apparent futility, there is hope. Knowing that the criteria holds under a limited set of circumstances is sometimes enough. Often it is the best that we can do.

Proof By Enumeration

In security and privacy parlance this limited scope is called the threat model.

Absolute security is impossible, so one would start by establishing the specific threats that they would like to deal with. For example when considering the security of a networked application one could start by drawing out a data flow diagram and then going over the different points at which data would be at risk of being compromised. Enumeration of vulnerabilities and threats in this way opens up the possibility of thoroughly examining the attack surface. Thus the previously intractable problem becomes finite.

As our understanding of threats improve, so does the effectiveness of the threat model. Historically, understanding this kind of problem space takes time and many – often costly – mistakes. The boundaries sometimes only become known when someone exploits it.

Similarly, when evaluating a squishy concept like inclusion and equity one would also start with a well defined scope that lends itself to examination. Just like before, often we only learn where to look for problems the hard way. So we come up with methodologies like carefully curated dimensions of identity. Enumerations like that give us a finite set of things to consider with our finite resources. Additionally these methodologies also grant us a taxonomy with which we can organize our hard earned knowledge on the topic.

Proof by Enumerating Product Outcomes

Enumeration based approaches establish clusters of possible product outcomes which we can then evaluate individually.

As an example, a product that processes pictures of people can be evaluated based on the outcome of handling pictures of people with different skin tokes, gender expressions, body types, ages, hair and accesories, visible disabilities, and lighting conditions. Coming up with these outcomes can be laborious and expensive, but it is tractable. Costs of curating suitably detailed test cases and data sets should amortize across their potentially broad applicability.

Proof By Construction

But sometimes the set of outcomes do not cluster effectively or are too numerous to exhaustively evaluate. In security this would be similar to asking whether some software system is free from buffer overruns – a question that is theoretically impossible to answer in general if the method of construction is not amenable to analysis. The best we could do is for suitably skilled persons to go through the code with a fine toothed comb looking for potential problems; a task that is both impractical as it is prone to error.

Proof by Process of Construction

Instead we could try using programming languages that provide certain safety guarantees: memory-safe languages. These safety guarantees establish the memory safety of the resulting behavior as a consequence of the method by which the underlying software is constructed. Another method we can use is to continue using an unsafe language¹, but invest in development practices like fuzz testing which give us sufficient statistical control over the behavior of our software.

Inclusion and equity can follow a similar paradigm where we establish certain process requirements that also yield a favorable outcome with a high likelihood. Google’s Building for Everyone outlines some of these processes criteria such as involving a diverse group of people during the ideation and design phases of a product, and involving a diverse group of people during testing. Once again, these approaches don’t guarantee success, but have a much higher likelihood of delivering a favorable outcome.

Amalgam Engineering

Increasingly software systems are built as amalgamations of systems that are themselves complex. The behavior of each constituent is difficult to reason about let alone the behavior of the amalgamation.

To establish some desirable property of the amalgamation, it becomes necessary to progressively establish the same or minor properties of the constituents. The point of establishing these properties is to yield a simpler more tractable model of a system that we can readily reason about. We can then infer or at least approximate the behavior of the amalgamation based on the simplified models that we derive of the contituents.

We don’t expect this situation to get better given that many of these contituent parts will be ML models that are too complex for us to understand. As enumerating product outcomes becomes less effective, we will need to rely ever more on process criteria and governance of construction to curb the product outcomes.

The accuracy of the resulting simplified models is all that we will have to grasp in order to have some say how technology will govern us in turn.

Continuing to use an unsafe language isn’t as dumb as it sounds. Large tech companies with massive software estates are often in the unenviable position of being unable to move enmasse to better platforms due to the cost. Thus less than perfect interim measures are not that silly.↩︎

Happy New Year 2024 and Year in Review 2023

2024-01-01T19:12:21Z

Happy New Year Everyone!

I hope everyone has a great year in 2024!

What We Are Leaving Behind

I’m acknowledging but not going into details of the cruelty we are inflicting on ourselves in Ukrain, the Gaza strip, Myanmar, Niger, Burkina Faso, Azawad, Tunisia, Sudan, Columbia, Afganistan, Pakistan, Somalia, Democratic Republic of Congo, Nigeria, Iraq, Rwanda, Burundi, Mexico, Camaroon, Chad, Syria, Yemen, Saudi Arabia, United Arab Emirates, Ethiopia, Haiti, and pretty much everwhere.

Instead let’s skip over to the the weirdness that happened in the tech sector.

Generative AI Has Entered The Chat

Educated people often make the mistake of underestimating the disruption that can happen in an industry. In this case, we are talking about generative AI and its disruption of the tech sector.

Transformer models have been around for a while (since before 2017) but they have mostly been a curiousity rather than a serious practical tool. It held much promise and excitement but was considered to be a technology that was in its infancy. The only real production workloads they handled were around machine translation.

However, a sudden disruption happened in 2023 where:

AI models become capable enough to give a somewhat convincing approximation of intelligence. This led people to conclude that generative AI is now ready for use in industry – a claim that is debateable.
Generative AI models became accessible to the general public largely in the form of ChatGPT though others followed quickly.¹
People started incorporating generative AI into their lives and businesses.² ³ ⁴
This triggered an influx of financial investment in generative AI.⁵ ⁶
Which was followed closely by legislation that aimed to mitigate the perceived harm of AI.⁷

All this in the span of a year.

I’m not directly involved in any ML work, so the way I experienced it was:

At the start of the year I had a couple of NLP projects that I had lined up to complete by the end of the year. The tasks were somewhat complex and involved fiddling with a lot of NLP.
By the end of the year the tasks were reduced to producing a prompt for an LLM and then parsing its result.

In addition to all the assistive coding tasks that are pretty much production ready at this point, the transformation of a complex coding task into simple English question and answer is kind of mind blowing.

I’m waiting to see what the new year is going to bring in terms of automation. We are truly living in the future.

“45% of US population surveyed is using generative AI” according to Salesforce.↩︎
See McKinsey’s the state of AI in 2023: Generative AI’s breakout year.↩︎
“ChatGPT just crossed 1 million users; it’s been 5 days since launch.”↩︎
“Three in four business leaders (74%) rank generative AI as the top emerging technology that will impact their business over the next year and a half.”↩︎
Funding for AI-related startups were more than $65 billion in 2023. See TechCrunch’s ‘Mega-deals’ could be inflating overall AI funding figures ↩︎
“Generative AI to Become $1.3 Trillion Market by 2023, Research Finds”↩︎
“In the 2023 legislative session, at least 25 states, Puerto Rico and the District of Columbia introduced artificial intelligence bills, and 15 states and Puerto Rico adopted resolutions or enacted legislation.”↩︎

Thirteen Years

2023-12-07T21:30:52Z

And just like that, another year sped by. It’s now been thirteen years since I joined Google.

Of the people who started at Google that day just a hair over 50% are still at Google. Glancing through the list¹ I see a couple of folks who have left for greener pastures or pursuing something on their own. Others who are still here have made their way up the organizational ladders.

But for everyone on that list the Google of today is not the Google we joined all those years ago. Someone who had been at Google much longer than I have recently published their thoughts after they left Google. They had good things to say about Google, but they also had some pretty harsh things to say. I can’t comment about some of the specifics they mentioned but the general sentiment checks out.

Avery Pennarun² wrote a post that I think touches on something relevant and important. The post is called What do executives do anyway?. To save you a read (though you really should add it to your reading list), the salient points are:

An executive’s only job is to “enforce culture and values for their whole organization.”³
The executives set and enforce the values. The rest of the company comes up with strategies accordingly.
“In a large organization, executives don’t set strategy.” … “It’s an illusion to believe you can enforce a strategy.”
Useful organizational values involve real tradeoffs.

People have a common misconception that all employees of big companies are constantly thinking of the bottom line. This is not the case. Precious few teams can draw a clear line from their day-to-day work to any measurable change in the company’s performance⁴. So at an individual level, decisions are influenced by concerns much closer to themselves: things like base pay, career, admiration of their peers, and their own sense of accomplishment.

There are many things that an individual can work on to satisfy their immediate concerns. The culture and values of a company help narrow these options down to those that benefit everyone.

The reason I brought up the topic of culture and values is because Google has a vacuum where an inspiring vision once was. Layoffs wreaked havoc with the trust that people had for each other and themselves. We are still trying to focus on the user; sharply, in fact. But it is evident that company and its people are increasingly acting out of self preservation,

There’s a mailing list for the folks who started on the same day.↩︎
Avery is also a Xoogler.↩︎
The article goes into detail about ratifying good decisions and arbitrating difficult decisions. But those are not relevant to this discussion.↩︎
You can, however, affect the stock price by getting something really really wrong. Noone wants to be in this position though.↩︎

Pillars And Governance Structures

2023-09-09T00:18:23Z

Note: Originally published on 2023 September 8. Revised 2023 November 12 with hopefully less rambling.

Re-orgs.

It’s a fact of life for large organizations. A popular metaphor making the rounds now is this concept of pillars. The metaphor makes sense to some extent.

A house without a foundation

Let’s start with a picture of how a sanitized version of a pillar structure would look like.

On the top is some kind of goal or an organization. In the case of an organization we can think of organization’s mission as the goal. Pillars support the goal.

Stage 2: Subpillars

Massive corporations can’t be organized around a handful of pillars. We need more structure: subpillars.

The subpillars are holding up the pillars. Together all of them hold up the goal.

Collectively what we are looking at is a system for organizing people. One that is hierarchical and tree structured.

We all know this. Because that’s what org charts have always looked like.

Team Topologies

In TeamTopologies ¹ Martin Fowler et. al. discusses different kinds of software teams.

One can think of pillars and sub-pillars as being higher level structures for organizing these teams. Individual teams at the granularity at which team topologies apply are too granular for pillars.

What I found compelling in “Team Topologies” were the concepts of Outcome Oriented teams and Activity Oriented teams. These have always had names – project teams and service teams.

Fowler’s article mentions

To put it in a nutshell,

Outcome oriented teams are teams that are working towards something. They hope to achieve some sort of goal, and once it is achieved the purpose of the team has been fulfilled and typically people move on to other things.

Outcome oriented teams have specific measurable goals like…
- build a thing,
- make specific qualitative change happen,
- research something, or
- achieve a sales goal.
Activity oriented teams are teams that perform some sort of activity for the organization. They do things like keeping the lights on, bug fixing, and occasional feature development. Activity oriented teams are typically in the service of enabling an outcome.

Activity oriented teams have qualitative goals like
- uptime,
- efficiency,
- user-satisfaction, and
- level of support.

There are more details and nuances and much hair-splitting. But the gist of it can be understood without diving too far into the weeds. What is important is that …

Outcome oriented teams and activity oriented teams can mutually depend on each other.

Also …

A single outcome oriented team can depend on multime activity oriented teams.

For example, a team tasked with building a new service will depend on multiple existing services that are under the purview of multiple activity oriented teams.
A single activity oriented team can depend on multiple outcome oriented teams.

For example, a team tasked with maintaining a critical service will at times need to implement large changes or features, which require multiple outcome oriented teams.

Uh oh. Our organization is not tree-shaped anymore.

It’s a Graph

The problem is that the the thing we are trying to organize is not tree structured. Each group has relationships that stretch vertically along the reporting chain, and also horizontally across “enabling” teams, support teams, infrastructure teams, or other teams that are solving similar problems.

An organization especially at this level of abstraction cannot represent all this complexity. A large number of free-floating small teams interacting with each other won’t be very efficient². Nor is it conducive to goals at a large scale because that would require a degree of cooperation that won’t appear spontaneously.

So we need to steer this pool of teams. It needs some structure so that some higher level of management can coordinate a number of smaller groups; so that they can collectively work towards a common but larger goal.

With this newfound knowledge we can once again approach the pillars but this time with goals and activities for the higher level teams.

It’s a Tree (Again)

However, the leadership needs to figure out which of these relationships to organize around.

This isn’t quite satisfactory yet because we can’t made things fit into the pillar system. Once people get stuck on pillars everything needs to be a pillar.

We can, however, stick teams into pillars. These teams would be outcome oriented if they are stuck into outcome oriented pillars. It follows that activity oriented pillars will need to be stuck into activity oriented pillars.

So it’ll kind of look like this:

And there you have it.

In reality, the structure can be made much cleaner once you start asking each team what their goals are.

But the gist of it is you can’t plan teams or pillars until you know what their goals are.

I don’t believe the concepts introduced in Team Topologies don’t apply universally even across software companies. I work in one where it doesn’t.↩︎
Even though in the aggregate, you can think of the open economy as being a very large number of small “teams” with a very dense network of interdependencies. Sure, it’s leaving people in poverty and is killing the planet; but it still kind of works.↩︎

A Good API: g9

2023-08-12T23:29:31Z

Once in a while, an API comes around that makes me stop and look at what I’m doing.

Like many aspects of design, when it comes to APIs, what is not there matters as much as what is. Engineers like to fiddle around with things. So they like to add knobs and dials for fiddling around with things.

People say platitudes like good API design has to do with balancing flexibility and simplicity. People throw around the word “simplicity” as if people just need to be reminded of it and all will be well. “Simplify your API” isn’t the same as “put on sunscreen.” It takes work. Lots of work. Intentional work with the goal of simplifying because there is little chance you are going to arrive at the right solution by accident.

An example of a good API is g9.

Before you look at it, think about how you would construct an API that:

… allows you to draw a diagram on an HTML canvas – perhaps with a limited set of drawing primitives.
… once the diagram is rendered, allows users to manipulate the diagram by dragging points around.

There are millions of ways of designing this. But g9’s approach takes the cake. You can take a shortcut and have a look at their API documentation or their examples. But basically, it is this:

Tell us what your mutable parameters are.
Draw your diagram based on the set of parameters.
…

That’s it. That’s all there is.

Now, you can sit there and come up with a million knobs that you claim make the API more powerful. And you might be right – you’ll be able to do more things with the API with the knobs you add.

But when you start treating simplicity as a feature, you’ll notice that adding knobs isn’t the same as making a more powerful API. All those additional knobs must be essentially invisible for people who don’t need them. Otherwise, every additional knob will waste more of your users’ time and energy for nothing.

The power of an API comes from letting users express more with less.

The Point of a Strategy

2023-07-24T15:00:12Z

Q: How do you win a soccer match?

A: Score more goals than the other team.

This is a dumb strategy, right? But why?

Compared to a more reasonable strategy like start preparing with plenty of time to spare, invest in a good team, hire a good coach and what not, why is scoring more goals a dumb strategy? That’s because the bad strategy didn’t reduce risk.

The whole point of a strategy is to mitigate risk. One could imagine gathering up a bunch of random people and somehow lucking out at the game and winning it. You need a strategy if you don’t want to leave everything to chance.

That’s why the next strategy you write down needs to account for these:

State your goal clearly.

A good goal should clearly articulate the outcome, and constraints. E.g. MVP available for limited testing (the outcome) by end of Q4 (the constraint).

It should be obvious to anyone when the goal has been reached. This is by design, since there should not be any question of what you are collectively trying to accomplish.
Identify the risks.

Mitigating risks begin by identifying them. Some risks are obvouls, but still write them down because you need to address them.

Failing to meet the goal is an obvious risk, but why would you fail to meet the goal?

A common reasons is simply that you need more time. E.g. you are at risk of not meeting the constraints due to underestimating the amount of work required.

Another reason could be technology you are using not performing as expected due to bugs. Every part of your tech stack is a gamble. A part that you don’t own is a specially big risk because you may not be able to fix it or work around a problem.
Figure out how to mitigate the risks.

At risk of schedules slipping? Have intermediate deliverables and milestones along the way. Instead of everything falling apart at the end, you’ll know when things are not going to schedule much earlier in the product life cycle. Knowing early means that you can address them before failures snowball into a giant ball of trouble.

Unreliable tech-stack? Choose stacks that have a solid user base and support structure. A project with bugs that are getting addressed is a lower risk than one that hasn’t seen any updates in a year.
Stack up your goal, risks, and mitigations.

There’s your strategy.

It’s not a strategy is risks are not identified and mitigated.

A common mistake is to write down a list of steps that, if followed, achieve a stated goal. A good first step. But the unless the strategy can reduce risk, it is not a useful.

Just Enough Technical Debt

2023-07-08T00:00:00Z

Zero technical debt sounds like a great target to aim for. In my earlier developer days I strongly believed that all projects should go for this target. At a company all-hands I heard from a senior engineer that having zero technical debt is often not what we want. I felt betrayed.

Why is it that zero technical debt isn’t what we want?

Memory Allocation Strategy: Leak All Allocations

Let’s take a moment and think of a hypothetical program. If you knew that your program ran once and terminated, would you go through the effort of deallocating each individual memory allocation at the end? You are about to end the process and drop the entire heap anyway. Cleaning the heap at this point is a waste. For short-lived programs who’s memory consumption is bounded and less than the available memory the more efficient strategy is to just leak all memory.

One might wonder if such a strategy would break reusability. “Leak all allocations” doesn’t mean that you shouldn’t call free() (or equivalent) in your code. It means that the memory manager would treat free() as a no-op. This principal also applies to garbage collected languages. After all, the job of memory management is to simulate infinite memory. If you are only going to use a finite amount of memory then you don’t need memory management.

What Does Just Enough Technical Debt Look Like?

Some of the motivation for stopping short of perfect comes from the Mythical Man Month.

For the human makers of things, the incompletenesses and inconsistencies of our ideas become clear only during implementation. Thus it is that writing, experimentation, “working out” are essential disciplines for the theoretician.

― Frederick P. Brooks Jr., The Mythical Man-Month: Essays on Software Engineering

The hardest part of the software task is arriving at a complete and consistent specification, and much of the essence of building a program is in fact the debugging of the specification.

― Frederick P. Brooks Jr., The Mythical Man-Month: Essays on Software Engineering

Software projects tend to be characteristically similar. Putting too much effort into a project features you don’t need and polishing up code that nobody will reuse is wasteful. It’s just a little bit harder to know when you’ve crossed the point of diminishing returns where cleaning up the code no longer makes sense.

Code As UI: The UX of config-as-code and everything-as-code

2023-07-01T00:00:00Z

Welcome to the UI review

Your UX person comes up to you with the new mocks for the configuration UI. It looks like this:

You: What is this? Is this modern?

UX: That’s be beauty of it. The user can type in the label and type in the value.

You: And how would they know what the labels are and what valid values are?

UX: We will write some documentation on what the labels and values should be.

You: What about validation?

UX: Hmm. That would be nice. But the UI doesn’t really know what the valid values are.

You: Come again?

UX: The UI is neutral. Again, the valid labels and values are documented.

You: Is it a good document?

UX: Let’s not get ahead of ourselves.

You: Is it a single document?

UX: We are thinking more along the line of leaning into crowd sourcing. Like there will be some documentation, but really the labels and values are consumed by the backend. We don’t really have much control of that.

You: So, the backend is responsible for keeping the documentation up to date? How has that worked out before?

UX: Not great.

You: So, any good news here? Why are we doing this?

UX: Because it’s config-as-code. It’s the new hotness.

You: Say no more fam.

So here’s the thing. Config-as-code isn’t new. That’s how it was before any GUIs existed. Anyone hows configured an old OS, and some new ones, will tell you that the only way some parts can be configured is by editing text files; sometimes using an esoteric editor called vi¹.

Then there was a glorious age of GUIs. At some point they started treating developers and sysadmins like real people. You might not believe this, but they even did user experience testing to make sure those sysadmins’ lives were as easy as possible. It was glorious.

And now, what’s old is new again.

The Pros

There are in fact good reasons for config-as-code. Here’s a few:

No more opaque configuration.

Fancy UIs often mean that the configuration data goes straight into an opaque configuration store somewhere. Sure it will come around and affect the behavior of whatever it was that is being configured. But that stuff usually doesn’t involve the user.

Usually the the platform doesn’t have much reason to keep configuration a secret. So there will be some schema and some method to storing and querying the configuration.

However when it comes to third-party or hosted services, this is not always the case. The how the configuration is stored and how it is reified into behavior is opaque.

With configuration-as-code (Can I call this CaC? That’ll be more convenient.) the user is in control of storage. How else would they write it down? The schema must necessarily also be made available to the user.
All the controls of source code.

Anyone who’s written some source code and used some source control tool will tell you how useful source control can be.

Almost trivially you get:
1. History and all that comes with it like the ability to revert back to known good states.
2. Blame/Praise and attribution of changes to specific authors – assuming there’s sufficient authentication and authorization in place.
3. Reviews/Approvals that verify that the changes were correct and appropriate.
Unless someone can be mandated to stand behind you and look over your shoulder taking notes, none of that comes for free with a sysadmin at a GUI.
Audits, validation, automation.

Follows from the above. Now that the configuration is inspectable and understandable, it is easily amenable to automation and validation.

One could argue that this was always the case. However, now the user is empowered to do so with the same fidelity as the authors of the system.

And the cons

The things that need fixing is usability.

We spent decades saying Fitt’s law this and “recognition over recall” that, and then threw everything away. It’s time we put all that experience into action.

Just because the configuration is in text files doesn’t mean that it has to be authored that way. Even if it did, it is still possible to substantially improve the UX there.

Why do people enjoy Mad Libs but wouldn’t enjoy writing the same content from scratch? The answer is “recognition over recall”.

An example from the original Mad Libs book. Source: Wikipedia

We can have the format and options documented somewhere and have the user consult the document at every step of the way. This sound reasonable at first. So people write things like “read all the documentation carefully before attempting to configure the thing.”

But the amount of information that a human being can keep track of at a given time is limited. What it ends up being is a tiresom sequence of flipping back and forth between reference and implementation.

Left to right to left to right …

Making things better

It turns out we already have some ideas on how to fix this, because the same problems plague software engineering. Or I should say the other areas of software engineering.

Here are several approaches I’ve seen in practice that make config-as-code more bearable.

Language Servers/Autocomplete

Pretty much the only useful development in coding in the past several decades is autocomplete. A good schema along with documentation can make a huge difference. Try editing the VS Code settings file to see it in action.

Templates

As the Mad Libs example shows, templates are much easier and more expedient than a blank empty document.

Other examples of templates include:

But templates in general only work the first time. So subsequent edits to the configuration will have to contend without the help of a template. This is a biggie since configurations can’t be updated when updating their corresponding templates.

Alternate UIs

We once used to have WYSIWYG UI editors. Now we have to write our UIs as code in one of any number of frameworks that in turn try to reduce the toil of writing UIs. It doesn’t have to be this way.

At least for configuration, just because the configuration is written down as code doesn’t mean it has to be edited as code. The same UIs that we once used can still generate those text files.

Vim is actually quite awesome. Neovim is slightly more or less awesome depending on who you are ask.↩︎

Promise Fatigue

2022-11-01T04:03:16Z

I’ll get it to you by the end of the day.
I’m working on it right now.
I’ll get it done today.

Even though the word “promise” doesn’t appear in any of these, they still represent a commitment you make to someone else. If you fail to keep your promise, you will disappoint someone or more than one.

But if you fail to keep your promises, eventually, you’ll stop disappointing people. That’s because they no longer expect you to keep your word. Of course, by then, you would have irreparably damaged your reputation – and people will no longer consider you as being reliable.

Before making a commitment, always think about promise fatigue. Setting low but realistic expectations is better than setting high but unrealistic ones.

The Loss of a Virtuous Culture

2022-10-28T21:13:19Z

Long ago, around 2011, we had an espresso machine in the Google micro kitchen – or MK as we called it – near my desk. There were no baristas at the time, so people made their own. Some folks thought we could use more sophisticated coffee beans than they had in the MK. A designated person would purchase these higher-quality beans using money that everyone collected in a jar in the MK.

This cash jar was not locked. Instead, it was in a conspicuous location, along with a note suggesting that people using the “good” beans should consider dropping some cash there so that people could continue purchasing premium beans.

One day, the cash in the jar went missing.

The response was swift. Donations poured from everywhere in the office, offering to replace the lost cash. Noone speculated on who took the money. Things were back to normal as soon as the incident was reported. The cash jar was re-instated.

The cynic in me wondered why the money didn’t get stolen sooner. But everything worked on the honor system at Google. And we were insistent on keeping it that way.

As far as I know, the money never went missing again, though the cash jar system went away as folks moved around inside the ever-expanding office.

Cash isn’t used as much anymore. However, would a similar system work now? We no longer place as much trust in each other as we did. We don’t even know most of the people we see walking around. But something is going on beyond that. Even if we did know the people, we are now more attuned to insider risk – and justifiably so. As a result, there is more of an air of distrust.

What would be the consequences of a breakdown in mutual trust among employees? In “Upside Decay,” Brian Lui argues that such a breakdown in virtue limits an organization’s ability to benefit from luck.

As the diagram indicates, a virtuous culture cultivates collaboration, leading to successful extra-curricular activity beyond their core work, which reinforces the culture. Notably, many of Google’s successful billion-dollar products came about or benefited from unexpected collaborations that were more or less luck. In a culture of mistrust, such associations outside of their core work would never have happened.

Leadership can identify, plan, and account for some contingencies in core work. Non-core work, on the other hand, relies on self organization within the company. That self organization needs trust. Every large project owes its existence to this non-core work.

And that’s not all. Some of this work may well become large products in their own right. Any large tech company will have tons of stories of this happening repeatedly.

The importance of “serendipitous collaboration” isn’t lost on executives. It is the reason behind many pushes for bringing tech workers back to the office.

The Lettergrade Rule

2022-09-16T00:27:57Z

How do you know when a code review is “done”?

Let’s say you are the reviewer and see many small changes you’d like the author to make. At some point, you’ll agree that the introduced behavior changes are correct or at least harmless. The remaining comments are largely stylistic or involve misalignments in minor architectural choices.

The letter grade rule is a lens for determining if any remaining comments justify blocking the change. It goes something like this:

Does addressing your comment or doing what you suggest improve the reviewed code by at least half a letter grade?

Your comments are not worth blocking the review if the answer is no. Instead, leave them as suggestions that the author can take and do with it what they will. Then, move on with your life.

This route is respectful of the author’s time and yours. It also implicitly places some amount of trust and responsibility on the author, which is empowering.

You might now wonder what constitutes a half-a-letter-grade improvement.

Here’s a non-exhaustive list of things it is not:

Stylistic choices that don’t violate the project’s style guide. Please choose a coding style and enforce it via a code formatting tool if this is a common cause of disagreement. Use a code formatting tool even if it isn’t a common cause of dispute. There should be no reason to format text with a spacebar in 2022.
Choice of identifier name; unless the name diverges from the project’s domain model and naming convention or objectively hurts readability.
“This isn’t how I would have done it.”
There are minor grammar issues or word choices in comments that don’t decrease the readability of the code. You should be able to trust the author to make those fixes and submit the change without you having to do another pass.

All natural numbers are interesting

2022-08-03T15:50:46Z

Theorem: All natural numbers are interesting.¹

Proof by contradiction.

Assume the theorem is false. Then there must be at least one number \in \N that is boring.

Let \mathbb{B} \subset \N be the set of boring numbers.

By the assumption, we have:

\mathbb{B} \ne \phi

Thus, because \mathbb{B} is non-empty \mathbb{B} \subset \N, the Well Ordering Principle tells us that:

\exists b \in \mathbb{B} \| b = \mathrm{inf}(\mathbb{B})

Thus this b is the smallest boring natural number in existence; which makes b interesting. A contradiction! ∎

Note that the same doesn’t trivially hold for the set of real numbers since you’d first have to prove that the set of boring real numbers is either finite or bounded, and that the infimum is a member of the set. Neither does this hold for \Z since a non-finite subset in \Z doesn’t necessarily have a minimum.

Not actually a theorem.↩︎

Use Git hooks to automatically push after every commit

2022-07-05T20:15:23Z

A Git hook is a shell script that is invoked at specific points during some Git operations. They are documented in the Git book.

The hook we are interested in is post-commit (documented here). It gets run after a new commit is made. This is exactly when we want to try and update our remote.

Our hook will:

Execute each time you commit something into your repository.
Avoid doing anything if the commit is made during a rebase or merge.
Push to a remote named origin (or whichever one you change the UPSTREAM shell variable to point).
Create or update a remote reference that matches the branch name that’s currently checked out.

Copy and paste the following script into a file named .git/hooks/post-commit in your repository directory.

IMPORTANT: Afterwards make the script runnable via chmod +x .git/hooks/post-commit.

DARPA's Heilmeier Catechism

2022-06-28T14:11:17Z

It seems the hardest part of engineering is the part where you are staring at a blank page with a vague notion of a problem statement in your head.

First you seek structure. After all your attention span can only hold so much at the same time. In order to tackle larger problems, you need to reduce large parts of the problem area into intuition. And in turn, to reduce parts of the problem into intuition, you need to break those pieces into smaller simpler pieces that you can chew on individually.

Losing sight of the forest for the trees is easy during this process.

Why were you trying to solve this problem in the first place? Would you have started if you knew then what you know now about the real scope of the problem?

Such concerns are also part of the problem statement; parts that help your prioritize and select the best solution amongst many. Throughout all, it’s always good to keep and eye on the prize. That being the end goal of the exercise. The goal must justify the effort involved. Once again, for that you need to reduce your motives into intuition.

The “The Heilmeier Catechism” is “a set of questions […] to help Agency officials think through and evaluate proposed research programs”. The list of questions is duplicated here for your convenience. It’s a short and terse list. Much like what your design sketch should be.

What are you trying to do? Articulate your objectives using absolutely no jargon.
How is it done today, and what are the limits of current practice?
What is new in your approach and why do you think it will be successful?
Who cares? If you are successful, what difference will it make?
What are the risks?
How much will it cost?
How long will it take?
What are the mid-term and final “exams” to check for success?

(Source The Heilmeier Catechism - DARPA).

Markdown should be readble without knowing Markdown

2022-04-05T17:02:12Z

Markdown started out as a way to write text files that could also be machine parsed and rendered into a higher fidelity format.

To quote from the Philosophy:

Readability, however, is emphasized above all else. A Markdown-formatted document should be publishable as-is, as plain text, without looking like it’s been marked up with tags or formatting instructions.

This particular aspect of the format wasn’t maintained very well. Immediately following the “Philosophy” section is “Inline HTML” which contradicts the “without looking like it’s been marked up” requirement.

The ease of writing goal was met head on. So much so that it is now the de-facto format for writing documentation that documents in general that has any kind of formatting at all. This blog post is also based on a markdown document.

People – of course – need more features. And features there were many. The “Variants” section of the Wikipedia page on Markdown touches upon some of the popular syntaxes that exist now.

On pet-peeve I have is that these new features don’t even try to adhere to the minimum markup philosophy. And hence just introduce HTML tags – or worse HTML-like but not HTML tags – into the format. Someone who’s not familiar with Markdown syntax – or for that matter whatever dialect you are using for some document – shouldn’t be thrown off by the additional markup.

Better Definition Lists?

I really don’t like the “Definition List” syntax. This feature isn’t in the original Markdown syntax, but is a part of many extended syntaxes. It looks like this:

Some term
: The definition of the term. Long lines can be
  broken down while preserving the indent. But
  the first character can't be ':' or it will become
  *another* definition.

See, it’s easy to see the : as some misplaced punctuation rather than markup when just reading the text. Instead how about maybe this?

Some term:
  The definition starts here. Long lines can be
  broken down like this.
    
  Maybe continue the definition here.

It looks more like a definition in a regular text file.

Better Code Blocks?

The original syntax for code blocks is to just add extra space in the middle, like so:


This is normal text.

    function ThisIsACodeBlock() {
    }

More normal text.

See, if you saw this in a text file, your eyes would easily follow the context switch. Instead, the most popular markdown formats use the following kind of markup:


This is normal text.

```
function ThisIsACodeBlock() {
}
```

More normal text.

See how the markup is in the way? There’s a reason for the special markup; it allows specification of additional attributes like the underlying language that the code block is in.

Instead, how about something like this?


This is normal text.

javascript code:

    function ThisIsACodeBlock() {
    }

More normal text.

It re-uses the some text: syntax that’s used for specifying links.

Better admonitions?

Admonitions aren’t a feature of regular Markdown. It’s available as an extension in select Markdown processors. The syntax is a little horrendous:

!!! type "optional explicit title within double quotes"
    Any number of other indented markdown elements.

    This is the second paragraph.

Okay the exclamation points kind of hints at what this is supposed to mean. But how about this?

NOTE: Optional explicit title
    Paragraph that contains extra details about the note.
    
    This is the second paragraph.

It’s obvious from looking at the plain text what this is supposed to be; no need to go look at the documentation for your Markdown dialect to figure out what on earth !!! title is supposed to be.

Better collapsed sections?

Okay, so this one is specific to GitHub Flavored Markdown, and it’s terrible. It comes down to the genre of just writing HTML:

<details><summary>CLICK ME</summary>
<p>

#### We can hide anything, even code!

    ```ruby
    puts "Hello World"
    ```

</p>
</details>

WTF? You might as well write the thing in HTML and call it a day. Maybe if they could’ve gone with something like:


Summary goes here.

Details:
  Details go here. and yes you can use fenced code blocks.
  
      Like this.
      
  Or the ugly way:
  
  ```
  Like this.
  ```

Once again it re-uses the some text: syntax, and it could probably use either details or Details. Either way, you don’t need to understand the markup to see where this is going.

On Gender Gap in Adolescent Mental Health

2022-03-01T19:51:34Z

Today is the first day of Women’s History Month.

Coincidentally there’s a paper making the rounds titled “The gender gap in adolescent mental health: A cross-national investigation of 566,829 adolescents across 73 countries” (Campbell et al., 2021), which makes some these interesting claims:

Girls have worse average mental health than boys across 4 mental health measures.
There is significant heterogeneity in the mental health gender gap size across countries.
The gap is most pronounced for psychological distress and life satisfaction.
More gender-equal countries have more gender gaps in mental health.

The last point, in particular, is a bit of an eyebrow-raiser. So let’s have a closer look.

The study uses the PISA 2018 dataset, introducing a couple of caveats. Namely:

All survey respondents were 15 years old; thus, the dataset doesn’t consider higher education and employment effects.
The data is from 2018; thus excludes the effects of the pandemic. It would be interesting to see how the responses change now that people are coming out of a lengthy lock down.
Gender was binary. There was no non-binary option.

The study (Campbell et al., 2021) derives five measures of mental health based on the PISA 2018 dataset as follows:

“Life satisfaction” measures responses to the question “on a scale of 0-10, overall, how satisfied are you with your life as a whole these days?”
“Psychological distress” measures responses to the question on how often they felt “sad”, “miserable”, “scared”, and “afraid” on a scale of “never”, “rarely”, “sometimes”, and “always”. Each response yields points ranging from 1 through 4, respectively. Summing the points for each question results in a score of 4-16.

The survey doesn’t appear to have clarified the difference between “sad”/“miserable” and “scared”/“afraid”. PISA 2018 mentions that:

All questions were translated into the languages of participating countries by two independent linguists and then reconciled by a third to ensure consistent meaning in all countries.
“Hedonia” measures the responses to the question on how often they felt “happy”, “lively”, “proud”, “joyful”, and “cheerful” on the same scale. This yields a combined scale of 5-20.
“Eudaemonic wellbeing” measures the degree of agreement with the below statements. The scale was “strongly disagree”, “disagree”, “agree”, and “strongly agree” with a combined score in the range 3-12:
1. “My life has clear meaning or purpose”.
2. “I have discovered a satisfactory meaning in life.”
3. “I have a clear sense of what gives meaning to my life”.

Details of the analysis are in the companion document (PDF link).

In figure S2 of (Campbell et al., 2021) shows the distribution of life satisfaction. Again, developed nations are not doing well compared with countries like Saudi Arabia.

Compare with:

I’m skeptical that these numbers represent reality. Unless there’s a part of Saudi Arabia where adolescents are consistently happy all the time, these numbers don’t make sense. My skepticism is also extended to the data from the UAE. Human Rights Watch’s report on the UAE indicates that the UAE is far from having any semblance of gender equality. Approximately 80% of the population consists of migrant workers who’s children don’t have access to formal education. It’s unlikely that these children of migrants participated in the study.

The curve fitting for the resulting graphs is what tells the stories that are summarized at the start of this post. The axes are arranged so that moving towards the right and moving upwards indicate better conditions. Hence it’s apparent from the curves that male life satisfaction is pretty consistent and not affected all that much by GII¹. Meanwhile, female life satisfaction trends downward as GII improves; hence the conclusion that more gender equality implies less happiness.

GSNI (Gender Social Norms Index) is also negatively correlated with quality-of-life measures (where we put “good” on the high end and “bad” on the low end).

All the graphs show a high variance. Only a small number of countries are present in the study. And whether the subjects of the study are representative of the population is in doubt. Overall I find the study unconvincing.

Campbell, O. L., Bann, D., & Patalay, P. (2021). The gender gap in adolescent mental health: A cross-national investigation of 566,829 adolescents across 73 countries. SSM-Population Health, 13, 100742.

GII is the Gender Inequality Index. It’s based on such factors like female and male shares of parliamentary seats, female and male population with at least secondary education, female and male labour force participation rates, maternal mortality ratio, and adolescent birth rate.↩︎

On Highlighting Syntax vs. Semantics

2022-02-12T00:00:00Z

Aesthetics aside, syntax highlighting in code performs an important role in improving the readability of code. Most syntax highlighting schemes including popular libraries are limited to using colors and formatting changes to distinguish between different syntax classes. This is what it would look like if we apply the same logic to English prose. Each word in this paragraph is highlighted using its respective syntactic purpose.

Not very readable, is it? The colors aren’t random. They highlights correspond to noun, adverb, gerund, preposition, adjective, and verb. This is basically how people use syntax highlighting.

Typographers have long known that punctuation alone can’t sufficiently convey emphasis and relative importance of prose. As (Truss, 2004) points out italics is “the print equivalent of underlining” (via (Wikipedia, 2021)). A number of uses exist for this font variation including providing emphasis, stressing important points, or indicating foreign words and phrases. When conveying emphasis the syntactic role of a word does not change; the word just is more important to make the central point than others.

These key important words alone can’t convey a story. Grammar and completeness demand accompanying words. It is the role of formatting, whitespace, and punctuation to guide the reader towards what is important. Without formatting and whitespace, prose would become an intimidating “wall of words”; tedious and unwelcoming.

And so it goes for code; some code is more important than others. The central logic of a snippet of code lies somewhere in the middle surrounded by boilerplate, setup, and tear down logic. If syntax highlighting were to make the central logic stand out, then it has to go beyond mere syntactic purpose of the tokens comprising the code.

What would such a scheme look like?

 bool ListValue::Set(size_t index, std::unique_ptr<Value> in_value) {
   if (!in_value)
     return false;
   if (index >= list().size())
     list().resize(index + 1);

   list()[index] = std::move(*in_value);

   return true;
 }

Perhaps not the best example, but you should get the gist of where I’m going with this. The highlighting quickly directs your eyes to the parts that need attention. You’d still need to do the legwork for the details. But for someone trying to figure out what some undocumented function is doing, this kind of highlight makes quick work of going past the boilerplate.

Truss, L. (2004). Eats, shoots & leaves: The zero tolerance approach to punctuation (p. 146). Penguin.

Wikipedia. (2021). Italic type — Wikipedia, the free encyclopedia. http://en.wikipedia.org/w/index.php?title=Italic%20type&oldid=1049142359.

Inktober - Stripes

2022-01-21T22:20:38Z

For Inktober 52.

Inktober - Wild

2022-01-13T20:35:39Z

For Inktober 52.

Inktober - Decay

2022-01-13T20:21:37Z

For Inktober 52.

Quickly grab a snapshot of your reMarkable tablet

2022-01-06T00:00:00Z

One everyday workflow that I wanted to optimize was quickly sketching out a diagram on the reMarkable tablet and including it in a document I was writing somewhere else.

My primary computer is a Mac. Here’s my solution:

Set up reSnap. This step requires changes on both the reMarkable tablet and the local machine. For the former, I followed the instructions in the README file. For the latter, I only really needed to install FFmpeg and ImageMagick.
Create a Mac Quick Action to invoke reSnap.
Bind a hotkey that grabs the snapshot of the reMarkable display, cleans it up, and sticks it in Mac’s clipboard.

Details instructions follow. I can now just hit ⌘+⌃+⇧+P to grab the snapshot and then hit ⌘+V to paste it into whatever application I am using, usually Google Docs.

Setting up `reSnap`

I have my fork of reSnap, which adds a few valuable tricks like cleaning up the image and placing it in the system clipboard. By default, the upstream reSnap script attempts to preview the image using feh (an image viewer I’ve never heard of). Immediately viewing the picture wasn’t useful to me, so I made that optional. You can grab my fork of reSnap here.

On the reMarkable tablet side, you must follow the instructions in the README file to install lz4 and head on the device.

It is important to set up SSH to use password-less logins since the hotkey method relies on being able to connect without prompting for a password. See instructions for setting up SSH for passwordless login.

On the local machine – which in my case is a Mac – you can install all the requirements using Homebrew as follows:

brew install ffmpeg imagemagick

Creating a Mac Quick Action

Start “Automator.”
Create a new “Quick Action.”
Set up the workflow options to receive no input in any application. All the other options are up to you, or you can leave them as-is.
Drag over the “Run Shell Script” action from the actions library.
Paste something like the following (with the suggested modifications to suit your local environment).
```
/usr/bin/env PATH=/bin:/usr/bin:/usr/local/bin:/sbin /path/to/reSnap/reSnap.sh --sketch --copy
```
Note that the reMarkable needs to be connected via a USB cable to capture a snapshot using the above command. If you want to be able to use WiFi you should use the --host option to specify either the local hostname (usually “remarkable”) or the IP address. In all cases the hotkey should only be expected to work when the reMarkable is not sleeping.
If you’d like, you can also add a “Display Notification” action, so you know when the script is done running. The FFMpeg invocation may take a few seconds.

Try it out with the “Run” button.

Binding a hotkey

Once you save your quick action, you can bind a hotkey like this:

Open “System Preferences” and navigate to “Keyboard” -> “Shortcuts”.
Click on “Services” and scroll to the bottom. You should see the new quick action you just made listed under the “General” category. It should have no keys assigned to it.
Double click on the “none” and type your hotkey.

That’s it.

Here’s what a snapshot looks like:

A couple of notes on the `--sketch` option

My fork of reSnap introduces a --sketch option that cleans up the picture a little bit. It currently does the following:

Erase the little menu widget from the top-left corner of the image. Since reSnap grabs the image right off the framebuffer this little artifact makes its way to the resulting image.
Erase a tiny little black dot that appears in the bottom-left corner of the image. I don’t know where this one comes from, but it’s there.
Make the background transparent.
Resample the image at 50% resolution with anti-aliasing. The original image is a stiff monochrome image with a white background and black strokes. The resampling after making the background transparent makes the edges smoother and a bit easier on the eyes. Otherwise all the edges look jagged.
Trim the image to remove all the empty borders thus making the image size just large enough to hold the actual image.

All of this is done via an invocation of ImageMagick.

Note-taking on a reMarkable 2

2021-12-30T21:10:30Z

After much dillydallying about which note-taker/ebook reader I wanted to get, I went ahead and got a reMarkable 2. I’ve been using it for about a month now. I must say that it’s the closest thing to writing on paper that I have encountered.

My requirements of a note-taking tablet were:

Replace my paper notebooks.
Quickly transfer diagrams and drawings to the computer/laptop.

Replacing a paper notebook is a tall order. A paper notebook is …

Ready as soon as I open it.
Lightweight.
Durable.
Can be used with a comfortable pen.
Doesn’t make my eyes tired.

An Unsuccessful Attempt With A Pixelbook

NOTE: The comments below only apply to the Chromebook Pixel’s utility as a note-taking device. It is, of course, far more than a note-taking device and will likely be a better choice for you if you want to use the web, consume lots of eBooks, or otherwise need the functionality of a modern laptop.

The Pixelbook ¹ was my first attempt at using a tablet for note-taking. Unfortunately, while it had other advantages, the Pixelbook falls far short on convenience.

Ad-hoc note-taking is a hassle. Getting started requires logging in. There are good reasons for needing signing-in, and it does have an option to use the built-in sketch app from the lock screen using the Pixelbook pen. But the sketch app doesn’t suffice as a note-taking app.

I do like the privacy model of the Pixelbook’s lock screen sketch app – namely, that the note drops into a black box that only the owner can open. However, I wish the reMarkable could have a feature where adding a new note doesn’t simultaneously require viewing every note on the device.

Once folded over, I get paranoid about the keys underneath. Can I place it on a table without worrying too much? It’s anxiety-inducing. It’s also a laptop, which means that someone who’s easily distracted would be … easily distracted.

So over to the reMarkable…

Moderate Success With The Remarkable 2

First of all, this is a note-taking device. You should not expect much more from it. I thought it would make for a good ebook reader and journal article reader – it is neither. Its support for ebooks is limited to PDFs and EPUBs.

I’m still looking for a satisfactory method of keeping a set of journal articles synchronized between my laptop and the reMarkable tablet. The current method is to go through their proprietary cloud service, which I can only do for my personal usage but not for work.

Once again, this is neither a laptop nor an ebook reader.

There are, however, two important points to note about the reMarkable tablet:

It’s the closest I’ve seen to writing on paper. The hardware is pretty much spot on.
There’s an active community of hackers who constantly push the tablet’s capabilities.

The second point bears repeating. The biggest thing going for the reMarkable tablet right now is the developer community surrounding it. Have a gander at the reMarkableWiki’s Tips and Tricks page or the Awesome reMarkable list. Some highlights include:

Ability to use third-party clouds or local storage for synchronizing your reMakable documents (via rclone or rm-sync or many other options).
Setting up an email address to which you can email documents which will appear on your reMarkable (via reMarkable Newsletter).
Stream your reMarkable screen (via pipes and paper, reStream, rMview, or others).

If you are into tinkering with electronics, the reMarkable will provide ample entertainment in addition to note-taking.

Workflows

The process of making myself at home with the reMarkable 2 involved coming up with good workflows for the things that I want to do with it. My goal here is to reduce the overhead as much as possible. Although a paper notebook is close to ideal for writing things down, it’s not easy to get that content out to an electronic format. I hope that the reMarkable 2 will keep the former as-is while significantly improving the latter.

Not all notes are equal. Some are ad-hoc sketches and doodles. Some need to be a bit more longevity, like daily diary entries. Some are drafts of design documents, diagrams, or blog posts that I’d need to move over to an electronic format. Then there are work-related things like interview notes and meeting notes. So we have three overall workflows:

Missives that I may want to keep around only to satisfy some sort of hoarding instinct.
Notes that I want to transcribe into some electronic form. If the originals are to be kept, they must be labeled or categorized somehow.
1. Pictures or diagrams.
2. Text.
Notes that, regardless of whether any transcription is needed, carry confidential or proprietary information.

Of these, the first one is trivial. Pull out the device, write stuff, move on with life. If necessary, I can always look at the recent notes, but I will likely never look at anything older than a month.

The second one is easy if you are using the Connect cloud. There are multiple integrations that they already have for moving content to other cloud providers, including conversion to different image formats. The Connect cloud also handles the work of extracting text from what you’ve written. In my experience, the handwriting recognition results have been quite good though they require the occasional cleanup.

The third one is tricky. Again, sidestepping the Connect cloud offering, the aforementioned hacking resources really help get material from the device onto your machine.

I’ll leave the details for another blog post, but the gist of it is that we can quickly transfer the contents of the display as a picture to a laptop or desktop computer via Wifi or via USB. From there, you can use any number of tools to extract the text or clean up the drawings. With a little bit of automation, the process was more accessible than the Connect cloud.

Conclusions

Pros:

Writing is as close to writing on a paper as one could get.
Tight integration with their cloud offering.
Great ecosystem of hackers who push the capabilities of the device far beyond the stock software.
The device lasts for days on a single charge. Their marketing materials mention two weeks. I haven’t pushed it that far due to occasionally connecting it to the laptop via USB for content transfer. I can, however, say that based on the battery remaining this claim seems to be correct in my case.
Can’t do much else other than note taking.
Relatively cheap compared to other note-taking devices.

Cons:

The nib on the pen wears out quickly.

They do include 9 additional nibs in the package, but I suspect that I’ll be buying more within a few months. Compare that with something like Supernote which boasts that their nibs provide the same level of paper-like friction without as much wear.
Requires a lot of hand tinkering unless you subscribe to their cloud offering.
Limited support for ebook formats mean that you likely can’t read your purchased ebooks on this device.
No support for labeling or organizing your notes – as of this writing.

In case the Pixelbook site goes offline, here’s the Wayback Machine’s snapshot from when I wrote this article.↩︎

Why 'Strong Opinions Weekly Held'

2021-09-28T00:00:00Z

Previously on this blog I wrote a short note on Why Strong Opinions Weakly Held.

I’m trying to get something out every week. Hence the new blog title.

Visualizing A Million People

2021-09-06T00:00:00Z

This article was formerly published as Visualizing Internet Users. This version is more generalized and up-to-date.

What if you introduce a bug and/or a feature that pisses off a million people?

The scenario isn’t all that hard to imagine these days if you work at a big tech company or even a small one with a considerable user base. It seems a million users aren’t all that hard to reach these days.

What is harder though is to imagine what a million people looks like. What if you were being stared down by the entirety of the affected user base? Or worse you had to stand on a stage in front of them and explain yourself?

If you have 1 billion users and you ship a bug that only manifests 1 in a million times, then you have yourself 1000 angry users. It’s good to know what that group looks like.

We can easily imagine a single person, 10, or even 25 people. But exceed a couple of hundred and our imagination falters. It becomes just a number and stops feeling like “users” or “people.” The perceived significance of the population thus starts to diminish.

So what’s the significance of a million people anyway?

World population is teetering on roughly 7.7 billion people (in 2019). So a million people is about 0.01% of the global population.

Internet users, on the other hand, come up to something like half of the world population based World Bank figures which imply that roughly 50% of the population should have internet access in 2021 ¹.

So a million users would be 0.02% of the population with some access to the internet. Still not that big.

But once you focus on a smaller scale like a geographic region or a user base of some application, then suddenly the million users looks much bigger.

1 to 49

Let’s skip this range since this is in the range of what you’d see in a mirror, can fit on a bicycle, can fit on a sofa, your team, gathering of friends, significant annual family event, or other familiar situation.

From 50 to 100,000

An example of visualizing populations for small numbers can be found at Visualizing Crowd Sizes. It goes from around 50 people up to 100,000 people.

While 100,000 is interesting we need to go all the way to 1 million.

200,000

200,000 was the estimated number of people who attended the March for Our Lives in D.C.. There aren’t any photos capturing all attendees in one shot, but there are numerous pictures in this CBS News article to give you a good idea.

In more terrible examples, this article on dw.com suggests that the crowd size pictured is roughly around 200,000 people.

More than 200,000 conservative Muslims gathered in Indonesia’s capital Jakarta on Friday to protest against the city’s Christian governor, who is facing trial for blasphemy, police estimated.

250,000

The March on Washington in 1963– an “event focused on employment discrimination, civil rights abuses against African Americans, Latinos, and other disenfranchised groups, and support for the Civil Rights Act that the Kennedy Administration was attempting to pass through Congress” – drew an estimated 250,000 people according to the National Park Service.

Rustin coordinated a staff of over 200 civil rights activists and organizers to assist in publicizing the march and recruiting marchers, organizing churches to raise money, coordinating buses and trains, and administering all of the other logistical details. In many ways, the March defied expectations. The number of people that attended exceeded the initial estimates made by the organizers. Rustin had indicated that they expected over 100,000 people to attend - the final estimate was 250,000, 190,000 blacks and 60,000 whites.

(From March on Washington for Jobs and Freedom – National Parks Service)

500,000

We are heading into shaky ground here. A 800*600 picture only contains 480,000 pixels. So we are now reduced to indicating population by inference. It is no longer possible to capture 500,000 people in a webpage sized picture while showing each person individually.

This Reddit post claims that there are 500k people in the picture, but I couldn’t readily find any good references backing up that claim.

Another event that drew 500,000 people is an anti-violence rally in Barcelona as reported by NBC. While there’s video, the content doesn’t quite capture the massive number of people. Also it’s “nearly 500,000” people.

However, the closest we’ll probably get to a somewhat believable 500,000 person crowd would be the March for Science. They posted the following picture on Facebook noting that “There are 500,000 people on strike in Montreal”.

Woodstock or “an Aquarian Exposition: 3 Days of Peace & Music” festival held in 1969 reportedly attracted “over 400,000”, “450,000”, or “500,000” people depending on the source. Let’s assume that the real number is somewhere in-between which gets us to somewhere in the neighborhood of 500,000 people.

There are numerous aerial photos of the Woodstock crowds, some of which you can find in this image gallery from Getty images.

750,000

Off to look at protests again. This time we are looking at Barcelona where 750,000 Catalan independence supporters are demanding the release of their leaders according to this article.

The crowd extends past the top of the frame and there’s no chance this is the entire crowd. But you get the general idea. If you ship a bug that affects 0.01% of the internet population, then this is a conservative lower bound of what that angry mob would look like.

But wait, the Guinness World Record for the largest charity walk/run is 283,171 people at a single venue. Have a look at that picture:

It seems that picture of 750,000 people doesn’t quite capture that many people because clearly there are more people in the picture that’s supposed to be 283k people. Which of course goes to show how hard it is to really capture how massive a crowd 750k makes.

In an article on Wired that discusses how hard it is to estimate crowd sizes, they note²:

Every time a ton of people gather in one place, there are all sorts of pronouncements made about how many folks really showed. The figures rarely agree; Glenn Beck’s rally on the National Mall this summer attracted a million people, according to Beck cohort Rep. Michelle Bachman. CBS News pegged it at more like 87,000.

And More

I’m going to give up at this point. Pictures purportedly of around one million people are mostly parts of crowds that have been estimated to be that big.

Note that figure 2 indicates that the estimate is 49% of the population would have internet access in 2017. It’s likely that by 2021 we’d at least have reached 50% based on the sharp growth curve.↩︎
How Many People Are in Tahrir Square? Here’s How to Tell ↩︎

Fart Limit

2021-06-14T15:30:18Z

Thanks to Vsauce for bringing this particular issue to our attention. The TL;DR of that video is:

Most of the gases in your farts are odorless.
The one that usually makes things stinky is Hydrogen Sulfide.¹
The content of H_2_S in a fart is around 50ppm.²
One fart can travel around 10 ft/s.

Enter OSHA:

H_2_S at a concentration of 50ppm (roughly what you’d get if you inhaled a fart directly into your lungs) you’d suffer Slight conjunctivitis (“gas eye”) and respiratory tract irritation after 1 hour. May cause digestive upset and loss of appetite.³.
Any practical dosage would be around 3ppm. At that rate it would be Odor becomes more offensive at 3-5 ppm. Above 30 ppm, odor described as sweet or sickeningly sweet.. The key point here is that around 10 times the usual concentration, a fart won’t stink. Instead it will smell “sweet.”
At 100ppm – which is around 30 times the usual dose – you won’t smell anything at all because you’ll lose your sense of smell (olfactory fatigue or paralysis).
At 1000ppm – 300 times the usual dose – it’s nearly instant death.

So I guess it is in fact possible.

As an aside, I found it quite disappointing that the Listen button on the “What’s a Fart” page on KidsHealth.org doesn’t make a fart sound. I mean, what else was that button supposed to do?

https://web.archive.org/web/20210614200353/https://www.houstonmethodist.org/blog/articles/2020/oct/flatulence-everything-you-wanted-to-know-about-farting/↩︎
https://www.chemistryworld.com/news/explainer-the-chemistry-of-farts/2500168.article↩︎
https://www.osha.gov/hydrogen-sulfide/hazards↩︎

Shannon Paper

2021-03-26T20:14:20Z

Claude Shannon’s A mathematical theory of communication is an oft cited classic in information theory. In fact, as of this writing there are 84’411 citations and 139 versions of the article on Google Scholar.

Let’s dive in and try to tease apart the “why”s that are often overlooked when people build on top of the introduced theory. They are definitely things that I didn’t consider to be obvious without the benefit of reading the paper.

The section headings below correspond to the sections in Shannon’s paper.

Introduction

The paper’s introduction states the purpose as:

… [To] extend [the general theory of communication] to include a number of new factors, in particular the effect of noise in the channel, and the savings possible due to the statistical structure of the original message and due to the nature of the final destination of information.

The two most salient points from this section are

the arguments for a logarithmic measure of information, and
what a measure of information actually measures.

Logarithmic measures are well suited for measuring information, Shannon states, because it’s “nearer to our intuitive feeling as to the proper measure.” Additional rationalization exists in the paper, but I found this to be the most convincing.

To intuit, adding a channel that’s equal in capacity to an existing channel increases the space of transferrable messages quadratically. However our intuition finds it easier to think of this as doubling the capacity. The latter is more in line with a logarithmic measure.

Information is what’s ultimately useful for the recipient. Hence a measure of information must be based on the ability gained by the recipient upon receipt of the message. Receiving a signal means that the recipient can now uniquely identify a message among a large number of possible messages, or to reduce the space of possible messages.

If the destination receives 10 bits of information, that means that the recipient is now able to uniquely select a message from 2^{10} possible messages. Or equivalently, the recipient can choose one out of 2^{10} outcomes.

Hence the measure of information is based on the ratio of the space of possible messages before and after receipt of the information.

The following sections focus on defining and measuring the space of messages.

Part I: Discrete Noiseless Systems

A discrete noiseless channel is one in which a selection of symbols from a finite alphabet can be transmitted from a source to a destination without loss. It’s not necessarily the case that each symbol consumes an equal amount of resources to send. Since the paper pertains to electronic communication Shannon states the cost in terms of time.

1. Discrete Noiseless Channel

Capacity of a channel is defined to be the maximum amount of information that can be transferred over that channel.

The capacity C of a discrete noiseless channel is defined to be:

C = \lim_{T \to \infty} \frac{\log N(T)}{T}

Here N(T) is the number of allowed messages that take a duration of T to transmit. Let’s see how this comes about.

From the prior definition of information, if the recipient receives C bits of information that means that they were able to distinguish between 2^C possible messages.

So we start with counting the number of possible messages that could’ve been transmitted over T time. If we denote this count as N(T), then we know that there were \log{N(T)} bits of information transferred during T time. Hence the rate is:

R(T) = \frac{\log{N(T)}}{T}

To generalize, we find the limit of R(T) as T tends to \infty.

Suppose the channel can transmit n symbols {S_1, S_2, ..., S_n} each of which consume {t_1, t_2, ..., t_n} units of time. How many possible messages can there be in T units of time? I.e. what’s N(T)?

To break this down a bit further, we can ask “how many possible messages could there be that end in S_i during time period T?”

Any message that ends in S_i can be written as a message consisting of arbitrary symbols s_1 s_2 ... s_k (where each s_i \in S) and ending with the symbol S_i. I.e.:

\underbrace{\underbrace{s_1 s_2 s_3 ... s_k}_{T - t_i \text{ seconds}} S_i}_{T \text{ seconds}}

As shown, the length of the s_1..s_k portion in time must be T - t_i since the remaining t_i time is consumed by S_i.

Hence there can be N(T - t_i) possibilities for the s_1 ... s_k prefix.

Thus we can state this problem recursively as follows:

N(T) = N(T - t_1) + N(T - t_2) + ... + N(T - t_n)

… where N(T) = 0 for all T \lt \min(t_1, t_2, ... t_n).

Solving this equation involves recognizing that this is a linear difference equation (not to be confused with a linear differential equation).

If all t_i are distinct, then the characteristic polynomial corresponding to this recurrence is:

X^{-t_1} + X^{-t_2} + ... + X^{-t_n} = 1

If X_0 is the largest real solution to the characteristic equation, then N(T) approaches X_0^T for large T.

(I need to check the math here since Shannon just says this is a well known result and moves on.)

So now we have:

C = \lim_{T \to \infty} \frac{\log N(T)}{T} = \log X_0

2. The Discrete Source of information

This section of the paper focuses on defining what at discrete source of information is. For our purposes it can be thought of as a stochastic process which produces a stream of symbols.

3. Discrete Approximations to English

Here we come to word and sentence generation using stochastic processes. In particular Shannon takes us through the progression of constructing messages via:

Zero-order approximation. Symbols are individual letters with the addition of a space. Each symbol is selected with equal probability.
First-order approximation. As above, but the relative frequencies correspond to actual letter frequencies in English.
Second-order approximation. As above, but selection of a symbol depends on the previous symbol. I.e. follows a digram structure. Probabilities correspond to English.
Third-order approximation. As above, but using trigrams instead of digrams.
First-order word approximation. Start with a list of words and pick them at random with probability corresponding to their usage in English.
Second-order word approximation. As above, but selection frequencies depend on the previous word. I.e. follows digrams.

By the time we arrive at second-order word approximations, the generated messages start to resemble natural English albeit nonsensical.

4. Graphical Representation of a Markoff process

Identifies the previous stochastic processes as Markoff processes. They are represented as a set of states. Each state can probabilistically transition to number of states with some probability distribution. Each transition emits a symbol.

If the emitted symbol doesn’t depend on any prior symbol, then we only really need one state. It will have a self-edge for each possible emitted symbol.

digraph {
  nodesep=1.5;
  S -> S [label="0.2\n\n 'A'"];
  S -> S [label="0.2\n \n'B'"];
  S -> S [label="0.6\n \n'C'"];
}

If the emitted symbol depends on the previous symbol, then there will be as many states needed as there are symbols. This is the case for digrams.

digraph {
  A -> A [label="0.33, A"];
  A -> B [label="0.33, B"];
  A -> C [label="0.33, C"];
  B -> A [label="0.33, A"];
  B -> B [label="0.33, B"];
  B -> C [label="0.33, C"];
  C -> A [label="0.33, A"];
  C -> B [label="0.33, B"];
  C -> C [label="0.33, C"];
}

For trigrams, since they depend on the previous two symbols, require n^2 states.

5. Ergodic and Mixed Sources

In an ergodic process every sequence produced by the process is the same in statistical properties. […] Roughly the ergodic property means statistical homogeneity.

6. Choice Uncertainty and Entropy

Here we come to the part that is most often cited. The definition of entropy.

Shannon lays out three requirements for a measure of information H(p_1, p_2, .. p_n) which measures the information content of a set of outcomes each of whose probability is p_i:

H should be continuous in the p_i.
If all p_i are equal, i.e. p_i = \frac{1}{n} then H should be a monotonic increasing function of n.
If a choice be broken down into two choices, then the original H should be the weighted sum of the individual values of H.

The only H satisfying the above three requirements is of the form: H = -K \sum_{i=1}^n p_i \log p_i

The constant K is merely a unit of measure and can be set to 1.

Some interesting properties of H:

H = 0 only when all but one p_i = 0, and the remaining p_j = 1.
For a given n, the maximum H is reached when all p_i = \frac{1}{n}.
The joint entropy of two random variables follow the relation: H(x,y) \le H(x) + H(y) The condition of equality implies that x and y are independent.
Any change towards equalization of probabilities p_i increases H. I.e. if p_1 \lt p_2 and \delta \lt |p_1 - p_2|, then the entropy of p_1, p_2, ... p_n is lower than the entropy of (p_1 + \delta), (p_2 - \delta), p_3, ... p_n.
For two variables x,y,there’s conditional probability p_i(j) of y having value j assuming x has value i. This is given by: p_i(j) = \frac{p(i,j)}{\sum_j p(i,j)} The corresponding conditional entropy H_x(y) is given by: H_x(y) = - \sum_{i,j} p(i,j) \log p_i(j) This conditional entropy then holds the relation: H(x,y) = H(x) + H_x(y)
From 3 and 5, we have: H(y) \le H_x(y)

7. The Entropy of an Information Source

Let S_i be a set of states each of which produce some set of outputs s_j with probability p_{i,j}.

Each state S_i has an associated entropy H_i representing the entropy of all possible outputs from that state. I.e.:

H_i = - \sum_j p_{i,j} \log p_{i,j}

Each such state also has a probability of occurrence P_i.

The entropy of the entire information source is the weighted sum of the state-wise entropies. The weight being the probability of occurrence. Thus:

\begin{align*} H & = \sum_i P_i H_i \\ & = - \sum_{i,j} P_i p_{i,j} \log p_{i,j} \end{align*}

If the source is operating at some definite time rate, then the entropy per second, or entropy rate is defined to be:

H' = \sum_i f_i H_i

where f_i is the average frequency of state i.

Shannon says “Clearly H' = mH where m is the average number of symbols produced per second.” But in order to make the connection you need to observe that f_i = m \cdot P_i.

This is followed by a derivation that takes some work to follow. Let’s go through it step by step:

Suppose that the information source consists of a single state and n symbols each of which is emitted with a probability of p_i.
Now take a string of length N emitted by this source. The expected number of occurrences for symbol s_i is N p_i.
A message of sufficient length where all symbols appear at their expected occurrence count would have a probability of occurring of:

p = \prod_i p_i^{N p_i}

From there we see that:

\begin{align*} \log p & = \sum_i N p_i \log p_i \\ & = N H \end{align*}

Sometimes It's the Interviewers Who Suck

2021-03-20T23:09:49Z

I’ve done hundreds of interviews. I’ve also served in numerous hiring committees over the years.

Tech companies tend to have cultures built around tech people interviewing other tech people, typically in what is not-so-affectionately known as the “Tech interview.” This is good because candidates would be evaluated based on their actual skills instead of keywords in their resume. Interviewers are in effect choosing their future coworkers which makes the interview serve the interest of both the interviewer and the interviewee. However candidates – at least the one who didn’t get accepted according to exit polls – tend to hate this interview format.

The goal of the tech interview is to assess how well the candidate will be able to perform in the role as a tech worker. Unfortunately the typical problems that one would encounter during the course of a career in tech aren’t easy to replicate within a 40-60 minute interview. So partly the criticism is around the fact that such interviews bear little resemblance to what the actual job would entail. Deciphering what happens during an interview and distilling it into a prediction of future success is quite difficult in the best of times.

Making this much worse is the undeniable fact that many if not most tech workers are just not good interviewers.

This is as true as a claim that most tech workers aren’t good managers – whether it’s managing projects or people. Interview follow-up don’t take this factor into consideration quite as much as they should.

My anecdotal evidence is that when presented with the artifacts from a bad interview, hiring committees will disproportionately blame the candidate for it. After all if the candidate was really good, they would’ve potentially salvaged the interview. The problem disproportionately affects candidates who are near the middle of the skill curve where things can probably go one way or the other depending on the combination of question asked, and caliber of the interviewer.

Interviews at Google famously involve a bunch of paperwork where each interviewer is expected to produce some verbose artifacts detailing what happened. Since these artifacts come from the interviewer, hiring committees need to exercise some due diligence to discern what actually happened based on the writeup. So how do we go about spotting bad interviews?

The writeup is terse.

Not in and of itself a conclusive sign, but if the writeup doesn’t have any details that a committee could use to form a decision, it’s likely a sign that the interview didn’t really gather much information about the candidate in the first place. Typically these kinds of writeups are from novice interviewers who don’t know what to write about.

The writeup uses adversarial framing.

A very frequent problem specially with novice interviewers is that they go into the interview thinking that it is some sort of competition between the interviewer and the interviewee. The interviewer might consider it a bad thing if a candidate were to hit their question out of the park. Some of this comes from a certain level of imposter syndrome where they don’t want to come across as less smart than the candidate. Either way this mentality comes across during the interview as hostile or condescending.

The interviewer takes too much pleasure in the interviewee’s misery.

Each mistake is gloated on while points at which the interviewee obviously shone are muted.

The question is unnecessarily hard.

Solving a hard question that requires a great deal of time and thought is very satisfying. But it isn’t what you do on a day-to-day basis.

The interviewer allows the interviewee to march into the weeds without helping them back.

The interviewer has the luxury of having seen the question beforehand and also having studied possible solutions to it. They should be familiar enough with the question that they could spot misunderstandings, misconceptions, and common mistakes. How a candidate notices a misstep and how they dig themselves out is important to note. However, letting the candidate languish in a dead-end while the clock ticks away the remaining minutes of an interview does not help anyone.

Obvious misunderstandings are not corrected.

It is the interviewer’s responsibility to explain the problem clearly. The problem itself doesn’t need to be fully explained since there’s much value in seeing how a candidate explores and understands the problem area. However if it is obvious that the candidate has fundamentally misunderstood some critical aspect of the question, the interviewer should step in and intervene early. We all misunderstand things, specially when they aren’t clearly explained. It’s not useful to knowingly send a candidate down the wrong path.

The interviewer seemed annoyed or angry.

This one should be self explanatory. The candidate is anxious enough as it is. The interviewer should not frustrate the interviewee’s attempts at asking questions and seeking clarity around the question. It’s likely that while the problem is obvious to one, the problem area may be unfamiliar to others who may need extra help figuring their way around. In fact the ability to resolve such ambiguities and seek clarity are good. If the interviewer actively discourages such inquiry that’s on the interviewer – not the candidate.

One interview diverges significantly from others.

There’s more than one interview for a reason. If one interview highlights an area as being weak while another interview highlights the same area as being strong, then there’s something going on that needs scrutiny. There can be many reasons why a candidate bombs an interview. Not all of them say anything bad about their ability to function well in their job. The causes should be teased out.

–

What happens when an interview is pretty much unsalvageable? It might be that the writeup is inconclusive due to issues such as those highlighted above. A committee could ignore that specific interview and base their decision on the other interviews, or failing that schedule another interview to account for the lack of decisive information.

From a candidate’s point of view, it is really important that they report any unpleasant interactions to one’s recruiter immediately after the interview. This allows a recruiter to go to bat for you and perhaps get you another interview if appropriate.

On WontFixing Bugs

2021-03-02T22:15:06Z

This was partly inspired by a Tweet by a co-worker. The tweet is:

https://twitter.com/miketaylr/status/1366762904851279877”

For projects that are under-resourced but has heavy usage it is in fact quite normal to accumulate a large number of issue reports over time. Take Chromium for example. There are (at the time of writing) ~95,000 open bugs. Many of these are task tracking bugs rather than tracking actual product bugs. But still that’s a lot of real bugs.

What’s even more interesting is the accumulation of bugs. Typically for a “successful” project the rate of new issues being reported will exceed the rate of issues being fixed. Hence the growth.

What Matters …

Issues are often annotated to indicate some measure of severity and priority. These are two different axes of measurement:

Severity refers to the degree of impact that an issue has. How many users are affected? Is the effect isolated or spread across a large user base? Severity is supposed to summarize the answer to these questions.

Priority refers to what the issue triagers deemed to be the relative importance of the issue. A bug that affects a large number of users – hence a high severity – might not be a high priority due to mitigating factors. Typically a project owner would tackle higher priority issues first. Under-resourced projects – which is the case for pretty much every successful project I’ve seen – will accumulate lower priority issues across time.

Neither severity nor priority are static either. An issue with a low severity might be elevated to a higher severity if the affected population of users is discovered to be larger than originally assumed. Or the priority may be raised if the issue is blocking the fix for another more severe issue. It’s not uncommon for some low priority issue to be suddenly elevated to critical status once someone figures out how to exploit it.

In practice perceptual priority is the only one that matters. It doesn’t matter what the issue tracker label or score is. What matters is that there’s a contributor who’s motivated to fix it. The decisions of a contributor are complex and subjective. So while the most important issues will likely get fixed, just because an issue is real or important doesn’t mean that it will be addressed.

What this means is that it is important for issue reports to stick around. At worst they will accumulate more evidence of severity or clues to what the issue is. Often the desired outcome isn’t universal either. So an issue report also functions as an opinion catchment area where interested parties can hash out their differences.

And What Doesn’t …

However, the health of the project then depends on not only handling new issues, but also in how they shepherd existing or known issues.

Much like any other todo list, the effectiveness of the issue tracker decreases when there’s an overwhelming number of items to look at. The cognitive overhead of correct prioritizing increases dramatically, as does the time it takes for triaging.

Strategies That Don’t Work …

So projects tend to use various techniques to deal with the growing pile of issues. The more overwhelming it is, the more drastic the mitigation strategy.

One such strategy in action is what we see in the Tweet above where there’s a bot that automatically closes old issues that haven’t seen any activity. At first glance this seems to make sense.

The assumption seems to be that if the issue was important enough then an endless stream of users would pipe up on the bug with “me too”s to keep the issue open. The issue tracker is now a battlefield where issue reporters have to continually work to keep the issue alive until a contributor takes pity and addresses it.

Issue reporters themselves who are often volunteers who are electing to participate in a project can also burn out. If the issues they file are automatically closed, it’s frustrating to the reporters. Chances are that with every few issues that are closed due to inactivity, the project loses a contributor.

Another strategy is to have new issues start at some sort of unconfirmed state. Once the issue has been acknowledged and accepted it can proceed to some other state which makes the issue become a candidate for contributor consideration. This one is a bit better in that it doesn’t discard confirmed issues.

However, eventually the real issues will accumulate to the point where some sort of bookkeeping is needed. Along comes WontFix and friends. This is a state where the issue is marked as “this is a real issue but we will probably never get around to fixing it.”

Reporters nevertheless find this state to be no-less frustrating than an outright closure. WontFixed bugs don’t show up by default in issue searches, so they are effectively undiscoverable to future issue reporters. All time invested in that issue is now lost.

And One That Does? …

The alternative is to just leave the issue open, but lower the priority down to something that reflects the current opinion of the project maintainers.

This strategy acknowledges that the issue is real. It serves as documentation that the product has a known defect, and in addition it can serve the purposes of accumulating evidence of severity along the way. If the issue becomes prominent enough then it can be retriaged.

That’s it. So far I haven’t seen alternatives that are much better than this. I’m generally against mass issue closing events like declaring “bug bankruptcy.” The bugs don’t care about the state of the issue report. All you’ve accomplished is to throw away the accumulated history and evidence of a defect.

Deriving the Poisson Distribution

2020-12-11T16:13:39Z

Where does the Poisson Distribution come from?

A little bit of research¹ tells us that the distribution was originally introduced by Abraham de Moivre in 1710 in an article called “On the Measurement of Chance, or, on the Probability of Events in Games Depending Upon Fortuitous Chance” ² (not the original title).

A few steps that will get us there is laid out below.

Let’s start with a simple “rate” problem.

How likely is it that you’ll receive 12 visits next week given that you had 10 visitors this week?

This is clearly not a question to be answered based on a lot of information. However when the assumption is that on average you get 10 visitors a week, then the question sounds a bit more reasonable.

While we say the rate is 10 visitors per week, that’s obviously not going to be uniformly distributed. Maybe given the nature of your site, people are more likely to visit over the week-end rather than during the week. However for the purpose of this analysis we are going to assume that the rate is uniform.

Simplifying things a bit more, we can ask the question in a different way:

How likely is that that the rate of visitors for your site will be \frac{12}{7} per day next week given that the usual rate is \frac{10}{7} per day?

Or in another way:

How likely is that that the rate of visitors for your site will be \frac{12}{7 * 24} per hour next week given that the usual rate is \frac{10}{7 * 24} per hour?

Why do this? Without much explanation this reformulation gets us to a rate that’s less than one per unit of time being considered. This way we can now look at each time unit and assume that we’ll either get 1 visitor or none any given hour.

So each hour, there’s a probability of \lambda that there will be a user, and 1 - \lambda that there won’t. Here we are using \lambda = \frac{10}{7 * 24} for convenience. The latter is also the “rate” of events or arrivals. The term arrivals or occurrences is common in literature.

The probability that there will be 12 visitors is the probability that there will be 12 hour slots where there’s an arrival and the rest will be slots without an arrival.

Treating this as a list of one-hour “slots”, the outcome would look like:

0	1	0	0	0	1	0	1	1	0	0	0	0	1	0	…	0

The number in each box is the number of arrivals during that slot which is either 1 or 0. So there are 2^n possible outcomes where n = 7 * 24. Each 1 has a probability of \lambda and 0 has a probability of 1 - \lambda.

So for there to be exactly k arrivals – where k = 12 which is the target – there needs to be k ones and n - k zeros.

So the probability of one of those events happening is \lambda^k (1 - \lambda)^{n-k}.

But there are lots of ways of choosing k boxes out of n. In fact there is exactly \binom{n}{k} ways.

So the probability of getting exactly k arrivals is:

\Pr(k) = \binom{n}{k} \lambda^k (1-\lambda)^{n - k}

With me so far?

Astute readers will note that we did some hand-waving up there. Specifically we assumed that the possibilities for any 1-hour block is either there being an arrival or there not being one. This isn’t always the case since there could be more than 1 arrival in any given hour.

One way to address this is to consider increasingly smaller units of time. For example, if we divide the one-hour block into m equal sized pieces then it’ll be a bit more likely – assuming m \gt 1 – that there will be at most 1 arrival during that time period. In fact if m is very large, then it is increasingly likely that the choices are either 1 or 0.

So the number of boxes is now n*m, and the probability of an arrival during any single timeslot is now \frac{λ}{m}. Substituting these, our new probability is:

\Pr(k) = \binom{nm}{k} \left( \frac{\lambda}{m} \right)^k \left(1 - \frac{\lambda}{m}\right)^{nm - k}

Alright. We are getting somewhere now. The next step is to increase m all the way to \infty.

\begin{align*} \lim_{m \to \infty} \Pr(k) & = \lim_{m \to \infty} \binom{n*m}{k} ( \frac{λ}{m} )^k (1 - \frac{λ}{m})^{nm - k} \\ & = \lim_{m → \infty} \frac{(nm)!}{k! (nm - k)!} ( \frac{λ}{m} )^k (1 - \frac{λ}{m})^{nm - k} \end{align*}

The right side looks a bit daunting, the we can simplify things quite a bit by looking at each part separately.

For example:

\begin{align*} \binom{nm}{k}\left( \frac{\lambda}{m} \right)^k & = \frac{(nm)!}{k! (nm - k)!}\left( \frac{\lambda}{m} \right)^k \\ & = \frac{nm \cdot (nm - 1) \cdot (nm - 2) \cdot ... (nm - k + 1)}{k!}\left( \frac{\lambda}{m} \right)^k \\ & = \left( \frac{nm}{m} \right) \left( \frac{nm - 1}{m}\right) ... \left( \frac{nm - k + 1}{m}\right) \frac{\lambda^k}{k!} \end{align*}

Furthermore:

\left( \frac{nm}{m} \right) \left( \frac{nm - 1}{m}\right) ... \left( \frac{nm - k + 1}{m}\right) \frac{\lambda^k}{k!} = \frac{\lambda^k}{k!}\prod_{i=0}^{k-1} \left(n - \frac{k - i}{m}\right)

As \lim_{m → \infty} we see that \frac{k-i}{m} tends to 0. This makes our lives a bit easier.

Now we have:

\begin{align*} \Pr(k) &= \lim_{m \to \infty}\frac{(n \lambda)^k}{k!} \left(1 - \frac{\lambda}{m} \right)^{nm - k} \\ &= \lim_{m \to \infty}\frac{(n \lambda)^k}{k!} \left(1 - \frac{\lambda}{m} \right)^{nm(1 - \frac{k}{nm})}\end{align*}

Remember that:

\lim_{n \to \infty} \left( 1 + \frac{x}{n} \right)^n = e^x

So substituting n \to nm and x \to - \lambda n gets us:

\lim_{nm \to \infty} \left( 1 - \frac{\lambda}{m} \right)^{nm} = e^{- \lambda n}

Now going back to \Pr(k):

\Pr(k) = \frac{(n \lambda)^k e^{- \lambda n}}{k!}

Let’s do another substitution. Remember that \lambda is the number of arrivals during n time periods. If we consider n time periods (originally hours) to be a single unit of time, then our rate of arrivals is \lambda n. So without loss of generality let’s substitute \lambda n \to \lambda.

This gives us:

\Pr(k) = \frac{\lambda^k e^{- \lambda}}{k!}

… which is the PDF of the Poisson distribution.

n.b.: In retrospect it would’ve been easier to consider the unit of time to be n hours in the first place, but I’m too lazy to go back and rewrite everything.

…

Also known as reading the relevant Wikipedia article. In this case that’s the History section of the article on the Poisson distribution.↩︎
The original article in Latin is available via The Royal Society Publishing.↩︎

Ten Years

2020-12-09T02:41:50Z

It’s been ten years since I started at Google. The work anniversary fell on 6th of December.

Ten years ago my wife and I made our way to Mountain View for my orientation; all excited for a brand new chapter in our lives. After spending a week in Mountain View / Palo Alto we both decided that California was not for us. But that’s beside the point.

I was elated. This was pretty much everything I dreamt of as a kid growing up in Sri Lanka. Except those days we dreamt of someday working for Microsoft or starting our own software company.

One afternoon during orientation – maybe it was the evening, I don’t remember – I was walking by a bunch of outdoor tables next to one of the larger café’s at the Google HQ. As I walked by I overheard a Googler on the phone telling someone how they just had a really shitty day. I remember being confused. How could someone have a bad day at Google? Inconceivable.

Obviously this feeling didn’t last. People normalize things quickly. The problem with achieving your dreams is that now you have to make more; an eternal treadmill. It’s nice to periodically look back though. The novelty isn’t there anymore, but it’s humbling to recall how much we dreamt for the things that we now take for granted.

Looking back at the last years, the things that I’m really proud of aren’t my code contributions. If any aren’t already obsolete then by now it’s become some legacy crap that some engineer has to maintain.

But the things that I really remember are the three times I talked hiring committees into hiring someone they had decided not to hire. Those were good candidates whom I thought didn’t get a fair shake.

Also I clearly remember the smart people who acknowledged and appreciated my work. I also remember the ones that took credit for my work or didn’t treat me with dignity, but I don’t think about them that often.

I remember the big projects that I initiated, the big ideas I sold, and the ones I managed to pull off.

Then there are the low points. Moments when I let people down or burned a bridge or two for the wrong reasons.

But overall, looking back there isn’t much there. It’s surprising how little really matters in the long run.

I should remember that the next time I feel like staying up late trying to get something done because I feel some sort of obligation or bigger purpose in doing it. Often there is none.

Visualizing Internet Users

2020-11-21T16:37:53Z

What does 0.3%¹ of internet users look like?

As of this writing the global population hovers around 7.7 billion according to the World Population Clock ².

The International Telecommunications Union “estimates that at the end of 2019, 53.6 per cent of the global population, or 4.1 billion people, are using the internet.”³

So a 0.3% of the internet would be a roughly 12,300,000 or around 12.3 million. A nice little number.

From 50 to 100,000

It’s interesting to try and visualize this. An example of visualizing populations for small numbers can be found at Visualizing Crowd Sizes. It goes from around 50 people up to 100,000 people.

While 100,000 is interesting we need to go all the way to 12.3 million.

200,000

A particularly macabre topic these days since the United States surpassed 200,000 deaths due to COVID-19 around September of 2020. ctpost ran an article online title “What does 200,000 deaths actually look like”. There are some interesting comparisons there, but none make for a good visual. It’s just an interesting number right now.

As for crowd sizes, 200,000 was the estimated number of people who attended the March for Our Lives in D.C.. There aren’t any photos capturing all attendees in one shot, but there are numerous pictures in this CBS News article to give you a good idea.

In more terrible examples, this article on dw.com suggests that the crowd size pictured is roughly around 200,000 people.

More than 200,000 conservative Muslims gathered in Indonesia’s capital Jakarta on Friday to protest against the city’s Christian governor, who is facing trial for blasphemy, police estimated.

500,000

We are heading into shaky ground here. A 800*600 picture only contains 480,000 pixels. So we are now reduced to indicating population by inference. It is nolonger possible to capture 500,000 people in a webpage sized picture while showing each person individually.

This Reddit post claims that there are 500k people in the picture, but I couldn’t readily find any good references backing up that claim.

As is tradition by now, another protest that drew 500,000 people is an anti-violence rally in Barcelona as reported by NBC. While there’s video, the content doesn’t quite capture the massive number of people. Also it’s “nearly 500,000” people.

Image source

750,000

Off to protests again. This time we are looking at Barcelona where 750,000 Catalan independence supporters are demanding the release of their leaders according to this article.

But wait, the Guinness World Record for teh largest charity walk/run is 283,171 people at a single venue. Have a look at that picture:

It seems that picture of 750,000 people doesn’t quite capture it because clearly there are more people in the picture that’s supposed to be 283k people. Which of course goes to show how hard it is to really capture how massive a crowd 750k makes.

In an article on Wired that discusses how hard it is to estimate crowd sizes, they note⁴:

Every time a ton of people gather in one place, there are all sorts of pronouncements made about how many folks really showed. The figures rarely agree; Glenn Beck’s rally on the National Mall this summer attracted a million people, according to Beck cohort Rep. Michelle Bachman. CBS News pegged it at more like 87,000.

And More

I’m going to give up at this point. Pictures purportedly of around one million people are mostly parts of crowds that have been estimated to be that big. Forget about 12.3 million.

Why 0.3% of the internet population? It’s a number that came up when trying to come up with a reasonable sample size for estimating the diversity of the internet population. There’s not much special about it other than that.↩︎
The world population clock. This link is for the current number at whatever time you clicked on the link assuming the site is still there. As of November 20, 2021, the number was 7.7 billion. I’d link you to an archived page but that doesn’t seem to work.↩︎
Statistics pages on itu.int project even higher numbers than 53.6% by the end of 2020, but it’s possible numbers won’t behave according to historical trends due to the effects of COVID.↩︎
How Many People Are in Tahrir Square? Here’s How to Tell ↩︎

Things You Can Do With Neovim and Vscode That You Can't Do With Neovim Alone

2020-09-19T00:00:00Z

This is the third time I’ve tried in earnest to switch to VSCode from Vim (or Neovim in my case) for reasons not all of which are relevant to this post. But I love Vim style modal editing and I’ve grown accustomed to some features that aren’t present or not well emulated in VSCodeVim.

Enter VSCode Neovim. I was intrigued by their claim to integrate Neovim instead of trying to emulate Vim. After going through the code a bit I thought this extension alone warranted a new attempt at switching to VSCode. So far it’s going great.

How VSCode Neovim Works

Here’s the gist of how this extension works. Feel free to skip to the good stuff below but this section is good background information for understanding why it works the way it does.

Invokes an embedded Neovim instance.

Code for this is in the main_controller.ts (linky) if you are curious about how exactly it’s implemented and the commandline options that are passed in.

You’ll also notice that the code invokes --cmd source ${neovimSupportScriptPath} which runs vim/vscode-neovim.vim (linky) before running your init.vim file. This script is what sets up the g:vscode flag and sources all the other vscode-*.vim scripts except vscode-options.vim. The latter runs after your init.vim in order to force some settings that would otherwise break the extension or cause Neovim to be confused.

So far this is pretty standard fare for how Neovim would be embedded for a custom UI. But things get a bit interesting from the next step.

Creates a new Neovim buffer and window corresponding to a VSCode editor.

The buffer thus created is a plain text buffer without any bells or whistles. Contents of the buffer comes from VSCode¹. VSCodeNeovim synchronizes the content between the Neovim buffer and the VSCode editor. Thus the relationship between the contents and the corresponding file on disk is mediated through VSCode. This is a key difference between other custom UIs and this one and one that opens many possibilities.

VSCodeNeovim defers to VSCode for handling insert mode. VSCode handles autocompletion and other assistive features. You’ll have to get used to your insert mode customizations not working, or replicating that with VSCode keyboard mapping customizations. When exiting insert mode the extension asynchronously synchronizes the buffer contents – which at this point would include the changes that were introduced – with Neovim. The code for synchronizing the VSCode editor contents and the Neovim buffer is in buffer_manager.ts (linky) which is also a good place to look if you want to know the details about buffer options that are being set.

Of course I’m eliding lots of detail here including integrations for other areas of UI. But I think this captures the pertinent details. The extension is a pretty impressive piece of work which is well worth a gander. Now on to the good stuff.

Responsive Neovim Experience Over A laggy SSH Connection

This gets to the reason why I’m trying to move to VSCode in the first place. I work on a project consisting of thousands of C++ source files. My home-office doesn’t have enough hardware to do regular builds, or in fact any build at all. The beefy machines are in a remote location accessible via SSH.

I’ve been using Neovim over SSH for a while, but the latency really bothers me. And no, neither SCP nor SSHFS scratches this itch because I won’t have access to language smarts like clangd which require access to all the source files and some build tooling.

VSCode Remote Development was a nearly perfect answer with the exception that VSCodeVim didn’t quite have all the functionality I needed. VSCodeNeovim addresses that last piece of the puzzle.

VSCode + Remote SSH + VSCodeNeovim + Neovim gives me:

On the remote machine:
- Source files.
- Build tooling.
- clangd process invoked via vscode-remote.
On the local machine:
- VSCode.
- VSCodeNeovim.
  - Neovim instance embedded by VSCodeNeovim.
    - Neovim plug-ins and normal mode keybindings.

Since the embedded Neovim instance is local the editors are super responsive. Completions via clangd is still laggy as one might expect. But that’s something I’m willing to live with.

Proportional Fonts With Neovim

Okay, so why would anyone actually want this? Turns out I’m not the only one. In my case I prefer monospace for regular code, but I’d much much rather read comments and prose in proportional font. Also, I now spend a lot of my time writing Markdown with LatexMath. Staring at monospaced prose all day is not my cup of tea.

This is what it looks like when I’m editing this post in VSCode (in Markdown).

The best part is that I can use my familiar Vim/Neovim keybindings and modal editing. I had to make a couple of customizations and I only have this set up for Markdown. But it works fairly well. One gotcha is that the cursor jumps horizontally when moving the cursor up or down since the columns are no longer neatly aligned.

Monospace Code With Proportional Comments?

What I really want isn’t proportional fonts everywhere. Rather I only want proportional fonts for prose. In code, this would take the form of comments.

Unfortunately, VSCode doesn’t support changing the font for specific syntactic tokens or scopes. That would’ve been ideal because it would let me configure things so that comments are in proportional font while the rest is in monospace. Also in the case of Markdown, I’d be able to make everything proportional except for inline code and fenced code blocks. Alas we are not quite there yet. Several feature requests already exist², but hasn’t seen much activity.

Takeaways

VSCodeNeovim is a functional replacement for the avid Neovim or perhaps Vim user who wants it both ways. Lots of caveats exist including the fact that Neovim mode only really works inside editor windows.
VSCodeNeovim + “Remote SSH” is an awesome combination for editing remote files using Neovim with the latency of a local Neovim instance.
You can use proportional fonts with VSCodeNeovim for super double awesomeness.

Not quite the whole story because it’s possible a buffer be created inside the Neovim instance and have it be mirrored in a VSCode editor. But that’s probably not what you’ll be using on a regular basis.↩︎
A quick search in the VSCode repository with this query yields several issues that request for pretty much the same thing: Allow setting fontName in tokenColorCustomizations. There’s also this curious experiment which patches in a different font for italics so that you can use the existing fontStyle: "italic" option in tokenColorCustomizations to switch fonts.↩︎

Why 'Strong Opinions Weakly Held'

2020-08-28T19:17:16Z

Origins of the phrase

The title of this blog is Strong Opinions, Weakly Held. The same concept shows up elsewhere are Strong Opinions Loosely Held. An eponymous essay by Paul Saffo in (2020) introduced the world to this concept.

In his essay – which isn’t very long if you would like to read the whole thing yourself – he lays out the concept as follows:

I have found that the fastest way to an effective forecast is often through a sequence of lousy forecasts. Instead of withholding judgment until an exhaustive search for data is complete, I will force myself to make a tentative forecast based on the information available, and then systematically tear it apart, using the insights gained to guide my search for further indicators and information.

Criticism

Cedric Chin’s critique (2020) comes down to two points that the author summarizes as follows:

In my experience, ‘strong opinions, weakly held’ is difficult to put into practice. Most people who try will either:

Use it as downside-protection to justify their strongly-held bad opinions, or
Struggle to shift from one strong opinion to another.

The reason it is difficult is because it works against the grain of the human mind.

So don’t bother. The next time you find yourself making a judgment, don’t invoke ‘strong opinions, weakly held’. Instead, ask: “how much are you willing to bet on that?” Doing so will jolt people into the types of thinking you want to encourage.

Michael Natkin on (2020) points out the fallacy of assuming that strong opinions will be reliably challenged:

The idea of strong opinions, loosely held is that you can make bombastic statements, and everyone should implicitly assume that you’ll happily change your mind in a heartbeat if new data suggests you are wrong. It is supposed to lead to a collegial, competitive environment in which ideas get a vigorous defense, the best of them survive, and no-one gets their feelings hurt in the process.

[…]

What really happens? The loudest, most bombastic engineer states their case with certainty, and that shuts down discussion. Other people either assume the loudmouth knows best, or don’t want to stick out their neck and risk criticism and shame. This is especially true if the loudmouth is senior, or there is any other power differential.

Reconciliation

Both cases above illustrate the damage caused by the disconnect between the holder of the opinion and others about the degree of confidence with which the opinion is held. A strongly held – and thusly communicated – opinion has a higher chance of being misinterpreted as conviction.

Saffo’s original claim still holds that having a potentially inaccurate position is better than having none at all. Such a position elicits arguments and opens up avenues for investigation that wouldn’t otherwise materialize in the absence of any other catalyst.

Cedric Chin’s essay points out a simple fix, where the holder of the opinion also states the degree of confidence. This is meant to dress up the opinion more as a target than a decree, thus inviting debate instead of suppressing. I believe this to be good advice.

Back in my college days, I remember a particularly engaging professor asking a question like “is claim A true or is claim B true?” After observing a lackluster show of hands he remarked that of the three groups of students – those that chose A, those that chose B, and those that chose neither – the group that will leave the lecture having learned the most is the group that gets the answer wrong. He advised all of us to “get some skin in the game,” even if you don’t know the answer.

There is some truth to this. Of course a corporate meeting where we need to elicit debate from a group of people with varying levels of confidence is a very different venue than a group of students answering a question. But a similar principle still applies. The goals might be different, but it’s still better to cajole everyone to get some skin in the game instead of not passive silence.

What ‘Strong Opinions Weakly Held’ Means To Me

Having an opinion is better than not having one. Accmulation of raw data is great, but it’s even better when you intentionally accrete them into something meaningful. Any interpretation of information beyond establishment of fact is probably going to be in the realm of opinion.
An opinion expressed is better than one that is not as long as it’s open for criticism. You wouldn’t find out if you were wrong if you didn’t tell anyone.
That it is weakly held is as important to communicate as the opinion itself. Doing so invites criticism and discussion instead of suppressing them.

References

Strong opinions loosely held might be the worst idea in tech - the glowforge blog. (2020, September 4). https://blog.glowforge.com/strong-opinions-loosely-held-might-be-the-worst-idea-in-tech/

Strong opinions weakly held : Paul saffo. (2020, September 4). https://www.saffo.com/02008/07/26/strong-opinions-weakly-held/

“Strong opinions, weakly held” doesn’t work that well. (2020, September 4). https://commoncog.com/blog/strong-opinions-weakly-held-is-bad/

Names for Name Conventions

2020-08-24T16:32:45Z

Names I’ve seen used to refer to naming conventions:

snake_case, hacker_case, unix_hacker_style : Everything is lower case, though exceptions exist. E.g. HTTP_foo_bar.
SCREAMING_SNAKE_CASE : Usually reserved for macros and constants.
camelCase : The first letter is lowercase.
PascalCase : The first letter is uppercase.
kabob-case : Like snake_case, but uses dashes instead of underscores. Common for command-line options¹, CSS styles, commands (e.g. git-receive-pack).

While looking idly looking for details on this, I stumbled on this Medium post which is relevant.

Opinions vary on whether the correct convention for command line options is snake_case or kabob-case. Long form options are often written as --long-option, but can also be --long_option (where the delimiter within the option is _).↩︎

How This Blog Works

2020-07-27T04:57:17Z

Requirements

Math. Preferably Latex.
Citations. I like to cite stuff.
Figures and pictures and references to thereof.
Be able to export to Github Flavored Markdown.

The Stack

All documents are in Markdown. Specifically Pandoc Markdown. Obviously, the source is processed via pandoc.
The site itself is compiled using Hugo.

The current version (as of this writing) doesn’t support any configuration when using Pandoc for markup processing. By default Hugo only passes the --mathjax option to the pandoc invocation. So I had to make some local modifications so that I can specify filters and some other options to support citations and references.

Pull request is here.
Hugo output – which is a static HTML site – gets pushed out to a separate private GitHub repository.
Netlify monitors this separate GitHub repository and updates this blog.

As a bonus, the generated static content is pushed out to my GitHub Pages site. So there’s an easier to remember URL than https://අ.com.
Secondary sites are fed via Pandoc which converts from Pandoc Markdown to Github Flavored Markdown.

Why secondary sites? For various reasons, the primary site for some of the articles need to be their own GitHub repository. E.g. https://github.com/asankah/identity-domains .

For all those sites, the primary source is the blog. That way I can use Pandoc, use citations, use math etc. and things will “just work” when publishing to another site that expects GitHub flavored Markdown.

The Comcast Technician Problem

2020-07-13T17:21:51Z

The Story

As is a rite of passage in these parts, I waited for the Comcast¹ technician to arrive during their scheduled appointment window. In preparation I had scheduled a day of working from home.

As the appointment window unceremoniously came to a close vacillated over when it would be a reasonable time to call Comcast again to see what’s going on. That’s when I noticed the technician’s truck pull up beside our house. They rolled down the window and surveyed the scene for a brief moment, looked at something inside their truck, rolled up the window and promptly drove off.

“That’s it,” I thought. That was the right time to call Comcast. That’s what I did. After a long wait² the friendly service person on the other end of the line told me that the according to the technician nobody answered the door. It took some convincing on my part to establish that:

I was there the whole time. Nobody rang the doorbell.
The doorbell works.
The technician — assuming that the person who drove by was the scheduled technician — did drive by, but they didn’t get out of the vehicle to ring any doorbell.
My phone didn’t ring, nor does it indicate any missed calls.
My phone works and has a signal.

They were kind enough to ask the technician to return, but noted that the appointment window would now be unpredictable due to prior appointments. Sure.

Much later that evening, the same technician who drove by earlier showed up. After they were done, I thanked them for showing up and mentioned what happened to the original appointment slot. I didn’t mention anything about seeing the same technician before. After all I was just happy that I was done with this and didn’t want to cause any trouble.

He said that technicians sometimes skip slots when they are running behind schedule.

The Problem

Looking at what happened from the perspective of the technician, the logic behind all this becomes a little clearer.

You see, had the technician had tried to honor the original appointment — one which they were already late for — they would’ve been even more late for the next one. This would continue down the line until by the end of the day pretty much all of his appointments that afternoon would’ve registered as late.

Had they skipped one – my appointment as the case turned out – then they’d potentially be on time for the rest of their appointments. The one they skipped could later be justified as one where the customer was not at home or any other reasonable excuses. Even if they were truthful and were penalized for missing an appointment, it’s likely that they’d still be better off than being late for so many more appointments. Either way, the outcome is both better for the technician, and more importantly, for their supervisor.

In the books, the two situations look like this:

Technician’s Choice	Successful appointments	Late appointments	“No-show”s ³
Honor all slots	g	l	n
Skip when late	g + l - s	0	n + s

The outcomes depending on the technician’s decision on what to do with late appointments.

g are the good slots where the technician is able to show up during the appointment window. l are those where the technician was not available during the appointment window. In either case some number of households are not going to be home or not ready for service. Those are n.

When the technician decides to skip late appointments some number – s in the table above – will turn into additional no-shows. However, skipping these will mean that the technician can make it to l-s appointments on time.

Assuming l-s > 0, then the outcome of the skip when late strategy is strictly better for the technician’s supervisor. I can’t imagine there being any incentive for the supervisor to penalize this strategy. So the technician also gets off scott free.

Why This Is Relevant

It’s easy to harp on Comcast⁴ and some random lowly technician. But this incentive pattern is everywhere. Even at Google and s.

Every time you measure someone’s success on how well they handled a number of issues or cases without taking into account the cost they incurred to get there, you are creating a Comcast Technician problem for your subordinates.

But Is It Bad?

Dropping tasks instead of unbounded accretion of delay is sometimes a better strategy.

One might even argue that this is the cost of doing business and it would actually makes sense. One customer’s (or user’s or client’s or patient’s) convenience doesn’t outweigh the convenience of many others.

There could be additional complicating factors involved:

What’s the cost of skipping or rescheduling a case? Is the cost bourne by you? How much of the cost is bourne by the customer?
How do you choose which case to skip? It might be that the case to skip isn’t the one that’s late, but one later down the schedule that’s much cheaper to reschedule or less damaging to cancel.

In either case, be forgiving when you inevitably find yourself as the one who got the short end of the stick.

It just might be that all parties involved are rational actors, and there’s really not a great alternative. Sometimes it’s just life.

The Verge. The worst company in America. (Accessed July 13, 2020)↩︎
Urban Dictionary. Long Wait (Accessed July 13, 2020)↩︎
“No-shows” are just going to be rescheduled. It’s not the end of the world.↩︎
The Verge. The worst company in America. (Accessed July 13, 2020)↩︎

Vim: Use Drop Not Edit, SBuffer Not Buffer

2020-07-07T16:02:20Z

I’m going to assume you are a Vim user.

Say you have a several windows open in Vim and you want to edit another file. Using :edit works if you want to open the file in the current window unconditionally.

But that’s often not what you want. If you have the same file open in another window, then the most efficient and least disruptive thing to do is to switch to that window.

drop

That’s where :drop comes in. In case you aren’t familiar with the drop command, have a look at the help page in your favorite editor (:help drop) or here on the web. But the gist of it is that for most people¹ :drop {file} will:

Switch to the window in which {file} is already open.
Switch the buffer in the current window to {file} otherwise.

The :drop command is intended to be used as the “open buffer” command for automated tooling like source browsers. For example, you can configure coc-nvim GitHub to use :drop as the command used for opening a source file by setting coc.preferences.jumpCommand to drop. See :help coc-configuration for details on how to set this, but basically it’s as simple as opening up the configuration using :CocConfig and then adding a "coc.preferences.jumpCommand": "drop", mapping to the configuration JSON file.

switchbuf or swb

Or perhaps configuring each plugin is too much work. Enter switchbuf. This is a global setting in Vim that controls how Vim switches between buffers.

Setting set switchbuf=useopen has the same effect when invoking buffer. Note that using sbuffer, sbnext, or sbrewind automatically behaves as if this is the case. I.e. they will use an existing window if one already has the buffer open.

Setting set switchbuf=usetab is similar to the effect of useopen except that it will look through windows in other tabs.

See :help switchbuf or here on the web for full details.

Assuming you have set hidden that is. See :help drop for more details on what happens when the current buffer cannot be unloaded.↩︎

Notes from Crash Course's Videos on Fact Checking Information You See On The Internet

2020-07-04T00:00:00Z

Fact Checking

Notes from Crash Course’s excellent series about fact checking information you see on the internet. The series starts here.

Three Questions

How is behind this information?
What is the evidence for their claims?
What do other sources say about the organization and its claims?

“A question wrongly put.”

Lateral Reading

All information is produced by someone, and is produced for a purpose. The lines between motives are unclear. There are always multiple motives, and they don’t always conflict.

“Who made this and why?”

We read websites like we read books. We read vertically. You are only seeing what the creator wanted you to see. It is often impossible to tell reliable information from unreliable information.

A better alternative is to leave the site and search for corroborating information. Look for conflicts of interest that are left undisclosed which is a red flag.

Newspapers can be a good start. These are now digital media companies. Sometimes they have their own point-of-view. Sometimes these are stated explicitly and/or are obvious. But that’s not necessarily always the case.

There is no magic arbiter of truth. All of these is created by humans and humans make mistakes.

“The media” doesn’t exist. It is a large and diverse industry. It is, however, possible to take these diverse points of view into account.

You should use Wikipedia. Not every article is perfect, but you should see verifiable citations.

Because no source is perfectly objective, one might conclude that no source is trustworthy. This is not true.

Deciding Whom To Trust

It is easy to be mislead. None of us have the time to be an expert in everything.

We have to trust information from outside of ourselves we have to find a way to accredit and trust experts, even though they will be wrong some of the time.

“Listen to me”bloviate””

Before we place trust in a source, we should verify the authority, and the validity of their evidence.

Investigating A Source’s Authority

The author or authors’ professional background.
The process they used to produce that information.
The systems that are in place to catch mistakes and correct them.

Everyone has a perspective influenced by their lived experience. This doesn’t necessarily mean that they are unfairly biased. We should take their perspective into consideration when examining their arguments. They may be presenting their information in a way that’s persuasive.

“Opinion”, “Analysis” etc. mean that the work intends to be persuasive rather than informative.

STOP, THINK, and LOOK AROUND.

Each bit of information we have about a source gives us a lens to filter out their perspective.

Using Wikipedia

When used correctly Wikipedia can be a great place to start when verifying the legitimacy of a source. It’s breadth can exceed any newspaper. It gives you the general lay of the land when exploring or researching a topic.

Wikipedia has become the internet’s largest general reference work.

Their content policies:

A neutral point of view.
No original research.
Verifiable.

Modern Wikipedia has rigorous and robust mechanisms to keep things in check.

Wikipedia is not a one-stop shop for in-depth research. Don’t cite Wikipedia. That’s not a good look. And encyclopedia is not a source.

Citations help bring up more information from reputable sources.

Treat Wikipedia as a launch pad.

Wikipedia is another tool in your information evaluation tool kit.

You go there for general overview a topic or a stepping stone to more references, or to use as one lateral reading source among several.

And as long as your know how and when to use it appropriately, Wikipedia can be a great friend.

Evaluating Evidence

As you get older “because I said so” doesn’t cut it anymore. You need to provide evidence and that evidence should be convincing.

“What is the evidence?”

Often when the evidence does not support the claim, it is misleading or disinformation. Lack of evidence should be suspicious immediately.

The mere existence of evidence doesn’t support the claim either.

Not all evidence is cerated equal.

The evidence a source provides should come from another reputable source.

“Thousands of people never conspire to do anything secretly.”

Spurious correlation: The implied causal relationship between events that are coincidentally linked.

The quality of our evidence like the quality of our information, affects the quality of our decisions.

Evaluating Photos & Videos

Photographs feel real and authentic. Even when images aren’t altered, they are selectively framed, or have their context falsified (e.g. with a false explanation, or date).

We are used to thinking that “seeing is believing”.

Thanks to their power, images are a very common form of online evidence. But just like data or text image-based evidence can be relevant and reliable, or irrelevant and unreliable.

When you encounter a suspicious image online it is crucial to investigate who is behind it and whiter they are a reliable source. We also must verify the surrounding context and whether it supports any claims being made.

Use reverse image search on Google to track down sources. Turn to fact checking sites like Snopes and Politifact.

Videos can also be dramatically altered. It is important to know where a video came from, who created it, and whether it’s been altered before you believe what you see.

Data and infographics

Easier CLI for ad-hoc Ansible tasks and playbooks

2020-06-01T00:00:00Z

This article is about using Ansible on the command-line.

Problem:

You …

… want a convenient way to do ad-hoc tasks on a bunch of machines.
… already have a few playbooks and machine groups defined in your inventory.
… don’t want to invest too much into Ansible because you have other things to do and Ansible is just a tool to deal with one aspect of what you want to do.
… live on the command-line.
… think ansible-playbook and ansible commands have too many knobs and too few knobs at the same time.

A Possible Solution

Taking a page from old school Posix tools, let’s make a script that encodes Ansible group names into the filename.

Then you can go from this:

$ ansible atlanta -m copy -a "src=/etc/hosts dest=/tmp/hosts"
$ ansible webservers -m service -a "name=httpd state=started"

To this:

$ atlanta copy "src=/etc/hosts dest=/tmp/hosts"
$ webservers service "name=https state=started"

Or from this:

$ ansible-playbook -l atlanta go-to-the-moon
$ ansible-playbook -l webservers do-the-other-things

To this:

$ atlanta go-to-the-moon
$ webservers do-the-other-things

Setting Up

These are just examples. Please don’t spend time bikeshedding names because you can use whichever names you like.

Download the contents of ansible-by-proxy from here. It’s a starter script that includes a bunch of sensible settings expressed via environment variables. Remove or change these to match your needs.

Note that the script expects to find a sub-directory called ansible in the same directory as the script which contains the playbooks and the inventory file.
Store this file somewhere on the system PATH.

Don’t forget to mark the file as executable if it isn’t already.
```
$ chmod +x ansible-by-proxy
```
Create symlinks to the file that have the same name as hostnames or groups as described in the includes:
```
$ ln -s ansible-by-proxy atlanta
$ ln -s ansible-by-proxy webservers
```
and so on.

That’s it. Now you can invoke playbooks and tasks just by using the name of host or group.

Identity Domains

2020-05-01T00:00:00Z

An Identity domain is a scope within which we consider it a given that the user has a shared or trivially joinable identity.

This is a privacy boundary, not a security boundary. Hence it assumes that where possible all sites share information.

Above is a diagram showing relationships between clusters of documents and identity domains.

foo.example and baz.example belong to the same first party set.
a.example has no first-party relationship with any other domain.

The Identity domain has the following properties:

All active documents in descendent browsing contexts belong to the same identity domain as the active document in the top-level browsing context.

See each cluster of browsing contexts in the diagram above.
All active documents in top-level browsing contexts that share the same site also share the same identity domain.

In the diagram above, foo.example in cluster 1 and baz.example in cluster 2 belong to the same first-party set, hence they are considered to belong to the same identity domain. However a.example in cluster 3 does not belong to the same identity domain because there’s no first party relation between a.example and any other top level domain in the diagram.
All private client state including but not limited to open sockets, socket pools, cookies, storage, permissions / content settings, transient caches of credentials, cached resources, service workers, and shared workers accessible to any document in an identity domain also belong to the identity domain.
Identity domains don’t span browser profiles.

See clusters 1 and 4 in the diagram above. Both share top-level origins, but don’t share identity domains because they are in two different browser profiles.
Identity domains don’t span browsers.

Derived from above.
Identity domains don’t survive browsing data erasures.

Erasing an identity domain – and thus an identity – requires destroying all private client state mentioned previously. Hence erasing an identity domain involves discarding all Document objects and workers in that identity domain.

Note that in the existing web privacy model third-parties can trivially associate identities across top-level contexts. Thus the entire internet essentially amalgamates into a single identity domain.

The boundaries discussed herein require moving to a different model of identity on the web. The privacy threat model guiding this model is discussed in PING’s Target Privacy Threat Model document which is a work in progress as of this writing.

Additional Notes and Observations

The identity domain is a privacy boundary. The boundary is something that’s asserted by the user-agent and doesn’t depend on the cooperation of sites. I.e. assumes that sites involved are worst case actors.
- Not to be confused with security boundaries like those imposed via Content Security Policy.
The definition assumes that sites can share information out-of-band. The identity domain boundary does not require sites to cooperate.
The same site can appear in multiple identity domains. For example, re-identification is equivalent to a single site joining an identity across disjoint domains.

One can also imagine a user-agent that maintains multiple identity boundaries for the same top-level site for the purpose of allowing multiple sets of cookies to be used from the same device.
Two identity domains can be joined by sharing a unique identifier between the two domains.
- WRT federated identity (as detailed in WebID) multiple relying parties join identity domains by virtue of shared unique identifiers like email address. Similarly identity providers can also generate and use unique identifiers linking domains.
  
  We likely need another term for describing externally joined identity domains. In such cases the UA doesn’t necessarily know or can prevent joining of identity domains.
Concepts like “logging out” can be defined in terms of destroying an identity domain on the User Agent.

Ephemeral Fingerprinting On The Web

2020-04-01T00:00:00Z

TL;DR:

Any web observable property whose changes are concurrently observable by multiple top-level sites can lead to cross site identity joining.
This method of identity joining does not require additional coordination between multiple first parties. A single third party embedded within multiple first parties can also use this method.

Background

Figure 1: Two sites observe a sequence of device orientation changes at times 𝒕₀, 𝒕₁,𝒕₂ .

All sites on the same UA instance share a clock and therefore can agree on the timestamps with a small margin of error. The triplet 𝒕₀, 𝒕₁,𝒕₂ has a high probability of uniquely identifying the user. The two sites can thus use these observations to conclude that the observations originate from the same user.

As illustrated above, one or more low entropy signals observed concurrently can be used to identify a user with a high degree of confidence. Let’s call these ephemeral fingerprints. This document discusses two types:

The sequence of timestamps corresponding to observed changes of a volatile surface can be used for identification. Let’s call these correlated events.
A stream of observations of a volatile surface can be identifying. Let’s call these unique event streams.

Signals considered for ephemeral fingerprinting don’t need to be highly identifying by themselves. The privacy budget proposal does not adequately account for fingerprinting based on concurrent observations of low entropy signals.

Device orientation, from our earlier example, can take one of two values (portrait or landscape) and is unstable. Thus a single sample of device orientation carries almost no information. I.e. A recorded observation of device orientation doesn’t help at all with identifying the user at a later time. However the timestamps corresponding to orientation changes could have identifying levels of entropy.

These are not new. For example, this is discussed by Van Goethem et. al. ¹

who calls these “Cross-Session Events” (§ 5 of linked paper). Potential ephemeral fingerprinting surfaces also get flagged during standardization discussions ( Example: Polling enumerateDevices, Example: Ambient light events).

Modelling Correlated Events

A correlatable event can be thought of as the tuple <surface-sample, timestamp>. The addition of the timestamp strictly increases the amount of information carried by the surface sample.

Modelling Event Streams

An event stream is simply a list of observed samples sample₀, sample₁, ....

Each additional observation strictly increases the amount of information.

Other Examples:

Accelerometer properties.
Sensor, onreading event.
BatteryManager.onlevelchange : Deprecated but still shipping.
Bluetooth.onadvertisementreceived
BroadcastChannel, all events.
MediaDevices.devicechange event.
GlobalEventHandlers.onfocus and onblur events can fire simultaneously when switching between two browser windows.

Mitigation

Permissions

Goal: Require informed consent from users.

There’s precedent for considering permissions ² to be sufficient mitigation for similar issues. For example, the Media Capture API specification includes the following:

For origins to which permission has been granted, the devicechange event will be emitted across browsing contexts and origins each time a new media device is added or removed; user agents can mitigate the risk of correlation of browsing activity across origins by fuzzing the timing of these events.

From §15 of Media Capture and Streams API specification.

Pros

Prevents drive-by fingerprinting.

Cons

UI for permissions don’t disclose the fact that all sites that have been granted access to the same resource can synchronize identifiers as a side-effect.
Doesn’t prevent identity joining once permission is granted.

Fuzzing Timing of Events

Goal: Deter correlation of events by injecting timing skew.

Mentioned in the snippet above from the Media Capture and Streams API and called out by Jeffrey Yasskin as a potential general mitigation in “desynchronize whole-browser events” in this issue filed against the WHATWG HTML specification.

Pros

Lowers confidence of identity equivalence.

Cons

Precise timing may not be required. I.e. the lowered level of confidence could still be sufficient for most uses.
Doesn’t address unique event streams.

First-Party Restriction for APIs

Goal: Deter identity correlation by third-party sites.

Restrict APIs to the origin of the top-level browsing context.

The latter may choose to explicitly delegate access to the APIs via feature policies. But third-party contexts can’t “reach across” browsing contexts via correlation of cross context events or attributes that may be made available by the API.

Pros

Makes it a requirement that first-parties actively cooperate with other first-parties or third-parties. I.e. Good first-party + bad third-party = safe.

Cons

Difficult to retrofit into existing APIs.
Browser-wide events and attributes are still visible to distinct top-level browsing contexts.
In practice top-level browsing contexts contain a fair amount of third party scripts which may access the same APIs. Furthermore there are financial incentives for first-parties to delegate API access to third-parties.

Limit API Access To Visible Browsing Contexts

Goal: Prevent background browsing contexts from skimming identifiable events.

The Page Visibility API defines the visibility state of a document as visible if the document is “at least partially visible on at least one screen”.

Restrict APIs to — possibly top-level — browsing context’s active document.

Pros

Is a convincing mitigation on mobile devices where only one top-level document can be visible at the same time.

Cons

There could be multiple visible browsing contexts which still leaves the door open for identity correlation across site boundaries.
The fact that having more than one browsing context open at the same time is a privacy risk is quite surprising for users.

Limit Events To Focused Top-Level Browsing Context

Goal: Limit firing correlatable events to a single top-level browsing context.

The HTML spec defines a concept of a currently focused area of a top-level browsing context. As defined, every top level browsing context has one regardless of visibility. A similar narrow concept could be introduced that recognizes the top level browsing context that has system input focus. There should be only one of these on a single device.

Let’s call the top-level browsing context that has system input focus as the focused top-level browsing context.

New specifications could restrict browser-wide events to the focused top-level browsing context.

Pros

Likely fits in well with the intended usage model for most APIs.

Cons

Does not address event stream fingerprinting via polling.

Limit API Access To Focused Top-Level Browsing Context

Goal: Limit access to sensitive APIs to a single top-level browsing context.

Similar to the above, but addresses issues around polling by disallowing access to the entire API or sensitive attributes by restricting the entire API instead of just events.

Pros

Resilient to event stream fingerprinting via polling.

Cons

Can’t be easily retrofitted to existing APIs since it requires defining behavior for “disabled” APIs.
May break critical use cases.

Secure-Context Restriction and Control via Feature-Policy

These should be pretty standard at this point.

Pros

Just makes sense.

Cons

Not sufficient by itself.

Spotting Ephemeral Fingerprinting Surfaces In Web Specs

Ephemeral fingerprints:

Require that multiple browsing contexts observe the same events or access the same volatile attribute.
These browsing contexts could involve a single third party in multiple first party contexts.
Does not require precise clocks nor agreement on the exact timestamps of the observed events. Depending on the fingerprintable surfaces involved the sequence of events could be identifiable by itself. Servers can roughly bucket observations by time periods, eliminating the need for client-side clocks.
Does not require an API to fire an event. A property with a volatile value that can be polled periodically is sufficient.

What to look for:

Events with external triggers. E.g. hardware based events.
Multiple distinct events that are fired in tandem to distinct browsing contexts. E.g. any processing model where multiple events are fired.
Volatile attributes that are visible across browsing contexts.
Volatile attributes that don’t share values across browsing contexts, but change simultaneously. E.g. Salting a volatile attribute is insufficient if its mutations can be correlated.

Example

Consider onfocus and onblur events.

The focus update steps involve firing up to three distinct events: change if the node losing focus is an input element, focus, and blur.

When focus traverses a browsing context boundary, these events may be fired simultaneously to two different browsing contexts. Browsers mitigate this by not firing blur for cross site tab switches, but they still fire blur when the browser itself goes out of focus. Thus identity can be correlated when switching browser windows.

Possible Mitigation

When the new chain and the old chain ³ are in different top-level browsing contexts whose active documents are not same-origin, queue but don’t fire change and blur events until focus returns to the old top-level browsing context.

Notes

Van Goethem, T. and Joosen, W., 2017. One side-channel to bring them all and in the darkness bind them: Associating isolated browsing sessions. In 11th {USENIX} Workshop on Offensive Technologies ({WOOT} 17). (PDF)↩︎
Either via a legacy permissions prompt or explicitly requiring the use of the Permissions API in the spec for sensitive APIs.↩︎
New chain and old chain are defined in focus update steps.↩︎

The Letter අ

Something Happened On The Way To Perfect History

False Precision and Vibe Coding

The Increasingly Inaccurately Named User-Agent

The User-Agent is supposed to be your buddy.

Modern User-Agents are technical marvels.

But they are bound by the same implicit social contract …

That the User-Agent must act ONLY on behalf of the user.

But how do we know how the UA balances conflicting interests?

It matters who owns the User-Agent.

What's your cog ratio?

Optimal line-length for publishing on the web

An Intuition on Shannon Entropy

Shannon entropy is a measure of uncertainty

Choices of measure for a single outcome

Adding independent pieces of information should correspond to adding entropy

More uncertainty should correspond to a larger entropy

No uncertainty should correspond to zero entropy

Putting it all together

Choice of measure for a probability space

Formalities

Incentives

Ode to Glue People

Soft Roles as Organizational Cross Beams

Emergence of Soft Roles

Draining Our Intellectual Catchment Areas

Migrations of Intellectual Capital

Decline

Act III: Flame

Act I: Meaning

Intermission - Q1 of 2025

Preface

Intrinsic and Extrinsic Motivation: An anecdote

Buffer Bloat

From Here To There On The Internet

Faster and Faster

Flow Control

Stuck On The Wire

Buffer Bloat

Fourteen Years

From Illegal Flower Tribute

To Project Nimbus

That’s Where We Are

I Moved This Blog To Eleventy

Growing Pains

The Exodus

Templating

WebC + Eleventy Is Broken

Nunjucks + Eleventy Works

Handrolling

Pandoc Again

Bundling Files

SASSy

Katex

Conclusion

Proof of Work

How to Solve a Problem in Three Steps

Asian Is Not a Useful Category

So what should I ask instead?

The Have-Loop

When Your Feature Gets Someone Hurt

Secret Life of Systems

The Problem (P)

The Model (M)

The Solution (S)

The Implementation (I)

Deployment (D)

Why I Hate Carousels

Layout Runneth Over

Sliiiiiide To The Right

Uh oh. Here come the dots of stupidity…

What’s Better?

In Summary

Inclusion/Equity and Evaluating Negatives

To Prove A Negative

Proof By Enumeration

Proof by Enumerating Product Outcomes

Proof By Construction

Proof by Process of Construction

Amalgam Engineering

Setting up `reSnap`

A couple of notes on the `--sketch` option