Inclusion/Equity and Evaluating Negatives
Privacy, security, inclusion, equity, accessibility, … all of these have an interesting property in common. When evaluation products against such questions you are usually trying to establish a negative.
To Prove A Negative
Proving a negative requires providing evidence of absence.
If one wanted to disprove the claim “all primes are odd” one only needs to prove “some primes are even”. The latter can be accomplished by finding a single counterexample – the number 2. On the other hand if one wanted to prove the claim, they would need establish that all primes are odd. Another example is the claim “there are no vermin in this house,” proving which requires an exahustive search of the premises. Once again, disproving the claim only requires a single piece of evidence – a mouse.
Going back to the beginning, asking someone whether a product is secure, private, inclusive, accessible etc… fall into the same category. Answering these questions require proving that there is nothing insecure, privacy violating, non-inclusive, or inaccessible about this product. These questions are even worse than the one about vermin from before because it’s not even clear where to start looking. People who have spent their entire careers studying the subject would find the criteria difficult or downright impossible to evaluate.
Despite this apparent futility, there is hope. Knowing that the criteria holds under a limited set of circumstances is sometimes enough. Often it is the best that we can do.
Proof By Enumeration
In security and privacy parlance this limited scope is called the threat model.
Absolute security is impossible, so one would start by establishing the specific threats that they would like to deal with. For example when considering the security of a networked application one could start by drawing out a data flow diagram and then going over the different points at which data would be at risk of being compromised. Enumeration of vulnerabilities and threats in this way opens up the possibility of thoroughly examining the attack surface. Thus the previously intractable problem becomes finite.
As our understanding of threats improve, so does the effectiveness of the threat model. Historically, understanding this kind of problem space takes time and many – often costly – mistakes. The boundaries sometimes only become known when someone exploits it.
Similarly, when evaluating a squishy concept like inclusion and equity one would also start with a well defined scope that lends itself to examination. Just like before, often we only learn where to look for problems the hard way. So we come up with methodologies like carefully curated dimensions of identity. Enumerations like that give us a finite set of things to consider with our finite resources. Additionally these methodologies also grant us a taxonomy with which we can organize our hard earned knowledge on the topic.
Proof by Enumerating Product Outcomes
Enumeration based approaches establish clusters of possible product outcomes which we can then evaluate individually.
As an example, a product that processes pictures of people can be evaluated based on the outcome of handling pictures of people with different skin tokes, gender expressions, body types, ages, hair and accesories, visible disabilities, and lighting conditions. Coming up with these outcomes can be laborious and expensive, but it is tractable. Costs of curating suitably detailed test cases and data sets should amortize across their potentially broad applicability.
Proof By Construction
But sometimes the set of outcomes do not cluster effectively or are too numerous to exhaustively evaluate. In security this would be similar to asking whether some software system is free from buffer overruns – a question that is theoretically impossible to answer in general if the method of construction is not amenable to analysis. The best we could do is for suitably skilled persons to go through the code with a fine toothed comb looking for potential problems; a task that is both impractical as it is prone to error.
Proof by Process of Construction
Instead we could try using programming languages that provide certain safety guarantees: memory-safe languages. These safety guarantees establish the memory safety of the resulting behavior as a consequence of the method by which the underlying software is constructed. Another method we can use is to continue using an unsafe language1, but invest in development practices like fuzz testing which give us sufficient statistical control over the behavior of our software.
Inclusion and equity can follow a similar paradigm where we establish certain process requirements that also yield a favorable outcome with a high likelihood. Google’s Building for Everyone outlines some of these processes criteria such as involving a diverse group of people during the ideation and design phases of a product, and involving a diverse group of people during testing. Once again, these approaches don’t guarantee success, but have a much higher likelihood of delivering a favorable outcome.
Amalgam Engineering
Increasingly software systems are built as amalgamations of systems that are themselves complex. The behavior of each constituent is difficult to reason about let alone the behavior of the amalgamation.
To establish some desirable property of the amalgamation, it becomes necessary to progressively establish the same or minor properties of the constituents. The point of establishing these properties is to yield a simpler more tractable model of a system that we can readily reason about. We can then infer or at least approximate the behavior of the amalgamation based on the simplified models that we derive of the contituents.
We don’t expect this situation to get better given that many of these contituent parts will be ML models that are too complex for us to understand. As enumerating product outcomes becomes less effective, we will need to rely ever more on process criteria and governance of construction to curb the product outcomes.
The accuracy of the resulting simplified models is all that we will have to grasp in order to have some say how technology will govern us in turn.