Entropy Tolerance: The Most Important Software Question You Aren't Asking

In software development, there's a lot of questions we ask ourselves. Who are my users? What are the requirements for this feature? What's the stack that I should use? How should I deploy? Where should I deploy? And in the age of AI, I think there's a missing question. And so we're going to discuss here in this talk what that question is and how it can help us make better choices when it comes to implementing AI, utilizing AI in software. So this talk is about entropy tolerance, the most important software question you aren't asking.

So for starters, we need to talk about the difference between deterministic and non-deterministic software. When we say the word determinism, that means that when we give a system a certain set of inputs, we always get the same outputs if we give the same inputs. The entire digital age is built on the trustworthiness of deterministic processes. For example, just a very simple example. Let's think about a calculator. Let's suppose you have a calculator that just adds two numbers. So I say, what's one plus one? And I get out a two. That's an expected answer. It's the correct one. And the next time I use the calculator, if I give it those same two inputs, I get the same output. The internal mechanisms of the system are deterministic. And for that reason, they are consistent, reliable and trustworthy.

Computational devices, the systems that we've used now for decades, are deterministic, at least at the abstraction level that we use them at. And that has allowed the digital age to flourish. And we rely on them. For example, if I was thinking about an autopilot on an airplane, it relies on a deterministic system. The autopilot gets inputs from the sensors on the plane and makes decisions. And we don't even think about it. We board a plane and we fly to where we need to go. And the pilot may use autopilot for a portion of the time. We don't know. We're not bothered by it. Deterministic systems are testable and reliable and trustworthy. And now we introduce the idea of non-determinism. So non-determinism is the property of a computational system where the same input may produce different outputs, and that could be due to some kind of randomness, probabilistic behavior, hidden state. There could be any number of reasons why a system is non-deterministic.

But for our purposes, let's use the same calculator example, but an example that's kind of more useful for thinking about large language models and AI. Let's suppose I have the same idea of a calculator that just adds two numbers. And I can give it inputs. What's one plus one? But the calculator, rather than doing a deterministic process, carries out a look at every possible answer that's been recorded out there. We'll say that it's a corpus of possible answers. Most of the answers are correct. Some of them are not. And the way this calculator functions is that when you give it two inputs, it essentially rolls a pair of dice and chooses from that corpus, from that set of possible answers that have been recorded in its memory, to give us then a final result.

Now, the first time we use the calculator, it looks accurate. But the next time we use the calculator, after enough times, eventually we may roll a die and get an incorrect answer selected, because the internals of the system have no concept of correct or incorrect, just probabilistic behavior based on a large set of possibilities. So we end up with a different output for the same inputs. That's non-deterministic behavior. In this case, due to probabilistic or stochastic behavior, it's inconsistent.

Here's the business question. How would you feel about a very fast calculator that sometimes, unpredictably, might give you the wrong answer? What are the pros and cons that you're weighing with the usage of this calculator? Well, it depends, really, on what it is you're trying to calculate. It's not about the calculator so much, but the process that you're using the calculator to support. If the process that you're using to support can handle things being incorrect every now and again, then the speed of the calculator might be a worthwhile risk.

But if you cannot tolerate incorrectness, then that speed, the value of that speed no longer matches the risk. Let's imagine a non-deterministic autopilot on a plane. Instead of having a deterministic system where you get those sensor inputs and always get the same output with particular sensory input, what if we replace that with a large language model that was just vibing away as our autopilot? Now, imagine you're sitting, boarding at a gate, ready to board the plane. And the announcement is made that on this plane flight, it will be run by an autopilot, which is actually a large language model which will receive the sensory inputs and then give the output to the plane's systems to navigate. And there's no human pilots in the seats. Would you board the plane? I'll tell you, I wouldn't. And I bet a lot of you wouldn't either. Because we know that the risk factor is too high for the value that you're getting from a non-deterministic process, a large language model, which is pulling possible values, a prediction machine based on a tremendous, huge corpus of data that it's been trained on. And yet, we are adding every day non-determinism to business processes without a way to judge or measure the danger.

So as technical leaders, we need to go beyond hype, beyond marketing of AI, and think critically about what we're adding to the underlying business processes when we add and integrate AI and LLMs into those processes. But we need a way to measure it. We have all these metrics for development. But the problem is we don't have an intrinsic way to measure this risk.

What I've done is I've actually introduced an idea, created a term, and I invite you to use it if you find it useful in your leadership. To explain it, first, we need to talk about the idea of entropy. Now, entropy has a few different meanings depending on the scientific or mathematical route that you're talking about. But in our case, we're talking about what's called Shannon entropy, named after Claude Shannon, which is where the Claude AI is named after. And that entropy has to do with information sciences.

The idea is that the average level of uncertainty of a value across a probability distribution is its entropy. Now that's a bit mathematical, but we're going to use that as an inspiration to give a somewhat more simplistic idea. But the entropy of something is the amount of random chance or probability in a process. So that's the definition we're going to go forward with for this purpose. So we introduced the fundamental question, the question you should be asking yourself as we integrate AI into software, into business processes. What is a process's entropy tolerance? That means how much random chance can the process you're carrying out handle?

It's really about the underlying processes that a particular software supports because software is really supporting some process underneath it always. So that process, does it tolerate potential inaccuracy? Does it tolerate the influence and effects of compounding probabilities? So what I'd like to do now is show you a decision framework that you can use with this idea of entropy tolerance.

This is the entropy tolerance decision framework. You identify a task, a task that software is going to support, and that perhaps you're going to integrate AI into. And this could be the development of software itself. So that could be an AI feature that's being added to a particular process or a particular piece of software. Or it could be how much AI code generation without human checks you're going to have involved in the process.

Whatever it is, you're going to identify the thing that you're doing, the process that you're trying to engage in. And you're going to ask yourself first, does this process tolerate randomness? And we'll look at a few examples in just a minute. But that's the core question. What is its entropy tolerance? If it's a low tolerance, that means you can't have a lot of probability going on. And if it's a high tolerance, then that's okay.

And if it's a high tolerance, then that's okay. We'll define three levels. We have a low tolerance, which means we got to be exact. This is a high risk to the value. If we have issues, if we have inexactness, if we have AI hallucinations, it's a high risk. Medium is somewhere in between. We have some flexibility. We could probably handle some inaccuracy, but not always.

And then we have very high tolerance for entropy, for probabilistic, non-deterministic processes. That could be very low risk processes where inaccuracy is actually okay. The next phase, no matter what the entropy tolerance then is, is what's the value to the process of using AI in the first place? That could be low, or that could be high. So low value or a high value of using AI in this process.

So if we have a low tolerance process, low entropy tolerance, we cannot deal, we cannot tolerate too much inaccuracy, hallucination, randomness, unexpected outputs in this process. Well, if there's not a lot of value to using AI, then just keep it human-centered. Have humans do it. Doesn't mean humans don't make mistakes. But as we know, with humans in the process, we have a better chance of catching those mistakes as we go because the amount of output of a human is at the rate of the human checking.

On the other hand, high value of using AI for a low tolerance. Well, then you may want to include AI, but you're going to have to check the AI's output. You're going to have to have humans in the loop. So it becomes AI-human collaboration if there's high value with low entropy tolerance. For medium tolerance, well, it's very similar. You might want to keep it human-centered if there's any risk at all, although this may be a choice that you have to make within your own organizations and structures and stakeholders, etc. And again, human in the loop collaboration, if there's a high value to using AI, if the value of adding AI processes is going to be high, then you might want to go ahead and deal with that tolerance. But human in the loop is going to keep you safe.

And if there's a very high tolerance, well, great. If you really don't get a lot of value from the AI, then you might have some kind of hybrid collaboration. Maybe some folks want to use AI for their particular work, others don't. If there's not a lot of value to AI and it's a very high tolerance, it really doesn't matter too much. But if there's a very high value of using AI, and it has a very high tolerance, a very high entropy tolerance, where inaccuracy is OK, it's not going to be the end of the world, it's not going to be a big risk to you, to the organization, to the business, to whatever it is you're leading, that could be an AI-led process. That could be a vibe-coded process.

So by having this decision framework, we can then make choices. For example, let's suppose that the task that we're going to do is dealing with customer requests for support tickets in some kind of ticketing system that starts with a chatbot. So the question is, does this process tolerate randomness? Well, it might depend on what the system is, what kind of tickets we're getting from customers. If it's a banking system where financial transactions are going on and we get a support ticket because something went wrong with the financial transaction, that might be very different from a support ticket for something far less urgent. Then, whatever that is, let's say we decide as an organization, as leaders, it's a medium tolerance, we have some flexibility, then the question becomes, what's the value? Well, let's suppose the value is very high...

So that would be, for this, perhaps you would decide it's human-in-the-loop collaboration. We have an AI perhaps be the first level of interface with the customer, but allow the customer to upgrade it to a human, or perhaps have the AI categorized by urgency, and then have humans review depending on the level of urgency, for example.

As we can see, by making those choices, by starting with the question, what is the entropy tolerance that helps define how we deal with AI within the process itself? There are other examples we could think of, of low, medium, and high entropy tolerance processes. For example, what might be some low tolerance processes? Medical diagnosis systems, very low tolerance for inaccuracy. Financial transaction processing, very low tolerance for slightly off numbers that aren't deterministic, that aren't consistent.

Now, security vulnerability auditing of code. Perhaps an AI system initially has some responses to that, some recommendations, but that doesn't mean a human doesn't also need to take a look, human-in-the-loop on whatever code's being generated. So, that might be low tolerance, and so primarily human-led with human-AI collaboration and human-in-the-loop. For medium, perhaps that could be like a customer service chatbot, like we mentioned. Maybe extracting data from PDFs, but just for reporting, not for updating data in a database.