Entropy Tolerance: The Most Important Software Question You Aren't Asking

In software development, there's a lot of questions we ask ourselves. Who are my users? What are the requirements for this feature? What's the stack that I should use? How should I deploy? Where should I deploy? And in the age of AI, I think there's a missing question. And so we're going to discuss here in this talk what that question is and how it can help us make better choices when it comes to implementing AI, utilizing AI, in software. So this talk is about entropy tolerance, the most important software question you aren't asking.

So for starters, we need to talk about the difference between deterministic and nondeterministic software. When we say the word determinism, that means that when we give a system a certain set of inputs, we always get the same outputs if we give the same inputs. The entire digital age is built on the trustworthiness of deterministic processes. For example, just a very simple example. Let's think about a calculator. Let's suppose you have a calculator that just adds two numbers. So I say what's one plus one? And I get out a two. That's an expected answer. It's the correct one. And the next time I use the calculator, if I give it those same two inputs, I get the same output. The internal mechanisms of the system are deterministic.

And for that reason, they are consistent, reliable, and trustworthy. Computational devices, the systems that we've used now for decades, are deterministic, at least at the abstraction level that we use them at. And that has allowed the digital age to flourish. And we rely on them. For example, if I was thinking about an autopilot on an airplane, it relies on a deterministic system. The autopilot gets inputs from the sensors on the plane and makes decisions. And we don't even think about it. We board a plane, and we fly to where we need to go, and the pilot may use autopilot for a portion of the time. We don't know. We're not bothered by it. Deterministic systems are testable and reliable and trustworthy. And now we introduce the idea of nondeterminism. So nondeterminism is the property of a computational system where the same input may produce different outputs. And that could be due to some kind of randomness, probabilistic behavior, hidden state. There could be any number of reasons why a system is nondeterministic.

But for our purposes, let's use the same calculator example, but an example that's kind of more useful for thinking about large language models and AI. Let's suppose I have the same idea of a calculator that just adds two numbers, and I can give it inputs. What's one plus one? But the calculator, rather than doing a deterministic process, carries out a look at every possible answer that's been recorded out there. We'll say that it's a corpus of possible answers. Most of the answers are correct. Some of them are not. And the way this calculator functions is that when you give it two inputs, it essentially rolls a pair of dice and chooses from that corpus, from that set of possible answers that have been recorded in its memory, to give us then a final result.

Now, the first time we use the calculator, it looks accurate. But the next time we use the calculator, after enough times, eventually, we may roll a die and get an incorrect answer selected because the internals of the system have no concept of correct or incorrect, just probabilistic behavior based on a large set of possibilities. So we end up with a different output for the same inputs. That's non-deterministic behavior. In this case, due to probabilistic or stochastic behavior, it's inconsistent.

Here's the business question. How would you feel about a very fast calculator that sometimes, unpredictably, might give you the wrong answer? What are the pros and cons that you're weighing with the usage of this calculator? Well, it depends really on what it is you're trying to calculate. It's not about the calculator so much, but the process that you're using the calculator to support. If the process that you're using to support can handle things being incorrect every now and again, then the speed of the calculator might be a worthwhile risk.

But if you cannot tolerate incorrectness, then that speed, the value of that speed, no longer matches the risk. Let's imagine a non-deterministic autopilot on a plane. Instead of having a deterministic system where you get those sensor inputs and always get the same output with particular sensory input, what if we replace that with a large language model that was just vibing away as our autopilot? Now, imagine you're sitting at a gate, ready to board the plane, and the announcement is made that on this plane flight, it will be run by an autopilot, which is actually a large language model, which will receive the sensory inputs and then give the output to the plane's systems to navigate. And there's no human pilots in the seats. Would you board the plane? I'll tell you, I wouldn't. And I bet a lot of you wouldn't either. Because we know that the risk factor is too high for the value that you're getting from a non-deterministic process, a large language model, which is pulling possible values, a prediction machine based on a tremendous, huge corpus of data that it's been trained on. And yet, we are adding everyday non-determinism to business processes without a way to judge or measure the danger.

So, as technical leaders, we need to go beyond hype, beyond marketing of AI, and think critically about what we're adding to the underlying business processes when we add and integrate AI and LLMs into those processes. But we need a way to measure it. We have all these metrics for development, but the problem is we don't have an intrinsic way to measure this risk. So, what I've done is I've actually introduced an idea, created a term, and I invite you to use it if you find it useful in your leadership. To explain it, first we need to talk about the idea of entropy. Now, entropy has a few different meanings depending on the scientific or mathematical route that you're talking about. But in our case, we're talking about what's called Shannon entropy, named after Claude Shannon, which is where the Claude AI is named after. And that entropy has to do with information sciences. And the idea is that the average level of uncertainty of a value across a probability distribution is its entropy. Now, that's a bit mathematical, but we're going to use that as an inspiration to give a somewhat more simplistic idea. That the entropy of something is the amount of random chance or probability in a process. So, that's the definition we're going to go forward with for this purpose.

So, we introduced the fundamental question, the question you should be asking yourself as we integrate AI into software, into business processes. What is a process's entropy tolerance? That means how much random chance can the process you're carrying out handle? Entropy tolerance is not about whether or not you should use AI code generation, not purely that. It's really about the underlying processes that a particular software supports, because software is really supporting some process underneath it always. So, that process, does it tolerate potential inaccuracy? Does it tolerate the influence and effects of compounding probabilities? So, what I'd like to do now is show you a decision framework that you can use with this idea of entropy tolerance. So, this is the entropy tolerance decision framework. You identify a task, a task that software is going to support and that perhaps you're going to integrate AI into. And this could be the development of software itself. So, that could be an AI feature that's being added to a particular process or a particular piece of software. Or, it could be how much AI code generation without human checks you're going to have involved. In the process, whatever it is, you're going to identify the thing that you're doing, the process that you're trying to engage in.

And you're going to ask yourself first, does this process tolerate randomness? And we'll look at a few examples in just a minute. But that's the core question. What is its entropy tolerance? If it's a low tolerance, that means you can't have a lot of probability going on. And if it's a high tolerance, then that's okay. So, we'll define three levels. We have a low tolerance, which means we got to be exact. This is a high risk to the value. If we have issues, if we have inexactness, if we have AI hallucinations, it's a high risk. Medium is somewhere in between. We have some flexibility. We could probably handle some inaccuracy, but not always. And then we have very high tolerance for entropy, for probabilistic, non-deterministic processes. That could be very low risk processes, where inaccuracy is actually okay.

The next phase, no matter what the entropy tolerance then is, is what's the value to the process of using AI in the first place? And that could be low, or that could be high. So, low value or a high value of using AI in this process. So, if we have a low tolerance process, low entropy tolerance, we cannot deal, we cannot tolerate too much inaccuracy, hallucination, randomness, unexpected outputs in this process. Well, if there's not a lot of value to using AI, then just keep it human centered. Have humans do it. It doesn't mean humans don't make mistakes. But as we know, with humans in the process, we have a better chance of catching those mistakes as we go. Because the amount of output of a human is at the rate of the human checking.

On the other hand, high value of using AI for a low tolerance, well, then you may want to include AI. But you're going to have to check the AI's output. You're going to have to have humans in the loop. So, it becomes AI-human collaboration if there's high value with low entropy tolerance. For medium tolerance, well, it's very similar. You might want to keep it human centered if there's any risk at all. Although this may be a choice that you have to make within your own organizations and structures and stakeholders, etc. And, again, human-in-the-loop collaboration, if there's a high value to using AI, if the value of adding AI processes is going to be high, then you might want to go ahead and deal with that tolerance. But human-in-the-loop is going to keep you safe.

And if there's a very high tolerance, well, great. If you really don't get a lot of value from the AI, then you might have some kind of hybrid collaboration. Maybe some folks want to use AI for their particular work. Others don't. If there's not a lot of value to AI and it's a very high tolerance, it really doesn't matter too much. But if there's a very high value of using AI and it has a very high tolerance, a very high entropy tolerance, where inaccuracy is okay, it's not going to be the end of the world, it's not going to be a big risk to you, to the organization, to the business, to whatever it is you're leading. That could be an AI-led process. That could be a vibe-coded process. So by having this decision framework, we can then make choices.

For example, let's suppose that the task that we're going to do is dealing with customer requests for support tickets in some kind of ticketing system that starts with a chatbot. So the question is, does this process tolerate randomness? Well, it might depend on what the system is, what kind of tickets we're getting from customers. If it's a banking system where financial transactions are going on and we get a support ticket because something went wrong with the financial transaction, that might be very different from a support ticket for something far less urgent. Then whatever that is, let's say we decide as an organization, as leaders, it's medium tolerance, we have some flexibility, then the question becomes, what's the value? Well, let's suppose the value is very high, because if we add AI and LLM to sorting the tickets, dealing with the tickets, responding to tickets, highlighting urgency, well, that could let us deal with other things, other problems that are more potentially important and can surface the really important things that we need to focus on.

So that would be, for this, perhaps you would decide it's human-in-the-loop collaboration. We have an AI perhaps be the first level of interface with the customer, but allow the customer to upgrade it to a human, or perhaps have the AI categorized by urgency and then have humans review depending on the level of urgency, for example. So as we can see, by making those choices, by starting with the question, what is the entropy tolerance, that helps define how we deal with AI within the process itself. There are other examples we could think of, of low, medium, and high entropy tolerance processes. For example, what might be some low tolerance processes? Medical diagnosis systems, very low tolerance for inaccuracy, financial transaction processing, very low tolerance for slightly off numbers that aren't deterministic, that aren't consistent.

A security vulnerability auditing of code. Perhaps an AI system initially has some responses to that, some recommendations, but that doesn't mean a human doesn't also need to take a look, human-in-the-loop, on whatever code is being generated. So, that might be low tolerance, and so primarily human-led, with human-AI collaboration and human-in-the-loop. For medium, perhaps that could be like a customer service chatbot, like we mentioned, maybe extracting data from PDFs but just for reporting, not for updating data in a database, and that would depend on the kind of decisions being made from the reporting.

Maybe code review assistance, suggestions as your developers are coding. Well, that could also be some kind of human-AI collaboration with a bit more tolerance for what the AI is doing. Then we have high. High would be software prototyping for user testing. Doesn't matter. You can vibe code a prototype, it's not going into production, you can test with users, and you could generate the entire thing with LLMs. That's fine. Marketing brainstorming and ideation, not going to hurt anything. Draft content creation for blogs, just getting some ideas out the door before they're edited and refined. Those have all very high tolerance, and so we can make a judgment about how to integrate AI in those processes.

So as technical leaders going beyond hype, going beyond any of those ideas that a marketing might produce, but rather being strong technical leaders having pragmatic solutions, then we're thinking about skepticism and realism when it comes to LLMs. We understand what they are. They aren't people that are thinking. They are prediction machines from large corpuses of text. And so because they're these non-deterministic probabilistic systems, we are skeptical of their outputs. We understand where we need verification. Where we need to have judgment of their outputs. And we integrate our AI, we weigh that integration according to entropy tolerance.