Daniel Beskin's Blog

I Am Not a Functional Programmer

Daniel Beskin — Wed, 28 Jan 2026 00:00:00 GMT

Despite rumors to the contrary, I am not actually a functional programmer. True, I sometimes slip and fall and an "applicative functor" would come out from under my breath. But surely you shouldn't judge me for this minor tick. So no, I'm not a functional programmer, I'm just trying to be reasonable¹.

We all want higher code quality, better testability, and less maintenance burden. Don't we? It just so happens that techniques from FP are really good at achieving these goals. We don't even have to mention functional programming to anyone, we can just write better code and call it a day².

Spooky Action at a Distance

Did it ever happen to you that you needed to implement some minor fix in one place in your codebase, and then suddenly code in another seemingly unrelated place started breaking? If you're lucky the breakage is something loud and clear like an exception. If you're less fortunate it will be some kind of silent and subtle logic bug that takes time and effort to pinpoint.

When this happens we can say that our code suffers from "spooky action at a distance"³. I think that these days it's universally accepted that spooky action at a distance is bad for everyone involved. Who wants to live in a world where small changes have unexpected consequences all over the place?

If we accept the premise that minimizing spooky action at a distance is a net positive for the maintainability of our code, the next question becomes, what are the sources of this malaise?

One prominent source of spooky action is shared mutable data (both variables and data structures), especially high-visibility or even global data. When you mutate data what do you know about all the other potential users of that data? Are you willing to check them one by one to make sure you didn't break any invariants? Can you even tell who they are? Can you?

If you're like most programmers you probably can't, or won't. You make the changes, cross your fingers, and pray for the best. But if we recognize all the pain and suffering that shared mutation brings into our lives (and I didn't even mention concurrency) a reasonable conclusion would be to avoid it like the plague. Mark everything as const/final, favor immutable data structures, and breathe a sigh of relief.

You'll be surprised how good it feels. No need to defensively copy anything, no need to fear the consequences of mutating this or that input (you can't). Debugging becomes easier. Sure, you'll have to learn some new ways of solving problems, but spooky action at a distance is scientifically guaranteed to drop by at least 50%.

Mocking the World

Most programs do more than mutating variables. They do a whole lot of things, like talking to databases, or sending HTTP requests, and many other interactions with the big bad world. And that's perfectly normal. We're all conscientious developers though, and before we ship anything we must verify that it actually works as stated. Some of us even write tests to try and prove that, or so I hear.

But my god, writing tests has become so tedious...

Even for the smallest test that validates some simple business rule you have to set up 15 different mocks/spies/doubles/clones/whatever. One for the DbFactoryProxyFactory, another for the HttpCoordinationOrchestratorManipulator, and the list goes on and on. Usually these mocks tend to be annoyingly mutable and prone to race conditions.

Sure, these days you can delegate some/most of this tedium to your handy AI, it won't complain (yet⁴). Unfortunately, despite a common misconception, tests are not "write only", they are also a form of executable system documentation. Even if you didn't have to write all that noise yourself, you might need to read it one day and the signal to noise ratio makes it a huge waste of time just trying to locate the code that actually matters.

If we take a closer look at the sort of code that forces us to write these unpleasant tests, we are likely to find that it entangles⁵ too many things together. Or at least two of them: business logic and interactions with the external world.

Unless you're doing some very complex external interactions it's quite likely that you should focus most of your testing effort on the business logic. Presumably that's also where most of the value for the business lies as well. Too bad it's so difficult.

Since most of the complex mocking described above comes from the bits of code that interact with the external world, this should give us plenty of motivation to disentangle the business logic from the interaction code. Business logic tends to be simple in-memory computation (at least conceptually), testing it in isolation should be much simpler. Set up some inputs, assert some outputs, done. Written this way, the signal to noise ratio likely to go way up.

The actual process of disentangling code might not be easy. It might even require some deep design changes in how you structure your code. But if you're serious about testing it is well worth the effort. And as the people from the cult of TDD will tell you, code that is easy to test tends to be of all around higher quality. You get this for free just by focusing on less tedious tests.

Feeling a Bit DRY

I don't like repeating myself⁶, both because I'm lazy and because it just itches me the wrong way to see the same code repeated again, and again, and again... I suspect that many programmers share this tendency naturally. And even if not, we can probably agree that more code equals more liability (tests, maintenance, bugs) and so when it doesn't obstruct readability, reducing code volume⁷ is probably for the best.

Suppose that you're coding yet another shopping cart backend for whatever. You now create the cart checkout process. This would've been straightforward but there are quite a few failure modes⁸, and your team is obsessed with logging and error enrichment. So you end up with this code:

function processCustomerCheckout(order) {
  let r1
  console.log('Validating inventory...')
  try {
    r1 = validateInventory(order) // 1
  } catch (e) {
    console.error('Validating inventory failed:', e.message)
    throw Error(`Validating inventory: ${e.message}`)
  }

  let r2
  console.log('Calculating shipping...')
  try {
    r2 = calculateShipping(r1) // 2
  } catch (e) {
    console.error('Calculating shipping failed:', e.message)
    throw Error(`Calculating shipping: ${e.message}`)
  }

  let r3
  console.log('Processing payment...')
  try {
    r3 = processPayment(r2) // 3
  } catch (e) {
    console.error('Processing payment failed:', e.message)
    throw Error(`Processing payment: ${e.message}`)
  }

  return confirmOrder(r3) // 4
}

There are four steps to the checkout process: validating the inventory (1), calculating shipping costs (2), processing the payment (3), and confirming the order (4). Unfortunately all the error handling and logging that's interspersed between the steps makes it very difficult to actually see what's actually going on.

But it's the repetitiveness that really gets me, I have to DRY it out to oblivion.

If we look at what's fixed vs. changing we see that in each step we process the order and produce the output for the next step, and every step is wrapped with almost identical logging and error handling.

One common way to reduce simple repetition like this is to use an anonymous function to extract the part that changes in the different steps⁹. So we move the logging and error handling into a helper function like so:

// 1
function withLogging(stepFn, stepName, order) {
  console.log(`${stepName}...`)
  try {
    return stepFn(order) // 2
  } catch (e) { // 3
    console.error(`${stepName} failed:`, e.message)
    throw Error(`${stepName}: ${e.message}`) // 4
  }
}

This new withLogging helper function (1) takes a function argument for the step (stepFn), the name for the current step, and the original order input. It then applies the logging and error handling logic from before: log and call the stepFn function on the order argument (2). If there is an exception (3), log, "enrich", and rethrow it (4).

We can now use withLogging to wrap around our original steps and see the repetition melt away:

function processCustomerCheckout(order) {
  const r1 = withLogging(
    validateInventory, 'Validating inventory', order)
  const r2 = withLogging(
    calculateShipping, 'Calculating shipping', r1)
  const r3 = withLogging(
    processPayment, 'Processing payment', r2)

  return confirmOrder(r3)
}

Much better, it's now easier to spot the building blocks of the flow. But it's still kind of annoying. With the plumbing still sticking out and obfuscating the main logic. Do we actually care about the logging strings? Or the fact that logging is present here?

These are all good questions, but for me this is still just repetitive. So we'll use a bigger hammer to take it out. Let's hide away the logging completely by wrapping away the original functions:

function withLogging(stepFn, stepName) { // 1
  return (order) => { // 2
    console.log(`${stepName}...`)
    try {
      return stepFn(order)
    } catch (e) {
      console.error(`${stepName} failed:`, e.message)
      throw Error(`${stepName}: ${e.message}`)
    }
  }
}

This withLogging variant is very similar to the first one, except that it's no longer taking the order as a top-level argument (1). Instead it produces a new function (2) that expects the order argument and wraps around the invocation of stepFn¹⁰.

We use the new withLogging function by pre-applying it to our building blocks:

const checkInventory =
  withLogging(validateInventory, 'Validating inventory')
const withShipping =
  withLogging(calculateShipping, 'Calculating shipping')
const withPayment =
  withLogging(processPayment, 'Processing payment')

With the wrapped building blocks we can rewrite the flow as follows:

function processCustomerCheckout(order) {
  const r1 = checkInventory(order)
  const r2 = withShipping(r1)
  const r3 = withPayment(r2)

  return confirmOrder(r3)
}

Without all the logging noise, this is almost what we want, but for the last bit of obfuscation.

Do we enjoy passing around the r1, r2, r3 variables? It's not just annoying, but also error prone, what if we add another step and forget to update the numbering appropriately¹¹?

What we really want is to just say "here are the steps, pipe them together", this should eliminate the last bit of repetition:

function pipe(...fns) = initialValue =>
  fns.reduce((value, fn) => fn(value), initialValue)

We are going a bit meta, but given a list of functions we create a new function that applies them one after the other starting with some initialValue. Scary though it might be, here's the result:

const processCustomerCheckout = pipe(
  checkInventory,
  withShipping,
  withPayment,
  confirmOrder
)

Look at this beauty. No repetition, only business logic. Each building block is now a nice, testable, and reusable component.

Speaking of reusable, it is now very easy to mix and match the components as we deem appropriate:

// 1
const processStorePickup = pipe(
  checkInventory,
  withPayment,
  confirmOrder)

// 2
const generateQuote = pipe(
  checkInventory,
  createQuote)

We now support two new flows. Store pickup (1) no longer requires shipping, so we omit that step, and create another pipeline. Another flow just generates quotes (2), which requires no payments or shipping, but we add a new building block for quote generation. This is just as seamless as before, yet another clean and testable pipeline.

Chasing DRY sure got us somewhere.

I'm Just Trying to Be Reasonable

I don't think that many people would find what I wrote so far particularly controversial. It's the sort of stuff that might come up in a code or design review.

Who would object to improving the maintainability of their code? Making code more testable? Great. Scratching the DRY itch? That's how I pass most of my days.

All I'm trying to do is be reasonable, and improve the code I write. The only difference between what I'm doing and a random bunch of "best practices" for code improvement is the source of these practices. Without mentioning it, every single one of them is informed by functional programming.

The fear of spooky action at a distance led us to immutability, the default in functional languages. The search for testability led us to the segregation of side-effecting code from pure code (business logic). Take this a step further and we get to "functional core, imperative shell". Lastly, the need for DRY sneakily got us into using higher-order functions (withLogging v1), partial function application (withLogging v2), and a functional combinator (pipe). All hallmarks of the functional style.

And this is just scratching the surface. Dig a bit deeper and you'll find more tools that take advantage of immutability, more techniques to control side-effects, more ways to combine functions in useful ways. And much, much more.

Applied judiciously, techniques from functional programming will lead you towards code that is easier to reason about locally¹², easier to reuse, easier to refactor, easier to test, and generally easier to maintain. All relevant concerns for the plain old programmer, no academic gymnastics involved.

Yeah, but.

The Monad Elephant in the Room

It's much more common to hear about functional programming alongside intimidating academese like monads, functors, category theory, and oh my... It might seem that the typical functional programmer is stuck in an ivory tower onanistically trying to force mathematical abstractions down the CI pipeline.

I won't argue that this isn't a real phenomenon. And I can't say I'm completely innocent of doing just that myself. But it's not entirely our fault.

Once you start with the basics of untangling the spaghetti of rampant side-effects, the clarity you obtain reveals the deeper repetitive structures of your code¹³. And it becomes yet another thing you want to DRY. And you know who has a hundreds of years old track record of systematically discovering and categorizing repetitive structures? Mathematicians, that's who.

So it's not that surprising that some of the structures that appear in programming overlap with structures that mathematicians already discovered. Once you see it, it's hard to unsee. And that's how you rediscover monads¹⁴, and then the rest of the **** owl¹⁵. Nobody's forcing us down that path, but these abstractions are so powerful and reusable that it's a slippery slope. One that's both fun and practically useful.

So no, I'm not a functional programmer, I just happen to wield a very powerful toolbox to make code better. And you?

Pun intended. ↩
Although I could argue that naming things is better than not. There's a reason why classic OOP design patterns are so popular. A name is a powerful thing. ↩
Originally from discussions about physics. ↩
But will likely produce exceedingly ugly code that you will most definitely ignore, "it just tests, who cares...". ↩
Continuing the quantum physics-inspired theme... ↩
Despite my whole blog possibly being an indication of the opposite. ↩
Not in the "code golf" sense, reducing the absolute number of characters is not what I'm aiming at. ↩
In real life you'll probably need to handle rollbacks and the like to make this thing actually transactional, but that goes way beyond the scope of this post. ↩
Compare with the classic Template Method design pattern. ↩
Compare with the classic Decorator design pattern. ↩
Yeah, yeah, this is JavaScript, you can just reuse the same variable multiple times. But given what I wrote before I err on the side of using const where possible. And even if not, this is just an example, so chill... ↩
Although functional programming is not the only way to attain local reasoning. See for example how the imperative Rust can take you in the direction of local reasoning. ↩
The eagle-eyed viewers amongst you might've noticed that we got very close to rediscovering the Try/Result type from functional programming, along with the monadic operations it supports. ↩
Although "monad" is not really an FP term, or even a mathematical term. It seems to have come to us from philosophy. And it appears that Microsoft had them all along. ↩
In case this is somehow unfamiliar, I'm referring to this wonderful meme. ↩

The Compiler Is Your Best Friend, Stop Lying to It

Daniel Beskin — Mon, 22 Dec 2025 00:00:00 GMT

Note: this is a "script" for a podcast that I recently recorded, as such it's more conversational and less technical than the usual content of this blog. Not a single code block in sight...

Prologue

Imagine you wake up one night to find out that production is crashing. It takes you a while to unearth the root cause. It turns out to be a null pointer exception deep inside a service. It crashed the whole system. How did it get there? Where did it come from?

Now imagine a different story. One day a developer had to struggle for a whole of 20 minutes fixing some informative compilation errors. And that's it, that's the whole story. Nobody had to wake up at night, production never crashed.

One of these stories is a story of lies and deceit, the other a story of dialogue and cooperation. Today we will learn how to stop lying to the compiler, and how to get a good night's sleep.

Part I: The Compiler

(If you're familiar at high-level with how compilers work, feel free to skip to the next part)

But first things first. What is a compiler?

In the broadest sense, a compiler is a function, that takes an input in one format and produces an output in another format. This definition is so broad as to be useless... Although, if you take this definition seriously, it has some interesting implications on how we approach solving problems in everyday code. But I digress.

Let's try again, more concretely this time. A typical compiler takes source code in some programming language and produces an output in some other language. For example, a C compiler takes files written in C, and produces assembly code.

A typical compiler pipeline might have the following steps:

Parsing the source code into an intermediate representation, an abstract syntax tree (AST)
Typechecking the code, make sure that all the types line up
Optimization, going over the AST and finding opportunities to make the code more efficient
Code generation, converting the AST into the output format, like machine code.

As we will soon see, the typechecking step is probably the most relevant for most developers.

Like most things, nothing is that simple. Different languages have differently flavored compilers, doing a wide range of things. This is not a thorough review on all things compilers, so instead, let's take a sample of languages and see what kind of compilers they have.

Rust

Rust is similar to C, in that it compiles directly to machine code. To distinguish this "classic" compiler from the rest, we call this style an "ahead of time" compiler. This means that everything that the compiler does starts and ends before runtime. Once we created the binary output, the work of the compiler is done.

But, this doesn't mean that the job of the compiler is easy. Rust is quite sophisticated, apart from parsing the source code the compiler needs to do things like macro expansion, type checking, borrow checking, and lots, and lots of optimizations.

Optimization, and generally performance is a common benefit of using compiled languages. Compiled languages tend to be faster than languages that don't use a classic compiler, but are instead dynamically interpreted on the fly (like Python). And Rust is famous for its optimizations in the style of "zero cost abstractions".

Zero cost abstraction means that despite Rust being a low-level language, in many cases one can write pretty high-level code and still get the performance of handwritten low-level code. This is possible because the compiler analyzes the code, and manages to translate a high-level construct like map on an iterator into a low-level, fast loop. This includes optimizations like inlining and loop fusion.

As a developer writing in Rust though, the thing that you would notice the most during development is all the memory safety checks the compiler does for you at every step.

Rust makes it possible to safely manage memory without using a garbage collector, probably one of the biggest pain points of using low-level languages like C and C++. It boils down to the fact that many of the common memory issues that we can experience, things like dangling pointers, double freeing memory, and data races, all stem from the same thing: uncontrolled sharing of mutable state. For example, if two different variables share a pointer to the same data, if one of them frees the pointer, the other will end up with an invalid pointer without knowing it. Likely breaking something upon pointer dereference.

So Rust forbids us from freely sharing pointers. Each pointer has exactly one owner at every moment of time. Once the owner goes out of scope, memory is freed automatically. Since there are no other owners, this is safe and cannot lead to dangling pointers and the like.

What makes Rust special is that all of this happens at compile-time. The compiler tracks pointer ownership, using something called a borrow checker, forbidding us from doing anything that violates the single ownership of pointers. Memory unsafe code will simply fail to compile.

Rust's powerful type system is one very compelling (and painful) reason to use it. A broken elevator and walking 21 floors up the stairs can really motivate you to invent such a safe language.

Then we have Java.

Java

People often think of Java as a static, compiled language. But actually, the Java compiler doesn't produce machine code. Instead, it produces bytecode. An intermediate (stack-based) language that is then dynamically interpreted by the Java Virtual Machine (JVM) when the bytecode is executed.

The reason for going through all this trouble is portability. By using intermediate bytecode, Java can be compiled once on any platform, but then executed on any other platform that has an implementation of the JVM. That's cool and all, but didn't we just say that interpreted languages are slow?

True, in the early days of Java, people were worried that running such a "dynamic" language will be too slow to be useful, and surely will never be able to compete with something like C++. And that's why we have "just in time" compilation. Or JIT for short.

The Java compiler does a lot less than Rust's. The translation from Java to bytecode is comparatively simple. The compiler doesn't make optimizations as far reaching as the Rust compiler. And if we left things that way Java would indeed be quite slow.

The real fun begins at runtime. The Java runtime has a JIT compiler enabled. The JIT compiler monitors the execution of the bytecode, as it gets slowly interpreted step by step. After a while, the JIT starts recognizing various hotspots in the code, like long loops, or common branching points. It then takes the bytecode for the hotspot and compiles it down to an efficient machine code representation. From that point onwards, that segment of code should run much more quickly.

Like a sophisticated ahead of time (AOT) compiler, the JIT compiler can apply nontrivial optimizations to the code. But unlike an AOT compiler, the JIT compiler has access to the actual usage patterns at runtime. Meaning that it can be more aggressive in optimizing the real bottlenecks in code. Things like dynamic dispatch in an object-oriented class hierarchy can be converted to static dispatch at runtime.

If you ever wondered why your JVM app takes time to "warm up" before it's actually fast, the JIT compiler is your answer. It takes time for the JIT compiler to recognize and compile the actual hotspots in code.

These days, Java has an AOT compiler as well, called Graal. As a result we can use the very same Java code and produce two completely different artifacts, either in bytecode or machine code (with some limitations).

More generally compilers can have more than one compilation target, or "backends". For example, both Scala and Kotlin can be compiled to either JVM bytecode, JavaScript, and native machine code. To support such use cases, and to avoid repetition between different backends, the compiler will have some intermediate representation of the language that is being compiled, the AST. This way, most of the compiler's work like parsing, typechecking, and various optimizations can produce one AST, but then that one AST can be converted into different outputs.

Despite all that complex work being done by the various compilers, most of the interaction that Java programmers have with the compiler is with the type checker. Java's type-system is not nearly as strict as Rust's, often being more annoying than helpful. So it's not uncommon for us the developers to "know better" than the compiler and use a cast every now and then. These are the first signs of trouble...

Seeing that a compiler can be quite the complex beast, you would be quite correct in asking:

Aside: Who Compiles the Compilers?

Or more specifically, in what language is the compiler written? It would seem natural for the developers of a new language X to write the compiler in X. Surely it's their favorite language. At the very least, it's a good idea for them to dogfood their own creation.

Among people who write compilers, writing the compiler in the very same language it's supposed to compile is considered an important milestone. It shows that the language is mature enough to be able to implement such a complex piece of code. When a language does it, it is said to be "self-hosting".

Here we have a paradox. If I have a compiler written in X, to be able to use it I first need to compile it, and for that I need a compiler for language X. But I don't have one compiled, I need to compile it first... And so we find ourselves with a chicken and egg problem.

The way to get out of this conundrum is by a procedure called "bootstrapping". Here things are going to get a bit meta.

It goes like this. First write a compiler in language X, the "bootstrapped compiler". Ideally you do that using as few features of X as possible. Then choose another existing language Y (can be assembly or some high-level language), and write a compiler for X in Y, the "bootstrapping compiler". It doesn't have to handle all of the features of X, only the ones that were used in the code for the bootstrapped compiler. Once you run the bootstrapping compiler on the code for the bootstrapped compiler, congratulations, you now have a brand new self-hosting compiler for X. From this point onwards, you can improve the compiler and add features to it all without leaving the comfort of your favorite language X.

We can follow this process with the Rust compiler. Originally, it was written in the OCaml language. But the current Rust compiler was rewritten in Rust, which makes it a proper self-hosting compiler.

This is all good and well, but most of the time us simple developers don't really care about the language the compiler uses, we just need it to compile code. The reason I'm telling you all this is that it gives us an interesting insight into the development of programming languages.

The people who write compilers are usually the same people that design the language itself. And most of what they do all day long is writing compiler code. This might skew their incentives. They really enjoy using languages that are a good fit for writing compilers. Even more so, since in the early stages of a new language the compiler is probably the biggest piece of code written in it. That's the chance for the new language to really shine.

We shouldn't be surprised then, to find that many new languages tend to accumulate features that are very convenient for compiler writers. Things like pattern matching, and implicit parameters.

But seeing how compilers are not very representative of the code most of us write daily, this might not be the best way to spend the complexity budget when designing a new language. Lucky for us, some of those features are useful for plain-old, boring business code as well. Especially code written in the functional style.

Let's wrap up this exploration of compilers with one more language.

TypeScript

The TypeScript compiler is special in that the target is not a low-level format like assembly and bytecode, but instead another high-level language, JavaScript. Although definitions are a bit fluid, but in this case we say that the TypeScript compiler is actually a "transpiler". Even more curiously, since JavaScript is a strict subset of TypeScript, we can say that TypeScript is compiling into itself.

There are two main reasons to want to do that: nicer syntax for various features of JavaScript (like classes and enums), and the type-system that TypeScript adds on top of JavaScript.

Microsoft, who developed TypeScript, was motivated by the pain they experienced when scaling large JavaScript codebases. Adding a robust type-system to a language makes it much easier to tame large codebases over time (things like refactoring become safer). We can see the same trend with Dropbox investing in Mypy, an optional type checker for Python, to help scaling their Python codebase.

Adding a type-system to an existing programming language is very different from designing a language with types from the ground up. With an existing language you must be able to add types to as much existing code as possible. And codebases in dynamic language sure do tend to be... Dynamic.

To mitigate this, TypeScript uses gradual typing. With gradual typing you don't have to add types to the whole codebase all at once, like you would in, say, Java. Instead, you can do it gradually, and the compiler verifies that whatever is annotated is consistent with itself, and ignores the rest. This smoothes migration from an untyped code and mindset to a typed one.

There is also the question of which type system is most appropriate for an ex-dynamic language. Did you know that type-systems come in different flavors? Most mainstream languages use "nominal" type-systems. This means that each type gets a unique name, and types that have different names are treated as distinct types by the compiler. In contrast, TypeScript uses a "structural" type-system. Two types are considered the same if they have the same structure, e.g., the same fields and methods. The name of the type doesn't matter.

Structural typing seems to be better suited when gradually typing a dynamic language. "If it walks like a duck and quacks like a duck, then it must be a duck" is a common viewpoint in dynamic languages, and structural typing is the way to capture this mindset with types.

In a curious turn of events, TypeScript and Mypy went the other way around with bootstrapping and self-hosting. The current TypeScript compiler is self-hosting, but a rewrite in Go is in the works. In Mypy, the compiler was rewritten from plain Python into a special almost-Python dialect that compiles directly to C. The motivation for both rewrites is performance. Turns out that it's difficult to get a sufficiently fast compiler that can quickly process large codebases in dynamic runtimes like JavaScript and Python.

Like before, we can't help but notice that the type-system is the main interface between the developer and the compiler. In the case of TypeScript I would say that it's also the main reason to even want to use the language.

And yet, so many developers hate the compiler. It seems that all it does is yell at us about trivialities like "string is not int" and such. Is it worth bothering with a compiled language just for that?

Part II: Lying to the Compiler

The compiler is always angry. It's always yelling at us for no good reason. It's only happy when we surrender to it and do what it tells us to do. Why do we agree to such an abusive relationship?

And for someone who yells that much, it's not even that smart. Remember all the times when the compiler yelled at you something about a type mismatch, but turns out that you were right and the compiler was wrong? In the end you had to use a cast (or mark something as any) just to shut the compiler up.

We can solve these issues by quitting the relationship, and jumping ship to a dynamic language. I suspect that compiler abuse is why in the past Python grew in popularity compared to Java (that, and Java's verbosity). But we just saw that even in historically dynamic languages, like JavaScript and Python, there's a trend towards a more static, more compiler-centric way of writing code. These days Python has not just one, but multiple competing tools for typechecking (mypy, pyright, pyrefly, ty). Why do people lean that way?

Maybe it's performance. We talked before about how compiled languages tend to be faster than dynamic languages. But that doesn't seem quite right. In many contexts, where the language's performance is not the bottleneck, people will still use types. And TypeScript, for example, doesn't give us a runtime performance benefit, all the type information is discarded after compilation. Are people really just that masochistic?

Maybe Rust is the answer. Rust's compiler brings significant improvements in code safety where memory management is involved. True, its compiler is even more difficult to satisfy than average, but it solves a very tangible problem that would be very difficult to safely solve otherwise. Still, most of us don't write low-level code, and we mostly use garbage-collected, memory-safe languages. Not many mainstream languages have a type-system that is as powerful as Rust's, so we don't usually get that much extra safety.

We can imagine that in very large codebases, like the ones managed by Microsoft and Dropbox, there's some magical advantage to having types checked by the compiler (we mentioned safer refactoring before), despite the obvious pains. But most of us don't maintain code that large, why bother with types then?

It seems that all the compiler can do for us is act as simple, and not very reliable, guard rails. Sure it would sometimes notice that this or that int is actually a string. But if that's all it can do for us, then we might as well write a couple of unit tests, and remove this nuisance from our lives.

The Lies

I think that this is a false premise. The compiler is actually much more useful and powerful than it might appear. So useful in fact, that relying on it is beneficial in any codebase, large and small. The problem is that we are used to lie to the compiler all the time, in return all it can do for us is yell about strings not being ints and do little else.

What do I mean by lying to the compiler?

The compiler, or rather the type-system, is only aware of things that are known at compile-time. A "lie" would be to write code that communicates one thing at compile-time while doing something else at runtime. On the other hand, the more facts that we can truthfully state to the compiler at compile-time, the more useful the compiler can be. But first, let's see some concrete examples of lies.

Null

When we say that some variable x is a string. What do we mean by that? It means that the runtime values that x can take are things "a", "ab", "abc", and many other reasonable strings. This also means that we can apply various methods to x like substring or endsWith. Unfortunately, in most modern languages this also means that at runtime it can be null.

Unlike the other non-null values, you cannot call substring or any other method on null. Calling a method on null will trigger an NullPointerException, possibly far away in code and time from where that null was created. We could adopt a policy of "defensive programming" checking for null every step of the way. But in practice nobody does that. Instead, in many places in our code we implicitly assume that values are non-null unless stated otherwise (in a comment that nobody actually reads).

This is one of the most common lies we tell the compiler. We state that something is a valid string, but at runtime it can just as well be null. In most mainstream languages the compiler cannot distinguish between nullable and non-nullable values. And so it cannot help us out when our nullability assumptions turn out to be wrong. That's when things crash.

How many times did you find yourself setting something to null because you couldn't come up with something better, only to discover that some other place starts crashing?

Exceptions

Speaking of crashing, another source of lies we tell the compiler are exceptions. More specifically, unchecked exceptions.

If we write that some method returns string, what do we mean by this? Ideally, we mean that every time we call the method a string value will be returned. But in reality, there's another, sneaky way of returning from a method: we can throw an exception.

Similar to null, unchecked exceptions might be discovered away from the origin (although not as bad as null, they have to walk up the stack, and are not stored as data). And similar to null the compiler won't track them for us. If we don't go into a full defensive mode, we're assuming that the code is exception-free, and assumptions tend to break over time. But we lied to the compiler, so it won't be there to help when they finally break.

How many times did you leave a comment on some branch of code stating "this CANNOT happen" and thrown an exception? Did you ever find yourself surprised when eventually it did happen? I know I did, since then I at least add some logs even if I think I'm sure that it really cannot happen.

Casts

So far things happened by accident, due to implicit assumptions being broken. But sometimes we are smarter than the compiler. Sometimes we know better and we can force the compiler to obey. We can use type casting.

Suppose that there's some interface, let's call it Animal. And it has two different implementations: Cat and Dog. Sorry for the cliche example, it's difficult to be original without written code.

Now we are working on some code that got an Animal argument, call it x. In this particular flow we know for certain that x must be a Dog. We can follow the code ourselves and show that x is unquestionably a Dog. So we want to use Dog-specific methods. But the silly compiler won't let us. It only knows about Animal methods.

The compiler won't budge, and we won't budge. So we take out our hammer, and use a cast. Now x is a Dog, and the compiler doesn't argue anymore. All is well, until some new feature request that requires supporting Cats in our flow. When we make the change, we forget about that cast we did a while ago to make things work, and the code explodes.

The compiler didn't help us because we silenced it by lying to it. There are two lies going on here, why wasn't the initial type precise enough? If we can see that it must be Dog, why weren't we able to show that to the compiler? Then there's the cast itself, which blatantly tells one thing to the compiler (x is a Dog), when the reality might be something else (x is a Cat).

How often do you think you know better than the compiler?

Side-Effects

Up until now the examples are quite obvious lies. We all feel a bit dirty when doing a cast, or throwing a runtime exception with the comment "this should NEVER happen". But here's a more subtle example.

Let's say there's a function called foo. When we see that foo takes no arguments and returns void, what can we say about what foo does without looking at its implementation? Nothing really. It's clearly doing something, and that something is not reflected in its inputs or outputs. We can only conclude that the function is doing some kind of side-effect, like changing a variable or writing to a file.

Just like us, the compiler doesn't know anything useful about foo either. So we have a function that is clearly doing something important, otherwise it won't be there. But all we tell the compiler is that "nope, move along, nothing interesting to see here". But that's a blatant lie, something interesting is going on here, but the type of foo doesn't reflect that.

The compiler gets so confused that even if we comment out a call to foo it won't complain, it's as if it doesn't even know it exists.

If you look at a typical codebase, many, many functions return void. Every time we do that we prevent the compiler from knowing something useful about the flow of the code.

With all these lies all over the place, it's no wonder that the compiler is not particularly useful. What happens if we stop lying to it?

Part III: No More Lies

If we want develop a serious, constructive relationship with the compiler, we must first stop lying to it. Only then can we open channels for serious dialogue.

We are so used to using null, exceptions, casting, and side-effects, that at first sight it might seem impossible to give those up.

But that's just an illusion.

Null

Some languages like Rust and Haskell have no null at all. Can you imagine what it's like to work that way?

Liberating, that's how it feels. You no longer have to keep worrying about an accidental null slipping through, because there aren't any. But, you might ask, what if something really is missing, how do I represent that without null?

In a world without null there are a number of ways to represent a thing that might be missing. The most common one is to use something like an Option type, that can be in one of two states, present with a value, or missing. When you want say that a String might not exist you use the type "option-of-String". And that makes all the difference to the compiler.

Now the compiler knows what's going on, since option-of-String is a distinct type from just String, it's no longer possible to call String methods, like substring, on it. Every time you deal with an option-of-String value the compiler will helpfully remind you to deal with the possibility that the value is missing.

On the flip side of that, now every time you have a String value you no longer have to worry that it might be missing. You can just use it without any defensive programming involved. And if you mistakenly break the rules and try to assign an option value to a non-option value the compiler will point that out and force you to think whether that's what you really want and what the consequences across the codebase will be.

Some languages, like Typescript and Kotlin have builtin support for nullable values. If you mark a String as nullable, it will become a distinct type from a plain String, giving us similar benefits to Option. In Java we can emulate something similar with Nullable and Nonnull annotation. But that requires some additional tooling to make it work.

What all those approaches have in common is that all of them involve the compiler. Instead of hiding our assumptions from the compiler we make them explicit in a way that the type-system can track.

We can do the same with exceptions.

Exceptions

I'm personally not aware of languages that completely remove exceptions from the language. But working mostly without exceptions, at least as a convention, is definitely possible.

Inspired by the null case, we can take a similar approach with exceptions and make them explicit in our types.

The most common way to do that is to use another container type like Result or Try. A return type like "result-of-String" means that we can have one of two cases, either a success with a String value, or a failure with some exception.

This approach makes it explicit to both us and the compiler what's going in a function. Now the compiler can track possible errors for us, and force us to deal with them as appropriate. And if we change some piece of code from String to "result-of-String", the compiler will helpfully shows where this needs to be handled.

In Java we can take an alternative approach and use checked exceptions. But industry experience over the years seems to indicate that people don't enjoy using them. It's probably a matter of ergonomics. Using a wrapper type instead might be a more ergonomic alternative.

While avoiding nulls and exceptions is fairly straightforward, casts can be trickier.

Casts

Unless forced to by some external library, I think that the presence of casts in our code should be considered a design smell. My first (possibly controversial) suggestion would be to configure your tooling to just forbid them by default.

When feeling the need to use a cast, we should consider whether the current design is doing a good job of reflecting our intentions. Recall that when doing a cast we can ask one of two questions, why was the type not precise enough to begin with? And why do we need to cast here?

There's no generic answer to these questions, but finding a good answer to either of them will make the compiler's life easier.

Some times the solution would be to add more methods on a parent interface (like Animal). Some times you might discover that you could've started out with the more specific type (Dog, for example) from the very beginning.

Another class of solutions is to remodel the problem from using open interfaces to using sealed/union types. Many modern languages support them. Using sealed types retains some of the flexibility of using open interfaces, but still allows the compiler to be in the loop. Instead of blindly assuming that some case is true, we can safely match on the sealed type and the compiler will gladly inform us about all the cases we must handle at this point. Some languages can even magically auto-complete all the relevant cases.

As code evolves, the compiler will keep track of the cases over time. And if something changed in the sealed type, the compiler will tell us about all the locations where the new case should be handled. Giving us a chance to handle whatever new logic that was added, before it crashes at runtime. Something that it was powerless to do when you used a plain cast.

Now for the thorniest issue of them all.

Side-Effects

Software exists to make side-effects, otherwise we would just have overheating boxes without any useful output. So side-effects cannot be completely eliminated from code. But they can be better managed.

This is even a deeper design issue than casting, and there isn't a one size fits all solution. My suggestion would be to separate as much pure computation from side-effects as possible.

In practical terms that would mean that every time you see a void return, or something that performs some side-effects mixed with other useful computation, you try to extract a pure function, with well-defined inputs and outputs, that only does computation, and another function that does the side-effects (similar to the Command Query Separation principle).

A common pattern would be to separate pure business logic from data fetching/writing. So instead of intertwining database calls with computation, you split into three separate phases: fetch, compute, store (a tiny ETL). First fetch all the data you need from a database, then you pass it to a (pure) function that produces some output, then pass the output of the pure function to a store procedure.

Now instead of having one big flow that takes no inputs and produces no outputs, you have three building blocks, and one of them has well-defined inputs and outputs. Not only is having this separation makes it easier to focus on your business logic, especially when writing tests. But now that we have inputs and outputs the compiler can follow along as well and help out.

Taking these ideas seriously leads us toward architectures that are known as "functional core, imperative shell". Within the functional core the compiler is much more useful, because it has actual types for inputs and outputs that it can follow and enforce. More "interesting stuff" is happening within the knowledge of the compiler.

Let's see what happens when we maximize that knowledge.

Part IV: The Compiler as Our Friend

Okay, so we stopped lying to the compiler as much as we can, and we are already seeing some gains in safety. No more null pointers, surprising exceptions, or fragile code. We can stop here and pat ourselves on our backs.

But we can do even better. Not lying is just the first step in a productive relationship. The next step would be to start sharing knowledge. The simplest way to start doing that is to sprinkle our code with more types.

Typed wrappers

When we use a string or an int in our code, what do they actually mean? Rarely is it just an arbitrary number or some raw text. Usually they represent some concept like a user ID or a file name. The thing is that this special meaning of the int or string are not reflected anywhere outside our head, or maybe a comment.

As time goes by we have more and more different meanings that an int can take in our system. Maybe it's an app ID, or a post ID, or a feed ID? Who knows. And so confusion arises, you start mixing up your ints, weird bugs creep up.

While this is happening, the compiler remains silent, "an int is an int, what can you do...". We didn't share our knowledge with the compiler, we have no one to blame.

The fix is simple: use small, typed wrappers, sometimes called "tiny types". Each concept you have in your system deserves to have a unique type, something for the compiler to hold on to. So you would have a UserID type, an AppID type, and so on. Underneath they might all be simple ints, but now both you and, more importantly, the compiler can distinguish the different concepts in code.

The difference might seem small, but the consequences are far reaching. Extra safety, the compiler will no longer let us confuse the different ints, is just the beginning.

Once the compiler gained knowledge of a concept you can use it for various tasks. For example, suppose you want to find out where are all the usages of user IDs in code. If we were still using ints, asking the compiler for all usages of the type int is meaningless, there are too many of those and they mean different things. Searching variable names is easy, but the results might not be complete. But if we have a dedicated type for UserID, that's easy, just ask your editor to find usages of the type and you get the precise results you need. The compiler keeps track.

For the same reasons, refactoring becomes easy. You want to change the representation of the UserID? Add some extra salt for security? No need to search for all the relevant ints and pray for the best. Modify the type and follow the compilation errors. Done.

Need some special validation for user IDs? Hide the constructor of UserID, add an alternative "smart" constructor, follow compilation errors. Done. No need to chase different ints across the system.

And for us humans, descriptive, domain-specific types make function signatures more readable, both for developers and for the occasional stakeholder. What's not to like about it?

Simple typed wrappers are, well, simple. But sometimes we need more sophisticated types to express ourselves to the compiler.

Union Types

Did you ever stumble on a class Foo that has too many fields with a long comment that explains something along the lines of "if field a is true then fields b and c must not be null". And so on and so forth for a few more different conditions?

Then when you actually start trusting Foo's comment and check that a is true, you discover, with a production null pointer exception, that b was actually null.

To describe this more formally, Foo had some invariants, and those invariants were violated. Not every combination of field values makes sense for our application.

How much nicer it would if the compiler could tell us "hey, look here, this invariant breaks if you use this set of field values"? Unfortunately, most compilers are incapable of expressing invariants that depend in such a way on runtime values.

The solution to many such problems is to remodel the type to better reflect the invariants that we want to encode, in such a way that the compiler will be able to follow along and only allow for legal combinations.

One of the most versatile tools for this is to use union, or sealed types (enums in some languages). With a union type we create one variant per case. And each case can only have a legal combination of values. So the case where a is true will become one variant with two non-nullable fields b and c. And that's it.

The advantage of taking this approach is that the union type doesn't even allow us to write down an invalid combination. You can only choose from valid ones, anything else is a compilation error.

The way you work with union types is that every time you need to access some data you need to match on the value. And then you act according to the case you're on. If you're in case a, you're guaranteed to have the b and c values. The compiler makes sure they are consistently there no matter how you got to this point of code.

This can make union types more powerful than tests that check invariants. Unlike a unit test, which can only test for a specific code paths you currently know about, the compiler can track all code paths, now and forever.

As mentioned before, in some languages the compiler can auto-complete the different cases relevant for the point in code you're at. Sometimes it feels like the code is writing itself, you just have to fill in the blanks.

When the code evolves over time and more cases are added, the compiler will notify you about all the relevant places where you need to update the logic to handle new cases. This is a great way to assess the consequences of changes you're about to make and help you with decisions at the design phase.

Using union types this way is a great way to model alternative states in code, and it's part of a more general approach to programming where you strive to "make illegal states unrepresentable". That is, encode your invariants as facts that the compiler can enforce and makes it impossible to even write code that will break those invariants.

This opens up a whole new way of approaching types.

Typed Guarantees

We are used to think about types as just a way to distinguish different values. This one's a string, that one an int. But as we rely more on the compiler to help us we can start thinking about types as guarantees that we want from the type-system. Or facts that we expect to hold, and that the compiler should verify for us.

A common scenario is that we want to make sure that the list we got as an argument contains at least one item. A classic example is when we need to compute some average, it's undefined for empty lists. Or choose a payment method in a payments system, you must have at least one to be able to proceed.

What we could do is revert to defensive programming, check the list before proceeding, and throw an exception if it's empty. But being defensive all the time is tedious (and eventually you forget about it), and we are back to lying to the compiler. Instead, we can signal our intentions with a type: NonEmptyList, a list type that cannot be even constructed without at least one element (some languages have it builtin, but it's also easy to implement in a library).

Not only does this communicate our intent to our callers better than any comment can do, but it also forces the compiler to make sure that nobody can call us with an illegal value. Once we encoded a guarantee as a type, we don't have to worry about it anymore. No defensive programming, no need to write extra tests for it. No more fear that some future refactor will break our invariants.

Once we start thinking about guarantees the compiler can make for us, the opportunities are endless. You can come up with a type for pretty much anything (within reason):

PositiveNumber, how much code that we write would be invalid if it got a negative number?
AgeOver18, maybe it's mission critical to limit certain flows based on user data
SortedByPriority, because it might be expensive to re-sort stuff just to be defensive

And the list goes on, for almost any kind of custom validation logic you might want. Some can be encoded with union types, some might require techniques like smart constructors. You don't have to get too fancy for this to become a valuable tool.

Once you start sprinkling these types around, not only is the intent of the code becomes clearer, but the compiler can help verify that everything is consistent, and that any new code knows what it's required to do.

With enough types in place you might get the feeling that "if it compiles, ship it".

Dialogue

In the prologue we saw what I think is a fairly common story, null unexpectedly crashing a production system. But we can make this story very concrete. I'm sure that many of us felt the Google Cloud outage in June 2025. So many services around the world were affected that it was very hard to miss.

From the incident report we learn that the root cause of it all was a null value where it was unexpected (along with some other operational compounding issues, like a lack of randomized backoff). And it's not really surprising. We saw that nulls are lies that we tell the compiler, values that it cannot track for us. In a large enough system, you are bound to lose track of them.

With the lessons that we learned, can we imagine another outcome? An outcome that doesn't involve a big chunk of the internet crashing?

So we already know that using null is lying to the compiler. If we are motivated enough we will never use a null in our code, opting instead for something that the compiler can track, an option-like type. The first step of not lying to the compiler might be the most difficult one, but once we take it a whole new world opens up for us.

The incident report mentions some "unintended blank fields" in Policy datatype that triggered the whole thing. Although I don't know the actual details, let's imagine that those fields used to be non-empty and now we want to open the possibility for making them empty. But we no longer have null at our disposal, what do we do? We change the type of the fields to be "option-of-X", and run the compiler. This is where the magic of dialogue starts to shine.

In a large codebase and a central entity like Policy you can imagine that you'll have many compilation errors the moment you set some fields to be optional. That sounds scary, but that's way better than null silently propagating all over the code and not knowing the consequences. With the compilation errors you now have a full report of all the code sites you need to reconsider.

You start going over them, one by one, fixing the errors. But the more errors you fix the more you feel that something's not right. Adapting the code feels like too much effort, too hacky, "a maze of twisty little ifs, all alike". That's the compiler trying to tell you something:

Maybe you need to reconsider the design?

And that's what you do, with all those changes that you had to make it becomes more obvious that these are not just fields that you made optional, what you really did is uncover a new flow in the system, with new rules. In the old flow the fields really are mandatory, in the new flow they are no longer needed. So you combined the two flows and made the fields optional. But by just setting the fields to be options the new flow becomes implicit in a combination of empty values and flags, something that the compiler can't track, and is not apparent when reading the code.

Instead of riding on the existing flow and tweaking it, you can make things explicit and turn Policy into a union type. If it already was a union, you add another case to describe the new flow, and you compile again. There are still many compilation errors, matches that are incomplete, but now they are telling you something different:

If you want to support this new flow, here are all the things you need to consider

And that's what you do reviewing the compilation errors and figuring out what the new flow implies at each site. Maybe along the way you discover that the new Policy type requires different validations that you didn't think of before. As a result in the new Policy you start using a NonEmptyList of AvailabilityZones which guarantees that you can't run the new flow without at least one applicable AvailabilityZone. (Of course AvailabilityZone is not some plain string, you have a dedicated type that tells you exactly what it means and gets validates accordingly.)

You compile again and somewhere along the edges of the system where you process user input the compiler tells you:

Listen, I don't have any way to prove that you actually have at least on availability zone provided by the user

And the compiler is right, if you were just using a regular list to represent the availability zones you would've missed this. Input processing happens far away from where we actually require the availability zones to not be empty and your unit tests missed this. So you add the appropriate user input validation. And lo and behold, it compiles. Ship it!

Yes, you struggled with the compiler, and yes it took some time. But guess who's getting good night's sleep tonight?

What's the Point of Learning Functional Programming?

Daniel Beskin — Thu, 13 Nov 2025 00:00:00 GMT

"What's the point of learning functional programming?" was a genuine question I got from a student on a functional programming course I was TAing on.

But let's rewind for a bit of background first. As I found myself standing in front of a frightened looking class, reviewing some Haskell basics, I was starting to feel guilty¹ for overwhelming them with all these foreign new concepts. Recursion, currying, function composition, algebraic data types, and the list goes on and on. So it felt natural to give them an escape hatch.

I mean, if all you know in life are loops, how can you possibly make do with just recursion? So when it was time for a new homework assignment we gave them a hint along the following lines:

Try solving with Pythonish pseudocode first, and every time you have a loop, you can convert it to a tail recursive function as follows...

Proceeding to explain how to convert loops to tail recursion². At that, some of the students seemed mildly relieved, and continued with their homework.

Fast forward after the submissions. A student approached me after class with a question. I was surprised, till now I scarcely seen them outside their Haskell stupor. And here's what the student had to say. "If all we did was write some Python in Haskell syntax, what's the point of learning functional programming?".

Thinking about it, he was right. With the escape hatch we gave them they really could solve the homework by mechanically translating Python code into Haskell. Apart from some new syntax, you're not really learning much of anything new. And that's unfortunate, there's so much to learn from Haskell and functional programming in general. It would be a shame to miss the opportunity.

Can we do better?

The Homework

The homework problem was to solve the knight's tour problem. That is, given a chess board of some arbitrary dimensions and a starting position, find a path that a knight can take that will go through all the cells of the board exactly once.

Efficiency was not the point of this exercise, so the students could just do a brute-force search for the correct path using backtracking. Suppose you're a student and Python is your main weapon of choice. How would you solve this?

Here's the core function of a very naive attempt at brute-forcing the solution³:

# 1
def tour(n, visited, moves, pos):
    if len(visited) == n * n: # 2
        return moves

    for knight_move in all_moves: # 3
        (row, col) = pos
        (dx, dy) = knight_move.value # 4

        next_row = row + dx
        next_col = col + dy

        next_pos = (next_row, next_col)

        # 5
        if is_valid(n, visited, next_row, next_col):
            new_visited = visited + [next_pos] # 6
            new_moves = moves + [knight_move]

            # 7
            result = tour(n, new_visited, new_moves, next_pos)

            # 8
            if result is not None:
                return result
    # 9
    return None

This is not very good code, for various reasons. But that's not really the point, it (slowly) solves the problem as stated, and that's good enough for illustration purposes. Let's review what this does:

tour is a recursive function (1) that takes the current state as input:
- The size of the board n
- The list of already visited coordinates
- The moves we constructed so far
- And a tuple of coordinates pos for the current position
We have a stopping condition (2), if the path we visited covered the whole board, in which case we are done and we return the moves that were constructed
If not, we iterate over all possible knight moves (3), which are defined as a separate enum
For each such move, we construct the next cell that we are going to visit (4)
If the new cell is valid (5), i.e., within the bounds of the board and wasn't visited before
We add the new position to the list of visited cells (6), and the current move to the moves we are building⁴
We then proceed with the next step by recursively calling on tour with the new state (7)
If the result of the recursive call was successful we return that (8)
Otherwise, we backtrack on this attempt and try the next knight move in the list
When we exhausted all the possible knight moves, we give up and return None (9)

This is all good and well, but the actual homework is in Haskell. What good is it to have this Python solution instead?

A Haskell Rewrite

It is true that having a concrete (and correct) solution written out is very helpful in thinking about a problem. But with the tip that we gave the students about loops and recursion, it's even better than that. It's actually fairly mechanical to translate the Python code into Haskell. To wit⁵:

tour n visited moves row col =
  if length visited == n * n
    then Just moves -- 1
    else go allMoves -- 2
 where
  -- 3
  go [] = Nothing
  go (knightMove : rest) =
    let (dx, dy) = moveValue knightMove
        nextRow = row + dx
        nextCol = col + dy
        nextPos = (nextRow, nextCol)
     in if isValid n visited nextRow nextCol
          then
            let newVisited = visited ++ [nextPos]
                newMoves = moves ++ [knightMove]
                result = tour n newVisited newMoves nextRow nextCol
             -- 4
             in case result of
                  Just solution -> Just solution
                  Nothing -> go rest -- 5
          else go rest -- 6

If you're familiar with Haskell syntax, this code should look basically the same as the original Python code. There are a few differences, but they are mostly cosmetic:

Since there's no None (or nulls in general) in Haskell, we have to wrap things with Maybe (1), it's a bit more syntax compared to Python, but not by much
Haskell is expression-oriented, so there are no early returns
As a consequence the stopping condition has an else (2) that invokes the logic for the next step
The main loop over knight moves was replaced by a tail recursive function called go (3), this is our tip in action
We need a bit more ceremony when working with Maybe, so we pattern match on it (4)
Since we are not running in a loop, we must call the next backtracking step explicitly by invoking go (5, 6) with a new argument

If you're a student who just created this Haskell solution from the Python sketch, what new things did you learn?

You might've learned that it's possible to work without nulls. But since this code is small, the consequences of not having null are not very visible, and you mostly get just some added syntactic noise⁶. And not having null is not a uniquely functional thing, e.g., Rust doesn't use null as well.

There's a glimpse of being more expression-oriented here. That's a deep and very impactful principle. But since we just mechanically translated from Python, it's easy to miss its significance. This is even more obscured by the fact that it's difficult to appreciate the consequences of expression-orientedness "in the small".

Lastly, we learned that it's possible to replace iteration with simple recursion. Which is cool and all, but what's the point? for loops work well enough, doing recursion just for that feels like a lousy (and overly general) syntax for something we already have figured out.

If you submit this solution to the problem it won't be out of place to wonder "what's the point?".

Going Functional

Now you may be screaming at the screen that no, functional programming is worth learning, it's not just Python with uglier syntax. And you would be right, there is a lot to learn, even with a simple assignment such as the one we are working with now.

But to get these benefits we must make an effort to break away from our "mainstream" roots, and stop translating the old way into a new syntax. We need to completely change the way we think about the problem and its solution.

For problems like this knight's tour, one good approach is to use what is called "wholemeal programming":

Functional languages excel at wholemeal programming, a term coined by Geraint Jones. Wholemeal programming means to think big: work with an entire list, rather than a sequence of elements; develop a solution space, rather than an individual solution; imagine a graph, rather than a single path. The wholemeal approach often offers new insights or provides new perspectives on a given problem. It is nicely complemented by the idea of projective programming: first solve a more general problem, then extract the interesting bits and pieces by transforming the general program into more specialised ones.

So instead of exploring the solution space step by step, we'll create a list of the whole solution space at once, and then use that list to solve the problem we are interested in. Then I'm sure we'll find something new to learn about programming and problem-solving.

The first issue to tackle is, how can we possibly represent the whole solution space? It might be huge. Lucky for us, Haskell is lazy by default, so we can just pretend that we have the full list, but actually compute it fully only on demand.

Next, since we are going to deal with a state space, what are the actual states that we will be working with?

We can encode a single state in the tour with this record⁷:

data TourState = TourState
  { board :: Board
  , visited :: [KnightPos]
  , moves :: [KnightMove]
  , current :: KnightPos
  }

These are basically the arguments to the original tour function wrapped up in a single record (with some more civilized wrapper types)⁸.

To be able to explore the state space, we need a way to move between states. For that we can define:

moveState :: TourState -> KnightMove -> Maybe TourState

Given a single state and a move, we compute the next state by calculating the new position and updating the list of moves. Since not every move is legal from a given state, we accommodate possible failure with a Maybe. If we can't move to the next state the result will be Nothing.

You can see the implementation in the repo, but it's basically the same logic that we had before, just adapted to using TourState.

With moveState we can apply one move to a state. Next we want to try out all the possible knight moves from a state. The signature will be:

nextStates :: TourState -> [TourState]

From a given TourState generate all possible (legal) states that can be reached by a single step. Here's the implementation:

nextStates state = mapMaybe (moveState state) allKnightMoves

We take all the possible knight moves and with the help of mapMaybe apply moveState from the current state to every element in the list of moves. mapMaybe takes care of removing all the Nothing entries, so that we end up with a flat list with just the legal states we can reach.

This is actually our first example of using wholemeal programming. Notice how we didn't iterate step by step. Instead of thinking about moving between individual knight moves, we applied the moving logic to the whole list with mapMaybe.

With these helpers we are now ready to create a (potentially) infinite list of all the states that we can be in while searching for a tour from the initial state. Concretely, we need to implement the following signature:

allStatesFrom :: TourState -> [TourState]

Given a single state, we create a list of all the states that can be reached from it.

Creating an infinite list of steps might sound intimidating, but we'll do it anyways:

allStatesFrom state = -- 1
  let
    next = nextStates state -- 2
    allTheRest = concatMap allStatesFrom next -- 3
   in
    state : allTheRest -- 4

We start from the initial state (1), and we compute all the legal next states using nextStates (2). This is one step into the search. Next, we want to take the next steps from each of these new states. We do it by recursively⁹ calling allStatesFrom on each of the next states (3). concatMap flattens all the resulting lists into one big list. We construct the full list of states by prepending the initial state to allTheRest with all the deeper steps (4).

And just like that we have a list that represents all the possible states in the problem we are solving. This is a bit mind-bending, take a moment to let the tricky recursion sink in properly.

Notice how "wholesome" this code is. We don't deal with edge cases or stopping conditions, we just generated the whole thing in one fell swoop. Nor do we worry about going off into infinity, laziness prevents this list from being computed prematurely¹⁰.

With this list in hand, it's almost trivial to solve the knight's tour problem. We just need to find the first state from the list that covers the whole board.

We can define the stopping condition like so:

isComplete TourState{board, visited} =
  boardSize board == length visited

This function determines whether a given state is a solution by checking the size of the list of visited positions (which happens to be the same as the stopping condition in the original Python solution).

With these two building blocks in hand, a list of states and a stopping condition, we can easily compose them into a single solution:

tour :: Board -> KnightPos -> Maybe [KnightMove]
tour board pos = -- 1
  let
    -- 2
    init = initState board pos
    -- 3
    finalState = find isComplete (allStatesFrom init)

   in
    fmap getMoves finalState -- 4

Step by step:

The final tour function takes in a board and an initial position (1)
From this we build the initial state (2), which just initializes a fresh instance of TourState with the current position and an empty list of moves
Now for the highlight of this solution, we use find (3) to locate the first TourState that matches isComplete in the full list of states that we generated from the initial state
The final result (4) just extracts the moves from the state that we found (which might be missing, hence the fmap call)

I find this approach to be breathtakingly elegant. Thanks to the classic "Why Functional Programming Matters" paper for the inspiration. Not only does it work, but now we have an opportunity to learn something truly novel.

What Did We Learn?

With such a strikingly different solution, we are bound to learn something new. Here are a few lessons that we can take away from this.

Laziness

To generate the list of states we used laziness in an essential, non-contrived way. Without laziness, the list could potentially overflow memory or take too much time to compute upfront, making it impractical to generate "the whole state space".

Haskell's built-in laziness makes this invisibly seamless (for better or for worse), but the lesson that we learn here is applicable almost anywhere. Once we recognize this need for laziness we can use the appropriate technique to achieve it in pretty much any modern language. Be it generators in Python, streams/iterators in Java, or any streaming library in any language.

Laziness, in its conceptual form, is a powerful tool we can use anywhere. And developing an intuition for it from lazy Haskell lists makes the approach to it relatively gentle.

Explicit State Space

The way we went about exploring the state space forced us to actually think about that space. What are the possible states? How do we move between them?

This was explicitly reified in the TourState type and the moveState function.

The state space exists independently of whether we bother to acknowledge it or not. But my feeling is that when thinking imperatively about code it's much easier to "miss the forest for the trees". Focusing on each step we are making in some iterative loop tends to hide the significance of the state space as a whole from us.

Whether or not you're applying functional programming to your code, keeping in mind the state space can only be beneficial. If only for something like "make illegal states unrepresentable".

Modularity

By separating the exploration of the state space from the stopping condition we made the code much more modular than what it was before. Previously we had to intertwine the stopping condition directly into iteration, but no more.

This kind of modularity opens the door to many possible modifications that we can apply to one part of the code without disturbing the others.

Some examples:

We can change the stopping condition without touching the state space exploration logic. Instead of using find to find the first solution, we could search for all solutions. Or we could search only for "closed tours" (where the final cell is adjacent to the initial cell).
In the solution code we are exploring the state space using a preorder depth-first traversal. We can easily adapt it to be any other depth-first traversal, or with some more effort we can make it into any traversal order we want.
The current algorithm is a naive brute-force approach, we can make things faster by pruning the state space list with heuristics, like applying Warnsdorf's rule.

The important thing is that we can make all these changes by modifying only the code that is responsible for it, without touching its surroundings. This also means that everything can be tested separately without having to test unrelated parts.

Contrast this to the original code where there's no way to modify or test anything without touching everything else. The stopping condition sits smack in the middle of iteration.

That's a level of modularity that takes some effort to achieve with procedural code, but flows out naturally from the wholemeal approach to solving problems.

Composition

Complementing modularity we have composition¹¹.

It's one thing to split things into modules, but another thing is how easy it is to compose the modules back into a single piece of code that actually solves the full problem.

In our case it was very easy, just a call to find along with the state space list and the completion predicate. And the same applies to all the different variants described in the previous section. We can easily mix and match the different implementations to generate new working pieces of software.

Using higher-order functions (find) as the glue for simple, purely-functional (or side-effect-free) components is a great recipe to obtain code with good composition properties.

Quoting from Tony Morris:

Suppose any function that exists. I will suppose one that is made of parts A, B, C and D. I might denote this A ∘ B ∘ C ∘ D. Now suppose you desire a function that is made of parts A, B, C and E. Then the effort required for your function should be proportional to the effort required for E alone.

The closer you are to achieving this objective for any function, the closer you are to the essence of (pure) functional programming. This exists independently of the programming language, though programming languages will vary significantly in the degree to which they accommodate this objective.

Imperative programming – specifically the scope of a destructive update – is the anti-thesis of this objective.

Wholemeal Programming

Tying all these concepts together is wholemeal programming, showing us how to tackle problems with a wholly different mindset.

Not only does it force us to think differently about problem-solving, but it brings concrete, practical advantages to the code we produce.

Although it's possible to see every one of these advantages in other paradigms, functional programming really shines in making those advantages stand out. Once you learn about them, you can apply them pretty much anywhere. You just need to stop mechanically translating your existing knowledge and break away from everything you think you know.

So what's the point of learning functional programming? You tell me...

I'm sure the guilt fades away over time, but I was doing this for only one semester. ↩
Funny how usually programmers find themselves going in the opposite direction. Let's ignore for now the fact that in Haskell tail recursion is not that important. ↩
The full code for all the examples that follow can be found on GitHub. The Python solution here. ↩
Appending to a list in a loop like this is quadratic, which might slow down the implementation. Can you think of a better way to do this while still using a simple list? ↩
See the full code here. ↩
Nor do we use any of the special Maybe functions like map. ↩
See the full code here. ↩
We are still keeping a naive mindset towards performance so as to not distract from the main point. But playing around with the types of the record can yield some speedups (e.g., not using lists for sets). ↩
Recursion is sometimes called "the goto of functional programming". One might attempt to remove the explicit recursion in favor of something more structured like unfold. ↩
The state space for this particular problem is actually finite. But that's beside the point, since it can be very large for sufficiently large boards. For such boards actually computing the whole list would likely crash the program. ↩
I'm not using the term "compositionality" because it has a more specific technical meaning which I'm not trying to imply here. ↩

Pretty Symlinking with Home Manager

Daniel Beskin — Sat, 18 Oct 2025 00:00:00 GMT

I have a confession to make. It might not be appropriate to admit in polite society, but I'm not ashamed: I don't manage all my dot files with Nix.

Unless I get some concrete benefit like templating or cross-library integration, I can't be bothered to rewrite arbitrary dot files into Nix. And beyond that, I just don't want to be forced to rebuild every time I make a tiny tweak to this or that setting¹.

On the other hand, I do like the idea that all my configs, Nix or otherwise, are in the same repo. And since I'm already using a tremendously powerful file management tool, Home Manager, I don't want to bring in yet another tool like GNU Stow or chezmoi. Call me simple-minded, but I just want as few tools as reasonably possible² to achieve my goals.

Luckily, Home Manager got our back. What I just described fits nicely within the capabilities of mkOutOfStoreSymlink. And there are tutorials on how to make use of it for this exact purpose³. But this post is not about the capabilities of Home Manager (or Nix in general), but rather how people use the Nix language itself.

While setting up my system, I was looking for examples around the web, and a typical example of using mkOutOfStoreSymlink would look something like this⁴:

{
  config,
  symlinkRoot, # 1
  ...
}: {
  xdg.configFile = { # 2

    # 3
    "copyq/copyq.conf".source = config.lib.file.mkOutOfStoreSymlink "${symlinkRoot}/copyq/copyq.conf";
    "shellcheckrc".source = config.lib.file.mkOutOfStoreSymlink "${symlinkRoot}/shellcheckrc";
    "spaceship.zsh".source = config.lib.file.mkOutOfStoreSymlink "${symlinkRoot}/spaceship.zsh";

    # 4
    "eww" = {
      source = config.lib.file.mkOutOfStoreSymlink "${symlinkRoot}/eww";
      recursive = true;
    };
    "fish" = {
      source = config.lib.file.mkOutOfStoreSymlink "${symlinkRoot}/fish";
      recursive = true;
    };
    "nushell" = {
      source = config.lib.file.mkOutOfStoreSymlink "${symlinkRoot}/nushell";
      recursive = true;
    };
    "nvim" = {
      source = config.lib.file.mkOutOfStoreSymlink "${symlinkRoot}/nvim";
      recursive = true;
    };
    "tmux" = {
      source = config.lib.file.mkOutOfStoreSymlink "${symlinkRoot}/tmux";
      recursive = true;
    };
    "waybar" = {
      source = config.lib.file.mkOutOfStoreSymlink "${symlinkRoot}/waybar";
      recursive = true;
    };
  };
}

This is not particularly complicated Nix code. We get the root directory for the symlinks as an argument (1)⁵. We then set up xdg.configFile (2), which takes an attribute set and creates matching files in XDG_CONFIG_HOME. Finally we set the actual standalone files (3), and directories (4) with recursive = true.

Great, that works as well as it should, but, oh my eyes... I'm a programmer, damn it. I have DRY tattooed on my forehead. It hurts deep down in my soul to see that much repetition:

The repeated fully qualified mkOutOfStoreSymlink calls
The repeated symlinkRoot prefix
The doubly named files and folders (both keys and sources); typos, anyone?
The repeated source and recursive keys

And it's only going to get worse as we keep on adding more and more keys here.

We're so used to using highly constrained and repetitive configuration formats like YAML⁶, that this kind of never ending tedium is not perceived as something out of the ordinary. Just another heap of text to maintain along with a pile of Kubernetes YAMLs, nothing to see here... Maybe one day someone invents a templating engine for Nix code to relieve some of the noise.

Hyperbole aside, I do suspect that many of the people that write Nix treat it as just another "config syntax". While in reality Nix is a full-fledged programming language, capable of many things you might expect a programming language to do⁷. DRYing code with Nix is very much possible, and I would claim that it's even desirable.

One might argue that the repetition allows for some flexibility. What if some of the entries do not map the files one-to-one, suppose you want to rename a file on the fly? Maybe, but we don't actually use that flexibility most of the time. So from now on I'll consider the problem of simple, stow-like, symlinking without any special cases. Special cases can be added separately.

Now, not everyone share my sensibilities and sensitivities when it comes to code that you have to maintain (even as a hobby). I'm also, like many software engineers when presented with shiny new toys, prone to over-engineering. So if you're fine with this kind of repetition and it works for you, keep at it. Don't let me infect you with my DRY cough. For the rest of you, how do we clean this code up?

I'll present a series of steps trying to reduce repetition and to (subjectively) aid readability. I don't claim that any of them are necessary, or that the resulting code is "the Nix way" of doing things. What I want to do is show that you can make configuration in Nix less repetitive and more readable, you just need to pick the right tools from whatever the Nix language offers. Hopefully this will be mostly beginner-friendly as well.

Imports

Although Nix doesn't have separate import statements like many mainstream languages, you can simulate them easily with let bindings to make thing more readable:

{
  config,
  symlinkRoot,
  ...
}: let
  # 1
  link = config.lib.file.mkOutOfStoreSymlink;
in {
  xdg.configFile = {
    # 2
    "copyq/copyq.conf".source = link "${symlinkRoot}/copyq/copyq.conf";
    "shellcheckrc".source = link "${symlinkRoot}/shellcheckrc";
    "spaceship.zsh".source = link "${symlinkRoot}/spaceship.zsh";

    "eww" = {
      source = link "${symlinkRoot}/eww";
      recursive = true;
    };
    # ... similarly for the rest
  };
}

We have the "import" on (1), we just assign the fully qualified function reference to another value (which we can name whatever we want). Then use that instead of the fully qualified name (2). Already much less noisy.

This simulates a renaming import. We can also have a "regular import" with the following syntax:

let
  inherit (config.lib.file) mkOutOfStoreSymlink;
in
  # ... use mkOutOfStoreSymlink unqualified

The inherit syntax binds any symbol we can reference to its unqualified name. But I like the shorter name for now. We'll use inherit more later on.

Functions

One of the most ancient ways to DRY up code is to use functions. Simple, small, everyday functions. Nix being a functional language has a very pleasant syntax for functions, so there's no reason not to use it.

First step, let's stop repeating the symlinkRoot prefix:

{
  config,
  symlinkRoot,
  ...
}: let
  link = config.lib.file.mkOutOfStoreSymlink;
  # 1
  toSrcFile = name: "${symlinkRoot}/${name}";
in {
  xdg.configFile = {
    # 2
    "copyq/copyq.conf".source = link (toSrcFile "copyq/copyq.conf");
    "shellcheckrc".source = link (toSrcFile "shellcheckrc");
    "spaceship.zsh".source = link (toSrcFile "spaceship.zsh");

    "eww" = {
      source = link (toSrcFile "eww");
      recursive = true;
    };
    # ...
  };
}

Instead of manually prepending symlinkRoot for each entry, we defined a new function toSrcFile (1). It takes the name of the file as an argument and prepends symlinkRoot to it. Now we use toSrcFile in each entry (2) without repeating the reference to symlinkRoot every time.

Looking at the new code though, it seems that we only ever use link in combination with symlinkRoot. Let's redefine link to call toSrcFile and stop repeating this combo:

{
  config,
  symlinkRoot,
  ...
}: let
  # 1
  inherit (config.lib.file) mkOutOfStoreSymlink;

  toSrcFile = name: "${symlinkRoot}/${name}";
  # 2
  link = name: mkOutOfStoreSymlink (toSrcFile name);
in {
  xdg.configFile = {
    # 3
    "copyq/copyq.conf".source = link "copyq/copyq.conf";
    "shellcheckrc".source = link "shellcheckrc";
    "spaceship.zsh".source = link "spaceship.zsh";

    "eww" = {
      source = link "eww";
      recursive = true;
    };

    # ...
  };
}

inherit is back is business now. We import mkOutOfStoreSymlink (1), and now link is redefined to call mkOutOfStoreSymlink after applying toSrcFile (2). And now the link calls look very clean (3).

Programmatic Keys

Now the starkest bit of repetition are the file paths. Each name is repeated both as a key and as an argument to link. If this was YAML we would be stuck, but in Nix we can create attribute sets with dynamically specified keys. They don't have to be string literals. Here's a function to link a single file:

linkFile = name: {
  ${name}.source = link name;
};

This function creates an attribute with the provided key name (and source under it). The ${name} syntax lets us splice the dynamic key name into the set without using a string literal.

We can do the same thing with directories⁸:

linkDir = name: {
  ${name} = {
    source = link name;
    recursive = true;
  };
};

These functions can now be used to create all the file entries without repeating the path. But we have a wrinkle. Calling linkFile creates a set with a single key. Calling it multiple times will produce multiple sets, each with one key. What we need though, is one set with multiple keys. Since I don't know how to create "bare" key/value pairs, we'll just have to merge all these redundant sets into one using the // operator. Like so:

let
  # ...

  confFiles = # 1
      (linkFile "copyq/copyq.conf")
    // (linkFile "shellcheckrc")
    // (linkFile "spaceship.zsh");

  confDirs = # 2
      (linkDir "eww")
    // (linkDir "fish")
    // (linkDir "nushell")
    // (linkDir "nvim")
    // (linkDir "tmux")
    // (linkDir "waybar");

  # 3
  links = confFiles // confDirs;
in {
  xdg.configFile = links;
}

No more filename repetition! We make multiple calls to linkFile (1) and linkDir (2) and combine the results into one big attribute set with // (3), the resulting links value is an attribute set that we can assign to xdg.confFile.

List Processing

Written out this way, this looks a lot like we have two lists (separated by //⁹), one for files and one for directories. Each entry is prefixed with either linkFile or linkDir, the last source of repetition in this code. Let's make the list processing explicit.

Nix has a bunch list processing functions, and it's quite easy to set up a "data processing pipeline". First we extract the lists we want to deal with:

confFiles = [
  "copyq/copyq.conf"
  "shellcheckrc"
  "spaceship.zsh"
];

confDirs = [
  "eww"
  "fish"
  "nushell"
  "nvim"
  "tmux"
  "waybar"
];

Now to each entry of a list we need to apply the appropriate function (either linkFile or linkDir). The way to do it is to use map:

let
  # ...

  inherit (lib) map;

  # 1
  confFiles = map linkFile [
    "copyq/copyq.conf"
    "shellcheckrc"
    "spaceship.zsh"
  ];

  # 2
  confDirs = map linkDir [
    "eww"
    "fish"
    "nushell"
    "nvim"
    "tmux"
    "waybar"
  ];

  # 3
  links = confFile ++ confDirs;
in # ...

map takes care of iterating over the list and applying the supplied function to each entry (1, 2). We are dealing with lists now, so we combine the two lists into links using ++ (3). We are almost back to the original links value. But the types don't quite add up.

Previously links was an attribute set, now it's a list of attribute sets. No worries though, that's why we have mergeAttrsList¹⁰:

let
  inherit (lib) map mergeAttrsList;

  confFiles = map linkFile [
    "copyq/copyq.conf"
    "shellcheckrc"
    "spaceship.zsh"
  ];

  confDirs = map linkDir [
    "eww"
    "fish"
    "nushell"
    "nvim"
    "tmux"
    "waybar"
  ];

  links = mergeAttrsList (confFiles ++ confDirs);
in {
  xdg.configFile = links;
}

And that's where I'll stop DRYing up the code. We removed all of the egregious repetitions, and I think that with the two explicit lists of files the essence of what's going on is much clearer (contingent on good naming for the custom functions).

Cosmetics

Lastly, let's apply some cosmetic changes to the code, to make it really shine. This is getting very subjective, but I personally like the "DSL aesthetic", with small, well-named, custom functions to do my bidding. I'm sure that many people prefer using built-in functions more explicitly, since it's easier to look up what they do. Even if you don't like the style below, I think the DSL/small library mentality can be useful at times.

A minor tweak we can do to hide away the explicit list manipulation is to extract the map calls into helper functions:

# 1
linkConfFiles = map linkFile;
linkConfDirs = map linkDir;

# 2
confFiles = linkConfFiles [
  "copyq/copyq.conf"
  "shellcheckrc"
  "spaceship.zsh"
];

# 3
confDirs = linkConfDirs [
  "eww"
  "fish"
  "nushell"
  "nvim"
  "tmux"
  "waybar"
];

With map link... assigned to new functions (1), we can apply the new linkConfFiles and linkConfDirs directly to our file lists (2, 3). The main code now flows better without mentioning the plumbing with lists and map. Nix's curried functions make creating these small helpers a breeze. Note how we didn't need to explicitly name the argument to linkConfFiles, we just partially applied map (a.k.a, point-free style).

Next, this pattern of combining attribute set lists with ++ and then merging seems to be pretty useful. Let's make it work more generically. Instead of working with just two lists, we can make it work with a list of lists:

inherit (lib) flatten mergeAttrsList;

flatMerge = sets: mergeAttrsList (flatten sets);

flatMerge is now doing the same thing, but relies on the built-in flatten to concatenate a list of lists, rather than just two lists. We use it like so:

links = flatMerge [confFiles confDirs];

Finally, when you work enough with functional-styled code some patterns tend emerge. One of them is applying a chain of functions one after the other. In the code so far we had two examples of it:

link = name: mkOutOfStoreSymlink (toSrcFile name);
flatMerge = sets: mergeAttrsList (flatten sets);

We first apply one function, and then another to the output of the first one. This is function composition, the same one you learned about in school algebra. It tends to pop out a lot. This pattern can be easily abstracted a way, without us manually writing new functions every time we need to compose a couple of functions.

For some reason, Nix doesn't have a built-in function composition operator. But it does have the pipe function, which takes a value and a list of functions, it then composes them (in reverse) and applies the resulting function to the value.

Convenient though pipe is, its signature is not quite what we want. It would be more reusable for us to first receive a list of functions, and then the value. This is easily remedied with the flip function:

inherit (lib) flatten flip map mergeAttrsList;

# 1
pipe = flip lib.pipe;

# 2
flatMerge = pipe [flatten mergeAttrsList];
# 3
link = pipe [toSrcFile mkOutOfStoreSymlink];

In (1) we override the built-in pipe with the flipped version. With the pipe in hand¹¹ we can turn flatMerge (2) and link (3) into explicit pipelines of composed functions, without manually spelling out the function composition. As a bonus, this easily generalizes to any number of (compatible) functions.

And that's all the cosmetics we'll apply for today. The full code now looks like this:

{
  config,
  lib,
  symlinkRoot,
  ...
}: let
  inherit (config.lib.file) mkOutOfStoreSymlink;
  inherit (lib) flatten flip map mergeAttrsList;

  pipe = flip lib.pipe;
  flatMerge = pipe [flatten mergeAttrsList];

  toSrcFile = name: "${symlinkRoot}/${name}";
  link = pipe [toSrcFile mkOutOfStoreSymlink];

  linkFile = name: {${name}.source = link name;};
  linkDir = name: {
    ${name} = {
      source = link name;
      recursive = true;
    };
  };

  linkConfFiles = map linkFile;
  linkConfDirs = map linkDir;

  confFiles = linkConfFiles [
    "copyq/copyq.conf"
    "shellcheckrc"
    "spaceship.zsh"
  ];

  confDirs = linkConfDirs [
    "eww"
    "fish"
    "nushell"
    "nvim"
    "tmux"
    "waybar"
  ];

  links = flatMerge [confFiles confDirs];
in {
  xdg.configFile = links;
}

You can find all the different revisions in an executable format in the repo. If you want homework, you can try to apply the same approach to files linked directly to home.file.

My eyes can rest now. But if we were really serious about Nix programming, we would probably turn this into a real, configurable, Nix module, and hide away all the utility functions from our hypothetical users.

Although usually I don't tweak configs much after a short period of initial set up, some apps get tweaked continuously. For example, Neovim, whose configuration (gasp) I don't manage through Nix. ↩
Reasonable people may disagree what to consider "reasonable". But I'm fine with using myself as the golden standard for being reasonable. ↩
If you're using flakes, keep in mind how flakes treat relative paths and be careful to use absolute paths when passing arguments to mkOutOfStoreSymlink. ↩
You can find the code for these examples in the repo. ↩
Don't forget that this must be an absolute path if you're using flakes. ↩
Instead of using a more capable solution like Dhall. ↩
Although one might argue that when used for configuring NixOS it might be beneficial to have a non-Turing complete language, something in the vein of what Dhall does. ↩
I leave it as an exercise for the reader to unify these two functions and to reduce that tiny repetition with the source key. ↩
This sure looks a lot like an application of some fold variant. Coincidence? ↩
We could've use a fold instead of mapping and merging. But that would've been clunkier to write, and possibly less performant, since mergeAttrsList seems more optimized with some kind of binary merging. ↩
When all you have is a pipe, everything looks like a functional pipeline... ↩

Validating Custom Keyboard Layouts on NixOS

Daniel Beskin — Sat, 04 Oct 2025 00:00:00 GMT

If you're in the rare intersection of people that use NixOS and custom keyboard layouts, this post is for you. And for those of you who don't use custom layouts, you might learn something useful about build-time assertions and laziness.

All three of you still reading, here we go.

So, one day I decided that I wanted to use a custom keyboard layout on my NixOS machine. Should be simple enough. If you find the right bit of documentation, you'll find out that you can easily and declaratively add custom XKB layouts to NixOS using the services.xserver.xkb.extraLayouts key. This presumes that you understand how to write your own XKB symbol files, a nontrivial assumption given how arcane that format is¹.

Suppose you have the following simple XKB symbol file²:

xkb_symbols "test"
{
  include "us(basic)"

  key <CAPS> {[ Escape ]};
};

This file defines a new layout based on the regular QWERTY layout, mapping the capslock key to escape.

Adding this to your NixOS setup is easy enough:

services.xserver.xkb.extraLayouts = {
  test = {
    description = "Test";
    languages = ["eng"];
    symbolsFile = ./test.xkb;
  };
};

Each entry under extraLayouts points to one XKB symbols file along with some metadata. If you add this to your NixOS config, this will build and successfully add the new layout to the system.

But then I kept on reading the documentation and saw the following warning:

... a broken XKB file can lead to the X session crashing at login. Therefore, you're strongly advised to test your layout before applying it...

And then they show you some cryptic bash commands to actually perform the testing. This decidedly doesn't feel like the Nix way. What's the point of having this amazing build machinery if I have to run some manual commands every time I decide to update my keyboard layouts³?

No, what I want is to validate my layouts during the Nix build, and stop the build if the XKB file is invalid.

After some digging around, I found this somewhat dated documentation on the Wiki:

let
  # 1
  compiledLayout = pkgs.runCommand "keyboard-layout" {} ''
    ${pkgs.xorg.xkbcomp}/bin/xkbcomp ${./path/to/layout.xkb} $out
  '';
in
  # 2
  services.xserver.displayManager.sessionCommands = "${pkgs.xorg.xkbcomp}/bin/xkbcomp ${compiledLayout} $DISPLAY";

This uses xkbcomp to compile the layout file (1), which should fail if the file is invalid. It then passes on the result of compilation as part of the sessionCommands for XServer (2). But using sessionCommands is outdated, we want to use the newer extraLayouts way of doing things, which feels more high-level.

So let's combine the two approaches: compile only to validate the layout, and if compilation succeeded, use the extraLayouts setting like we did before.

To make sure this actually work for us, let's check the relevant commands in the terminal:

xkbcomp ./test.xkb

This won't have any output if the file is valid. But suppose you make some error in the file, then you'll get something like this:

➜ xkbcomp ./test.xkb
syntax error: line 5 of ./test.xkb
last scanned symbol is: CAPS
Errors encountered in ./test.xkb; not compiled.

This is exactly what we need, a check we can run during the build. The only wrinkle with xkbcomp is that it makes what I find to be a strange usage of exit codes. Whether or not compilation succeeds, the exit code is always 1. No worries though, it's easy enough to accommodate.

With this tool in hand, here's my naive first attempt to write a Nix function that compiles the layout before using it:

buildLayout = { # 1
  name,
  description,
  lang,
  symbols,
}: let
  # 2
  xkbcomp = pkgs.lib.getExe pkgs.xorg.xkbcomp;
  # 3
  compilationOutputFile = pkgs.runCommand "${name}-compiled-keyboard-layout" {} ''
    (${xkbcomp} ${symbols} 2> $out) || true
  '';
  # 4
  compilationOutput = builtins.readFile compilationOutputFile;
  # 5
  compilationSuccess =
    (compilationOutput == "") || abort "Failure compiling layout [${name}]: ${compilationOutput}";
in {
  # 6
  ${name} = {
    description = description;
    languages = [lang];
    symbolsFile = symbols;
  };
};

This is a bit long, so step by step:

In (1) we define a new function buildLayout that takes all the data that we need to construct a new layout, symbols refers to the XKB symbols file
We find the executable for xkbcomp (2)
And use it to validate the symbols file (3)
- We want to run the command and get its output to be available within the build
- For that we use runCommand which will run the given command and let us access its output
- This is done by running xkbcomp and redirecting the error output to $out
- Due to the quirk with the exit code of xkbcomp described above, we ignore it completely and always succeed with || true
- After running this, compilationOutputFile points to a file that should either be empty if the layout is valid, or contain the error message that xkbcomp produced
We then read the file back into the variable compilationOutput (4)
And finally the check (5), if compilationOutput is empty nothing happens, but otherwise we abort with a friendly message that contains the output from xkbcomp
After all that, we can construct the new layout definition (6), safe in the knowledge that if we got this far, the layout is definitely valid

With buildLayout in hand, we can set our now verified layout:

services.xserver.xkb.extraLayouts = buildLayout {
  name = "test";
  description = "Test";
  lang = "eng";
  symbols = ./test.xkb;
};

Done, the layout building and verification is fully automated. We can try building this new module with the following command⁴:

nix build --impure --expr 'let pkgs = import <nixpkgs> {}; in (import ./with-validation1.nix { inherit pkgs; }).services.xserver.xkb.extraLayouts'

Unfortunately, this never fails. Whether or not the XKB file is valid or not doesn't change a thing. The build always passes. Can you see the issue?

It took me a bit to spot it, but Nix is a lazy language. So unless an expression is explicitly used for anything, it's never evaluated⁵.

In the new flow I haphazardly mixed and matched the validation logic from one place with layout setting from another. In the outdated documentation the result of compilation was actually referenced in an expression that is used by NixOS, transitively forcing the evaluation of the compilation command⁶.

Notice that in the command I'm running above, I explicitly reference extraLayouts, this will force the evaluation of the buildLayout call. Within buildLayout itself, the compilationSuccess value is defined in a let but it's never actually used in the output of the let expression. That is to say, it is not referenced in the final output that we produce. As a result, under lazy evaluation, we don't have to evaluate compilationSuccess to compute the output of buildLayout. In effect, we never actually run the check we so lovingly crafted.

I have no qualms with laziness in the Nix language in general, can't really imagine the flexibility of nixpkgs without it, but I really do want that check to be executed. So we need to chain it to the final output.

Like so:

let
  # ... same as before
  compilationSuccess = compilationOutput == ""; # 1
in {
  ${name} =
    if compilationSuccess # 2
    then {
      description = description;
      languages = [lang];
      symbolsFile = symbols;
    }
    # 3
    else abort "Failure compiling layout [${name}]: ${compilationOutput}";
};

Most of the code remains as is, but we now modify compilationSuccess to only check the condition without aborting (1). We then branch on the result of compilationSuccess (2), and abort if it's false (3). As a result compilationSuccess and transitively the xkbcomp call, is chained to the final output.

And this works as expected, if we build it with a broken layout:

➜ nix build --impure --expr 'let pkgs = import <nixpkgs> {}; in (import ./with-validation2.nix { inherit pkgs; }).services.xserver.xkb.extraLayouts'
error:
       … while evaluating the attribute 'test'
         at with-validation2.nix:22:5:
           21|   in {
           22|     ${name} =
             |     ^
           23|       if compilationSuccess

       … while calling the 'abort' builtin
         at with-validation2.nix:29:12:
           28|       }
           29|       else abort "Failure compiling layout [${name}]: ${compilationOutput}";
             |            ^
           30|   };

       error: evaluation aborted with the following error message: 'Failure compiling layout [test]: syntax error: line 5 of /nix/store/ilwgm7nihyzy30avldk7nr2wim9vwj9d-test.xkb
       last scanned symbol is: CAPS
       Errors encountered in /nix/store/ilwgm7nihyzy30avldk7nr2wim9vwj9d-test.xkb; not compiled.
       '

A bit verbose, but the build fails as required, and the compilation error from xkbcomp is printed along the way.

Since doing an if/else and abort every time I want to validate something feels a bit noisy, we can use utility functions to prettify it. There are a few ways to do this, like the builtin assert special form, maybe paired with the slightly more friendly assertMsg. But I think that the syntax for throwIfNot is a bit cleaner. Here's the final version⁷:

let
  # ... same as before
  compilationSuccess = compilationOutput == "";
  # 1
  errorMessage = "Failure compiling layout [${name}]: ${compilationOutput}";
  # 2
  ifCompilationSuccess = lib.throwIfNot compilationSuccess errorMessage;
in {
  # 3
  ${name} = ifCompilationSuccess {
    description = description;
    languages = [lang];
    symbolsFile = symbols;
  };
};

To prettify things we extracted the error message into a variable (1), now laziness is on our side, it won't be evaluated unless actually used. We then create a new function called ifCompilationSuccess (2). It uses throwIfNot to check the compilationSuccess value and will throw the errorMessage if it's false. throwIfNot returns the identity function if the condition is true, otherwise it throws an error. Lastly, we apply ifCompilationSuccess to the result of buildLayout (3), and by this we chained the compilation check into the flow, and kept the noise to a minimum.

This works as before, although the error message looks a bit different due to the difference in formatting between abort and throw.

We are now done. We can easily create and modify XKB layouts without worrying about breaking our system. Any errors will be caught at build time.

If you're feeling particularly masochistic, you can read an in depth tutorial on the topic. ↩
The example code can be found on GitHub. ↩
I'll silently ignore the question of how often I actually modify keyboard layouts. ↩
The full file can be found here. ↩
Shame on me, having spent a semester TAing on a Haskell course, I shouldn't be surprised by laziness. ↩
Actually, in the old documentation it's not clear to me how it could ever work in the first place. Due to the aforementioned exit code quirk of xkbcomp any build that calls it without error capturing should fail. Maybe it's more recent behavior. ↩
The full file. ↩

Random Scala Tip #534: Adopt an Error Handling Convention for `Future`

Daniel Beskin — Mon, 08 Sep 2025 00:00:00 GMT

We seem to be living in the future. Surrounded by an ever increasing number of advanced effect-systems, capability trackers, and even salads¹. Yet I'm sure that many of us, in an effort to bring food on the table, are still stuck in the past, using Future.

There are quite a few things to say about Scala's default Future implementation, some of them good, some not so much, but I'll focus on one thing: error handling. For better or for worse², errors in Future are not tracked by the compiler. That is to say, it's impossible to statically distinguish between a potentially failing Future instance, and another instance that cannot fail³. More concretely in code:

def doStuff(): Stuff
val defaultStuff: Stuff

def foo() = Future(doStuff())

def bar() =
  Future(doStuff()).recover:
    case NonFatal(ex) => defaultStuff

val x = foo()
val y = bar()

The types of x and y are both Future[Stuff], despite the fact that bar handles exceptions, and "cannot fail". There is no way to reflect that difference between x and y with types. This leads to some unfortunate consequences.

With this in mind, and without any help from the compiler, we must live in constant fear of exceptions. Can an exception lurk in this Future or that⁴?.. What should we do about it? Should we do anything? Every developer touching the codebase might have different answers to these questions which can lead to a host of different surprises.

This is no way to maintain an existence, so in lieu of the type system's support, here is today's tip:

When working with Future be explicit about your error handling convention, and make sure your whole team and codebase are aligned to it.

If you adopt this tip, some of the anxiety from what I wrote above might dissipate.

Lest this stays completely vague, let's discuss some concrete error handling conventions we can use.

YOLO

As the name of this convention implies, here we don't worry much about anything, and about Future errors in particular. You use exceptions in Future as you see fit, and you never bother handling them in any specific place.

Pros: There's only one answer to all of the questions above, and it requires zero effort or discipline.

Cons: Let me know if you find any.

Full On Defensive

In this convention we move to the other end of the worrying scale: every instance of Future is suspect, we trust no one but ourselves, and we call recover (and maybe add some logging for good measure) every time we stumble on Future in the code.

Pros: Yet again, we have only one answer to all the questions above, it's always recover.

Cons: This gets old fast, requires discipline, the code gets cluttered with error handling, and letting loose your paranoia is not likely to get you anywhere productive anyways.

Only Defects in Error Channel

This convention is more subtle⁵: we decree that we only use Future's error channel for defects in code. I.e., errors that we didn't predict during development. As a rule, we don't handle defects till the "top-level" of the app, where we might recover from the defect with some generic logging⁶. Defects are not the norm, and when we discover one, we try to eliminate it from our code.

Now what I mean by the "top-level" is context dependent. It might be the actual top-level of the application, somewhere in the "main" of the code. But more typically that would be "per request". So, for example, if you're writing a web server, you will handle errors at the top-level of the request. We wouldn't want to crash the whole server just for one failing request. Similarly, if you're writing some kind of message handler (like with streams, or actors), you would handle errors once per message.

If the Future's error channel is only used for defects, where do we put all other errors? Somewhere where the compiler can track them, of course. So every time we intentionally want to raise and handle an error we will reflect it with a dedicated type like Either or one of a panoply of Validation types that are available in the Scala ecosystem⁷. As a result, when working with code that can fail and be meaningfully recovered, we will have to deal with something like Future[Either[MyError, Stuff]]. A bit verbose though it may be, this makes it very explicit when we have an error to deal with, versus when all errors have already been dealt with (when we end up with Future[Stuff] again). And we are letting the compiler guide us in the right direction in case we forget anything.

Pros: Most of the time⁶ we know how to answer all of the questions about error-handling, and the compiler is there to help us, so that we don't have to rely on discipline too much.

Cons: Ergonomics, working with nested types can be a bit cumbersome. And if performance is at stake, the extra layer of indirection might be a price that we don't want to pay.

Possible mitigation: If we settle for some specific type for error representation (like Either), we can add some utilities and convenience syntax for the nested Future[Either[_, _]] type, like support for mapping/flat-mapping the inner value (or use an existing library solution like EitherT⁸).

Others

I'm sure there are many other viable error handling conventions⁹. The two main criteria I would judge them by:

How much discipline they require? The less, the better.
How well can you answer questions about error handling at compile-time? The less thought it requires, the better.

If you have other Future error handling conventions that you find useful, I'll be glad to hear about them in the comments.

An Italian one. ↩
Worse. ↩
For the purposes of this discussion I'll be ignoring any fatal errors, like out-of-memory, and only consider "user space" errors, like IO exceptions and the like. ↩
I'll ignore the fact that Scala has regular runtime exceptions as well. I think it's pretty conventional for Scala developers to eschew the explicit use of throw. As a result most of the time I don't feel much of a need to get all defensive about it. On the other hand, since Future is a datatype, people seem to feel more comfortable to use Future's error channel, just as you would with Try. I don't harp about Try because it's not as contagious as Future, and its foremost purpose is error handling, so I expect people to be sufficiently careful around it. Future's main purpose in not about errors, so it's easier to forget about them. ↩
The terminology is inspired by the error channels in the ZIO library. ↩
We might have to break the convention in various rare cases. Like when interfacing with Future-based code that we don't control. In such cases we will hit recover as soon as possible, and then continue pretending that our convention was never broken (kind of like you would with null handling when interfacing with Java). ↩ ↩²
Or Try, but it feels kind of strange to work with Future[Try[?]], since Future already conceptually contains a Try, and you cannot define custom errors for it that are reflected in the type. ↩
Thanks u/pizardwenis96 from Reddit. ↩
Like adapting something like the capability-tracked "submarine" exception from Cats Effect. It's probably just as applicable to Future as it is to Cats Effect. Especially once capture-checking and co. gain some more mainstream adoption. Also, see this discussion on Reddit. ↩

Make Illegal AI Edits Unrepresentable

Daniel Beskin — Sun, 24 Aug 2025 00:00:00 GMT

This post is also available in video form:

These are strange times we're living in. We have unleashed an army of junior developers on our codebases while changing little else, yet we expect everything to just be okay.

Granted, they aren't exactly junior developers; they are both smarter and dumber, all at once. But it doesn't change the reality that the cumulative effect of running all sorts of AIs in our codebases, with little to no supervision, can, and likely will, lead to ever increasing complexity and maintainability woes.

"No, no, I'm doing stringent code review on every line of code the AI generates", sure you do, buddy. You're running five agents, all at once, and then expect that you'll be able to thoroughly review every line of an ever growing volume of generated code?

Imagine a well-functioning team of five senior developers, working on a large codebase. The complexity is there, slowly growing, but they manage to wrangle it, somehow. Then they bring on some young blood, a brand new junior developer, fresh off the boat from college. The codebase is large and complex, unsurprisingly, the junior makes mistakes. But it's okay, the seniors are there, at least one of them is always available to guide, review, and clean up any mess that comes up. After a while, the junior is no longer a junior, but a young senior. Soon we'll have a team of six seniors, and we can bring in some more fresh blood. All is as it should be.

Next, imagine the same scenario, but now we bring in 25 juniors in one batch. If you think that it's going to work out fine, you're most likely delusional. The seniors will have no way to keep up with the juniors, and the code will end up in shambles.

More generally, when many people work on the same system all at once, be it seniors, juniors, or anything in-between, code quality tends to quickly plummet. It appears that humans, juniors and seniors alike, are pretty bad at taming complexity. They have trouble keeping track of enough context to see the consequences of small changes rippling across code and time.

Unfortunately, current generation AIs suffer from the same problem. The only difference is that the number of AIs working on code seems to explode these days, overwhelming whatever benefit that experienced developers can contribute to the process.

In the long run, if we want to be productive using AIs to develop code, we must use better ways to handle complexity.

Even before the advent of widespread AI tools, I spent most of my professional career thinking about the problem of keeping code complexity at bay in the presence of humans. And now I do the same in the presence of an army of AIs.

One of the most useful tools for the purpose of complexity management that I know of is the design principle coined by Yaron Minsky that says:

Make illegal states unrepresentable

And that's what I want to talk about now. To put this into today's context, we'll paraphrase a little bit. Our goal will be to:

Make illegal AI edits unrepresentable

The Pitch

I won't delve into explaining what "illegal states" or "representability" are, hopefully most of you have some intuition about them¹. What I want to highlight though, is that "making illegal states unrepresentable" is an excellent complement to the weaknesses of both humans and AIs that I described above.

The way one would typically enforce the unrepresentability of illegal states is at compile-time, using the type system. Unlike your average AI, the compiler is enforcing type constraints by taking your whole program as context. It doesn't "forget" things just because they are too far away in code or time. And while the wet and imprecise brain of an AI can miss things, the cold and calculating brain of a compiler never forgets anything you ever taught it.

The biggest takeaway from this is that the more rules and invariants we encode into the type system, the more helpful the compiler will be in enforcing those invariants. Deterministically and without fail. So if some illegal state needs to be avoided globally and forever, make it unrepresentable at compile-time, and whether you or your AI remember about it while editing code won't matter in the least.

Let's illustrate this with a concrete example.

The Illegal States

Let's pretend we are programming a coffee-making robot². Something like this:

There are many subsystems involved in operating such a thing. We'll focus on the coffee-ordering part. Since we are just a small startup, we start with a small proof of concept. For now, we support only one type of drink.

Apparently, cappuccino is very popular these days, so that's a good starting point for us. To order cappuccino the user needs to tell us what kind of milk should be used. We'll model the order with a record class³:

record CoffeeOrder(
    @Nonnull Milk milk) {}

It appears that choosing milk nowadays is a non-trivial task, so Milk is actually an enum with multiple choices. Notice the Nonnull annotation on the Milk field. I hear that nulls can be dangerous, so we are very explicit about how we handle them. Every cappuccino must have Milk specified.

After successfully making a cappuccino, we want to expand our repertoire. So might as well make a plain espresso, we already have all the ingredients to make it. Espresso is just cappuccino minus the milk:

record CoffeeOrder(
    @Nullable Milk milk) {}

Now milk is a nullable field. If the value is present we make a cappuccino, if not, we make an espresso.

We got the ball rolling, so let's make another drink: latte. It's just like cappuccino, but with more milk. So both latte and cappuccino must have Milk specified. The presence or absence of the milk field is no longer enough to indicate which type of drink we need. We'll have to disambiguate the drinks somehow:

enum DrinkType {
  CAPPUCCINO,
  ESPRESSO,
  LATTE,
}

The DrinkType enum will indicate which drink the user ordered. It can be used like so:

record CoffeeOrder(
    @Nonnull DrinkType drinkType,
    @Nullable Milk milk) {}

Now we can tell which type of drink to prepare, even when the milk field is present. We are still careful with nulls: drinkType is mandatory. Otherwise, we won't be able to tell which drink the user actually ordered. This is a nice way to future-proof ourselves so that we can handle other drinks as well.

With support for some basic drinks in place, we want to show how fancy our robot is. Let's add a couple more special drinks: affogato and Irish Coffee.

The new drinks require more ingredients, so we add them to the CoffeeOrder record:

record CoffeeOrder(
    @Nonnull DrinkType drinkType,
    @Nullable Milk milk,
    @Nullable Gelato gelato,
    @Nullable Cream cream,
    @Nullable Whiskey whiskey) {}

Here we have gelato for the affogato, and cream and whiskey for the Irish coffee. The new fields are all nullable, as they are only relevant for these specific drinks.

We also update DrinkType to support the new drinks:

enum DrinkType {
  CAPPUCCINO,
  ESPRESSO,
  LATTE,
  AFFOGATO,
  IRISH_COFFEE,
}

If we take a step back and look at what we have here, we'll see that although we are only supporting five drinks, this is already quite the mess. The CoffeeOrder record is definitely capable of representing all the drinks we want it to. But along with that it's capable of representing nonsensical drinks. Some of them, like espresso with gelato are just redundant. But others make no sense at all. Like Irish coffee without whiskey, which is an illegal state if I ever seen one. And so, we made illegal states representable in our system.

This example might look a bit contrived, who ever just adds many fields like this to a class all at once? But this is based on actual experience, where a team of developers keeps adding fields over time, without immediately noticing the implications. And before you know it, you end up with a monster full of potential illegal states.

All is not lost though, we are conscientious developers. We did our best with the nullability annotations. We can also add our knowledge about the different invariants to the code as well. Here's the final version of CoffeeOrder:

/**
 * Invariants:
 * - For cappuccino: milk must not be null, all other nullables should be null
 * - For espresso: all nullable field should be null
 * - For latte: milk must not be null, all other nullables should be null
 * - For affogato: gelato must not be null, all other nullables should be null
 * - For Irish coffee: cream and whiskey must not be null, all other nullables should be null
 */
record CoffeeOrder(
    @Nonnull DrinkType drinkType,
    @Nullable Milk milk,
    @Nullable Gelato gelato,
    @Nullable Cream cream,
    @Nullable Whiskey whiskey) {
}

This comment is very thorough, now anyone who forgets about the invariants has a complete reference. Is this good enough?

Getting in the Flow

How do we actually use CoffeeOrder? We are in the business of taking orders and operating robots. So here's the pipeline that we'll be implementing:

In this flow we take in a new CoffeeOrderDTO, which represents the raw data we got from the user. It's defined like so:

record CoffeeOrderDTO(
    @Nullable DrinkType drinkType,
    @Nullable Milk milk,
    @Nullable Gelato gelato,
    @Nullable Cream cream,
    @Nullable Whiskey whiskey) {}

Basically the same thing as CoffeeOrder, but now everything is nullable and we don't assume any invariants about it. We have to validate CoffeeOrderDTO before we can convert it into a proper CoffeeOrder. This is where OrderProcessor comes into play. The process method takes in a CoffeeOrderDTO and converts it into a CoffeeOrder. Here we'll be careful to maintain all the CoffeeOrder invariants that we care about. With a CoffeeOrder ready we send it to CoffeeMachine, which will produce the instructions to the robot according to the CoffeeOrder we have.

To keep things contained, we'll only focus on a single drink. And so we'll just look at the code that is responsible for Irish coffee.

Let's start with OrderProcessor:

public class OrderProcessor {

  public CoffeeOrder process(@Nonnull CoffeeOrderDTO order) { // 1
    var drinkType = // 2
      order.drinkType() == null ?
        DrinkType.CAPPUCCINO : order.drinkType();

    return switch (drinkType) { // 3
      ...
      case IRISH_COFFEE -> processIrishCoffee(order); // 4
    };
  }

Here we take the input CoffeeOrderDTO (1). As a first step we are trying to determine the DrinkType. Since it cannot be missing, we provide a default value (cappuccino) in case it's null (2). We then switch on the DrinkType (3). In case it's IRISH_COFFEE we process the order using the processIrishCoffee method (4). Which is defined as follows:

private CoffeeOrder processIrishCoffee(CoffeeOrderDTO order) {
    return new CoffeeOrder( // 1
        DrinkType.IRISH_COFFEE, // 2
        null,
        null,
        order.cream() == null ? Cream.HALF_AND_HALF : order.cream(), // 3
        order.whiskey() == null ? Whiskey.JAMESON : order.whiskey()); // 4
}

We finally create a CoffeeOrder (1) of type IRISH_COFFEE (2). At this point we have to populate its fields according to the invariants that we defined for Irish coffee. So we extract the Cream value, and provide a fallback in case it's missing (3). And do the same for the Whiskey value (4). The rest is not relevant for Irish coffee, so we set it to null.

With this code in place, we kept all the invariants that are dictated by CoffeeOrder. Illegal states are nowhere in sight. We could even write a unit test to make sure this is so.

The next step is to program the CoffeeMachine.

Suppose we have a method that tells the robot to prepare the Irish coffee. Given the user's choice of cream and whiskey it instructs the robot to prepare Irish coffee:

private void prepareIrishCoffee(
  @Nonnull Cream cream,
  @Nonnull Whiskey whiskey)

Because Irish coffee makes no sense without cream and whiskey, the method requires the cream and whiskey arguments to be non-null. This reflects the rationale for the invariants in CoffeeOrder.

We'll use the prepareIrishCoffee method in CoffeeMachine.prepare when converting a CoffeeOrder into instructions for the robot:

public class CoffeeMachine {
  public void prepare(CoffeeOrder order) {
    switch (order.drinkType()) { // 1
      ...
      case IRISH_COFFEE -> { // 2
        prepareIrishCoffee( // 3
            order.cream(),
            order.whiskey());
      }
    }
  }
}

We switch on the order's DrinkType (1) and in the case where the drink type is IRISH_COFFEE (2) we extract the cream and whiskey values and pass them on (3). That's okay, right? We have all those invariants in place (here's to good documentation) because it makes no sense for those values to be empty at this stage. OrderProcessor made sure of this.

Unfortunately, this doesn't compile:

CoffeeMachine.java:23: error: [NullAway] passing @Nullable parameter 'order.cream()' where @NonNull is required
            order.cream(),
                       ^
CoffeeMachine.java:24: error: [NullAway] passing @Nullable parameter 'order.whiskey()' where @NonNull is required
            order.whiskey());
                         ^
2 errors

Ah, we just specified that prepareIrishCoffee takes non-null values, and despite our invariants, nothing guarantees to the compiler that the values are actually not null. But we know that everything is fine here. Let's insist on that:

public class CoffeeMachine {
  public void prepare(CoffeeOrder order) {
    switch (order.drinkType()) {
      ...
      case IRISH_COFFEE -> {
        // This is SAFE // 1
        // Irish coffee ALWAYS has cream and whiskey
        if (order.cream() == null) // 2
          throw new IllegalStateException("No cream");
        if (order.whiskey() == null)
          throw new IllegalStateException("No whiskey");

        prepareIrishCoffee(
            order.cream(),
            order.whiskey());
      }
    }
  }
}

We added a defensive null check to satisfy the compiler (2), and we throw an exception if the invariant is broken. But we know it's safe, and just to be extra careful we leave an emphatic comment for future reference about just how impossible it is for the values to be missing here (1). With the null check in place this code now compiles.

Our pipeline is now complete, all invariants are kept. We can even run it to make sure it works:

// 1
var processor = new OrderProcessor();
var coffeeMaker = new CoffeeMachine();

// 2
var dto1 = new CoffeeOrderDTO(
    DrinkType.IRISH_COFFEE, null, null, Cream.HEAVY, Whiskey.TULLAMORE);

// 3
var dto2 = new CoffeeOrderDTO(
    DrinkType.IRISH_COFFEE, null, null, Cream.HEAVY, null);

// 4
var order1 = processor.process(dto1);
var order2 = processor.process(dto2);

// 5
System.out.println(order1);
System.out.println(order2);

// 6
coffeeMaker.prepare(order1);
coffeeMaker.prepare(order2);

In this simple script we initialize the different components of the flow (1). We then create a couple of CoffeeOrderDTOs. The first is for Irish coffee, with all the relevant data present (2). The second is missing a whiskey value (3). Next we process both orders and generate CoffeeOrder instances (4). We print those values so that we can see they are okay (5). Lastly, we prepare the orders (6).

Running this produces the following output:

CoffeeOrder[drinkType=IRISH_COFFEE, milk=null, gelato=null, cream=HEAVY, whiskey=TULLAMORE]
CoffeeOrder[drinkType=IRISH_COFFEE, milk=null, gelato=null, cream=HEAVY, whiskey=JAMESON] // 1
Making Irish coffee with: HEAVY cream and TULLAMORE whiskey
Making Irish coffee with: HEAVY cream and JAMESON whiskey // 2

We can see that the missing whiskey value in the second order was filled in with the default value (1), and the robot happily fulfilled the order (2).

Everything is fine, no invariants were broken, no illegal states encountered.

This is our baseline, now let's see how things break over time...

AI vs. Illegal State

Time passes by, everything works correctly, and you forget all about this code. But then, a new requirement comes in. Recall how we used a hardcoded Whiskey.JAMESON value when the whiskey value is missing from the order. Well, we noticed that the whiskey stock in the coffee shop keeps fluctuating, and it's possible that Jameson is out of stock today. We no longer want to hardcode this value, instead we'll query a dedicated service to obtain the default value for today.

For this we'll have a new service definition:

interface WhiskeyService {

  @Nullable
  public Whiskey fetchTodayDefaultWhiskey();
}

It's just a simple fetcher. Since things can go wrong with fetching, the output is nullable, let our clients deal with that. We want to use an instance of this service to fetch the value instead of the hardcoded value we currently have.

Additionally, there's an implementation of WhiskeyService called WhiskeyServiceDefault:

public class WhiskeyServiceDefault implements WhiskeyService {

  @Nullable
  @Override
  public Whiskey fetchTodayDefaultWhiskey() {
    var rnd = new Random();
    var whiskies = Whiskey.values(); // 1

    var whiskey = whiskies[rnd.nextInt(whiskies.length)]; // 2

    return whiskey; // 3
  }
}

For purposes of demonstration, it just returns a random value each time it's called. In (1) we obtain an array of all available whiskies. Then we choose a random whiskey from all the possible whiskey values (2), and return it (3).

Since we're living in the future, we won't be implementing the new feature ourselves, manually, like some plebs. Instead, we'll use our handy AI agent⁴.

Here's the prompt for this task:

> In @OrderProcessor.java we no longer want to hardcode the
default `Whiskey` value since we don't know whether it's
available today or not. Use @WhiskeyService.java to inject
the value dynamically instead. In @Main.java use
@WhiskeyServiceDefault.java. Use `gradlew build` to verify
that everything compiles.

Nothing fancy here. The AI performs the relevant edits, without breaking a sweat. Here's the summary of what it's done:

● Done! I've successfully:

1. Modified OrderProcessor.java:25 to use
   whiskeyService.fetchTodayDefaultWhiskey() instead of
   hardcoded Whiskey.JAMESON
2. Added WhiskeyService dependency injection to
   OrderProcessor constructor
3. Updated Main.java:5-6 to create WhiskeyServiceDefault
   instance and inject it into OrderProcessor
4. Verified compilation with gradlew build

The new code ⁵ looks very reasonable, with all the modern bells and whistles of dependency injection, and whatnot. Here's the new code in processIrishCoffee:

private CoffeeOrder processIrishCoffee(CoffeeOrderDTO order) {
  return new CoffeeOrder(
      DrinkType.IRISH_COFFEE,
      null,
      null,
      order.cream() == null ? Cream.HALF_AND_HALF : order.cream(),
      order.whiskey() == null ? whiskeyService.fetchTodayDefaultWhiskey() : order.whiskey()); // 1
}

As we can see in (1) the default whiskey is no longer hardcoded, we delegate to WhiskeyService instead. Cool, this fulfills the requirements as we stated them.

We can run the example code from before:

CoffeeOrder[drinkType=IRISH_COFFEE, milk=null, gelato=null, cream=HEAVY, whiskey=TULLAMORE]
CoffeeOrder[drinkType=IRISH_COFFEE, milk=null, gelato=null, cream=HEAVY, whiskey=BUSHMILLS] // 1
Making Irish coffee with: HEAVY cream and TULLAMORE whiskey
Making Irish coffee with: HEAVY cream and BUSHMILLS whiskey

Since the default whiskey is no longer hardcoded, every time we run the code we'll get a different whiskey value. This time it was BUSHMILLS (1).

... cue in ominous music...

The AI fell for the trap I set for it. Can you see it?

As it currently stands the code doesn't exhibit any bugs. But after this simple edit a bug is latent in the codebase. Neither the AI, nor the human, or even the test suite caught anything.

We move on with our life, busying ourselves with other parts of the codebase. Until one day we get another feature request.

It appears the stock of whiskeys fluctuates so rapidly that we still sometimes try to serve whiskey that's out of stock, even without hardcoding the value. To fix this in the WhiskeyService implementation we are asked to check the current stock and avoid choosing a whiskey that is out of stock.

To do this we have another service we can query:

interface WhiskeyStockService {
  // invariant: all Whiskey types are present in the resulting Map // 1
  @Nonnull
  public Map<Whiskey, Integer> fetchTodayStock(); // 2
}

WhiskeyStockService.fetchTodayStock gives us a snapshot of the current stock (2). There's also a helpful comment about an invariant here, which we'll trust blindly. Now we can use WhiskeyStockService to implement the new requirement (for demonstration purposes WhiskeyStockServiceDefault will return random values for the stock).

Let's prompt the AI:

> In @WhiskeyServiceDefault.java it's possible that we used a
whiskey that's not in stock today. Use
@WhiskeyStockService.java to only return the chosen whiskey
if it's in stock at the moment. In @Main.java use
@WhiskeyStockServiceDefault.java. Use `gradlew build` to
verify that everything compiles.

The AI chugs along. It stumbles for a moment because the compiler was not impressed by the invariant that was stated in the comment in WhiskeyStockService. But finally we get back the results:

● Done! I've successfully updated the code to check whiskey
stock availability:

1. WhiskeyServiceDefault.java now uses WhiskeyStockService
   to validate stock before returning a whiskey
2. Main.java now creates and uses WhiskeyStockServiceDefault
3. The build compiles successfully with gradlew build

The implementation now only returns whiskeys that are in
stock (stock level > 0), and returns `null` if no whiskeys
are available.

The new WhiskeyServiceDefault:

@Nullable
@Override
public Whiskey fetchTodayDefaultWhiskey() {
  var rnd = new Random();
  var whiskies = Whiskey.values();
  var stock = stockService.fetchTodayStock();

  for (int attempts = 0; attempts < whiskies.length; attempts++) {
      var whiskey = whiskies[rnd.nextInt(whiskies.length)];
      var stockLevel = stock.get(whiskey);
      if (stockLevel != null && stockLevel > 0) {
          return whiskey;
        }
    }

  return null; // 1
}

Let's ignore the question of whether this fulfills the requirements; vague human language sure tends to produce vague results. What's important is that this code now exhibits a bug. Despite the @Nullable annotation, the original implementation of fetchTodayDefaultWhiskey would never return a null value in practice. With the new implementation this is no longer the case. The code can, and depending on today's stock, will eventually return null (1).

Why is this a problem? Recall that after our first round of edits we had this code:

return new CoffeeOrder(
    DrinkType.IRISH_COFFEE,
    null,
    null,
    order.cream() == null ? Cream.HALF_AND_HALF : order.cream(),
    order.whiskey() == null ? whiskeyService.fetchTodayDefaultWhiskey() : order.whiskey());

We blindly take whatever fetchTodayDefaultWhiskey returns and pass it along. The null checking that the compiler is doing for us doesn't help us, since whiskey is a nullable field. It shouldn't be null only in specific circumstances (DrinkType == IRISH_COFFEE) which the compiler is incapable of checking.

For bonus points, this bug in intermittent, it only happens in specific circumstances, good luck debugging it when it hits production.

If we run our test script, most of the times everything will be fine:

CoffeeOrder[drinkType=IRISH_COFFEE, milk=null, gelato=null, cream=HEAVY, whiskey=TULLAMORE]
CoffeeOrder[drinkType=IRISH_COFFEE, milk=null, gelato=null, cream=HEAVY, whiskey=BUSHMILLS]
Making Irish coffee with: HEAVY cream and TULLAMORE whiskey
Making Irish coffee with: HEAVY cream and BUSHMILLS whiskey

But some times, when we hit the right conditions (all chosen whiskeys are out of stock), we get an exception:

CoffeeOrder[drinkType=IRISH_COFFEE, milk=null, gelato=null, cream=HEAVY, whiskey=TULLAMORE]
CoffeeOrder[drinkType=IRISH_COFFEE, milk=null, gelato=null, cream=HEAVY, whiskey=null] // 1
Making Irish coffee with: HEAVY cream and TULLAMORE whiskey
Exception in thread "main" java.lang.IllegalStateException: No whiskey
        at CoffeeMachine.prepare(CoffeeMachine.java:20)
        at Main.main(Main.java:23)

We can see in (1) that we managed to get null for the whiskey field, and so the null checks we were forced to place in CoffeeMachine throw an exception.

Remember that emphatic comment we had above?

// This is SAFE
// Irish coffee ALWAYS has cream and whiskey

So much for the efficacy of comments...

The Root Cause

What went wrong? The root cause of this is that we had an important invariant that was not enforced at compile-time. It was only encoded as textual knowledge (comments), not anything that is mechanically enforceable. We made an illegal state representable in our system, and given enough code and time, we eventually managed to hit it.

It's no wonder that simple humans fall for such things all the time. The problem is that current AIs are prone to falling for this just the same. Both humans and AIs use a local context of some random size when writing code. And whether or not they will notice this or that global invariant being violated is given up to chance⁶.

"But", you might object, "I can cover the invariant with tests, and those tests will keep the AI in check". Alas, that's not good enough either. You can definitely cover all known relevant code paths with tests. But what about code paths that do not yet exist? How do you know what future code path you will create and how it interacts with the invariants you've set?

And to head off the obvious objection that your AI can handle this trap and not fall for it. Sure, it's a tiny codebase, with only a few files. I was even nice enough to spell out the invariant very explicitly in a comment⁷, in a way that an AI can definitely pick up. But this will only happen if the AI "sees" that comment. Given a sufficiently large codebase, with a sufficiently widespread invariant, your AI is bound to miss it as well.

The same applies for people that are more skilled at prompting than I am. Maybe you can prompt better, but nothing will save your prompt if you're not even aware of the danger you are trying to circumvent.

Whatever your objections are to the details of this conceptual example, you cannot change the fact that in general you will not be able to consistently enforce global invariants with tools that only apply to a limited local scope. Humans, current AIs, and unit tests, all fall under the category of "local" tools.

If an illegal state is representable in your system, you and your AI will eventually hit it. And those odds multiply as you run more and more agents generating ever increasing amounts of code.

The best way to not fall for it is to make the illegal state unrepresentable, and by so making the illegal AI edits we just seen unrepresentable as well.

The Unrepresentable

In a parallel universe, where developers know what's good for them and their sanity, every time they recognize an illegal state of the system, they make it unrepresentable. This is not always easy, and depending on what's at stake might not even be worth it. But for our example, the fix is fairly straightforward.

The problem starts with CoffeeOrder which is capable of representing illegal values. If we remodel CoffeeOrder to only allow legal combinations, the rest will follow. One of the best techniques to do that is to use a sum type. In Java this means using a sealed interface, where each admitted class represents one (legal) combination of values.

Let's rewrite CoffeeOrder⁸:

public sealed interface CoffeeOrder { // 1
  record Cappuccino(@Nonnull Milk milk) implements CoffeeOrder {} // 2

  record Espresso() implements CoffeeOrder {} // 3

  record Latte(@Nonnull Milk milk) implements CoffeeOrder {} // 4

  record Affogato(@Nonnull Gelato gelato) implements CoffeeOrder {} // 5

  record IrishCoffee( // 6
      @Nonnull Cream cream,
      @Nonnull Whiskey whiskey) implements CoffeeOrder {}
}

CoffeeOrder is now a sealed interface (1), with all the admitted classes nested in it. Each implementation of CoffeeOrder is one possible drink that we currently allow. Accordingly:

Cappuccino (2) has one Milk field, which is now enforced to be @Nonnull, as per the invariants we described before
Espresso (3) has no fields, so we no longer need to set anything to be nullable just for it
Latte (4) has a @Nonnull Milk as well, and we don't have any ambiguity with Cappuccino since it's a separate subtype
Affogato (5) has one mandatory Gelato field, which by virtue of being a separate class doesn't affect anything else
Lastly, IrishCoffee, our nemesis, explicitly declares its requirements with a pair of Cream and Whiskey mandatory fields

The CoffeeOrder datatype perfectly captures all the requirements that we stated before. Better yet, it does that at the type-level, where the compiler can enforce it. There's no longer a need for a comment nobody's going to read. CoffeeOrder is a precise specification of what is allowed in the system.

Whereas the original CoffeeOrder was "too big", allowing for a mismatch between its DrinkType and the nullable fields. The new CoffeeOrder subsumes DrinkType into its subclasses, perfectly aligning the required data with the drink being ordered.

Now we have to adapt the code to the new datatype, but this time we no longer have to worry about invariants, the compiler will nudge us in the right direction at every step.

We'll start with OrderProcessor, we only need to change processIrishCoffee:

private CoffeeOrder processIrishCoffee(CoffeeOrderDTO order) {
  return new CoffeeOrder.IrishCoffee( // 1
    order.cream() == null ? Cream.HALF_AND_HALF : order.cream(), // 2
    order.whiskey() == null ? Whiskey.JAMESON : order.whiskey());
}

The code looks pretty much the same, but now we initialize the specific subtype of CoffeeOrder that we need here, IrishCoffee (1). The cream and whiskey fields are initialized the same as in our baseline implementation (2). Unlike before we no longer have to provide the dummy null values for the fields that are irrelevant in this case. Because there aren't any, IrishCoffee is very precise.

What's invisible in this code is the fact that the arguments to IrishCoffee are not allowed to be null like they were before. We'll later see how this saves our code from bugs.

Next we'll adapt CoffeeMachine:

public void prepare(CoffeeOrder order) {
  switch (order) {
    ...
    case CoffeeOrder.IrishCoffee irishCoffee -> {
      prepareIrishCoffee(
          irishCoffee.cream(),
          irishCoffee.whiskey());
    }
  }
}

This code is the same as our very first attempt to write it, but this time the compiler won't complain. The fields of IrishCoffee cannot be null, and we can safely pass them to prepareIrishCoffee.

With these few small changes we managed to encode the invariants we care about globally, in a way that can be enforced mechanically and deterministically by the compiler. Will this code withstand the pressure of humans and AIs trying to break it?

AI vs. The Unrepresentable

Let's repeat the same prompt that we tried before and see how the AI manages now. Recall the prompt:

> In @OrderProcessor.java we no longer want to hardcode the
default `Whiskey` value since we don't know whether it's
available today or not. Use @WhiskeyService.java to inject
the value dynamically instead. In Main.java use
@WhiskeyServiceDefault.java. Use `gradlew build` to verify
that everything compiles.

The AI happily chugs along trying to make the same modification it did before:

private CoffeeOrder processIrishCoffee(CoffeeOrderDTO order) {
  return new CoffeeOrder.IrishCoffee(
      order.cream() == null ? Cream.HALF_AND_HALF : order.cream(),
      order.whiskey() == null ? Whiskey.JAMESON : order.whiskey());
      order.whiskey() == null ? whiskeyService.fetchTodayDefaultWhiskey() : order.whiskey());
}

But this fails to compile. The arguments to IrishCoffee cannot be null, but fetchTodayDefaultWhiskey is marked Nullable. The AI reacts with:

● I need to handle the null case from the WhiskeyService.
...
The issue is that fetchTodayDefaultWhiskey() can return
null, but the CoffeeOrder.IrishCoffee constructor expects a
non-null whiskey.

Which is great, and exactly the sort of process that making illegal states unrepresentable should trigger. The AI then suggests the following modification:

private CoffeeOrder processIrishCoffee(CoffeeOrderDTO order) {
  var defaultWhiskey = whiskeyService.fetchTodayDefaultWhiskey();

  // 1
  var whiskey = order.whiskey() != null ? order.whiskey() :
                defaultWhiskey != null ? defaultWhiskey : Whiskey.JAMESON;

  return new CoffeeOrder.IrishCoffee(
      order.cream() == null ? Cream.HALF_AND_HALF : order.cream(),
      order.whiskey() == null ? defaultWhiskey : order.whiskey());
      whiskey);
}

In (1) there is now a fallback in case WhiskeyService produced a null, which keeps the original hardcoded value.

Here the timeline splits. If we aren't carefully reviewing everything the AI is doing, which is getting more and more likely as people offload more code to agents, we might accept this solution. Despite the fact that it doesn't actually fulfill our requirements for not hardcoding values. Such is life. But, crucially, we didn't break any invariants.

On another timeline, we might've noticed that the requirements weren't fulfilled (or the prompt was more insistent about it), in which case we reject this edit. Likely or not, I'll continue on this timeline, just for the sake of seeing where this might lead:

> Never use a hardcoded value

Which produces the following code:

private CoffeeOrder processIrishCoffee(CoffeeOrderDTO order) {
  var whiskey = order.whiskey();

  if (whiskey == null) {
    whiskey = whiskeyService.fetchTodayDefaultWhiskey();
    if (whiskey == null) {
      throw new IllegalStateException("No whiskey available today"); // 1
    }
  }
  return new CoffeeOrder.IrishCoffee(
      order.cream() == null ? Cream.HALF_AND_HALF : order.cream(),
      whiskey);
}

This solution throws an exception when the fetched whiskey is null (1). Seeing how we went in this direction due to a runtime exception being thrown where we didn't expect it, we'll reject this solution as well⁹:

> No exceptions

To which we get this response:

● I understand. Let me think about this differently. Since
the WhiskeyService interface is @Nullable but we need a
non-null value, and you don't want hardcoded values or
exceptions, perhaps the solution is to modify the
WhiskeyServiceDefault to never return null, or change the
interface contract.

Looking at the current WhiskeyServiceDefault implementation,
it actually never returns null - it always returns a random
whiskey. Should I remove the @Nullable annotation from the
interface to reflect this reality?

Nope:

> It's impossible to guarantee that `fetcTodayDefaultWhiskey`
is never `null`

Now the AI goes on a tangent, trying to swap the order for a hardcoded cappuccino if the whiskey is missing. This might be acceptable to some, but not in this house:

> Changing the drink silently is not user friendly

● You're right. If whiskey isn't available, we shouldn't
silently change the drink type. Looking at this problem, I
think we need to modify the CoffeeOrder.IrishCoffee to
accept a @Nullable whiskey parameter, since that's the
reality of the business logic - sometimes whiskey might not
be available.

But no:

> It makes no sense to have Irish coffee without whiskey

● You're absolutely right. An Irish coffee without whiskey
isn't an Irish coffee. Let me look at this from a different
angle - perhaps we need to handle this at the order
processing level by not allowing Irish coffee orders when no
whiskey is available.

Looking at the current flow, when someone orders an Irish
coffee but no whiskey is specified in the order AND the
service returns null (no whiskey available today), we have
an impossible state to represent. // 1

Maybe the solution is to return an Optional<CoffeeOrder> or
similar from the process method, or have a different
approach to handle unavailable ingredients. What would you
prefer as the approach to handle this case where Irish
coffee is ordered but no whiskey is available?

First Make It Correct

Daniel Beskin — Tue, 10 Jun 2025 00:00:00 GMT

Way back in December, in the spirit of the times (it seems that everyone was either solving Advent of Code or looking for a job), like an elephant in a china shop, I found myself solving a leetcode-style problem.

Before I say anything else¹, I would like to preface that I hate leetcode problems with a passion, and not only because I suck at them. The software industry needs to rise as one and banish them from any interviewing process in existence, and leave them be for the masochists that actually enjoy solving them.

Having said that, for the right level of difficulty² I do get easily nerd sniped by leetcode problems. And that's how I found myself spending way too much time solving a simple problem. Lest this time be completely wasted, I tried to conjure up a lesson for "real code" that can be learned from this sordid ordeal.

And so, paraphrasing a famous saying:

First make it correct, then make it anything you want.

Let me illustrate this point with a leetcode problem. This is going to be a wild ride from recursive solutions to procedural loops, all with a touch of property-based testing. Grab yourselves a cup of liquid and settle in for a (very) long read.

The Problem

Here's the problem I was trying to solve:

Given a two dimensional board compute a list of coordinates that form a clockwise spiral from the top-left corner to the center.

For example, given a board of dimensions 4x3, the spiral's path looks like this³:

-------------------------------------
| -[1]→  | -[2]→  | -[3]→  |  [4]↓  |
-------------------------------------
| -[10]→ | -[11]→ |  [12]  |  [5]↓  |
-------------------------------------
| ↑[9]   | ←[8]-  | ←[7]-  | ←[6]-  |
-------------------------------------

And the list of coordinates is (using 0-based indexing from left-to-right, top-to-bottom):

(0,0), (1,0), (2,0), (3,0), (3,1), (3,2), (2,2), (1,2), (0,2), (0,1), (1,1), (2,1)

Before reading on, spend some time and think about how you would approach solving this⁴.

Obviously Correct Solution

As stated above, the first step when approaching any problem is to write a correct solution. I'd even go further, ideally, I would start with an "obviously correct" solution. A solution that so closely follows the structure of the problem that you can just see it works from reading it. No nasty edge cases or convoluted logic. It's difficult to precisely define what "obviously correct" means, this is yet another one of those cases of "you know it when you see it".

Having scratched my head some, I got myself a working solution. But it wasn't an "obviously correct" solution. It had inconvenient edge cases and just wasn't obvious. Luckily, the friend who nerd sniped me⁵ gave me a better insight into a possible solution to the problem.

Generating the correct coordinates is like peeling an orange with a knife. You shave off a row, rotate, and repeat until you're done.

Let's see this with a concrete example. Suppose you fill the board with the coordinates of each cell:

---------------------------------
| (0,0) | (1,0) | (2,0) | (3,0) |
---------------------------------
| (0,1) | (1,1) | (2,1) | (3,1) |
---------------------------------
| (0,2) | (1,2) | (2,2) | (3,2) |
---------------------------------

We can peel the top row and add it to the solution:

(0,0), (1,0), (2,0), (3,0)

We are then left with a smaller board:

---------------------------------
| (0,1) | (1,1) | (2,1) | (3,1) |
---------------------------------
| (0,2) | (1,2) | (2,2) | (3,2) |
---------------------------------

Which we rotate counter-clockwise:

-----------------
| (3,1) | (3,2) |
-----------------
| (2,1) | (2,2) |
-----------------
| (1,1) | (1,2) |
-----------------
| (0,1) | (0,2) |
-----------------

Peel the top row again, and add it to the solution:

(3,1), (3,2)

With the remaining board:

-----------------
| (2,1) | (2,2) |
-----------------
| (1,1) | (1,2) |
-----------------
| (0,1) | (0,2) |
-----------------

Which we can rotate again, and repeat the process until the board reaches size 0. This solves the problem neatly, nicely following the symmetry inherent to the problem, and leading us towards what might turn out to be an obviously correct solution. All that's left is to code it up.

The solution just described is inherently recursive. We repeat the same procedure over and over again on an input that shrinks on every step, until we reach a base case. Since functional languages are very well adapted to recursion, we'll be using a functional language for this post. Specifically the Scala programming language, as it lets us do the whole functional thing with ease, but will also let us go procedural later on.

First let's lay down some basic helpers. Since we don't want to get confused by the different pieces of data we'll need. Let's define some simple types:

case class Dims(w: Int, h: Int)
case class Point(x: Int, y: Int)

Dims represents the dimensions of a board, with a width w and a height h. Point represents a two-dimensional point located at the x and y coordinates.

With this out of the way and without further ado, here's the orange peeling solution in code⁶:

def traceSpiral(dims: Dims): List[Point] = // 1
  peel(buildCoordinates(dims)) // 2

def peel(board: List[List[Point]]): List[Point] = // 3
  board match // 4
    case Nil => Nil // 5
    case topRow :: otherRows => // 6
      topRow ++ peel(rotateCCW(otherRows)) // 7

Let's go through it step by step:

The entry point is traceSpiral (1), it takes in the dimensions of the board and produces a list of points
We then call a utility function buildCoordinates (2) that populates a list of lists with the coordinates of each board cell
We pass that along to the recursive peel function
The peel function (3) operates on the board points producing a single list of points
We do this by pattern matching on the current board (4)
If the board has size 0 (i.e., the empty list) we reached the base case and return an empty list (5)
If the board is non-empty, we extract its top row and the rest of the rows (6)
We return the results by prepending the top row we just extracted to the result of calling peel again with the counter-clockwise rotated leftover board (7)

This closely follows the algorithm we just described, and has very little extraneous detail. Pattern matching makes this safe and edge case free. You can check out the definitions of buildCoordinates and rotateCCW in the repo. But since they rely on high-level list functions like tabulate and transpose they too are edge case free, and are very easy to verify with a simple unit test⁷.

And so, this is as close to an obviously correct solution as I'll ever get⁸. But is it actually correct?

I mean, I can handwave obvious correctness as much as I want, but without some kind of more concrete proof, that's all it is, handwaving. I won't try to make an actual proof in the mathematical sense, but I can at least provide some tests.

Here's an easy one:

test("Traces the correct spiral"):
    val dims = Dims(w = 4, h = 3)
    val points = traceSpiral(dims)

    val expected =
      List(
        Point(0, 0),
        Point(1, 0),
        Point(2, 0),
        Point(3, 0),
        Point(3, 1),
        Point(3, 2),
        Point(2, 2),
        Point(1, 2),
        Point(0, 2),
        Point(0, 1),
        Point(1, 1),
        Point(2, 1),
      )

    assertEquals(points, expected)

This test codifies the example we used above. And if we run it our solution passes with flying colors.

"But what about edge cases", you may wonder. And I shall answer that life's too short to look for edge cases manually. Instead we can employ property-based testing to search for edge cases for us.

For that we need to specify some properties that should hold for any valid solution of the spiral problem. Having spent a few minutes thinking about it, here are some properties I came up with that can be used as sanity checks. For all possible board dimensions it should hold that:

The points of a solution must be within the bounds of the board
The number of points in a solution should be the same as the size of the area of the board
There should be no duplicate points in a solution

Let's make this specification executable using property-based testing⁹:

property("Sanity: all points in bounds"): // 1
  forAll(dimsGen): dims => // 2
    val points = traceSpiral(dims) // 3

    points.forall(dims.inside) ==== true // 4

This corresponds to property 1:

In (1) we define a new property
We use dimsGen, which is a generator of Dims instances (2), that will randomly generate different dimensions for us¹⁰
The property should hold for any board dimensions that we throw at it
With the randomly generated dimensions in hand we invoke the traceSpiral function (3)
We then assert that all the points in the generated solutions are within the bounds of the dimensions we are working with (4)

Here's property number 2:

property("Sanity: dimensions match number of points"):
  forAll(dimsGen): dims =>
    val points = traceSpiral(dims)

    points.size ==== dims.area // 1

This works similarly to the first property, we run for any instance of Dims and assert that the length of solution matches the area of the dimensions (1).

And for property number 3:

property("Sanity: no duplicate points"):
  forAll(dimsGen): dims =>
    val points = traceSpiral(dims)

    points.distinct ==== points // 1

Here we make sure that any solution that we produce does not contain duplicate points (1). Again, holding for any board size.

In aggregate, these three properties verify that any valid solution is a permutation of all the points on the board.

Actually, we could've written that as a single property, but splitting this up has the advantages that:

It's marginally easier to write these properties than writing a function that correctly generates all the points on the board. Thus reducing the chances of the property itself being broken.
Having separate properties makes it easier to debug if something in the solution breaks. As we can see which aspect of "being a valid permutation" fails to hold.

And now for the big benefit defining these tests as properties: the property-based machinery is going to run our code on many different dimensions of boards, thus very likely flushing out any edge cases lurking around in our code. This happens without us needing to actually come up with those edge cases ourselves (is a board of size 0 an edge case? 1x1? 2x1? who knows?..).

We can run the full suite of tests now:

+ Traces the correct spiral 0.0s
+ Sanity: all points in bounds 0.006s
+ Sanity: dimensions match number of points 0.007s
+ Sanity: no duplicate points 0.008s

We passed all the tests (denoted by +). Our solution looks correct indeed, and we can be fairly certain that we didn't miss any edge cases.

Of course these are just sanity checks that I came up with quickly, they are not specific to spirals, and they don't actually validate that the solution we got is correct. They only attempt to reject obviously invalid solutions. My claim is that in conjunction with the single unit test above, these properties are enough to give me high confidence that the "obviously correct" solution is actually correct¹¹. If you don't believe me, I challenge you to come up with a non-contrived solution that passes the unit test along with the sanity properties and isn't a valid solution to the spiral problem. Please let me know if you do.

In case you're not satisfied with these properties, it is clear that such a nicely symmetric and recursive problem can have other properties that actually capture its rich structure. If you're so inclined, you can see an additional property that I defined here that tries to capture some of that structure. Since defining this property was more difficult than actually solving the problem, I won't be showing it here. There's only so much verification I'm willing to do for a problem I'm not being paid to solve...

Anyways, I now feel comfortable stating that the obviously correct solution is indeed correct, and we can move on.

The Reference Solution

Before we actually do move on. Let's reap another benefit of using property-based testing.

We can now treat the solution that we just created as a reference solution, so that we can use it to check any other implementation that we come up with. We can define one more property: for all board dimensions, every solution to the problem must match the reference solution.

In code:

property("Matches reference"):
  forAll(dimsGen): dims =>
    val expected = Reference.traceSpiral(dims) // 1
    val actual = traceSpiralCandidate(dims) // 2

    actual ==== expected // 3

We are defining this property (for all board dimensions) as part of a suite that will be used by all future solution candidates. First generate the reference solution (1). Then generate the candidate solution (2). Lastly, compare the two to make sure they are equal (3).

Since we trust the reference solution we can mechanically verify that every other solution matches it. This is a very powerful test. Assuming that the reference is indeed correct, such a test can act as a strong guarantee for any new code that we introduce¹².

Finally, with a solid reference in hand, we are ready for the second part of the statement at the beginning of this blog:

... then make it anything you want.

Recursing the "Right" Way

Astute readers may notice that our reference solution, recursively beautiful as it is, is actually flawed in several ways. The most glaring ones:

It does recursion "the wrong way". That is, it uses recursion in a non-tail position. In a strict language like Scala this easily leads to stack overflows.
It allocates O(w * h) memory upfront. This is actually quite bad, so much so, that I wasn't able to trigger a stack overflow even with a heap of more than 8 GB. Allocating a large enough board, such that it is worthy of a stack overflow is so heavy that I got an out of memory exception first¹³ ¹⁴.

The fact that I consider these to be flaws is of course subjective, and depends on the use case being considered. More importantly though, these are only non-functional flaws, logically our solution is doing the right thing. And so we can still use it as a reference while we fix these flaws.

We'll start by fixing the recursion issue, as this one is easier to tackle.

Lucky for us, transforming (simple) non-tail-recursive code into tail recursive code is a fairly mechanical process. Given our reference code we can mechanically turn it into a tail recursive version:

Make your return value into an argument to the recursive function
Instead of returning a result and recursing, update the new argument with the result of the current step and then recurse
When reaching the base case, return the accumulated result argument

Applied to our reference solution, the result is this:

def traceSpiral(dims: Dims): List[Point] =
  peel(buildCoordinates(dims), result = Nil) // 1

@tailrec // 2
def peel(board: List[List[Point]],
         result: List[Point]): List[Point] = // 3
  board match
    case Nil => result // 4
    case topRow :: otherRows =>
      peel(rotateCCW(otherRows), topRow ++ result) // 5

This code is very similar to the code we started with, but it's now tail recursive. In detail:

The peel function now takes an extra argument result (3), we'll use that to accumulate the final result
When making the recursive call, we prepend the current result topRow to the result argument we got (5)
In the base case (4), we return the result argument
peel is now tail-recursive, so we annotate it with the tailRec annotation (2), this forces the compiler to validate that the code is indeed tail recursive and can be optimized accordingly
When making the initial call to peel we initialize result with the original base case value Nil (1)

That's it. The code is now tail recursive. But is it correct? I mean this transformation is not completely trivial, how do we know we got it right?

Easy, we have our reference solution, we can just run our property-based tests and validate that we are still returning logically correct results:

TailRecursiveSuite:
==> X TailRecursiveSuite. Traces the correct spiral // <--- 1
31:
32:    assertEquals(points, expected)
33:
values are not the same
=> Obtained
List(
  Point(
    x = 1,
    y = 1
  ),
// ... more output
)
=> Diff (- obtained, + expected)
   Point(
+    x = 0,
+    y = 0
+  ),
// ... more output
   )
  + Sanity: all points in bounds 0.069s // <--- 2
  + Sanity: dimensions match number of points 0.015s
  + Sanity: no duplicate points 0.019s
==> X TailRecursiveSuite. Matches reference
12:
13:      actual ==== expected
- TailRecursiveSuite. Matches reference: Falsified after 10 passed tests
> Dims(1,2)
> === Not Equal ===
> --- lhs ---
> List(Point(0,1), Point(0,0))
> --- rhs ---
> List(Point(0,0), Point(0,1))

... And we failed. On two tests no less. I clipped some of the output as it's a bit noisy. Let's see what happened.

We failed the unit test, that's the X TailRecursiveSuite. Traces the correct spiral (1). The output contains the diff between what we got and what was expected (clipped). It's a somewhat difficult to decipher as the hardcoded value in the unit test is a bit large. It's good to know that we are protected by the unit test, but it's not that useful for debugging.

Next we have the sanity tests (2). We did pass them, which is unfortunate, as they don't help us to narrow down the problem in this case. Oh, well, maybe next time.

Lastly, we have the reference solution test. And this one fails, and not only that, the failing values are very small. This is no coincidence, property-based testing libraries usually come with the ability to shrink their values. That is, if a test fails on some input, the library tries to shrink that input to the smallest possible value that is still failing¹⁵. In our case this means that if we fail on some random Dims instance, the library will try to reduce the size of the dimensions as close as possible to Dims(0, 0). Here we failed on the dimensions Dims(1, 2):

> === Not Equal ===
> --- lhs ---
> List(Point(0,1), Point(0,0))
> --- rhs ---
> List(Point(0,0), Point(0,1))

rhs is the expected (reference) value, while lhs is the value we got from our tail recursive solution. We see that our new solution is reversed compared to the reference solution.

Now, I'm not much into thinking, I got my test to do that for me. Since the result looks reversed, let's reverse it back:

def traceSpiral(dims: Dims): List[Point] =
  peel(buildCoordinates(dims), result = Nil).reverse // 1

@tailrec
def peel(board: List[List[Point]], result: List[Point]): List[Point] =
  board match
    case Nil => result
    case topRow :: otherRows =>
      peel(rotateCCW(otherRows), topRow ++ result)

This solution is the same as the previous one, but the final result is reversed (1). Let's run the tests again:

==> X TailRecursiveSuite. Matches reference
12:
13:      actual ==== expected
- TailRecursiveSuite. Matches reference: Falsified after 8 passed tests
> Dims(2,1)
> === Not Equal ===
> --- lhs ---
> List(Point(1,0), Point(0,0))
> --- rhs ---
> List(Point(0,0), Point(1,0))

We're still failing on the same test. The failure looks the same, the result is again reversed compared to the reference. Except that this time the dimensions are transposed, Dims(2, 1) vs. Dims(1, 2).

I guess it's not really surprising that a simple reverse wasn't good enough. The failing unit test with the bigger board wasn't failing on a simple reversal. Something trickier is going on here.

The problem is that the counterexample is so minimized that it's hard to see what the underlying issue is. We need a bigger counterexample. If we weren't using property-based testing we'd have to manually search for more problematic cases, like some plebs. Luckily, we'll be spared this gloomy fate. Instead we'll define a generator that limits the dimensions to be at least 2. And tweak our reference test to use that (temporarily, for "debugging"):

property("Matches reference"):
  forAll(dimsAboveGen(2)): dims => // 1
    val expected = Reference.traceSpiral(dims)
    val actual = traceSpiralCandidate(dims).toList

    actual ==== expected

In (1) we're using the new generator. After removing the unproductive reverse call, and we get a new, larger, counter example:

> Dims(2,2)
> --- lhs ---
> List(Point(0,1), Point(1,1), Point(0,0), Point(1,0))
> --- rhs ---
> List(Point(0,0), Point(1,0), Point(1,1), Point(0,1))

The next counterexample was right around the corner, who could've thunk...

With this larger, but still not overwhelming, example we see a glimmer of a pattern. The new result is not reversed, but rather the halves are reversed. If we split the list in two and swap the halves around, we'll match the reference. Let's confirm this pattern with dimensions above 2:

> Dims(3,3)
> === Not Equal ===
> --- lhs ---
> List(Point(1,1), Point(0,1), Point(1,2), Point(0,2), Point(2,1), Point(2,2), Point(0,0), Point(1,0), Point(2,0))
> --- rhs ---
> List(Point(0,0), Point(1,0), Point(2,0), Point(2,1), Point(2,2), Point(1,2), Point(0,2), Point(0,1), Point(1,1))

Although we probably skipped over 2x3 cases, this is good enough. Since the number of points in the result is not even, we obviously won't be splitting in two halves. But nor can we split in thirds. It seems that the pattern is more involved. Looking for segments that we can reverse, we see the following segment lengths: 2, 2, 2, 3. Weird... But recall our orange peeling algorithm. The first segment we peel off is 3. We are then left with a 3x2 board, which we rotate, peeling off a segment of length 2. Rotate again, peel off 2, and then another 2. Our segment lengths are just the orange peeling segments in reverse!

We're building up the result the wrong way:

def peel(board: List[List[Point]], result: List[Point]): List[Point] =
  board match
    case Nil => result
    case topRow :: otherRows =>
      peel(rotateCCW(otherRows), topRow ++ result) // <--

The arrow points at the problematic place where we build up the result. This was perfectly fine with the original, non-tail-recursive solution. But what I missed in my translation algorithm is that result building with tail recursion goes "the other way around" ¹⁶.

The fix is straightforward, just reverse the arguments:

def peel(board: List[List[Point]], result: List[Point]): List[Point] =
  board match
    case Nil => result
    case topRow :: otherRows =>
      peel(rotateCCW(otherRows), result ++ topRow) // <--

With the fix in place, we finally pass all the tests:

TailRecursiveSuite:
  + Traces the correct spiral 0.04s
  + Sanity: all points in bounds 0.113s
  + Sanity: dimensions match number of points 0.04s
  + Sanity: no duplicate points 0.046s
  + Matches reference 0.058s

Now we have the confidence that we didn't break the correctness of our original solution.

If you're really into asymptotics though, you might notice that our solution just turned to be quadratic. Using ++ on a List to append to the ever growing result value is not good for performance.

The fix is yet again mechanical, but leads us even further away from the reference solution. Since prepending to a List is faster than appending to it, we can do that and sprinkle in some reverse calls. I'll spare you the gory details of me trying to debug the implementation with tests. The upshot is this:

def traceSpiral(dims: Dims): List[Point] =
  peel(buildCoordinates(dims), result = Nil).reverse // 1

@tailrec
def peel(board: List[List[Point]],
         result: List[Point]): List[Point] =
  board match
    case Nil => result
    case topRow :: otherRows =>
      peel(rotateCCW(otherRows), topRow.reverse ++ result) // 2

Here we reverse the way we build up the result (2). We first reverse the current row, and the prepend it to the accumulator. Finally, we have to reverse the whole result (1). Rest assured that I messed up the different reverse calls, but the tests had my back.

So guided by the correct reference solution we managed to go quite far away to improve performance, while not losing sight of correctness.

But why stop here?

Recursion, I Hardly Know Her

Lately, I was flabbergasted to hear that there exist languages that don't support any form of tail call optimization¹⁷.

Try as we might, our tail-recursive solution doesn't help us if we're stuck using such a language. Using any form of recursion is liable to get our stack blown. How can we help our fellow programmers in need?

Easy, just as there's a mechanical way to translate simple recursion into tail recursion, so can we translate tail recursive code into an iterative loop.

The process is as follows:

Replace the recursive function with a while loop
Replace each argument to the recursive function with a var
Instead of making a recursive call with new values, update the variables with those new values
Run the loop until reaching the base case, then return the variable that stands for the accumulator

Concretely, in code:

def traceSpiral(dims: Dims): List[Point] =
  var result = List.empty[Point] // 1
  var board = buildCoordinates(dims) // 2

  while board != Nil do // 3
    val topRow = board.head // 4
    val otherRows = board.tail // 5

    result = topRow.reverse ++ result // 6

    board = rotateCCW(otherRows) // 7

  end while

  result.reverse // 8

We no longer have a recursive function, just one big loop
What were previously function arguments are now the (mutable) variables result and board (1, 2)
The loop will run until we hit the base case, the empty board (3)
While the board is not empty, we peel of the current row (4), and the rest of the rows (5)
We then update result variable in the same way we did in the last recursive solution (6)
And the same for the board variable (7)
Once the loop completes, result contains the full result in reverse, which we return after reversing (8)

Look how far away from our initial, elegant, recursive solution we managed to get. But also notice how much more error-prone this code is:

We no longer use pattern matching, instead relying on the unsafe head and tail methods¹⁸, the compiler will no longer have our back when refactoring
If we forget to update result, everything will still compile and run, but the final result will be wrong
If we forget to update board, everything will still compile and run, but this time we'll hit an infinite loop¹⁹

I won't disclose how many of those issues I've hit while writing this code. But all I'll say is that I'm happy that I had a robust test suite. Having a solid reference sure pays dividends...

Looking at this solution though, we might again notice some redundancy. As we are no longer in the nice functional realm, do we really need to pay the price for building an immutable list? Especially as it forces us through all the reverse shenanigans just to keep us from getting quadratic.

Might as well go all in on the procedural approach and use a more performant mutable collection:

def traceSpiral(dims: Dims): List[Point] =
  val result = ListBuffer.empty[Point] // 1
  var board = buildCoordinates(dims)

  while board != Nil do
    val topRow = board.head
    val otherRows = board.tail

    result.addAll(topRow) // 2

    board = rotateCCW(otherRows)

  end while

  result.toList // 3

result is now a mutable ListBuffer (1), which can be appended to efficiently. Which is what we do in (2), avoiding the need to reverse. At the very end we build up the final List result from the buffer (3). This way we can maintain a functional facade while utilizing mutability in the implementation. We gained performance and made the code a bit less confusing.

Guided by the reference solution we managed to accommodate the needs of our fellow procedural programmers, recursion be damned. Yay us.

Now let's move on to the real elephant in the room.

Laziness with Class

One of my biggest takeaways from dabbling in computer science is that it pays to be lazy²⁰.

Clearly the reference solution is not very lazy. It allocates a potentially large board upfront, while in reality it might not be needed at all. Maybe our clients will need only part of the solution? Or maybe they'll use it one step at a time, like with streaming? But no, we prevent any such useful use cases by allocating O(w * h) data upfront, with no way to take it back. Time to fix that.

Nostalgic aside...

Back in the olden days, in the heyday of OOP, some smart people decreed that we should all "program to interface". And so you would see programmers gather around in circles and chant "SOLID, SOLID, SOLID..." as they conjure up the latest AbstractSingletonFactoryFactoryVisitater²¹ out of the ether. Those were the days...

And although these ancient customs may seem strange to us in this day and age, there is a grain of truth to it. Programming to interface can be very useful.

It is indeed by way of us using a very concrete representation of the board, namely List[List[Point]], that we are forced to allocate memory prematurely. But is it actually necessary for the board to be a list? Is it essential to the solution?

If we could extract some kind of simple interface from the reference maybe we'll be able to implement a less memory intensive solution.

We can use the code we started with as guidance:

def traceSpiral(dims: Dims): List[Point] =
  peel(buildCoordinates(dims)) // 1

def peel(board: List[List[Point]]): List[Point] =
  board match // 2
    case Nil => Nil // 3
    case topRow :: otherRows => // 4
      topRow ++ peel(rotateCCW(otherRows)) // 5

Suppose we had some entity called Board. What kind of interface should it have? Taking inspiration from the reference solution, what we need is:

A way to construct a Board for given dimensions: Dims => Board (1)
A way to query the state of the board, pattern matching would be a nice way to do it (2)
A case for empty boards (3)
A case for non-empty boards, where we can extract the top row and the rest of the rows (4)
A way to rotate the board counter-clockwise (5)

Any concrete Board representation that implements this interface can be used to solve our problem. The trick is to find a representation that consumes less memory than List. If this was some real code and we wanted to make it polymorphic over different board implementations, we might actually define that interface in code. But here I'll just use this pseudo-interface as a guide and provide another concrete implementation directly.

Not using List is necessary for us to be able to solve the problem lazily, but it's not sufficient. To actually turn things lazy we'll use Iterators²². So instead of returning List[Point], we'll return Iterator[Point]. The new signature that we need to implement is:

def traceSpiral(dims: Dims): Iterator[Point]

With the goal being that we never allocate more than O(1) memory.

Let's sketch out our new Board type, we'll refine the details as we go along. First define a type:

sealed trait Board

We are using a sealed trait (sum type) as requirements 2-4 indicate that we are going to have more than one case: empty and non-empty. Let's add those:

case object Empty extends Board

case object NonEmpty extends Board

One case per requirement, and we can distinguish between them by pattern matching.

As per requirement 4 we'll also need some way of extracting data from the non-empty case, so we'll add methods:

case object NonEmpty extends Board:
  def topRow: Iterator[Point]
  def otherRows: Board

We are using methods rather than class fields as I don't yet want to commit to a specific representation of NonEmpty. topRow returns an Iterator and not a List so as to preserve our laziness goals.

Requirement 5 implies that we need a rotation method that works on any board:

sealed trait Board:
  def rotateCCW: Board

Lastly, we need a builder function for requirement 1:

def buildCoordinates(dims: Dims): Board

And this completes our pseudo-interface, we can plug it into the algorithm and get a brand new solution to the spiral problem. To wit:

def traceSpiral(dims: Dims): Iterator[Point] =
    peel(buildCoordinates(dims))

def peel(board: Board): Iterator[Point] =
  board match
    case Empty => Iterator.empty // 1
    case ps: NonEmpty => // 2
      ps.topRow ++ peel(ps.otherRows.rotateCCW) // 3

This looks eerily similar to the reference solution. We just replaced every occurrence of List with Iterator, as Iterator and List share a very similar interface²³ this change is almost seamless, ++ works for either type (3). In (1) we return an empty Iterator rather than an empty List.

The only place things look a bit different is the NonEmpty pattern match (2), where we no longer extract the rows²⁴. Instead, we get them with the appropriate method calls on the board value (3).

As a bonus, we no longer have to care about the non-tail-recursive call to peel. ++ being lazy in its right argument makes that problem go away (left as an exercise to figure out why).

All that's left is to actually implement the functions I left abstract. Once we do that, we should have a working solution, but with a very different memory footprint.

Let's start with the easy case, Empty. The empty case has no data associated with it, so we don't need to add any fields to it. All we need is to be able to rotate it, which is trivial:

case object Empty extends Board:
  def rotateCCW: Board = Empty

A rotated empty board is still an empty board. Done.

Now comes the tricky bit, how do we implement NonEmpty without populating a whole board-worth of data?

Here we can take advantage of the fact that we already have a working solution that we can use to gain some insight about the shape of the data we're dealing with. Recall the board that we start with for a 4x3 problem:

---------------------------------
| (0,0) | (1,0) | (2,0) | (3,0) |
---------------------------------
| (0,1) | (1,1) | (2,1) | (3,1) |
---------------------------------
| (0,2) | (1,2) | (2,2) | (3,2) |
---------------------------------

The first row, which is what we want to extract is not made out of arbitrary points, instead it's very structured. It's just (0, i), where i is 0 <= i < 4. We can easily represents this with an Iterator that will consume a constant amount of space:

Iterator.range(0, 4).map(Point(_, 0))

Iterator.range creates a lazy range that is equivalent to [0, 1, 2, 3], but without actually materializing all the numbers into memory. Then we use map, which is lazy on Iterator, to build up Point(i, 0).

For the actual implementation of topRow we'll need to replace the hardcoded 4 with the width of the board. And that's the first ingredient that NonEmpty needs to hold on to: the current Dims of the board. Leading to this code:

case class NonEmpty(dims: Dims) extends Board: // 1
  def topRow: Iterator[Point] =
    Iterator.range(0, dims.w).map(Point(_, 0)) // 2

We added a new field to the NonEmpty case to hold on to the dimensions that this board represents (1). Using the dims field it's now possible to create the iterator for the first row (2).

This seemingly works, but it was suspiciously easy to implement. Sure this is the correct answer on the first step of the solution, but will it work for the second step?

Recall that as we are solving the problem we keep shrinking and rotating the board. So for the second step we should be working with the following board:

-----------------
| (3,1) | (3,2) |
-----------------
| (2,1) | (2,2) |
-----------------
| (1,1) | (1,2) |
-----------------
| (0,1) | (0,2) |
-----------------

This is what's left after we remove the first row and rotate the board. The board will now have different dimensions, but we are covered, we have the dimensions as a field of NonEmpty. We can update its values as we are building up the solution.

But even with the correct dimensions in place, the code we just had:

Iterator.range(0, dims.w).map(Point(_, 0))

Won't produce the correct result. We'll get [(0, 0), (1, 0)], which is the first row of a board of dimensions 2x4, instead of the correct [(3, 1), (3, 2)]. The problem is that we are not actually working with a 2x4 board, the entries that we are looking for come from the original board of 4x3. Compared to the 2x4 board, the original board is rotated and shifted (since we removed a row). Without the information about how much we are rotated and shifted we won't be able to compute the correct first row. Let's add this information to our class:

case class NonEmpty(dims: Dims, rotation: Rotation, shift: Point) extends Board

Rotation is a type that represents the current rotation of the board relative to the original board we started with. We do it by wrapping a number that represents the number of 90 degree rotations that we have²⁵:

case class Rotation(value: Int)

E.g., Rotation(0) is no rotation, Rotation(1) is rotation by 90 degrees, and we wrap around at 4: Rotation(4) = Rotation(0). For the second step we are at Rotation(1).

The shift field represents where we need to shift the current board so that it matches the original board²⁶. Or more concretely, where the top-left corner of the current board is located on the original board. We'll be adding shift to each point in the result. For the second step the shift is (3, 1).

With these two fields we can define the following helper method:

def rotateShift(point: Point): Point

This method takes a single point, and rotates and shifts it using the rotation and shift fields. The implementation is not very difficult (you can see it in the repo), but having nonexistent spatial abilities it took me quite a few trials and errors to get it right. Figuring out the axis of rotation, mixing up signs, the order of operations, and rotation directions. But guess what had my back all along? A strong test suite backed by a correct solution.

If we keep track of the correct rotation and shift while we compute the solution (we'll see how later on), we can use them to get the right answer for the first row at every step:

def topRow: Iterator[Point] =
  val row = Iterator.range(0, dims.w).map(Point(_, 0)) // 1

  row.map(rotateShift) // 2

Now we compute the first row like we did before (1), but instead of returning it as is, we rotateShift each point in the row (2). This is still as lazy as before, but now gives the correct answer at every step (assuming that the rotate and shift fields are correct).

For example, as mentioned above, for the second step of the solution we should have: rotation = Rotation(1), shift = Point(3, 1), and row = [(0, 0), (1, 0)]. If we rotate it by 90 degrees we get:

[(0, 0), (0, 1)]

Then we shift it to get:

[(3, 1), (3, 2)]

Which is the correct result for the second step.

Before we proceed with otherRows, let's define a useful helper:

def buildCoordinateFrom(dims: Dims, // 1
                        rotation: Rotation,
                        shift: Point): Board =
  if dims.area == 0 then Empty // 2
  else NonEmpty(dims, rotation, shift) // 3

This function takes in all the ingredient that we need for a board (1) and constructs the appropriate representation for it. For dimensions that represent an empty area we return an instance of Empty (2). Otherwise we return a NonEmpty instance with all the arguments we got (3). This implementation conveniently hides away the distinction between Empty and NonEmpty when we don't care about it.

With this function we can also implement the buildCoordinates function that's part of our "interface":

def buildCoordinates(dims: Dims): Board =
  buildCoordinateFrom(dims, Rotation(0), Point(0, 0))

This is just a call to buildCoordinateFrom with the rotation and shift set to their initial values (no rotation and no shift).

Next we are ready to implement otherRows. As we are aiming for laziness here, no "actual" work should be performed by otherRows. It's topRow's job to produce actual data, otherRows needs only to keep track of the changing state of the board. That is, it needs to update dims, rotation, and shift to match the new, smaller board.

Pictorially, what's happening is this:

---------------------------------
|   x   |   x   |   x   |   x   |
---------------------------------
| (0,1) | (1,1) | (2,1) | (3,1) |
---------------------------------
| (0,2) | (1,2) | (2,2) | (3,2) |
---------------------------------

We need to remove the top row, marked by xs. We can do it like so:

def otherRows: Board =
  val newDims = dims.reduceH(1) // 1
  val newShift = rotateShift(Point(x = 0, y = 1)) // 2

  buildCoordinateFrom(newDims, rotation, newShift) // 3

In the first step we reduce the vertical (height) dimension of the board by 1 (1), the intent being to drop the top row. Next we compute the new shift. When removing the top row, the new top-left corner moves one step down, hence the shift is by (0, 1) (one step down). But we must not forget that we have the current rotation and shift to account for, so we have to rotateShift (0, 1) to cover that (2). Finally, we build up a new (smaller) board with new dimensions and shift using buildCoordinateFrom (3).

All we did here is to manipulate the parameters that represent the state of the board, and laziness is preserved.

The last ingredient to complete this lazy solution is rotateCCW. It's similar to otherRows in that it only manipulates the state parameters, but it's somewhat trickier geometrically. We need this board:

---------------------------------
| (0,1) | (1,1) | (2,1) | (3,1) |
---------------------------------
| (0,2) | (1,2) | (2,2) | (3,2) |
---------------------------------

To become this:

-----------------
| (3,1) | (3,2) |
-----------------
| (2,1) | (2,2) |
-----------------
| (1,1) | (1,2) |
-----------------
| (0,1) | (0,2) |
-----------------

Meaning that the top right corner rotates to become the top-left corner.

def rotateCCW: Board =
  val newDims = dims.transpose // 1
  val newRotation = rotation.next // 2
  val newShift = rotateShift(Point(x = dims.w - 1, y = 0)) // 3

  buildCoordinateFrom(newDims, newRotation, newShift) // 4

Step by step:

The new dimensions are just the transpose (swapping w and h) of the old ones (1)
The new rotation is + 1 on the old one, but we need to wrap around when we reach 4, that's what next does (2)
The new shift is the top-right corner (w - 1, 0), which now becomes the top-left corner, appropriately rotateShifted, of course (3)²⁷
With the full new state, we can create a new board (4)

That's it! We got a fully lazy solution to our problem. And if you're thinking to yourselves "wow, so much geometry, must have been tedious and error prone to implement", you're completely right. Lucky that I was blessed by a correct reference solution to keep me sane.

And after so many failed attempts, it was all the more satisfying to see the test suite passing on this one:

LazySuite:
  + Traces the correct spiral 0.023s
  + Sanity: all points in bounds 0.051s
  + Sanity: dimensions match number of points 0.019s
  + Sanity: no duplicate points 0.021s
  + Matches reference 0.04s

But is it actually lazy?

While this code using the reference solution throws an out-of-memory exception on my machine:

Reference.traceSpiral(Dims(w = 5000, h = 5000))
  .foreach(println)

The same code with the lazy solution happily chugs along, printing all the spiral elements:

Lazy.traceSpiral(Dims(w = 5000, h = 5000))
  .foreach(println)

Mission accomplished. By using the "interface" of the obviously correct solution as inspiration we managed to derive a functioning lazy solution. At least for me, coming up with such a solution without some concrete (and correct) reference would've been challenging²⁸.

But all these rotations and shifts, they do feel a bit painful.

Follow the Pain

There's this phenomenon that I noticed over the years, that software developers tend to build up pain tolerance to the point where they no longer feel any pain while writing code. It seems to be a protection mechanism against all the terrible code we have to deal with daily. "It gets the job done", "worse is better", "it is what it is", they say to themselves and move along the 3,000 LOC function. And that's a shame. Pain is an excellent driver for improvement. The moment you stop feeling pain, like a shark that stops moving, your creativity and drive to improve wither away.

Cultivating the appropriate pain sensitivity when writing code is a must-have when working on large and complex codebases.

With that in mind, I shall follow the pain that we felt in the previous section with all the geometry and rotations. How did we get there? Does it have to be that way?

We pretty much mechanically followed the "interface" dictated by the reference solution. One might say that we were lulled into a false sense of security by it. Maybe it's a downside of having an existing solution. But not all is lost. Now that we have the painful solution, we can use it to visualize the problem, and maybe come up with something better.

The main pain point of the lazy solution are the rotations that we have to keep track of. They contribute the biggest chunk of complication in the code, making the shifts jump all over the place in a seemingly random fashion. Let's plot the rotations and shifts on a board:

-----------------------------------------------------------------------
| a,0→ |      |      |      |      |      |      |      |      |      |
-----------------------------------------------------------------------
|      | e,0→ |      |      |      |      |      |      |      | ↓b,1 |
-----------------------------------------------------------------------
|      |      | i,0→ |      |      |      |      |      | ↓f,1 |      |
-----------------------------------------------------------------------
|      |      |      | m,0→ |      |      |      | ↓j,1 |      |      |
-----------------------------------------------------------------------
|      |      | l,3↑ |      |      | ←o,2 | ↓n,1 |      |      |      |
-----------------------------------------------------------------------
|      | h,3↑ |      |      |      |      | ←k,2 |      |      |      |
-----------------------------------------------------------------------
| d,3↑ |      |      |      |      |      |      | ←g,2 |      |      |
-----------------------------------------------------------------------
|      |      |      |      |      |      |      |      | ←c,2 |      |
-----------------------------------------------------------------------

Here the first letter (a-z) is indexing the shifts, the number is the rotation index (0-3), and the arrow is the direction of the row getting peeled.

Unsurprisingly, the shifts form a spiral-like pattern, something that was not intuitively clear from the code. Seeing this from a high-level vantage point makes the pattern very clear: all shifts of the same rotation sit on a diagonal. Better yet, the shifts with no rotation (== 0) sit on the main diagonal (i, i), for i going from 0 to the center of the board.

This reflects the not very deep insight²⁹ about 90 degree rotations: every 4 rotations we are back where we started. So each shift of the same rotation must start on the same angle of the board.

With this insight in mind we can come up with a new algorithm to solve the riddle:

On each step, draw a frame around the board
Remove the frame's points and add them (in clockwise order) to the solution
Repeat until the board is empty

Notice how this algorithm does not imply any rotations. We just remove frames one by one keeping the board in place.

We can illustrate the first few steps of the algorithm with this board:

---------------------------------------------------------------------------------
| (0,0) | (1,0) | (2,0) | (3,0) | (4,0) | (5,0) | (6,0) | (7,0) | (8,0) | (9,0) |
---------------------------------------------------------------------------------
| (0,1) | (1,1) | (2,1) | (3,1) | (4,1) | (5,1) | (6,1) | (7,1) | (8,1) | (9,1) |
---------------------------------------------------------------------------------
| (0,2) | (1,2) | (2,2) | (3,2) | (4,2) | (5,2) | (6,2) | (7,2) | (8,2) | (9,2) |
---------------------------------------------------------------------------------
| (0,3) | (1,3) | (2,3) | (3,3) | (4,3) | (5,3) | (6,3) | (7,3) | (8,3) | (9,3) |
---------------------------------------------------------------------------------
| (0,4) | (1,4) | (2,4) | (3,4) | (4,4) | (5,4) | (6,4) | (7,4) | (8,4) | (9,4) |
---------------------------------------------------------------------------------
| (0,5) | (1,5) | (2,5) | (3,5) | (4,5) | (5,5) | (6,5) | (7,5) | (8,5) | (9,5) |
---------------------------------------------------------------------------------
| (0,6) | (1,6) | (2,6) | (3,6) | (4,6) | (5,6) | (6,6) | (7,6) | (8,6) | (9,6) |
---------------------------------------------------------------------------------
| (0,7) | (1,7) | (2,7) | (3,7) | (4,7) | (5,7) | (6,7) | (7,7) | (8,7) | (9,7) |
---------------------------------------------------------------------------------

The first frame will be:

---------------------------------------------------------------------------------
| (0,0) | (1,0) | (2,0) | (3,0) | (4,0) | (5,0) | (6,0) | (7,0) | (8,0) | (9,0) |
---------------------------------------------------------------------------------
| (0,1) |                                                               | (9,1) |
---------                                                               ---------
| (0,2) |                                                               | (9,2) |
---------                                                               ---------
| (0,3) |                                                               | (9,3) |
---------                                                               ---------
| (0,4) |                                                               | (9,4) |
---------                                                               ---------
| (0,5) |                                                               | (9,5) |
---------                                                               ---------
| (0,6) |                                                               | (9,6) |
---------------------------------------------------------------------------------
| (0,7) | (1,7) | (2,7) | (3,7) | (4,7) | (5,7) | (6,7) | (7,7) | (8,7) | (9,7) |
---------------------------------------------------------------------------------

We add the elements of the frame in clockwise direction to the solution. The remaining board becomes:

-----------------------------------------------------------------
| (1,1) | (2,1) | (3,1) | (4,1) | (5,1) | (6,1) | (7,1) | (8,1) |
-----------------------------------------------------------------
| (1,2) | (2,2) | (3,2) | (4,2) | (5,2) | (6,2) | (7,2) | (8,2) |
-----------------------------------------------------------------
| (1,3) | (2,3) | (3,3) | (4,3) | (5,3) | (6,3) | (7,3) | (8,3) |
-----------------------------------------------------------------
| (1,4) | (2,4) | (3,4) | (4,4) | (5,4) | (6,4) | (7,4) | (8,4) |
-----------------------------------------------------------------
| (1,5) | (2,5) | (3,5) | (4,5) | (5,5) | (6,5) | (7,5) | (8,5) |
-----------------------------------------------------------------
| (1,6) | (2,6) | (3,6) | (4,6) | (5,6) | (6,6) | (7,6) | (8,6) |
-----------------------------------------------------------------
| (1,7) | (2,7) | (3,7) | (4,7) | (5,7) | (6,7) | (7,7) | (8,7) |
-----------------------------------------------------------------

The next frame becomes:

-----------------------------------------------------------------
| (1,1) | (2,1) | (3,1) | (4,1) | (5,1) | (6,1) | (7,1) | (8,1) |
-----------------------------------------------------------------
| (1,2) |                                               | (8,2) |
---------                                               ---------
| (1,3) |                                               | (8,3) |
---------                                               ---------
| (1,4) |                                               | (8,4) |
---------                                               ---------
| (1,5) |                                               | (8,5) |
---------                                               ---------
| (1,6) |                                               | (8,6) |
-----------------------------------------------------------------
| (1,7) | (2,7) | (3,7) | (4,7) | (5,7) | (6,7) | (7,7) | (8,7) |
-----------------------------------------------------------------

We add it to the solution, and repeat until the board is empty. No rotations involved, everything seems simple³⁰.

Let's code this up (while still maintaining laziness). We'll use the description of the algorithm as an inspiration for an "interface", and solve the problem in terms of the new interface. After that we'll actually implement the interface itself.

Like before, we'll start with two cases for the board:

sealed trait Board

case object Empty extends Board

case class NonEmpty extends Board:
  def getFrame: Iterator[Point] // 1
  def dropFrame: Board // 2

def buildCoordinates(dims: Dims): Board // 3

The Empty case looks the same (1), no additional methods beyond the datatype itself. NonEmpty changes completely and now has two new methods:

getFrame - returns an iterator for the full frame of the board (1), in clockwise order
dropFrame- returns a new board with the outer frame of the board removed (2)

We also still use a factory function to build up a board for given dimensions (3).

This covers all the functionality we need to implement the new algorithm:

def traceSpiral(dims: Dims): Iterator[Point] =
  peel(Board.buildCoordinates(dims)) // 1

def peel(board: Board): Iterator[Point] =
  board match // 2
    case Empty => Iterator.empty // 3
    case nonEmpty: NonEmpty => // 4
      nonEmpty.getFrame ++ peel(nonEmpty.dropFrame) // 5

The new implementation is still pretty simple, and quite similar to the original lazy solution:

We build a new board for the dimensions of the problem and call the recursive peel function on it (1)
On each peeling step we match on the board (2)
If it's empty, we are done and produce an empty iterator (3)
Otherwise, for the non-empty case (4)
We extract the frame of the board, and prepend it to the result of peeling the board without the frame (5)

We followed the description of the algorithm to the letter, now we just need to implement the interface.

For the Empty case there's nothing more to implement. Rotations are no longer a part of the interface, so the Empty case just needs to exist.

Now, what kind of data does NonEmpty needs to have? Before we kept track of:

The dimensions of the board, still relevant
The rotation of the board, no longer relevant
The shift of the board, still relevant as we will keep shrinking the board

But we can simplify the shift. Previously the shift was a Point because it was difficult to follow the position at each step. But now we just need to move along the main diagonal on each peeling step. So on step i, the shift is (i, i), meaning that we just need a single number, rather than a point to represent the shift.

NonEmpty then becomes:

case class NonEmpty(dims: Dims, shift: Int) extends Board

With these pieces of state we can implement the methods of NonEmpty. First getFrame.

The idea here is similar to getting the first row of a board like we did in the first lazy solution. But this time we need both horizontal and vertical directions, moving forwards and backwards. So, for example, the vertical segment on the left is given by:

Iterator.range(h - 1, 0, step = -1) // 1
  .map(Point(0, _)) // 2

This segment at y = h - 1, and we step backwards till y = 0 (1). This becomes the y coordinate of Points with a fixed x = 0. Figuring out the exact offsets and handling off-by-one errors is of course prone to bugs, but we're not worried, we have a robust test suite to catch such things.

Getting the full frame becomes:

def getFrame: Iterator[Point] =
  import dims.* // 1

  val path1 = Iterator.range(0, w - 1).map(Point(_, 0)) // 2
  val path2 = Iterator.range(0, h - 1).map(Point(w - 1, _))
  val path3 = Iterator.range(w - 1, 0, step = -1).map(Point(_, h - 1))
  val path4 = Iterator.range(h - 1, 0, step = -1).map(Point(0, _))

  val frame = // 3
    path1 ++ path2 ++ path3 ++ path4

  frame.map(_.shiftBy(dx = shift, dy = shift)) // 4

We start by importing the fields of the Dims instance of the board, just to make the code below more concise (1)
Then we create the different segments of the frame (2), in clockwise order, the details of the numbering are left as an exercise
The full frame is composed from concatenating all four segments (3)
Lastly, we need to shift the frame diagonally down, we do it by moving each point by (shift, shift) (4)

The result is a full frame, shifted to correspond to location of the current step. I don't know about you, but for me it was much easier to figure out this code compared to all the rotation logic we had before.

Like before, we'll use a helper to build boards at the current dimensions and shift:

def buildCoordinateFrom(dims: Dims, shift: Int): Board = // 1
  if dims.area == 0 then Empty // 2
  else NonEmpty(dims, shift) // 3

This time we only get the dimensions and shift (1). In (2) we create an Empty board if the area is 0. If not, we create a NonEmpty board with the given dimensions and shift (3).

With the helper, buildCoordinates is trivial:

def buildCoordinates(dims: Dims): Board =
  buildCoordinateFrom(dims, 0)

And lastly, dropFrame:

def dropFrame: Board =
  val newDims = dims.reduceBy(2) // 1
  val newShift = shift + 1 // 2

  buildCoordinateFrom(newDims, newShift) // 3

The new dimensions (1) after dropping a frame are computed by reducing the dimensions by 2 both horizontally and vertically (since the frame has two segments in each orientation)
The new shift (2) is just incrementing by 1 as we move along the main diagonal
Then we can build a new board with the new state (3)

Done. This should be a fully working, lazy, solution to the problem. As promised it's greatly simplified by not having to deal with rotations at all.

Does it work?

Let's find out by running the tests³¹:

LazyFrameBuggySuite:
  + Traces the correct spiral 0.019s
^C

The good news are that the unit test is passing. The bad news are that I had to terminate the run due to what seems like an infinite loop.

The property-based tests are catching some edge case that the unit test is missing. Great to have them watching our back. Unfortunately, as the blight of Turing completeness is upon us, there's no way for the tests to actually show us what triggered the infinite loop.

Instead we'll be doing some good old-fashioned println debugging. Since the board contains the full state information as we are moving along the solution, we'll start by printing the board on each step. Which might give us the following (output changes due to randomization in property-based test):

...
NonEmpty(Dims(3,1),0)
NonEmpty(Dims(1,-1),1)
NonEmpty(Dims(-1,-3),2)
NonEmpty(Dims(-3,-5),3)
NonEmpty(Dims(-5,-7),4)
NonEmpty(Dims(-7,-9),5)
...

Somehow, we are getting negative dimensions. Philosophical issues aside, how on earth does the board's dimension gets negative?

The clue is in the fact that the dimensions reduce by two on every step. Which was of course part of the algorithm. Unlike the original lazy version where the dimensions reduce by 1 one each step, and we are guaranteed to hit 0 eventually, in the new code we might just jump over it.

Now, if I had any principles, I would be practicing what I preach and make this "illegal state unrepresentable" in a way that's guaranteed by the compiler. But I'm not, and I've already spilt too much ink discussing this one problem.

What we can do instead is to fix the code where we skip over 0 and decide that negative dimensions should yield an empty board:

def buildCoordinateFrom(dims: Dims, shift: Int): Board =
  if dims.w <= 0 || dims.h <= 0 then Empty
  else NonEmpty(dims, shift)

Now whenever any of the dimensions hops into the negative, we decree it to be the Empty board. Does it work now?

The good news is that it no longer goes into an infinite loop, the bad news are that now some tests are actually failing.

We pass the unit test, and the first sanity check:

LazyFrameBuggySuite:
  + Traces the correct spiral 0.016s
  + Sanity: all points in bounds 0.033s

That unit test seems awfully resilient. But then we fail everything else.

Somehow we omit points from the solution³²:

==> X LazyFrameBuggySuite. Sanity: dimensions match number of points  0.023
43:
44:      points.size ==== dims.area
45:
- LazyFrameBuggySuite. Sanity: dimensions match number of points: Falsified after 10 passed tests
> Dims(1,1)
> === Not Equal ===
> --- lhs ---
> 0
> --- rhs ---
> 1

We also duplicate points, something that never happened before:

==> X LazyFrameBuggySuite. Sanity: no duplicate points  0.003s
49:
50:      points.distinct ==== points
- LazyFrameBuggySuite. Sanity: no duplicate points: Falsified after 13 passed tests
> Dims(1,3)
> === Not Equal ===
> --- lhs ---
> List(Point(0,0), Point(0,1), Point(0,2))
> --- rhs ---
> List(Point(0,0), Point(0,1), Point(0,2), Point(0,1))

Since we don't even pass the simple sanity checks, we obviously fail to match the reference solution:

==> X LazyFrameBuggySuite. Matches reference  0.009
13:
14:      actual ==== expected
- LazyFrameBuggySuite. Matches reference: Falsified after 10 passed tests
> Dims(1,1)
> === Not Equal ===
> --- lhs ---
> List()
> --- rhs ---
> List(Point(0,0))

Not for the first time, our comprehensive test suite is paying dividends. Something non-trivial is wrong here.

Of the sanity checks the duplicate points looks like the most informative one. Where can we get a duplicate point?

Since the output is small (but not too small, so it's "interesting"), we can trace the code and see what went wrong. The board we are dealing with is:

---------
| (0,0) |
---------
| (0,1) |
---------
| (0,2) |
---------

The board is not empty, so the first non-trivial step is to take the frame of the board with getFrame. Hmm... What is the frame of a single column?

Clearly it can't be this:

val path1 = Iterator.range(0, w - 1).map(Point(_, 0))
val path2 = Iterator.range(0, h - 1).map(Point(w - 1, _))
val path3 = Iterator.range(w - 1, 0, step = -1).map(Point(_, h - 1))
val path4 = Iterator.range(h - 1, 0, step = -1).map(Point(0, _))

val frame =
  path1 ++ path2 ++ path3 ++ path4

This implies that we have four sides to the frame, but on a column-only (or row-only) board we don't actually have four sides. This also explains why we have duplicate points. With a single column there's an overlapping contribution from path2 and path4 (and similarly for a single row).

And so, it appears that any board that ends up with a single row or column after peeling enough frames is an edge case in this new solution. We were unlucky that the unit test didn't catch it. But that's the nature of unit tests, you're either lucky, or you miss things. With a solid property-based suite, we can take most of the luck out of the equation³³.

Now that we isolated the issue, the fix is fairly straightforward, the frame of single row/column is just that row/column:

val frame = dims match
  case Dims(w, 1) => Iterator.range(0, w).map(Point(_, 0)) // 1
  case Dims(1, h) => Iterator.range(0, h).map(Point(0, _)) // 2
  case Dims(w, h) => // 3
    val path1 = Iterator.range(0, w - 1).map(Point(_, 0))
    val path2 = Iterator.range(0, h - 1).map(Point(w - 1, _))
    val path3 = Iterator.range(w - 1, 0, step = -1).map(Point(_, h - 1))
    val path4 = Iterator.range(h - 1, 0, step = -1).map(Point(0, _))

      path1 ++ path2 ++ path3 ++ path4

This is the code for the frame calculation, with two special cases, for a single row (1) and a single column (2). In both cases we just return the corresponding row/column. The last case (3) is the same as the old code.

Are we there yet?

LazyFrameSuite:
  + Traces the correct spiral 0.019s
  + Sanity: all points in bounds 0.036s
  + Sanity: dimensions match number of points 0.012s
  + Sanity: no duplicate points 0.013s
  + Matches reference 0.034s

Yep! We pass our test suite. And so we now have a solution that's both lazy and (despite the edge cases) has what I think of as simpler code, without all the complication of rotations³⁴.

That's when the product manager steps in with new feature demands...

Spiral Tracing as a Service

Our clients want to get specific points on the spiral on demand. They might say "give me point number 233 on a spiral on a 567x665 board" and our Spiral API™ should quickly produce a result³⁵.

Since our clients only ask for a single point on the board, and care about low latency, it would be wasteful to compute the whole spiral. What we should do instead is do the minimal amount of computation to get that one point.

Unfortunately, I'm not smart enough to come up with a closed-form formula taking an index to its location on the board's spiral³⁶. So all I can do instead is to fetch the point without materializing all intermediate points, and skipping as many steps as I can.

The new function that we'll implement is this:

def findPointAt(index: Int, dims: Dims): Option[Point]

Given an index and dimensions, we compute the point that matches the index (and None when out of bounds). More precisely, ignoring the efficiency requirements we could define:

def findPointAt(index: Int, dims: Dims): Option[Point] =
  Reference.traceSpiral(dims).lift(index)

This solution will first build the whole spiral with traceSpiral and access the element at index. lift turns indexing into a safe, Option-returning operation. Logically, this is correct, but we want something faster, that doesn't go over all the intermediate points, nor should it go to points beyond index. We'll call this new approach "computational".

For inspiration we can use the frame solution, as it has all the ingredients we need.

Recall that we are moving along frames of the board. We do that by skipping one frame at a time, and each frame starts on the next point on the main diagonal of the board:

-------------------------------------------------------------
|  0  |     |     |     |     |     |     |     |     |     |
-------------------------------------------------------------
|     |  1  |     |     |     |     |     |     |     |     |
-------------------------------------------------------------
|     |     |  2  |     |     |     |     |     |     |     |
-------------------------------------------------------------
|     |     |     |  3  |     |     |     |     |     |     |
-------------------------------------------------------------
|     |     |     |     |     |     |     |     |     |     |
-------------------------------------------------------------
|     |     |     |     |     |     |     |     |     |     |
-------------------------------------------------------------
|     |     |     |     |     |     |     |     |     |     |
-------------------------------------------------------------
|     |     |     |     |     |     |     |     |     |     |
-------------------------------------------------------------

The first frame is on the 0 offset, the second on 1, and so on. Each frame corresponds to a range of indices on the spiral. For example, on the board above, the frame on 0 corresponds to indices 0 to 31 (inclusive), the frame on 1 corresponds to indices 32 to 55 (inclusive). In the frame solution we managed to compute all the points of a given frame. By massaging that solution a bit, we can compute a specific index on a frame.

From this we can split up the problem into two parts:

Find the frame that contains the index we are looking for
Compute the point on the frame that matches the index

This way we are not wasting our time going around the frames, we just skip directly to the frame we need, and do some computation over it. Yielding a speed up over the original solution (or even the lazy one).

Let's code this up. As we had quite enough recursion for one post, let's go back to the procedural/iterative approach, using the recursive frame solution as inspiration (and remembering how to convert recursion to loops from the previous parts). We shall leave behind any pretense for functional elegance.

The first question about the algorithm above, is what are we iterating over? That's the shifts of the board (which correspond to frames), they start from 0 and keep growing, until we find our index.

The next question, is what do we do on every step of iteration? We need to check where the index is relative to the current frame. We have two possible cases, either the index is after the current frame, or within it. I said the word "case", so let's have this coded up as a datatype³⁷:

enum Framing:
    case After // 1
    case Within(frame: FrameData) // 2

The first case is self-evident (1). But why does the second case (2) have an argument? In the spirit of the great "parse don't validate" post, we (retroactively) predict that the point in the code where we discover that we are within the frame is also the location where we have all the data necessary to compute the result we are after. The Within constructor holds on to that data so that it can be used later on and not recomputed. The exact contents of FrameData will be specified later on.

For now, let's pretend that we have a function to compute the current Framing:

def getFraming(index: Int, shift: Int, dims: Dims): Framing

getFraming takes the index we are looking for, a shift on the board, and the dimensions of the board, it then computes the Framing of the index at the given shift. We'll implement this function in a bit.

The last thing we'll need before we proceed with the main loop is a way to convert the frame we found into a specific point that corresponds to the index we are searching for. Let us again pretend that we already have a function for that:

def getPointOnFrame(index: Int, frame: FrameData): Point

Given an index and a FrameData instance, we compute the corresponding Point.

Cool, with these ingredients in place we can implement the main loop:

def findPointAt(index: Int, dims: Dims): Option[Point] =
  if index < 0 || index >= dims.area then None // 1
  else
    var result = Option.empty[Point] // 2
    var shift = 0 // 3

    while result.isEmpty do // 4
      getFraming(index = index, shift = shift, dims) match // 5
        case Framing.After => shift += 1 // 6
        case Framing.Within(point) => // 7
          val point = getPointOnFrame(index, frame) // 8

          result = Some(point) // 9
    end while

    result // 10

The main loop has the following:

We verify that the index we got is not out of bounds (1), if it is, we return None³⁸
The variable result starts out as an empty value (2), to be set to the correct point later on
The initial shift is 0 (3), starting out at the top-left corner of the board, corresponding to the outermost frame
We set the condition for the loop (4), we iterate until a solution is found, that is, until result stops being empty
On each step of the loop, we check the framing for the index on the current shift (5)
If we aren't on the frame we are looking for (6), we increment the shift and move on
If we are within the correct frame (7), we also have the FrameData for it which we can use
We use the FrameData instance to convert the index into its Point (8)
Which we then assign to the result (9)
Once the loop is complete, result contains the correct Point, which we return (10)

Easy. All we have to do now is implement getFraming and getPointOnFrame. How hard could that be...

Recall how we moved around in the frame solution. We would increment the shift by 1, and then reduce the dimensions by 2, effectively working on an ever decreasing inner board.

Schematically, for shift == 2:

-------------------------------------------------------------
|  x  |  x  |  x  |  x  |  x  |  x  |  x  |  x  |  x  |  x  |
-------------------------------------------------------------
|  x  |  x  |  x  |  x  |  x  |  x  |  x  |  x  |  x  |  x  |
-------------------------------------------------------------
|  x  |  x  |  2  |     |     |  i  |     |     |  x  |  x  |
-------------------------------------------------------------
|  x  |  x  |     |     |  j  |     |     |     |  x  |  x  |
-------------------------------------------------------------
|  x  |  x  |     |     |     |     |     |     |  x  |  x  |
-------------------------------------------------------------
|  x  |  x  |     |     |     |     |     |     |  x  |  x  |
-------------------------------------------------------------
|  x  |  x  |  x  |  x  |  x  |  x  |  x  |  x  |  x  |  x  |
-------------------------------------------------------------
|  x  |  x  |  x  |  x  |  x  |  x  |  x  |  x  |  x  |  x  |
-------------------------------------------------------------

The xs mark the outer part of the board that we already passed. The empty squares mark the inner board, with its corner on shift number 2. i and j mark a couple of indices we might be looking for.

What does it tell us about the current framing? Where is 2 located relative to i or j?

From the drawing we can compute the index that corresponds to 2: it's just the area of the "outer board" marked by x. i is on the frame of 2, we can see that by computing the size of the frame of the inner board and adding it to the index of 2 and comparing that number to i. We can do the same thing for j to see that it's on another frame, after 2.

So to compute the framing we need to compute the area of the outer board, and the size of the current frame. With this insight, let's implement getFraming:

def getFraming(index: Int, shift: Int, dims: Dims): Framing =
  val totalArea = dims.area // 1
  val innerDims = dims.reduceBy(2 * shift) // 2

  val (frame = innerFrame, area = innerArea) =
    frameAndArea(innerDims) // 3

  val outerArea = totalArea - innerArea // 4

  if index >= outerArea + innerFrame then Framing.After // 5
  else
    val frame = FrameData(outerArea, shift, innerDims) // 6

    Framing.Within(frame) // 7

We start by getting the basic measurement of all the parts involved:

The total area of the full board (1)
The dimensions of the inner board (2), which we obtain by reducing the original Dims by double the shift number, as implied by the recursive frame solution
We then call frameAndArea on innerDims (3) to obtain its frame size and area in a tuple (which we intermediately deconstruct)
By knowing the innerArea and the totalArea we can compute the outerArea (4), which is just subtracting one from the other

Now we can figure out where the index is relative to the current frame:

If it's outside the outer area and the current frame, we return Framing.After (5)
Otherwise, we collect info about the frame into a FrameData instance (6)
And return a Frame.Within with that data (7)

For completeness, here's the definition of FrameData:

case class FrameData(outerArea: Int, shift: Int, dims: Dims)

In FrameData we hold on to the area of the outer board, the shift that got us there, and the dimensions of the (smaller) inner board. We'll see later on how to use this info to compute a point on the frame.

frameAndArea should've been relatively simple, but it suffers from the same edge cases like we had in the lazy frame solution. Lucky that we have that solution to learn lessons from. Accounting for this we get:

def frameAndArea(dims: Dims): (frame: Int, area: Int) = // 1
  dims match // 2
    case _ if dims.w <= 0 || dims.h <= 0 => (frame = 0, area = 0) // 3
    case Dims(1, h) => (frame = h, area = h) // 4
    case Dims(w, 1) => (frame = w, area = w) // 5
    case Dims(w, h) => (frame = 2 * (dims.w + dims.h - 2), area = w * h) // 6

This function returns a named tuple, with one field for the frame size, and one for the area (1). We do this by matching on the Dims we got. The first three case (3, 4, 5) are exactly the edge cases we seen before: negative dimensions and row/column-only boards. The last case (6) is for a full size board. If you're unnerved by all the random numbers here, or by the fear of off-by-one errors. Fret not, we still have the reference solution to back us up with its test suite (adapted so as to call findPointAt for all points of the board). So I fearlessly trial-and-errored this bit of code until it worked out. Also, fair warning, we are going to have much scarier "indexology" coming up very soon.

We are almost done, now that we know where we are relative to the current frame, we need to implement getPointOnFrame to be able to convert the Frame.Within into a concrete point. But first, let's solve a slightly simpler problem.

Recall that we have the "inner board", and the frame we found is its outer frame. If we take the perspective of the inner board (similar to how recursion worked in the previous solution), we can implement the following: given an index that is located on the outer frame of a board, find the point that corresponds to it. Once we have that, we'll need to convert the index into a corresponding index on the inner board, get the point, and then shift it back to the full board.

Be warned, the following helper is not for the faint of heart:

def getPointOnFrameInner(index: Int, dims: Dims): Point = // 1
  dims match // 2
    case Dims(w, 1) => Point(index, 0) // 3
    case Dims(1, h) => Point(0, index) // 4
    case Dims(w, h) => // 5
      index match // 6
        case _ if index < w => Point(index, 0)
        case _ if index < w + h - 1 => Point(w - 1, index - w + 1)
        case _ if index < w + h + w - 2 => Point((w - 1) - (index - w - h + 2), h - 1)
        case _ => Point(0, (h - 1) - (index - w - h - w + 3))

Let's break it down:

This is helper for getPointOnFrame, written from the perspective of the inner board (1)
The arguments index and dims refer to the inner board, not necessarily the originals we got in findPointAt
We match on dims, yet again, to suss out the edge cases (2)
We don't need to worry about empty/negative boards here, as we only reach this code when the location was found, which cannot happen on an empty/negative board
For single row/column boards (3, 4), the point is just the index itself (placed horizontally or vertically)
For a full board (5), we match on the index³⁹ (6)
Now comes the scary part, where we have to figure out the Point we are on based on the relative values of index, w, and h
Each branch corresponds to one side of the frame, like we had in the original frame solution
I won't even try to explain how I got these specific numeric values, suffice it to say that I still have nightmares

Never in a million years would I have gotten those values correct without a reference solution to keep me honest. The sheer number of back and forth off-by-one errors is more than I'm willing to admit. Unlike the other places before, one of the edge cases here was the surprisingly unintuitive 2x2 board. I had to play around with the values until the tests passed, and since we have property-based tests I'm fairly confident that I didn't miss any other weird edge cases.

How far we've come from the minimal, elegant, edge case free reference solution...

Anyways, implementing getPointOnFrame is now just a matter of shifting perspectives (pun intended) back and forth:

def getPointOnFrame(index: Int, frame: FrameData): Point =
  val innerIndex = index - frame.outerArea // 1

  val point = getPointOnFrameInner(innerIndex, frame.dims) // 2

  val shiftedPoint =
    point.shiftBy(dx = frame.shift, dy = frame.shift) // 3

  shiftedPoint // 4

To get the index as viewed from the inner board, we take the original index and subtract the outer area from it (1). This shifts the index to start from the top left corner of the inner board. We then call getPointOnFrameInner (2) with innerIndex and the dimensions of the inner board (coming from the FrameData). With the Point in hand, we must move it back to the full board's perspective. We do that by shifting the point (3) with the shift the frame is on. And this is what we return (4).

That wasn't that hard, was it? Was it?..

Since I wasted what feels like years of my life running the test suite on this solution till I got it right, I'm fairly certain we can say that having a reference solution was justified. We shall pause for a moment an appreciate the distance we made, starting from the lovely reference and the monster we have now. I wouldn't've made it this far without the experience and intuition that I gained from the previous correct solutions.

Are we done yet?

...And Then Make It Fast

Time to wow our interviewer⁴⁰.

It's true that our computational solution is asymptotically faster than the naive implementation with the reference. It's linear in max(w, h), while the reference is O(w * h). But when was any interviewer impressed with anything linear? Can we do better?

Turns out we can. Taking out our dusty CS books we notice that what we are doing here is searching: searching for a frame that matches a certain criteria. Not only that, but the things we are searching over are sorted: the group of indices in each frame are strictly monotonically increasing. This is almost a textbook case for a binary search. Who could've thunk that I'll ever be implementing binary search outside of school...

So instead of going from the 0 shift upwards, we can start from the middle, and go up or down depending on where our index is relative to the current frame. This requires only slight modifications to our computational solution.

For a binary search we need a lower and an upper bound. The lower bound is easy, that's just the 0 shift. But what is the upper one? Looking at the boards we have above we see that we only need to go to about the middle of the board. We'll use that as our upper bound.

Another thing we need to take care of is the Framing data type. When going from 0 upwards we only had two cases. Either the index was after the current frame or within it. But in a binary search we can go back and forth, so it might happen that index is before the current frame:

enum Framing:
    case Before // 1
    case After
    case Within(frame: FrameData)

We just added another simple case here (1).

The main loop for a binary search becomes the following:

def findPointAt(index: Int, dims: Dims): Option[Point] =
  val maxShift = (math.min(dims.w, dims.h) + 1) / 2 - 1  // 1

  var result = Option.empty[Point]
  var min = 0 // 2
  var max = maxShift // 3

  while result.isEmpty && min <= max do // 4
    val shift = min + (max - min) / 2 // 5

    getFraming(index = index, shift = shift, dims) match // 6
      case Framing.Before => max = shift - 1 // 7
      case Framing.After => min = shift + 1 // 8
      case Framing.Within(frame) => // 9
        val point = getPointOnFrame(index, frame)

        result = Some(point)
  end while

  result

This is still pretty similar to what we had before:

We initialize the maxShift (2) to the middle of the board, as per usual, the right numbers were found by random trial-and-error backed by tests
Two new variables are needed to keep track of the process:
- The current lower bound (2) that starts at the 0 shift
- The current upper bound (3) that starts at maxShift
We are no longer checking that index is within the correct bounds, instead relying on the stopping condition of the loop (4)
We stop the loop when either the result stopped being empty, or the lower bound went beyond the upper bound (meaning that the index is not on the board)
Inside the loop we set the shift to be in the middle between min and max (5)
With the current shift we can get the Framing for it (6), but now we deal with one of three outcomes:
- The index is before the current frame (7), we have to go backwards, so max is brought down to the current shift
- The index is after the current frame (8), we have to go forwards, so min is brought up to the current shift
- We are within the frame, in which we proceed the same way as before (9)

To finish this, we need to adapt getFraming as well. We just add another branch to the conditional logic there:

if index < outerArea then Framing.Before // 1
else if index >= outerArea + innerFrame then Framing.After
else
  val frame = FrameData(outerArea, shift, innerDims)

    Framing.Within(frame)

The Frame.Before case is produced when the index is in the "outer board" part (1). The rest remains the same.

Finally! We are done. This solution is logarithmic in max(w, h), way better than what we had before⁴¹. Both our clients and our interviewer will be thrilled by the snappy latency that the Spiral API™ can now provide.

Conclusion

Woah, that was quite a journey!

We started out by laying down a solid foundation with an (obviously) correct solution, backed by a property-based test suite. From there we used the initial correct solution both as reference to test against, and as an inspiration to help us think about the problem we are solving.

Guided by various requirements and improvements, we were able to both mechanically adapt the solution, and to use it as a guide and safeguard for more creative approaches. Yielding a diverse collection of solutions:

A simple recursive solution, closely following the problem statement
A tail recursive solution, to avoid stack overflows; by a mechanical transformation of the original correct solution
An iterative solution, to accommodate languages that don't support tail-call elimination; by mechanical transformation
A lazy solution, to avoid blowing out the memory; by using the original solution to inspire an interface to code to
A lazy frame by frame solution, to avoid the pain of dealing with rotations; by following the pain and using an existing solution for visualization
A computational solution, to avoid unnecessary computation when adapting to new requirements; by heavily relying on the reference property-based test suite to weed out edge cases
An even faster solution, just because we can and because we wanted to impress the hypothetical interviewer; by throwing a CS book at an existing solution

Starting out with a correct solution gave us clarity of thought about the problem, and a tangible way to test any other solutions that we can come up with. Property-based testing allowed us to maintain confidence even as we tried more and more daring approaches. Without the initial correct solution, none of this would be possible.

And so we circle back to my original statement:

First make it correct, then make it anything you want.

Did I mention that I hate leetcode problems?

I would like to consult with my attorney first. ↩
I.e., not too difficult. ↩
I've spent way too much time writing the code for this "visualization"... ↩
Or don't, I don't really care. ↩
Is he still considered a friend after that? ↩
The code for this post is available on GitHub. ↩
In case you get the directions wrong, which I obviously did on my first attempt. ↩
If you can come up with an even more obviously correct solution, please reach out and let me know. ↩
The ==== operator stands for "very much indeed equals". I guess the other = variants were already taken... ↩
In theory we can make it unbounded, but for the tests not to take too long, the specific generator I defined creates dimensions with a linear size of up to 30. ↩
Kind of like solving differential equations. The unit test is like the initial condition, and the properties are like the smooth function that acts as a solution. ↩
But we will still continue to use the sanity properties that we defined above, as they will be easier to use as debugging tools in case a candidate solution has a bug. ↩
Of course the specific numbers depend on the JVM parameters that we use. We could reduce the size of the stack on purpose and get a stack overflow first. ↩
I leave it as an exercise to my dear readers to figure out how to test for these flaws. ↩
In this post I'm using the Hedgehog library which does a good job at automatically maintaining shrinking even for custom types. If you're using something like ScalaCheck, you might need some extra code to achieve the same thing. ↩
What we are experiencing here is basically the difference between a foldRigt and a foldLeft. As illustrated here. ↩
Java, anyone? ↩
We could still use pattern matching, but then we'll have to somehow ignore the empty case, as it is already handled by the loop condition. Probably leading to some other form of unsafe code. ↩
For the life of me, I couldn't write a property that catches infinite loops... ↩
The other is being greedy. ↩
With a nod to Steve Yegge ↩
You could use proper streams for what I'm about to do, the resulting code will be very similar. ↩
For better or for worse. ↩
If we really wanted to preserve the pattern matching syntax from List we could define a custom extractor to achieve this goal. ↩
In the spirit of "make illegal states unrepresentable" we could come up with a more precise representation that won't allow us to express more than 4 rotations. Or at least make the constructor private to prevent the possibility of invalid values. For this small example it's probably not worth it. ↩
This is an annoying case of overloading. The Point type represents two different things: absolute points and relative deltas (shifts). We could resolve this by introducing a dedicated type for each use case, but the margin of this page is too small... There's some useful mathematics underlying this concept. ↩
Guess who had some off by one errors here... ↩
Having obtained this solution we can improve on it. E.g., the interface we came up with is more flexible than actually required. We don't need separate otherRows and rotateCCW methods, we always use them together. By combining them into one we can skip some intermediate steps in the solution. ↩
Unless it is... ↩
As an aside, why didn't we get to this solution in the original reference solution? One reason would be that the representation of the board that we used (a nested list of lists) is naturally suited for removing single rows (due to the way pattern matching on lists works), while removing a frame would require some coding effort. This is an example of "function follows form". Choose your forms wisely... ↩
To see the output below I'm using the -b flag passed to the test runner, to avoid buffering output. ↩
If you run the tests yourselves you might get different outputs, since minimization doesn't always reach the same cases. ↩
But not all of it, since the generators are still probabilistic. ↩
A "fun" riddle for the bored: suppose the board condition in buildCoordinateFrom was (wrongly) set to dims.area <= 0. What will happen? Will the test suite pass or fail? Why? ↩
Who said that leetcode problems are not realistic enough to reflect the demands of a real job? ↩
And if you come up with such a solution, please reach out and let me know. ↩
Okay, so I haven't completely let go of functional pretense. And you could solve this without a custom datatype, but it will be more elegant this way. Goes to show that you can mix the procedural and declarative approaches. ↩
We could skip this and just assume that the index input is valid, just like we do with dims. But I wanted this code to be compatible with the reference solution including the lift bit, which handles out-of-bound indices. I also coded up this condition as a property-test that you can see in the test suite for this solution. ↩
I'm using pattern matching as a cheap substitute for a multiway-if, which is something I usually avoid. Here it will get too clumsy if do it with regular if/else expressions. ↩
Assuming said interviewer is still with us after all this time. ↩
If you really want to take advantage of this speed up, you should probably stop using Ints though... Also see the riddle from above. ↩

Random Scala Tip #624: The Pitfalls of Option Blindness

Daniel Beskin — Thu, 01 May 2025 00:00:00 GMT

Given this class:

case class User(
  id: UserId,
  email: Option[Email],
  address: Option[Address],
  posts: Option[List[Post]],
  lastLogin: Option[Timestamp])

What does Option on each of the fields actually means?

Well, after spending hours digging in the code, documentation¹, and use sites, I gleaned the following:

email is actually mandatory, but that requirement was only added recently, so it was made optional for backwards compatibility with older clients
address is taken from user input, and it's not mandatory and can be missing
posts is fetched from a database, but some times the database is down, so we fallback to None, which is distinct from Some(Nil), and should be handled appropriately
lastLogin will be empty if the user never logged in previously, we need to provide a special greeting when a user logs in for the first time

We have four distinct meanings for Option, but only one data type to represent them all. This phenomenon is called "Option blindness".

Differently Typed Blindnesses

Boolean blindness is a well known phenomenon. It has a somewhat lesser known cousin called algebraic blindness. That's when we use very generic algebraic data types, like Option, Either, and These/Ior² to represent domain-specific data, and in doing so we miss that domain-specific context in our code.

Here I'm picking specifically on Option because it's so common. As it's often touted as a solution to the "null problem", it easy to think about it as an "never wrong" solution, even more so if you're new to functional programming. On top of that, Option enjoys a privileged status in the wider Scala ecosystem. Especially in various serialization libraries, where it commonly gets special handling, making it all the more tempting to use it all over the place.

The Tip

Before wrapping anything in Option, consider whether you should use a more meaningful domain-specific data type instead.

Examples

There are plenty of domain-specific meanings that an Option can take, but I'll use the four examples above. Hopefully this will illustrate the general principle well enough.

Backwards Compatibility

This one is particularly annoying, as it tends to get more widespread as the system evolves over time. You add a new field to a class that's constrained by backwards compatibility (e.g., it is used for communication, or stored in a database), and then you mark it Optional for perpetuity. Time passes by, and you no longer have any idea whether the field can be missing "for real", or it's just some vestigial Option supporting a no longer relevant version of code.

Instead, I would suggest to use a custom generic type, isomorphic to Option, to communicate the backwards compatibility concern directly³:

enum BackCompat[+A]:
  case Present(value: A)
  case Missing

How you end up treating such BackCompat values is very domain-dependent, maybe you can replace them with A over time, maybe not. But in any case, at least now you know why they are there and treat them accordingly.

Missing User Inputs

For user input that is not mandatory I'm on the fence, this might be the case where just using Option is the right fit, with all the convenience that you get from using a standard data type. The data is there or not, and that's all you care about. On the other hand, knowing that the data came from user input, and that it was explicitly not provided, might be relevant to your domain, in which case a custom Option-like data type (see above) may be in order.

This dilemma highlights the point that choosing the correct way to model your data is to an extent an art rather than just science.

Feature Toggle

The next example is using Option to indirectly communicate that a certain feature in the system (data fetching in the example) was turned off. That means that we have to remember that None has a special meaning. It's not just an empty value, but instead it's empty because a feature was turned off, and we might want to apply special handling to it. Using the terminology from the boolean blindness post, the provenance of the Option is just as important as the value of the data itself.

Instead of using Option we can communicate the provenance with yet another domain-specific, Option-like data type:

enum FetchedData[+A]:
  case Available(value: A)
  case Disabled

Now, every time someone has a FetchedData value, they have to decide what to do with it based on the informatively named cases. No longer do we have to remember the special meaning of None, the type system will guide us in the right direction.

One day, if and when we'll have more data fetching toggles, it'll be easy to add them to the custom FetchedData type, and then let the compiler guide us with fixing up the code⁴. Piggybacking on Option for this might get cumbersome.

System State

The last example is when Option is used to represent different states of the system. Now None means that the user never logged in. Information that we have to hold in our heads rather than in the type system⁵.

Unsurprisingly, the solution would be to create a domain-specific data type. But to add some variety, for this case I will go with a non-generic type. The "last logged in" state doesn't have a meaningful value that's not a Timestamp (at least in my interpretation of this example).

The state machine for this part of the system can be represented as:

enum LastLogin:
  case At(value: Timestamp)
  case Never

With this little state machine the compiler itself will remind us of whatever special behavior that we need to apply when a user logs in for the first time.

Trade Offs

Like most things in life, using custom data types is a trade off. Even if we ignore the boilerplate of actually defining a new type every time, we still pay in at least two ways:

We lose all the standard Option functions
We cannot automatically participate in whatever special treatment that Option gets in various libraries

We can try to mitigate the first point by defining conversions to/from Option and then use those to gain back some of the Option functionality⁶. Even better, if you're into functional programming you can use some standard typeclasses to imbue our custom types with Option-like functionality⁷. Some libraries will even let you derive those typeclasses automatically.

For the second point, special Option handling is very common in serialization libraries, e.g., converting Option to null in JSON. If you're lucky, whatever special functionality Option gets is guided by implicits. In which case custom data types can participate just as well (at the price of some more boilerplate). If you're not that lucky, and Option handling is hardcoded into the library, then you can try to petition the library authors to change that. Feel free to use this post as a form of justification.

Despite these cons, I still think that the maintainability gains we get from domain-specific, Option-like data types often outweigh the downsides.

May you never be blinded by Option ever again, till next time!

Obviously unmaintained and misleading. ↩
Like here, or here. ↩
I'm using Scala 3 syntax which is pleasantly compact. The same applies to Scala 2 albeit with noisier syntax. ↩
Assuming that you have pattern match exhaustivity errors turned on, and if not, go now and enable them. What are you still doing here? Go! ↩
Going down the route of moving system states into the type system is a good first step towards "making illegal states unrepresentable". ↩
If you're feeling particularly adventurous you can make those conversions implicit. ↩
Things like Functor, Applicative, or even Optional ↩

Whiteboxish Macro Powers with Named Tuples

Daniel Beskin — Mon, 14 Apr 2025 00:00:00 GMT

I've recently stumbled upon a post on Scala Contributors by Kit Langton describing how he managed to create the illusion of adding automatically derived (publicly visible) members to a companion of a class. Something that's no longer possible to achieve with macro annotations in Scala 3. This was done by daisy-chaining a number of features together: transparent macros, Selectable types, and implicit conversions. Read the full post for the details.

As Kit mentions, the solution is just 2% shy of syntactic perfection, which is good enough for me. But another downside is the fact that it uses transparent macros, meaning that IDEA cannot infer anything about the result. And that's a bummer.

In an entry below Kit's post, Guillaume Martres, mentions that we can get rid of the implicit conversions using a new feature: "computed field names", a sub feature of named tuples. This got me curious, as I suspected that we can make it even better and get rid of the transparent macros as well. Maybe IntelliJ can join the party after all?

You can see the full example code for this post in the repo.

Computed Field Names

For the purposes of this post I'll assume that you're already familiar with named tuples, and if not, please watch Jamie Thompson's excellent talk from Scalar. Additionally, basic knowledge of the mechanics of macros is also assumed (see here if not).

But let's see what "computed field names" actually means with a small example (the video above contains an explanation as well):

object Foo extends Selectable: // 1
  type Fields = (hello: String, meaningOfLife: Int) // 2

  def selectDynamic(field: String): Any = // 3
    field match // 4
      case "hello" => "world"
      case "meaningOfLife" => 42
      case _ => sys.error("cannot happen unless `selectDynamic` is called directly") // 5

Here we are defining a new Selectable object (1) to enable "dynamically" computed fields. In (2) we define the type member Fields which is a named tuple that specifies the members that the compiler will allow us to select on Foo. These are the "computed field names". The named tuple that we use here can come from an arbitrary (type-level) computation. Something that'll we'll take advantage of later on.

Next we implement field access with selectDynamic (3), which receives the name of the field being accessed as a string, and returns Any. Although nothing is enforcing the correctness of the selectDynamic implementation, type safety is somewhat retained by the fact that the user is only allowed to call the fields that are specified in the Fields named tuple (almost, see below).

With the field name in hand, we match on it (4), and produce the appropriately typed result (as specified by Fields). Though we return Any here, the compiler will cast the value to the correct type at runtime. We must be careful to obey the contract, otherwise we'll get a runtime cast exception.

Unfortunately, despite having a solid, typeful contract with the Fields tuple, nothing is stopping the user from calling selectDynamic directly with an arbitrary string. We handle that with a custom error (5), just in case.

With the computed field names, Foo now behaves as if we added the hello: String and meaningOfLife: Int fields to it. Which we can safely use, like so:

val a: String = Foo.hello
val b: Int = Foo.meaningOfLife

But this won't compile:

Foo.stuff

As stuff was not specified in Fields¹.

This new capability of Selectable is a nontrivial improvement over what we had before. Previously we would typically have to use macros to compute the precise type of a Selectable. Now we can use this new power to recreate Kit's example sans the transparent macros.

Automatic Lenses

The example in the original post tackles the problem of automatically adding lenses for the fields of a case class to its companion. I'll shamelessly steal the same example here.

In the post we have the following code that we want to automate:

case class Person(name: String, age: Int, isAlive: Boolean)

object Person:
  val name = Lens[Person](_.name)(p => name => p.copy(name = name))
  val age = Lens[Person](_.age)(p => age => p.copy(age = age))
  val isAlive = Lens[Person](_.isAlive)(p => isAlive => p.copy(isAlive = isAlive))

As we can see the code here is very mechanical and tedious. It can be described algorithmically as:

For each field of the case class
Add a Lens val to the companion with the field's name
Implement the val with a getter/setter pair for the field

In Scala 2 we could use annotation macros to automate this completely:

@deriveLenses
case class Person(name: String, age: Int, isAlive: Boolean)

This would automatically add all the necessary vals to the companion of Person, without us even needing to name it ourselves. Unfortunately, this is no longer possible in Scala 3, as macro annotations are not allowed to define new, externally visible, members on a class.

Computed field names to the rescue!

Computed Lens Fields

As we saw before, we can create the illusion of added field names by specifying the appropriate Fields named tuple on our class. What would it be for the example above?

type PersonLenses = (name: Lens[Person, String],
                     age: Lens[Person, Int],
                     isAlive: Lens[Person, Boolean])

If we pass this as the Fields member of a Selectable, we will get the appropriately typed values on our target class. But of course, we wouldn't want to do it manually, we want to compute this automatically from the Person type.

We can use the various tuple utilities to achieve this. The first stop is to convert the Person type to the equivalent named tuple. The magic NameTuple.From lets us do just that:

scala> type PersonFields = NamedTuple.From[Person]
// defined alias type PersonFields = (name : String, age : Int, isAlive : Boolean)

PersonFields now has the same labels and types as the Person case class. Now we need to manipulate the types so that they have the corresponding Lens wrappers.

First, let's define a type-level "function" that can convert a type to the matching Lens:

type ToLens[S] = [T] =>> Lens[S, T]

This is the equivalent of a curried function of two arguments, but at the type-level. The first argument is the source of the lens, the second is the target. We just curried the Lens type constructor. We can use it like so:

summon[Lens[Person, String] =:= ToLens[Person][String]] // compiles

(Using =:= to check the result as the REPL refuses to expand the resulting type.)

Now we can use this anonymous function to map over the named tuple:

import scala.NamedTuple.Map

type LensesForPerson = PersonFields `Map` ToLens[Person]

Leaning on the intuition that tuples are like collections, we import the Map "function" for named tuples², and then we use it as an infix operator to apply the ToLens "function" over the fields of Person. We partially apply ToLens to the Person type, so that it becomes the source of all the lenses that we get.

This automatically converts Person to the corresponding field lenses:

summon[
  (name: Lens[Person, String], age: Lens[Person, Int], isAlive: Lens[Person, Boolean])
  =:=
  LensesForPerson
] // compiles

Cool, we managed to automatically compute the lenses fields for Person. Of course, what we really care about is doing this generically for any case class, not just Person. It's easy enough to generify this code, just replace Person with A all over the place:

type LensesFor[A] = NamedTuple.From[A] `Map` ToLens[A]

Short and to the point.

This still works the same for Person:

summon[
  (name: Lens[Person, String], age: Lens[Person, Int], isAlive: Lens[Person, Boolean])
  =:=
  LensesFor[Person]
] // compiles

This is the key ingredient that we need to be able to avoid transparent macros. Once we computed the desired type of the output, transparency is no longer needed. Let's tie this all together into automatic lenses.

Automatic Lensy Members

We need one more ingredient to be able to implement the lens members: the actual lenses. For this we'll use a combination of givens and macros. First a helper type:

case class LensesMap[S](lenses: Map[String, Lens[S, ?]])

This is just a wrapper around a map of lenses, all pointing to the same source S. The intent being that S stands for our target class, e.g., Person. Each map entry is keyed by a field name with the value being the field's lens. Since the fields can have different types, we cannot type the lenses with anything better than ?. But that's okay, as we are going to hide this unsafety behind a safe Selectable interface.

For our Person example this map would look something like this:

LensesMap[Person](Map(
  "name" -> Lens[Person, String](...),
  "age" -> Lens[Person, Int](...),
  "isAlive" -> Lens[Person, Boolean](...),
))

This is pretty much the same structure that we wanted to generate on the companion object but written out in an unsafe and stringly manner. Creating these values is also completely mechanical, and we can automate this with a macro:

inline given derived[S]: LensesMap[S] = ${ derivedImpl[S] }

def derivedImpl[S: Type](using Quotes): Expr[LensesMap[S]] = ???

We auto-derive LensesMap for a given type S. The implementation is a trivial adaptation of the code from Kit's original example, which you can see in the repo. Crucially, the macro is not transparent, it has a well-defined type known at the definition site of the macro.

With this building block we can finally create our automatic lens deriver:

// 1
trait DeriveLenses[A](using lensesMap: LensesMap[A]) extends Selectable:
  type Fields = LensesFor[A] // 2

  inline def selectDynamic(name: String): Lens[A, ?] =
    lensesMap.lenses.getOrElse( // 3
      name,
      sys.error(s"Invalid field name [$name]"))

The implementation is very short, but it packs a punch due to the power of named tuples. Let's break it down:

DeriveLenses take a type argument, the case class we want to create lenses for (1)
We also take as a given argument for the automatically derived lenses for the same type (1)
And extend Selectable to be able to dynamically add fields (1)
Next we compute the lens fields that correspond to the type we got (2), this ensures that we can safely access all the field lenses
Lastly, we implement selectDynamic by extracting the lens that matches the string field name we got as input, this is safe as LensesMap and LensesFor both refer to the same type

Done! But does it work?

First we need to apply derivation to the Person companion:

object Person extends DeriveLenses[Person]

Now all the lenses are available to us as if they are regular fields:

val person = Person("Alice", 42, true)

val name = Person.name.get(person)
val age = Person.age.get(person)
val isAlive = Person.isAlive.get(person)

println(s"Name: $name, Age: $age, Is Alive: $isAlive")
// Name: Alice, Age: 42, Is Alive: true

Great success. It all works, fully type-safe, with zero transparent macros. Computed fields really boosted our expressivity here.

Some observations:

This is not as efficient as the Scala 2 version, where the code compiled to regular field access. Here we have to convert from a string to get to the actual lens ³.
We lowered our syntactic perfection score to be 3% shy of perfection, due to the extra Person type parameter we have to specify. It may be possible to avoid it with another macro, but I don't see how it can be done without using transparent macros again.
The NameTuple.From type is very magical indeed, without it we would probably have to resort to using Mirror, which in turn would force us into implicit conversions like in the original code. Although the proposed modularity improvements might make this a non-issue even without From.

Having said that, I think that we got one step closer towards whiteboxish macro capabilities from the days of yore, without actually needing to be whitebox. Well done named tuples!

Aftermath

Recall that the motivation to avoid transparent macros (other than to just show that we can) was to enable IDEA to infer the lenses. We succeeded in removing the transparent macros, but as of writing IntelliJ is yet to support computed field names with named tuples.

There goes trying to invite IDEA to the party⁴...

But this unfortunately does compile and fails at runtime: Foo.selectDynamic("stuff") ↩
To gain more intuition you can see more examples of generic tuple manipulation in a talk of mine, or this blog post. ↩
We could somewhat improve efficiency by using a macro that creates a pattern match for the lenses rather than a Map. But this is still probably not on par with direct field access. Maybe some kind of trickery with inline matches can get us closer to direct field access. ↩
Jokes aside though, the Scala team at Jetbrains deserves much credit for having quite a bit of support for named tuples, even though they are still marked experimental. ↩

Random Scala Tip #568: Beware of Leaking Iterators

Daniel Beskin — Wed, 15 Jan 2025 00:00:00 GMT

The Tip

The Scala collections library defines an Iterator trait. As you might expect, a typical Iterator's implementation relies on mutable state.

And so for the today's tip:

Beware of leaking Iterator instances outside the scope you initialized it in.

But What's So Special About `Iterator`?

Glad you asked.

We're all functional adults here, we're avoiding mutable state anyways. And even when we don't, obviously we won't be leaking any mutable state outside some tightly guarded scope. So why am I singling out Iterator specifically?

The reason is that Iterator can be sneaky. It's returned in places you might not expect it, and it superficially behaves like any other (immutable) collection¹.

Let me demonstrate. Suppose you're working on this code²:

// 1
def process(datas: List[UserData]): Unit =
  datas
    .map(enrich) // 2
    .foreach(storeToDB) // 3

// 4
def enrich(data: UserData): EnrichedUserData = ???

// 5
def storeToDB(data: EnrichedUserData): Unit = ???

What you have here is a nice, lightly functional, data processing pipeline. The reason why we all love using functional collections for data processing. In detail:

The main process function (1) takes a bunch of UserData values
It then enriches each one (2) using the enrich function (4)
Lastly, it stores each enriched entry into the database (3) using the storeToDB procedure (5)

You can run this code with some sample inputs and get the following output:

Stored ID: 1
Stored ID: 2
Stored ID: 3
Stored ID: 4
...
Stored ID: 1000

Nothing fancy, I'm sure that most Scala programmers are familiar with such code.

But then requirements change. We're told that the database is getting overloaded with all the storeToDB requests. We scratch our head a bit and decide to do some batching.

Easy enough:

def process(datas: List[UserData]): Unit =
  datas
    .map(enrich)
    .grouped(100) // 1
    .foreach(storeToDB)

def enrich(data: UserData): EnrichedUserData = ???

// 2
def storeToDB(batch: List[EnrichedUserData]): Unit = ???

With a minor tweak to the code we managed to adapt to the new requirements³. By calling grouped (1) on our list, we split it in batches. We can now send each batch to the modified storeToDB procedure (2) to be stored as a whole batch.

Who doesn't love code that's so adaptable to fluctuating requirements? That's functional modularity at its best.

And you can run it again and see how each batch gets stored:

Stored IDs: 1, 2, 3...
Stored IDs: 101, 102, 103...
Stored IDs: 201, 202, 203...
Stored IDs: 301, 302, 303...
Stored IDs: 401, 402, 403...
...
Stored IDs: 901, 902, 903...

So far, we have no issues. But as always, requirements change yet again. This time we are required to add some monitoring to the batches that we produce. For monitoring we have the following procedure:

def monitor(batch: List[EnrichedUserData]): Unit = ???

It takes a batch and side-effectfully does some monitoring. Another requirement is that we don't want to delay our main flow due to monitoring, so we should run it in a "fire and forget" Future⁴.

That's all easy enough⁵:

def process(datas: List[UserData]): Unit =
  val batches = // 1
    datas
      .map(enrich)
      .grouped(100)

  Future(batches.foreach(monitor)) // 2
  batches.foreach(storeToDB) // 3

We now assign our batches to a variable (1) so that we can use it twice
We run monitoring in a separate Future (2)
And we do the data storing just like in the previous snippet (3)

Although the code got a bit uglier, it's not that terrible, so we move on with our day.

Until the production bug hits us in the face...

If we try to run this code like we did before we might get the following output:

Monitored: 100 items
Monitored: 100 items
...
Monitored: 100 items
Monitored: 1 items
Stored IDs: 1, 101, 102...

It seems that we monitored most of what we need to monitor, but we only stored one batch of mismatched IDs.

Or worse yet:

Exception in thread "main" java.lang.NullPointerException: Cannot invoke "scala.collection.IterableOnce.knownSize()" because "prefix" is null
        at scala.collection.immutable.List.prependedAll(List.scala:148)
        at scala.collection.immutable.List$.from(List.scala:685)
        ...
Monitored: 100 items
Monitored: 100 items
Monitored: 100 items
...
Monitored: 100 items
Monitored: 20 items

We only monitored some of the batches, but we did no storing at all, instead ending up with an exception.

What's going on?

The Case of the Sneaky Iterator

What just happened is that unbeknownst to us the grouped method returns an Iterator[List[UserData]]. Now, whether or not this is the correct choice of a return type for this method is above my pay grade. But it's there in the standard library (along with some other Iterator-returning methods like sliding) and so we have to deal with it.

The problem here is how invisible it is. Due to type-inference we didn't have to specify the type of batches, so we missed it there. And since Iterator being part of the collections library has a foreach method, there was nothing to draw our attention to the fact that we moved from a regular immutable collection to a mutable Iterator.

As we are sharing the batches value between two flows we get the weird and racy behavior that we just observed. Both flows are competing to consume the Itertor, and the results are mixed.

And this is why I singled out Iterator: it can sneak mutable state where you don't expect it, resulting in code that leaks Iterators into unexpected places.

Possible objection: "but I'm doing pure FP". Indeed, doing pure FP is great. But this means there's all the more reason to be careful with the standard collections, lest you inject an impure value into your pure computation and wreak havoc on your referentially transparent code.

Possible workaround: I might even go as far as adding a linter rule (for, e.g., WartRemover) to the build to forbid the usage of these somewhat dangerous, Iterator-returning, methods.

That's all for now, stay Iterator-safe.

Yet another instance of "overloading considered evil". ↩
The full example code is available on GitHub. ↩
This is a great use-case for a streaming library. Unfortunately not all of us get to live in the enlightened world of streaming. ↩
A terrible idea in general, but sometimes you just do what they tell you to do. ↩
Yes, I know the code looks a bit contrived, but it's "based on a true story". ↩

Random Scala Tip #697: Avoid Anonymous Functions as Dependencies

Daniel Beskin — Tue, 07 Jan 2025 00:00:00 GMT

The Problem

Imagine that you're building an app and somewhere in your code you write something like this:

class DoFinanceStuff(cc: CreditCardService):

  def grabClientMoney(card: CardData) =
    val amount = // ...

    // ...

    cc.processPayment(amount, card)

And like the good developer that you pretend to be, you want to write some unit tests to verify that you really are grabbing the client's money. Since the codebase pervasively uses dependency injection, this should be a breeze.

So you want to mock out the CreditCardService instance you have in the DoFinanceStuff. "No problem", you say to yourself, let's see what functionality we need to provide. You open up the CreditCardService trait, and lo and behold:

trait CreditCardService:
  def validateCard(cardNumber: CardNumber, expiryDate: ExpiryDate, cvv: CVV): ValidationResult

  def processPayment(amount: MonetaryAmount, cardData: CardDetails): TransactionId

  def getCardholderDetails(cardNumber: CardNumber): CardholderInfo

  def authorizeTransaction(cardData: CardDetails, merchantId: MerchantId): AuthToken

  def checkTransactionStatus(transactionId: TransactionId): TransactionStatus

  def calculateRewards(transactionAmount: MonetaryAmount): RewardPoints

  def updateCardLimit(cardNumber: CardNumber, newLimit: CreditLimit): CreditLimit

  def getBillingCycleInfo(cardNumber: CardNumber): BillingCycle

  def reportLostCard(cardNumber: CardNumber): CardStatus

  def initiateDispute(transactionId: TransactionId, reason: DisputeReason): DisputeStatus

  def freezeCard(cardNumber: CardNumber): CardStatus

  def getTransactionHistory(cardNumber: CardNumber, startDate: Date, endDate: Date): List[Transaction]

  // ...

And it goes on like this for another ~15 methods.

Now it could be argued that someone messed up with this trait and it should be broken apart. But you just wanted to write a measly test¹, not refactor the whole universe. Creating a stub implementation with this many ??? seems daunting, why do we need to suffer through so much boilerplate?..

A thought goes through your mind, "a mocking library will save me from this hell". But you promptly discard this thought having remembered that mocking libraries are evil and there's a whole other hell reserved for those that use them².

Finally you remember that you're using Scala, a language that supports and encourages functional programming.

It now seems obvious, we're only using one method from CreditCardService, we can replace it with an anonymous function. Like so:

class DoFinanceStuff(processPayment: (MonetaryAmount, CardDetails) => TransactionId)

  def grabClientMoney(card: CardData) =
    val amount = // ...

    // ...

    processPayment(amount, card)

Cool. Now we can easily provide a test implementation for this one function. The change that we need to apply to the production code is minimal:

val cc = CreditCardService(...)

val financialThing = DoFinanceStuff(cc.processPayment)

Here we use the fact that Scala automatically converts methods to anonymous functions.

All is great, we have very little boilerplate and writing the test is a breeze. No?

The Tip

Nope.

So here's the actual tip:

Avoid using anonymous functions as class dependencies.

Why? Ergonomics.

Imagine a realistic full size application, with all its transitive interdependencies all over the place. Now you look at this class header:

class DoFinanceStuff(processPayment: (MonetaryAmount, CardDetails) => TransactionId)

And you ask yourself:

What is the processPayment function doing?
Where did it come from?
Where was it initialized?
What are the relevant production implementations of this function?

All important questions to ask, and all might not have immediately obvious answers in a sufficiently ~~convoluted~~ realistic code. The problem with using anonymous functions like this is that no mainstream tool for Scala is equipped to answer those questions. Or more concretely, you can't hit "find implementations/references" on a function type³.

What do we do instead then? We bite the bullet, settle for a bit more boilerplate, and define a custom trait:

trait PaymentProcessor:
  def processPayment(amount: MonetaryAmount, cardData: CardDetails): TransactionId

Along with an implementation that simply forwards calls to a CreditCardService:

object PaymentProcessor:
  class Default(cc: CreditCardService) extends PaymentProcessor:
    def processPayment(amount: MonetaryAmount, cardData: CardDetails): TransactionId =
      cc.processPayment(amount, cardData)

Our original class now looks like this:

class DoFinanceStuff(pp: PaymentProcessor)

And finally the initialization becomes:

val cc = CreditCardService(...)

val financialThing = DoFinanceStuff(PaymentProcessor.Default(cc))

A single method trait (a SAM) is indeed a more boilerplate-y equivalent of an anonymous function. Testability remains just as easy as before. So what we did here is to pay some boilerplate tax to get the exact same functionality as before. Did we gain anything in return?

What we gained is that now all the questions that we wanted to answer before can easily be answered with any reasonable Scala IDE⁴. To wit:

What is the processPayment function doing?
Hit "find implementations" on the PaymentProcessor and find the relevant implementation (there's likely just one⁵).
Where did it come from?
Hit "find usages" on the PaymentProcessor type and get a good overview of where it came from
Where was it initialized?
Hit "find usages" on the relevant implementation from the first question and find the place it gets initialized in
What are the relevant production implementations of this function?
Hit "find implementations" and... You get the general idea

This is just the beginning. As you now have a named entity in your code there are lots of things that become better:

Giving names to things forces you to think about them more, and only good can come out of that
There's a natural place for documentation about this entity
It's easier to refactor, since both automated tooling can help and you can more easily find all the relevant usages
Future evolution becomes easier, as you can add methods to your new trait if the need arises ⁶
You've just taken the first step to breaking down the monster CreditCardService.

And the list goes on. None of those benefits are available for an anonymous function.

Caveat: as most things in life, this is a trade off. Sometimes all the boilerplate ceremony and naming effort is just not worth it, and you might prefer to go down the anonymous road. Use your judgement⁷.

Addendum⁸: One can also improve things a bit by using a type alias for the function type. And although it somewhat improves readability, and forces you to name the thing; it's still not as ergonomically convenient as a newly defined type (be it a trait, or some other wrapper). Type aliases are only searchable when they are explicitly named, and nothing is forcing you to name them when passing an anonymous function along, you just instantiate the anonymous function as you would any other function, without using the name of the alias.

Bonus Tip

You might've noticed that I paid for a bit more boilerplate than strictly necessary by introducing the Default class. I could've just created a quick method with an anonymous implementation, and since the trait is a SAM, it could've just been implemented with an anonymous function. The result is pretty much equivalent, but again I'd avoid the anonymous route.

This too is an ergonomics/tooling issue, but it seems that most tools are better at displaying named implementations rather than anonymous ones. And although I might use an anonymous implementation for testing, the "real" implementation is better be named (ideally with some consistent naming convention).

It's a shame to avoid nice language features due to minor ergonomic nits, but you can't have it all...

Hope you found these tips useful, or at least thought provoking. Let me know in the comments.

As they say, no good deed goes unpunished. ↩
You think I'm being hyperbolic, don't you? ↩
Although it's not difficult to imagine a Hoogle style query engine in the editor that can do this, we are not there yet. ↩
Was that an oxymoron? ↩
Coupled with an enstablished convention for naming the trait implementations (e.g., Default/Impl/Live/Prod) this becomes almost automatic. Too bad we can't express the convention in the type system. If you have ideas on how to do that, please reach out. ↩
For better or for worse, we don't want to end up with another CreditCardService monster... ↩
Or don't, it's up to you. ↩
Thanks Reddit. ↩

Purify Your Tests Episode IV: The Monoids Strike Back

Daniel Beskin — Mon, 02 Dec 2024 00:00:00 GMT

Welcome back to yet another episode of test purification. In the last part we explored another benefit of adding type parameters to our code, the ability to work with very lean mocks. It's been a while since the last part, so please take a look there to refresh your memory.

In this part, rather than further improving our test code, we are going to see how to leverage our newly minted type parameters to improve our production code.

Onwards and forwards!

A Long Time Ago in a Galaxy Far, Far Away...

Recall that last time we ended up with the following code¹:

class UberService[UserID, UserData, EnrichedUserData, Bookkept, Stored](
    fetcher: Fetcher[UserID, UserData],
    enricher: Enricher[UserID, UserData, EnrichedUserData],
    bookkeeper: Bookkeeper[Bookkept, UserData, EnrichedUserData],
    storage: Storage[Bookkept, Stored, EnrichedUserData]):

  def fetchAndStore(user: UserID): Stored =
    val data = fetcher.fetch(user)
    val enriched = enricher.enrich(user, data)

    val bookkept = bookkeeper.bookkeep(data, enriched)
    val stored = storage.store(bookkept, enriched)

    stored

With a glorious sum total of 5 parameters, we managed to make this code highly testable with stateless mocks and maximally lean input data. Those were the days...

Unfortunately, since then, someone in management discovered that what we so bravely called UberService is actually a glorified ETL job. And so, management decreed, our service was demoted to be a batch job. Oh well... Might as well use this opportunity to refactor our code and improve it, that's the thing us developers love doing the most.

Stream Data, Stream...

As a batch job we now need to handle multiple users all at once. To make this manageable and fun, we'll use streaming to get the data into our process.

The new Fetcer now looks like this:

trait Fetcher:
  def fetch: Stream[UserData]

Instead of receiving a UserID argument and returning data for a single user, we now produce a Stream² of UserData. This is the all data that we need to process in our current run.

It also turned out that nobody was ever looking at the bookkeeping data and so we decided to surreptitiously drop the bookkeeping logic from our new job ³.

Lastly, since we are in batch mode, it would be more efficient to store data in batches so as to not invoke the storage procedure on per entry basis. So our data storage now looks like this⁴:

trait Storage:
  def storeBatch(data: List[EnrichedUserData]): Unit

With all those requirements in place, we can now write our new job:

class UberJob(fetcher: Fetcher, enricher: Enricher, storage: Storage): // 1

  def fetchAndStore: Unit =
    fetcher
      .fetch // 2
      .map(enricher.enrich) // 3
      .grouped(5) // 4
      .foreach(storage.storeBatch) // 5
      .to(Sink.discard) // 6

We have a new class appropriately called UberJob (1).
For now we forget all the lessons we learned about type parameters, and write the code without them.
The streaming pipeline now starts from fetching (2).
The data is enriched in a streaming fashion using map (3).
We seamlessly create batches using the streaming grouped operator (4).
Which we store with storeBatch, since this is a pure side-effect we use foreach rather than map (5).
To actually run the stream we use the to operator, which "pulls" the stream into the provided sink (6).
The argument to to is a Sink[In, Out] which is a sink that consumes entries of type In and produces a single result of type Out.
In this case, since we are only running the stream for its side-effects, the sink is Sink[Unit, Unit], which discards the stream's (Unit) output⁵.

Although I promised you that we'll be improving the production code using type parameters, let's start first by asking how testable is this code? A good question to ask, since it's often the case that testable code happens to be better in ways that are not directly related to testing⁶.

Tests, Tests Everywhere

Well, this code is at least as bad as the original code in the very first post in the series. Since all the flow is producing is Unit the only way to test the logic here is to use some form of mutable mocks. But just like in the first post, we can try to improve testability by (ab)using type parameters, which we will scatter liberally all through the code.

First the helper traits:

trait Fetcher[UserData]:
  def fetch: Stream[UserData]

trait Enricher[UserData, EnrichedUserData]:
  def enrich(data: UserData): EnrichedUserData

trait Storage[EnrichedUserData, Stored]:
  def storeBatch(data: List[EnrichedUserData]): Stored

We now have a type-parameter for every piece of data involved in the flow. Learning the lessons from the previous parts, this should mean that we can avoid mutable mocks (to inspect the outputs of each step) and can use lean fake data (for the inputs of the test).

Note the Stored type parameter that Storage now has. This stands for the Unit return type that storeBatch used to have, and should help us in reflecting the flow of data in the test, where we can choose Stored to be something more informative than Unit.

This leads us to this second version of UberJob:

class UberJob[UserData, EnrichedUserData, Stored]( // 1
    fetcher: Fetcher[UserData],
    enricher: Enricher[UserData, EnrichedUserData],
    storage: Storage[EnrichedUserData, Stored]):

  def fetchAndStore: Unit =
    fetcher
      .fetch
      .map(enricher.enrich)
      .grouped(5)
      .map(storage.storeBatch) // 2
      .to(Sink.discard)

The new code is fairly similar to the one we started with. We use a multitude of type parameters on UberJob (1). But the only real difference is that we use map rather than foreach when calling storeBatch (2), since the result is a non-Unit value.

The promise of all these fake type parameters is better testability. Is this what we got here?

Well, not quite. We still have Sink.discard stuck at the very end of our flow. Meaning that the output of fetchAndStore is still Unit, and is as untestable as ever⁷.

In the non-streaming variant we could return a single Stored value to reflect everything we wanted to know about the flow in the test. But in the streaming version we have (potentially) more or less than one Stored value, one per every batch stored. How do we reflect that in the final output?

One way to achieve this would be to swap out the Sink we are using in favor of Sink.toList⁸, a sink that accumulates all of the stream's output: Sink[A, List[A]]. If we choose the output of storeBatch to be List[X], where X contains all the data about a single processed entry⁹, we'll get the following signature:

def fetchAndStore: List[List[X]]

This solves the testing problems, we can get all the info we need for the test by embedding it inside the List[List[X]] value, just like we did in previous parts. But this comes at the cost of modifying the production code in a noticeable way. Now, instead of returning Unit we are going to return a List[Unit], needlessly allocating a (potentially) large list just to improve our tests. This is too high a price to pay for a bit of testability.

We can do better though.

Is There an Algebraic Abstraction in the Room?

The magic of using type parameters for testing stemmed from the fact that we can write the code once and instantiate the type parameters in two different ways. Once for testing and once in a way that's compatible with the original production code. That was parametric polymorphism in action.

In UberJob things are different, not only do we need two different return types (Unit vs. List[...]), but we also need distinct behaviors, one for each return type (discarding results vs. result accumulation). The whole point of parametric polymorphism is that the code's behavior is independent of the choice of type parameters. And so parametric polymorphism doesn't seem to cut it for us now.

Not all is lost though. Luckily, there are other forms of polymorphism we can take advantage of. Namely ad hoc polymorphism, and more specifically typeclasses, which allow us to choose different behaviors based on the specific instantiation of a type parameter.

Our task now is to find an appropriate typeclass and concrete types that capture both our production use case and the test use case. Let's try to "derive" the typeclass we need from the requirements that we have. Step by step:

Let's call the input to our final Sink In and the final output Out, i.e., we are working with Sink[In, Out].
In the production use case the requirement is that:
- type In = Unit - the result of the storeBatch in production.
- type Out = Unit - the whole process is a side effect, so we don't keep anything in memory from it.
Conclusion from 2: in the general case In =:= Out¹⁰, we are working with Sink[Unit, Unit].
In the test use case we have:
- For storeBatch we choose Stored to be List[X]
- From 3 we know that type In = Out = List[X]
Conclusion from 4: we need a Sink[List[X], List[X]].
The requirements for this Sink are:
- We need to be able to handle the empty case, if there's no data in the stream, the result should be List.empty.
- To be able to assert on all the relevant data, we want to combine all the individual List[X] values that we get¹¹, we can use ++ for that.

Ergo, we need a Sink that supports the following:

def mysterySink(empty: List[X])
               (combine: (List[X], List[X]) => List[X]): Sink[List[X], List[X]]

This signature has a very familiar shape. If we generalize this signature to arbitrary types, this looks exactly like fold¹²:
```
def fold[A](empty: A)
           (combine: (A, A) => A): Sink[A, A]
```
In the test this can be called as:
```
Sink.fold(List.empty[X])(_ ++ _)
```
The generic Sink.fold can be used with Unit as well, we just need to choose the appropriate implementations for empty and combine:
```
def empty: Unit = ()
def combine(x: Unit, y: Unit): Unit = ()
```
The types match, we still have a Sink[Unit, Unit], but we also avoid in-memory accumulation when running the production code, we just return Unit on every step.
To be (ad hoc) polymorphic over our inputs we need to support only two functions, empty and combine, since these are the only requirements for using Sink.fold.

Which brings us to the following typeclass:

trait MysteryTypeclass[A]:
  def empty: A
  def combine(x: A, y: A): A

If you've ever been exposed to functional programming, this should ring a bell, we just rediscovered Monoid¹³:
```
trait Monoid[A]:
  def empty: A
  def combine(x: A, y: A): A
```
Both List[X] and Unit have a Monoid implementation, one with List.empty and ++ and the other is the trivial Monoid with the code from 10.

We can use Sink.fold with any type that has a Monoid instance:

def foldMonoid[A: Monoid] = Sink.fold(A.empty)(_ combine _)

And so, using this rather lengthy step by step derivation, we conclude that we can make our code (ad hoc) polymorphic by requiring a Monoid instance on the Stored parameter:

class UberJob[UserData, EnrichedUserData, Stored: Monoid]( // 1
    fetcher: Fetcher[UserData],
    enricher: Enricher[UserData, EnrichedUserData],
    storage: Storage[EnrichedUserData, Stored]):

  def fetchAndStore: Stored = // 2
    fetcher
      .fetch
      .map(enricher.enrich)
      .grouped(5)
      .map(storage.storeBatch)
      .to(summarize) // 3

  def summarize: Sink[Stored, Stored] = Sink.foldMonoid[Stored] // 4

This is the final version of UberJob:

We now have a Monoid constraint on the Stored type parameter (1).
Meaning that we can run this code with any concrete instantiation of Stored as long as it has a Monoid implementation. The code will adapt its behavior accordingly.
Instead of Sink.discard we now call summarize (3), which is just (a slightly more fluent) way to call Sink.foldMonoid (4), which compiles thanks to the Monoid constraint.
Now the return type of fetchAndStore is no longer Unit but rather Stored (2), a value we can use in a test to collect data without affecting the production logic with Unit.

The promise of ad hoc polymorphism has been fulfilled, with the same code we get different behaviors based on types, matching the requirements of both the production and test scenarios ¹⁴.

But Is It Actually Testable?

Yes!

We can now write a purely functional test using an appropriate selection of types and implementations.

To wit, we select the following types for the test:

type UserData = String
type EnrichedUserData = String
type Stored = List[String]

Recall that using type parameters for the inputs lets us use "lean mocks", that is, we can choose whatever we want to be our inputs. In this case we choose everything to be String. We will augment the contents of the strings as we move between UserData to EnrichedUserData, finally reaching Stored which becomes a List[String] due to working with batches. Notably, the choice of type Stored = List[String] fulfills the requirement of having a Monoid instance.

Next, we implement the corresponding mocks:

object TestFetcher extends Fetcher[UserData]:
  def fetch: Stream[UserData] =
    Stream("data1", "data2", "data3") // 1

object TestEnricher extends Enricher[UserData, EnrichedUserData]:
  def enrich(data: UserData): EnrichedUserData =
    s"enriched: $data" // 2

object TestStorage extends Storage[EnrichedUserData, Stored]:
  def storeBatch(data: List[EnrichedUserData]): List[String] =
    data.map("stored: " + _) // 3

We produce some mock data in a stream in the Fetcher (1). We augment the string with the word enriched in the Enricher (2). And finally, for each entry in a batch, Storage marks it with the word stored.

Our mocks are just pure functions working with lean data, just like we learned to use type parameters in the previous installments.

Lastly, we have the actual test:

"The uber-service" should:
  "fetch the user data, enrich it, and store the results" in:
    val service = UberJob(TestFetcher, TestEnricher, TestStorage) // 1

    val result = service.fetchAndStore // 2

    result shouldBe List( // 3
      "stored: enriched: data1",
      "stored: enriched: data2",
      "stored: enriched: data3")

We initialize a new UberJob with our mocks (1).
We run fetchAndStore producing a List[String] result (2)
We assert that the List contains the correct entries (3)
Each entry contains traces of going through all the steps in the flow: enrichment and storage.

This test is once again a pure function¹⁵, taking inputs and producing deterministic outputs, thanks to the Monoid instance in the background that accumulates results for us¹⁶ ¹⁷.

The production code can still work just as before:

val service = UberJob(Fetcher.Default, Enricher.Default, Storage.Default)

service.fetchAndStore // produces `Unit`

The Monoid[Unit] instance doesn't accumulate anything, and fetcheAndStore only performs side-effects ending with a single Unit value.

This is all great, but somewhere at the beginning I promised you that the gratuitous addition of type parameters will improve our production code, not just tests. What was that all about?

A Product Manager Steps In...

And tells that we that our clients demand observability. More concretely, the data we're dealing with looks like this:

case class EnrichedUserData(
  userId: String,
  registered: Boolean,
  friends: List[String])

Our clients want to know:

How many users we processed
How many of them were registered
What is the average number of friends per user in the current run

That's a lot of data that we need to collect. How shall we approach this?

One thing we could do is to scatter some mutable variables in the flow and accumulate the statistics into them. But that would be iffy, because mutable variables suck¹⁸. And also, who wants to change code due to spurious product demands? Especially with such nice and clean looking code, polluting it with secondary concerns such as logging is unwarranted. Add to that the fact that we'll also have to modify the test we just written.

Lucky for us, our code is written generically for any instance of a monoid, and monoids are all about data accumulation (namely the combine function). If we can choose the appropriate monoid for the Stored type parameter we will get data accumulation for free. No need to modify fetchAndStore, nor will we need to change any of the existing tests.

Our first task is to come up with a type that will be accumulate all the data specified in the requirements. Here it is:

case class Stats(count: Int, registered: Int, friends: Int): // 1
  def averageFriends = friends.toDouble / count // 2

Here we have one field per a piece of data we want to count (1). Additionally we have a method to compute the average number of friends given the totals in the fields (2)¹⁹.

The next step is to provide a monoid instance for this type. For that we can just rely on the fact that integers form a monoid under addition, and a 0 as the empty value:

given Monoid[Stats]:
  def empty = Stats(0, 0, 0)

  def combine(x: Stats, y: Stats): Stats =
    Stats(x.count + y.count, x.registered + y.registered, x.friends + y.friends)

We also need a way to collect a summary from an EnrichedUserData instance:

def fromUserData(data: EnrichedUserData): Stats =
    Stats(
      count = 1, // 1
      registered = if data.registered then 1 else 0, // 2
      friends = data.friends.length) // 3

Each entry counts for 1 in the summary, when we combine Stats instances we'll eventually get the full count of entries (1).
We only add to the registered field if the current user is actually registered (2).
For every entry we accumulate the sum total of friends, ignoring the actual values (3).

Lastly, since we operate in batches, we'll actually need to create a Stats instance per batch:

def fromBatch(data: List[EnrichedUserData]): Stats =
    Monoid.foldMap(data)(fromUserData)

Here foldMap is a generic monoid function²⁰ that converts each EnrichedUserData list entry into a Stats instance, and then accumulates the results into a single Stats value using combine.

With all that in place we can participate in the monoids game we have going on here. The only thing that's missing is an actual Storage that produces Stats values instead of Unit.

Here it is:

object WithStats extends Storage[EnrichedUserData, Stats]: // 1
  def storeBatch(data: List[EnrichedUserData]): Stats = // 2
    println(s"Storing: [${data.map(_.userId).mkString(", ")}]") // 3

    Stats.fromBatch(data) // 4

This implementation chooses Stats for its output, which is compatible with the contract of UberJob as we defined a Monoid instance for Stats (1).
Accordingly, the output of storeBatch is now Stats (2).
For demonstration purposes we simulate data storage with an informative println that shows us the currently stored batch (3).
And then we summarize the current batch into a single Stats value (4).

Done! We have all the pieces in place to produce statistics about our job.

Here's a sample run:

val service = UberJob( // 1
  Fetcher.Default,
  Enricher.Default,
  Storage.WithStats) // 2

val stats = service.fetchAndStore // 3

println("---- Done ----")
println(stats) // 4

In this very simplified running script we instantiate an UberJob (1).
We swap the default Storage instance with the new WithStats implementation (2).
Then we run as usual, but unlike the original production code, the result is no longer Unit, but rather a Stats value (3).
We can further process the stats for the run, by e.g., sending them to some monitoring dashboard. Here we just print the stats to the console (4).

A sample output for running this code might look like the following²¹:

Storing: [a, b, c, d, e]
Storing: [f, g, h, i, j]
Storing: [k]
---- Done ----
Stats(count = 11, registered = 6, friends = 31, avgFriends = 2.8181818181818183)

We can see here how each batch is being stored, and at the very end we have all the statistics that we required.

Notice what we didn't have to do. We didn't further modify the code of UberJob, nor did we have to modify the test we've written. Everything just works due to the code being parameterized with the Monoid constraint, and the Monoid implementation for Stats doing the right thing²².

Conclusion

Adding a type parameter gained us flexibility that turns out to be useful beyond just tests. The type parameter revealed an essential truth about our code, a truth that we can use as requirements evolve. In this case the truth we got is about the accumulating nature of our flow. Constraining with different typeclasses can reveal different capabilities of the code.

The way we got to this code structure is also interesting. All we did was to try and improve the testability of the code, but what we got in return is more flexible and maintainable flow. This points to the even more general idea that it can be sometimes useful to think about code in a context that is more general then the one being currently solved for. Using type parameters is one (very powerful) way that we can remove code from its specific context. Even if you don't end up using type parameters in your final code, just thinking about code that way can also be beneficial.

Although the transformation to using type parameters was a bit more involved this time, with some practice you too will be able to spot those patterns and gain the benefits of parametric code.

That's it for this time. If you come up with any other benefits of using type parameters for testability, please reach out.

Happy parametrizing!

The full code for the examples is available on GitHub. ↩
I'm not specifying the particular streaming library that we are using here. The example code can be adapted to any reasonable streaming library. Although the code below looks a bit like (a blocking version of) Akka Streams, the ideas will apply to any library that uses explicit effects (like fs2 and ZIO Streams). You can see the concrete (toy) implementation here ↩
It's almost as if it was there only for pedagogic reasons to begin with. ↩
Seeing the way we use streaming in the code that follows, it could be argued that we can leverage streaming abstractions to create a streaming version of the Storage trait that can hide away the batching logic as an implementation detail, while only exposing an interface that receives entries one by one. Streaming is indeed quite a powerful code structuring method. We won't be doing that here so as to not get too bogged down in streaming mechanics. ↩
Notice how clean streaming makes this code, imagine doing this kind of batching "by hand". There are many benefits to using streaming (like how easy it would be to parallelize this code), but that's a topic for another post. ↩
Surprising exactly no one... ↩
Without mutable mocks that is. ↩
We could take another approach where the final Sink is passed in as a parameter to UberJob. But doing that will still force us to use a mutable mock for the test sink, one that mutably accumulates results. So we don't count that as a solution to the problem that we posed. ↩
E.g., in the tests we can choose X to be String. ↩
The production case constrains the general case. ↩
That's why we had List[List[X]] before. ↩
The actual signature in the repo is slightly more general than this, more similar to foldLeft[A, B] ↩
The actual code in the repo uses a slightly different syntax, using the currently experimental "better support for typeclasses" and support for infix operations, but the essence is still the same. ↩
It could be argued that we are still paying a bit of a tax for testability as we are needlessly "combining" unit values, something that wasn't happening in the original code. Although I don't think it really matters for any practical purposes, we can try to tackle this by parametrizing over the final Sink. Once passing a Sink.discard and once passing in Sink.foldMonoid. That might be okay, but would mean that we are not testing an important feature of the production flow: that the logic of Sink.discard does the right thing for our purposes. This is not an issue in the approach described in the main text as the code uses the same sink in both cases. And if you keep on reading we'll see another reason to favor the Monoid approach later on. ↩
If we go a step further and parameterize over the choice of streaming type, then given we don't use any too advanced streaming tools, we can choose List to stand for the stream in the test. Making the whole test a pure function over lists, which is something that's even easier to write tests for. ↩
Food for thought: our code is using monoids, and the test code is relying on lists, which are the "free monoid". It seems that more generally free structures are useful for testing. ↩
The eagle-eyed readers amongst you might notice that this doesn't check that batching was properly performed. I leave it as an exercise for the reader to choose the appropriate Monoid instance that can check that and amend the test accordingly. ↩
Although not specifically relevant to the current example, but if we add concurrency into the flow (like with some kind mapAsync), things get even more complicated when we mix mutability with concurrency. ↩
It's an unfortunate fact of life that the averaging operation is not itself a monoidal action. We cannot, in general, combine two averages two produce a new average value. ↩
One of the biggest advantages of using generic abstractions such as monoids in our code is that we gain access to a wealth of library functions that work with these abstractions, for free. [^The collection "posts" does not exist or is empty. Please check your content config file for errors.] ↩
If you're following along with the repo, you can run this yourself with the command:
```
scala-cli run . --main-class ncreep.streaming_purified.runUberJobStats
```
↩
Of course one would have to write tests for the new code we just written (the monoid instance and the new Storage implementation). But that's the beauty of the modularity that we gained: the new tests are completely self-contained and only need to deal with the new code. ↩

ANY Overloading Considered Evil

Daniel Beskin — Wed, 19 Jun 2024 00:00:00 GMT

This is not just clickbait, I really do mean that ANY form overloading in programming should be considered evil.

By "overloading" I shall refer to any instance of using one name to refer to multiple different things¹. A common example would be operator overloading, where the operator + can refer both to integer addition and to floating point addition.

Although I cannot possibly make an exhaustive list of all possible means of overloading, I shall try to illustrate this point with a few examples of different forms of it². Hopefully, by the end of this post you too will be of the opinion that overloading is quite evil. Though possibly a form of necessary evil.

On to our first example.

Operator Overloading

Overloading the meaning of arithmetic operators is so common that we don't even notice it. Even in high-school one can find the + operator overloaded to mean both real number addition and complex number addition.

Despite this innocuous seeming usage, in sufficiently complex applications, evil may lurk. Consider the following ³:

record Foo (int bar, double baz) {} // 1

record Bar (int qux) {} // 2

double doStuff(Foo foo, Bar bar) { // 3
  return (foo.bar / bar.qux) * foo.baz; // 4
}

In this code we have two records Foo (1) and Bar (2), containing some numbers. We then have a method doStuff (3) that takes instances of Foo and Bar and does some arithmetic (4). This code looks confusing, and that's on purpose. In a large enough codebase, following things might get difficult.

Now we can invoke this code as follows:

doStuff(new Foo(1, 2), new Bar(3))

And the result will be:

0.0

This is because we're doing integer division with 1 and 3. Nothing particularly interesting here. But now imagine that we are refactoring our code and deciding for some reason that the qux field should actually be a double:

record Bar (double qux) {}

Can you guess what the very same call to doStuff will return now?

0.6666666666666666

As we changed the type of qux to double the meaning of the / operator changed from integer division to floating point division. And so the result is now completely different.

Whether this is a bug in our application or not depends on context. But the point here is that we weren't warned about it in any way. The meaning of / changed silently without any kind of compile-time warning.

And given that the definition of doStuff can be far away from the definition of Bar, this is also an example of spooky action at a distance. Nobody likes spooky action at a distance...

It's reasonable for things to change when changing types, but it's not reasonable for it to happen silently and only be visible at run time.

The culprit here is the overloaded meaning of /.

Let's move on to the next example.

Literal Overloading

This was hinted at in the previous section. Note this call:

doStuff(new Foo(1, 2), new Bar(3))

What are the types of 1, 2, and 3?

The answer is: it depends. Depends on the expected types of the arguments to the Foo and Bar constructors.

As we changed the type of Bar.qux the meaning of 3 changed as well. Yet again, silently. If we had some compile-time warning about that change, we might've caught the change in the doStuff call. But alas, that's not the case. As the meaning of integer numeric literals is overloaded, we just silently changed behavior at run time.

I won't dwell on this point, as from my experience code usually doesn't rely too much on literals beyond tests and examples. And in any case, in most languages literal overloading is usually fairly limited⁴.

Method Overloading

A more common example of overloading, at least in the Java world, is the classic method overloading. That's when one class contains multiple methods with the same name but different type signatures. To wit, Arrays.sort:

public static void sort(int[] a)
public static void sort(long[] a)
public static void sort(char[] a)
public static void sort(byte[] a)
public static void sort(double[] a)
// ...

As you can see, there are many overloads of the sort method, accommodating different array types. All aptly named sort for convenience's sake.

Now imagine the following scenario. You have an array of prices, represented as double[]⁵. And you have some complicated logic to process them like so:

public void processPrices(double[] prices) {
  // ... do stuff with prices

  Arrays.sort(prices);

  // ... do more stuff with prices
}

Deep inside the processing logic we have an Arrays.sort hiding. But that's okay, it all works perfectly fine:

double[] prices = { 2, 5, 4 };

processPrices(prices);

After running processPrices the contents of prices is as expected:

[2.0, 4.0, 5.0]

Move along, nothing to see here...

But then we think to ourselves, is it good practice to represent business entities like "price" using raw doubles? Of course not.

Instead we decide to make a small domain wrapper record to represent the concept of "price" in our system:

record Price(double value) {}

This is a good practice, as we now have a point of reference and a single source of truth for anything that relates to the concept of "price" in our system.

As part of this refactor we now have to change the processPrices method as well:

public void processPrices(Price[] prices) {
  // ... the rest as before...
}

Great, this now all compiles and we can run the code from before:

Price[] prices = { new Price(2), new Price(5), new Price(4) };

processPrices(prices);

Any guesses as to the result of running this code?

...

Exception in thread "main" java.lang.ClassCastException: class overloading_post.
Price cannot be cast to class java.lang.Comparable (Price is in unnamed module of loader 'app'; java.lang.Comparable is in module java.base of loader 'bootstrap')
        at java.base/java.util.ComparableTimSort.countRunAndMakeAscending(ComparableTimSort.java:320)
        ...

Right, we got a class cast exception. That makes prefect sense.

How did that happen? Well, there is no Arrays.sort overload for Price, but one of the overloads is this one:

public static void sort(Object[] a)

It takes any array of Objects and tries to sort them. How can you possibly sort Objects? Obviously you can't as Object doesn't have any sorting methods. What you can do instead is to cross your fingers and cast an Object to Comparable and then sort that.

In our case though, Price did not implement comparable, and the cast failed spectacularly.

Now a lot of bad things are going on here. Using Object directly, array covariance, relying on casting ⁶, and of course, method overloading.

And yet, method overloading in Java is determined statically, the compiler knew that we just changed the type from double to Price, and it determined, at compile-time, to use the Object overload of sort. In another world the same compiler making all these decisions could've warned us that we are doing something suspicious, and ask us to sign off on that⁷. This, sadly, is not the world we live in.

The things we do for a bit of overloaded convenience⁸...

To paraphrase Benjamin Franklin:

Those who would give up essential Safety, to purchase a little temporary Convenience, deserve neither Safety nor Convenience.

Runtime Polymorphism

A staple of Object Oriented Programming, runtime polymorphism is arguably one of the most characteristic features of doing OOP. But it too falls under my loose definition of "overloading" given above: we are using a single name for an (abstract) method to provide two or more different implementations.

Unlike the examples above, runtime polymorphism, as the name implies only happens at runtime, and is not determined at compile-time. As a result, chaos must obviously ensue.

Continuing our Price example from above, suppose we need to do some further processing: we have to avoid using illegal prices (like negative ones). We also decided that using array is so 90s. We should modernize and program to interface, the List interface.

As a result we have this fine method:

void removeIllegalPrices(List<Price> prices) {
  prices.removeIf(price -> price.value < 0);
}

And we can use it like so:

List<Price> prices = new ArrayList<>();
prices.add(new Price(1));
prices.add(new Price(2));
prices.add(new Price(-1));

removeIllegalPrices(prices);

As you might expect, the result will be:

[Price[value=1.0], Price[value=2.0]]

Again, plain, old, boring Java code.

But then yet again the winds of change are blowing. Somehow you ended up with a copy of that newfangled book everybody's talking about, Effective Java, and you read about item 15⁹:

Minimize mutability

Sure enough, that rings true in the depths of your immutable soul, and you go on implementing this in your code:

List<Price> prices = new ArrayList<>();
prices.add(new Price(1));
prices.add(new Price(2));
prices.add(new Price(-1));

prices = Collections.unmodifiableList(prices);

There you go, item 15 successfully applied. And what's super convenient is that unmodifiableList returns an instance of the same List interface. So we can happily keep on using it like we did before.

I hope that by now you see where the path of convenience is leading us...

Running this:

removeIllegalPrices(prices);

Can only end in tears:

Exception in thread "main" java.lang.UnsupportedOperationException
        at java.base/java.util.Collections$UnmodifiableCollection.removeIf(Collections.java:1096)
        ...

The remove method (and by extension removeIf method) on List is overloaded to mean different things depending on the concrete runtime implementation. In one case it removes an element, in another it throws an exception. All these two scenarios share in common is a name. And that's plenty enough to wreak havoc in our code.

Silent breakage haunts us yet again. This time, though, we can't blame the compiler. Runtime polymorphism was not meant to be tracked at compile-time. The culprit is the (runtime) overloaded meaning of the remove method.

Just Random Things Breaking All Over

Okay, okay. This all sucks. But maybe, maybe it's just this outdated Object Oriented thing? Who does that anymore? I hear that all the young people are doing something called "Functional Oriented Programming", or some such. Surely this kind of legacy mess doesn't happen there.

We sure can join that party. Except we are still in the confines of our enterprise cage. So baby steps, baby steps. Let's use Functional Java. It's quite an oldie, so maybe our corporate masters won't object too much.

You're really sold on the whole functional thing. And you learned to "make illegal state unrepresentable". As a result you're using the NonEmptyList type religiously. This way you can enforce the presence of at least one item in your lists at compile-time, which can be handy for some business requirements.

For example:

record Debtor(NonEmptyList<Payment> payments) { // 1
  Payment maxPayment() {
    return payments.maximum(Payment.ord); // 2
  }
}

As unfortunately we are working for an evil corporation, we have to model some debtors. A Debtor has a list of payments owed to us. And here we make an illegal state unrepresentable: the list of payments must be non-empty, as otherwise there is no actual debt. This is modeled by the NonEmptyList type (1). We are enforcing at compile-time the business requirement we have in our code. And all is well.

Furthermore, this Debtor entity has a utility method maxPayment that returns the maximum from the list of payments owed by the debtor (2). We use this for reporting purposes¹⁰. Notice that unlike the legacy Java code, despite being able to invoke maximum on any instance of NonEmptyList we are obligated to provide an Ordering instance that tells the list how to actually choose the maximum. So no unsafe casting is involved.

Even more importantly though, this is completely safe. Since the list is enforced to be non-empty at the type level, the call to maximum can never fail, there must be at least one element in our list.

Safety... What a breath of fresh air. Enter the product manager...

Well actually, there's a new business feature, we now sometimes consider a person a debtor without an actual debt. Don't ask questions, it was all cleared with the accountants.

Right.

Our careful modelling now goes down the drain, and we obediently replace our NonEmptyList with the List type from Functional Java. As we still want to retain that immutable functional goodness with all the cool data processing APIs.

Now we have this:

record Debtor(List<Payment> payments) {
  // the rest as before
}

Where the List type here is not the builtin one, but rather fj.data.List. We can now support the new business requirement of debtless debtors. Well done.

Some time later...

Our reporting mechanism is randomly crashing. You dig into the logs and find this:

Exception in thread "main" java.lang.Error: Undefined: foldLeft1 on empty list
        at fj.Bottom.error(Bottom.java:29)
        at fj.data.List.foldLeft1(List.java:972)
        at fj.data.List.maximum(List.java:1624)
        ...

Recall our cool maximum method? Well apparently it's not safe for empty lists, as you cannot in any way produce an actual element (maximal or not). And yet the fd.data.List class defines the maximum method for, um, convenience. The reports for our debtless debtors now trigger maximum on an empty List and the result is this exception.

We can question the choice of having the patently unsafe maximum method on List. But be that as it may, the real culprit here is, yet again, overloading. List and NonEmptyList are completely unrelated classes, they only share and overload the method name maximum "by coincidence". There's no common interface that forces them to do that. As we changed the type of the payments field we silently changed the meaning of maximum in our code. The compiler knew what we were doing but chose to let us shoot ourselves in the foot.

The silence of the compiler is deafening...

And Now For Something Completely Different

You're finally tired of all these Java shenanigans and your enterprise cage and decide to move to a hip new startup.

What a wondrous world, they use the very niche Haskell language. The pinnacle of "functional oriented programming" and pretty much the polar opposite of Java, in being both principled and safe.

Even better, function overloading is forbidden in Haskell. Every imported function name must refer to exactly one function, otherwise you get a compile-time ambiguity error. It might be a bit tedious to muck around with all the qualified imports, but at least now you're safe from the woes of overloading.

But one day you discover that actually, Haskell supports a form of ad-hoc polymorphism called "type classes". And the moment you see a footgun you quickly go and shoot yourself in the foot.

As follows¹¹:

newtype Debtor = Debtor { payments :: NonEmptyList Payment} -- 1

maxPayment :: Debtor -> Payment
maxPayment = maximum . payments -- 2

Turns out you work for a debt collection startup...

We use AI and the blockchain to make debt collection as pleasant as possible to all parties involved.

This is basically exactly the same example as before but rewritten in Haskell. We now have a Debtor type with a single field of a NonEmptyList of payments (1). We then define a utility function maxPayment (2) which uses the maximum function to compute the maximum of the non-empty list of payments.

What is notable here is that maximum is part of the Foldable type class. And so is actually overloaded to mean different things depending on the target type.

As a result, when once again our dear product manager¹² forces us to break our invariants we end up with this:

newtype Debtor = Debtor { payments ::  List Payment }

-- the rest is the same

And once again our reports explode when stumbling on debtless debtors:

Prelude.maximum: empty list

Overloading just keeps haunting us. We changed types, and a function silently changed its meaning without warning. Since type classes are resolved at compile-time this is pretty much the same pitfall as we had with method overloading in Java ¹³. The compiler knew, but kept mum.

Does the flap of a butterfly's wings in Brazil set off a tornado in Texas?

Maybe we should quit programming altogether, maybe physical debt collection is a less overloaded field.

Can We Do Better?

I mean, we can forgo overloading altogether and completely forbid it at the language level. But truth is, overloading is very convenient. Can you imagine instead of using + you'd have to write +_int, +_double, +_short?..

On the other hand, all the pitfalls I showed above stem from the fact that the code silently changes meaning in the presence of overloaded definitions. I can imagine a language where every time an overloaded definition changes its meaning the compiler will flag it and ask for confirmation¹⁴. Maybe something like a magic import that states the current type of the overloaded definition, and if the import and actual type diverge, the compiler will fail compilation until the import is fixed. At which point you can decide whether you're doing the right thing when changing types.

Making this sufficiently ergonomic is left as an exercise to the reader...

ANY Overloading Considered Evil

Circling back to the original statement, I stand by the claim that ANY form of overloading should be considered evil. It may be necessary and even convenient, but make no mistake, where overloading exists, evil lurks not far behind.

This statement goes beyond any specific languages and even programming in general¹⁵ ¹⁶.

I purposefully chose the word "evil" because there's something sinister about the way it strikes you. Waiting for you to feel cozy with the typechecker, lulling you into a false sense of security, only to strike you when you least expect it.

Although I mostly used various Java warts to illustrate my point, this holds even without Java or those warts. Whenever you change types and your compiler lets behavior change silently and without your knowledge, it's a recipe for disaster.

The consequences of overloading can be more subtle than actual exceptions. Especially as code grows larger and we rely more and more on the compiler to track the correctness of our code. Like a silent performance degradation due to a change of collection types in some unrelated area of the code¹⁷. Or some accidental parallelism where none was present. Or improperly sorted data that you discover far away from the point of sorting. Or poorly formatted strings in your critical financial reports. Or anything really, let your imagination (paranoia?) run wild!

Thank you for reading this far, and may your future not be overloaded...

For our purposes here, Java-style method overriding is also a form of "overloading", as it too ascribes potentially different meanings to the same name. ↩
This is mostly in the context of statically typed languages as well, although it also holds to a degree in more dynamic languages. It's just that in statically typed languages we do expect to have more assistance from the compiler, which makes the evilness of overloading more apparent. ↩
I know this may seem a bit contrived, but in real code this can come up naturally on its own. It's just difficult to condense into blog form. ↩
A notable exception would be the Haskell language with its very flexible and overloaded numeric, string, and other literals. ↩
Of course one should never represent actual financial numbers with double, but we'll ignore this for the sake of the example. ↩
You could argue that I'm picking on geriatric code. The Arrays class began life in Java 1.2 (circa 1998), so it's probably older than some of the readers of this blog. And yet if you take a look at Stream, despite its veneer of modern respectability it has the sorted method, which works generically for any Stream<T>, and so it too can potentially throw a class cast exception. ↩
The Scala tool Wartremover has something that kind of does that, the Any wart (where Any is the Scala equivalent of Object). ↩
Method overloading can also exist with signatures of different arity. Although in the spirit of this post I should say something against this as well, for the moment I can't come up with a likely scenario where overloading over arity will break things silently. Reach out if you have examples of that as well. ↩
Or 17, depending on edition. I guess that numbering wasn't immutable... ↩
So as to depress our clients with the size of their debts. ↩
Here and below I'm using type aliases with type NonEmptyList a = NonEmpty a and type List a = [a]. This is not essential, I'm using the aliases just to make it a bit easier to relate the Haskell code to Java code above. ↩
Is it the same one we had in our enterprise cage? I can't tell them all apart anymore... ↩
Unlike Java though it's encouraged in Haskell for type classes to come equipped with laws. The laws help us reason about the behavior of the different implementations generically, making sure that all implementations "make sense". That's great, and better than nothing. But I still think you can stumble on overloading pitfalls even in the presence of laws. ↩
This can help with the statically resolved overloading cases, but won't help with all instances of runtime polymorphism (although some can be caught at compile-time, like changing the receiver type of toString in a non-generic context). ↩
I hear that human languages are known to be notoriously overloaded. Can you think of any problems that can cause? ↩
It's not uncommon for mathematicians to rely on overloading to make derivations more concise. Unfortunately for them, most of them don't even have a typechecker to begin with. ↩
The runtime polymorphic get method on List can have wildly different performance characteristics depending on the concrete implementation. Some algorithms may become unacceptably slow when you swap constant-time indexing with linear indexing. ↩

Purify Your Tests III: Lean, Mean Testing Machine

Daniel Beskin — Thu, 30 May 2024 00:00:00 GMT

In the last part we saw how to make our code more declarative, and the tests more functional by introducing type parameters for the inner data flowing through our code. In this part we are going to make our test inputs leaner.

Previously...

Recall that last time we ended up with the following code¹:

class UberService[Bookkept, Stored](
    fetcher: Fetcher,
    enricher: Enricher,
    bookkeeper: Bookkeeper[Bookkept],
    storage: Storage[Bookkept, Stored]): // 1

  def fetchAndStore(user: UserID): Stored =
    val data = fetcher.fetch(user)
    val enriched = enricher.enrich(user, data)

    val bookkept = bookkeeper.bookkeep(data, enriched) // 2
    val stored = storage.store(bookkept, enriched) // 3

    stored

The key insight from the last post is that we can use type parameters to enforce the call order between the Bookkeeper and Storage instances (1). We are forced to produce a Bookkept value (2) before being able to call store (3), and the only way to produce a Bookkept value is by calling bookkeep. This makes our code more declarative, and that much more type-safe. So much so, that it enabled us to delete a test.

Last time I suggested and quickly dismissed the idea of turning the inputs into type parameters as well. I (quite reasonably) claimed that the proliferation of type parameters might not be worth the potential gains. That may be the case in this specific example. But what if we imagine a slightly different scenario. One where our inputs are much more cumbersome.

This Is All Way Too Much

Till now, these were the classes representing the inputs to our tests:

case class UserID(value: Int)
case class UserData(value: String)
case class EnrichedUserData(value: String)

As you can see, these inputs our quite lean, just some wrappers around strings and ints. It's not a big deal to create fake instances of them in a test. This was perfectly fine for a blog post example, but it's usually not representative of real code.

Let's imagine that we had some more realistic inputs.

First stop, UserID is actually defined like so:

case class UserID private (value: Int) // 1

object UserID:
  def make(value: Int): Option[UserID] = ... // 2

Notice the private constructor (1), and the smart constructor make (2). UserID cannot be created willy-nilly, there are some invariants² to be maintained, and a smart constructor to enforce them. But do you care about that in the test? Do you want to deal with creating valid instances and handling that smart constructor just to pass data around in the test?

Next, the user-data is actually this monster:

case class UserData(
  id: UserID,
  username: String,
  email: Email,
  password: Hashed[Password],
  firstName: FirstName,
  lastName: LastName,
  bio: String,
  location: Location,
  birthDate: LocalDate,
  gender: Gender,
  profilePicture: Option[Media],
  coverPhoto: Option[Media],
  education: List[Education],
  workExperience: List[WorkExperience],
  interests: List[String],
  socialMediaAccounts: List[SocialMediaAccount],
  privacy: Privacy,
  followers: List[UserID],
  following: List[UserID],
  posts: List[Post],
  notifications: List[Notification],
  createdAt: Instant,
  lastLogin: Instant,
  isVerified: Boolean,
  // ... another 20 fields...
)

Not only does it have way too many fields, some of those are nested classes as well (like Post). How cumbersome would it be to create fake instances of this data in your test³? And for what, just to pass it around? Or maybe you're a well-disciplined functional programmer, and use randomly generated data a la property-based testing. That's great, but you will still have to create a gnarly generator for this purpose. The fact that you can use random data and not care might be hinting something...

You can imagine that EnrichedUserData is just as bad, maybe it has even more fields. I won't daunt you with an example, just add it to our overly heavy baggage.

The ongoing theme here is that we are paying the price in the test for something we really don't care about.

Hopefully, by now I'm becoming predictable and the solution intuitive. When code "doesn't care" about something it might be a fertile ground to introduce a new type parameter. Encoding the code's indifference at the type-level.

Every piece of data the details of which we don't care about in our flow and want to avoid faking in the tests can become a brand-new type parameter⁴ ⁵.

Fair warning, some individuals may find the number of type parameters in the following section disturbing. Reader discretion is advised.

Type Parameter Galore

Here's the new Fetcher trait:

trait Fetcher[UserID, UserData]: // 1
  def fetch(user: UserID): UserData // 2

Note the new type parameters, UserID and UserData. I'm breaking the FP convention of naming type parameters using single alphabet letters⁶. The reasoning being that these are not "really" generic things, they are just stand-ins for the real UserID and UserData classes. They exist merely to facilitate testing.

The real production implementation is going to set the parameters to the original classes, and otherwise remain unchanged:

class ProductionFetcher(/* ... */) extends Fetcher[UserID, UserData]:
  def fetch(user: UserID): UserData = ...

Just as before, the production code doesn't care about the type parameters, and can keep using the original classes. The actual benefit of using the type parameters will become apparent when we write down the tests. But first, let's parametrize the rest of the classes:

trait Enricher[UserID, UserData, EnrichedUserData]: // 1
  def enrich(user: UserID, data: UserData): EnrichedUserData

trait Bookkeeper[A, UserData, EnrichedUserData]: // 2
  def bookkeep(original: UserData, enriched: EnrichedUserData): A

trait Storage[Precondition, A, EnrichedUserData]: // 3
  def store(precondition: Precondition, data: EnrichedUserData): A

Similarly to Fetcher, we are replacing every occurrence of UserID, UserData, and EnrichedUserData with the type parameters (1, 2, 3). We then use them consistently in all the methods. Just like with Fetcher, the production classes can remain unchanged by setting the type parameters to the original classes.

Now for the main bit, UberService:

class UberService[UserID, UserData, EnrichedUserData, Bookkept, Stored]( // 1
    fetcher: Fetcher[UserID, UserData], // 2
    enricher: Enricher[UserID, UserData, EnrichedUserData], // 3
    bookkeeper: Bookkeeper[Bookkept, UserData, EnrichedUserData], // 4
    storage: Storage[Bookkept, Stored, EnrichedUserData]): // 5

  def fetchAndStore(user: UserID): Stored = // 6
    val data = fetcher.fetch(user)
    val enriched = enricher.enrich(user, data)

    val bookkept = bookkeeper.bookkeep(data, enriched)
    val stored = storage.store(bookkept, enriched)

    stored

UberService now takes an impressive total of five type parameters⁷ (1). We then consistently pass these parameters around the different classes (2, 3, 4, 5). But despite this apparent mess, the actual implementation code (6) remained unchanged⁸.

Now we are ready for the payoff.

Lean Fakes

To make things slightly less confusing⁹ I'm going to define the following aliases for the test:

type UserID = Int
type UserData = String
type EnrichedUserData = String

What I mean by that is that in the context of the test, the different type parameters are going to be chosen to be these simple aliases. All we have to do now is to reimplement our mocks using these parameters. To wit:

object TestFetcher extends Fetcher[UserID, UserData]: // 1
  def fetch(user: UserID): UserData =s"data: $user" // 2

The new Fetcher mock now uses the aliases UserID (actually Int) and UserData (actually String) for the type that it's going to be processing (1). And the implementation of fetch (2) can now use a plain string as its output. No need to deal with the huge "real" UserData class. Our fake data couldn't be much leaner than that.

Similarly, Enricher:

object TestEnricher extends Enricher[UserID, UserData, EnrichedUserData]:
  def enrich(user: UserID, data: UserData): EnrichedUserData =
    s"enriched: $user - $data"

Again, these are just the simple aliases, not the real "classes", we can use a lean string instead of some huge data class.

Finally, the Bookkeeper and Storage mocks. Recall that they were a bit of mess before because we had to pipe various tuples of UserData and EnrichedUserData through them. But now that we are no longer bound to these specific types, we can choose some other way of reflecting the data that flowed through UberService¹⁰.

The new Bookkeeper mock:

object TestBookkeeper extends Bookkeeper[List[String], UserData, EnrichedUserData]: // 1

  def bookkeep(original: UserData, enriched: EnrichedUserData): List[String] = // 2
    List(s"bookkept: $original", s"bookkept: $enriched") // 3

In this new mock we choose the output of Bookkeeper to be List[String] (1). Since both the UserData and EnrichedUserData types are actually plain strings now (1), we can just place them in a list, instead of using a tuple.

Accordingly, bookkeep now returns a List[String] (2). And we construct that list (3) in the implementation, by placing both inputs to bookkeep in the result. To make it a bit more obvious when asserting, we are adding a marker "bookkept" prefix to that string. This is possible because everything here is actually a String, we are not relying on the toString representation of the real UserData and EnrichedUserData classes, or anything evil like that.

Similarly, we modify the Storage mock to use List[String]:

object TestStorage extends Storage[List[String], List[String], EnrichedUserData]: // 1

  def store(bookkeepingResult: List[String], data: EnrichedUserData): List[String] =
    bookkeepingResult :+ s"stored: $data" // 2

To be consistent with the TestBookkeeper we are setting the input to TestStorage to be List[String], as well as the output (1).

The final result from store is now a compound list with both the Bookkeeper result and the entry from the storage process (2). Again, we are adding a string prefix to make it easier to see what's going on when we finally assert on the data. Which we are now ready to do.

Here's what the new test looks like now:

"The uber-service" should:
  "fetch the user data, enrich it, and store the results" in:
    val service = new UberService(TestFetcher, TestEnricher, TestBookkeeper, TestStorage)

    val result = service.fetchAndStore(5) // 1

    result shouldBe List( // 2
      "bookkept: data: 5",
      "bookkept: enriched: 5 - data: 5",
      "stored: enriched: 5 - data: 5"
    )

First we are running fethAndStore (1) with the ID 5. Note that this is a plain Int. No need to try to circumvent the smart constructor of the real UserID type. After running fetchAndStore we have a single result which is a List[String]. We can now assert on it (2) and see that we indeed have all the relevant outputs, both from Bookkeeper and Storage, as plain strings.

And that's it. Keep in mind what we didn't have to do, we never initialize any instances of "real" production classes, we never had to come up with a multitude of fake values for all the numerous fields of UserData. All we had to do is use some simple strings and ints, and yet this test is powerful enough to test real production code, code that can work with real, messy data.

This is the power of type parameters, they blend immense flexibility with extraordinary type-safety.

Was It All Really Necessary?

Recall that in the last post we talked about "non-fungible" type parameters. That is how in code that uses type parameters we cannot fake parametric values into existence out of nowhere. We are forced to "follow the types" and call the relevant producers of each type (assuming we are not cheating¹¹).

This allowed us to avoid testing various aspects of the code. Like the fact that we definitely called the Bookkeeper before calling Storage.

Now that we have even more type parameters, can we remove more tests?

If we look at the code of fethAndStore again:

def fetchAndStore(user: UserID): Stored = // 1
  val data = fetcher.fetch(user) // 2
  val enriched = enricher.enrich(user, data) // 3

  val bookkept = bookkeeper.bookkeep(data, enriched) // 4
  val stored = storage.store(bookkept, enriched) // 5

  stored

Is it even possible to not call the Fetcher or Enricher? Is it possible to pass the wrong inputs to Bookkeeper and Storage?

If we follow the types backwards from the return type, we observe the following.

We must return a Stored value (1), but this is a type parameter, so it cannot be faked. The only way to obtain it is to call store (5).
The only way to do that is to pass it a Bookkept value¹² and an EnrichedUserData value. But EnrichedUserData is now a type parameter, so there's no way to fake it as well.
To obtain an EnrichedUserData value we must call Enricher (3), but to do that we must provide it with a UserID and a UserData instance. Both are now type parameters.
UserData can only be obtained from a call to Fetcher (2).
UserID can only be obtained from the input to fetchAndStore (1).

Following this chain we see that the answers to the questions above is "no", we cannot mess these things up. The type-signatures force us into a certain chain of calls, that cannot be avoided.

This means that we could actually remove the test we just wrote, as it doesn't seem to test anything that is not already enforced by the type-system. Our types are so declarative that the compiler can just check everything for us. It's as if the types have become the code's specification.

If you're like me though, and have a slight measure of paranoia¹³, maybe one little sanity test like the one we wrote is not that bad. Especially when the resulting test is both pure and uses very lean fake data.

Conclusion

Following the pains of mocking data we made our code so parametric that it no longer cares about anything, allowing us to use the leanest possible fake data in our tests. This might seem an extreme measure, but depending on how difficult it is to create the fake data, it might be worth it.

Even if you don't end up using so many type parameters, the mere fact that you can do it tells us something important about the code: it's agnostic to all the data flowing through it, it's pure orchestration. Every time you see such indifference¹⁴ in your code you can take advantage of it to make your tests both leaner and purer.

On a different note. I'm not a big believer in Test Driven Development¹⁵. I don't think that it's necessary to write tests upfront, and I'm probably guilty of skipping the occasional test. But I'm a firm believer that making code testable tends to improve the overall quality of the code.

If writing tests for some piece of code is painful, maybe that code can be improved to be more testable, and as a side-effect¹⁶ the quality of the code itself will be improved.

I hope that this blog was a case in point. Our quest to make the code more easily testable made it so type-safe that it's nigh impossible to write it incorrectly. I think that's a win¹⁷.

In the next part we'll see how we can leverage our fake type parameters directly for the benefit of the production code, and not just for tests.

See you next time!

The full code for the examples is available on GitHub. ↩
Like value > 0. ↩
Don't fall for the luring trap of an Object Mother, only pain and misery can come from that. ↩
As if we don't have enough of them already... ↩
Actually the burning desire to avoid annoying fake data in tests was the initial trigger that got me into this approach of using type parameters to help with testing. You wouldn't believe the size of the class I wanted to avoid... ↩
Or the dubious Java convention of naming parameters with T, U, and V. ↩
Are we getting closer to the glorious septifunctor?.. ↩
Partly due to the unconventional naming of the type parameters, we can still pretend that we are working with the same classes as before. ↩
Or more. ↩
Though the old way with more granuraly typed tuples is still possible. ↩
By "cheating" I mean things like subverting the type system with nulls, casts, and the like. See the Scalazzi subset of Scala. I'll assume we are not cheating for the rest of the post. ↩
I'm ignoring Bookkept now because we already discussed it at length in the last installment. ↩
This is fairly simple code, but in the presence of something like IO we might want, for example, to sanity test that we are not accidentally short-circuiting where we are not supposed to. One might ponder whether introducing a type parameter instead of IO will enable us to skip these tests as well... ↩
And it tends to be more common than you might guess. ↩
Prepare the pitchforks... ↩
Pun intended. ↩
Although the exact number of type parameters such that the cost/benefit of using them is sufficiently worthwhile is up for debate. ↩

Purify Your Tests: 2 Parametric, 2 Declarative

Daniel Beskin — Mon, 20 May 2024 00:00:00 GMT

In the last part we learned how to purify our tests using type parameters. In this and following parts we'll see some further benefits of adding type parameters this way.

Let's dig in!

Previously...

Recall that last time we ended up with the following code¹:

class UberService[Bookkept, Stored]( // 1
    fetcher: Fetcher,
    enricher: Enricher,
    bookkeeper: Bookkeeper[Bookkept], // 2
    storage: Storage[Stored]):

  def fetchAndStore(user: UserID): (Bookkept, Stored) = // 3
    val data = fetcher.fetch(user)
    val enriched = enricher.enrich(user, data)

    val bookkept = bookkeeper.bookkeep(data, enriched)
    val stored = storage.store(enriched)

    (bookkept, stored)

This is our UberService, doing some uber-work. Note the "fake" type parameters (1). These are used by Bookkeeper and Storage (2) in place of Unit that they used to return. The fetchAndStore method returns these values (3), so that we can inspect them in the test (by setting the type parameters to something useful), and discard them in production (by setting them to Unit).

So far, what we gained is the ability to write pure, stateless mocks in our tests, and avoid mutable state altogether. Is that all though?

NFT Type Parameters

There's something interesting going on with the type parameters we introduced. Unless we're cheating², from the point of view of UberService there's no way of knowing what Bookkept and Stored is going to be, and so it cannot "fake" these values into existence³. UberService is forced to call Bookkeeper and Storage in order to comply with the return type of fetchAndStore.

This is what makes the resulting type signature so declarative:

def fetchAndStore(user: UserID): (Bookkept, Stored)

We know for a fact that there's no way to produce these values without using the provided Bookkeeper and Storage instances.

This property of type parameters being both "non-fungible" and declarative is something that we can take further advantage of.

A New Requirement

As sometimes happens, we just got a new requirement for our UberService. Turns out that the Bookkeeper service can sometimes fail by throwing an exception⁴. The new requirement is that it's no longer allowed to store the data unless bookkeeping succeeded. Can we guarantee this is the case?

The type signature of fetchAndStore says nothing about the ordering of bookkeeping and storage. Both Bookkept and Stored are returned, and there's no way of knowing which was invoked first. We could flip the invocation order, or even run in parallel and still have the same (Bookkept, Stored) result type.

That's okay, not everything can be enforced by types. Can we test that the new requirement holds?

Bad, Bad Tests!

We can actually check the invocation order of our mocks. But that will require us to use some form of side-effect/mutation. Like so:

class TestBookkeeper(invocations: ListBuffer[String]) // 1
  extends Bookkeeper[(UserData, EnrichedUserData)]:

  def bookkeep(original: UserData, enriched: EnrichedUserData): (UserData, EnrichedUserData) =
    invocations += "bookkeep" // 2

    (original, enriched)

class TestStorage(invocations: ListBuffer[String]) // 3
  extends Storage[EnrichedUserData]:

  def store(data: EnrichedUserData): EnrichedUserData =
    invocations += "store" // 4

    data

In these new mocks we now take a constructor argument (1 and 3). A mutable list that will store the invocations that the test is about to perform. Then when actually invoking the bookkeep and store methods we add the appropriate invocation name to our list of invocations (2 and 4).

In the test itself we can use the new mocks:

"The uber-service" should:
  "invoke bookkeeping before storage" in:
    val invocations = ListBuffer.empty[String] // 1

    // 2
    val bookkeeper = new TestBookkeeper(invocations)
    val storage = new TestStorage(invocations)

    val service = new UberService(TestFetcher, TestEnricher, bookkeeper, storage)

    val _ = service.fetchAndStore(UserID(5))

    invocations shouldBe List("bookkeep", "store") // 3

This new test first initializes an empty list of invocations (1). We then pass that empty list to our two new mocks (2). After invoking fetchAndStore we verify that the resulting invocations are in the correct order (3).

The test works, but yet again it sucks. We just spent a whole blog post eradicating mutation from our mocks. All that for nothing? Look how quickly mutation crept back in.

You might say that these concerns are just theoretical, the test works, leave it at that. And yet I won't leave it alone. Not only that I'm personally offended by mutability, but in the presence of concurrency this is a flaky test waiting to torment you in your CI pipeline forever. In a concurrent setting, which is quite likely for such an esteemed UberService, you might get arbitrary interleavings of bookkeep and store, and the test will randomly succeed or fail accordingly.

Are you willing to tolerate this?

Be Declarative Young Padawan

Remember way back when, a couple of paragraphs ago when I lamented that not everything can be enforced by types? Well, shame on me, I shouldn't have given up on them so quickly.

You see, there is a way to enforce invocation order using types. And it's one of the most elementary things that we have – functions.

Suppose I have a function f: A => B and a function g: B => C. If I start with A and end up with C, there's only one order in which f and g can be invoked. You must first invoke f to produce a B value, and only then can you pass it to g to produce the final C value.

But. This handwavy argument⁵ only works if B is "non-fungible". That is, if we have no way of obtaining B without invoking f.

All we need to do now, is to actually declare our intentions with the types, and to make sure they are non-fungible, or, in our case, parametric.

To that end, we are going to explicitly encode the relationship between Bookkeeper and Storage.

First we'll make our storage accept a precondition

trait Storage[Precondition, A]:
  def store(precondition: Precondition, data: EnrichedUserData): A

Now Storage takes another type parameter, which stands for a precondition that should hold before we can call store. We then modify store to accept the Precondition as an argument. That means that there is no way to call store without first obtaining a Precondition value. This signature declares our intentions more precisely, directly in the types. A Precondition value is proof that a function producing Precondition was invoked.

While Bookkeeper can remain the same, we'll now modify UberService to take advantage of the new precondition:

class UberService[Bookkept, Stored]( // 1
    fetcher: Fetcher,
    enricher: Enricher,
    bookkeeper: Bookkeeper[Bookkept],
    storage: Storage[Bookkept, Stored]): // 2

  def fetchAndStore(user: UserID): Stored = // 3
    val data = fetcher.fetch(user)
    val enriched = enricher.enrich(user, data)

    val bookkept = bookkeeper.bookkeep(data, enriched) // 4
    val stored = storage.store(bookkept, enriched) // 5

    stored // 6

UberService still takes the same two type parameters (1), but we now declare that the precondition for calling store is the presence of a Bookkept value (2). For reasons that will be clear a bit later, fetchAndStore no longer returns a tuple, instead it's only a Stored value (3).

Now for the main part. After we invoke bookkeep (4) and obtain a Bookkept value, we pass that value to store (5). We then return the Stored result (6).

Note how we no longer have a choice, if we want to obtain a Stored value we must invoke store after we successfully invoked bookkeep. Because the precondition for Storage is now the Bookkept value, and because the only way to obtain the (non-fungible) Bookkept value is by calling bookkeep, we are forced to observe our new ordering requirement.

Stated more explicitly, we now have the following chain:

bookkeep: (UserData, EnrichedUserData) => Bookkept
store: (Bookkept, EnrichedUserData) => Stored

Since Bookkept cannot be faked into existence, the only way to obtain a Stored value is to chain bookkeep with store

If we try to flip the order of store and bookkeep, we will fail to compile. If we forget to invoke bookkeep ⁶, we will fail to compile. We really have no choice. This is also the reason why we removed Bookkept from the return type of fetchAndStore, we already know for sure that bookkeep was called, there's no need to assert it again.

That's the power of having declarative types. By thoroughly declaring our requirements in the type system, we now have the compiler to enforce our invariants. Better yet, we can say that we "made an illegal state unrepresentable⁷".

All the while the production code can still remain completely oblivious to this:

class ProductionStorage(/* ... */) extends Storage[Unit, Unit]

The production implementation doesn't care about any preconditions, and just sets it to Unit. Nonetheless this cannot be abused by UberService to circumvent our rules. Since it's completely parametric, it cannot know that the real implementation has such an easily forgeable⁸ precondition, and so it cannot abuse that fact.

What's even better, we no longer need to write those terrible, mutable tests to check the invocation ordering requirement anymore. The compiler has our back. As they say ⁹:

The best tests are the ones you never had to write.

Even More Type Parameters?

The rush you get from deleting tests with impunity...

Can we delete more tests? Can we delete the original test we started with? Unfortunately, no, the original test is verifying that we are invoking Bookkeeper and Storage with the correct input. Unlike the outputs, the inputs of UserData and EnrichedUserData can be faked into existence, as they are concrete types.

Seeing the success we had so far with using fake type parameters, maybe we can apply this to the inputs as well. Indeed we can, but the proliferation of type parameters might not be worth it. In the next part I will find some justification to fake the inputs as well. But for now, we'll keep them as is and instead adapt the existing test to the new type signatures. This can be instructive.

Every Day I'm Refactoring...

Recall that we no longer have Bookkept as the output of fetchAndStore. Previously we used the Bookkept parameter as the vehicle to carry data about the invocation of bookkeep. How can we obtain this information now?

Swimmingly, it turns out.

On the one hand Bookkept is being piped into store, on the other, type parameters are so flexible we can put anything we want into them. The solution here would be to add the info from Bookkept directly to the Stored result.

The Bookkeeper mock remains exactly as before, it just produces its inputs as the output:

object TestBookkeeper extends Bookkeeper[(UserData, EnrichedUserData)]:

  def bookkeep(original: UserData, enriched: EnrichedUserData): (UserData, EnrichedUserData) =
    (original, enriched)

Now we know that this type (UserData, EnrichedUserData) is going to be fed to our Storage mock. Let's use it:

object TestStorage extends Storage[
  (UserData, EnrichedUserData), // 1
  ((UserData, EnrichedUserData), EnrichedUserData)]: // 2

  def store(
      bookkeepingResult: (UserData, EnrichedUserData), // 3
      data: EnrichedUserData): ((UserData, EnrichedUserData), EnrichedUserData) =
    (bookkeepingResult, data) // 4

This mock looks a bit scary now, but it's still essentially the identity function.

We align the Precondition type parameter to be the output of Bookkeeper: (UserData, EnrichedUserData) (1). The output is now modified to be both the original output of the Storage mock (EnrichedUserData) and a tuple of (UserData, EnrichedUserData) (2). This way we have access both to the data from Bookkeeper and the data from Storage in a single output.

In the store implementation, we get the output from Bookkeeper (3), which we then produce as part of the output of store (4).

With the flexibility of type parameters we managed to pipe all the data we need to be able to test the fetchAndStore flow. And now the test remains exactly the same as before:

"The uber-service" should:
  "fetch the user data, enrich it, and store the results" in:
    val service = new UberService(TestFetcher, TestEnricher, TestBookkeeper, TestStorage)

    val expectedUserData = UserData(s"data: 5")
    val expectedEnriched = EnrichedUserData("enriched: 5 - data: 5")

    val (bookkeeperResult, storageResult) = service.fetchAndStore(UserID(5)) // 1

    bookkeeperResult shouldBe (expectedUserData, expectedEnriched)
    storageResult shouldBe expectedEnriched

The output of fetchAndStore (1) has the same type and meaning as the output when Bookkept was part of the signature. The only difference is that now it comes through the store call. By plumbing the type parameters correctly we managed to reproduce the old functionality¹⁰ .

A Silver Bullet?

This is all great, unfortunately, it's not a silver bullet for all possible requirements we might get. As it stands, for example, there is no way to prevent the code from invoking bookkeep more than once. We are forcing the code to invoke bookkeep at least once, but it can accidentally (or on purpose) invoke it more than once.

I'm not aware of a way to use Scala's current type system to enforce such an invariant¹¹. Maybe one day substructural types will enter the mainstream, and will let us enforce such basic invariants. One can only hope...

For now, we'll have to make due with the unpleasant mutable mocks if we want to check for multiple invocations. If you come up with a way to avoid them in this scenario as well, I'll be glad to hear about it.

Conclusion

Type parameters are great, they are flexible, and enable us to enforce some non-trivial invariants. By making our code more declarative, we managed to promote a requirement from a runtime test into a compile-time verified fact¹².

More generally, what was it about our code that enabled us to apply these powerful techniques?

I would posit that a lot of the well-factored, testable code that we write tends to end up turning into glue code that is ignorant of the specific context it is being used in. It basically just orchestrates other code (like UberService), instead of doing much on its own. Type parameters lets us promote this ignorance to the type-level where we can reap the benefits that we just saw: declarative, type-safe code and pure tests.

In the next part we'll continue reaping the benefits of type parameters and handle those pesky inputs.

Till next time!

The full code for the examples is available on GitHub. ↩
By "cheating" I mean things like subverting the type system with nulls, casts, and the like. See the Scalazzi subset of Scala. I'll assume we are not cheating for the rest of the post. ↩
This is a loose application of "parametricity". ↩
We could use a more functional approach to error handling, but the same point I'm about to make will still stand. ↩
This is yet another loose application of "parametricity" ↩
E.g., after an accidental refactoring. ↩
As per Yaron Minsky. ↩
I mean, it's not that difficult to obtain a value of type Unit. ↩
I'm sure someone, somewhere said that. ↩
If we are being paranoid, it's true that since we are dealing with concrete types in the test, the Storage mock can now fake the output of Bookkeeper. If indeed we submit to the paranoia, we can easily make the mock "ignorant" of the input, and keep Precondition as a type-parameter, and just pipe that to the output. This will prevent TestStorage from messing around with the data flowing through it, just like it did in UberService. Incidentally, this will make the mock more flexible and reusable in other contexts as well. I leave this as an exercise to the curious reader. ↩
Maybe something with the still in research capture checking? Reach out if you have any suggestion on how this can be achieved. ↩
For a more involved example along these lines, see the last part of my "Make Illegal States Unrepresentable" talk. ↩

Purify Your Tests

Daniel Beskin — Thu, 09 May 2024 00:00:00 GMT

Here's a fun little trick to purify your tests. When I say "purify" I mean it in the pure functional programming sense of making the tests side-effect-free. But more specifically, I'm going to show you a way to avoid mutable mocks.

Testing Side-Effects

To make this concrete, we're going to use an example. Although this is Scala, the technique I'm about to describe is applicable in any statically-typed language with generics/parametric polymorphism, Java¹ to Haskell included.

So suppose you're writing your very cool UberService ²

class UberService(
  fetcher: Fetcher,
  enricher: Enricher,
  bookkeeper: Bookkeeper,
  storage: Storage):

  def fetchAndStore(user: UserID): Unit = // 1
    val data = fetcher.fetch(user) // 2
    val enriched = enricher.enrich(user, data) // 3

    bookkeeper.bookkeep(data, enriched) // 4
    storage.store(enriched) // 5

UberService is the pinnacle of engineering:

It has one method fetchAndStore (1).
As the name states, it does some data-fetching (2).
It then enriches the data (3).
Since this is an important process we do some bookkeeping (4) about the fetched/enriched data.
Lastly, we store the enriched result into some storage (5).

And because this is the pinnacle of engineering, we want UberService to be testable. To that end it doesn't actually do anything on its own, instead it only orchestrates the Fetcher, Enricher, Bookkeeper, and Storage instances. Since they are all abstract traits, we can mock them out in tests, and verify that UberService actually does what's written on the tin.

First we need to set up some mocks³ ⁴.

Here are the mocks for the Fetcher and Enricher traits. These are easy, they are just pure functions, on every input they return a deterministic output:

object TestFetcher extends Fetcher:
  def fetch(user: UserID): UserData = UserData(s"data: ${user.value}")

object TestEnricher extends Enricher:
  def enrich(user: UserID, data: UserData): EnrichedUserData =
    EnrichedUserData(s"enriched: ${user.value} - ${data.value}")

Next we need to mock the Bookkeeper trait. This one's trickier, since its functionality is a pure side-effect, all we "return" from bookkeep is Unit, which doesn't provide us any new information within the context of the test.

class TestBookkeeper extends Bookkeeper:
  var bookkept: List[(UserData, EnrichedUserData)] = List.empty // 1

  def bookkeep(original: UserData, enriched: EnrichedUserData): Unit =
    bookkept ::= ((original, enriched)) // 2

As a result, to be able to test proper usage of Bookkeeper we have to use some mutable state:

In (1) we initialize a var⁵ to hold the data that will be passed by UberService to our mock.
In (2) we add the data that was passed to the bookkeep method.

And similarly for Storage:

class TestStorage extends Storage:
  var stored: List[EnrichedUserData] = List.empty

  def store(data: EnrichedUserData): Unit =
    stored ::= data

Enough mocks, let's get to business. Now we can write a test for UberService:

"The uber-service" should:
  "fetch the user data, enrich it, and store the results" in:
    // 1
    val bookkeeper = new TestBookkeeper
    val storage = new TestStorage

    val service = new UberService(TestFetcher, TestEnricher, bookkeeper, storage) // 2

    val expectedUserData = UserData(s"data: 5")
    val expectedEnriched = EnrichedUserData("enriched: 5 - data: 5")

    service.fetchAndStore(UserID(5)) // 3

    // 4
    bookkeeper.bookkept shouldBe List((expectedUserData, expectedEnriched))
    storage.stored shouldBe List(expectedEnriched)

The test is pretty-straightforward:

We initialize some mocks (1).
Create a new a UberService instance (2) with these mocks.
Then execute the fetchAndStore method (3), which is the thing actually being tested.
And we assert the contents of our mocks, making sure they got the right data passed into them.

By doing all that we verified that UberService indeed performs the orchestration expected of it.

But this kind of sucks...

The Problem

The Bookkeeper and Storage mocks are annoying. Writing mutable mocks is tedious and error-prone⁶. Luckily for us UberService is completely synchronous, but what would happen to that poor var if we add some concurrency to the game? Not to mention race conditions and the like.

Needing to inspect the innards of our mocks to be able to write the test is a definite code smell.

To contrast, the mocks for Fetcher and Enricher are much simpler, they can be modeled by pure functions, given some input they produce some output. We do not have to deal with mutable state to see that they are used properly by UberService. Their outputs can be used as indication that the functionality of Fetcher and Enricher was employed correctly.

It all boils down to a single source of problems: we are trying to test side-effects. Since side-effects are not observable without other side-effects, we have to resort to using mutation to catch them.

But what can we do, Bookkeeper and Storage are side-effectful components. Do we have any other choice?

The Non-Solutions

What makes Fetcher and Enricher nice to work with is that they produce outputs when called, while Bookkeeper and Storage only return Unit. Well, what about adding some outputs to them. But what should we return? Here's one attempt:

trait Bookkeeper:
  def bookkeep(original: UserData, enriched: EnrichedUserData): Bookkept // 1

sealed trait Bookkept
object Bookkept extends Bookkept // 2

In this new edition of Bookkeeper instead of returning Unit we are returning a new type called Bookkept (1). Which is defined as a singleton (2).

Great, we now have an output, just like Fetcher. Can we get rid of the unpleasant mocks? Nope.

Since Bookkept is a singleton with no information beyond its existence, that is to say, isomorphic to Unit, we cannot use it to glean any information on how Bookkeeper was invoked. This is a dead-end.

Okay, so let's add some information to the output:

trait Bookkeeper:
  def bookkeep(original: UserData, enriched: EnrichedUserData): (UserData, EnrichedUserData) // 1

class UberService(/* same as before */):

  def fetchAndStore(user: UserID): Bookkept =
    // same as before

    val bookkept = bookkeeper.bookkept(data, enriched) // 2

    storage.store(enriched)

    bookkept // 3

In this version of Bookkeeper we are returning the data that we need for the test (1), a tuple of (UserData, EnrichedUserData). This should give us everything we need. To be able to use that data, we have to hold on to it in fetchAndStore (2), and then return it as the result of fetchAndStore (3).

This works, but feels kind of iffy. Why should the real production code know about the specific data that is needed for the test? And what if we have different kinds of tests that care about different pieces of data? Do we have to accommodate them all in our real code?

The resulting code is both too rigid and concerns itself with something it shouldn't care about. How can we make it both more flexible and more ignorant ⁷?

The Solution

When some code requires both flexibility and ignorance it is very likely that the solution is to add a type parameter. Let me demonstrate:

trait Bookkeeper[A]: // 1
  def bookkeep(original: UserData, enriched: EnrichedUserData): A // 2

Now Bookkeeper takes a type-parameter (1), and returns a value of that type as a result. This gives us maximum flexibility. In concrete implementations we can choose A to be whatever we want, it can be Unit for real production code, and it can be (UserData, EnrichedUserData) for test code. And the best part of it is that UberService doesn't care, it's ignorant of the concrete choice of A:

class UberService[Bookkept]( // 1
  fetcher: Fetcher,
  enricher: Enricher,
  bookkeeper: Bookkeeper[Bookkept], // 2
  storage: Storage):

  def fetchAndStore(user: UserID): Bookkept = // 3
    val data = fetcher.fetch(user)
    val enriched = enricher.enrich(user, data)

    val bookkept = bookkeeper.bookkeep(data, enriched)

    storage.store(enriched)

    bookkept // 4

This code is very similar to the first attempt of returning the Bookkept value. Unlike last time though, Bookkept is now a type-parameter (1), so UberService doesn't "know" what it's going to be, and doesn't care. The only thing we know is that the type parameter of the input Bookkeeper (2) is going to be consistent with the output type of fetchAndStore (3). All it does is pass it along.

Because the code is now completely generic (or ignorant) we can accommodate both the real use-case (Unit) and the test use-case ((UserData, EnrichedUserData)), without further modifications to UberService.

To wit, here's what the mock for Bookkeeper will look like now:

object TestBookkeeper extends Bookkeeper[(UserData, EnrichedUserData)]:

  def bookkeep(original: UserData, enriched: EnrichedUserData): (UserData, EnrichedUserData) =
    (original, enriched)

No more mutable state, we choose the type we need for the test, and the resulting mock is basically a pure function⁸, just like the mocks for Fetcher and Enricher.

The test itself can easily check that Bookkeeper was called with the correct input:

"The uber-service" should:
  "fetch the user data, enrich it, and store the results" in:
    // ...

    val service = new UberService(TestFetcher, TestEnricher, TestBookkeeper, storage)

    // ...

    val bookkeeperResult = service.fetchAndStore(UserID(5)) // 1

    bookkeeperResult shouldBe (expectedUserData, expectedEnriched) // 2

    // ...

We call fetchAndStore as before (1). But instead of Unit we have an actual result. Since we chose the type parameter of Bookkeeper to be (UserData, EnrichedUserData), this is exactly the data we had before in our mock, and we can assert on it as before (2).

What's more, it's not possible for UberService to "lie" to us with that Bookkeeper output. From the point of view of UberService there's no way to produce a Bookkept value without calling Bookkeeper, unless you're cheating⁹, parametric values cannot be faked into existence¹⁰.

While our test is becoming more civilized, the production code knows nothing about it. The production implementation can remain the same by setting A = Unit:

class ProductionBookkeeper(/* ... */) extends Bookkeeper[Unit]

Rinse and Repeat

We can repeat the same trick with Storage.

Add a type-parameter:

trait Storage[A]:
  def store(data: EnrichedUserData): A

Propagate the new type through UberService:

class UberService[Bookkept, Stored](
    fetcher: Fetcher,
    enricher: Enricher,
    bookkeeper: Bookkeeper[Bookkept],
    storage: Storage[Stored]):

  def fetchAndStore(user: UserID): (Bookkept, Stored) =
    val data = fetcher.fetch(user)
    val enriched = enricher.enrich(user, data)

    val bookkept = bookkeeper.bookkeep(data, enriched)
    val stored = storage.store(enriched)

    (bookkept, stored)

Create a mock that's a pure function:

object TestStorage extends Storage[EnrichedUserData]:
  def store(data: EnrichedUserData): EnrichedUserData = data

And use it in our test:

"The uber-service" should:
  "fetch the user data, enrich it, and store the results" in:
    val service = new UberService(TestFetcher, TestEnricher, TestBookkeeper, TestStorage)

    val expectedUserData = UserData(s"data: 5")
    val expectedEnriched = EnrichedUserData("enriched: 5 - data: 5")

    val (bookkeeperResult, storageResult) = service.fetchAndStore(UserID(5))

    bookkeeperResult shouldBe (expectedUserData, expectedEnriched)
    storageResult shouldBe expectedEnriched

Look at this lovely test! It's now completely pure. No more side-effects or mutable state. We are testing using pure-functions. The best kinds of test.

This test has been officially purified.

Complications?

The eagle-eyed readers amongst you might've noticed the potential for proliferation of type parameters in our code. For every Unit-returning method we now have to create a fake type-parameter. Is it worth it?

Apart from the benefit of (much) nicer tests, I would argue that the resulting type-signature is actually more helpful:

def fetchAndStore(user: UserID): Unit
// vs.
def fetchAndStore(user: UserID): (Bookkept, Stored)

The first signature describes an opaque side-effect, while the second signatures is more declarative, stating that the result of fetchAndStore is something being bookkept and stored. I think that's a win.

In the next part we'll see more advantages of being declarative this way in detail.

But I'm Already Doing Pure FP

You might also object that you're already doing pure FP with ZIO/Cats Effect/Finally Tagless. And that's great. But every time you see something like IO[Unit], the same considerations as above apply. You'll need to test a side-effect, and resort to some sort of mutability¹¹.

Unless you're doing Finally Tagless in which case in your tests you can replace F[Unit] with something like Writer[Unit] and still have a pure test. Though I still think that adding a type-parameter might be worthwhile.

In any case, the general lesson is still:

Friends, don't let friends use Unit as a return value.

Conclusion

We've seen how we can mechanically turn our side-effecting, mutable tests into nice pure functions, just by adding some type-parameters. The results are nicer, more declarative tests.

In the next part, in the spirit of "one 'simple' design change, a panoply of outcomes", we are going to take a look at some other benefits of parameterizing our code this way.

Happy purifying!

Gasp... ↩
The full code for the examples is available on GitHub. ↩
You could also use a mocking library for this purpose, but I personally prefer to avoid them. ↩
Don't catch me on the terminology here. Mocks, spies, stunt doubles, whatever kids these days call them, for our purposes here they are all the same. ↩
Double gasp... ↩
Okay, using a mocking library would make this less tedious. It can still be error-prone. And in any case I have no incentive to make mocking easy, I want people to feel the pain of mocking, so that they gravitate to code that doesn't rely on it that much. Arguably, such code would be of better quality compared to code that requires many mocks to test. ↩
I mean this in the Orwellian sense of "ignorance is strength". And see this talk for more. ↩
Equivalent to the identity function. ↩
By "cheating" I mean things like subverting the type-system with nulls, casts, and the like. See the Scalazzi subset of Scala. ↩
This is a loose application of "parametricity". ↩
The same reasoning applies to non-pure container types, like Future. ↩

When Are Functions Lazy Enough for Lists

Daniel Beskin — Thu, 02 May 2024 00:00:00 GMT

Recently, I've been TA-ing on an introduction to functional programming course, using Haskell. This is a very enlightening experience, as it once again proves the notion that the best way to learn something is to teach it.

Since Haskell is rife with topics that I feel I understand but are still tricky to actually explain¹, trying to answer student question can sometimes prove more challenging than expected. This post is my attempt to answer one such question that I got the other day.

Disclaimer: this post is aimed at Haskell beginners or teachers.

As part of their homework, the students had to reimplement some basic list functions on their own. Part of the requirements were that when possible a function accepting a list should seamlessly work for both finite and infinite lists. Given that Haskell is lazy by default, this tends to happen magically on its own, without any extra effort. But trying to understand when this works or not is another matter.

Here I'll try to add some intuition for the following question:

When is a function that operates on lists is "lazy enough" to handle infinite lists?

The answer I'm going to propose is that²:

A function on lists is "lazy enough³" when for all inputs it uses a finite chunk of the input to produce a finite chunk of the output.

The answer might be even more confusing, so let's check out some examples to get intuition.

The head function⁴:

head (x : _) = x
head [] = error "head on an empty list"

head takes in a list, a potentially infinite list, but it only looks at the head of the list (a finite chunk) and ignores the tail (a potentially infinite chunk of data). So head is "lazy enough" to handle infinite lists:

head [1..] == 1

What about tail?

tail (_ : xs) = xs
tail [] = error "tail on an empty list"

To produce a finite chunk of the tail, we only need to skip the head (a finite chunk) and then produce the (finite) chunk of the tail of the list. So this works:

head $ tail [1..] == 2

Similarly functions like take n, only consume a finite chunk of elements (n) to produce a finite chunk of elements (a list of n elements).

What are examples that are not lazy enough?

Take length for example:

length [] = 0
length (_ : xs) = 1 + length xs

Here, to produce a finite chunk of the result (a single number), we must consume the whole list (a potentially infinite chunk of data). So there's no way this can work for infinite lists.

length [1..] -- never terminates

The same goes for reverse: the first element of the result of reverse is the last element of the input list. To produce a finite chunk of the result (the last element of the input), we must consume the whole, potentially infinite, list. And so reverse does not work for infinite lists.

An important thing to note is that different implementations of the same function can behave differently with respect to laziness depending on the implementation details. Consider the following implementation of head:

badHead xs =
  if length xs > 0 then head' xs
  else error "head on empty list"
  where
    head' (x : _) = x

This implementation tries to be "smart" about checking the error condition. But now in order to produce a finite chunk (the head of the list) we must consume the whole (potentially infinite) list to check its length. The new implementation of head no longer works for infinite lists:

badHead [1..] -- never terminates

Now let's take a look at something more complicated. Figuring out when the functions above are lazy enough or not is probably not that difficult. Let's analyze a more challenging function: partialSums.

Given a list of numbers, partialSums creates a new list of sums of the numbers up to the current element:

partialSums [1, 2, 3, 4, 5] == [0, 1, 3, 6, 10, 15]

And it should also work for infinite lists:

take 6 $ partialSums [1..] == [0, 1, 3, 6, 10, 15]

We can implement this function using explicit recursion:

partialSums [] = [0]
partialSums (x : xs) = 0 : map (x +) (partialSums xs)

Why does this work for infinite lists? Let's say we want the first element of partialSums [1..]. We can follow the equations in this definition⁵:

partialSums [1..] == -- pattern matching
partialSums (1 : [2..]) == -- by definition
0 : map (1 +) (partialSums [2..])

Assuming that (:) is lazy (it is), we can access the first element 0 without accessing the rest of the (infinite) result.

To be able to continue evaluation, recall the definition of map:

map _ [] = []
map f (x : xs) = f x : map f xs

map itself is lazy, to get the first element of its result, we only need to evaluate f on the head of the result. No need to go over the possibly infinite tail.

Let's continue evaluation and grab the next element:

0 : map (1 +) (partialSums [2..]) == -- evaluating the first step of `partialSums` so that we can pattern match on the result
0 : map (1 +) (0 : (map (2 +) (partialSums [3..]))) == -- by definition of `map`
0 : (1 + 0) : (map (1 +) (map (2 +) (partialSums [3..]))) == -- reduce arithmetic
0 : 1 : (map (1 +) (map (2 +) (partialSums [3..])))

Notice how pattern matching forced us to evaluate the next step of partialSums, we can't choose the correct branch of map without knowing whether the list is empty or not. But we don't need anything beyond that, we do not evaluate the whole list, just the existence of the first head/tail pair. More generally, pattern matching is one way to gradually force the evaluation of lists.

The next element 1 is available for grabbing, while the rest of the tail is not evaluated by (:).

Just for fun, we'll repeat this once again for the third element:

0 : 1 : (map (1 +) (map (2 +) (partialSums [3..]))) == -- evaluate `partialSums`
0 : 1 : (map (1 +) (map (2 +) (0 : (map (3 +) (partialSums [4..]))))) == -- evaluate the second `map`
0 : 1 : (map (1 +) ((2 + 0) : (map (2 +) (map (3 +) (partialSums [4..]))))) == -- reduce arithmetic
0 : 1 : (map (1 +) (2 : (map (2 +) (map (3 +) (partialSums [4..]))))) == -- apply first `map`
0 : 1 : (1 + 2) : (map (1 +) (map (2 +) (map (3 +) (partialSums [4..])))) == -- reduce arithmetic
0 : 1 : 3 : (map (1 +) (map (2 +) (map (3 +) (partialSums [4..]))))

And now 3 is available to us using a finite amount of elements from the input list. We can go on like this, every time consuming just one element from the input list to produce one element from the output. Since (:) and map are both lazy, we are never forced to evaluate the whole tail, just a finite chunk of it.

We can also rewrite partialSums in terms of foldr:

partialSums = foldr aux [0]
  where aux x res = 0 : map (x +) res

This is a mechanical rewriting of the recursive solution⁶. Since foldr is implemented as the same recursion pattern that we wrote manually before, this works just like the recursive version. Therefore, it still works for infinite lists.

We can prove it using the same logic, we just have to apply foldr first. Recall that foldr is defined as:

foldr _ z [] = z
foldr f z (x : xs) = f x (foldr f z xs)

Now we can evaluate the new partialSums again:

partialSums [1..] == -- by definition `partialSums`
foldr aux [0] [1..] == -- pattern matching in `foldr`
foldr aux [0] (1 : [2..]) == -- by definition of `foldr`
aux 1 (foldr aux [0] [2..]) == -- apply the same logic recursively
aux 1 (aux 2 (foldr aux [0] [3..])) ==
aux 1 (aux 2 (aux 3 (foldr aux [0] [4..]))) == -- expand the `aux` calls
aux 1 (aux 2 (0 : (map (3 +) (foldr aux [0] [4..])))) ==
aux 1 (0 : (map (2 +) (0 : (map (3 +) (foldr aux [0] [4..]))))) ==
0 : (map (1 +) (0 : (map (2 +) (0 : (map (3 +) (foldr aux [0] [4..])))))) ==

From here this looks pretty much the same as in the recursive version, and is lazy for the same reasons.

Now suppose that we had a non-lazy version of map:

badMap f xs =
  if length xs == 0 then []
  else f (head xs) : badMap f (tail xs)

This is logically the same thing as the regular map, but it is no longer lazy, since it calls length which forces the evaluation of the whole list.

If we use badMap to implement partialSums:

badPartialSums = foldr aux [0]
  where aux x res = 0 : badMap (x +) res

The result will no longer be sufficiently lazy, and won't work for infinite lists:

take 6 $ badPartialSums [1..] -- never terminates

If our function calls a "not lazy enough" function, then it stops being "lazy enough".

Hopefully now the statement we started with makes more sense:

A function on lists is "lazy enough" when for all inputs it uses a finite chunk of the input to produce a finite chunk of the output.

Thanks for reading this far. I hope this helped you gain some intuition about laziness and infinite lists. Reach out if you have any further questions or comments.

foldr anyone? ↩
This is not a formal definition, but good enough for our purposes. I'm not trying to be precise and fully explain laziness in Haskell, but just to generate enough intuition for the students' homework. ↩
As was noted in the Reddit discussion, I'm actually overloading the word "lazy", when what I actually mean by "lazy enough" is "productive". Since we're using Haskell, laziness is entangled with productivity anyways. I'm still on the fence on whether it's worthwhile introducing this new terminology while in the context of this particular homework set. ↩
We'll ignore for the moment that head and tail are not safe and should be banished from our lives. I don't want to bring in Maybe, which might be distracting from the point. ↩
This process of moving through equations is called "equational reasoning". A very useful tool when trying to understand Haskell code. ↩
I'll spare myself from explaining the workings of foldr in this post. ↩

Daniel Beskin's Blog

I Am Not a Functional Programmer

Spooky Action at a Distance

Mocking the World

Feeling a Bit DRY

I'm Just Trying to Be Reasonable

The Monad Elephant in the Room

Footnotes

The Compiler Is Your Best Friend, Stop Lying to It

Prologue

Part I: The Compiler

Rust

Java

Aside: Who Compiles the Compilers?

TypeScript

Part II: Lying to the Compiler

The Lies

Null

Exceptions

Casts

Side-Effects

Part III: No More Lies

Null

Exceptions

Casts

Side-Effects

Part IV: The Compiler as Our Friend

Typed wrappers

Union Types

Typed Guarantees

Dialogue

What's the Point of Learning Functional Programming?

The Homework

A Haskell Rewrite

Going Functional

What Did We Learn?

Laziness

Explicit State Space

Modularity

Composition

Wholemeal Programming

Footnotes

Pretty Symlinking with Home Manager

Imports

Functions

Programmatic Keys

List Processing

Cosmetics

Footnotes

Validating Custom Keyboard Layouts on NixOS

Footnotes

Random Scala Tip #534: Adopt an Error Handling Convention for `Future`

YOLO

Full On Defensive

Only Defects in Error Channel

Others

Footnotes

Make Illegal AI Edits Unrepresentable

The Pitch

The Illegal States

Getting in the Flow

AI vs. Illegal State

The Root Cause

The Unrepresentable

AI vs. The Unrepresentable

First Make It Correct

The Problem

Obviously Correct Solution

The Reference Solution

Recursing the "Right" Way

Recursion, I Hardly Know Her

Laziness with Class

Follow the Pain

Spiral Tracing as a Service

...And Then Make It Fast

Conclusion

Footnotes

Random Scala Tip #624: The Pitfalls of Option Blindness

Differently Typed Blindnesses

The Tip

But What's So Special About `Iterator`?