#8 - Idempotency

A simple idea with a scary name. Let's make sure no one trips you up with this fancy jargon.

May 11, 2020

Hey all,

Hope your week is off to a great start (and that you did something awesome for Mom!).

In addition to being Monday, it’s also “Day 11” of Summer of Shipping. We met last Thursday for our first demos and are building momentum!

Summer of Shipping @summer_of_ship

Just had our Week 1 Meeting! - Presentations featuring ideas from Andy Grove, "The Cathedral and the Bazaar" - 3 Demos to kick off our demo "open-mics"! Check it out:

youtu.beSummer of Shipping - Week 1Summer of Shipping is under way! 🚀 New here? Checkout the kickoff: https://youtu.be/tfbaDguurH0 If you want to be a part of this, sign up here 👉 https://form...

School’s just wrapping up for a lot of folks this week, so we anticipate more building will happen soon! 🏗

You might be interested in…

If you have ambitions of one day starting a profitable side project, check out the Young Makers newsletter from Gabriel Chuan. His newsletter teaches you how successful makers make profitable projects. Give it a look, and level up those business skills! (They’ll help on the job too 😉)

Now onto today’s topic - a simple concept with a name so fierce it might scare you away. It’s called idempotency.

Idem-what?

Idempotency is a property of operations. It means that when an operation is applied twice, the result is the same as if the operation was only applied once.

That’s a mouthful, but let’s make it clear with some dumb examples.

Dumb Examples

Let’s take a simple operation, called times2:

const times2 = (value) => value * 2

As its name suggests, it takes a value and returns a new value that is twice the input.

Is times2 idempotent?

const a = 1;
const result1 = times2(a);       // returns 2
const result2 = times2(result1); // returns 4

// result1 != result2

No. It is not. Applying it twice changes the result.

Now let’s take a similar operation, called times1:

const times1 = (value) => value * 1

As its name suggests, it takes a value and returns that value.

Is times1 idempotent?

const a = 1; 
const result1 = times1(a);       // returns 1 
const result2 = times1(result1); // returns 1  

// result1 == result2

Yes. Yes it is.

A math-y look

Another way to think about idempotency is to consider a function “f”. We say that “f” is idempotent if the following is true:

f(x) = f(f(x))

And by inductive reasoning, we can also say that:

f(x) = f(f(x)) = f(f(f(x))) = f(f(f(f(x)))) = ... and so on

So basically, we can apply “f” all we want, and it’s the same as applying it just once.

Other math-y operations that are idempotent:

Multiplying by 0
Absolute value

Who cares?

Given these examples, you might be led to believe that idempotency really isn’t all that potent after all. Who cares about some funny property of basic arithmetic operations? Who really cares that multiplying by 0 always returns 0?

For those who are familiar with Pokemon, sounds like something that is as useful as Magikarp’s Splash “attack”.

But don’t let these examples fool you. Idempotency is essential to consider when computers communicate across an unreliable network.

Miscommunication

When sending a message from one computer to another, you must anticipate network failure. You may never run into an error personally, but at scale - you better bet that some of your users will.

Here are some potential network failures:

Your message fails to arrive
You fail to receive an acknowledgement (ACK)
You receive a false error response (the downstream service has processed your message, but they accidentally sent an error back anyways. This can happen when that downstream service accidentally introduced a bug)

In all cases, a robust system would kick off a retry. But in examples #2 and #3, the message was already processed. A retry would result in the operation being done twice!

Building with Idempotency

Think about all the things your company does that you would only want to happen once. You might only want to send one shipment out of your warehouse. You might only want to dispatch a single driver to a pickup location. And you might want to charge a customer once, and only once.

Processing these requests more than once could result in a serious operational cleanup headache. Thankfully, adding idempotency is simple.

We ask that callers of our service send us a unique key for each request that they want processed once and only once. For every request, we check to see if we’ve seen this key. If this key is new, we process the request. But if we’ve seen it before, then nothing is done and the result from last time is returned.

This allows our caller to perform many retries without accidental duplicate requests. It’s a feature you should consider implementing on your API (and with all things API, refer to Stripe for inspiration).

Conclusion

Idempotency is a simple idea with a scary name and serious ramifications. In the future, when you’re designing systems - make sure to ask “is there anything that must happen once and only once?”

You don’t want to be alerted early one morning with thousands of duplicate payments on your hands. Nah, solve that problem once and for all by making sure you’ve designed with idempotency in mind.

Alright, until next time!

Phil

Software Mentor