Scale and scalability

I’ve been thinking a lot about scaling and the economics of the cloud recently after reading this. Specifically, this quote:

The costs for most SaaS products tend to find economies of scale early. If you are just selling software, distribution is essentially free, and you can support millions of users after the initial development. But the cost for infrastructure-as-a-service products (like Segment) tends to grow linearly with adoption. Not sub-linearly.

To unpack this a little, most Infrastructure-as-a-Service providers make “linear scaling” a big part of their sell. That is to say, their pricing is based on pure volume billing. You use more RAM, or CPU cores, or networking bandwidth, or transactions on the database service, or whatever, and your bill goes up proportionately. This speaks to the desire for predictability, and also to a couple of déformations professionelles.

First of all, if you program computers you learn pretty quickly that it’s easy to come up with solutions that work, but that become disastrously inefficient when you apply them on a large scale. Because computers do things quickly, this isn’t obvious until you try it – it might be fast because it’s right, or fast because the test case is trivial. Identifying the scaling properties of programs is both something that is heavily emphasised in teaching, and something that will invariably bite you in the arse in practice. One way or the other, everyone eventually learns the lesson.

Second, it’s quite common in the economics of the Internet that you meet a supply curve with a step function. Back in the day, BT Wholesale used to sell its main backhaul product to independent ISPs in increments of 155Mbps. As traffic increases, the marginal cost of more traffic is zero right up to the point where the 95th percentile of capacity utilisation hit the increment, and then either the service ground to a halt or you sent thousands of pounds to BTw for each one of your backhaul links. When the BBC iPlayer launched, all the UK’s independent ISPs were repeatedly thrashed by this huge BT-branded cashbat until several of them went bust.

So everyone is primed to look out for a) nasty scaling properties and b) cliff-edge pricing structures. The IaaS pricing model speaks to this. But it has a dark side.

When we talk about “economies of scale” we mean that something becomes cheaper to produce, per unit, as we produce more of it. This is, in some sense, the tao of the Industrial Revolution. To put it in the terms we’ve used so far, something that has economies of scale exhibits sub-linear scaling. The dark side of the cloud is that its users have got rid of the risk of pathological scaling, but they’ve done it by giving up their right to exploit sub-linear scaling. Instead, the IaaS industry has captured the benefits of scale. Its costs are sub-linear, but its pricing is linear, so as it scales up, it captures more and more of its customers’ potential profitability into its own margins.

There’s a twist to this. Really big IaaS customers can usually negotiate a much better deal. At the other end of the curve, there’s usually a free tier. So the effective pricing curve – the supply curve – is actually super-linear for a large proportion of the customer base. And, if we go back to the original blog post, there’s also a dynamic element.

Because outsourcing infrastructure is so damn easy (RDS, Redshift, S3, etc), it’s easy to fall into a cycle where the first response to any problem is to spend more money.

One of the ways sub-linear scaling happens in IT is, of course, optimisation. But here you’re being encouraged not to bother, to do the equivalent of throwing hardware at the problem. Technical improvement is also being concentrated into the cloud provider. And, of course, there’s a huge helping of confusopoly at work here too. The exact details of what you’re paying for end up being…hazy?

In our case, this meant digging through the bill line-by-line and scrutinizing every single resource. To do this, we enabled AWS Detailed billing. It dumps the full raw logs of instance-hours, provisioned databases, and all other resources into S3. In turn, we then imported that data into Redshift using Heroku’s AWSBilling worker for further analysis.

I heard you liked cloud, so I built you a cloud to analyse the bills from your cloud provider. In the cloud.

So here’s a thought. Want to know how employees at Apple and Google are more productive? The answer may just be “monopoly” or more accurately, “oligopoly”. And as Dietrich Vollrath likes to point out, under standard assumptions, more monopoly means lower measured productivity economy-wide.