Search Engines, Integrity Of, and Economics

Over in the shabbily luxurious senior common room that is Crooked Timber, they’re discussing the role of search engines (the internet’s swiss army knife) in economics, society and culture. Not only that, but one of them’s been asked to edit an issue of a scholarly journal devoted to the topic. A suggested topic is the question of whether or not there is a “viable public alternative to the search engine market dominated by private actors”. Now, this interested me at once, and offers an opportunity for my continuing efforts to get more economics into the blog.

For a start, why would anyone want a “public” search engine? Aren’t they public enough already? That’s obviously a misunderstanding, but it’s an illuminating one. Search engines are free to the user, but that doesn’t make them public. They certainly feel like it, though. You don’t pay anything, you don’t need to join or register, you just turn up – if you use Firefox you hit Ctrl-T and then Ctrl-K and clap in your search request, and up come two million Google results. However, there is famously no such thing as a free lunch, and there must be a revenue stream somewhere. After all, unlike a blog where if you are feeling militant you can quite simply provide all the makings yourself, a web-scale search engine requires infrastructure – warehouses stuffed with servers tended by pale and sweaty sysadmins. Google famously operates some 100,000 servers organised as a cluster operating system. (Link to seminal blog entry on this) That costs money, so someone must be paying. Google, and in fact all other publicly-available search engines, earns money by showing adverts to searchers. And this is where the first batsqueak of doubt kicks in.

A search engine’s product is search, right? Its customers are us – the searchers. But where does the money come from? Advertisers. Now, there’s clearly a problem of incentives here. If the searchers were paying customers, the search engine’s owners would have a direct financial interest in providing a product they would pay for. Their interests would be aligned with those of the users. Equally, if the search engine was a membership organisation, the user-members would have every interest to exercise supervision of the management. But Google, Yahoo!, MSN Search and the rest all make their money from ads and answer to shareholders. We do here perceive a divided duty! After all, it is in the advertiser’s interest to get as many users channelled to their site as possible. It’s therefore also in the search engine operator’s interest to rig the search process to that end. In fact, the contract with the advertiser may specify that payment is by results, measured by hits or clickthroughs. And the standard oversight mechanism – responsibility to shareholders – will not prevent this, as the same incentives apply.

In fact, the only constraint on such behaviour is that, if it is carried to extremes, users may stop coming. But for this to occur, the manipulation would have to be blatantly obvious – a fall in the hit count might be acceptable to advertisers if the clickthrough rate went up. There is also the point that, if it became known that certain content was being favoured, some users might be attracted. But it’s unlikely that the mass of users would be this sophisticated. Searching the web is generally an imperfect experience – you get perhaps a couple of very good results, twenty or so less goods, a mass of barely relevant stuff and usually a few random outliers. It’s likely that quite a serious degree of manipulation might go unnoticed except by a sophisticated minority of geeks.

Why is this important? The internet became an important business perhaps at the moment that search engines became indispensable – when some critical mass, both in diversity and in quantity of information, was passed and automatic search became the only practical way of retrieving relevant facts. It would seem that the critical mass when the net became a popular phenomenon and a serious economic proposition arrived at much the same time. Search is the primary means of accessing the internet. It’s one of cyberspace’s public utilities. Because of that, its integrity must be maintained. Without honest search, both free speech and free markets on the net are threatened. The problem is similar to the journalistic division of facts and comment: though there may well be a place for commercially-enhanced search, it’s a place clearly marked as such and distinct from the real thing.

What is the answer? Regulation? How? The difficulty of getting search engine operators to disgorge their underlying search algorithms would seem insuperable, and anyway this itself would be subject to manipulation. One possible idea would be to prepare a worldwide standard for search, perhaps under the auspices of the Internet Engineering Task Force or the WWW Consortium. But no-one could be compelled to be standards-compliant. One might hope that users would choose search engines bearing the “kitemark” (a magnifying glass, perhaps, or a pair of binoculars as that seems to be a de facto standard symbol for “search”) for preference. But this is hardly robust.

A public-service search engine, then? A BBC of search engines?

The first and obvious problem would be – who would pay? And who would do? Obviously someone would have to take responsibility and make the arrangements. Would any government do so – and would that be a trustworthy solution? A public service search engine, of course, would be in a position to disclose its methodology without prejudice to its income, or for that matter to go open-source. The question would remain, though, of whether the ultimate owners might fiddle with it. That might be solved, but the question remains of exactly who would do it.

An open-source search engine?

Part of the solution, clearly. This would allow for continuous development, open and verifiably honest search, and open access. But, though, this is where we run into our infrastructure problem. Open-source software doesn’t need an infrastructure – the users supply their computers. Open-source search would need a server farm somewhere, employees, money…and soon we’re back at square one. There are two answers to that: one is an imperfect but feasible solution, and one is more idealised. The first: build the open-source search engine and place it under a trusted organisation, for example Creative Commons or some similar outfit – if they could get the funding. Right. But – again – could the open source community pay, especially as the whole thing’s mission is to provide search without commercial bias? The second, a more distant proposal, would be a peer-to-peer open-source search system – its costs distributed across the user world, without a central owner capable of interference. This, however, would sound technically challenging, not to mention the possible privacy and security problems.

Another search-oligopoly issue, of course, is raised by spinoffs from the central technology, like all the cool stuff Google keeps launching. Wait a moment – I’m writing this on Blogger and my email address ends .gmail.com – as Gary Larson’s cow put it, IT’S GRASS! WE’VE BEEN EATING GRASS!