Yes, Network Effects Are a Problem for Open Formats
Published 10:10, 11 October 12
As we know, lock-in is one of the biggest obstacles to moving from closed, proprietary formats, to open ones. But so far as I know, no one has tried to quantify the extent to which people cling to old formats. That makes the following piece of research useful, at least as a first stab at finding out what is really going on:
we analysed a corpus of over 2.5 billion resources corresponding to the UK Web domain, as crawled between 1996 and 2010. Using the DROID and Apache Tika identification tools, we examined each resource and captured the results as extended MIME types, embedding version, software and hardware identifiers alongside the format information. The combined results form a detailed temporal format profile of the corpus, which we have made available as open data. We present the results of our initial analysis of this dataset. We look at image, HTML and PDF resources in some detail, showing how the usage of different formats, versions and software implementations has changed over time.
The key question was as follows:
whether formats last a few years and then die off, or whether (on the web at least) network effects take over and help ensure formats survive.
The results are not encouraging:
A large number of formats have persistent for a long time (47 formats have been around for 15 years), and that since 1997, roughly six new formats have appeared each year while fewer have been lost (roughly 2 per year).
most formats last much longer than five years, ... network effects to appear to stabilise formats, and ... new formats appear at a modest, manageable rate.
As the research notes, this is a indication that networks are very powerful - exactly as we might have surmised given the experience in the open source world. What this means in practice is that it's going to be really difficult to displace proprietary formats like .doc and .xls, even with the best will in the world. It also shows the importance of getting governmental affirmative action that explicitly recommends open formats in order to overcome these network effects - or at least speed things up.