The Welcome Return of Research Papers to Software Craft

Over the last two to three years a small, but very important and noticeable trend, has started around the world – a growing appreciation of the importance of primary research and academic papers among software practitioners. Those that are crafting software are spending more and more time understanding, learning from, and reflecting on research from the past and present.

Like many people working in industry I have a somewhat love hate relationship with academia. On one hand I love and value independent research, and have a strong appreciation of true academic rigour. On the other hand, I do sometimes find some academics completely disconnected from the reality of how industry works. It is, however, this very disconnection that allows many academics, and researchers in various commercial organisations such as Google, IBM and Facebook the freedom to think differently, challenge convention and come up with radically different approaches.

The level of practitioner interest in research papers has risen to a point that the opening keynote at QConLondon this week was delivered by Adrian Coyler, author of The Morning Paper and a venture partner at Accel. Adrian walked people through a number of his favourite papers and challenged people to think a little differently about what is coming in the future.

It is hard to pinpoint quite what caused this renewed interest, but it is safe to say that the emergence of Papers We Love, with the associated meetup groups, frequent discussions on forums such as Hacker News and blogs such as Adrian’s has created a wonderfully curated entry point to research papers for the curious. When people such as Werner Vogels at AWS remind us of the importance of papers, people sit up and take notice. As an industry we have had, at times, a tendency to forget to look at problems in detail, and instead focus on the quickest time to getting a product out the door.

One of the most recent Papers We Love talks came from Bryan Cantrill, CTO of Joyent, where he talked about BSD Jails and Solaris Zones, and as he noted at the start of his talk, while reminiscing about soda at the journal club his Dad, a physician, hosted:

I always felt that it was really interesting that medical practitioners did this, and I always try to draw lessons from other domains, and medicine is very good about this, and we are not very good about this. We in computer science and software engineering are not nearly as good about reading work, about discussing it.

However, a new generation of practitioners want to go deeper than this superficial level. As Adrian Coyler put it in our discussion about The Morning Paper:

I do believe there’s a growing hunger to learn something deeper and more lasting than just the syntax of the latest APIs.

And Bryan noted in his talk

when papers we love kicked up… a really, really exciting development

Papers We Love

Taking a closer look into whats happening with the community around Papers We Love, we see chapters gradually popping up around the world. What is really interesting here is to see the geographic spread of these meetups.

The cumulative worldwide growth for the Papers We Love meetups, and the interest in the github repo where many of the papers get posted to is following a really nice, strong trajectory as well.

The Morning Paper

As I mentioned I had the chance to discuss The Morning Paper with Adrian Coyler, and he kindly shared some stats about the blog. The Morning Paper gets between 7 and 12K unique visits a week, depending on the papers Adrian has chosen.

The Morning Paper mailing list is fast approaching 2,500 subscribers (you can subscribe here). Indeed, Adrian pointed out that several academics have told him how delighted they are to see their paper listed – the audience exposure is far, far greater than they normally receive from presenting a paper at an academic conference.

Industrial Causes?

One of the factors that may be contributing to our increased interest in research papers is the sheer speed to market that we now see, in particular around the big data space. Take a company like Databricks as an example, the research behind Apache Spark is only a few years old, but we have a commercial offering out for quite some time at this point. We see a similar pattern with companies such as HortonWorks, MapR, Cloudera, Confluent, Mesosphere and so forth.

But more generally there is something else going on – businesses have become to appreciate the value of engineering excellence, and the value of having staff members that understand how the new systems they are building their businesses are created. People that understand the primary research.

The relative numbers may be small, but they are statistically significant – and the practitioners reading these papers all want to learn. And that can only be a good thing.

Thank You

I would also like to say thank you to Adrian Coyler of Accel for sharing his thoughts and statistics on the morning paper and the growth in interest in research papers we are currently seeing.

Some notes on the analysis.

1. We looked at the growth of meetup groups where data is available on meetup.com. Groups that use facebook and other platforms for organising meetups were not included.

Disclosures: IBM, Cloudera and AWS are RedMonk clients.