Remember that time when thousands, or even Oh My Goodness tens of thousands of downloads a month was worth mentioning as a measure of project or product health? It’s not so long ago, but the game is irrevocably changing. What happens when everyone is contributing code, builds are automated, and libraries are stupid easy to include in your code? With good engineering practice you won’t be open to a shit show like left-pad but this post isn’t about dependency management. It’s more about the absurdity of the scale we’re dealing with in the age of package managers like npm.
This post by Algolia caught my eye yesterday – serving one billion javascript library downloads. Algolia is a text search company that focuses really deeply on performance and developer experience. Like all smart companies today it develops in the open. So of course it makes open source contributions.
“Algolia is developing multiple open source projects (like InstantSearch.js) to simplify the integration process of our search engine. This month, with the help of jsDelivr, we reached a milestone 1 billion downloads (that’s 26TB!) across all of our libraries.
Now seems like a good time to explain how this works and the choices we made.” [italics mine]
So with this kind of scale around library consumption, which would have been unprecedented even 10 years ago, we now have specialised CDNs for delivering open source libraries – see jsDeliver. For an old guy like me this is pretty mind-blowing.
As Algolia’s blog explains
“jsDelivr is a free CDN built exactly for this. It offers production quality of service and natively integrates with npm.”
Advanced users now bundle packages using Webpack. Others are still just cutting and pasting though – the terrifying, awesome basis of the modern web. To be honest, these days a lot of Javascript feels like a metaphor for the modern era – an edifice of cut and paste, which you worry could fall over any minute.
The relationship between Algolia and jsDelivr is mutually beneficial – with jsDelivr also using Algolia as a search mechanism for the Javascript libraries it hosts.
According to the post on Algolia, by the numbers
“jsDelivr serves about 19 billion requests every month. That’s almost 500TB of bandwidth of small ~3kb js files — all that from 176 locations all around the world.”
Services in a related space include Google Hosted Libraries and Microsoft Ajax CDN.
Check out Docker Hub download numbers for more high scale craziness in libraries and packages. Here is a great post by Brian Christner on the subject – showing how to track Docker Hub metrics with Prometheus and Grafana. By 2016, there were more than 6Bn pulls to Docker Hub. 2017 it must be north of 10Bn. I will get with the team here and verify.
This really is a different world, with new domain specific tools needed for new ways of writing, consuming and managing software. The scale of decomposition is truly staggering.
for disclosure purposes I should mention Docker is a client.
No Comments