{"id":4387,"date":"2017-05-05T16:45:59","date_gmt":"2017-05-05T16:45:59","guid":{"rendered":"https:\/\/redmonk.com\/jgovernor\/?p=4387"},"modified":"2017-05-05T16:48:20","modified_gmt":"2017-05-05T16:48:20","slug":"lessons-on-data-culture-from-ocado","status":"publish","type":"post","link":"https:\/\/redmonk.com\/jgovernor\/lessons-on-data-culture-from-ocado\/","title":{"rendered":"Lessons on data culture from Ocado"},"content":{"rendered":"

\"\"<\/a><\/p>\n

Data transformation is the new digital transformation<\/a>. It affects everyone.<\/p>\n

Ocado is an interesting company to learn from because it was a \u201cFull Stack Startup<\/a>\u201d before they were a thing, a vertically integrated business that builds all its own technology and take a new approach to a market. It runs everything from logistics to web site through warehouse routing and optimisation, with a team more than 1000 software engineers.<\/p>\n

Ocado Technology also provides fulfilment services to third parties, which makes the ability to scale critical. It was an early Google Cloud Platform (GCP) reference, and adopts Google technology as aggressively as can, given the limitation that it has a corporate rule against using beta technology in production. The company now has \u00bd a Petabyte of data in its Cloud Dataproc<\/a> managed Spark cluster, 20x what it previously had its Oracle-based datawarehouse, growing by 250M events per day.<\/p>\n

So that\u2019s the infrastructure – but what about the people, and the transformation that led the company to start hiring data scientists for every development team, rather than keeping them as a separate group.<\/p>\n

Daniel Nelson, Ocado head of data, said:<\/p>\n

\u201cTraditionally we got data, gave it to a data scientist, who put it on their laptop, they went off for a few months, wrote some code in Scala or Erland or whatever and then gave it to a dev team and said could you optimise it, any problems and it had to go back in backlog.\u201d<\/p><\/blockquote>\n

Not exactly agile then.<\/p>\n

Ocado tried a different approach when it started a project to try and identify ways to improve customer contact response times. Originally agents responded to emails in first in first out fashion – about 2000 per day – which was fine on an ordinary day but not if volume was much higher, for example because of bad weather.<\/p>\n

What I like about the story is that the new approach was driven by engineers. Some developers from the call center team while they were \u201cat the water cooler\u201d (probably the pub!) asked their peers from the then separate data science team – can you think of a way to solve this?<\/p>\n

Ocado ran with it, and put some of their data data engineers into the contact center to learn the job, colocating them with its software engineering team. The solution was automatic tagging of incoming emails for sentiment, pushing complaints to the top of the work list. Ocado got early access to the Google natural language API but at the time it wasn\u2019t sophisticated enough to do all the tagging.<\/p>\n

Ocado therefore took around 1M emails, 3 years worth, and pushed it into the cloud. The six month project was in 3 phases<\/p>\n

1 month for domain knowledge<\/p>\n

3 months for prep<\/p>\n

2 months to build and industrialise<\/p>\n

Ocado took a software engineering approach, making sure everything was reusable, with a full test suite. The teams used Git, and used Ocado\u2019s normal delivery pipelines.<\/p>\n

\u201cData Science is Software – it’s reproducible and maintainable\u201d<\/p><\/blockquote>\n

Ocado spent a month training the model, took a pragmatic approach. Get it working, deploy and test. It therefore built is own solution using Tensorflow running on Google Compute Engine. As stated above (as above Ocado couldn\u2019t use Google\u2019s managed data services because they were still in alpha at the time). Today\u2019s alternative would be Google Dataflow and ML APIs.<\/p>\n

The results- now if Ocaco is dealing with with urgent negative sentiment emails it does so four times more quickly. It was also able to move 4 people to more creative customer support roles. It is also seeing fewer refunds.<\/p>\n

The key insight culturally for me is that Ocado now hires data scientists directly into its development teams in different parts of the business, rather than keeping them as a separate group. Well that is, when it can onboard. It is of course hiring<\/a>\u2026<\/p>\n

It\u2019s also worth mentioning that Ocado Technology isn\u2019t just hiring. It\u2019s working on the future pipeline with it\u2019s Code for Life<\/a> kids coding initiative.<\/p>\n

 <\/p>\n

Related –<\/p>\n

Data Transformation is the New Digital Transformation<\/a><\/p>\n

That giant sucking sound? Hadoop moving into the cloud<\/a><\/p>\n

Measure what you manage\u2013data, storage and admin<\/a><\/p>\n

Google is a client.<\/p>\n","protected":false},"excerpt":{"rendered":"

Data transformation is the new digital transformation. It affects everyone. Ocado is an interesting company to learn from because it was a \u201cFull Stack Startup\u201d before they were a thing, a vertically integrated business that builds all its own technology and take a new approach to a market. It runs everything from logistics to web<\/p>\n","protected":false},"author":5,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false},"categories":[1],"tags":[],"class_list":["post-4387","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_featured_media_url":"","jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p9wfjh-18L","_links":{"self":[{"href":"https:\/\/redmonk.com\/jgovernor\/wp-json\/wp\/v2\/posts\/4387","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/redmonk.com\/jgovernor\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/redmonk.com\/jgovernor\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/redmonk.com\/jgovernor\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/redmonk.com\/jgovernor\/wp-json\/wp\/v2\/comments?post=4387"}],"version-history":[{"count":0,"href":"https:\/\/redmonk.com\/jgovernor\/wp-json\/wp\/v2\/posts\/4387\/revisions"}],"wp:attachment":[{"href":"https:\/\/redmonk.com\/jgovernor\/wp-json\/wp\/v2\/media?parent=4387"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/redmonk.com\/jgovernor\/wp-json\/wp\/v2\/categories?post=4387"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/redmonk.com\/jgovernor\/wp-json\/wp\/v2\/tags?post=4387"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}