Sometimes signal comes at you from interesting directions. We had a kick off call with a new client called Datical yesterday. They’re trying to solve a first 500 mile problem. You might call it Continuous Database Integration or DataOps. Datical calls is Database Release Automation.
Modern software development is all about Continuous Integration/Continuous Deployment (CI/CD) pipelines, but the test-driven development approach has been mostly used in new apps, targeting newer runtimes and likely a NoSQL datastore. When you’re running Docker on your laptop, writing a new web app, it’s unlikely you’re targeting a traditional relational database such as Microsoft SQLServer or Oracle. A lot of CI/CD is aimed at building somewhat stateless, cloud native apps, rather than more traditional ORM style stuff. As Stephen points out, when it comes to 12 Factor and Cloud Native apps the database isn’t even an afterthought – it’s an absence. So how are you going to do DataOps?
In the public cloud we’ve seen some solid work from Heroku in allowing cloning or “forking” of Postgres images for developers to work more effectively with pipelines. Enterprises though are still writing transactional apps that do target relational databases. So how can they get with the move to CI/CD? The last thing DBAs want to deal with is developers making a ton of changes and trying to push code every couple of hours. Never mind the DBA – the last thing a web developer wants to do is touch an Oracle database. There be dragons.
As per the chart above Datical is automating some of the work, the idea being a developer can treat database code like application code. Changes should be error free and compliant with DBA best practices. Datical offers database code packager, a developer or DBA will commit changes to a source code repo, then those changes can be introduced to the pipeline through change simulator. If changes won’t run they won’t run what rule was violated then they can rewrite that change, before turning it into a distributable package.
Another vendor working in adjacent spaces, which partners with Datical is Delphix – Data virtualisation and data masking. That’s another thing about enterprise data of course, unlike on the web in new businesses that tends to be a much strong call for data governance, particularly in regulated industries obviously. You can’t just go slinging clones of databases full of customer data around for developers to play with – data needs governance, whether or not forking makes for simplicity and better apps.
We’ll be working with Datical on positioning and product strategy, and they already have some solid technology to work with. One of their approaches is targeting folks using Liquibase, an open source tool for database refactoring.
Going back to the partner question and my intro, one thing that did strike me was that XebiaLabs showed up prominently on their partner slide. Second time this week people have been talking about the firm this week. XebiaLabs is a GUI tool for creating development pipelines that integrates with pretty much everything. Of course that’s important – there is huge fragmentation in the developer tools market – it’s not like you can just support Jenkins and be done with it. Developers need to be able to choose their own tools – long gone are the days when the CIO could say “we’re using Rational end to end for ALM”. So XebiaLabs is about making it easier to automate pipelines across the tools choices engineering organisations are making, from unit testing to service and support. I asked why XebiaLabs had made the cut, and the answer was great.
“You can’t teach hustle”. Hustle – that’s definitely something want you want in a business partner.
Anyway RedMonk is pleased to have Datical onboard – we’re doing a fair bit in the data space at the moment. We even have an experienced DBA on the team – Rachel Stephens.
This is what she had to say in Slack during the briefing.
1) it would have been AMAZING if I could have updated my databases with this degree of visibility/auditability.
2) I had to write all my own changes…. is that not how it normally works? maybe this is why I never had any angry developers? :slightly_smiling_face:”
And following up today:
“My auditors would have loved that tool even more than me”.
Internal auditors getting along with CI/CD tooling? Whatever next.
Oracle and Heroku are both clients.
Dj Walker-Morgan says:
May 22, 2017 at 4:41 pm
Oh Hai there James,
Over at Compose, we’ve been cloning databases for a while. We’ve been doing it with the very practical purpose of restoring databases from backup; we do it properly and never restore into an existing database deployment.
That feature, also means you can clone from any backup stored by the system to make a clone of the database at the time of the backup and you can trigger an on demand backup um… on demand.
So with just a couple of clicks (or two API calls) you can backup and clone and be running in no time at all, ready to run your integration tests, analysis or whatever else you want to do with your cloned data. And all you pay for is the hours your cloned database exists for.
We’ve been told customers have already baked the process into their workflows, which is great.
Dj