It should not be controversial to say that the way software is written in 2018 is materially different than in 2008, let alone 1998. Not just in terms of the underlying technologies, which have inevitably evolved and been refined, but the associated processes and methodologies. Whether driven by the introduction of transformative technologies ranging from distributed version control systems to cloud infrastructure, or by hard won insights from the likes of a Kent Beck or Martin Fowler, the software development world today would be largely unrecognizable to a developer from even a decade ago. It’s important to note, of course, that this software future is distributed unevenly, and there are thousands of businesses today which are profoundly uncomfortable with the present and future directions. But the direction of travel in the industry is clear, and it does not look like traditional waterfall processes, basic three-tier architectures or manual operational cycles.
As has been discussed in this space, however, the same cannot be said of the database world. Much as database customers once resisted the intrusion of open source in a manner quite distinct from other categories such as application platforms or programming language frameworks, database operators and their customers have continually pumped the brakes on the types and rate of change that are taken for granted in the application development world.
There are signs, however, that that resistance is coming to an end. Whatever the nascent reconsideration and reevaluation of database technologies, processes and procedures comes to be called – DataOps feels almost inevitable, even if it would have to be coopted and dramatically broadened – change is coming.
There are many factors that will reshape the database category as we know it.
- Most obviously, there’s the cloud. It’s no accident that Amazon’s fastest growing services are databases, that database market incumbents IBM, Microsoft and Oracle have rushed to get hosted databases to market, that non-traditional databases like MongoDB are spinning up online services, or that would be Amazon challengers such as Google are leveraging their database offerings as a differentiator: demand for database-as-a-service offerings is skyrocketing. Much as resistance to open source databases within the enterprise seemed to ultimately crack under the weight of rampant adoption, so too are enterprises increasingly recognizing that the horse has left the barn, that whether they like it or not, DaaS is part of their future.
- Separately but often in conjunction with cloud offerings, there are a variety of vendors attacking aspects of database modernization problems. Vendors like Citus, Datical, Datos, DBMaestro, Delphix or Treasure Data, for example, are focused on everything from scalability to release automation to data integration, portability, migration and security. While many of these vendors have been around for some time, the climate they operate in – one which is increasingly focused on velocity, which means emphasizing speed at every operational layer – is at once improving their visibility and their own ability to push for organizational change.
- Lastly, but most importantly, is the customer feedback. In conversations with five CIOs in the last three quarters each has, in either private conversations or presentations, acknowledged first that their data infrastructure needs to evolve and quickly, and second that they’re in the midst of executing on these changes now. The precise motivations varied, but the most common element cited was the need for improved velocity. And in every case, the lack of innovation within database operational and design structure relative to its application development counterpart was cited as a major issue.
If we assume, if only for the sake of argument, that the database world is poised for evolution if not revolution, the obvious question is what this means for the men and women who operate and manage databases today: the DBAs. Two points of evidence point to significant, systemic impacts to the category moving forward.
- First, we have the historical example of DevOps to extrapolate from. While measuring its precise impact within a given organization or across the industry as a whole is a challenge, we know that DevOps has typically come with changes in job content and responsibility for developer and operator populations alike.
Second, are the more salient experiences of customers like the ones mentioned above. In every case in which a CIO or other executive has driven or authorized substantial investments in service-based database infrastructure, changes in DBA roles have followed. As two financial industry executives put it at a conference in Jersey City and re:Invent, respectively, their DBAs are all being moved to doing more generic DevOps-style roles, roles that involve more architecture and engineering than traditional database administration. This is the logical outcome of a scenario in which making a database fault-tolerant with 6 copies across three availability zones with continuous backup is now merely a product feature instead of a full time job or jobs.
That change is coming is clear. The question for DBA and other database practitioners therefore isn’t whether their roles will change, but whether said changes will be new opportunity or existential crisis. The answer will, as in any industry, depend on an individual’s willingness to change, to adapt and to learn. Those that insist on practicing nothing more than what they do today are likely to face substantial headwinds in their careers, as the conflation of software and services that is as-a-service offerings obviates the need for work that was once the sole province of human administrators.
For those willing to expand their horizons, however, the opportunities both intellectually and compensation-wise should be substantial. Just ask the developers or operators who astutely positioned themselves as qualified DevOps resources by expanding their capabilities to meet the market demand for broader, more versatile skillsets.
With every new wave of automation comes fear, uncertainty and doubt about its ability to destroy jobs. But much as Automated Teller Machines resulted in net growth of jobs for human tellers, it seems probable that DBAs that want to learn new skills to replace those that have been automated away will have ample opportunity to do so.
Disclosure: Amazon, Citus, Datical, Delphix, Google, IBM, Microsoft, MongoDB and Treasure Data are RedMonk clients. DBMaestro is not a client.
Robert Treat says:
January 9, 2018 at 5:42 pm
The recent wave of cloud services and DBaaS should hardly qualify as an existential crises for those who have chosen the DBA field as their career path. As with all automation, reducing one aspect of the workload doesn’t eliminate the need for work to be done, but frees people up to focus on the next level of problems. Here are just a few outcomes to expect over the next five years as DBaaS offerings continue to pick up steam.
1. DBA’s who have spent the last five years focused on dealing with multiple systems, failover, upgrade scripts, and other operational concerns, will now be free to focus on other tasks. After all, it’s not like access control, network access, garbage collection, and other traditional operational concerns are all going to disappear. Even in the cloud you still need to test your restore procedures!
2. That said, once you aren’t writing massive amounts of shell or config management scripts, you can now focus on other database needs such as query tuning and schema design. As a bonus, most cloud services make it easier to make copies of data and/or spin up resources for testing; and from a business standpoint, focusing on improved performance of the instances you have rather than just constantly spinning up new instance means straight cash homey for your bottom line.
3. The growth of DBaaS solutions isn’t as simple as Amazon wiring up a Postgres instance and pressing go; we’ve also seen a rise in alternative database solutions within the cloud as well. If you can reduce the operational overhead of running different kinds of database systems, you increase the need for having people who can architect across an entire set of similar services like Redshift, Aurora, and Postgres RDS. Not to mention understanding when you shouldn’t use any of those and instead use S3 (maybe with a Foreign Data Wrapper!)
4. One of the big reasons NoSQL exploded onto the scene was that traditional RDBMs made running distributed services rather difficult. People (all too quickly IMHO) tossed out all the benefits that 30 years of RDBMS had evolved for theoretical reliability guarantees. Once you put those systems in the cloud, those advantages kind of go away, and I think we’ve started to see an early signs of people returning back to the RDBMS world.
So yes, once again the database industry is undergoing changes, but no, it won’t be the end of databases and don’t worry, you aren’t going to see DBA’s out begging for burritos in the mission district. It is change, and change means opportunity with the right caveat; you have be someone who can see the tradeoffs and who is focused on taking advantage of what is unfolding.
This Week in Data with Colin Charles 23: CPU security continues to draw attention - Prosyscom Hosting says:
January 12, 2018 at 1:57 pm
[…] else worth thinking about, Stephen O’Grady writes Whither the DBA. What do you think? Don’t forget to look below the fold for Robert Treat’s comment […]
January 16, 2018 at 6:25 pm
As an ex-DBA I know that two things would get me fired
1. Loss of service availability for the shared data asset
2. Data loss
Having to guard a shared facility where every user of that facility has no consideration for other users is a tough. It gets really bad where there is a strange belief that data husbandry happens by some automagical process. Things are blamed on DBAs that really aren’t a DBAs concern such as data quality. Hands up who has business stakeholders that play an active role in ensuring that data quality concerns are addressed? Liars the lot of you. Users of data want the quality to be good but they don’t see that they should have to play a role in ensuring that it is good.
Why are DBAs slow to do stuff? Read The Phoenix Project. They are probably doing stuff that needs to be done to ensure continuation of service (because they’ll get blamed/fired if the service halts) but is really nothing to do with them.
Developers and business people aren’t thick, they know a spade job when they see one so no-one is keen to listen to a DBAs concerns, much less action them because that means volunteering for the tasks they’d sooner blame the DBA for when it doesn’t work.
Lets take source control as an example. Developers use Git day in day out. Imagine that you don’t. You are willing to step up to the challenge of putting DBs under source control. What you need is someone to mentor you through the process, branching, merging, release packages etc. If you are doing it every day of your life it is really easy. If you are trying to learn it and (more importantly learning to do it properly) then its a bit more challenging.
Database stuff would be really easy if you didn’t have to worry about the data.