Riptano‘s Matt Pfeil discusses Cassandra with me over crab-stuffed mushroom caps and drinks.
Download the episode directly right here, subscribe to the feed in iTunes or other podcatcher to have episodes downloaded automatically, or just click play below to listen to it right here:
I ask Matt to tell us where Cassandra fits into the database world, what people are using it for, and how it works. We also discuss the new commercial company around Cassandra, Riptano.
Full Transcript
As usual, I haven’t looked over this transcript extensively to check and correct it: if we say something insane, check the audio before assuming so. There are some time-codes through-out where the transcription service had problems.
Michael Coté: Well, hello everybody! This is another episode of ‘make all’, the little podcast about fun and interesting stuff, and I guess computers, mostly software, but I don’t want to cut out like I don’t know routers, if I ever get excited in that one day.
So this is as always one of your — this is the host, Michael Coté, available at PeopleOverProcess.com, and I am joined here in Austin, which is very exciting, by the guest.
Matt Pfeil: So my name is Matt Pfeil and I am the Co-Founder of Riptano, we are an Apache Cassandra company.
Michael Coté: So we were just talking about something that’s totally non-Cassandra or Riptano related, we were talking about the office space that you guys have, and you have got a dress-design firm there.
Matt Pfeil: Yup!
Michael Coté: And the Web 2.0 medical thing, which you were joking about.
Matt Pfeil: And these are friends, so I want to be nice there.
Michael Coté: Yeah. Well, I should say I was joking.
Matt Pfeil: Okay.
Michael Coté: About Web 2.0 medical devices there, or for doctors it would be like, it’s all in good fun.
Matt Pfeil: Definitely!
Michael Coté: Anyhow, so like this is the favorite topic of mine, is the Austin office space. And let me ask you, so how easy or difficult was it to find an office that you liked, that was affordable?
Matt Pfeil: So I actually did it the easy way, and the friend that I just mentioned, whenever we were starting Riptano, I just got breakfast with him at 00:01:14 Coffee, just sort of ask for advice in general. And it was actually three for one special, because he gave advice, which was great.
Michael Coté: Did he buy the coffee?
Matt Pfeil: I don’t remember that part. But he introduced us to our lawyer today, and then he also 00:01:30 said, yeah, my office is upstairs, come check it out, if you ever need any space. So it was a great 30-minute, got a lot going.
Michael Coté: This is what I found is most everyone I know who is happy with their Austin office space, other than like big companies and things like that, like they get it through — like they are talking with someone and they are like, oh, I have got a good deal for you. Like they direct them somewhere.
Anyhow, so we were going to talk about Cassandra, and when you say it, you say Cassandra or Cassondra, like what do you like?
Matt Pfeil: I say Cassandra, but it’s one of those things where I know everyone says it differently, which is why we named the company Riptano, so you can say Ripteno and Riptano, so we wanted to mimic the project name.
Michael Coté: So you don’t want to lock yourselves in one technology necessarily.
Matt Pfeil: Exactly!
Michael Coté: And so you guys have a — while we are on that, you guys have a rhino as your logo, right, like what’s the story — what’s up with the corporate identity?
Matt Pfeil: So I have got you even more beat than that. First of all, the way that we came up with the name Riptano is we did a random string generator and took out the words that were pronounceable and then cross-referenced that with available dot-com domain names. And so it was about a 15-minute process.
Michael Coté: That’s a service right there.
Matt Pfeil: That’s a service. So for $799, we got everything we needed to start a company.
Michael Coté: That’s fantastic!
Matt Pfeil: And then we got the logo, because we have a really cool graphical designer, and he sent us one idea for a logo and we are like, we don’t like that, and the second shot was the rhino, and it stuck.
Michael Coté: Yeah. So how did you get involved in doing Cassandra stuff?
Matt Pfeil: So Jonathan also is my Co-Founder and we both worked at Rackspace together. Jonathan is the Project Chair at Apache for it, and he was working on the source code while there. I was on the strategy team, working on an effort to build out a multi-tenant deployment of Cassandra internally, so the 20 or 30 Rackspace development teams didn’t have to worry about scaling their own databases anymore, they could have a service where they could utilize that.
Michael Coté: So you guys are looking to centralize that database.
Matt Pfeil: Basically, yeah; basically build an internal sort of SDB. 00:03:23 is literally there to help out the development teams. And at the end of a phone call one day he mentioned who is going to be leaving Rackspace to do a startup on Cassandra. So I took him to lunch on Friday thinking I was going to convince him not to do it, and I asked him and said, who the hell is going to run the business side for you? And he looked at me and said, well, I was thinking about you, and we sort of took off from there.
Michael Coté: Oh, it’s a match made in heaven.
Matt Pfeil: It was a match made in heaven.
Michael Coté: Over lunch.
Matt Pfeil: Over lunch. He did buy that lunch.
Michael Coté: So were you at the Austin Rackspace or the San Antonio one?
Matt Pfeil: Austin Rackspace. I was part of a company called Webmail.us, that Rackspace acquired in 2007. I moved down here in October of 2009, to start to build out the development team for Rackspace.
Michael Coté: Right. So I mean, that’s a good segue into like the kind of — I like to start — when I try to understand the technology, I like to start out with like — the boring way of putting it is, what problem does it solve? Like what is it that it does really well and like why are people using it?
And you were alluding to the first use-case you encountered with Cassandra, which is, you have all these different teams for Rackspace. You are basically providing a bunch of services and many of the centralized data store that kind of fits in. So I mean, what is it that you are finding Cassandra fits for, like where do you slot it in?
Matt Pfeil: So I don’t even think it’s necessarily the — I think you can — the centralized data store wasn’t what they were doing. The biggest problem is, scalability is hitting everyone today. You have issues where you know that you are not going to be able to scale past MySQL Server on day one rather than like year five.
(00:04:57)
So your traditional options are, fine, I will go buy it and I will put it on top of a SAN, and that sounds like really an expensive to do. So it’s just not feasible. So people are looking at all these different options that are cropping up.
So on one side, you could do something like shard MySQL, which is a real pain in the butt, both on an operational and development side, or you can start with something that will handle that for you and utilize that.
Michael Coté: What do you think is driving that scale? I mean, there’s sort of the obvious high profile things like Twitter or Facebook that’s just a huge amount of data accessed all the time. But I mean, is that really the core nut of it, or are there other things that drive that scale need?
Matt Pfeil: I think simplistically it’s that, we are in the era of big data and data is growing faster than hardwares at this point. So what used to be a big boy problem, because they had so much of that —
Michael Coté: Today it’s impossible for individuals to generate a lot of data.
Matt Pfeil: Exactly! Today, we have got startups, we have got a couple here in town, who are starting with Cassandra, because they know that they are just expecting mass amounts of data and they need a scale of that.
Michael Coté: Right. And that makes sense. So first out, I mean, not to get too much into both — I mean why is it that like — so Oracle is expensive to scale up, and then the problem you have with, like you are saying, like sharding MySQL, is that it’s difficult to do. Like expensiveness is easy to explain, why that’s annoying, but what is it that’s difficult about scaling up MySQL?
Matt Pfeil: Okay. So there’s a couple of pieces to the scalability there. First, on one hand, most relational databases just aren’t good at handling large number of writes. With typical — using MySQL as the example, you can have a nice slave form that will let you scale your reads, but you still have to put all your writes at one place.
So then you start to talk about sharding, and that means that, first, you have to run a lot more code to do that for you, which isn’t really fun and it’s a time consumer. And then once you get live and each of those shard grows, you have to start to migrate data between them. So now you have an operational nightmare.
Then lastly, in terms —
Michael Coté: You have got a transactional engine that you are basically keeping consistent. I mean, you are kind of in charge of what the database was supposed to do in the first place, which means you have to be very smart about managing the consistency of it, and the transactional nature and everything.
Matt Pfeil: Exactly! And as each shard starts to grow in there, which hopefully every company wants to grow, so hopefully your data needs are going up, you still have to migrate pieces of those data between shard and that just takes time and effort that is really not a core focus for most people.
Then the last piece I think that you see in the high end right now is, you have multiple data centers available to you, and if you have large amounts of data, you don’t want it in a centralized plant, you probably want to keep it as close to your user as possible. You want it in different areas to make sure that it’s highly available.
And not only that, but multiple data centers are now sort of commoditized. Amazon did a great job with availability regions with Amazon EC2s offering, where you and I can go launch machines in three different data centers across the planet right now and it would cost us like three dollars. So that’s a luxury that is available to anyone at this point.
Michael Coté: So that’s another cluster of motivation, so to speak, is that a lot of the software people are writing is very networked and global, if you will. So when you have kind of an Internet bound piece of software, you start getting obsessed with what data is geographically located and therefore you are pushed towards using a bunch of nodes and different like — you want to disperse your data out, not have it centralized somewhere.
Matt Pfeil: Exactly! I mean, not only for the speed sake, but also the disaster recovery. Data centers go down outside of your control sometimes. So a lot of companies recognize that, and they say, fine, I will just make sure that if it happens, I don’t care.
Plus, I mean, it’s just a really fun computer science problem to have to worry about, replication between those data centers, because you just want to have some fun.
Michael Coté: Right. I remember even back when I was doing, as it was then called J2EE, like I would always get frustrated with people, because you could tell — because — I wasn’t trained as a programmer, as it were, and people were obsessed with like the hard problems of transaction stuff and all this stuff and it was just so frustrating for me.
I mean, it’s frustrating, because I can tell it’s really exciting if you like that kind of thing, but then it’s also a huge distraction from just worrying about the application. But you can tell people it’s a delightful thing they like dealing with.
So given that kind of context, like how does Cassandra fit into that? Like how is it satisfying the needs that you were seeing at Rackspace and now with the stuff that you are doing over time, how are you seeing it fix the problem?
Matt Pfeil: So on the scalability side, in terms of amounts of data or volumes of request for that data, whether it be writes or reads, the nice thing about Cassandra is, as those demands go up, you simply add more commodity hardware to your cluster and the problem is alleviated. So you can grow very incrementally, as you need to, very easily.
Michael Coté: So you can throw hardware at it.
(00:09:54)
Matt Pfeil: Exactly! But you don’t have to throw — so throw hardware, that traditionally means, you throw expensive hardware, that’s not the case usually. It will run on top of a cloud technology really well. The typical machine that most people use is a Dual or Quad Core with maybe 8 or 16 gigs of RAM and then SATA hard drives, so not expensive,
Michael Coté: So on that note, why is it that you can throw commodity hardware at it, like what is it about it that makes it so you don’t have to buy like — I don’t know, a spark box or something, like what’s the magic that allows you to use commodity hardware, or is it just the realization that — or is commodity hardware just cheap enough or that expensive hardware doesn’t necessarily exist as much as they use them?
Matt Pfeil: I definitely still think it’s cheap hardware. I think that — so on a Cassandra cluster, you have any number of nodes. Each node has the exact same role as every other one. So there are no master nodes that have to scale vertically.
What that means is, as your demands go up, you can just add more of those nodes in a reactive mode or hopefully proactive through that new demand that you are seeing. So it literally makes use of whatever resources are available to it on a local machine, and then you add more of those and take full advantage of those new resources.
Michael Coté: I mean, I guess the point being that, if you have a computer that has a network, this network on the network, and obviously has a processor and some storage, you don’t really need — there is no special hardware that you need. Hardware nowadays runs fast enough that whatever Cassandra stuff you are using, it doesn’t need like special fast stuff to work with.
Matt Pfeil: That’s all valid, but if — what I think is really cool is, or at least how I think most software should work is, you take use of whatever resources you have available to you — you just got some good-looking food.
Michael Coté: Yes, some crab stuffed mushrooms.
Matt Pfeil: Delicious!
Michael Coté: Here you go!
Matt Pfeil: Thank you. You simply add machines, and your software takes advantage of whatever resource is available to it, and if you need more, you add more hardware. Because if you think about it, you are going to have hardware probably running an application for multiple years. The hardware that you have on day one, in terms of quality compared to year three, is going to be vastly different.
But that doesn’t mean that you have to ditch your hardware every time you update, you bought that hardware, you are using it, so why not just continue to utilize it, and as you need more, you add whatever is the latest and greatest and it performs whatever percentage better.
Michael Coté: Right.
Matt Pfeil: I strongly feel that the hardware layer should be very abstracted, and you just get whatever you have got, and get the most out of it.
Michael Coté: Right, right. So then the question is like, if you throw expensive hardware at it, does it get better, does it kind of scale that way?
Matt Pfeil: I got asked that question when a client called, and someone said, have you ran a SAN with Cassandra yet? I was like, well, no. A, because it’s a pretty distributed thing and even expensive building blocks fail at some point, so you have single points of failure there. But B, I was like, it’s not really designed for it, but theoretically you could. I guess you could have one SAN at each data center and use Cassandra to upgrade your data.
Michael Coté: I mean, it’s sort of like — the point being there is that really the — if it’s faster, it will be better, but if it’s just like raw horsepower, then like — I mean, if the network is faster or the storage is faster or all these things are faster, then it will obviously be an improvement, but it’s kind of like — I guess it’s kind of like buying tires for your car, right? Like there’s a certain point you get to where the tires are good enough, it will get you to Point A to Point B. And then if you are like a race car driver, you will get the really good tires, that have better handling or whatever. But at some point, it doesn’t really matter much if you have got really specific needs.
Matt Pfeil: Yeah, I tell you, it’s not so much like the quality of the tire in the car, it’s the fact that — that’s a weird analogy, but you could have four tires on the car, but imagine if you could just keep adding tires to your car, and you got better gas mileage on them. So it’s almost like not which one is better, it’s which one gives you more bang for your buck, but they can both achieve the same goal.
Michael Coté: Right, right. So we have got the thing that we can handle reads and writes really fast, and we have got distributed high availability essentially or failover or whatever. All these kind of concepts collapse under the same thing, that you can kill off certain parts of the network and everything is fine.
So then like, what’s the nature of Cassandra? I mean, it falls in this NoSQL category of things, which implies that it’s not like a relational database. So what is — I guess there’s another way of asking this question, like what does it look like when you are using Cassandra? Like how is it different than a traditional database?
Matt Pfeil: So at heart you can retrieve data based off of today a primary fee. So that’s a unique identifier. In the next release, which I think is scheduled to go in beta in like two or three weeks from now, will allow you to look up by secondary index or secondary attribute.
So what that really means is, if you really need a lot of relational things, or you don’t want to change how you thought about your problem from your traditional database, Cassandra is just not the fit for you. I don’t know many people who would build a billing system on top of Cassandra potentially. You can do it, but if you have a large amount of data, and you are okay with learning about the concepts of denormalization to really do that, then it can really solve a lot of your scalability problems for you.
(00:15:02)
What we see as a company is people willing to think about their problem in a different way in order to achieve that scalability.
Michael Coté: Right, right, right. So you said something interesting that if you have a lot of denormalization, it works that way as well, right?
Matt Pfeil: If you can denormalize.
Michael Coté: Yeah, yeah. And those kind of problems typically are — so what are those kind of issues typically that you are seeing where you can’t denormalize the data, like what kind of applications would that look like?
Matt Pfeil: We actually see a wide variety of applications. It’s generally when there is a lot of or something close to user data. That doesn’t mean that this is all Web 2.0 crowd; it means you have data that you want to retrieve, that has something that joins most of it together, or one thing that it’s relating to.
So for example —
Michael Coté: Something that’s very — it’s centric on an individual, like it’s almost as if —
Matt Pfeil: Or entity, I mean —
Michael Coté: Yeah. It’s almost as if you want to relate, I am putting this in air quotes, everything to one individual entity, whether that’s a person or maybe a state, it could be a car or something where — and typically, I guess in a relational database, this would of course be a join. You have got, one table represents the very basics of some entity and you are joining in all this information about that entity.
So with the person it might be you are joining in like all the purchases of this entity, all the places they have been; like this is a very consumer centric thing, all of the different places they have been, the friends that they have, all these weird things, and somehow you are going to pull all that data together and make some conclusion about it, beyond presenting it.
I mean, obviously you are going to present it, but the more interesting thing is to say, now that we have all this data together at one place or this one entity or individual, we can sort of conclude something. Like given all that data, we will make something up.
Matt Pfeil: You made the comment about friends, and it’s almost as if you have a list of friends and I have a list of friends, and we might have this list stored in two different places rather than just have a join table that brings them together. So that’s sort of the — that’s very consumeresk, but that’s an easy example.
Michael Coté: Well, you know what we should do, we will make this a ‘make all’ first, we should pause so we can eat our food, before it gets cold.
Matt Pfeil: I completely agree.
Michael Coté: Alright. Well, we have devoured our crab-stuffed mushrooms, which we were just commenting on were delicious here at the 00:17:34.
Matt Pfeil: You are just trying to get like a free gift card or something out of it with your blog.
Michael Coté: You have got to take all the advantages you can. I mean, for you, it’s basically the best spice in existence.
Matt Pfeil: Free, as in beer.
Michael Coté: So we were talking about — we were saying, if you have sort of non-relational things, not non-relational, if you can normalize things out, those workloads work really well. So I mean basically —
Matt Pfeil: Denormalize things out.
Michael Coté: Yeah, denormalize. I always get my database stuff mixed up. But in looking at — in sort of 00:18:09 I have looked at Cassandra stuff, I mean it’s basically like a big hash state essentially or a hash store, a eValue store. What’s interesting there is, like a relational database is not a natural way to think about organizing data. Like I remember way back when I was figuring out what the hell all this SQL stuff was, like trying to sort that out, it’s actually very challenging and it seems like — and tell me if I am wrong with your experience, but it seems like a key value way of storing data is a bit easier to think about.
But I don’t know, like what’s the friction you see happening in people’s heads when they know SQL database is really well and they move to a key value pair, like is there some magic in understanding that shift or like how does that happen?
Matt Pfeil: It’s definitely not just an overnight thing. It takes a little bit of learning and readjusting to the fact that your data is going to be in multiple places simultaneously. But honestly, today, it’s a key value store.
The next release is going to let you look up by secondary attributes, which I think will really help breakdown the adoption barrier. Because, for example, today, let’s assume that you have got people in the United States or citizens in the United States, they are all supposed to have a unique identifier, that is your Social Security number. Whether that’s true or not —
Michael Coté: That can be used to identify you by any other system.
Matt Pfeil: Exactly!
Michael Coté: It’s that favorite geek joke, right?
Matt Pfeil: Yeah. But in the next version you will be able to look up and say, give me people who are also born in this year, for example. That’s a very natural way of thinking about a problem.
Michael Coté: So concerting together a bunch of things.
Matt Pfeil: Right. It’s very object-oriented to think about your data in terms of its attributes and not how maybe attributes connect to various tables to get the results that you really want.
(00:19:54)
Michael Coté: Not to think about the structure of how they are stored and how you need to like figure out that structure to look up what you want.
And it sounds like what you are saying is, in the next release, the fact that it’s like a key value store is going to be kind of abstracted away.
Matt Pfeil: Yes.
Michael Coté: To some extent it sounds like the evolution, and correct me if I am wrong, and I am guessing it, but based on that one data point, which is a great way of predicting the future, it sounds like the evolution is, maybe Cassandra, as we know it, is more of an engine for a generic data source, that you can look things up generically, and it doesn’t really matter what the exact implementation underneath it is, as far as the key value store, and all your super columns and families and all that business.
Matt Pfeil: Exactly!
Michael Coté: It’s more like you have got some data schema that you stick in there.
Matt Pfeil: Yes, and an arbitrary one, which is — Jonathan Ellis, who is my Co-Founder, was on a CC email for a customer the other day. I mean, this is probably a couple of weeks ago. And I made the comment that Cassandra is a key value store. He replied and said, don’t call it a key value store anymore. He is right. At this point, it’s becoming — it’s moving away from just a traditional, one way to get your data. And that’s really powerful for most of the developers out there. It’s how you want to think about your data, and you store data, especially because different objects at this point will have different attributes, whereas in a traditional relational database, your whole table has to have the same features or the same columns rather.
Michael Coté: And it’s fixed as well.
Matt Pfeil: It’s fixed and moving —
Michael Coté: It’s extremely difficult to add a column or to add an attribute, to use the word you were using earlier.
Matt Pfeil: Especially in a large dataset, where your ALTER TABLE had a large deployment, by taking months to roll out to every single server. So yeah, now you can do that very — if you can define your data with different attributes very much on the fly, like you can in Cassandra, you should be able to start to retrieve it very arbitrarily and very easily.
You really just want to get the data based on unexpected or even unanticipated ways.
Michael Coté: Yeah. So correct me if I am wrong, but it seems like the basic atomic unit you start with in Cassandra is a name value there, right?
Matt Pfeil: Yes.
Michael Coté: Then that gets stuck into a column, if I remember. Like walk us up from there, instead of me embarrassing myself with my paucity of knowledge.
Matt Pfeil: That’s fine. It’s actually — this is an area where if I say anything wrong, make sure Ellis kills me. But you start with this key value pair and your key is this look up that returns your row from the row, you have columns on it. Those columns could be thought of as your attributes. Those attributes or columns can be different for every single row.
Michael Coté: Right. So you have a row and the columns are like a bag of name value pairs essentially. Can you have unlimited name value pairs in there or only one?
Matt Pfeil: Inside of a single row you can have an arbitrary number of name value pairs. So using our Social Security example, my Social would retrieve a column or I will try a row and the columns could be everything from my first name to middle name to mother’s name to address. And you can add those on the fly.
Michael Coté: And each of those things, like your mother’s name, is a column and a row, is that right?
Matt Pfeil: Yes, exactly! But there is no guarantee that you and I have the same columns or number of columns. I might have a sister column, you don’t have that.
Michael Coté: And that’s another key thing is, there is no enforced schema to use a relational database, there is no enforced, you have to have these attributes in this order or whatever, it’s all dynamic, or kind of made up, if you will.
Matt Pfeil: Exactly! That’s awesome, because you know, who are we to guess what data we are going to store a year from now. Things change. So you should have a data store that allows you to change those changes without pain.
Michael Coté: So to get a bit parenthetical on that, what is the — so let’s say you are doing the upgrade from Version 4.8 of your application into Version like 5, are there typically like database upgrade scripts that you do or how do you handle that?
Matt Pfeil: So the nice thing about Cassandra and the fact that your data in general is replicated between nodes is — and since every node has the same role in a cluster, you can do a rolling upgrade, where you will take a node out of the cluster, upgrade it, and then bring it back, and you can just do that around the cluster of servers without affecting the —
Michael Coté: And that’s how like the gossipy aspect of it works, right?
Matt Pfeil: Yes.
Michael Coté: And by that I am kind of — I don’t know if I am making that, but it’s sort of like, the nodes talk to each other and they are kind of like, hey, this is what I know about the state of the world, and they tell it to the other node and then that replicates out basically.
Matt Pfeil: And that just really matters in terms of when you are thinking about life and death. So if a node is not available, you want to make sure that the other nodes don’t think it’s around, and that way you handle failures really smoothly. So taking a node out of an application —
Michael Coté: So you don’t have false gossip, as it were.
Matt Pfeil: Exactly! False gossip about important events.
Michael Coté: Right, right. So then, the way you do a database upgrade, so to speak, is you more see the upgrade. So you might tell one node, here is the current state of the world, like this is the new way — and I am kind of retrofitting these words in here, and here is the new schema, even though there are not schemas, like here are the new attributes that are on something.
(00:25:03)
Matt Pfeil: Here’s the new engine, how about that?
Michael Coté: Right, right. And then that spreads to the rest of your database, if you will. I keep doing air quotes here.
Matt Pfeil: Air quotes are important, it’s what bring us together in life. No. I think the biggest thing around the rolling upgrades is that, today Jonathan and the rest of the community have done a really good job at making sure that future versions are backwards-compatible.
If Jonathan was here he would say, I am not going to promise that forever, but they have focused on making sure that you can upgrade without having to shut down the entire cluster simultaneously.
Michael Coté: Right, right.
Matt Pfeil: Because that would sort of suck, because then you would have to turn off your whole cluster.
Michael Coté: Right. Well, that would be one of the major benefits, if you have high availability or whatever you want to call it, right? That gets your 0.9% downtime.
Matt Pfeil: Yes. No one wants that 0.9%. But literally, you can turn off one machine at a time, upgrade it, bring it back, and just do that around this cluster, so that you never experience downtime with the end user.
Michael Coté: Right, right. So then I interrupted you explaining, so you have got the name value pairs, which is a column and that translates to a row, essentially.
Matt Pfeil: Yes.
Michael Coté: Then what are the other sorts of data models?
Matt Pfeil: So there is the notion of a column family, which is equivalent to a table in relational speaks. So in other words, an organizational unit for a collection of entities. There’s also a super column in there, which is basically when you have pieces of data inside of a single column. So you can think of that, to some extent, as an associative array. That’s one where hopefully Ellis doesn’t also come back and say, you describe your problem, but that’s how I understand it. So full disclosure on the business side of this.
Michael Coté: That’s right.
Matt Pfeil: Yeah.
Michael Coté: Alright. So that’s kind of like the collection of stuff that you have. So then the other angle is — and this is all like an Open Source 00:26:55 Apache and you can download it and whatnot. So you have got sort of like the library and the raw software, if you will, the framework I should say. Usually with a database, there is a whole bunch of other tools that come around that are sort of like managing it and deploying it and keeping up with it. Like everything that is frustrating about it, not about the actual software itself, but making the software happy.
So like, what does this probably look like as far as — like what do sys admins do with this, like when you hand this over to production and they need to make sure it’s up and running, and you have got to do that thing like deploy the new versions of the nodes, like what does the management of it look like?
Matt Pfeil: Honestly, today, it’s not great, and that’s actually an area where a ton of things that we can really add value, to view as an opportunity. Or what we are working on right now is sort of management software and tools that we will be bundling with our support offerings today. And the goal there is literally to make the end user’s life, and that today being the op/admin, as good as possible. And the best support is proactive, where you can tell that there’s going to be an issue ahead of time. So that means things like upgrades or adding nodes to the cluster, so you can increase your capacity.
Michael Coté: So you can do planning and things like that.
Matt Pfeil: Hey, planning, who wants to do planning though? Someone said do it live, right?
Michael Coté: That’s right. Well, I mean, that’s like — I hadn’t thought about it in this detail till now, but that’s one of the big divides between development and operations, is operations people are like planning, that’s what I like doing.
Matt Pfeil: Developers are like, I will just show up to work at 1 o’clock and do whatever for the next four hours, like play ping-pong.
Michael Coté: That’s right. Planning is a way to limit what a developer can do. Whereas for operations people, it’s very liberating to know what the plan is. Because otherwise you are sort of at the whim of those crazy developers essentially.
Matt Pfeil: Developers don’t care about five nines as much as operations guys do.
Michael Coté: So then the other thing that gets mentioned a lot in this area, and I feel like it’s kind of a cliché to talk about it, but it would be very inconsistent, as it were, to talk about it, is the eventual consistency. We have kind of like danced around it, as far as like the nodes getting updated and the gossip and things like that. So like, what’s that all about?
Matt Pfeil: I love it! So this is the biggest misnomer about Cassandra that I am so happy we can talk about, because there is this notion right now that eventual consistency means that your data, if you do an update or a write, and then I try to retrieve that data, I am going to get a different result than what I just put there.
Michael Coté: Right, because you are going to lose items from the shopping cart.
Matt Pfeil: Yes, you are going to lose items from the shopping cart. Cassandra offers tunable consistency. Basically what it means is, when you start your project, you pick a replication factor. So you decide system-wide how many copies of each data you want. Most people choose three if they are living inside of a single data center, because that means that you can have one node down for either failure or maintenance and still have high availability. Or people who have multiple data centers usually will do two copies of each, just because of cost.
But the cool thing about Cassandra is, per query, so read or write, you have got this option to do tunable consistency. And we will go through a few cases to probably explain this the best.
(00:30:04)
First of all, if you want your data to always be consistent, all you have to do is make sure that when you write your data, you write to at least half of the nodes, and when you do your read, you read from at least half of the nodes.
So in other words, if you have a value N, that is your replication factor, then you have a read value, that is the number of machines you will read from, and a w for the number of machines you are going to write to, R+W have to be greater than N, and then you will always get the most up-to-date data. So it’s good for an example there.
Let’s say our replication factor is 3. If I write to 2 — oh, that’s bad, yeah, if I write to 2 nodes and then I read from 2 nodes, I will always get one of the most up-to-date copies and we will always get the most up-to-date data.
Michael Coté: Right.
Matt Pfeil: Now, performance matters to some people more than others. Who are you and I to decree how important your read performance is, and obviously writing to two machines is slower than writing to one machine, and just like reading from two machines is slower than reading from one machine.
And so an option is, if maybe that consistency doesn’t matter and you can have a small window of five or ten seconds, or usually in Cassandra it’s your networks, so we are talking about milliseconds, you can have a small window where your data might be inconsistent, but it’s fine for those people, and they just really want to get that read as fast as possible.
And so on each query, whether it’s read or write, you can determine how many nodes you want to have respond to the initial request. So if your replication factor is three, I could do a write to all three copies, and I know that no matter what, any read after that is going to return the most up-to-date value.
I could also just write to one data before I return to the user and then Cassandra in the background will start to replicate to the other two machines, and you will have a small window of time where there is a chance that, that data is inconsistent. But it’s completely tunable to you, the developer, to achieve what you are really looking for the most.
Michael Coté: Right. I mean, that’s one of the interesting things about the eventual consistency is, you never want to sort of market the — you never want to — it’s difficult to market this kind of thing. So it makes sense why it doesn’t come up, but it seems like part of it is, you don’t really — when you are using a relational database, you are not given the choice to do that. Like there’s only one way of doing it.
Matt Pfeil: You sort of are though. Because if you think about it, like let’s use MySQL replication as an example.
Michael Coté: Right, well, in that sense.
Matt Pfeil: Yeah.
Michael Coté: But if you are just using like one single database, as it were, without caching and replication and things like that, it’s kind of — the whole point of using a database is that it’s always consistent, and you can’t really tune it, if you will. So it seems like that’s one of the interesting things about Cassandra and other NoSQL stuff is, they kind of throw that assumption out. They are kind of like, you don’t really need that as much as you think you need. I mean, you don’t need it to the degree to which a database provides it to you.
Matt Pfeil: Anytime you bring a distributed system into play, you run into this notion of, what do you do about data not necessarily being in every location across the network simultaneously. If it’s on a single machine, you don’t have that problem.
Cassandra simply says, we know it’s an issue, you can pick how you want to handle it, and there is a lot of options you can do based on your specific needs.
Michael Coté: Right, that makes sense. Well, I think that’s a pretty good overview. Do you think there is anything important that we left out?
Matt Pfeil: No, I think it was good. There is a lot of smart people working on the project right now, 00:33:24, and it’s sort of really nice to see that these people who just have a common interest are working together to solve a problem, and at the end of the day it’s free for the world to take. So it’s sort of cool that we have these extremely smart people working on one of the more difficult 00:33:38 problems. And a lot of these guys aren’t even getting paid for it.
Michael Coté: Yeah. Who would have thought that there would be like a revolution in database — or evolution in database technologies, one of the more tweedy parts of the programming world.
Matt Pfeil: Someone will have to tweet that and we can re-tweet it to make ourselves seem cooler.
Michael Coté: That’s right. I mean, since you are the, as you said several times, the business side of Riptano, as it were. I mean, what are you guys up to, like what are you guys providing? How do you fit into this whole crazy thing?
Matt Pfeil: We just sit on the side and watch everyone talk. We don’t actually do anything. So today we have a very service-based business, where we offer solutions like training and consultation and then support contract, so if you are using Cassandra or want to learn more about it, we can either help teach you and really get you up to speed with it. And then we can help you sort of, as you are designing your application, we are here to make sure you are doing it the right way.
Michael Coté: Give some architectural guidance.
Matt Pfeil: Exactly! And then once you are up and running in production, if something does go wrong, we can provide you with a number to call or someone to really make sure that you can get help when you need it.
We are also working on some of those tools in management software we talked about to make your life easier, and that stuff that’s in development right now.
Michael Coté: Kind of like the Cloudera way of going about things.
Matt Pfeil: Very similar.
Michael Coté: Right, that makes sense. Well, thanks for taking all this time to talk.
Matt Pfeil: Oh, this is great. Thank you.
Michael Coté: And we will see everyone next time, hopefully at some place that has equally good appetizers.
Disclosure: The Apache Software Foundation, where Cassandra is homed, is a client. See the RedMonk client list for other relevant clients.
http://www.truereligions.in/