What happens when a software engineer decides to track his dog’s bathroom habits with open-source tools? You get Prometheus for Pets, a hilarious, surprisingly educational talk from Ronald, who turns everyday pet care into an exploration of where IoT, data analytics, and your pets meet. Using ESP32 boards, Prometheus, Grafana, MQTT, and a few LEGO bricks, Ronald shows how to build smart pet gadgets, from door sensors to water-bowl monitors, while blending humor, the rare Monktoberfest live demo and genuine love of tinkering.

Transcript

This is my first time at Monktoberfest. I want to say first of all, this is amazing. Like there have been these talks on like life and death and I developed my personal philosophy and solved three underlying physics problems along the way, and I’m a little bit intimidated because my talk is basically, I can tell you when my doggy goes pee-pee, so we’re going to see what happens!

     This is Prometheus for pets and we’re going to talk about data — you’re actually thinking is this your dog and the answer is, no, I would never turn my dog into a board. And no, this is a drawing. Think before you ask questions!  Come on.

     This is my dog. The closest to making him into a board that I’ve ever done is put my glasses on him and I got a withering death stare as a result. By the way, I rehearsed this a couple of days ago at home and when this slide came up, he growled the entire time. So he does not like this picture, he did not like me sharing this. But he’s not actually a board.

     So what am I going to talk about here? Well, what is this talk? Could be keeping tabs on your pet. It could be things at looking at water consumption or how long they’re outdoors, but in actual fact, it’s about having a fun project to get into IoT, Internet of Things technology, data analytics, things like that. I know whoa I’m trying to learn something or trying to play with new technology, it makes it a lot more interesting to me to have a solid problem to engage with, something real to do.

     I will say this is going to touch on hardware and software and one of the weird things about being in an audience like this or having an audience like this, is that probably everybody in here knows more about at least one of those topics than I do, but my experience has been that not everybody knows both. And like I was a software guy for a long time. I went to school to get a computer science degree. I wrote code for decades. I never really grokked hardware, I never really understood it. So that’s why it’s kinda weird that ten years ago I went and worked for an IoT company and at that point I started to learn hardware and it wasn’t as scary as I thought it was. And I’ve gotten to the point now, ten years later, where I can successfully nine times out of ten pick up a soldering iron without burning myself, so I’m doing great.

    [applause]

     Disclaimer first of all, I’m not a veterinarian, none of this is medical advice. Please please please do not try to make something at home your only way of keeping tabs on your pet’s health. I hope nobody would do that. Your pet loves you, they want your attention, you should engage with them. Don’t try to automate them away. But other than that, let’s get going. We’re going to talk about both hardware and software. Everything I will talk through I will give more detail about the time that we go to use it in the demos. So we’re going to use some IoT dev boards based on the ESP32. If you don’t know what it is, that’s fine. We’re going to use some cheap off the shelf sensors to get data into these things. We’re going to use some cheap, off the shelf devices to get data out, think lights and sounds, things like that. So I’m going to run this on a laptop, they can run on a desktop, it can run on a Raspberry Pi. Everything I’m going to show is very light. It doesn’t need a whole lot of power to run and on the software front, everything here is open source, so we’re going to look at Prometheus, we’re going to use MQQT and we’ll talk about that later and a visualization engine to look at the data.

     I hope everybody gets to take something out of this.

     So I’m going to start with the hardware front and these are microcontrollers, these are actually development boards with microcontrollers on them. Maddog held up a single-board computer earlier. These are the kind of things he was talking about. And the reason I love these is that these are powerful and cheap. They’re powerful for the size that they are. They’re not powerful compared to the processer in your computer or the processer in your phone, but they’re actually not that bad, if one of these things were a desktop computer 40 years ago, it would blow everything out there away. The ESP32 which is the chip I’m focusing on has two processers. It runs at 240 megahertz, it can have megabytes of storage, megabytes of RAM and it has wifi and Bluetooth and a bunch of other things in it. These things are insanely powerful compared to what we had everywhere running businesses 40 years ago, and on the flip side they’re super-cheap. The quarter there is for scale, it’s not because they only cost a quarter. But these development boards can run you two or three dollars. I think the most expensive one I’m going to show in this demo was around $16. You don’t have to worry about playing with them. If you break one, OK, who cares, you can put it into a project, you can leave it there and they’re insanely low-power, as well, so you can use a USB battery like you would use from a phone charger to run one of these things for days, weeks, even months, depending on the workloads you’re running on it. They’re super-cool devices, I get excited.

     Software side we’re going to focus on Prometheus, MQTT and Grafana. It’s just basically where we’re going to store data we collect from these devices. We need a place for that data to live, Prometheus is going to be that place. MQTT stands for message queuing telemetry transport. We’ll talk a lot about that later. Don’t worry if you don’t know that. And Grafana is the visualization engine for this. Prometheus, all kinds of APIs, and give you a set of tools to see all the data in one place.

     And what are we going to do with these things? I’m going to build some toys. We’ll build a pet door monitor, so we’ll start nice and simple, just track how often your dog or your cat or your lizard goes in and out of the house so we can look at trends in that data.

     Then we’ll build a water monitor, we’ll see how much your pet is drinking and how frequently. We’ll build an alerting system. So we have data, what can we do with it? Well, we can say when something goes wrong and finally we’ll take some of that data and perform analytics on it to see even cooler things we can do as you add more and more data.

     So the title of this talk is Prometheus for Pets. I should probably talk about Prometheus at some point. So what is Prometheus? It’s an open source project, and if you are anywhere in the cloud computing area, you’ve probably used it already, but if not, Prometheus is sort of the de facto metric storage database. So it’s a time series database, time series data is anything with a time stamp and data, so think of a weather report. it’s a certain temperature in the morning and it goes up and down throughout the day. Same thing with investment data.

     Hey, give me the data for last week, or tell me what the average was over time for this series of points.

     It does have a lightweight UI it’s got a web UI which we’ll look at briefly. Most of the time you’ll be using it with Grafana, they go together like peanut butter and jelly.

     Prometheus reaches out and says give me data, give me data, give me metrics. Prometheus for some reason calls this scraping. So if you hear me talking about scraping data or scraping it’s not some weird medical thing I’m doing to my arm, it’s just what Prometheus calls this.

     And I know this is absolutely unreadable. You don’t need to read it, don’t worry. I wanted to talk about why Prometheus has become so popular. And a lot of it is it’s incredibly easy to produce metric in Prometheus format. So this is actually an excerpt from one of my systems at home from a computer that I have running in my garage and you can see — well, you can’t read, but hopefully you can see, this is just a text file. It’s plain ASCII text. What the computer sends back is really three things. It sends a metric name, so CPU utilization, for example.

     And then finally it sends a number, and it’s always a number, it’s never a text string or anything else. And it’s always a 64-bit floating point number, so there’s not a lot of options in producing Prometheus data, but it makes it very, very simple to add this to a program.

     So it’s kind of caught on like wildfire because it’s very easy to integrate with.

     All right, so let’s actually build something. Sneak preview, it’s this. This is just a cat flap that I ordered on the internet. You can see it’s a window with a microcontroller and — so the microcontroller is the brains here. This is the processer that is doing all the work. In this case, this is a board from a company called Adafruit. I’m not affiliated but I do love their stuff. They have a lot of good documentation, they make it really easy to get started.

     But this board is called the cutie pie and the reason it’s called cutie Py is because it runs Python already. If you’re a Python developer, it’s a simple language to learn and use, and the real advantage of it here is that Python is interpreted. You work with Python in real time. You write a statement in Python and it executes immediately. So you can plug this board into your computer, and you can start writing code without compiling, without any extra steps and it will just run on this device, so it’s really cool for learning and experimenting.

     And then attached to that, we’ve got an accelerometer and an accelerometer is just a sensor that measures acceleration.

     So why acceleration? Well, in this case we want to know what orientation the device is sitting in, and it’s measuring acceleration in three axes. So the cool thing is, we can measure the acceleration from Earth’s gravity using this. So if this thing is sitting like this and we’re measuring the force in this direction, it’s 0, there’s nothing pushing the device right now, but if I lift it up like this, now there’s an 9.8 meter-per-second force pulling down on it, just gravity, right? I pull it the other way my arm doesn’t bend that way, but we can measure the force of gravity and use that to tell what orientation the device is in. And of course we’re going to use Prometheus to store the data. As I said before, Prometheus is going to scrape the data from the microcontroller, pull it back, and store it.

     Looking at the code, like I said, this is Python code, so it’s very straightforward-easy to use. You don’t need to read this. I’ve published this on GitHub, I’ll give you folks a link at the end. But I Harped earlier about Prometheus being so simple to build metrics for in part so I could get away with what I’ve done in my code here. It’s Python, there are Prometheus libraries, I could have pulled a library in. But in this case, I’m literally just tracking how many times was this door flipped up and down. It’s so simple I don’t need to pull in a big library. Instead what I do is the first three lines are start up a web server, connect it to wifi, listen for connections on port 8080, and the rest of this is every time somebody comes asking for metrics, just send back a plain text representation of that Prometheus format and append the values to it. So I’m just saying, it opened, it opened, it opened, and I’m just spitting out plain text, so I can create a Prometheus instrumentation in like six or seven lines of Python. It’s really easy.

     The rest of the code on this device is just capturing the metrics. It’s — I’m going to loop forever. It says “while_true” so it works forever. If that value is greater than 4, then we’re going to assume the door is open. Why 4? Again we’ve got 9.8 meters per second. If it’s 90 degrees out, I’m going to say if it’s halfway, then it’s probably. If it’s four in the other direction, the negative direction, it’s probably open this way and otherwise we’re going to assume the door is closed. So that’s it. We’re going to run that forever and ever and ever and every time Prometheus wants something on that metric, it’s going to pick it up. Everybody cross your fingers. The most nerve-wracking part of any live demo is the live demo. It’s my Yogi Barra moment for the day.

     Oh, sorry, I’ve got to grab an adapter.

     USB-C and USB micro do not fit together. All right, I’m going to plug this thing in. All right, this is my nothing up my sleeve moment. Not connected to my laptop …

     Now, I have Prometheus running locally on my laptop, like I said, this is very, very lightweight, it doesn’t take a lot of resources to run, so I’m going to simulate some data. I’m going to go in and out my door a couple of times as if I am my dog.

     All right, now, I’ve got Prometheus set up to scrape this data every five seconds. So hopefully I have talked long enough that it has done a scrape and if I just search for the door use — yeah, that’s what I called it, door_use_counter_total. you can see here, probably can’t read it, says entries of 2, one that says exits of 1.

    [applause]

     Come on, it’s not that great. We’ll do more interesting things later, trust me. Prometheus will let you graph some of this, too. Let’s zoom out and look at the last minute. I’ll flip it a couple more times.

     And if I have waited long enough I will come back and refresh this and we’ll see that it continues. Yeah, so it is good, go us, pat on the back, we’ve got some data, but we can do more. So let’s leave the pet door for now and let’s build something more interesting.

     All right, another thing we can monitor with our pets is their water consumption, right? Water has weight. We know that we can measure weight with scales, so let’s do that. In this case I’m going to use a different microcontrollers, I kind of did this on purpose. I’ve got several different vendors, several different microcontrollers, just to show there’s a variety that you can use. You’re not tied into one ecosystem. In fact I should show earlier I did limit myself to things that you don’t have to solder, so all of the things that I’m showing, literally, they come with wires, you plug them in together, not that solder is that scary.

     So this device is from a company called M5. This is the M5 stick C+, great name, and instead of Python this is going to run something called Arduino. So it allows us you to write C++ now, you might be saying why would I write C++ when I could write Python? As I said before Python is an interpreted language, there’s something on your board to run that code and write it. So you have to have a fairly big board to run the Python board and you’re losing some of the clock cycles on that board to the interpreter. If I write it on C++, yeah, I do have to compile it but then it’s just running natively on the board, I can get away with a smaller, cheaper piece of hardware. Another real reason to use Arduino, though, is it’s been around for a long time and has a huge community around it and that community has developed a lot of libraries, so if you want to do anything — you want to use any sensor, any weird sensor that you can find, somebody has written an Arduino library for it. You want some weird output device, somebody has written a Arduino library for it. So it makes it really easy to go grab these things, plug them together and just build what you want to focus on.

     Now, I’m going to connect this to a digital scale. And I’ll talk more about the scale in just a moment. Let’s move on to the rest of the architecture here. Again we’re going to capture this data in Prometheus, but this time I’m going to switch it up, instead of having Prometheus scrape data from the device, I’m going to push data from the device to Prometheus. People have strong opinions about whether pulling or pushing data is the right thing and holy wars are fought over it, I’m not going to get into it. You can do what you want and this time we’ll add Grafana onto it to visualize the data.

     Now, finally, finally, you get to see why I have my LEGO bricks sitting in my closet since the 1980, it’s because the I was going to 3D print a stand for the scale, but then I realized I had LEGOs, and I could just build it and that was way more fun, so this is a picture of kind of the underside of the scale. The scale itself is sitting in between the weird basket on the top and the red bricks on the bottom. It’s just hard to pull apart and get back together on stage.

     So if we look at the code for this, this is C++ instead of Python, but it’s kinda got the same structure. There’s a bunch of boilerplate, again you don’t need to read all this, but it’s basically just setting things up. I’ve run this one thing when the board starts and it’s like initialize the board, connect the scale, turn on the LCD, connect to wifi, things like that, so it’s basically boilerplate, it’s the sort of thing you’re going to repeat in every project. But then we’re going to loop over and over and over, and in this, basically it’s the same idea. I’m going to get the value from the scale, I’m going to print it on the LCD and I do a little bit of math here that says, OK, if it’s less than a third full I’m going to print a message that says it’s empty, and if it’s more than 2/3 full I’m going to say it’s full. We’re going to push that value to Prometheus and wait five seconds and do it again. We don’t need to check the dog’s water every five seconds. Five seconds is more than enough. If one demo is scary, two is terrifying. All right, I’m not going to bother putting the display in a way you can read it, because it’s too small for anyone to see anyway.

     But, I’ve got a scale, I’ve got a water bowl on it, and of course, I have whatever it is, that they use in paper towel commercials

    [laughter]

     I assume this is dog Gatorade, so we’re going to give it to Ruggles.

     OK, I’ve filled it up enough that hopefully you can see a little bit of the blue liquid, and what we’re going to do this time is go to Grafana to look at this, so let me log in. Please don’t try to guess my password, it is definitely admin. And what I’m going to do is I’m just going to quickly make a dashboard to visualize this. So create a dashboard, I’m using Prometheus as my data source, I believe I called this weight, yeah, weight in grams, and that’s it. If I refresh this data  — OK, good. Let’s shrink this down, because looking at the last six hours isn’t going to tell us much. All right, this is the last five minutes. Cool, maybe we add a little bit of water.

     And hopefully — I added it at just the wrong time. I have to stall for five more seconds. We should see that line bump up a little bit. So now we’ve got the basics of we can see when the dog goes in and out —

    [applause]

     Thank you, thank you, we can see when the dog goes in and out and we can see when the dog is drinking water. So the final part that we want to do is make use of this data, right? So this is where I’m going to bring in the final piece of software, MQTT, it’s basically just a lightweight message transport system, so if I wanted to do something like have a digital scale that measures water and maybe make a bright light or a sound play when the water bowl gets too empty, I need those two things to talk to each other, right and I could write a bunch of code on each of these devices that says, hey, go see if I’m online, yup, I’m online, I can write all that code, but to be perfectly honest, it’s a pain in the butt and I don’t want to do it. What MQTT does is passes messages from a publisher to a subscriber, which is any system that receives a message and I can have dozens or hundredses or thousands of subscribers, and MQTT will deal with all of the connections and I don’t have to think about it. Now, how do I know as a subscriber if I’m getting the message I want, every message has a topic, basically like a title. Think of it as nested folders. In this case I’m going to call them dog/alerts, because I’m not that creative. So again we’re going to start with a microcontroller, this one is called the M5 atom matrix, also runs Arduino and this time I’m just connecting it up to an audio device. It’s just an MP3 player, like I literally put an SD card in there with a sound file on it, and the microcontroller can say play, pause, next track, etc. And then we’re going to have Grafana and MQTT to send the data to that device. I’ll grab it and turn them on. Speaker works.

1. My device — this is — I was trying to make something that indicated that it was connecting to wifi, but it didn’t work well, OK, so hard to see but there’s a little green smiley face. How does this work? Well, we’ve got MQTT like I said, and so the two things that I really need are getting a message and doing something with it. So if I get a message, all I’m going to do is look at that topic and say, is this the dog alerts topic. If it isn’t, just leave it alone. If it is. I could put a JSON parser on here, and pull all the relevant data out, but I’m lazy and one of the fields is called status. If that field exists in there and you see the word “firing” I’m going to assume it’s firing. If not, it’s not firing, we’re not going to get an alert. So five lines of C to be able to do that.

     And then the board itself is going to run in an infinite loop. It’s going to say, hey, do I have a message from MQTT.

     So there’s not a lot to this stuff, but you can start to see how it builds up and becomes cool as you connect them together, right?

     So, OK, everybody’s tired of demos at this point, me too. It’s the last physical demo, I promise. So I built this dashboard before, I hope I’m not leaking water, I don’t know why it’s going down slightly. May need a cleanup on aisle 3. Let’s treat an alert rule on this data. Oh, no, I have to save my dashboard first. I usually give things descriptive names, but new alert rule.

1. So we’ve got this metric set up. Alert connection, if it is below — let me see what it is now. It’s currently 503 grams so we’ll pretend that, I don’t know, 490 grams is when it’s empty. Grafana by default stores everything in folders for organization so we’ll just call this dog alerts.

     I have to create this and call it dog alerts. One of the things that we want to do is shrink the window of how frequently this is going to be checked to as short as possible. Normally you know, if you’re tracking health of a system or something, you probably don’t want to get alerts every ten seconds. Like, five minutes is probably plenty. I want this to happen quick. So we’re going to shrink everything down as fast as it can go, and the final part is I’m going to send this to MQTT. So I’ve preconfigured MQTT in Grafana so that it knows how to send a message, but that should be it.

     So let’s look at our alerts — actually let me go back to the dashboard. What I’m going to do is — I’m going to have a dog come in and drink some of this water.

     And this is a dog, everyone recognizes it as a dog, right? It’s very thirsty.

     All right, we’re now at 471, and this will evaluate every ten seconds, so worst case you have to listen to me try to stall it for ten seconds. (Woof!  I’m thirsty!)

     And that’s my dog —

    [applause]

     And if I come back and fill it up, again we should see in ten seconds after you listen to the dog a few more times, we’ll see when it was triggered and we see when it was cleared. Again we’ve got some cool things we could use. This could actually legitimately be useful as it is. I want to do the data analytics part. I’m going to have to rush through part of this, I’m definitely talking longer than I should. We’re going to apply data analytics to make this more useful. So the one place I’ll cheat and not run something locally is I am using Grafana cloud which is just a hosted service for Grafana, Prometheus, etc. because I wanted to simulate data for this for several weeks and I didn’t want to leave my laptop open for that long so I vibe coded a Python script that pretended to be a dog going in and out of that dog door and so what it does is sleeps all night, wakes up and goes out. So I’ve got three weeks of simulated data. And again, I can do simple things with this, and I can say look at when the dog is going in and out and see if there are any interesting patterns here.

     Maybe it’s more useful to see how it’s changing day over day, but it gets way more interesting when I start to add other data in here. So I could look at the door use versus temperature, for example, is my dog going out more often or less often when it’s cold outside? But the real power of this is once you amass a decent amount of data, you can start doing cool things like machine learning on this. So I took this three weeks of data and I ran machine learning against it, just a Facebook prophet algorithm and of course, in proper conference fashion I played with this two days ago and broke any demo data, so fortunately I had a screenshot of what I built. So — nope, not share. Slideshow. So what this is, the orange line here is the actual data, the data that was recorded by the dog going in and out. And there’s kind of a fuzzy blue area around it, that’s what the machine learning algorithm predicts the data will be in. And you can see it does a pretty good job. When the dog is asleep, it says I don’t expect there to be change in this data, when the dog is awake, it says I do expect there to be a change in this data.

     But then I changed I the simulation to say I want the dog to go in and out more frequently. I can set up an alert based on this but that alert is probably much more important this might indicate that all of a sudden my dog is going out to pee a whole lot more than normal and I want to be aware of that to be able to take them to the vet and get checked out. It’s also possible that there’s just squirrels in the yard and the dog is going crazy. That’s much more likely with my dog, but the data that you collect like this can start to become useful and start to show trends that can actually be actionable. And that’s it. If anybody wants to look at the code or try these things out, it’s on my GitHub, it’s pinned there and otherwise I know I’m between everybody and beer. So thank you very much.

    [applause].