RedMonk Radio #45: Really Simple Integration with SnapLogic

In this episode of RedMonk Radio, Stephen O’Grady and I talk with SnapLogic‘s Chris Marino and John Bennett.

Really Simple Integration

We start out with an overview of what SnapLogic is and does. As Chris says, “SnapLogic is an open source data integration framework that allows people to very quickly build data integrations that replaces the hand-coded, point to point integrations they’re using today.” The idea that SnapLogic bills itself under is “Really Simple Integration,” which drives the choice to be a highly web-influenced architecture. Put simply, SnapLogic lets you pull data from various sources, transform them, and then provide them as a simple URL with RESTful features.

We discuss SnapLogic’s choice of python as the base language which I found interesting when I first heard about it. Chris comments that the language is keyed in nicely to enabling data integrations and transformations.

Using SnapLogic

I then ask Chris to walk us through a scenario of using SnapLogic to combine two data streams together – to make a mash-up, as it were. Building on the idea that SnapLogic is very web architecture centric, Chris explains how you would define “resources,” for example, reading out of a database, a file, or another source of data. If needed, there are transformation resources if needed. The two are combined together into pipe-lines that eventually spit out in the RESTful interface. As I note and then we discuss each of these components are then available to be re-used and re-combined if that makes sense.

Though SnapLogic Chris and John don’t cover it in depth here – this being an audio only podcast – the Flex-based SnapLogic IDE is worth checking out a screencast of SnapLogic in action.

Getting Started with SnapLogic

Stephen then asks how developers can get started with SnapLogic. As Chris notes, SnapLogic is GPLv2’ed so you’re free to go download it and start using it. He gives us another scenario for pulling data from a database and then using the SnapLogic tool to assign the results of the query to a URL, something like http://theurl/cote/database_query. Again, the idea being that your end-goal is get a URL that returns the desired data in formats like XML, CSV, plain-text, or other representations.

John adds to this the idea of building up a library of these resources to re-use with different people who’re interested in consuming the data. This segways into a discussion of the meta-data features that SnapLogic has. They expose meta-data about services in a web page, describing the data service, listing related services, and then allowing you to cross-launch into the web-based design tool.

Enterprise Features

I then ask about SnapLogic’s enterprise-grade features. Ostensibly, you can compare what SnapLogic does to Yahoo! Pipes, but, being targeted at businesses and on-premise use, there are, no doubt, additional management and security options. Chris notes that part of the benefit of using a web-based architecture is that SnapLogic can use the same security and access management as other web-based applications.

After discussing some cross-organizational scenarios they’ve seen people using SnapLogic for, I ask Chris and John to compare the way SnapLogic does data integration with the traditional methods. We then get into a mini-discussion of FTP, EDI, CSV, and other exciting data formats and data extract scheduling.

SnapLogic and WaveMaker

Pulling back to more modern times, Chris talks about working with WaveMaker at Web 2.0 Expo last week – also, some brief video here. As I’m always painting out the need for RIA layer people to get a back-end – that is content and data – this topic is interesting.

Cloud Computing

Stephen then brings up the topic of cloud computing, asking how developers who’re doing cloud work could use SnapLogic. This gets to an interesting discussion about using SnapLogic not only for data integration, but also as a sort of URL-accessible “stateless compute resource”: that is, submitting asynchronous or synchronous jobs to a SnapLogic layer for processing.

John adds in that the web-centric nature of SnapLogic makes it a good fit for the cross-platform nature of most cloud architectures: since everything is accessible by a URL, there aren’t exactly the same cross-platform problems and re-combing two different parts of a system together is, hopefully, easier to do with URLs than direct connections over traditional APIs.

SnapLogic 2.0

Rounding up the general overview of what SnapLogic does, I ask Chris and John to go over the SnapLogic 2.0 release. John starts by saying they “completely re-engineered the product” with an eye towards performance and delivering on the really simple integration idea, for example, with the meta-data directories. Also, while being in python, they’ve now added a Java library and are targeted to add more languages. They’ve also improved the designer tool. Finally, in addition to the community edition, they’ve introduced different commercial packaging options: developer and enterprise editions. Chris adds that they’ve added several different representations that data can find itself in as well as new sources of data, such as screen-scraping web pages.

When then talk about how SnapLogic fits in as a sort of data-access middleware layer for ISVs who’d want to bundle SnapLogic into their own product. The idea here being that SnapLogic can sort out the data access issues, allowing the ISV to focus on the use of that data, for example for dashboards, business intelligence, and other ways of looking through and using data.

Since Chris was at Web 2.0 Expo last week, before wrapping up, he gives us a brief overview of his thoughts. He was suprised at the amount of enterprise people there, though he heard some carping that there was perhaps too much enterprise. As I say, it sounds like there was plenty of “2.0” there, but a mix of “Web” and “Enterprise” in front of it.

Disclaimer: SnapLogic is a client and sponsored this podcast.

Technorati Tags: snaplogic, eai, rest