jQuery

jQuery has been in existence for years. It’s nothing new, and it’s certainly not exciting. However, it’s the industry standard for DOM manipulation, and it’s under active development. Not long ago they gave some previews of jQuery 2.0. It’s nice to see it active and in constant improvement (unlike poor jQuery UI).

Require.js

As JavaScript applications get larger and developers need a large organized codebase, require.js steps in and fills that gap nicely. Require.js will help you to organize a large JavaScript project, and comes with a great tool to minimize and optimize your entire application, including CSS. James Burke has been tirelessly adding new features and doing great things to promote AMD patterns that are being adopted in many well-known libraries.

Backbone.js

Backbone provides a great MVC framework for helping you organize your large JavaScript applications. It’s even better when bundled with jQuery and Require.js. Recently I used it on a project to connect directly to a CouchDB instance to provide a live-updating data-driven application that makes a really nice and relatively simple stack.

D3.js

Data intensive applications need a lot of charting and graphing so that user’s can get better insight into their data. D3 provides a strong foundation that other libraries can be built on top of. Unfortunately, there haven’t been many quality libraries built for it, but it still provides a great feature set for building custom data visualizations.

Meteor.js

I’ve had my eye on this one for a long time. It provides a framework for quickly building a full client-server application that allows live-collaboration. I still don’t feel like its ready for prime-time yet, but it’s a very exciting and ambitious project. Once they tighten down the security, and allow it to be scale-able, I think it will turn into a big player.

Socket.io

This library provides great socket functionality between a Node.js server and a JavaScript application. It’s got great backward-compatibility support for older browsers and is simple to use (though difficult to master).

Conclusion

Here at Nodeable we use all of those libraries except for Meteor.js. They have been very helpful to organize a large code-base for our ajax driven web application as well as our mobile web application.

That’s it for my quick list of JavaScript libraries that are the foundations of JavaScript development. We use all of those except for Meteor.js at Nodeable, and they serve us pretty well.You should ignore all other JS libraries (except for js.js … Joking!!) I know there are probably some that I am missing, and I would love to learn what libraries you use for the foundations of your applications. Post the libraries you use in the comments, and teach me something I don’t know!

StreamReduce is Nodeable’s real-time aggregation and processing engine that enhances and analyzes data streams as a cloud-based complement to batch processing via Apache Hadoop and Amazon Web Services Elastic Map Reduce.

“Nodeable’s cloud-based complement to Hadoop means users can process multiple data streams in real-time, enhancing data and making MapReduce jobs more efficient,” said Mike Olson, CEO of Cloudera. “As the demand for Hadoop and big data insights continues to grow, Nodeable’s pre-processing engine helps large enterprises get more from Hadoop faster and its real-time cloud delivery brings the benefits of big data analytics to companies without the dedicated IT staff and resources of the Global 2000.”

Market researchers estimate that big data is a market expected to grow to $16.9 billion by 2015, with the Hadoop software sector alone exploding at a compound annual growth rate of more than 60 percent.

But data-driven companies need real-time useful information – a capability not available in batch-oriented tools like Hadoop. With StreamReduce from Nodeable, users can complement their Hadoop deployments by applying continuous computation to aggregate custom, structured and unstructured data streams, revealing key real-time insights, anomalies and summaries that help make data-driven businesses more efficient and agile.

StreamReduce is also designed to be easy-to-use for customers who want sophisticated data analytics without the high expense of dedicated big data scientists on staff and the enormous capital costs of building large Hadoop clusters.

“For almost a year the engineering team at Nodeable has worked closely with more than 400 beta users who’ve told us that a real-time analytics complement to Hadoop is a top priority,” said Dave Rosenberg, Founder and CEO of Nodeable. “Batch workflows are too slow for turning data into useful, actionable information. StreamReduce solves that problem with a simple cloud-based solution.”

With StreamReduce, customers can gain invaluable insights into their big data before Hadoop processing – giving users the benefits of immediate analysis of their data streams. Nodeable beta users are taking advantage of the StreamReduce for a variety of use cases including:

• Log and clickstream analysis;
• Anomaly detection in Amazon Web Services EC2 instances
• Security and fraud detection;
• Mobile and geo-location measurement;
• Pinpointed advertising and marketing.

Nodeable’s StreamReduce service is available now. Pricing starts at $99/month.

About Nodeable
Nodeable provides real-time data stream processing for big data applications. Nodeable’s StreamReduce engine aggregates, normalizes and analyzes data before it reaches Hadoop, significantly accelerating batch processing under MapReduce. Customers are up and running in under 15 minutes and scale large workloads easily with Nodeable’s cloud-based deployment. Founded in 2011, the company is backed by True Ventures.

The value in analytics scales with the mountains of data we amass.  It’s akin to something I wrote about years ago for CNET: 21st Century businesses thrive by driving abundance and then selling minimization of complexity inherent in that abundance.  It’s what Red Hat does with open source, what Google does with search, and what Facebook does with social.  It’s also what companies like Nodeable do with all the data your marketing/IT operations/sales/etc. systems throw off.

There are a few interesting stories buried in the growth of analytics, but perhaps the biggest is Hadoop.  While the overall analytics market grew by 14 percent in 2011, IDC has the Hadoop market growing by 60 percent each year through 2016.  Admittedly, that’s off a small base, but at that pace the Hadoop ecosystem, which Forrester already sizes at $1 billion per year, will be very, very big.

Big money for Big Data.

And for such a comparative pittance.  Hadoop, as I’ve argued before, has democratized data.  Big Data analytics used to be the province of expensive data warehousing systems, complete with proprietary software, expensive, proprietary hardware, and a smiling salesperson with their palm out, waiting for you to mortgage your house.  Not anymore.  Hadoop is open source and is run on commodity hardware.  It’s a game even cash-strapped organizations can play.

It’s not perfect.  Hadoop can’t do real-time, for example, which is why Nodeable buttresses its fantastic batch-processing capabilities with the real-time computation heroics of Storm.  Storm, of course, is also open source.

Which is what is so fascinating in this Big Data gold rush: it’s being driven by free and open-source software.  No wonder the market is growing in such dramatic fashion: it’s not being gated by vendors anymore.  That’s good news for all of us…including vendors.

After all, given that 32 percent of Karmasphere’s survey participants are running Hadoop clusters smaller than 2 terabytes, and 55 percent are running clusters of 10 terabytes or less, the “Big” in “Big Data” really isn’t.  Not yet, anyway.  That will likely change as enterprises move from toe-dipping to diving into Hadoop and Big Data in a big way, but for now the workloads aren’t huge, and real-time tools like Storm might be ideal for managing them.

These workloads are also not necessarily being run behind the firewall.  While both Cloudera and Hortonworks are booming due to enterprises keeping their Hadoop jobs running primarily in their data centers, Amazon is already managing in excess of one million Hadoop clusters with its Elastic MapReduce service.  This is perhaps not surprising given that the majority of Big Data users tend to be business users, not hard-core IT people, according to Karmasphere’s survey.  These people are apparently very comfortable having their data processed in the cloud.

Interestingly, while there are numerous great applications for Hadoop, the majority seem to be using it for marketing-related functions:

All of which brings me back to the point I made earlier this week: some data are best analyzed in real time, not batch.  For many things, you’ll actually want both: a real-time view into what’s happening with your website, HR systems, etc., as well as a deeper, Hadoop-based analysis that is done in batch, after the fact.

Real-time analytics tools like Nodeable (based on the open-source Storm project) are not a replacement for Hadoop.  They’re complements.

Given that so much of the data being analyzed with Hadoop are still relatively small and marketing-focused, not to mention being analyzed in Amazon’s cloud, I’d argue that more of today’s data, not less, should be run through real-time analytics systems, and particularly hosted systems.  After all, while it’s useful to know how aspects of your online retail site are working hours or days or months after the fact, you actually want the “next click” to reflect real-time analysis, as Yahoo CTO Raymie Strata argues:

With the paths that go through Hadoop [at Yahoo!], the latency is about fifteen minutes.…[I]t will never be true real-time.  It will never be what we call “next click,” where I click and by the time the page loads, the semantic implication of my decision is reflected in the page.

Again, this isn’t to denigrate the importance of Hadoop.  At all.  It’s simply to suggest that for many applications, relying on batch-oriented Hadoop alone is an incomplete strategy.  Real-time is required for many applications, particularly those where Hadoop is being used today, and that real-time capability is delivered through Storm-based Nodeable or other real-time analytics systems.

It’s not Storm or Hadoop.  It’s Storm and Hadoop.

Like Storm, the project Twitter inherited when it acquired BackType in 2011.

At Nodeable we use Storm to surface real-time insights from system data, whether that system is GitHub or AWS or Salesforce.com or Twitter or an infinite number of data sources.  We operate under the assumption that users need real-time insights (Storm) and timely information (Hadoop).  It would be impossible to crunch all of a business’ data in real-time, and frankly not necessarily all that useful.  So Hadoop’s batch approach to data mining is great for a wide variety of jobs.

But not when you need to know something right now.  As Metamarkets CEO Michael Driscoll noted at a recent Churchill Club event, “Hadoop is not like having a conversation with your data.  Instead it’s like having a pen pal that you write from time to time.”  For things like clickstream analysis, IT early-warning systems, security and fraud detection, etc., that’s not fast enough.  So Storm is a great complement.

There are alternatives to Storm, of course.  Hstreaming, Streambase, and Yahoo S4 each offer real-time complements for Hadoop, though S4 is arguably the most like Storm.  We opted for Storm for many of the reasons Dan Lynn highlights in a presentation he gave at Gluecon. It’s open source.  It works really well.  And it gives our engineering team a great deal of flexibility.

But whether you use Storm or something else, you likely do need to figure out how to complement Hadoop with real-time.

Nodeable can help.  Nodeable provides real-time data streaming for Hadoop, which means that we provide front-end processing — summaries, counts, anomalies, status, trends — before data hit Hadoop and turns those data into useful information.  In other words, we not only give you real-time insight into your systems, but also normalize and enhance your data to make your Hadoop batch processing much more efficient.

Please sign up for beta access and tell us what you think.

For example, consider just a few of the trends sweeping the industry, particularly those that are finding their way into job specifications.  HTML5, NoSQL (e.g., MongoDB), cloud computing, Hadoop, etc.  Spend a few minutes on TechCrunch and you’ll start to feel that YOU MUST BUY INTO THESE RIGHT NOW!

And, in some cases, you should.

But not always.  If you’re building a cloud application, for example, you’re likely going to want the scale-out capabilities of a NoSQL database like MongoDB.  But Oracle has a good point arguing that you wouldn’t want to build a checking application for a bank with NoSQL technology.   SQL has its place, and NoSQL has its place.  It just happens to be dominant for new school applications, which may not be the kind you’re developing.

Or consider Hadoop, which has dramatically lowered the (economic) bar to data mining/analytics.  Hadoop is fantastic technology, but it’s batch-oriented.  If you need real-time analytics, you’re likely going to want to couple Storm with Hadoop, as we do here at Nodeable.  Or maybe you should embrace Red Hat’s JBoss Data Grid 6, which “as an in-memory, key-value store…is much more optimized to handle the operations that Hadoop simply can’t: transactions like the kind found in e-commerce and financial trading systems.”

Does this mean you dump Hadoop and swap it for Data Grid or Storm?  Of course not.  But it does mean that developers need to look beyond the hype to determine what is the best tool for a particular job.

The same thing holds true in mobile, where HTML5 promised to be the end to mobile’s fragmentation problem.  Instead, it turns out native apps dominate the smartphone space, while content-friendly tablets are much more likely to be friendly to HTML5.

The list goes on.  Some swear by Node.js, but it’s best for server-side app development, and not a panacea, according to Nodeable CEO Dave Rosenberg.  Cloud?  You probably shouldn’t choose private or public, but both, declares Citrix’s Peder Ulander.  Etc. etc.

Each of these technologies is trending because it solves real needs in novel, useful ways.  But this doesn’t mean they’re right for you.  Of course, one of the great things about cloud and open source and other megatrends like these is that they tend to skew open, such that you can try before you buy.  As a vendor, this is a huge boon, as I’d much rather customers buy what they need and are happy, rather than getting duped into buying hype that doesn’t fit their needs.

That’s the old world: buy into the hype and regret is later.  The new world lets you regret your decision right away.

Dave summarized the panel discussion on CNET, and highlights Cloudera CEO Mike Olson’s call to arms for Hadoop-based applications.  It’s something Olson has said before, including here on this blog, but it was particularly poignant against the backdrop of a deep, engaging discussion about Hadoop’s pros (powerful, open source) and cons (batch-oriented, complex, somewhat inefficient).

And it’s why I think Nodeable is a sign of the times.

We’re an application that depends upon Hadoop.  But we’re also a technology that improves Hadoop by front-ending it with Storm.  Hadoop is powerful but limited to batch-oriented processing of data.  Storm actually crunches data in real-time, in the stream.  The combination of the two is potent, and something that we only discovered while building out our application to ingest systems data and extrapolate insights via in-stream data analytics.

In the near future the back-end data processing via Storm and Hadoop will be managed behind the scenes by cloud applications, as Workday co-CEO Aneel Bhusri tweeted from the Churchill Club event.  For now, companies like Nodeable are helping to bridge the divide between complex infrastructure and simplified applications.

But the real question is whether an organization is culturally ready to embrace the cloud. If so, the necessary infrastructure follows and, importantly, it’s not necessarily always going to look like Amazon. As Mark Thiele writes:

For a legacy IT organization to adopt cloud solutions without significant organizational realignment and improved business participation, the benefits would largely be wasted. It’s akin to thinking you can put a modern 500-horse power engine in a 1970’s economy car and get all the same performance and protection characteristics you would enjoy in a 2012 model year luxury sedan.

In fact, the introduction of cloud without organizational improvements would likely increase enterprise risk and potentially cost. The real opportunity of a cloud operating model comes from the alignment of technical solutions, people, and process.

In other words, cloud isn’t something for IT to hatch in seclusion from the business side of the enterprise. Cloud is, in an ideal world, truly a function of what the business needs.

We’re starting to see this play out in the rise of DevOps, but ultimately the integration of the enterprise across functions will go even deeper. Cloud computing should demolish the walls IT has put up to protect its turf (and sanity). IT will need to work hand-in-hand with the business to build out the right cloud tools for a particular job, whether public, private, or a hybrid of the two.

Andy Jassy, senior vice president of Amazon Web Services (AWS), dismisses the notion of private clouds altogether:

If you look deep into what [private cloud vendors] are offering, you will see that it’s basically an internal data center that is virtualized and has some management tools. Organizations that have private cloud systems will have missed out on all the advantages and benefits of going into the cloud.

But this is easily said by the vendor best-positioned to capitalize on public cloud computing. Amazon doesn’t need to worry about a potpourri of hardware and software choices, built up over years. Bias argues that this is one of the great strengths of AWS and, indeed, of all big web companies like Google and Facebook that have been able to build their clouds from the ground up.

Within the average enterprise, however, years or decades of legacy hardware and software choices must be balanced against the new imperative of the cloud. And so they consider the cloud for resource bursting or carve out a private cloud for new applications. Will they run as efficiently as Amazon? Almost certainly not. But that’s not really the goal, is it?

IT can play an essential role in helping the business to rationalize its existing assets and complement them with cloud resources. I suspect one area that can help bridge the gap between IT and the business is better tools to express what is happening in cloud systems, and what this means for the business.

At Nodeable, we’re trying to build monitoring tools that go beyond mere reporting of what’s happening to express why things are happening, and how these cloud systems impact the business. One of the key reasons for our Twitter-like interface is that we want the system to be approachable to non-technical users. Because, frankly, cloud systems shouldn’t be isolated to IT folks.

The cloud has the potential to democratize IT, and bring the business into the IT conversation. Part of this cultural shift can be complemented by the right tools, tools that don’t drown users in arcane minutiae but instead present insights into how things are working, and why. This is the recipe for cloud success, and it’s something we as an industry are just now starting to figure out.

Until this week, I hadn’t realized just how serious IBM was about DevOps.  But on Tuesday I was fortunate to spend some time talking with IBM’s Bill Higgins.  As we talked, I started searching the web for more information on IBM’s involvement, and found it…everywhere.  Yes, there were the obligatory conference talks, but there was also smart discussion about how to pull off DevOps within traditional enterprise IT.

And a whole IBM blog devoted to the topic of DevOps.  Imagine that!

Yes, it just started.  But it’s great to see such a trusted enterprise brand do more than merely slap the DevOps label on a tired, old product line, hoping that customers won’t notice.  I’m sure IBM is doing that, too, but not Higgins and his team.  It sounds like there’s a very real, concerted effort to embrace DevOps within IBM and within its customer base.

I don’t know where this leaves the counterculture.  But I think it means DevOps is more than some passing fad.  And that maybe, just maybe, IBM is shrewdly embracing the counterculture again, just as it did with Linux and open source years ago.  That bet paid off big time for Big Blue.  I imagine its bet on DevOps will do the same.

UPDATE: Donnie Berkholz, part of Redmonk’s awesome analyst team, attended IBM’s Pulse conference earlier this year, and praised IBM for being out in front of its stodgy peers in terms of DevOps and other important trends:

IBM’s people really get it. They understand trends that are happening at the frontlines of tech today in startups and in open-source development. IBM is way out in front on enabling DevOps in big enterprises, and the teams working on DevOps inside Tivoli as well as Rational (which builds tools for developers) are outstanding. A lot of my experience with enterprises is that they’re slow-moving and often lagging trends by years, to the point where it’s nearly laughable, but in this case IBM is definitely a front-runner.

In a new research report (executive summary available for free), Gartner articulates this shift away from IT-as-king to IT-as-facilitator, and how the cloud fosters the trend:

In 2010, organizations were compelled to consume IT services from external cloud providers to achieve their business, budget, and IT goals. But organizations realize that external cloud computing is not a panacea. They still need internal data centers to house critical applications and data. However, the use of external cloud providers has conditioned organizations to expect IT resources — whether internal or external — that are offered in an on-demand, self-service manner. Therefore, IT organizations are forced to offer IT services by using the same consumption model or otherwise risk extinction.

Poor IT.  First it had to deal with the rise of open source, and now cloud.  Both trends have forced IT to loosen its stranglehold on the software, hardware, and services used with the enterprise.  All of the arguments that failed against open source are also failing to stem the tide of cloud computing.

And rightly so.

But none of this means that IT is dead.  It just means, as Gartner points out, that IT’s role needs to change.  For example, DevOps is a very real phenomenon, but I suspect few organizations will have the know-how or brazen courage to take the Netflix route and embrace DevOps completely.  The trick is to give developers more flexibility without imposing on them the burden of setting up and managing all infrastructure themselves.  Yes, some will want precisely this.  But most just want greater influence over the tools they use.

They’re not going to get this from OpenView or any of the legacy tools from legacy vendors.  They’re just not going to blend private and public cloud resources, host their code on GitHub, and be bothered with clunky old management tools.  It’s not going to happen.

So IT needs to redefine its new role.  And it needs to get new tools for doing so.  Or it’s going to evaporate into obsolescence.