The “Connected Data Platforms” that Hortonworks introduced on March 1 are its well-known Hortonworks Data Platform (HDP) Hadoop distribution and its Hortonworks DataFlow (HDF) platform aimed at collecting, curating and routing real-time data from any source to any destination. HDP and HDF can be used independently, but here’s how they fit together to become a cohesive platform for managing and analyzing streaming and historical data.

Interest in streaming data analysis has been growing steadily in recent years, but the emergence of Internet of Things (IoT) opportunities has interest soaring. The thing is, streaming-data use cases such as connected-cars, smart oil fields, smart utilities and precision medicine often require analysis of historical data, which brings context to the real-time insights. That’s why HDF and HDP need to be connected.

Inside Hortonworks Connected Data Platforms

This week Hortonworks introduced HDP’s 2.4 release. Notable upgrades include support for and bundling of Apache Spark 1.6 software as well as improved system management and remote optimization capabilities through Apache Ambari 2.2 and SmartSense 2.2. Ambari, the open source management software, gained an Express Upgrade feature that lets you quickly stop jobs, update software and restart the cluster and running jobs all within one hour, even on large systems. SmartSense is a “phone home” capability that relays system-performance parameters to Hortonworks, which can diagnose problems and offer more than 250 recommendations on optimizing system performance and availability.

The biggest development with HDP 2.4 is a new distribution strategy with two separate release cadences. Core Apache Hadoop components including HDFS, MapReduce and YARN as well as Apache Zookeeper will be updated annually, in line with other members of the ODPi consortium. Hortonworks is expediting other, newer capabilities through new “Extended Services” releases, which will be offered as quickly as they can be made available. One example of an Extended Service is support for Spark 1.6. Other candidates for this release approach will include Hive, HBase, Ambari “and more,” says Hortonworks.

MyPOV on HDP 2.4: I like this two-pronged strategy with the stable, slower moving core complemented throughout the year by extended services. Hortonworks has lagged behind Cloudera in the past in adding certain new capabilities that customers have been anxious to use. This is a good approach to fast tracking capabilities that are in demand (although they presumably can’t require changes to Hadoop core components). The approach also simplifies matters for other distributors of ODPi-based distributions.

Hortonworks DataFlow 1.2

HDF is Hortonwork’s streaming data platform based on Apache NiFi and adapted from last year’s Onyara acquisition. Upgrades with the move HDF 1.2, which will be available later this month, include the integration of Apache Kafka and Apache Storm streaming analytics engines. The release also gains support for Kerberos for centralized authentication across applications. On the near-term roadmap is support for Spark Streaming, which should be available by early summer, according to Hortonworks.

MyPOV on HDF: There’s much to like in Hortonworks DataFlow, including a drag-and-drop approach for developing the routing, transformation and mediation within dataflows. It also offers built-in data-security and data-provenance capabilities. One exec described it as “a FedEx for streaming data,” providing the digital equivalent of a logistics system for routing streaming data and tracking sources and changes to digital information along the way. The ecosystem seems strong, with support for more than 130 processors for systems including Kafka, Couchbase, Microsoft Azure Event Hub and Splunk.

How HDP and HDF are Connected

Hortonworks wants to be a multi-product company, so it has stressed that HDP and HDF will be sold and can be used independently. HDF can route data to (and draw from) other Hadoop distributions, databases such as Cassandra and cloud-based sources, such as Amazon S3.

When use cases span data in motion and data-at-rest, HDP and HDF have commonalities that makes them easier to use together. For example, both HDP and HDF share more than 70 data processors and both use Ambari for system deployment and management. What’s more, Hortonworks is promising that SmartSense, and the Ranger and Atlas security and governance projects will also support both platforms.

MyPOV on Connected Platforms: The need for the combination of streaming and historical data analysis is popping up in many quarters. It was touted as a benefit of Spark Streaming 2.0 at the recent Spark Summit East event, and MapR also has a strategy to address both forms of data in one platform.

Hype around streaming data opportunities is nothing new. More than a decade ago, complex event processing systems were touted as “ready to go mainstream.” At long last, I think we’re finally seeing signs that streaming data analysis is emerging. The mobile, social, cloud and big data trends set the stage and maybe, just maybe, the promise of IoT possibilities is pushing it over the top.

PS: Hortonworks also spotlighted two promising Spark related developments this week. First, it’s shipping a preview of Apache Zeppelin with HDP 2.4, providing a coding-free UI for visualization and a notebook-style approach to working on Spark. This is a usability improvement and democratization tool that Spark sorely needs. Second, in a partnership with HP Enterprise Labs, Hortonworks will bring to open source an optimized shuffle engine for Spark that HP Enterprise says will offer 5X to 15X performance improvements as well as optimized use of memory. This tech doesn’t have project status yet, let alone acceptance from the Spark community, but Hortonworks says it will ship the software with HDP later this year.

Related:
Spark Summit East Report: Enterprise Appeal Grows
Strata + Hadoop World Report: Spark, Real-Time In the Spotlight

It’s going to be a big week for big data as Strata + Hodoop World 2015 kicks off this week at New York’s Jacob Javits Center. Ahead of the show there were two big announcements from Cloudera on Monday as it introduced RecordService and Kudu.

RecordService is an all-new security layer for Hadoop designed to ensure secure access to data and analysis engines that run on top of Hadoop, including MapReduce, Apache Spark, and Cloudera Impala. Where Cloudera’s previously available Sentry security component supported the definition of access-control permissions, RecordService complements Sentry by enforcing access down to the column and row level. It also supports dynamic data masking, according to Cloudera, brining uniform, granular access control to any framework or analysis engine that’s plugged into the RecordService API.

RecordService and Kudu are being released as beta software downloads this week, and Cloudera says it intends to donate both open-source projects to the Apache Software Foundation (ASF). General availability will depend on feedback and revisions required, according to Cloudera, and it did not specify when it will contribute the technologies to ASF.

MyPOV on RecordService and Kudu

Despite the open-source billing, there’s little doubt that both of these technologies will be promoted predominantly by Cloudera and used mostly by its customers. Cloudera rival Hortonworks previously introduced the open source Ranger project for data-access authorization, control and auditing within Hadoop. Which security project is likely see wider support? Cloudera mentioned MapR, which currently supports Impala, as showing interest in RecordService. Hortonworks, meanwhile, is a member of the Open Data Platform initiative (ODPi), which counts IBM, Pivotal, SAS and Teradata as founding members. But that group has yet to name Ranger as one of the components of the ODPi Common Core, which currently includes HDFS, YARN, MapReduce and Apache Ambari.

The good news is that Cloudera customers can now look forward to granular data-access controls across data and analysis engines, something that’s crucial to secure use of the data in Hadoop in production environments. As for disunity on Hadoop, get used to Cloudera and Hortonworks going separate ways, much like Ford and Chevy or RedHat and Ubuntu.

The bottom line for Cloudera is that it’s the leading Hadoop distributor and a deep-pocketed partner of Intel. It can be confident that its technology will be see adoption even there are questions about vendor lock in and dependence on commercial Cloudera Manager software.

Real-time and streaming applications are another front on which Cloudera and Hortonworks are headed in different directions. Kudu looks like a promising complement to the current Hadoop stack, and it’s aimed at simplifying the complicated LAMBDA architectures that have emerged. Matt Brandwein, Cloudera’s director of product marketing, tells me Kudu will work with Apache tools including Kafka and Spark Streaming and will enable developers to remain within the Hadoop stack when building streaming applications.

That sounds like a jab at Hortonworks DataFlow, which is based on its recent Onyara acquisition and Apache NiFi. While both Kudu and DataFlow are aimed at streaming use cases, they are in no way comparable. While Kudu is a storage engine, DataFlow is a much more comprehensive platform aimed at managing streaming data flows from edge sensors and devices to core systems, ensuring tracking of data provenance, and ensure the secure delivery and flow of data across multiple systems despite bandwidth disruptions or offline conditions. It’s also separate from Hadoop and has yet to support YARN.

If Kudu and DataFlow have anything in common, it’s that it’s early days for both projects. A lot of development and integration work lies ahead before either technology will see significant real-world adoption. The good news for practitioners is that the Hadoop ecosystem is maturing quickly to support a range of big data and low-latency applications with all the security features needed for enterprise production deployments.

We’re looking for innovators in eight categories, but as the chair of judges in the “Data to Decisions” area, I’m most looking forward to reading about innovators who are using data to make informed business decisions or to create all-new business models.

All finalists of the SuperNova Awards are awarded one complimentary ticket to Constellation’s Connected Enterprise executive innovation summit.* The application takes only about 15 minutes to complete. Gather all the information about your disruptive technology project, and submit your application now!

Apply Here: https://www.constellationr.com/node/3137/apply

Revised timeline

• August 7, 2015 August 14, 2015 last day for submissions.
• September 2, 2015 finalists announced and invited to Connected Enterprise.
• September 14, 2015 voting opens to the public
• October 2, 2015 polls close
• November 4, 2015 Winners announced, SuperNova Awards Gala Dinner at Connected Enterprise
• We’re searching for the rebels, catalysts, and innovators that are shaping the future of technology! If you led the implementation of a disruptive technology for your organization, we want to hear your story!

Complete the SuperNova Award application in 3 steps

1. Create a Constellation account
2. Gather the information required to complete your SuperNova Award application. Preview the application and download an application checklist here.
3. Submit your application by August 14, 2015.

Apply Here: https://www.constellationr.com/node/3137/apply

*limit one per finalist. Transportation and lodging not included. 

Why is Apache Spark, an open-source project that only reached its 1.0 release a little more than a year ago, getting so much attention?

IBM, for one, announced a big commitment to the platform at this week’s Spark Summit in San Francisco. And as IBM execs told analysts at the company’s new Spark Technology Center here, it’s an all-in bet to integrate nearly everything in the analytics portfolio with Spark. Other tech vendors betting on Spark range from Amazon to Zoomdata, even as real-world deployments number in the hundreds and are mostly experiments and proof-of-concept projects.

Describing Spark as “an operating system for analytics,” IBM execs cited Spark strengths including:

• Abstracted data management. Spark lets data scientists focus on the analysis, not data movement, as data pipelines can access data where it lies, whether that’s in Hadoop, in a database, or on Spark’s own cluster.
• Rich data-processing functionality. The Spark Core provides a flexible data-processing platform supporting distributed task management, scheduling, and basic I/O functionality as well as transformations such as map, filter, reduce and join.
• In-memory performance. The Spark Core delivers up to 100 times faster performance than Hadoop MapReduce. In iterative machine learning and other predictive analyses, you can squeeze in that many more processing cycles against all of the data, not just samples of data.
• Plentiful analytics options. IBM execs didn’t dwell on this strength, but Spark offers multiple analytic libraries than run on top of the core, including MLLib (machine learning), Spark SQL, Graph X, Spark Streaming and, released last week, Spark R. IBM plans to run its own software on top of the platform, including SPSS, IBM Streams, and (soon to be open sourced) SystemML, all of which are being ported to run on Spark.

In contrast to IBM, analytics rival SAS views Spark as a competitor. For more than three years, SAS has been working on its own big-data-capable, in-memory analytics platform, the SAS LASR Analytics Server, which runs SAS Visual Analytics (VA) and SAS Visual Statistics (VS) as well as SAS analytics libraries. The SAS LASR Analytic Server can be deployed on a single server, on a dedicated cluster, or on top of Hadoop (see my just-published report, “The Era of Self-Service Analytics Emerges: Inside SAS Visual Analytics and Visual Statistics”).

When I asked SAS about its views on Spark earlier this week, product manager Mike Ames offered the following (lukewarm) statement: “While Spark is currently an immature technology, it shows promise with rapid adoption as a result of its data processing capabilities. SAS and Spark are very capable of coexisting, with products such as SAS Data Loader for Hadoop, which can push transform logic to Spark.”

SAS is right about Spark’s immaturity. I’ve talked to practitioners and integrators who acknowledge that the technology is still green. Like a lot of 1.X open-source software projects, Spark is still buggy and it doesn’t have all the table-stakes systems-management, security and high-availability features that many enterprises would insist upon before running mission-critical workloads.

That’s not to say that Spark is incapable of running production workloads reliably or at scale. Hundreds of companies are doing just that, but they tend to be pioneers with an appetite for innovation and strong engineering teams that are willing to fix bugs and develop best practices where none exist.

Plenty of early adopters presented at Spark Summit, including Airbnb, Baidu, Edmunds.com, NASA, NBC Universal, Netflix, Shopify, Toyota Motor Sales, and Under Armour. Keynoter James Peng, principal architect at Baidu, the Google of China, described that company’s 1,000-plus-node, petabyte-scale Spark deployment, which is delivering 50 times faster performance than the conventional MapReduce processing it previously relied upon. Baidu is also pioneering the use of Spark SQL and the Tachyon caching layer.

MyPOV on Spark

Yes, it’s early days for Spark, but there’s good reason why IBM described it as “potentially the most significant open-source project of the next decade.” SAS acknowledged Spark’s data-processing capabilities, but that’s just the starting point. Even IBM’s characterization of Spark as “an operating system for analytics” seems like a left-handed compliment.

With all those libraries on top of the Spark Core – machine learning, SQL, graph, streaming and R — Databricks and the Spark community are trying to build out an all-purpose analytics platform capable of supporting many forms of analysis and blended analyses. By blending machine learning and streaming, for example, you could create a real-time risk-management app. What’s more, Spark supports development in Scala, Java, Phython and R, which is another reason the community is growing so quickly.

At Spark Summit, Amazon Web Services announced a free Spark service running on Amazon Elastic Map Reduce, and IBM announced plans for Spark services on BlueMix (currently in private beta) and SoftLayer. These cloud services will open the floodgates to developers, and IBM’s contributions will surely help to harden the Spark Core for enterprise adoption.

In short, it’s hard to see any open-source project matching Spark on depth and breadth of analysis and development flexibility (despite a prominent tout of Apache Flink at last week’s Hadoop Summit). And that’s why you’re hearing so much about Spark.

Hadoop is enterprise viable and growing quickly. That’s the message Hortonworks CEO Rob Bearden communicated from the start of his opening keynote at the June 9-11 Hadoop Summit in San Jose, CA. Billing Hadoop as “one, central platform for batch, interactive, and real-time applications,” he focused mainly on the big-picture possibilities for innovation and “game-changing new business models.” But he also acknowledged “incredible progress” in Hadoop basics such as operational performance, security and data governance over the past 18 months.

Progress certainly has been made, but Hadoop is best described today as entering its awkward teenage years. Its (analytical) voice is breaking and it sometimes throws tantrums when faced with adult (enterprise workload) expectations. Talking to customers here (running on Hortonworks and Cloudera), I heard tales of cluster crashes and cranky, uncooperative operational behavior.

Hortonworks introduced the 2.3 release of the Hortonworks Data Platform (HDP) at Hadoop Summit. But executives didn’t spend much keynote time talking about the improvements to Ambari systems management, project Ranger security and access controls, or Apache Atlas metadata management. The list of upgrades also includes broader SQL coverage, visualization of SQL queries, and easier installation and configuration of HDFS, Yarn, Hive and Hbase.

To discuss such basics would only underscore the relative immaturity of Hadoop. Instead Bearden and others focused on support for cutting-edge options like Apache Spark and streaming analysis with Kafka and Storm. All three were featured in on-stage demos.

Thankfully, Hortonworks also didn’t beat its chest in public about the Open Data Platform (ODP) initiative. Announced in February, this is the partnership led by Hortonworks, IBM, Pivotal and Infosys and since joined by Telstra, BMC, DataTorrent, Syncsort, Unifi, zData, and Zettaset. The goal is to get behind a stable core of Apache Hadoop components that all members use to promote interoperability and to speed adoption of Hadoop.

ODP members insist that they have invited all Hadoop distributors to join the group, but the starting-point was agreeing to interoperate with HDFS, YARN, MapReduce and Ambari. Ambari is the sticking point, as Cloudera and MapR have their own management software, so they’ve declined to join ODP. The choice of Ambari gets back to Hortonwork’s “100% open source” ethos, but this sort of intra-Hadoop-family drama also does not inspire confidence in Hadoop. In my view, Hortonworks and other ODP partners would do well to pursue this initiative from a purely technical perspective rather than using it as some sort of branding seal of approval.

As the proud parent of a maturing Hadoop distribution, Hortonworks was smart to put the public emphasis at Hadoop Summit on enterprise adoption. More than 75 Hortonworks customers presented at the event. Luminaries included Progressive insurance, oilfield services firm Schlumberger, online auto buying site TrueCar, telco giant Verizon, and web-measurement firm Webtrends. Progressive is analyzing all that Progressive Snapshot driving data IoT style, helping good drivers to save money on car insurance. TrueCar studies everything from the color of a car to localized buying trends to accurately price vehicles. In a real-time scenario, Webtrends can spot abandoned shopping carts and lost shoppers on e-commerce sites within seconds so retailers can respond and try to save the sale before customers leave the site.

MyPOV on Hortonwork’s Progress

Hortonworks is nothing if not consistent. It sticks to its mission of delivering 100% open source Hadoop software. In some cases it gets to features and functions after chief rival Cloudera has developed something first. Ranger, for example, provides access controls that Cloudera previously introducing in Sentry. And Cloudbreak, the cloud-deployment tool/service introduced in HDP 2.3, follows in the footsteps of Cloudera Director, introduced last fall.

First does not always mean best. Ranger, for example, provides more granular access control (and auditing) than does Sentry across all the components of Hadoop. Given the impressive and growing customer list, it’s clear plenty of companies like Hortonworks’ approach and are confident in the roadmap.

As for the some of the latest features from Hortonworks and competitors including MapR, let’s hope routine system-admin and security features won’t be what ultimately differentiates Hadoop distributions. These are the sorts of features that enterprise customers just expect to be there. When Hadoop is truly mature, the boundaries between the menagerie of projects within Hadoop will disappear. The complexity that still confronts Hadoop administrators and day-to-day users will diminish. And the competition will center on ease of data management, ease of workload management, breadth of analytical capabilities, and emerging, next-generation big data applications.