Despite challenges including a new location and a nasty Nor’easter that put a crimp on travel, Spark Summit East managed to draw more than 1,500 attendees to its February 7-9 run at the John B. Hynes Convention Center in Boston. It was the latest testament to growing adoption of Apache Spark, and the event underscored promising developments in areas including machine learning, deep learning and streaming applications.

The Summit had outgrown last year’s east coast home at the New York Hilton, but the contrast between those cramped quarters and the cavernous Hynes made comparison difficult. As I wrote of last year’s event, the audience was technical, and if anything, this year’s agenda seemed more how-to than visionary. There were fewer keynotes from big enterprise adopters and more from vendors.

Mataei Zaharia of Databricks recapped Spark progress last year, highlighting growing adoption and performance improvements in areas including streaming data analysis.

The Summit saw plenty of mainstream talks on SQL and machine learning best practices as well as more niche topics, such “Spark for Scalable Metagenomics Analysis” and “Analysis Andromeda Galaxy Data Using Spark.” Standout big-picture keynotes included the following:

Mataei Zaharia, the founder of Spark and chief technology officer at Databricks, gave an overview of recent progress and coming developments in the open source project. The centerpiece of Zaharia’s talk concerned maturing support for continuous applications requiring simultaneous analysis of both historical and streaming, real-time information. One of the many use cases is fraud analysis, where you need to continuously compare the latest, streaming information with historical patterns in order to detect abnormal activity and reject possibly fraudulent transactions in real time.

Spark already addressed fast batch analytics, but support for streaming was previously limited to micro-batch (meaning up to seconds of latency) until last February’s Spark 2.0 release. Zaharia said even more progress was made with December’s Spark 2.1 release with advances on Structured Streaming, a new, high-level API that addresses both batch and stream querying. Viacom, an early beta customer, is using Structured Streaming to analyze viewership of cable channels including MTV and Comedy Central in real time while iPass is using it to continuously monitor WiFi network performance and security.

Alexis Roos, a senior engineering manager at Salesforce, detailed the role of Spark in powering the machine learning, natural language processing and deep learning behind emerging Salesforce Einstein capabilities. Addressing the future of artificial intelligence on Spark, Ziya Ma, a VP of Big Data Technologies at Intel, offered a keynote on “Accelerating Machine Learning and Deep Learning at Scale with Apache Spark.” James Kobielus of IBM does a good job of recapping Deep Learning progress on Spark in this blog.

Ion Stoica, executive chairman of Databricks, picked up where Zaharia left off on streaming, detailing the efforts of UC Berkeley’s RISELab, the successor of AMPLab, to advance real-time analytics. Stoica shared benchmark performance data showing advances promised by Apache Drizzle, a new streaming execution engine for Spark, in comparison with Spark without Drizzle and streaming-oriented rival Apache Flink.

Stoica stressed the time- and cost-saving advantages of using a single API, the same execution engine and the same query optimizations to address both streaming and batch workloads. In a conversation after his keynote, Stoica told me Drizzle will likely debut in Databricks’ cloud-based Spark environment within a matter of weeks and he predicted that it will show up in Apache Spark software as soon as the third quarter of this year.

The Apache Drizzle execution engine being developed by RISELabs promises better streaming query performance as compared to today’s Spark or Apache Flink.

MyPOV of Spark Progress

Databricks is still measuring Spark success in terms of number of contributors and number of Spark Meetup participants (the latter count is 300,000-plus, according to Zaharia), but to my mind, it’s time to start measuring success by mainstream enterprise adoption. That’s why I was a bit disappointed that the Summit’s list of presenters in the CapitalOne, Comcast, Verizon and Walmart Labs mold was far shorter than the list of vendors and Internet giants like Facebook and Netflix presenting.

Databricks says it now has somewhere north of 500 organizations using its hosted Spark Service, but I suspect the bulk of mainstream Spark adoption is now being driven by the likes of Amazon (first and foremost) as well as IBM, Google, Microsoft and others now offering cloud-based Spark services. A key appeal of these sources of Spark is the availability of infrastructure and developer services as well as broader analytical capabilities beyond Spark. Meanwhile, as recently as last summer I heard Cloudera executives assert that the company’s software distribution was behind more Spark adoption than that of any other vendor.

In a though-provoking keynote on “Virtualizing Analytics,” Arsalan Tavakoli, Databricks’ VP of customer engagement, dismissed Hadoop-based data lakes as a “second-generation” solution challenged by disparate and complex tools and access limited to big data developer types. But Tavakoli also acknowledged that Spark is only “part of the answer” to delivering a “new paradigm” that decouples compute and storage, provides uniform data management and security, unifies analytics and supports broad collaboration among many users.

Indeed, it was telling when Zaharia noted that 95% of Spark users employ SQL in addition to whatever else they’re doing with the project. That tells me that Spark SQL is important, but it also tells me that as appealing as Spark’s broad analytical capabilities and in-memory performance may be, it’s still just part of the total analytics picture. Developers, data scientists and data engineers that use Spark are also using non-Spark options ranging from the prosaic, like databases and database services and Hive, to the cutting edge, such as emerging GPU- and high-performance-computing-based options.

As influential, widely adopted, widely supported and widely available as Spark may now be, organizations have a wide range of cost, latency, ease-of-development, ease-of-use and technology maturity considerations that don’t always point to Spark. At least one presentation at Spark Summit cautioned attendees not to think of Spark Streaming, for example, as a panacea for next-generation continuous applications.

Spark is today where Hadoop was in 2010, as measured by age, but I would argue that it’s progressing more quickly and promises wider hands-on use by developers and data scientists than that earlier disruptive platform.

What’s the business case for Apache Spark? After the opening (general-session) day of Spark Summit East 2016 in New York, I was thinking that Spark promoter and Summit host Databricks needed to do a better job of telling that story. Then on the final day, executives from Capital One, Synchronoss and eBay offered compelling, business-use-case focused keynotes that knocked it out of the park.

Chris D’Agostino, VP of Technology at Capital One, a top-ten bank and cutting-edge user of data and analytics, outlined the bank’s use of Spark to generate tabular and graph representations of data at high scale. These views of data are used in low-latency fraud-detection, anti-money-laundering and know-your-customer analyses.

Suren Nathan, senior director of Big Data Platforms and Analytics Frameworks at Synchronoss, a cloud services company, outlined four eras of data pipeline maturation. It started with the “V1” traditional ETL/data warehouse era and then moved on to the V2 appliance-assisted era (think Exadata and Netezza, with comparatively high costs yet lacking support for multi-structured data). In the V3 early Hadoop era we suffered with slow batch processing in MapReduce. In today’s V4 era, you get scale on Hadoop but also superior performance with support for both batch and streaming workloads thanks to Spark.

Seshu Adunuthula of eBay described how “the other e-commerce giant” is making use of Spark within its evolved enterprise data platform. eBay’s move into selling more new items at fixed prices (as opposed to used items through auctions) has led it to take a catalog-oriented, rather than search-oriented, approach. That has necessitated a more structured yet dynamic data modelling approach. eBay remains one of the largest users of Teradata in the world, but Adunuthula said the company is moving many workloads into Hadoop. Spark was added to support fast batch analysis and streaming-data services. “Once we introduced Spark, we saw rapid adoption, and now we’re seeing more and more use cases as adoption grows,” Adunuthula concluded.

The opening-day keynotes were offered by vendor execs who had mixed success in painting a big picture. IBM’s Anjul Bhambhri, VP, Big Data, came close, explaining that “the beauty of Spark is that all components [Spark SQL, Spark Streaming, Spark R, MLLib, etc.] work together in a seamless way. You don’t need half a dozen products; you need just one, foundational platform.”

Matai Zaharia, CTO of Databricks, stuck to detailing Apache Spark accomplishments in 2015 and reviewing new capabilities coming in Spark 2.0, set for release in late April or early May. Highlights of the coming release include whole-stage code generation aimed at improving Spark performance and throughput by as much as 10X. Spark Streaming improvements in 2.0 will support the common scenario of mixed streaming and batch workloads, as when you want to track the state of a stream while also firing off SQL queries. Finally, Spark’s DataFrame and Dataset APIs will be merged in 2.0, simplifying matters for developers by presenting fewer libraries and concepts to worry about.

Ali Ghodsi, co-founder and recently promoted to CEO of Databricks, confined his remarks to Databricks’ own services, highlighting the appeal of using Spark through the company’s cloud-based commercial offering. He also introduced a free (sandbox-scale) cloud-based Databricks Community Edition. This led to a demo by Databrick’s Michael Ambrust, who touted Databricks’ cloud offerings as a great way to avoid the pain of deploying Spark software.

MyPOV: With its cloud-based platform, Databricks competes, in a way, against the open-source software it helps to create and promote. Commercial companies behind open source software more typically support on-premises software deployments. Is it a coincidence that Spark customers and third parties including Cloudera and IBM seem to be carrying the load of making the case for enterprise Spark adoption? Plenty of Databricks execs offered technical presentations on Apache Spark software capabilities and roadmaps, but they were preaching to the choir. At Spark Summit East, Databricks missed a chance to be more of a champion of Apache Spark in the enterprise.

Thinking back, it took a partnership with O’Reilly to turn Hadoop World into Strata + Hadoop World, an event with a broader audience and a higher-level purpose. I’ll grant that much of what Databricks does to promote and contribute to a healthy Apache Spark ecosystem goes on behind the scenes. And you can’t argue with the project’s success. But it seems to me that Spark is ready for a bigger stage.

PS: Interesting tech seen at Spark Summit included h2o.ai, which supports distributed machine learning on Hadoop or Spark, and Data Robot, which automatically generates predictive models (using leading open-source algorithms in R, Python and Spark) and tests, validates and selects the most accurate ones, speeding predictive and easing the data science talent shortage.  Also interesting was SnappyData, a company recently spun out of Pivotal that has ported the open source Geode (formerly GemFire) in-memory, distributed database to run on Spark. It offers SQL querying in a persistent store that is part of/runs on Spark rather than requiring separate infrastructure.

The most talked about topic at Spark Summit NOT on the agenda or announced at the event was Apache Arrow, a new project which promises an in-memory, columnar data layer that can be shared by multiple open source projects, eliminating redundant infrastructure, cost and copies of data while enabling fast analytics across many workloads. The project launched with support from a whopping 13 open source projects, including Cassandra, Drill, Hadoop, HBase, Impala, Parquet, Pandas and Spark.

Why is Apache Spark, an open-source project that only reached its 1.0 release a little more than a year ago, getting so much attention?

IBM, for one, announced a big commitment to the platform at this week’s Spark Summit in San Francisco. And as IBM execs told analysts at the company’s new Spark Technology Center here, it’s an all-in bet to integrate nearly everything in the analytics portfolio with Spark. Other tech vendors betting on Spark range from Amazon to Zoomdata, even as real-world deployments number in the hundreds and are mostly experiments and proof-of-concept projects.

Spark offers unified access to data, in-memory performance and plentiful processing and analysis options.

Describing Spark as “an operating system for analytics,” IBM execs cited Spark strengths including:

• Abstracted data management. Spark lets data scientists focus on the analysis, not data movement, as data pipelines can access data where it lies, whether that’s in Hadoop, in a database, or on Spark’s own cluster.
• Rich data-processing functionality. The Spark Core provides a flexible data-processing platform supporting distributed task management, scheduling, and basic I/O functionality as well as transformations such as map, filter, reduce and join.
• In-memory performance. The Spark Core delivers up to 100 times faster performance than Hadoop MapReduce. In iterative machine learning and other predictive analyses, you can squeeze in that many more processing cycles against all of the data, not just samples of data.
• Plentiful analytics options. IBM execs didn’t dwell on this strength, but Spark offers multiple analytic libraries than run on top of the core, including MLLib (machine learning), Spark SQL, Graph X, Spark Streaming and, released last week, Spark R. IBM plans to run its own software on top of the platform, including SPSS, IBM Streams, and (soon to be open sourced) SystemML, all of which are being ported to run on Spark.

In contrast to IBM, analytics rival SAS views Spark as a competitor. For more than three years, SAS has been working on its own big-data-capable, in-memory analytics platform, the SAS LASR Analytics Server, which runs SAS Visual Analytics (VA) and SAS Visual Statistics (VS) as well as SAS analytics libraries. The SAS LASR Analytic Server can be deployed on a single server, on a dedicated cluster, or on top of Hadoop (see my just-published report, “The Era of Self-Service Analytics Emerges: Inside SAS Visual Analytics and Visual Statistics”).

When I asked SAS about its views on Spark earlier this week, product manager Mike Ames offered the following (lukewarm) statement: “While Spark is currently an immature technology, it shows promise with rapid adoption as a result of its data processing capabilities. SAS and Spark are very capable of coexisting, with products such as SAS Data Loader for Hadoop, which can push transform logic to Spark.”

SAS is right about Spark’s immaturity. I’ve talked to practitioners and integrators who acknowledge that the technology is still green. Like a lot of 1.X open-source software projects, Spark is still buggy and it doesn’t have all the table-stakes systems-management, security and high-availability features that many enterprises would insist upon before running mission-critical workloads.

That’s not to say that Spark is incapable of running production workloads reliably or at scale. Hundreds of companies are doing just that, but they tend to be pioneers with an appetite for innovation and strong engineering teams that are willing to fix bugs and develop best practices where none exist.

Plenty of early adopters presented at Spark Summit, including Airbnb, Baidu, Edmunds.com, NASA, NBC Universal, Netflix, Shopify, Toyota Motor Sales, and Under Armour. Keynoter James Peng, principal architect at Baidu, the Google of China, described that company’s 1,000-plus-node, petabyte-scale Spark deployment, which is delivering 50 times faster performance than the conventional MapReduce processing it previously relied upon. Baidu is also pioneering the use of Spark SQL and the Tachyon caching layer.

MyPOV on Spark

Yes, it’s early days for Spark, but there’s good reason why IBM described it as “potentially the most significant open-source project of the next decade.” SAS acknowledged Spark’s data-processing capabilities, but that’s just the starting point. Even IBM’s characterization of Spark as “an operating system for analytics” seems like a left-handed compliment.

With all those libraries on top of the Spark Core – machine learning, SQL, graph, streaming and R — Databricks and the Spark community are trying to build out an all-purpose analytics platform capable of supporting many forms of analysis and blended analyses. By blending machine learning and streaming, for example, you could create a real-time risk-management app. What’s more, Spark supports development in Scala, Java, Phython and R, which is another reason the community is growing so quickly.

At Spark Summit, Amazon Web Services announced a free Spark service running on Amazon Elastic Map Reduce, and IBM announced plans for Spark services on BlueMix (currently in private beta) and SoftLayer. These cloud services will open the floodgates to developers, and IBM’s contributions will surely help to harden the Spark Core for enterprise adoption.

In short, it’s hard to see any open-source project matching Spark on depth and breadth of analysis and development flexibility (despite a prominent tout of Apache Flink at last week’s Hadoop Summit). And that’s why you’re hearing so much about Spark.