Data Engineers Can Create Continuously Optimized, Apache Spark-Based Pipelines with 85% Less Code

PALO ALTO, Calif. – July 17, 2019 – Ascend today announced its launch from stealth and introduced the world’s first Autonomous Dataflow Service, the only solution allowing data engineering teams to quickly build, scale, and operate continuously optimized, Apache Spark-based pipelines. Powered by Ascend’s Dataflow Control Plane, users combine declarative configurations and automation to manage cloud infrastructure, optimize pipelines, and eliminate maintenance across the entire data lifecycle. Customers of all sizes and industries are using the Ascend Service to leapfrog their data architectures and build the data foundation to fuel their digital transformations. 

Pipelines are an essential part of any modern data architecture. With ever-changing infrastructure technologies, larger data volumes, and a growing variety of data formats, today’s pipeline development and operations require far too much coding and a disproportionate amount of time for maintenance and optimizations. Changes in the data itself, infrastructure updates, or logic tweaks often produce brittle and problematic pipelines. For many data engineering teams, this means combing through code and logs and constantly tuning parameters just to keep things running. As these systems scale and interoperate over time, the challenges grow exponentially, eclipsing what can be manually constructed and managed. 

“Over the past decade, a number of trends have emerged as engineering teams shift from monolithic to modular, manual to automated, and imperative to declarative. We have seen firsthand how these new approaches result in more stable foundations for critical components within businesses,” said Sean Knapp, Founder and CEO of Ascend. “The Autonomous Dataflow Service we’re announcing today mirrors that trend for data pipeline development. Now data engineering teams of any size can build, scale, and operate complex pipelines with 85% less code. This fundamentally changes what these teams are able to deliver, freeing them from the monotony of pipeline plumbing and arming them with the capabilities required to drive the next wave of data innovations.”

“Our research shows that data integration is the biggest obstacle for those wanting to make any degree of data-centric digital transformation,” said Mike Leone, Senior Analyst, Enterprise Strategy Group (ESG). “There’s been great innovation in Business Intelligence, Data Processing, Data Warehousing, and other applications, but data pipelines remain far too complex. It feels as though the orchestration layer has remained largely ignored. At ESG, we appreciate Ascend’s work to rectify this challenge with its Autonomous Dataflow Service and Dataflow Control Plane. By automatically optimizing and intelligently orchestrating data pipelines, Ascend enables data engineers to focus on the data itself, truly accelerating digital transformation.”

The Ascend Autonomous Dataflow Service automates development and maintenance of large-scale data pipelines, enabling pipeline creation with 85% less code and with zero maintenance burden. The Ascend Service brings together four foundational elements that enable this new standard in data development:

  • Fully Managed, Portable Cloud Service: Ascend deploys and runs seamlessly across Microsoft Azure, Amazon Web Services, and Google Cloud Platform. Customers have the option of fully managed SaaS or private deployments, and all development in Ascend is fully portable across clouds. 
  • Elastic Data Fabric: Ascend runs an auto-scaled microservices architecture built on Kubernetes and Apache Spark for on-demand processing and auto-orchestration to handle near-limitless scale.
  • Multi-Language: Ascend supports SQL and Python (Lambda functions and PySpark), with more language support coming soon. Languages and frameworks can be used interchangeably to build sophisticated pipelines with full lineage and dependency management. All development is operationalized by default, with no need to recode logic or manually schedule runs. 
  • Dataflow Control Plane: The powerhouse of Ascend keeps all development continuously running through the combination of declarative configurations and automation to manage the underlying cloud infrastructure, optimize pipelines, and eliminate maintenance across the entire data lifecycle. 

“The availability of Ascend on Microsoft Azure is a huge win for users already running on Azure, as well as for all those looking to migrate to the cloud,” said Shaloo Garg, Managing Director, Microsoft for Startups. “The speed and ease with which users can now build sophisticated, Azure-based data pipelines that feed directly to other applications and services, including the Azure SQL Data Warehouse, is a game-changer. I can’t wait to see the innovation this Microsoft-Ascend collaboration unlocks for Azure users.”

“I’ve seen firsthand at Twitter and Square the challenges of high-volume pipelines and the burden these deficiencies impose on data engineering teams, limiting what they can accomplish. If we want to continue to cope with the rapidly increasing volumes of data being generated, as an industry we must remove these limitations,” said Steven Parkes, Chief Technology Officer, Ascend. “We’re enabling data engineers to be vastly more productive in the development, deployment, and ongoing operation of data pipelines. Much as compilers have virtually eliminated assembly language programming, we have introduced a new development layer that drastically reduces the complexity of building and operating data pipelines, allowing these teams to build reliable and performant solutions, regardless of changes to the infrastructure, data formats, or targeted applications.”