If you’ve read this week’s release notes, you’ll have seen “Added ability to configure or override connection details at the Data Service level in BigQuery and Databricks Data Planes.” So what do we mean by that?
Let’s first brush up on Ascend product terminology.
- Data Service – is a group of Data Flows. At this level, users configure team members, connections to other systems, and event notifications.
- Data Flow – is an individual data pipeline. Data Flows include the ingestion of raw data, transformation via SQL and python, and sharing of finished data products.
- Data Plane – is the data cloud where the data pipeline workload runs, like Databricks, BigQuery, and Snowflake.
Before these new features, Ascend sent pipeline workloads to a single Data Plane that was set up in the initial configuration by the Ascend SRE team.
Then we added Data Plane configuration for Snowflake.
Last fall, we quietly added the ability for you to connect individual Data Services to different Snowflake databases. This database then hosts the workloads for the Data Flows in that Data Service. As a result, you can distribute workloads across more infrastructure, and align your Ascend teams with their access to different Snowflake resources.
Now we add Data Plane configuration for Databricks and BigQuery.
Starting this week, you can also direct workloads for each individual Data Service to Databricks or BigQuery. This means that Ascend becomes your single pane of glass to automate data pipelines in all three leading data clouds simultaneously: Snowflake, Databricks, and BigQuery.
Ascend will generate and automatically orchestrate the pipeline workloads in each of these data clouds, and consolidate the metadata from each step, including resource utilization with which to quantify the costs of your data products.
In short, with these new capabilities, organizations can distribute each data pipeline to the data cloud best suited for its use case, while staying entirely within the powerful data pipeline automation framework of Ascend.
