Data teams are consistently challenged by a rapidly evolving technological landscape and escalating demands. To navigate this environment, staying attuned isn’t just beneficial — it’s a necessity. This means not only understanding where you stand, but also recognizing how the evolving patterns in the broader industry might align with or diverge from your own data programs.
Over the past four years, we have conducted an industry-wide DataAware Pulse Survey to capture the current state of data teams. In this article, we detail the six main patterns from the 2023 results. The objective is to provide data teams with the essential insights to refine their strategies and maintain a competitive edge.
For those seeking an exhaustive analysis, the full 2023 DataAware Pulse Survey is available for download.
Download: 2023 DataAware Pulse Survey
Pattern 1: Data Teams Are at Capacity
Teams are consistently operating at their limits:
- Persistent Capacity Strain: For four consecutive years, an overwhelming 95% of data teams have reported operating at or above their work capacity. While 57% indicate they’re either somewhat above or significantly over their capacity.
- Disproportionate Burden on Data Engineers: Delving deeper, it’s evident that data engineers are feeling this strain most acutely. They’re three times more likely to report feeling significantly overburdened compared to their counterparts in data architecture and data analytics.
- Hiring Isn’t Catching Up: The gap between organizational demands and team growth is widening, especially for data engineers. They’re 27% more likely than others to report a stark disparity between the growth of their team and the escalating company demands. Simply put, the demand for data is outpacing team growth at an alarming rate.
Pause for a moment and evaluate the workload dynamics within your data team. Do these findings resonate with your current situation? Are you sensing a similar strain when it comes to capacity and workload? Especially for the data engineers on your team, is there a palpable feeling of being consistently overburdened? Compare the growth rate of your team with the increasing company demands. It’s essential to discern if your team mirrors this pattern, as recognizing it is the first step toward strategic adjustments.
Pattern 2: Data Teams Are Stuck in Reactive Mode
Data teams, rather than driving innovation forward, find themselves constantly playing catch-up:
- Maintenance Over Innovation: Instead of pioneering new solutions, the majority of data engineers, alarmingly, allocate 50% or more of their time to merely maintaining existing programs and infrastructure.
- Reactive Metrics: Data teams are stuck in a reactive mode. They primarily gauge their impact by the number of errors fixed — with 42% saying this metric is their top performance indicator.
- The Engineer’s Dilemma: Taking a closer look, engineers are the most impacted by this trend. 46% cite “number of errors fixed” as the most common metric used to measure their success, underscoring a prevalent culture of squashing bugs rather than creating value.
Reflect on the mode in which your data team currently operates. Are you finding yourselves entrenched in a reactive loop, constantly addressing errors rather than pioneering new solutions? How does your team’s focus on maintenance versus innovation compare with the insights shared? Consider whether your metrics and day-to-day operations truly align with driving innovation forward or if there’s an imperative need for a shift toward proactivity.
Pattern 3: Productivity Is Stagnant
Despite advances in data technology, there’s a pressing concern in the data community about stagnant productivity:
- Tool Sprawl: Data teams now use five tools on average in their data stack and 68% plan to maintain or increase this number in the next year. With so many tools to jump between just to do basic tasks, teams are falling further behind. Individual contributors were 2.1x more likely than their managers to say they want to cut tools out of their stack.
- Rising Inefficiencies: Surprisingly, respondents are twice as likely to highlight a decline in their productivity this year when compared to previous years.
- Engineers at the Forefront: Among the roles, data engineers feel the dip most acutely, reporting a 2.2x drop in their productivity year-over-year.
- Tools or Hindrances?: Data engineers, in particular, identify two core issues stymieing their productivity: constraints within the tools they use and a glaring absence of effective automation.
Are you noticing a similar trend of stagnation, or perhaps even a decline, despite leveraging new technologies? Consider the experiences of your data engineers in particular. Are they confronting issues with the tools they use or lamenting a lack of impactful automation? Assessing whether your team’s experiences align with these identified patterns can be a crucial step in pinpointing areas for improvement and intervention.
Pattern 4: Data Automation as the Answer
Data professionals are seeking solutions to their productivity issues, and automation consistently emerges as a sought-after remedy:
- Surge in Automation Interest: 91% of survey participants are either currently employing, or have intentions of leveraging data automation technologies in the next 12 months.
- Momentum Grows, Yet Challenges Persist: There’s a marked 20% uptick in respondents who are ‘very likely’ to integrate data automation technologies within the coming year. Remarkably, a 110% increase was observed in the current use of these technologies compared to 2022. However, a disparity emerges when considering past year’s projections: 40% professed a strong inclination to adopt automation, yet a mere 5% followed through. This stark difference suggests that, despite the genuine aspiration, there may be a gap in resources or expertise without the aid of supporting automation platforms.
- Top-Down Approach and Disparity: Data team leadership is 74% more inclined to prioritize and fund automation projects. This automation imperative is felt acutely by larger enterprises and teams, with such entities being 1.3 times more likely to experience the setbacks from an automation deficit.
- Engineers and Architects Champion Automation: Data engineers and architects are twice as likely to pivot to automation for augmenting team capabilities, as compared to their data analyst counterparts.
Take a moment to evaluate your organization’s journey toward automation. Are you among the majority expressing keen interest, or have you been able to fully integrate automation solutions? Given the evident surge in interest and its perceived value, how closely does your team’s progression match the general trend?
It’s particularly worthwhile to probe into the reasons if there’s a lag in the actual implementation despite intentions. Does your data team, especially the engineers and architects, resonate with this call for automation?
Pattern 5: Managers Bridge the Gap for Data Alignment
A significant chasm exists between data professionals on the ground and their executive counterparts, evidenced by variances in perception, objectives, and strategic approaches. This underscores the pivotal role managers and directors must play in bridging the gap and fostering alignment.
- Disparate Perspectives on KPIs: A striking difference emerges when considering top impact measures. Executives gravitate towards tangible outputs, being 3x more inclined to highlight presentations created and 1.5x more drawn to new dashboards as the primary impact measure for their role. Contrastingly, individual contributors prioritize error resolutions and ticket closures.
- Perceptions of Time Efficiency: Executive frustration is palpable, with many feeling basic data tasks are simply taking too long. This sentiment is twice as pronounced among them than the individual contributors on their teams. Further exacerbating their concerns, executives invest an average of 17.8 hours weekly looking for data to do their jobs — 5.6 hours more than individual contributors.
- Strategic Dissonance: Disagreements on major initiatives surface. For instance, individual contributors are five times more likely to dismiss plans to adopt data mesh/fabric, while executives are enthusiastic about its imminent implementation. Skepticism around data mesh benefits is also prevalent among team leads, being 2.8 times more widespread than among executives.
- Tooling Preferences: A tug-of-war ensues over data stack preferences. Individual contributors are up to 2.1x more likely to favor cutting tools. Team leads favor consolidating to platforms while retaining all functions. On the flip side, executives are up to 1.6x more likely to favor adding more tools to the stack.
- AI’s Role: The data team sees high potential for Generative AI. The top use cases include test automation, code generation, and documentation. However, individual contributors are 200% less likely to report believing that Generative AI can increase their impact on the business.
As you reflect on your organization’s structure and communication dynamics, consider the alignment between your data professionals and executive leaders. Are the differences in perspectives on KPIs, tool preferences, and strategic initiatives evident within your team?
If you’re in a managerial or directorial role, assess how effectively you’re mediating these distinct viewpoints. If you’re an individual contributor, are the discrepancies in priorities palpable? For executives, consider the insights you gain from your middle management.
Recognizing where your organization mirrors or contrasts with this pattern can offer insights into potential areas of focus to enhance understanding and collaboration across your data teams.
Pattern 6: Expenses Keep Growing
The increasing reliance on cloud infrastructure is undeniable, but with it comes the challenge of escalating costs:
- Proliferation of Data Pipelines: A staggering 89% of respondents anticipate a growth in the number of data pipelines over the next year, underscoring the continual expansion of data infrastructure.
- Cost-Saving Strategies Emerge: As expenses rise, teams are strategizing to curtail them. A prominent 48% of teams are leaning towards optimizing data pipelines as their top tactic to mitigate cloud computing costs, seeking efficiency in operations.
- Consolidation Over Volume: Meanwhile, 38% are setting their sights on consolidating data sources. By reducing data volumes, they aim to achieve a dual goal of streamlining operations and saving costs.
- Divergence in Strategies: Interestingly, while data team executives, at a rate 1.9 times higher than others, feel confident about consolidating data sources, the broader technical team expresses a preference for pipeline optimizations as their cost-control measure.
Pause for a moment to assess your organization’s data strategy in the cloud. Are you experiencing a similar surge in data pipelines and the associated costs? How does your current approach to cost management align with the findings above? As expenses rise, it’s crucial to contemplate if your team is adequately equipped with the strategies to optimize operations and manage costs effectively.
In Closing: The Road Ahead for Data Engineering
Recognizing prevailing patterns is essential to navigating the complexities of the industry. From teams consistently operating at capacity, to the challenges of stagnant productivity, the push for automation solutions, and the indispensable role of management in fostering alignment, each of these patterns provides a unique lens through which we can evaluate and refine our approaches to data.
As you reflect on these findings, consider how they resonate with your own experiences and challenges. Are these patterns reflective of your data team’s realities? Or do you see divergences in your organization’s journey? We encourage you to delve deep, assess, and recalibrate where necessary — and join the conversation by sharing your thoughts in the comments below!
