There’s no other way to put it: companies are already having a very sobered 2023.
Companies are tightening their budgets — even more than they already did in late 2022. Their board and investors are putting immense pressure on them to do more with less. To focus on impact, outcomes, and efficiency.
Below, I draw on my prior knowledge, discussions with customers, and conversations with other leaders to outline six key data engineering trends to be aware of (and capitalize on) in 2023.
1. Efficiency and productivity will be paramount in 2023
Teams will get extremely cost-conscious if they haven’t already. For the data engineering community, that translates to creating a more optimal foundation for data engineering, one that minimizes infrastructure spend and maximizes production of valuable data products.
Engineering leads should be prepared to produce significantly higher impact per person. Any rote activities should be as automated as possible to speed up workflows. Effective communication with other teams will become even more important in order to prioritize projects based on outsized impact on the business.
Establishing and sustaining that environment will be hard. It will require highly skilled and highly motivated engineers that drive their teams forward with focus and pragmatism.
2. The data engineer role will specialize
Over time, “software engineer” became too generic a term. Today, the average software engineering careers pages have frontend, backend, infrastructure, mobile, and even product-related roles.
This year, we’re going to see a similar fragmentation of the “data engineering” title. There will be new terms to distinguish between data engineers working higher up the stack and those deep in the technical depths. We’re already seeing surging demand for analytics engineers, very similar to the rise of frontend engineers in years prior, who are highly skilled and laser-focused on the last mile of data products.
Filling these specialized roles may seem counterintuitive when most companies are doing layoffs. But hiring the right people for the right roles can lead to what I call “impact engineering” — reducing the number of people and dependencies required to get a job done. Finding diamonds in the rough will get you to an impact engineering state faster while helping you retain talent by giving them work they are good at and enjoy.
3. Developer platforms will overtake loosely connected tools
Fragmentation cannot survive in a cost-conscious climate. Developers tire of maintaining dozens of integrations over time, and seek out more consolidated, if not all-in-one solutions. Such solutions will become increasingly appealing to both individuals and companies as they strive to streamline workflows and increase productivity.
Over the past few years, we’ve also seen the pendulum swing from highly technical solutions to low or even no-code tools. I predict that developer platforms that can provide flexible interfaces and experiences to meet the needs of a wide range of engineers will excel in the coming year.
4. The battle for the Data Cloud heats up
To be honest, this is the prediction I’m most excited about. The competitive dynamic between Snowflake and Databricks is pushing both companies faster than ever before to innovate and differentiate themselves. This fierce competition will benefit end-users and software companies integrating with these platforms by not only delivering new features, but investing in increasingly diverse and rich partner ecosystems.
For prospects and users of both tools, this period will be delightful and epitomizes why competitive markets are so beneficial for consumers. It will also push teams to think hard about their long-term architecture, and hopefully solidify the benefits of continually moving upstack to ride the innovation wave of major platforms.
5. Catalogs and MDM will resurface
Although you may not have heard much about catalogs and MDM lately, you will soon. In the coming year, I think we’ll see a rebirth of importance and a resurgence of investment in both areas.
Traditional players like Alation will face increasingly stiff competition from vertically integrated offerings such as the Databricks Unity Catalog, Amazon and Google’s Data Catalogs, as well as many other emerging open-source tools. With this resurgence will also come an increased focus on data lineage, data reliability, and potentially even end-to-end data costing.
6. Businesses will start adopting data mesh and data fabric
I already mentioned fragmentation several times, but I’ll mention it again.
The truth is that fragmented data strategies are happening faster than enterprise companies can consolidate them. Instead of fighting it, I predict organizations will begin to accept that reality and start meaningfully examining data mesh and data fabric.
Although both terms have gotten a lot of buzz in recent years, most CTOs have discussed them on more of a theoretical level than a practical one. This year, that’s going to change. Even if they may not have money to spend on these tools immediately, tech teams will commit to exploring what data mesh and data fabric really do and how they could improve their team’s agility.
