Last week our team had a chance to fly over to London to attend the UK’s biggest data industry gathering - Big Data LDN. It was a great opportunity to meet some old and new faces and hear from the data experts on where the industry is going.

TLDR, “business value” continues to be the main thing everyone talks about, the good trend however, there seems to be a shift from the theory to more practical discussions.

‍

Conceptual Data Modeling is Making a Comeback

One thing is for sure, conceptual data modeling is coming back. From the event’s chairman Mike Fergusson to Joe Reis (co-author of Fundamentals of Data Engineering), and industry experts like Andrea Gioia - everyone agrees that we need to start with a less technical approach when it comes to data initiatives.

It's no longer just about getting into the technical nitty-gritty right from the start. There's a need to take a step back, look at the bigger picture, and understand data from a broader perspective.

As highlighted by Joe Reis during his session, the co-author of Fundamentals of Data Engineering, we need to be adaptable. While there might be myriad ways to approach data modeling, flexibility is the key. He emphasized that organizations should not be shoehorned into a singular approach, given the diverse needs and objectives that vary from one organization to the other.

Moreover, Reis mentioned the significance of bridging the gap between the technical and non-technical stakeholders. Data modeling shouldn't be a realm exclusive to techies. It should be accessible, understandable, and actionable for all stakeholders involved. Physical modeling, while crucial, isn't the be-all and end-all. The addition of both conceptual and logical modeling into the workflow is paramount.

‍

The Surge of Data Products and Democratization of Data Engineering

Data products, popularized by Data Mesh, were a buzzword at Big Data LDN. Many speakers pointed out the rise of data fabric software, which allows for the creation of these data products with an intent to build once and repurpose across varied analytical tasks.

Another pivotal takeaway was the democratization of data engineering. As this field becomes more accessible, data modeling will invariably become a core skill, central to data literacy initiatives. If the goal is to enable domain-based citizen data engineers to construct data engineering pipelines effectively, then ensuring that they're adept in data modeling becomes essential.

Furthermore, the event’s chairman, Mike Ferguson, underscored the potential growth of internal data marketplaces. Such platforms will not only act as a repository for data products but will also foster collaboration by sharing analytical tools ranging from ML models to BI reports.

‍

Final Thoughts

As the question of ROI (Return on investment) continues to grow in the enterprise data space, it is extremely interesting to see where will the industry head in the coming years. If there’s one thing for certain though, is that things have to change, and we need to reflect on what has gone wrong in the past decade!

P.S. We’ll be back at Big Data LDN in 2024! 👏

‍