Over the last 40 years, Organizations have shifted their focus multiple times to continuously generate value from data while adapting to the rapid change of technologies. In the 1980s, the focus was on storing data in relational databases and data warehouses. In the 1990s, Organizations attempted to get insights from their data by leveraging OLAP and Business Intelligence systems. In the 2000s, it was acknowledged that it was necessary to deal with big data. In the 2010s, the buzzword was data science & AI. However, we had to wait until the 2020s to see the focus on Data Governance and Ethics.
In fact, those Organizations that started working out their own data strategy soon realized that benefits were hard to achieve, and data were still treated inefficiently. In more practical terms, data teams adopted and developed outputs based on new technologies such as Power BI dashboards and complex data science models, but the business continued to make decisions based only on day-to-day experience, and Excel remained the reference technology. From this point, it became clear that a data strategy was needed to enable the business to utilize its own data and that the business needed to be in charge of it.
Another important factor that affected Data Governance’s popularity was the recent boom in data protection regulations. For example, in 2018, the General Data Protection Regulation affected Europe and the UK and the new Swiss Federal Act on Data Protection (nFADP) came into effect on September 1, 2023. Furthermore, sector-specific regulations require companies to stay in control of their own data.
Other important factors that led CIOs to develop Data Governance also included enhancing customer experience and user trust in data, reducing costs, and improving operational efficiency.
In the last two years, instead, we have witnessed the boom of generative AI. This boom came as the icing on the cake of the international debate about AI & ethics. Furthermore, it has been a few years since legislators around the world discussed the topic and agreed that some sort of AI governance is needed. The European Commission decided to play a leading role in this by proposing the first-ever legal framework on AI, which aims to address the risks of AI. The regulatory proposal aims to provide AI developers, deployers and users with clear requirements and obligations regarding specific uses of AI. Following the Commission’s proposal in April 2021, the regulation could enter into force in a transitional period by the end of 2023. Standards would be required and formulated throughout this period, and the operational governance structures would be established. The regulation might be applicable to operators as early as the latter half of 2024, once the standards are in place and initial conformity assessments have been conducted.
To complete the scenario that explains why data governance became the hot topic of the 2020s, we also need to consider how Data Catalogs evolved. Data Catalogs have become increasingly popular since 2015 when Alation introduced the first technology that enabled end-users to curate business metadata and find data assets (such as business definitions or SQL tables) as easily as internet users search for items on Amazon.com.
With the additional push of technology vendors, organisations also realised the value of Data Catalogs as tools that facilitate data democratization and ultimately generate efficiency. To explain it more clearly with an example, IBM stated that Organizations can spend up to 80% of their time searching and preparing for data. It means that Organizations pay a very high salary to Data Scientists mainly to find out where the data is, and only 20% of the time is to generate the actual insight. Furthermore, Data Catalogs bring to the table a new opportunity for lower maturity Organizations to reshape the IT’s role, which historically was believed the owner of the Organization’s data.
Data Catalogs are technologies that are continuously being developed to meet corporate governance requirements as well as data governance requirements. It’s believed that Generative AI will remove some significant barriers to a company’s adoption by taking on a good part of the heavy lifting that businesses normally reject due to a lack of time. Furthermore, multiple vendors are adjusting the prices of Data Catalogs to market conditions, which will support the proliferation of this technology along with good data governance practices.
Although data governance is not new, it has gained more attention and importance in the 2020s due to various factors. These factors include the rapid evolution of data technologies, an increasing demand for data-driven insights, a growing awareness of data ethics and regulations, and the emergence of generative AI and data catalogs. Data governance can help organizations manage their data assets effectively, ensure data quality and security, foster data culture and literacy, and leverage data for innovation and value creation. Therefore, data governance is necessary and an opportunity for organizations to gain a competitive edge in this digit
Sandro Scordo
Head of Data Strategy & Governance Excellence Center
