cdCon+GitOpsCon 2023

Building a high quality software product is a breeze compared to building a high quality data product. Because software development being an evolved field has defined development best practices and the tools to implement these practices throughout the software lifecycle. On the contrary, most of the data teams have difficulty incorporating best practices like testing changes in isolation, reproducing a data error or rolling back in case of quality issues in production. With an ever-increasing number of organizations relying on data to make business decisions, the cost of data error in production is significantly higher too. Thus it becomes imperative for data teams to ensure the accuracy, resiliency and observability of data in production. So how does one go about extending software development best practices to data products? In this talk, you will learn about industry best practices for data lifecycle management and the open source tools such as Jenkins and lakeFS that help implement them.