Before diving in, let us discuss what data quality entails. IBM puts it so well - Data quality measures how well a dataset meets criteria for accuracy, completeness, validity, consistency, uniqueness,
This post aims to provide a historical picture of the evolution of the typical data stack over a span of about 5 decades. A lot has happened, but I will try to keep it as simple and digestible as possible. Fig 1: Evolution of the tools, concepts and...
Introduction This article will discuss compression in the Big Data context, covering the types and methods of compression. I will also highlight why and when each type and method should be used. Diving in What is compression? According to the gene...
There's often a huge fuss about making data-driven decisions, leveraging data analytics, using data science and data-centred thinking. From a technological point of view, data is usually stored and accessed using databases. Databases, often referred ...
This post draws extensively from a book I am currently reading and my experience managing data systems (primarily, core data infrastructure). The book is Designing Data-Intensive Applications by Martin Kleppmann. Hey! What is even a Data System? Firs...
Data Source: National Grid ESO
Brief This project aims to show the components of the Climate Stripes and the visual process of building the infamous Climate Stripes. The Climate Stripes, created by Professor Ed Hawkins, is a set of bars of varying colours stacked together, with al...
Complete Visualisation: Tools used: R programming: ggplot, maps, patchwork libraries Brief This project uses the intersection of two studies to show the geographical biodiversity of the Aeshna Caerulea species over the years in the UK. The Aeshna Ca...
For deep learning enthusiasts, working with the MNIST dataset is the equivalent of writing the "hello world" program when learning a new scripting language like Python, JavaScript etc. In this project, I experiment with the vast and highly dimensiona...
