To achieve full mastery with any tool, a craftsman must deeply understand its purpose, its abilities, and its limitations. A strong conceptual model enables its correct use.

The use of data requires recognizing its purity, its organization. If it is well organized, it can be used efficiently. If it is not well organized, if it is mixed in content, it requires more robustness in its use.

Data Wash organizes and analyzes data.

The degree of organization, of purity, of a dataset is measured in a concept called data entropy. In physics, if a system has a random distribution of things, it has high entropy; it is not very useful. If it is well organized, it has low entropy. And is very useful. But in physics, there is a movement from low to high entropy. There is no dynamic movement to data entropy. It is simply a description of the data.

The application of the concept of entropy to data was performed in a 1948 paper on communication theory by Claude Shannon, often called the “father of information theory.”

Data entropy is important because it measures disorder or randomness in data, which is crucial for applications like machine learning, cryptography, and data analysis. It helps in building accurate models by identifying the most informative features, securing communications through strong encryption keys, and analyzing files to identify compressed or encrypted content. According to research by IBM, poor data quality costs the US economy $3.1 trillion annually!

The terms “data entropy” and “information entropy” are often used interchangeably, but "data entropy" typically refers to the disorder in a specific dataset, which we are talking about here, while "information entropy" is the more formal concept from information theory that quantifies the average uncertainty of information in a random variable. Information entropy is a theoretical measure, perhaps of data flow, whereas data entropy is the practical application of that concept to the nature of the elements in a given dataset.

Information entropy addresses the predictability of transmitted information, of data flow. It is a measure of complexity, as highly chaotic, randomly distributed systems have high entropy and highly ordered ones low entropy. Information entropy of a random variable quantifies the uncertainty in that random variable. For probability p, entropy is highest when p = 0.5 and lowest when p = 0 or p = 1.

Data entropy is a measure of disorder or randomness within a dataset. High data entropy indicates more chaos, which makes it harder to analyze and derive insights. It is used in data science to understand the variability and structure of data. If a dataset is perfectly predictable (e.g., all values are the same), its data entropy is zero, meaning there is no uncertainty.

It is common to discuss homogeneous and heterogeneous data sets. Heterogeneous data structures can contain elements of different data types, say, integers and decimals. Homogeneous datasets can only contain data of the same data type. They are easy to work with because they are more efficient; we don’t have to check for data type. Heterogeneous structures require more checking but can apply to more complicated situations.

But, what if the data set is not one dimensional, but two, as with images?

Data Wash is designed to deal with complicated, multi-dimensional datasets.

Data Wash's tools apply entropy analysis to two dimensional images. In certain situations you may wish to reduce entropy, perhaps to increase coherence and reduce noise of a single class. And you could use entropy analysis to find class bifurcation suggestions. In other situations you may wish to increase entropy. For example, prior to training a deep learning model you may want to shuffle your entire, multi-class dataset to improve training performance and mitigate overfitting.

This breaks new conceptual ground, but most of the ideas and considerations about entropy that hold true for numerical datasets can also be true for image datasets.

In information theory, a symbol is the fundamental component that combines with other symbols to form messages. Symbols are the discrete carriers of information, used to build up a message, that is then transmitted and decoded by a receiver. The amount of information a symbol conveys is inversely related to its probability. A symbol with a probability of one (a certainty) provides no information.

In deep learning models (DLMs), tokens are an analog to symbols. Tokens represent the fundamental, discrete information forms input to a model.

The fundamental, discreet inputs analyzed by Data Wash are the information represented on any two dimensional pixel array. Commonly, these are image files, but the image could include 2D plotted time series data as easily as it can cat pictures.

The terms “images” or “inputs” will refer to the discrete units within datasets. And we use the terms "dataset", "population set", “population”, and "set" interchangeably.

To bridge the understanding of information and data entropy to that of images, we can relate entropy to the homogeneous / heterogeneous state of the population. Imagine an image dataset with a completely homogeneous mixture of 2+ classes, perhaps images of digits with a uniform mixture of numbers "3" and "4". If this homogenous mixture were to output component images, the dataset would produce a purely random (p = 0.5) output signal – it would not be possible to predict the next output.

The spectrum of homogeneous versus heterogeneous data has another important distinction. The population may have images belonging to a single class, cats, or we may have a mixture of images belonging to multiple classes, cats and dogs, or animals and plants.

A mixture of 2+ classes, each of which was homogeneous, would produce a purely random (p = 0.5) output signal and exhibit a high entropy. If the dataset was a homogeneous set of 1 class, then its output would be completely predictable (p = 1), exhibiting low entropy.

No one-class dataset is exactly homogenous because an ideal class is not comprised of exactly the same, that is duplicate, images. In fact duplicates are bad. An ideal class comprises images that are similar enough to truly fit the semantic label of the class, cars, while encompassing the natural variability of that class, domestic and foreign, gas and electric. We should consider a one-class set as "homogeneous" or "heterogeneous" depending on the model builder's intent for that class. Therefore the threshold of inferential similarity between members of a single class (that is considered "homogeneous" or "heterogeneous") depends upon the model builder's needs.

The extension of entropy to images is a measure of similarity and difference that correlates to the homogeneity or heterogeneity of the dataset assessed.

As with numerical datasets, entropy is only one tool to be used in harmony with others. One should not over-optimize entropy at the expense of the other characteristics your dataset must represent to achieve a robust, generalizing model. A class with very similar inputs will have low entropy, which is good from the perspective of class coherence and low noise. But one should not eliminate the important feature or natural feature variation necessary for strong generalization. A biased dataset might become more biased if entropy is reduced in certain scenarios.

Your goal isn’t always to minimize the entropy. It needs to be balanced with your model's other needs.

Luckily, Data Wash provides a robust toolkit of image dataset cleaning and preparation tools to help you do just that!