About the Importance of Data in Machine Learning

Everything You Always Wanted to Know About Data * But Were Afraid to Ask

Photo by ThisisEngineering RAEng on Unsplash

Data refers to stored and unprocessed information. We are currently living in the “data age,” where data is being generated constantly, every second, everywhere. It is estimated that each person contributes to the creation of 1.7 MB of data every second.

For AI and machine learning systems, data acts as the fuel that powers them, keeping them operational. The quality of the data supplied is crucial for these systems to function effectively. Essentially, every learning system is only as proficient as the data it is fed.

Let’s begin with some fundamental data terminology.

Terminology

• Direct data is collected directly from a source and utilized without any modifications.
• Indirect data refers to data that is repurposed and used for a different objective than the one for which it was originally collected.

For example, in a translation application, the input is word sequences. These word sequences sent to the application are direct data. During backend operations, this application may source information from reference tables, dictionaries, or other data within the database. These sources are considered indirect data.

The origin and collection method of direct data are clear, ensuring the data is precise and accurate. However, the volume of directly collected data is often limited, tending to comprise smaller datasets. Additionally, there may be challenges in accessing the data consistently, as it may not always be readily available.

Indirect data can be available in very large volumes and accessed continuously without interruption. However, potential biases or measurement errors that might have occurred during the original data collection process may not be known or identifiable.

• Static data is fixed and remains unchanged after its collection. It is historical, recorded at a specific moment in time, and does not reflect the current state or present conditions.
• Dynamic data is continually updated and evolves. With each update, the entire dataset transforms, consistently striving to maintain and reflect the most current information.

Consider a presentation you delivered at your company last year as an example of static data. It represents the information relevant to the date of the presentation. When you review it today, it does not provide any new insights.

On the other hand, think of Google Analytics as dynamic data. It offers a continuous stream of data regarding the traffic on your website, ensuring that the information you receive is always current and up-to-date.

• Raw data refers to the original, untouched data as it is collected. This can include various forms such as photographs, Word documents, or unprocessed signals from IoT devices.
• Derived data is the result of processing raw data. This involves operations such as merging and joining. An example would be creating a new, comprehensive Excel file by mapping and integrating specific columns from multiple Excel files.
• Tidy data represents the refined version of a dataset, organized and structured in a way that makes it ready for statistical analysis or use in machine learning projects.

We begin by collecting raw data, from which we derive new datasets. These datasets are then processed and refined to obtain tidy data, suitable for analysis or modeling.

• Attributes: Features, or columns in a dataset.
• Examples: Observations, or rows in a dataset.

Attributes and examples. Image by the author.

In a machine learning project, the dataset is typically partitioned into subsets.

• The training dataset is the largest portion and is utilized by the learning algorithm to train the model.
• The test dataset is withheld from the model during training and is used to assess the unbiased performance of the trained model. The model’s predictions, based on the test data observations, are compared with the actual labels to evaluate its accuracy.
• The validation dataset is employed to refine and tune the final model. The model’s hyperparameters are adjusted based on the performance metrics obtained from the validation dataset, aiding in the prevention of overfitting and ensuring the model’s generalizability.

Train-test-validation split. Source

The data partitioning process should adhere to certain conditions to ensure its effectiveness.

• The partitioning operation should be performed on the raw data. Subsequent data processing operations must be applied to each split subset separately. This approach is crucial to prevent data leakage and ensure the integrity of the training and evaluation process.
• Shuffle the data before splitting it, provided that the nature of the problem allows for this approach. This ensures a randomized distribution of data across the training, validation, and test sets.
• The validation and test datasets should have similar distributions to ensure that the model’s performance is consistently evaluated and that the insights gained from the validation phase are applicable during the final testing phase.
• If multiple identical observations exist, they should all be grouped into the same subset (either training, validation, or test) during the splitting process. This prevents a scenario where the model learns specific observations during training and then falsely appears successful when tested on the same observations.

In practice, 70% — 15% — 15% or 80% — 10% — 10% split ratios are preferred.

Data Collection

During the data collection process, certain questions emerge that require answers. The responses to these questions uncover important parameters regarding the data and the data collection methodology.

When considering your access to data, it’s essential to address questions such as:

1. Can you access the necessary data?
2. Is the data pre-existing, or does it need to be collected?
3. Is access to the data one-time, or can it be renewed?
4. Is the data purchased?
5. Is the data subject to copyright?
6. Does the data pose any ethical issues?
7. Does it comply with laws on personal data, or does it need to be anonymized?

These considerations are crucial in understanding the limitations and responsibilities associated with the data you intend to use.

Once access issues are resolved, the next challenge is assessing the scalability of your data:

1. How frequently can you generate new data?
2. What volume of data is required for your model to yield meaningful results?
3. What is the optimal size of the dataset needed?

Learning curves vs Training examples. Source

Utilizing learning curves can help you determine the necessary amount of data for a functional model. When you observe that the model’s performance starts plateauing on the learning curve graph, it suggests that adding more data does not significantly impact the model’s performance. In such cases, you may consider the following possibilities:

• Lack of Informative Features: The dataset may lack certain informative features. In this case, feature engineering may be necessary to extract more valuable information from the existing data.
• Maximum Information from Dataset: The plateau may indicate that you’ve extracted the maximum information the current dataset can provide. You can explore options for adding, discovering, or creating new features to enhance the dataset.
• Model Complexity: If the model used is relatively simple, you might reach a performance plateau. Transitioning to a more complex algorithm could yield better results.

In practice, there are some general thumb rules regarding the minimum size of a dataset.

• 10 times the number of features in your dataset.
• 100 or 1000 times the number of classes or categories in your classification task.
• 10 times the number of trainable parameters in your model.

These rules provide a rough guideline to ensure that your dataset has sufficient diversity and size to train a machine-learning model effectively. However, it’s important to note that these are rough guidelines, and the actual data requirements can vary depending on the complexity of the problem and the specific algorithms used.

Assessing the usability of the dataset is a critical step in data preparation. You should consider the following aspects:

1. Data Quality: Evaluate the quality of the dataset by checking for issues such as missing values, outliers, and inconsistencies.
2. Missing Values: Identify if there are any missing values in any of the features and calculate the ratio of missing data.
3. Feature Types: Ensure that the data types of each feature are consistent and correct. Address any data type discrepancies.
4. Tidiness: Assess how tidy the dataset is. Is it well-organized and structured for analysis, or is it in a raw format that requires preprocessing?
5. Duplicate Data: Check for duplicate records in the dataset and determine how to handle them if found.
6. Class Distribution: For supervised learning problems, examine the distribution of classes. Determine whether it is balanced or imbalanced, as this can impact model training and evaluation.

Addressing these issues and ensuring data quality is essential for building reliable and accurate machine learning models. Data preprocessing steps may be required to clean and prepare the dataset for analysis.

Assessing the inclusivity of the dataset is crucial, especially in applications like face recognition systems. You should consider whether the dataset represents a diverse and inclusive range of examples. For example:

1. Diversity: Ensure that the dataset includes a wide range of variations, such as different genders, ages, ethnicities, and expressions, to make the model robust and inclusive.
2. Representation: Check whether all relevant categories or classes are adequately represented in the dataset. For instance, if the goal is to recognize faces of all genders, it should include both male and female photos.
3. Bias and Fairness: Be mindful of potential bias in the dataset. Ensure that the dataset is balanced and does not favor one group over others. Addressing bias and fairness issues is critical for ethical and unbiased AI systems.
4. Real-world Scenarios: If the face recognition system is intended for real-world applications, the dataset should reflect real-world scenarios and variations.

It’s essential to have a deep understanding of your data, including the relationships between features and their relevance to the domain problem. This involves identifying which features provide valuable information about the problem at hand and leveraging domain knowledge to interpret the data effectively.

The reliability of the data is indeed a critical and fundamental consideration. When working with data, especially in applications like IoT systems, it’s important to assess the reliability of the measuring system. Key factors to consider include:

1. Measuring System Reliability: Verify whether the measuring system used to collect the data is reliable. This involves assessing its accuracy, precision, and consistency over time.
2. Maintenance and Calibration: Check whether the measuring system has been properly maintained and calibrated regularly. Maintenance practices can significantly impact data reliability.
3. Testing and Quality Control: Ensure that appropriate tests and quality control procedures have been conducted on the measuring system to validate its accuracy and reliability.
4. Data Validation: Implement data validation checks to identify and handle incorrectly measured signals. This can include outlier detection and data cleansing processes.
5. Impact on Model: Recognize that unreliable data can lead to inaccurate and unreliable models. Using unreliable data to build a model can result in issues when deploying the model in production.

Addressing data reliability issues is essential to ensure that the models built on this data perform as expected and provide trustworthy results when deployed in real-world scenarios.

Problems

Acquiring data can be a challenging task, and you may likely encounter difficulties along the way.

Get Okan Yenigün’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Gathering labeled data can often be a costly endeavor, and acquiring unlabeled data also involves its challenges. However, tagging unlabeled data adds an extra layer of complexity.

For instance, let’s consider the scenario of creating a dataset for categorizing shops on a street. We mounted a camera on a car and collected a substantial amount of image data, which constitutes our unlabeled dataset containing images of the shops. Yet, in order to assign labels to each shop’s sector, we face two options: either manual labeling by an employee or the development of an automated labeling system. Both options require additional efforts for the labeling task.

In certain instances, your dataset may exhibit low quality. Consider the example mentioned earlier with the camera-mounted car: the images captured by the camera could be of poor quality, with some shops appearing distorted. Alternatively, if you entrusted the tagging task to an employee, errors, and inaccuracies may have crept into the labeled data due to negligence or oversight.

Noise, characterized by factors like blurred images, distorted text, audio recordings overwhelmed by background noise from a crowd, or even sarcastically filled-out questionnaires, can significantly impact the quality of your dataset. This influence is particularly pronounced in small datasets, where noise can lead to overfitting issues in machine-learning models.

Your dataset may lack the necessary expressiveness and possess limited predictive power. For instance, if your goal is to recommend appropriate movies to users, but your dataset only contains user names and email addresses, it falls short of providing the essential information needed for effective recommendations. In such cases, it won’t work.

In a real-time application, maintaining up-to-date data is crucial. Consider a scenario where you’ve predicted the price of Bitcoin for the next day and offered this information as a service to your customers. However, on the following day, the Federal Reserve raises interest rates. If you fail to update this critical information in your dataset, your subsequent forecasts are bound to be inaccurate and unreliable, potentially resulting in significant setbacks for your customers.

Concept drift is indeed another significant issue to consider. Even if you regularly update your data and retrain your model, the accuracy of the model may remain relatively stable. However, concept drift can occur when there’s a fundamental change in one of the features in your dataset. For instance, if access to your website is suddenly restricted for individuals below a certain age, this can lead to a shift in the distribution of the age feature. Over time, the model’s performance may deteriorate, and it may no longer perform as well as it did previously. In such cases, it’s important to initiate a new learning study, possibly updating the model’s hyperparameters to adapt to the changing data distribution and concept drift.

Outliers can significantly impact the performance of certain algorithms. While it’s common to address outliers during the feature engineering stage after the dataset is created, it’s advisable, when feasible, to take preemptive measures against outliers during the dataset creation stage itself. For instance, integrating an anomaly detection system into an IoT system can serve as a proactive approach to filter out outliers at the data collection phase, ensuring that the dataset is less susceptible to the adverse effects of outliers on subsequent analyses and modeling.

Bias

Bias refers to inconsistencies or inaccuracies in the data that can affect its alignment with the domain problem. It’s important to note that bias can manifest in various forms, leading to different types of bias within the dataset.

Selection bias arises from errors or mistakes made when choosing samples from a population. These errors in selection choices can result in the introduction of bias into the dataset, potentially impacting the representativeness of the sample.

A classic example illustrating the issue of selection bias occurred during the 1936 American presidential election. Literary Digest, a renowned magazine at the time, conducted a poll by soliciting opinions from its readers regarding the presidential race between Alfred Landon and Franklin D. Roosevelt. Surprisingly, their mail-in poll indicated that Landon was ahead. However, when the actual elections occurred, the results were the opposite, with Roosevelt winning by a significant margin.

Source

Indeed, the issue in the Literary Digest magazine’s poll was that its readership did not accurately reflect the diversity of the entire population. The selected sample consisted primarily of middle to upper-class individuals, which led to a skewed representation of the population. This discrepancy highlighted the importance of ensuring that the sample used in polling and surveys is representative of the broader population. This principle remains relevant today, where online platforms like Twitter may not accurately represent the diversity of opinions and preferences within the entire population, as seen in various political contexts. We couldn’t even imagine that Trump would be elected president just by looking at the Twitter population.

To avoid such errors, it is essential to scrutinize the dataset creation process and the method of selection. We must assess the extent to which our sample represents the population within the domain space, examine the scenarios it encompasses, and actively work on enhancing and broadening the inclusivity of these factors.

Another form of this bias is self-selection bias. Take, for instance, restaurant reviews. These comments are not random; instead, they are provided by individuals who have chosen to leave feedback. Consequently, the comments do not represent the broader population accurately.

Sampling bias is a similar one. Imagine you have a multi-class problem with 10 distinct classes. However, two of these classes make up 85% of the dataset. In such cases, the model may exhibit a bias toward classifying inputs into one of these two dominant classes. It’s crucial to strive for balanced class proportions to mitigate this bias.

Omitted variable bias is another form of bias, where a crucial feature is unintentionally left out of the dataset. These omissions can impact other features and ultimately influence the model’s performance.

Socrates quotes. Source

How do we know what we don’t know? Detecting what we don’t know can be challenging. One approach to address this issue is to begin by including all available data. Subsequently, we can thoroughly analyze feature elimination and aim to incorporate as many relevant features as possible into the model. However, this approach may lead to a sparse dataset.

To mitigate experimenter bias, it is essential to formulate well-structured survey questions that encourage respondents to provide answers. Avoiding judgmental or leading questions can help create a more unbiased survey experience.

Systematic value distortion arises when the measurement device introduces errors, such as a camera displaying green as yellow during sunset. To address this issue, thorough maintenance and testing of the measuring instrument are crucial. A reliable approach is to utilize multiple measuring instruments to cross-validate the data and minimize systematic distortions.

Prejudice bias is particularly prevalent in datasets derived from historical sources, often reflecting deep-seated issues like gender inequality and racism. When working with such raw data, it may be necessary to address this bias by either under-sampling or over-sampling relevant data or, in some cases, completely removing biased data points.

There is also a phenomenon known as funding bias. When collecting data from the public domain, the influence of funding from certain institutions can skew the order or priorities within that public space. For instance, a battery manufacturer might provide funding to a journalist to continuously produce news about electric vehicles, thereby shaping the agenda in favor of their products.

In such situations, it becomes crucial to understand the inter-company relationships and closely monitor public spaces and agendas. Gathering inside information, perhaps through whistleblowers or journalists, can also significantly contribute to the success of the study.

In the example mentioned earlier, if one of our employees responsible for labeling has a bias related to the subject they are working on, it can introduce a labeling bias into the tagging process.

Data Leakage

Also known as target leakage, this issue poses a critical challenge to the model’s success. To put it simply, if external information infiltrates the training dataset during training, it is referred to as data leakage. When such leakage occurs, the model acquires information it shouldn’t have access to under normal circumstances. As a result, we may achieve successful and optimistic results during testing, but the model’s performance deteriorates when deployed in a production environment.

Leakage can manifest in various ways, with the three most common causes being:

Target is a function of any feature. Consider Einstein’s famous formula, E=mc², where energy (E) is calculated based on mass (m) and the square of the speed of light ©. If our goal is to estimate energy but we already have information about mass in our dataset, the model might predict energy with high accuracy. However, in this scenario, the target variable has effectively leaked through the feature of mass. In such cases, developing a machine-learning model is unnecessary.

E = mc2. Source

The feature is hiding the target. In some cases, the feature itself contains information about the target variable. For example, if our goal is to predict gender, and one of our features is the person’s first name, then names like “Michael” are likely to be associated with males. This kind of feature can hide the target variable, making the prediction task trivial.

Bankruptcy. Source

Some features are from the future! The classic example of this type of leak is as follows:

Consider a scenario where you’re building a model to predict whether customers will repay their debts to a bank. You have a feature in your dataset called “late_payment_reminder,” and you include this feature in your model. During testing, everything appears to be working well, and your model performs superbly. However, once the model is deployed in production, it fails to make accurate predictions.

Subsequently, you discover that the “late_payment_reminder” feature consistently has a value of zero in the production environment. This occurs because the bank is utilizing your model before extending any loans to customers, meaning that there are no reminders sent.

Data Drift

Data drift occurs when the statistical characteristics of the training data for a machine learning model change over time, leading to a deterioration in model performance. This phenomenon arises when the data initially used to train the model no longer effectively reflects the evolving real-world conditions.

• Distribution Drift (covariance shift): It happens when the distribution of any input data changes over time.

Distribution Drift. Image by the author.

• Concept Drift: The underlying concept being modeled changes over time. For example, in a fraud detection system, the types of fraudulent activities may evolve, requiring the model to adapt to new patterns.

Types of Concept Drift. Source

• Seasonal Variations: Data collected at different times of the year may have different statistical characteristics.

Addressing data drift is crucial as machine learning models trained on historical data can lose their predictive accuracy over time. To ensure the ongoing effectiveness of these models in production settings, regular monitoring and evaluation are essential for detecting and mitigating the effects of data drift.

Data Quality

How can we assess the quality of our data? High-quality data exhibits specific characteristics.

• Good data is informative enough.
• It generalizes the problem adequately.
• It accurately represents the inputs required for the production process.
• No bias
• No leakage or feedback loop
• Has consistent labels
• The volume should be enough.
• None (or controlled) drift.

Conclusion

Having more data generally leads to better model performance.

As mentioned earlier, data is the lifeblood of all intelligent systems. Data management often takes precedence over the models themselves in many machine-learning projects. When collecting data, it’s crucial to adhere to standard practices, avoid bias, and be vigilant about data leakage.

That concludes our discussion for now. Thank you for reading thus far. Please don’t hesitate to reach out or leave comments if you have any questions or thoughts.

Read More…

How To Handle Missing Values?

What Are The Feature Transformation Techniques?

What are Outliers and Anomalies?

The Most Used Feature Selection Methods

A Detailed Catalog of Dimensionality Reduction

References

Machine Learning Engineering by Andriy Burkov

https://as-ict-ria-yves.weebly.com/direct-and-indirect-data.html

https://www.cience.com/blog/static-vs-dynamic-data-in-b2b-sales

https://www.jarad.me/consulting/2019/12/20/data

https://en.wikipedia.org/wiki/Selection_bias

https://en.wikipedia.org/wiki/1936_United_States_presidential_election

What is Data Quality and Why is it Important?

Mlearning.ai Submission Suggestions

In Plain English 🚀

Thank you for being a part of the In Plain English community! Before you go:

• Be sure to clap and follow the writer ️👏️️
• Follow us: X | LinkedIn | YouTube | Discord | Newsletter
• Visit our other platforms: CoFeed | Differ
• More content at PlainEnglish.io