AI Explainer: What Is Data Cleaning?

In a previous blog post, which was a glossary of terms related to artificial intelligence, I included this brief definition of "data preprocessing":

Data Preprocessing: The process of cleaning, transforming and organizing data to make it suitable for analysis or machine learning.

It is common for people familiar with these matters to talk about not having clean data. When dealing with AI for whatever your needs are, clean data is crucial for the quality of results. Garbage in, garbage out, as they say. So, let’s dive into what it means to have clean data.

In the context of data preprocessing, "cleaning data" refers to the process of identifying and correcting errors or inconsistencies in a dataset to ensure that it is accurate, complete and ready for analysis or use in machine learning models. Data cleaning is a critical step because real-world data is often messy and can contain various issues that need to be resolved.

Here are some common tasks involved in cleaning data:

• Handling Missing Values: Missing data points are common in datasets. Data cleaning may involve strategies such as filling in missing values with defaults, removing rows with missing values, or imputing missing values based on statistical techniques.
• Dealing With Outliers: Outliers are data points that deviate significantly from the norm. Data cleaning may involve identifying and either removing or transforming outliers to avoid skewing the analysis or machine learning model.
• Standardizing Data: Data may come in different formats or units. Cleaning data often includes standardizing units, scales or formats to ensure consistency.
• Removing Duplicates: Duplicate records can skew analyses and models. Data cleaning involves identifying and removing duplicate entries.
• Handling Inconsistent or Incorrect Data: Data may contain inconsistent values or errors. Cleaning data includes identifying and rectifying these inconsistencies, such as correcting typos, mismatches or inaccuracies.
• Addressing Data Entry Errors: Data may have entry errors, such as incorrect dates, values or categorical labels. Cleaning data involves identifying and correcting these errors.
• Validating Data: Ensuring that data adheres to predefined validation rules or constraints, such as date ranges, numerical limits or data types.
• Dealing With Categorical Data: Encoding categorical variables into numerical values or using one-hot encoding to represent categorical data appropriately for machine learning models.
• Text Data Cleaning: For natural language processing tasks, text data cleaning involves tokenization (see below for more on this), removing stop words (insignificant words), stemming or lemmatization (identifying root words), and handling special characters.
• Addressing Inconsistent Data Types: Data columns may have inconsistent data types (e.g., mixing numbers and text). Cleaning data includes ensuring uniform data types within columns.

What is tokenization, you ask?

The concept of tokenization as it relates to data cleaning is actually pretty straightforward. It is a fundamental step in data preprocessing and natural language processing (NLP) that involves breaking down a text document or a sequence of text into individual units called tokens. These tokens can be words, phrases, sentences or even subword units, depending on the level of granularity required for analysis. Tokenization is an essential step in data cleaning and text processing because it allows you to work with and analyze text data more effectively.

Here's a brief overview of tokenization and its significance in data cleaning:

• Breaking Text Into Tokens: Tokenization divides a continuous string of text into discrete tokens. In the case of word tokenization, it separates the text into individual words. For example, "the quick brown fox," would be tokenized into the tokens: "the," "quick," "brown" and "fox."
• Sentence Tokenization: In addition to word tokenization, you can perform sentence tokenization, which splits text into sentences. This is valuable when you want to analyze text at the sentence level.
• Subword Tokenization: In some cases, especially in NLP tasks involving languages with complex morphology (e.g., agglutinative languages like Turkish), subword tokenization is used to split words into smaller meaningful units (subword pieces or characters) to improve text processing and analysis.

Why is tokenization important (in multiple areas)?

• Data Cleaning: Tokenization assists in identifying and cleaning noisy or inconsistent data. For example, if you're tokenizing text data, you can easily identify words or phrases that require further preprocessing, such as stemming, lemmatization or the removal of stop words.
• Text Analysis: Tokenization is a foundational step for various NLP tasks, such as text classification, sentiment analysis, information retrieval and machine learning. By converting text into tokens, you create discrete units that can be analyzed and processed.
• Feature Extraction: In machine learning and text analysis, tokens serve as features for training models. Each token can be considered a feature, and the frequency or presence of specific tokens can be used to represent the data.
• Text Search and Retrieval: In information retrieval systems and search engines, tokenization helps break down queries and documents into smaller units, facilitating more precise search and retrieval.
• Language Processing: Tokenization is essential for understanding the structure of a language, including word boundaries, grammar and syntax.

Tokenization is a crucial data preprocessing step that simplifies text data, making it more manageable and facilitating various text analysis and language processing tasks. Depending on your specific goals and the characteristics of your text data, you can tailor the tokenization process to suit your needs, whether that involves word, sentence or subword tokenization.

Back to the big picture of data cleaning. In summary, data cleaning is a crucial step in the data preparation process because the quality of the data directly impacts the quality and reliability of the analysis or machine learning models built upon it. Failure to clean data properly can lead to inaccurate results and unreliable insights. When you put garbage in, you can expect to get garbage out. So in your AI journey, remember that data cleaning is a fundamental aspect of data preprocessing that ensures that the data is in a suitable form for subsequent analysis or modeling.