Preparing Data for NLP
- several cleaning techniques used to ensure the data that powers the model is valid
Standardization
Standardization is the process of transforming data into a consistent format or structure to improve accuracy and improve downstream language processing tasks.
- e.g., removing capitalization
- may want to keep certain characters based on the project
- e.g., economics may want to keep currency symbols
Tokenization
Tokenization is a de-identification method where a unique token is substituted for real data.
- is the process of breaking up a text document into chunks of information called tokens
- first step in any NLP pipeline
- A tokenizer breaks unstructured data and natural language text into chunks of information that can be considered as discrete elements
- Tokenized text can immediately be turned into numerical data structures suitable for machine learning through vectorization
- Tokens can be used:
- directly by a computer to trigger useful actions and responses
- in a machine learning pipeline as features that trigger more complex decisions or behavior
- can separate sentences, words, characters, or subwords
- splitting the text into sentences is called sentence tokenization
- Splitting the text into words is known as word tokenization
- simplest way to tokenize text is to use white space within a string as the “delimiter” of words
- Punctuation, word and sentence, and treebank-based delimiters are also commonly leveraged
- popular libraries in Python:
- HuggingFace Tokenizers
- NLTK
- spaCy
- Gensim
- SentencePiece
- popular library in R:
- tokenizers
Stop Word Removal
Stop words are common or high-frequency words that are used in the business or general language, but not appropriate for including in the data read by a machine learning model.
- words that do not add much information to the text
- represent some of the most common words in any language
- e.g., articles, prepositions, pronouns, conjunctions
- removing stop words:
- reduces the size of data
- gives more clarity
- can be detrimental if working with contextual models
Stemming
Stemming is a natural language processing (NLP) technique that removes the most common endings of words, reducing them to base form for tasks like text normalization, query processing, and information retrieval.
- is the process of removing the morphological affixes to words, thus reducing a word to its stem or root word
- removes the inflection from longer words and leaves only the root
- e.g., helping → help
- One caveat of many stemming packages:
- the resulting stem word may or may not be a valid word in the given language
Lemmatization is a natural language processing (NLP) technique that reduces a word to its base or root form for tasks like text normalization, query processing, and information retrieval.
- converts words to their root form
- but a check is made to ensure the resulting root is a valid word for the given language
Regular Expressions
Regular expressions (RegEx) are a group of characters that describe how to execute a specific search pattern on a given text.
- powerful tool enabling to efficiently search, match, and manipulate text data
- provide a concise and expressive syntax for defining patterns within text
- invaluable for tasks like data cleaning, feature extraction, and sentiment analysis