Preparing Data for NLP


  • several cleaning techniques used to ensure the data that powers the model is valid

Standardization

Standardization is the process of transforming data into a consistent format or structure to improve accuracy and improve downstream language processing tasks.

  • e.g., removing capitalization
  • may want to keep certain characters based on the project
    • e.g., economics may want to keep currency symbols

Tokenization

Tokenization is a de-identification method where a unique token is substituted for real data.

  • is the process of breaking up a text document into chunks of information called tokens
  • first step in any NLP pipeline
  • A tokenizer breaks unstructured data and natural language text into chunks of information that can be considered as discrete elements
  • Tokenized text can immediately be turned into numerical data structures suitable for machine learning through vectorization
  • Tokens can be used:
    • directly by a computer to trigger useful actions and responses
    • in a machine learning pipeline as features that trigger more complex decisions or behavior
  • can separate sentences, words, characters, or subwords
    • splitting the text into sentences is called sentence tokenization
    • Splitting the text into words is known as word tokenization
  • simplest way to tokenize text is to use white space within a string as the “delimiter” of words
    • Punctuation, word and sentence, and treebank-based delimiters are also commonly leveraged
  • popular libraries in Python:
    • HuggingFace Tokenizers
    • NLTK
    • spaCy
    • Gensim
    • SentencePiece
  • popular library in R:
    • tokenizers

Stop Word Removal

Stop words are common or high-frequency words that are used in the business or general language, but not appropriate for including in the data read by a machine learning model.

  • words that do not add much information to the text
  • represent some of the most common words in any language
    • e.g., articles, prepositions, pronouns, conjunctions
  • removing stop words:
    • reduces the size of data
    • gives more clarity
  • can be detrimental if working with contextual models

Stemming

Stemming is a natural language processing (NLP) technique that removes the most common endings of words, reducing them to base form for tasks like text normalization, query processing, and information retrieval.

  • is the process of removing the morphological affixes to words, thus reducing a word to its stem or root word
  • removes the inflection from longer words and leaves only the root
    • e.g., helping → help
  • One caveat of many stemming packages:
    • the resulting stem word may or may not be a valid word in the given language

Lemmatization is a natural language processing (NLP) technique that reduces a word to its base or root form for tasks like text normalization, query processing, and information retrieval.

  • converts words to their root form
  • but a check is made to ensure the resulting root is a valid word for the given language

Regular Expressions

Regular expressions (RegEx) are a group of characters that describe how to execute a specific search pattern on a given text.

  • powerful tool enabling to efficiently search, match, and manipulate text data
  • provide a concise and expressive syntax for defining patterns within text
  • invaluable for tasks like data cleaning, feature extraction, and sentiment analysis