Part 1: Introduction to NLP and Preprocessing

This part introduces NLP concepts and essential preprocessing techniques.

1. What is NLP?

• NLP is a Machine Learning technology that enables computers to understand and interpret human language.
• It allows machines to comprehend human intent in text, speech, or other language data.

2. Why is NLP Important?

• Large organisations handle massive data—emails, invoices, messages, audio, video.
• Manually interpreting such data is infeasible.
• NLP automates interpretation, intent recognition, and response generation.

3. Real-World Applications:

• Chatbots: Understand user queries and generate automated responses.
• Voice Assistants: Process spoken queries (e.g., "Is it sunny or rainy?").
• Response Generation: Enables conversational systems to reply intelligently.
• Language Translation: Translates languages in real-time (e.g., Tamil to Malayalam).

4. Why Preprocessing is Necessary

• Machines can't interpret raw data directly.
• Preprocessing prepares text for model input, involving:
  • Feature encoding
  • Feature extraction

5. Key Preprocessing Techniques:

• Tokenisation: Breaks text into individual units (tokens), e.g.,

  "Today is a rainy day" → ["Today", "is", "a", "rainy", "day"].

• Stemming: Strips word endings to get the root form, e.g.,

  "running" → "run", "jumps" → "jump".