Understanding AI and NLP Basics

After discussing Prompt Engineering in the previous article, in this one I want to step back to get a clearer view: How does AI actually process language?

Many people (including myself when I first started) often think AI “understands” language like humans do. But that’s not quite the case. Understanding how AI operates at a basic level will help us:

• Have more realistic expectations of AI
• Understand why there are very good answers, but also sometimes incorrect or nonsensical ones
• And more importantly, know how to work with AI more effectively

1. How Does AI Process Language?

AI doesn’t understand language in the sense of “comprehending meaning” like humans. Instead, it operates based on pattern recognition and statistical probability.

Large Language Models (LLMs) are trained on massive amounts of text data. From this, they learn to:

• Predict the next word with the highest probability of appearing
• Recognize familiar patterns in context
• Generate text that looks like it was written by humans

When you input a question, the processing typically follows these steps:

1. Receive the input text
2. Convert text into a format machines can process
3. Analyze context based on learned data
4. Generate the final response

This explains why AI can write very fluently, but sometimes answers incorrectly on the essence of the problem.

2. Core Technical Concepts

To understand AI operation at a basic level, there are three important concepts frequently mentioned.

Tokenization

This is the first step in language processing. Text is broken down into tokens, which can be:

• A word
• Part of a word
• Or even a character

AI doesn’t work directly with complete sentences, but with these tokens.

Embeddings

After tokenization, tokens are converted into numerical vectors in a multi-dimensional space.

This approach helps AI recognize semantic relationships between words. For example:

• Words with similar meanings will have vectors close to each other
• Words with different meanings will be farther apart

Thanks to embeddings, AI can understand semantic relationships, not just simple keyword matching.

Model Training

AI learns through various methods, most commonly including:

• Supervised learning: labeled data
• Unsupervised learning: finding patterns in data autonomously
• Reinforcement learning: improving based on feedback

Typically, a model goes through:

• Pre-training to learn language and general knowledge
• Fine-tuning to serve specific purposes like conversation, code writing, or data analysis

3. Popular Large Language Models

Currently, there are many different LLMs, each with its own design philosophy and strengths.

• GPT Strong in conversation, summarization, content writing, and programming support. This is the most widely used model group today.

• LLaMA Designed to achieve high performance while optimizing resources, suitable for flexible research and deployment.

• Claude Focuses heavily on safety, ethics, and reducing bias in responses.

Understanding the differences between models helps you choose the right tool for each need, rather than using one model for everything.

Conclusion

When you understand how AI processes language and how models are built, you will:

• Be more realistic when working with AI
• More easily accept model limitations
• And know how to craft appropriate prompts to leverage their strengths

This is also an important foundation for using Prompt Engineering more effectively.

Illustrative Example

You can imagine AI like a code translator.

When you give it a sentence, it doesn’t “read” text like humans do, but converts each part of the sentence into numbers on a massive map.

By comparing the distances between those numbers, AI knows which words are related to which, and from there assembles a seemingly reasonable answer.