Overview of natural language processing

Overview and definition of NLP

Natural language processing (NLP) is a fascinating branch of AI that enables people and machines to communicate with one another in everyday language. The way you give voice commands to Siri or receive translations on Google are both examples of NLP in action, as both demonstrate software understanding and responding to human language.

How does NLP relate to AI?

Most people are not programmers or advanced software users, which is one of the reasons why natural language processing is so useful. Software can be difficult to learn, and NLP enables you to use even complex software without needing to become an expert. In fact, NLP enables you to simply describe what you want to an AI in the same way you might explain it to a friend.

The term AI encompasses a wide range of technologies, but the NLP branch concentrates on the challenges of human language. To be genuinely useful, NLP solutions must do more than simply catch the words you say. NLP systems must understand the context and intent behind your words. To make that possible, NLP developers use other AI technologies such as machine learning and deep learning.

Why is natural language processing important?

Natural language processing enables you to use technology that might otherwise be difficult to utilise. It also enables computers to understand text and speech in a way that they could not before NLP. Here are some of the benefits of NLP:

NLP helps people to be more productive

At work, natural language processing can increase your efficiency by enabling you to use automation for repetitive or time-consuming tasks. For example, a customer service department might use NLP-powered chatbots to deal with routine customer enquiries. Alternatively, an accounts department might use NLP-enabled systems to extract key information from invoices and receipts and use it to populate a database or a spreadsheet.

As additional benefits, automating data entry and processing reduces the likelihood of human error and speeds up workflows. When a system can understand human language well enough to take over basic tasks, it increases productivity by allowing users to focus on higher-value tasks.

NLP helps improve customer experiences

When you try to contact a business by phone but cannot get past a confusing phone menu, you are experiencing a poorly configured interactive voice response system. However, a well-trained NLP chatbot can provide customers with a more intuitive experience. An e-commerce site with an NLP-enabled system, for example, can analyse which products a customer has been browsing on the site. By understanding which products are of interest to a customer, the system can then suggest products that the customer is likely to want.

NLP-powered customer service chatbots enhance experiences by answering questions or resolving problems promptly. These chatbots can be designed to have a complete history of a customer’s interactions and recognise problems the customer has had in the past.

NLP creates new insights

If you have ever sent an email to a business to ask a question, offer advice or register a complaint, it can seem as though no one even reads it. And that might be the case, because few businesses have the time to read every piece of customer feedback they receive. But NLP-enabled AI does have the time. It can sift through enormous datasets such as customer conversations on websites. It can then provide businesses with a reliable summary of those discussions so that the business can rectify the issue.

How does natural language processing work?

Here’s how NLP makes conversations between humans and machines possible. The following steps apply to language in the form of text. NLP for spoken language is slightly different but follows the same general principles.

Let’s use a sample sentence to show how the process works:

“I enjoy demonstrating how natural language processing works.”

The algorithm begins with text pre-processing.

Text pre-processing

Text preprocessing refers to simplifying the text that people create to make it easier for NLP algorithms to process human language.

• Tokenisation

Tokenisation is the process of breaking down the words and punctuation in a sentence into tokens. Tokenisation is important because it is more efficient for NLP algorithms to process tokens than text when performing tasks such as indexing and search. The sample sentence contains eight words: “I like demonstrating how natural language processing works.” It also has a full stop, so you get eight tokens by counting the words and one by counting the full stop, for a total of nine tokens.

• Converting to lower case

Lowercasing is the process of converting all tokens into lower-case tokens to make the dataset simpler. In the sample sentence, one of the tokens was “I”. To avoid ambiguity and increase efficiency, the lowercasing step converts those uppercase tokens into “i”. The rules for using lower case become more complicated in other parts of the process.

• Removing stopwords

Another way of simplifying text for natural language processing is to remove words that do not have significant meaning, which are called stop words. In the sample sentence, the words “i” and “how” would typically be designated as stop words. When an algorithm removes them, you have seven tokens left, which are “like”, “demonstrating”, “natural”, “language”, “processing”, “works”, and “.”.

• Stemming and lemmatisation

Even with the seven tokens remaining there’s room for further simplification. One method, stemming, is cutting a word down to its base or root form. The token “demonstrating” is a word that builds on the stem “demonstr” just as “natural” builds on “natur”, so the model replaces the original tokens with the stems “demonstr” and “natur”.

A word can have different meanings in different contexts, and lemmatisation is the process of determining the correct meaning in a particular context. The sample sentence includes “like”, which can mean “enjoy” or “similar to”. In this case, lemmatisation would result in assigning the context where “like” means “enjoy”.

Text representation

The next step is text representation, which involves converting words into a numerical format that a machine can process. In addition to numbers, text can be converted into vectors or embeddings, which are more complex formats that provide information such as context.

• Bag of words (BoW)

Computers are excellent at counting, and the BoW measurement counts how many times a word appears in a document. If the words team, game, and score appear frequently in a document, for example, the context is more likely to be sport. The sample sentence contains only one occurrence of each word. The BoW representation would show that each word only occurs once like this:

{“i”: 1, “like”: 1, “demonstrating”: 1, “how”: 1, “natural”: 1, “language”: 1, “processing”: 1, “works”: 1}

• TF-IDF (term frequency–inverse document frequency)

TF-IDF uses a formula based on how often a word appears in an overall dataset made up of many documents to determine a word’s importance. The more frequently a word appears, the lower its TF-IDF weight, and the less important it is in an individual document. Words such as “the” and “a” appear quite frequently, and so are less important. You would represent the weights of the sample sentence in a form rather like this, showing that common words have lower weights and uncommon words have higher weights:

{“i”: 0.1, “like”: 0.1, “demonstrating”: 0.3, “how”: 0.1, “natural”: 0.2, “language”: 0.2, “processing”: 0.5, “works”: 0.1}

Text analysis

Text analysis is the point at which an NLP algorithm extracts meaning from text. It’s how an algorithm can produce appropriate responses to the user's enquiry.

• Named entity recognition (NER)

Having carried out some quantitative analysis, NLP algorithms then look for words that are recognisable as naming something. The word apple refers to a type of fruit, but the word Apple refers to a specific company, and an NLP algorithm needs a way of recognising the difference. The sample sentence, “I like demonstrating how natural language processing works,” includes the words “natural language processing”, which people might recognise as naming an AI technology. This is represented as follows:

NER Output: [(“natural language processing”, “Technology”)]

• Sentiment analysis

Some datasets, such as a catalogue of lawnmower parts, may not possess a significant emotional tone. However, a film review might have a strong emotional tone. If a dataset does have a tone, sentiment analysis is the stage in natural language processing that captures it. For the sample sentence, the sentiment analysis might look like this:

Sentiment Output: Positive

Syntax analysis

Every sentence has a grammatical structure. Syntax parsing is the process of analysing that structure to find nouns, verbs, subjects, etc. This is important because different languages use different syntax, so syntax parsing is vital to machine translation. For the sample sentence, syntax parsing might produce this result:

Syntax Tree: (ROOT (S (NP (PRP I)) (VP (VBP like) (S (VP (VBG demonstrating) (SBAR (WHADVP (WRB how)) (S (NP (NNP Natural) (NNP Language) (NNP Processing)) (VP (VBZ works)))))))))

Depending on the specific algorithm, there will typically be additional steps. The result is a conversation in which the machine appears to understand your words and intentions and responds to you in natural language.

What are examples of natural language processing tasks?

Controlling a drone with voice commands

Even the most powerful technology has limited value if you do not know how to use it. Natural language processing makes technology more accessible. It reduces the need for specialised technical knowledge to gain the benefits of advanced software or hardware. Users can interact with NLP-enabled systems through natural conversation instead of relying on complex commands, coding, or physical controls.

For example, a remote-control application for small drones can allow you to simply tell a drone to do something like perform a flip in the air, without the need to learn the intricate controls you would otherwise need to use. The ability to use simple voice commands enables more people to use technology.

Gaining better insights for brand management

People express their thoughts and preferences every day, and businesses have access to much of that data. Businesses already use data such as product sales to understand certain aspects of customer behaviour. With natural language processing solutions, computers can also turn social media conversations and online customer reviews into actionable information.

The insights from NLP are a different kind of information from traditional sales analytics. Businesses use sales figures for operational insights such as forecasting or resource management, but NLP-based analysis can be more powerful in brand management or enhancing customer experiences.

Preventing information overload

If you have an email address that you use for your work, information overload can seem inevitable. The average employee receives more than 120 emails a day, so it is not surprising that about 60% of employees simply ignore internal company emails. However, if your email application has NLP capabilities, it can help you avoid information overload. It can filter, categorise, and prioritise emails so that the most important ones receive the attention they need.

Similarly, NLP systems integrated into collaboration software can transcribe and summarise meetings. They can even recognise and capture the key points made during a meeting and report on assigned action items. This type of NLP-enabled automation helps individuals save time and increases overall organisational efficiency.

Natural language processing use cases

NLP is transforming industries. NLP applications improve employee efficiency, enhance customer experiences, and enable strategic decision-making, spanning a wide range of sectors. Below are some notable use cases for NLP across different industries.

Natural language processing approaches

Most approaches to natural language processing fall into one of two broad categories. They either take a rules-based approach or a machine learning–based approach.

Rules-based NLP

This approach seeks to identify a set of linguistic rules that a computer can follow to reliably understand and generate human language. As a result, it is heavily oriented towards linguistic aspects of computational linguistics. Rules-based approaches can be highly effective if you are using datasets where the language is both controlled and predictable, such as legal documents or technical manuals.

Machine learning–based NLP

A machine learning–based approach to NLP employs a variety of statistical methods and algorithmic techniques. Instead of establishing the rules beforehand, the aim is to enable a computer to learn how to communicate based on a vast dataset. The idea is that once a computer has processed enough examples of human language, the computer will identify the patterns that make for good human language. If you have sufficiently large datasets, machine learning–based approaches to NLP can be quite flexible and remarkably effective.

A brief history of natural language processing

Natural language processing as a branch of AI began to develop in the 1940s. In the 1980s and 1990s, computing solutions became more powerful and machine learning began to mature. More recently, the rise of deep learning, neural networks, and various forms of generative AI has completely transformed natural language processing.

Events in the evolution of NLP

• 1940s | The first NLP developments, including the Weaver Memorandum on machine translation, emerge
• 1950s | “Universal grammar” introduced; Turing test proposed; deep learning emerges
• 1960s | Chatbots such as ELIZA emerge and linguistic theories underpinning NLP evolve
• 1970s | Case grammar theory and semantic networks emerge; rule-based systems mature and chatbots proliferate
• 1980s | Early machine learning algorithms are developed; speech recognition technologies evolve; computational linguistics expands and RNN theory emerges
• 1990s | Statistical models and LSTMs emerge and machine translation gains momentum
• 2000s | Machine learning evolves rapidly and neural language models develop; vast amounts of spoken and written data become accessible
• 2010s | A substantial rise in available data and computing power enables deep learning to develop rapidly
• 2020s | Large language models, pre-trained language models, and transformers gain prominence

Technology overview of natural language processing

Natural language processing is an umbrella term that covers a range of technologies and techniques that make it possible for machines to understand and produce human language. Every technology that enables NLP falls under one of those two capabilities.

NLP subcategories

Natural language processing encompasses a range of technologies and techniques. But the primary purpose of NLP is to enable machines to understand and produce human language. These two capabilities are the main components of natural language processing.

• Natural language understanding (NLU): Natural language understanding focuses on comprehending and interpreting human language. To do that, NLU systems must be able to parse syntax, analyse semantics, and understand the way context affects meaning in human language. This might take forms such as understanding spoken questions or carrying out tasks based on spoken instructions.
• Natural language generation (NLG): Natural language generation focuses on producing human-like text or speech. To do that, NLG systems must be able to turn unstructured data into natural-sounding language. This might mean summarising information or even holding conversations.

The role of machine learning in NLP

What could a computer do if it were able to teach itself new skills? That is what machine learning is. Machine learning is when computers learn how to carry out tasks on their own without any specific instructions.

For NLP, machine learning takes the form of creating models that enable both natural language understanding and natural language generation. It uses techniques including supervised learning, which refers to training models on data that has labels, and unsupervised learning, which is training models on data that does not have labels.

The role of deep learning in NLP

Deep learning is a particular form of machine learning. It uses neural networks that have multiple layers, which is why “deep” is in the name. The “learning” refers to the use of algorithms that identify and then model complex patterns in data sets. Deep learning is important in NLP because it has significantly improved NLP in certain tasks. These include translating between languages, analysing the sentiment in a dataset, and generating text.

How neural networks enhance NLP

Neural networks are based on the idea of using the human brain as a model for how to process data. Neural networks enable NLP systems to be highly accurate in both understanding and generating human language. Neural networks can have various architectures and are key to enabling applications such as a virtual assistant, chatbot, or automated text analysis.

Computational linguistics and NLP

Computational linguistics is the field of study that combines computer science and linguistics to focus on natural language processing. It establishes a theoretical foundation for enabling computers to understand human language.

• Syntax

Studies the structure of sentences and the rules that make sentences grammatical or ungrammatical.

British English syntax: “The cat sits on the mat.”

Incorrect English syntax: “Cat the on mat sits.”

• Semantics

Studies meaning in language, including how words and phrases represent objects, actions and ideas.

Sentence: “The cat is on the mat.”

Semantic interpretation: The meaning is that there is a cat physically situated on top of a mat.

• Pragmatics

Studies how context influences the interpretation of language.

Sentence: “Could you pass the salt?”

Pragmatic interpretation: Although it’s a question about ability, the context shows that you should understand it is a polite request for someone to pass the salt.

Computational linguistics is important because it connects the dots between linguistic theory and real-world applications of NLP.

FAQ