Power of Transfer Learning in NLP: Build a Text Classification Model Using BERT

Rate this content
Bookmark

The domain of Natural Language Processing have seen a tremendous amount of research and innovation in the past couple of years to tackle the problem of implementing high quality machine learning and AI solutions using natural text. Text Classification is one such area that is extremely important in all sectors like finance, media, product development, etc. Building up a text classification system from scratch for every use case can be challenging in terms of cost as well as resources, considering there is a good amount of dataset to begin training with.


Here comes the concept of transfer learning. Using some of the models that has been pre-trained on terates of data and fine-tuning it based on the problem at hand is the new way to efficiently implement machine learning solutions without spending months on data cleaning pipeline.


This talk with highlight ways of implementing the newly launched BERT and fine tuning the base model to build an efficient text classifying model. Basic understanding of python is desirable.

This talk has been presented at ML conf EU 2020, check out the latest edition of this Tech Conference.

FAQ

Transfer learning is a machine learning technique where a model trained on one task is repurposed for a second, related task. This method leverages the knowledge gained from the first task to improve performance on the second task.

The presenter is Jayita Pudukonda, a Senior Data Scientist at Intelliint US Inc., based in New York City.

The main challenges in NLP include handling ambiguity, synonymity, coreference, and the syntactic rules of the English language. These complexities make it difficult for a computer program to understand and process human language accurately.

BERT (Bidirectional Encoder Representations from Transformers) is the first deeply bidirectional model trained in an unsupervised way on plain text. Its significance lies in its ability to understand the context of words in a sentence by looking at both the left and right sides simultaneously, making it highly effective for various NLP tasks.

Common applications of NLP include machine translation, chatbots, text-to-audio and audio-to-text conversion, building knowledge trees, intent classification, natural text generation, topic modeling, clustering, and text classification such as sentiment analysis.

Data preprocessing in NLP is crucial because it ensures that the data is clean, grammatically correct, and semantically meaningful. This step includes handling extra spaces, tokenization, spell check, contraction mapping, stemming, lemmatization, and removing stopwords. Proper preprocessing leads to better model performance.

Transfer learning is useful because it helps when there is a scarcity of labeled data, which can be expensive and time-consuming to create. It allows the use of pre-trained models on large datasets to be repurposed for smaller, related tasks, thereby saving resources and improving efficiency.

Word embeddings are feature vector representations of words in a training corpus. They capture the context of a word in a document, its semantic and syntactic relationship with other words, and its meaning. This helps in tasks like clustering similar words and understanding word analogies.

BERT handles the context of words by being bidirectional, meaning it looks at both the left and right sides of a word simultaneously. This allows BERT to understand the meaning of a word based on its surrounding words, improving the accuracy of tasks like sentence prediction and question answering.

BERT is trained on two main tasks: masked language modeling and next sentence prediction. Masked language modeling involves predicting missing words in a sentence, which helps in understanding the language context. Next sentence prediction involves determining if one sentence logically follows another, which helps in understanding the relationship between sentences.

Jayeeta Putatunda
Jayeeta Putatunda
35 min
02 Jul, 2021

Comments

Sign in or register to post your comment.

Video Summary and Transcription

This video talk delves into the intricacies of Transfer Learning with BERT in Natural Language Processing (NLP). BERT is a deeply bidirectional model that learns from plain text in an unsupervised manner, making it adept at understanding word context. The talk highlights its applications in tasks like text generation, machine translation, chatbots, and topic modeling. It also covers essential text preprocessing techniques, such as tokenization, stemming, and lemmatization, which are crucial for effective NLP. Transfer learning is emphasized as a way to leverage pre-trained models for new tasks, particularly when labeled data is scarce. The video also discusses the importance of clean data and understanding baseline NLP modeling. BERT's ability to handle context-based words and synonyms is explored, along with its limitations in strict classification labels or conversational AI. The talk concludes with practical advice on understanding transformers and BERT by focusing on the problems they solve and gradually exploring their implementation in business scenarios.

1. Introduction to Transfer Learning and NLP

Short description:

Hello, everyone! Welcome to this session of Transfer Learning with Burt. Today, we'll learn about Transfer Learning and NLP. NLP is a subfield of linguistics and AI. We'll discuss its challenges and how it can be used for text classification.

Hello, everyone, and welcome to this session of Transfer Learning with Burt. With me, I'm very excited that you all are here and join me in the session. So let's see what we learn about Transfer Learning and Burt today.

Before we start, a quick introduction about me. Hi, my name is Jayita Pudukonda. I work as a Senior Data Scientist at Intelliint US Inc. We are based out of New York City. And to give you an overview, Intelliint, we are a customer driven professional services company. We specialize in tailor made software and tech solutions. We work with a lot of data analytics and AI based companies and build some of our tools and softwares around those states. So you can connect me on Twitter and LinkedIn if you have any questions about the session later on and we can discuss about that.

Great. So for today's session, we're going to talk about like 20 minutes for the session and then we have some Q&A. I'd also have some code that I can share over my GitHub. So do reach out if you want to take a look at those later after the session. So now the exciting part, I say this as an NLP is hard and you would say why. Now take a look at this picture. What's the first inference that comes to your mind? I know like human beings can make like, you know, great connections and references. This is a very trivial task for us. But if you think from a perspective of a computer program, this is a very daunting challenge to kind of understand this complexity of English language. So here in the picture it says, I am a huge Metal fan. So us like humans, we would know that, okay, this is the Metal fan is like, you know, a personifying electric electric component and says that, okay, I'm a metal fan, but it can also refer to that you are a huge metal, you know, music fan. So how do you, how the computer differentiates between these two meaning of the same terminology. So there's ambiguity, there's synonymity, there's coreference. And of course the syntactic rules of English literature, that kind of hampers, or makes it more daunting task for a computer program. So for today's agenda, we'll just quickly go over what's NLP, how, where it is used, how transfer learning can be used. And we look at a simple case of utilizing part for a text classification model. So let's jump right into it. For NLP, I feel that this image very well describes it. It's a subfield of linguistics, artificial intelligence.

2. Introduction to NLP Applications and Techniques

Short description:

There's computer science and also in information engineering. NLP has had an exponential growth in the last two years. It's used in machine translation, chatbots, intent classification, text generation, topic modeling, clustering, and text classification. To work with NLP, we need to handle extra spaces, tokenize text correctly, perform spell check, and use contraction mapping.

There's computer science and also in information engineering. So basically helps all the machines to understand and, you know, communicate back and forth with human beings in a free-flowing speech without losing context or references. And if you see NLP has had an exponential growth in the last, I would say two, two, years, like the huge models that Google, OpenAI and Vidya has been working on and releasing has like humongous amount of parameters. So just in May, 2020, OpenAI came up with a 175 billion parameter model, which is, you can understand how much text has gone into the processing of that model and how much work that it can do with so much accuracy.

So where is NLP used? I know that most of you are definitely familiar with it, but I just wanted to give a quick description of where I feel that it's being used the most. And I've worked hands-on on those areas. So definitely machine translation, like text to audio, audio to text, there's chatbot, building of knowledge trees, intent classification, there's also, you know, natural text generation. I'm sure when you use Gmail, you have seen that the prompt that keeps coming up when you're writing an email, says that, oh, that this two next words, I think would be good for the sentence that you're trying to complete. That's like, you know, text completion prompts that you get. It's also used a lot in topic modeling, clustering, understanding with the context of the whole or what kind of insights can be generated from huge amount of text data. And also text classification, which is like, you know, do you want to do a sentiment analysis? Like how do you understand what the general idea, say Yelp reviews or Amazon product reviews. So those have a lot of implications and good applications by NLP problems.

So how do we do it? I know that, these can sound a lot, underlying, but this is very important that we do it in all kinds of business cases, or business problems that we're trying to solve using NLP, so just a quick idea. We need to handle when we have, say, all our texts, make sure that we handle extra spaces. Then we also need to look after how we tokenize our text. So, tokenization just using, by spacing is the very traditional norm, but we also need to take care of use cases, like say, the whole world as United States of America, if we tokenize it just by the spaces, sometimes it can happen that that doesn't make sense in the context that we're trying to work through it, right? So then we need to keep that whole United States of America as a whole phrase, rather than tokenize it by a sentence, so then the, so that the information extraction works much better than, than otherwise if it's tokenized by word. The next step would be spell check. So I'm referring here directly to a very, like a great tool that Peter Norvig from Google created. It's a spell checker. Basically the idea is to, you know, kind of compare the distance between multiple words. And see that, okay. Does this spelling make sense and how close it is to a similar spelling or a similar word meaning that's there in the, you know, the vector space of the whole NLP corpus. So you can see here that when I pass the wrong spelling with a single L, the return value would be a spelling with a double L and also for corrected with the K not with the K and the final value would be corrected with a C. So this can actually help in making sure that our data is clean. And like they say that in NLP it's like garbage in and garbage out. Sorry about that garbage in and garbage out. So we need to make sure that the clean data makes sense with a grammatical sense. Syntactic sense and also semantic meaningfulness. The next step would be contraction mapping. This might seem that why we need to do it.

QnA