[NLP] Lemmatization (표제어 추출), Stopwords(불용어)

Lemmatization란?

단어의 다양한 형태(굴절, 파생)로부터 의미적으로 정확하고 문법적으로 올바른 기본 사전형 단어, 즉 표제어(Lemma)를 찾아내는 것이다.

running,ran,runs -> run(동사) / better -> good(형용사) 등 변환이 가능하다.

 

words=["eating","eats","eaten","writing","writes","programming","programs","history","finally","finalized"]
for word in words:
    print(word+"---->"+lemmatizer.lemmatize(word,pos='v'))

 

결과:

eating---->eat
eats---->eat
eaten---->eat
writing---->write
writes---->write
programming---->program
programs---->program
history---->history
finally---->finally
finalized---->finalize

 

Stopwords란?

텍스트에서 자주 등장하지만 분석에 중요한 의미를 가지지 않는 단어들을 제거하여 데이터의 양을 줄이고 핵심 단어에 집중하기 위한 것이다.

예를들어 the, a, is, are, 은, 는, 이, 가 <- 와 같은 단어들을 제거하는 것이다.

분석에 꼭 필요한 데이터만 남겨놓는 것.

 

예를들어 아래 corpus로 stopwords를 확인해보자.

paragraph = """I have three visions for India. In 3000 years of our history, people from all over 
               the world have come and invaded us, captured our lands, conquered our minds. 
               From Alexander onwards, the Greeks, the Turks, the Moguls, the Portuguese, the British,
               the French, the Dutch, all of them came and looted us, took over what was ours. 
               Yet we have not done this to any other nation. We have not conquered anyone. 
               We have not grabbed their land, their culture, 
               their history and tried to enforce our way of life on them. 
               Why? Because we respect the freedom of others.That is why my 
               first vision is that of freedom. I believe that India got its first vision of 
               this in 1857, when we started the War of Independence. It is this freedom that
               we must protect and nurture and build on. If we are not free, no one will respect us.
               My second vision for India’s development. For fifty years we have been a developing nation.
               It is time we see ourselves as a developed nation. We are among the top 5 nations of the world
               in terms of GDP. We have a 10 percent growth rate in most areas. Our poverty levels are falling.
               Our achievements are being globally recognised today. Yet we lack the self-confidence to
               see ourselves as a developed nation, self-reliant and self-assured. Isn’t this incorrect?
               I have a third vision. India must stand up to the world. Because I believe that unless India 
               stands up to the world, no one will respect us. Only strength respects strength. We must be 
               strong not only as a military power but also as an economic power. Both must go hand-in-hand. 
               My good fortune was to have worked with three great minds. Dr. Vikram Sarabhai of the Dept. of 
               space, Professor Satish Dhawan, who succeeded him and Dr. Brahm Prakash, father of nuclear material.
               I was lucky to have worked with all three of them closely and consider this the great opportunity of my life. 
               I see four milestones in my career"""

 

from nltk.stem import PorterStemmer

from nltk.corpus import stopwords

import nltk
nltk.download('stopwords') // 불용어 목록 다운로드

stopwords.words('english') // 불용어 목록(영어)

 

결과:

['i',
 'me',
 'my',
 'myself',
 'we',
 'our',
 'ours',
 'ourselves',
 'you',
 "you're",
 "you've",
 "you'll",
 "you'd",
 'your',
 'yours',
 'yourself',
 'yourselves',
 'he',
 'him',
 'his',
 'himself',
 'she',
 "she's",
 'her',
 'hers',
...
 "weren't",
 'won',
 "won't",
 'wouldn',
 "wouldn't"]

 

이제 stopword가 아닌 단어들만 lemmazitation을 진행해보자.

 

from nltk.stem import WordNetLemmatizer
lemmatizer=WordNetLemmatizer()

## stopwords가 아닌 단어들만 lemmazation 진행
for i in range(len(sentences)):
    words=nltk.word_tokenize(sentences[i])
    words=[lemmatizer.lemmatize(word.lower(),pos='v') for word in words if word not in set(stopwords.words('english'))]
    sentences[i]=' '.join(words) // 단어들을 조인하여 최종 문장으로 변환

 

words=[lemmatizer.lemmatize(word.lower(),pos='v') for word in words if word not in set(stopwords.words('english'))]

이 코드를 해석해보면,

1. for word in words -> 단어들에 대해서

2. if word not in set(stopwords.words('english'))] -> stopword(불용어)에 포함되지 않는 단어라면

3. lemmatizer.lemmatize(word.lower(),pos='v') -> 동사(v)로 단어를 lemmatize 진행한다.

 

결과:

['i three visions india .',
 'in 3000 years history , people world come invade us , capture land , conquer mind .',
 'from alexander onwards , greeks , turks , moguls , portuguese , british , french , dutch , come loot us , take .',
 'yet do nation .',
 'we conquer anyone .',
 'we grab land , culture , history try enforce way life .',
 'why ?',
 'because respect freedom others.that first vision freedom .',
 'i believe india get first vision 1857 , start war independence .',
 'it freedom must protect nurture build .',
 'if free , one respect us .',
 'my second vision india ’ development .',
 'for fifty years develop nation .',
 'it time see develop nation .',
 'we among top 5 nations world term gdp .',
 'we 10 percent growth rate areas .',
 'our poverty level fall .',
 'our achievements globally recognise today .',
 'yet lack self-confidence see develop nation , self-reliant self-assured .',
 'isn ’ incorrect ?',
 'i third vision .',
 'india must stand world .',
 'because i believe unless india stand world , one respect us .',
 'only strength respect strength .',
 'we must strong military power also economic power .',
...
 'my good fortune work three great mind .',
 'dr. vikram sarabhai dept .',
 'space , professor satish dhawan , succeed dr. brahm prakash , father nuclear material .',
 'i lucky work three closely consider great opportunity life .',
 'i see four milestones career']

 

stopword가 제거되고 남은 단어들이 lemmatize 되어 문장으로 구성된 것을 알 수 있다.