# Decode School

## Decode School - The School for Problem Solvers

Welcome to my blog. Subscribe and get my latest blog post in your inbox.

# Centroid based Text summarization in Python

June 2, 2020, 2:29 p.m. | Text Mining

### Introduction

It is an Extractive summarization, which extracts important words from document(s) to form a summary. Centroid-based summarization works as identifying the most central sentences in multiple documents that give the necessary and sufficient amount of information related to the main theme of document(s). A common way of identifying the central sentences is to represent the sentences in vector space.

Centrality of sentence is always defined in terms of centrality of word. TF X IDF scores are used to measure the centrality of words. Here, words that have TF X IDF scores above a predefined cosine threshold are centroid of cluster.

In centroid based summarization, the sentences which have more words from centroid of cluster are considered as central sentences. Finally, those sentences are produced as summary of multiple documents.

Centroid based summarization is introduced by Radev, Blair-Goldensohn, and Zhang.

### Implementation of Text Summarization in Python

```

```
from os import listdir
import string
import math

"""Method to calculate Inverse Document Frequency Score"""
def calculate_idf(word):
files = [f for f in listdir("E:\Works\Products\doc") ]  #Specify the directory where the documents located
count,wcount=2,1
for file1 in files:
file=open("E:\Works\Products\doc\\" +file1,'r')     #Specify the directory where the documents located
if(word in page):
wcount+=1
count+=1
idf=count/wcount

return math.log(idf,10)

"""Method to calculate Centroid Score of sentences"""
def calculate_centroid(sentences):

""""Compute tf X idf score for each word"""
tfidf=dict()
for sentence in sentences:
words=sentence.split()
for word in words:
if word in tfidf:
tfidf[word]+=calculate_idf(word)
else:
tfidf[word]=calculate_idf(word)

"""Construct the centroid of Cluster
By taking the words that are above the threshold"""

centroid=dict()
threshold=0.7
for word in tfidf:
if(tfidf[word]>threshold):
centroid[word]=tfidf[word]
else:
centroid[word]=0

"""Compute the Score for Sentences"""
senctence_score=list()
counter=0
for sentence in sentences:
senctence_score.append(0)
words=sentence.split()
for word in words:
senctence_score[counter]+=centroid[word]

counter=counter+1
return senctence_score

"""Splitting Documents as sentences"""
files = [f for f in listdir("E:\Works\Products\doc") ]
page=""
for file1 in files:
file=open("E:\Works\Products\doc\\" +file1,'r')
file.close()
sentences=page.split(".")
senctence_score=calculate_centroid(sentences)

"""Printing Sentences which has more central words"""
for i in range(len(sentences)):
if(senctence_score[i]>15):
print(sentences[i])
```
```

### Implementation Details

I have implemented the Centroid based text summarization in python, the above-mentioned code is purely for the text summarization which contains method for calculating IDF score of every word and centroid score of every sentence. To understand Python basics refer Python Programming Examples

Centroid score of every sentence is calculated by TF X IDF score of words in the sentence. To easily understand the algorithm of centroid based text summarization, preprocessing techniques like stop word removal and stemming are not given in the above implementation