TF-IDF implementations in python

TF-IDF in Python: Unlocking Document Relevance with Practical Implementations

Explore the theory and practical Python implementations of TF-IDF (Term Frequency-Inverse Document Frequency) for text analysis and information retrieval. Learn how to extract meaningful insights from your text data.

TF-IDF, short for Term Frequency-Inverse Document Frequency, is a statistical measure that evaluates how relevant a word is to a document in a collection of documents. This value increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus, which helps to adjust for the fact that some words appear more frequently in general. It's a cornerstone technique in information retrieval and text mining, widely used for tasks like document ranking, keyword extraction, and text summarization.

Understanding the TF-IDF Formula

The TF-IDF weight is a product of two terms: Term Frequency (TF) and Inverse Document Frequency (IDF). Each component plays a crucial role in determining the overall relevance score.

Term Frequency (TF): This measures how frequently a term appears in a document. There are several ways to calculate TF, but a common approach is the raw count of a term in a document, or the normalized frequency (raw count divided by the total number of terms in the document).

Inverse Document Frequency (IDF): This measures how important a term is across the entire corpus. Words that are common across many documents (like 'the', 'a', 'is') will have a low IDF score, while words that are unique to a few documents will have a higher IDF score. The formula typically involves the logarithm of the total number of documents divided by the number of documents containing the term.

flowchart TD
    A[Document Collection] --> B{Pre-processing}
    B --> C[Tokenization]
    C --> D[Calculate Term Frequency (TF)]
    D --> E[Calculate Inverse Document Frequency (IDF)]
    E --> F[Multiply TF * IDF]
    F --> G[TF-IDF Score for each term in each document]
    G --> H[Applications: Ranking, Keyword Extraction]

Flowchart illustrating the TF-IDF calculation process.

Implementing TF-IDF with NLTK and Scikit-learn

Python offers powerful libraries for implementing TF-IDF. NLTK (Natural Language Toolkit) is excellent for text pre-processing, while Scikit-learn provides a highly optimized TfidfVectorizer that handles the entire process from tokenization to TF-IDF calculation. We'll demonstrate both a manual approach using NLTK for understanding and then the streamlined Scikit-learn method.

import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from collections import Counter
import math

def compute_tf(word_dict, bag_of_words):
    tf_dict = {}
    bow_count = len(bag_of_words)
    for word, count in word_dict.items():
        tf_dict[word] = count / float(bow_count)
    return tf_dict

def compute_idf(documents):
    N = len(documents)
    idf_dict = dict.fromkeys(documents[0].keys(), 0)
    for document in documents:
        for word, val in document.items():
            if val > 0:
                idf_dict[word] += 1
    
    for word, val in idf_dict.items():
        idf_dict[word] = math.log(N / float(val))
    return idf_dict

def compute_tfidf(tf_bow, idfs):
    tfidf = {}
    for word, val in tf_bow.items():
        tfidf[word] = val * idfs[word]
    return tfidf

documents = [
    "The quick brown fox jumps over the lazy dog",
    "Never jump over the lazy dog again",
    "The dog is very lazy"
]

# Pre-processing
stop_words = set(stopwords.words('english'))
tokenized_documents = []
for doc in documents:
    words = word_tokenize(doc.lower())
    filtered_words = [word for word in words if word.isalnum() and word not in stop_words]
    tokenized_documents.append(filtered_words)

# Create word dictionaries for each document
word_dicts = [Counter(doc) for doc in tokenized_documents]

# Compute TF for each document
tf_docs = [compute_tf(word_dict, doc) for word_dict, doc in zip(word_dicts, tokenized_documents)]

# Compute IDF across all documents
all_words = set()
for doc_dict in word_dicts:
    all_words.update(doc_dict.keys())

# Create a unified word dictionary for IDF calculation
idf_input = []
for doc_dict in word_dicts:
    temp_dict = dict.fromkeys(all_words, 0)
    for word, count in doc_dict.items():
        temp_dict[word] = count
    idf_input.append(temp_dict)

idfs = compute_idf(idf_input)

# Compute TF-IDF for each document
tfidf_results = [compute_tfidf(tf_doc, idfs) for tf_doc in tf_docs]

for i, tfidf_doc in enumerate(tfidf_results):
    print(f"\nTF-IDF for Document {i+1}:")
    for word, score in sorted(tfidf_doc.items(), key=lambda item: item[1], reverse=True):
        if score > 0: # Only show words that appear in the document
            print(f"  {word}: {score:.4f}")

Manual TF-IDF calculation using NLTK for tokenization and custom functions.

from sklearn.feature_extraction.text import TfidfVectorizer

documents = [
    "The quick brown fox jumps over the lazy dog",
    "Never jump over the lazy dog again",
    "The dog is very lazy"
]

# Initialize TfidfVectorizer
# stop_words='english' automatically handles common English stop words
# lowercase=True is default
# analyzer='word' is default
vectorizer = TfidfVectorizer(stop_words='english')

# Fit and transform the documents
tfidf_matrix = vectorizer.fit_transform(documents)

# Get feature names (words)
feature_names = vectorizer.get_feature_names_out()

# Print TF-IDF scores for each document
for i, doc_vector in enumerate(tfidf_matrix):
    print(f"\nTF-IDF scores for Document {i+1}:")
    # Get the non-zero TF-IDF values for the current document
    feature_index = doc_vector.nonzero()[1]
    tfidf_scores = zip(feature_index, doc_vector.data)
    
    # Sort by score in descending order
    sorted_scores = sorted(tfidf_scores, key=lambda x: x[1], reverse=True)
    
    for idx, score in sorted_scores:
        print(f"  {feature_names[idx]}: {score:.4f}")

Streamlined TF-IDF calculation using Scikit-learn's TfidfVectorizer.

Applications and Best Practices

TF-IDF is a versatile tool with numerous applications in natural language processing and information retrieval. Some common uses include:

• Search Engine Ranking: Identifying documents most relevant to a user's query.
• Keyword Extraction: Pinpointing the most important terms in a document.
• Text Summarization: Helping to identify key sentences or phrases.
• Document Similarity: Comparing documents based on their TF-IDF vectors.
• Recommendation Systems: Suggesting similar articles or products.

Best Practices:

1. Thorough Pre-processing: Clean your text data rigorously (remove noise, normalize case, handle stop words, stem/lemmatize) to ensure meaningful TF-IDF scores.
2. Corpus Size: TF-IDF performs better with a sufficiently large and diverse corpus. A small corpus might lead to less reliable IDF values.
3. Parameter Tuning: For TfidfVectorizer, experiment with parameters like min_df, max_df, ngram_range, and use_idf to optimize performance for your specific dataset.
4. Sparsity: TF-IDF matrices are often sparse (many zero values). Libraries like Scikit-learn handle this efficiently, but be mindful when performing operations on these matrices.
5. Domain-Specific Stop Words: Consider adding domain-specific stop words if general stop word lists don't adequately filter out common but uninformative terms in your particular field.