TF-IDF¶
Importing libraries¶
from nltk.corpus import stopwords
from collections import Counter
import numpy as np
import pandas as pd
import math
Using numpy arrays for faster access and pandas to make use of the dataframe class.
Loading the data¶
def load_data(path):
dataframe = pd.read_csv(path)
return dataframe
PATH = 'C:\\Users\\maxel\\OneDrive\\Search_Engine\\Version_1\\processed_TED_Talks.csv'
df_info = load_data(PATH)
N = len(df_info)
processed_exposition = np.array([np.array(row['exposition'][1:].split(',')) for i, row in df_info.iterrows()])
processed_transcript = np.array([np.array(row['transcript'][1:].split(',')) for i, row in df_info.iterrows()])
Convert the processed expostition and transcript into a numpy array for a faster speed.
Calculating DF for both the exposition and and the transcript¶
DF - document frequency - how many documents the word appears in.
DF = {}
for i in range(N):
tokens = processed_exposition[i]
for w in tokens:
try:
DF[w].add(i)
except:
DF[w] = {i} # Set type
tokens = processed_transcript[i]
for w in tokens:
try:
DF[w].add(i)
except:
DF[w] = {i} # Set type
for i in DF:
DF[i] = len(DF[i]) # Get the number of documents the token appeared in
total_vocab_size = len(DF)
total_vocab_size
total_vocab = list(DF.keys())
We need a function that will return the document frequency of a word
def get_DF(word):
try:
return DF[word]
except:
return 0
Calculating TF-IDF¶
Where:
- tf is the term frequency
- f is the frequency of a given word
- d is the given document
The term frequency is given by the equation above. The choice can be made as to which equation is used. The first equation is simply a fraction of the words that is the token. The second gives a logarithmic way of describing the frequency of the words. We shall be using the logarithmic equation.
Where:
- idf is the inverse document frequency
- N is the number of documents
- n is the document frequency of the given token
There is also choice in the way that inverse document frequency is calcuated using the equations above. Jusitification for the use of this have included attempts to connect it to Zipf's law https://en.wikipedia.org/wiki/Zipf%27s_law, information theory or a probability. See more here https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Justification_of_idf.
We shall be using the smooth idf equation. The use of the plus one to the denominator is so that an undefined error does not occur. 1 will also be added to N to ensure that we don't have to compute the log of 0 (although this is not written in the equation).
We can now make a function which will calculate the tf-idf weights of each of the documens.
We will calculate the TF-IDF weights separately for the exposition and the transcript then merge then according to some given weight. The weight (called alpha) will try best describe how much more useful the exposition is at descibing the talk compared to the transcription per word.
Calculating TF-IDF for transcript¶
tf_idf = {}
for i in range(N):
tokens = processed_transcript[i]
counter = Counter(list(tokens) + list(processed_exposition[i]))
words_count = len(tokens) + len(processed_exposition[i])
for token in np.unique(tokens):
tf = counter[token]/words_count
df = get_DF(token)
idf = 1 + np.log((N+1)/(df+1))
tf_idf[i, token] = tf*idf
Calculate the TF-IDF for the exposition¶
tf_idf_expo = {}
for i in range(N):
tokens = processed_exposition[i]
counter = Counter(list(tokens) + list(processed_transcript[i]))
words_count = len(tokens) + len(processed_transcript[i])
for token in np.unique(tokens):
tf = counter[token]/words_count
df = get_DF(token)
idf = np.log((N+1)/(df+1))
tf_idf_expo[i, token] = tf*idf
tf_idf[(1948,"quantum")]
tf_idf_expo[(1948,"quantum")]
len(tf_idf)
Merging the TF-IDF according to the weights¶
Merge them into one weight matrix according to the alpha weighting
alpha = 0.7
for i in tf_idf:
tf_idf[i] = (1 - alpha) * tf_idf[i]
try:
tf_idf[i] += alpha * tf_idf_expo[i]
except:
pass
Matching score simularity¶
Here, the preprocess import is for a python script which simply contains the preprocessing functions written earlier.
from preprocess import preprocess_query
Matching score is simply how close the search query is in terms of its tfidf weights. Think of it as how close two points are in 2d or 3d space, you may struggle imagining a 40 thousand dimension space. The program below uses Manhatten distance (the blocky distance) where rather than drawing a straight line between the points, lines are drawn parallel to the axis and the sum of these distances are used. You can think of this as going along the edges of a cube rather than cutting throught the middle to get between opposite corners.
def matching_score(k, query):
preprocessed_query = preprocess_query(query)
tokens = preprocessed_query
print("Matching Score")
print("Search Query:", query, '\n')
query_weights = {}
for key in tf_idf:
if key[1] in tokens:
try:
query_weights[key[0]] += tf_idf[key]
except:
query_weights[key[0]] = tf_idf[key]
query_weights = sorted(query_weights.items(), key=lambda x: x[1], reverse=True)
for i in query_weights[:k]:
print(i[0], df_info['headline'][i[0]])
matching_score(10, "Quantum biology")
These results seem to be giving the right answers. But we can improve this even more.
TF-IDF Cosine Similarity Ranking¶
The equation above can be used to find the angle between two vectors. This can be used to find the angle between the search query and the document vectors. From this we can deduce that the smaller the angle the better the query matches the document. Cosine similarity is used when the magnitude of the vector doesn't matter. This is so since we are don't mind how long the talks are.
def cosine_sim(a, b):
cos_sim = np.dot(a, b)/(np.linalg.norm(a)*np.linalg.norm(b))
return cos_sim
D = np.zeros((N, total_vocab_size))
for i in tf_idf:
try:
ind = total_vocab.index(i[1])
D[i[0]][ind] = tf_idf[i]
except:
pass
def gen_vector(tokens):
Q = np.zeros((len(total_vocab)))
counter = Counter(tokens)
words_count = len(tokens)
query_weights = {}
for token in np.unique(tokens):
tf = counter[token]/words_count
df = get_DF(token)
idf = math.log((N+1)/(df+1))
try:
ind = total_vocab.index(token)
Q[ind] = tf*idf
except:
pass
return Q
def cosine_similarity(k, query):
print("Cosine Similarity")
preprocessed_query = preprocess_query(query)
tokens = preprocessed_query
print("Search Query:", query, '\n')
d_cosines = []
query_vector = gen_vector(tokens)
for d in D:
d_cosines.append(cosine_sim(query_vector, d))
out = np.array(d_cosines).argsort()[-k:][::-1]
for i in out[:k]:
print(i, df_info['headline'][i])
Q = cosine_similarity(10, "Groundwater")
We don't want to be computing the TF-IDF matrix every time we want to search something so we will save it to a file (although this process itself does take some time to load up again). We will use the npy file extension which has faster read times than a plain text or csv file.
# Save to a file
np.save('tf_idf_dict.npy', tf_idf)
read_tf_idf = np.load('tf_idf_dict.npy', allow_pickle = True).item()