Introduction
The first application written with Clusternet was an example to produce weights for terms in a corpus of ascii books. The example is developed using 3 steps to transform the results in separate MapReduce Jobs. This example can actually be run in any Hadoop cluster.
TF-IDF Algorithm
The term frequency, inverted document frequency is a well-known information retrieval algorithm for giving weights for terms of documents. It is important to note that the algorithm must not consider stop words, and for this reason, we will be using filtering the terms in the input for the algorithm with the Google Stop words. The algorithm is described as follows:
weight(t,d) = tf * idf tf = # terms t in (d) / total # of terms in (d) = tf