Minimal implementations of a couple of classic text analysis tools (TF-IDF and cosine similarity). Everything is done in memory so this library is not suitable for large-scale use. Instead, the goal is to create something simple that can be used to explain or experiment with the techniques, using a small set of documents. For a detailed and interactive explanation, see this Observable notebook.
The term weighting scheme is BM25, as described in this technical report by Stephen Robertson and Karen Spärck Jones.
A basic set of English stopwords is included, and you can specify your own list of stopwords to add. In the interest of keeping this "tiny" (and fast enough to run in the browser) there are some useful things that I didn't implement, most notably:
- phrases (bigrams, trigrams, etc), e.g. "proof of concept"
- stemming or lemmatizing, e.g. reducing "concept" and "concepts" to the same root
I am open to adding either if there's a tiny way to do it!
Note: I'm still actively developing this code (and documentation), and the API is likely to change/evolve up until version 1.0.
;const corpus ="document1" "document2" "document3""This is test document number 1. It is quite a short document.""This is test document 2. It is also quite short, and is a test.""Test document number three is a bit different and is also a tiny bit longer.";// print top terms for document 3console;// result'bit' 19939850399669656'three' 13113595307890855'different' 13113595307890855'tiny' 13113595307890855'longer' 13113595307890855'number' 06556797653945428'also' 06556797653945428'test' 02721316901570901'document' 02721316901570901
For many more usage examples, see this Observable notebook.
Disclaimer: this is an ES6 module and is mostly intended for use in the browser, rather than with Node.js (more background on ES6 modules and Node).
Example with Node v12.6.0 :
node --experimental-modules --es-module-specifier-resolution=node test.js
This is the main class that you will use directly. It takes care of creating a
Document for every text and also manages
Stopwords for the collection. It calculates term frequencies, term weights, and term vectors, and can return results for a given query.
constructor(names, texts, useDefaultStopwords = true, customStopwords = , K1 = 2.0, b = 0.75):
textsare parallel arrays containing the document identifiers and the full texts of each document;
customStopwordsare optional parameters that are passed along to the
Stopwordsinstance (see below);
bare optional tuning parameters for term weighting that are explained in the reference technical report
getTerms(): returns an array containing the unique terms used in the corpus (excluding stopwords)
getCollectionFrequency(term): returns the number of documents in the collection that contain the given term
getDocument(identifier): returns the
Documentobject for the given
getDocumentIdentifiers(): returns an array of all identifiers in the corpus
getCommonTerms(identifier1, identifier2, maxTerms = 10): returns an array of the terms that the documents with these two identifiers have in common; each array entry is a pair of a term and a score, and the array is sorted in descending order by the score, with a maximum length of
maxTerms(which is optional and defaults to 10)
getCollectionFrequencyWeight(term): returns the collection frequency weight (or inverse document frequency) for the given
getDocumentVector(identifier): returns a
Mapfrom terms to their corresponding combined (TF-IDF) weights, for the document with the given
identifier(this is used by the
getTopTermsForDocument(identifier, maxTerms = 30): returns an array containing the terms with the highest combined (TF-IDF) weights for the document with the given
identifier; each array entry is a pair of a term and a weight, and the array is sorted in descending order by the weight, with a maximum length of
maxTerms(which is optional and defaults to 30)
getResultsForQuery(query): returns an array representing the highest scoring documents for the given
query; each array entry is a pair of a document identifier and a score, and the array is sorted in descending order by the score. The score for a document is the total combined weight of each query term that appears in the document.
getStopwords(): returns the
Stopwordsinstance that is being used by this corpus (for inspection or debugging)
The other methods in the class (whose names start with
_calculate) are intended for internal use.
This is used by the
Corpus class for each of the given texts. It is independent of any stopword list or term weights (which are managed at the corpus level) and only maintains the document-level term frequencies. Terms can contain only letters or numbers; they are filtered out if they contain only 1 character or if they start with a number.
constructor(text): expects a single one of the texts originally passed into
getTermFrequency(term): returns a count of how often the given term appears in this document
getText(): returns a string containing the full text of this document (e.g. for display)
getLength(): returns the total number of terms in the document (including stopwords)
getUniqueTerms(): returns an array of the unique terms that appear in the document (including stopwords)
The other method,
_calculateTermFrequencies, is intended for internal use.
constructor(useDefaultStopwords = true, customStopwords = ):
customStopwordsare optional parameters, as specified in the constructor for
Corpus, which control whether the default stopword list should be used, and to specify any custom stopwords. If the default stopword list is to be used, any custom stopwords are added to that list; if not, the custom stopwords are used instead of the default list.
trueif the current stopword list contains the given
getStopwordList(): returns an array of the stopword list currently in use (for inspection or debugging)
An optional addition: once you have a
Corpus you can use
Similarity to calculate the pairwise similarity between the documents in the corpus, resulting in a distance matrix (distance = 1 - similarity).
constructor(corpus): expects an instance of
getDistanceMatrix(): returns an object with properties
identifiers(an array of identifiers for the items in the matrix) and
matrix(an array of arrays, where the values represent distances between items; distance is 1.0 - similarity, so 0 = identical)
There is also a static method,
cosineSimilarity(vector1, vector2), which calculates the similarity between a pair of documents (as the cosine of the angle between their vectors). Each vector is represented as an ES6
Map from each term to its combined (TF-IDF) weight for the corresponding document. It is currently only used to calculate individual entries in the distance matrix. The other method,
_calculateDistanceMatrix, is intended for internal use.