In this guide you’ll learn when to use TF-IDF instead of semantic search, how to configure and query it, and understand its limitations.
When to Use TF-IDF
Basic Usage
src/tfidf-basic.ts
Key Differences from VectoriaDB
Important: Reindexing
TF-IDF requires reindexing after document changes to update IDF (Inverse Document Frequency) values:src/tfidf-reindex.ts
Configuration Options
src/tfidf-config.ts
Search Options
src/tfidf-search.ts
TF-IDF Algorithm
TF-IDF (Term Frequency-Inverse Document Frequency) works by:- Term Frequency (TF): How often a term appears in a document
- Inverse Document Frequency (IDF): How rare a term is across all documents
- TF-IDF Score: TF x IDF - terms that are frequent in a document but rare overall get high scores
- Common words like “the”, “is”, “a” get low scores (low IDF)
- Unique terms specific to a document get high scores
- The query is matched against TF-IDF vectors using cosine similarity
Languages and tokenization
Documents and queries go through the same tokenizer, so a query term matches only a term from a document. The default tokenizer,defaultTokenizer, gives lowercase terms in any script:
- Letters, combining marks, digits and
_make up terms; everything else separates them. Terms of one character are left out. - Text is NFKD-normalized: compatibility forms are folded (
fileisfile, full-widthcatsiscats), and marks on Latin and Greek letters are dropped, soVögelmatchesVogelandcafématchescafe. Other marks are kept: Cyrillicй, and Hebrew, Arabic and Devanagari words, stay as written. - Chinese, Japanese, Thai, Lao, Khmer and Myanmar are written without spaces between words, so there are no word
boundaries to split on. A run of Chinese characters gives each character and each pair of neighbors (
查找狗is查,找,狗,查找,找狗). A run of kana or of Thai, Lao, Khmer or Myanmar gives each pair of neighbors. Pairs give the same terms in every runtime, where a dictionary segmenter likeIntl.Segmenterdepends on the runtime’s ICU data.
src/tfidf-languages.ts
[A-Za-z0-9_] only, so words in other scripts had no terms. ASCII text gives the same
terms as before.
A custom tokenizer
Passtokenizer to change how text is split. Wrap defaultTokenizer to add your own rules, such as leaving out stop
words:
src/tfidf-stop-words.ts
loadSnapshot() uses a snapshot’s stored vectors only when
that id is the instance’s own. Otherwise it re-indexes the snapshot’s documents from their text. Give a custom
tokenizer a tokenizerId, and change it whenever the tokenizer’s terms change, so cached snapshots stay usable.
Without one, snapshots of that index are always re-indexed. Snapshots from before 2.3.4 have no id, and are re-indexed
once when loaded.
Example: Tool Discovery
src/tfidf-tool-discovery.ts
Limitations
- No semantic understanding - “car” won’t match “automobile”, and a query in one language won’t match a document in another
- Reindex requirement - Must call
reindex()after changes - Limited to keywords - Misspellings and synonyms aren’t handled
- Memory for large vocabularies - IDF tables grow with vocabulary size
Hybrid Approach
For best of both worlds, you can use TF-IDF as a pre-filter before semantic search:src/tfidf-hybrid.ts
Related
Welcome
Getting started
Search
Semantic search options
Storage
Storage adapters