Skip to main content
Learn how to use the TF-IDF variant for keyword-based search without external dependencies.
In this guide you’ll learn when to use TF-IDF instead of semantic search, how to configure and query it, and understand its limitations.

When to Use TF-IDF

Basic Usage

src/tfidf-basic.ts

Key Differences from VectoriaDB

Important: Reindexing

TF-IDF requires reindexing after document changes to update IDF (Inverse Document Frequency) values:
src/tfidf-reindex.ts
Forgetting to call reindex() after changes will result in incorrect search results.

Configuration Options

src/tfidf-config.ts

Search Options

src/tfidf-search.ts

TF-IDF Algorithm

TF-IDF (Term Frequency-Inverse Document Frequency) works by:
  1. Term Frequency (TF): How often a term appears in a document
  2. Inverse Document Frequency (IDF): How rare a term is across all documents
  3. TF-IDF Score: TF x IDF - terms that are frequent in a document but rare overall get high scores
This means:
  • Common words like “the”, “is”, “a” get low scores (low IDF)
  • Unique terms specific to a document get high scores
  • The query is matched against TF-IDF vectors using cosine similarity

Languages and tokenization

Documents and queries go through the same tokenizer, so a query term matches only a term from a document. The default tokenizer, defaultTokenizer, gives lowercase terms in any script:
  • Letters, combining marks, digits and _ make up terms; everything else separates them. Terms of one character are left out.
  • Text is NFKD-normalized: compatibility forms are folded (file is file, full-width cats is cats), and marks on Latin and Greek letters are dropped, so Vögel matches Vogel and café matches cafe. Other marks are kept: Cyrillic й, and Hebrew, Arabic and Devanagari words, stay as written.
  • Chinese, Japanese, Thai, Lao, Khmer and Myanmar are written without spaces between words, so there are no word boundaries to split on. A run of Chinese characters gives each character and each pair of neighbors (查找狗 is 查, 找, 狗, 查找, 找狗). A run of kana or of Thai, Lao, Khmer or Myanmar gives each pair of neighbors. Pairs give the same terms in every runtime, where a dictionary segmenter like Intl.Segmenter depends on the runtime’s ICU data.
src/tfidf-languages.ts
Before 2.3.4, the tokenizer kept [A-Za-z0-9_] only, so words in other scripts had no terms. ASCII text gives the same terms as before.

A custom tokenizer

Pass tokenizer to change how text is split. Wrap defaultTokenizer to add your own rules, such as leaving out stop words:
src/tfidf-stop-words.ts
A snapshot records the id of the tokenizer that built it. loadSnapshot() uses a snapshot’s stored vectors only when that id is the instance’s own. Otherwise it re-indexes the snapshot’s documents from their text. Give a custom tokenizer a tokenizerId, and change it whenever the tokenizer’s terms change, so cached snapshots stay usable. Without one, snapshots of that index are always re-indexed. Snapshots from before 2.3.4 have no id, and are re-indexed once when loaded.

Example: Tool Discovery

src/tfidf-tool-discovery.ts

Limitations

  1. No semantic understanding - “car” won’t match “automobile”, and a query in one language won’t match a document in another
  2. Reindex requirement - Must call reindex() after changes
  3. Limited to keywords - Misspellings and synonyms aren’t handled
  4. Memory for large vocabularies - IDF tables grow with vocabulary size

Hybrid Approach

For best of both worlds, you can use TF-IDF as a pre-filter before semantic search:
src/tfidf-hybrid.ts

Welcome

Getting started

Search

Semantic search options

Storage

Storage adapters