> ## Documentation Index > Fetch the complete documentation index at: https://docs.agentfront.dev/llms.txt > Use this file to discover all available pages before exploring further. # Semantic Search with VectoriaDB > Index your FrontMCP tools with VectoriaDB for offline semantic discovery and personalized tool recommendations. VectoriaDB is FrontMCP's in-memory vector database built on transformers.js. Use it to surface the right tool, prompt, or document snippet from natural-language queries without shipping data to an external service. This guide shows how to fold VectoriaDB into a FrontMCP server for always-on semantic discovery. Embeddings run locally via transformers.js, so your users' data never leaves the server and you avoid API quotas. Strong generics ensure every document you index keeps the same shape as your tool metadata. Built-in rate limits, batch validation, HNSW indexing, and storage adapters keep the index production ready. ## What you'll build * A typed document shape for every tool, app, or resource you want to search * An indexing routine that stays in sync with `toolRegistry.getTools(true)` * Semantic queries with metadata filters, score thresholds, and pagination controls * Persistent caches (file or Redis) so restarts do not require re-embedding everything * Tunable HNSW search for large inventories The default Xenova `all-MiniLM-L6-v2` model is \~22 MB. The first initialization downloads and caches it under `cacheDir`; subsequent boots reuse the local copy. ## Prerequisites * Node.js 22 or later (Node 24 is recommended and is what FrontMCP tests against) * An existing FrontMCP server with at least one app and tool registry * Ability to install npm packages in the server workspace * Optional: writable disk or Redis if you plan to persist embeddings between restarts ## Step 1: Install & initialize VectoriaDB Install the package alongside your server: ```bash npm theme={"theme":{"light":"snazzy-light","dark":"dark-plus"}} npm install vectoriadb@^2 ``` ```bash pnpm theme={"theme":{"light":"snazzy-light","dark":"dark-plus"}} pnpm add vectoriadb@^2 ``` ```bash yarn theme={"theme":{"light":"snazzy-light","dark":"dark-plus"}} yarn add vectoriadb@^2 ``` FrontMCP 0.5 depends on `vectoriadb@^2.0.1`. Use the `^2` range to stay compatible with the SDK and automatically pick up 2.x optimizations and fixes. Initialize a singleton database during server startup (for example inside your custom bootstrap or provider): ```ts theme={"theme":{"light":"snazzy-light","dark":"dark-plus"}} import { VectoriaDB, DocumentMetadata } from 'vectoriadb'; interface ToolDocument extends DocumentMetadata { toolName: string; owner: string; tags: string[]; risk: 'safe' | 'destructive'; } export const toolIndex = new VectoriaDB({ cacheDir: './.cache/transformers', defaultSimilarityThreshold: 0.4, }); await toolIndex.initialize(); // downloads and warms the embedding model once ``` `initialize()` must run before `add`, `search`, or `update`. Calling it twice is safe because VectoriaDB short-circuits if it is already ready. ## Step 2: Index your tools Collect metadata from the tool registry (apps, plugins, adapters, or scopes) and write it into the database. Each document needs a unique `id`, the natural-language `text` you want to vectorize, and `metadata` that extends `DocumentMetadata`. ```ts theme={"theme":{"light":"snazzy-light","dark":"dark-plus"}} import type { ToolEntry, ToolRegistryInterface } from '@frontmcp/sdk'; function collectToolDocuments(tools: ToolEntry[]): Array<{ id: string; text: string; metadata: ToolDocument }> { return tools.map((tool) => { const docId = `${tool.owner.id}:${tool.name}`; return { id: docId, text: [ tool.metadata.description ?? tool.name, `Inputs: ${Object.keys(tool.metadata.inputSchema ?? {}).join(', ') || 'none'}`, `Tags: ${(tool.metadata.tags ?? []).join(', ') || 'none'}`, ].join('\n'), metadata: { id: docId, toolName: tool.name, owner: tool.owner.id, tags: tool.metadata.tags ?? [], risk: tool.metadata.annotations?.destructiveHint ? 'destructive' : 'safe', }, }; }); } export async function indexTools(toolRegistry: ToolRegistryInterface) { const tools = toolRegistry.getTools(true); // include hidden tools so you can filter later await toolIndex.addMany(collectToolDocuments(tools)); } ``` `addMany` validates every document, enforces `maxBatchSize`, and prevents duplicates. Use it after deployments or whenever your tool inventory changes. When using the CodeCall plugin, tool examples are automatically indexed for semantic search. Example descriptions and input values receive 2x weight, so tools with relevant examples rank higher in search results. See the [Tools documentation](/frontmcp/v/0.7/servers/tools#tool-examples) for how to define examples. ## Step 3: Run semantic search Query the index anywhere you can run async code (for example inside a custom MCP tool that recommends next actions): ```ts theme={"theme":{"light":"snazzy-light","dark":"dark-plus"}} const matches = await toolIndex.search('reset a billing password', { topK: 5, threshold: 0.45, filter: (metadata) => metadata.owner === 'billing' && !metadata.tags.includes('deprecated'), }); for (const match of matches) { console.log(`${match.metadata.toolName} (${match.score.toFixed(2)})`); } ``` `search` returns the best matches sorted by cosine similarity. Use `filter` to enforce authorization, `includeVector` to inspect raw vectors, and `threshold` to drop low-confidence hits. Keep the index current with `updateMetadata`, `update`, or `updateMany`. Metadata-only updates never trigger re-embedding, while text changes re-embed only the affected documents. ## Persist embeddings between restarts Avoid re-indexing on every boot by swapping the default in-memory adapter with the provided file or Redis adapters plus a deterministic tools hash. ```ts theme={"theme":{"light":"snazzy-light","dark":"dark-plus"}} import { FileStorageAdapter, createToolsHash } from 'vectoriadb'; export async function warmToolIndex(toolRegistry: ToolRegistryInterface) { const documents = collectToolDocuments(toolRegistry.getTools(true)); const toolIndex = new VectoriaDB({ storageAdapter: new FileStorageAdapter({ cacheDir: './.cache/vectoriadb', namespace: 'tool-index', }), toolsHash: createToolsHash(documents), version: process.env.npm_package_version, }); await toolIndex.initialize(); if (toolIndex.size() === 0) { await toolIndex.addMany(documents); await toolIndex.saveToStorage(); // persist embeddings to disk } return toolIndex; } ``` `toolsHash` automatically invalidates the cache when your tool list or descriptions change. Call `saveToStorage()` after indexing; `initialize()` transparently loads the cache on the next boot. Need a shared cache across pods? Swap in `RedisStorageAdapter` with your preferred Redis client and namespace. TTLs and key prefixes are configurable per adapter. ## Scale & tune search * Enable `useHNSW` for datasets above roughly ten thousand documents. HNSW provides sub-millisecond queries with more than 95% recall. * Adjust `threshold` and `topK` per query to trade recall for precision. * Guard resource usage with `maxDocuments`, `maxDocumentSize`, and `maxBatchSize` (VectoriaDB enforces these automatically). * Set a custom `cacheDir` if your runtime has strict filesystem policies. ```ts theme={"theme":{"light":"snazzy-light","dark":"dark-plus"}} const toolIndex = new VectoriaDB({ useHNSW: true, hnsw: { M: 16, efConstruction: 200, efSearch: 64 }, maxDocuments: 150_000, maxBatchSize: 2_000, }); ``` ### Complete Configuration Options | Option | Type | Default | Description | | ---------------------------- | ------- | --------------------------- | ------------------------------ | | `modelName` | string | `'Xenova/all-MiniLM-L6-v2'` | Embedding model to use | | `cacheDir` | string | `'./.cache/transformers'` | Model cache directory | | `dimensions` | number | Auto-detected | Vector dimensions | | `defaultSimilarityThreshold` | number | `0.3` | Minimum similarity score | | `defaultTopK` | number | `10` | Default results limit | | `useHNSW` | boolean | `false` | Enable HNSW index | | `maxDocuments` | number | `100000` | Max documents (DoS protection) | | `maxDocumentSize` | number | `1000000` | Max document size in chars | | `maxBatchSize` | number | `1000` | Max batch operation size | | `verboseErrors` | boolean | `true` | Enable detailed errors | ### HNSW Configuration | Option | Default | Description | | ---------------- | ------- | ---------------------------------------------------------- | | `M` | 16 | Connections per node in layer > 0 (higher = better recall) | | `M0` | 32 | Connections for layer 0 (typically M \* 2) | | `efConstruction` | 200 | Candidate list size during construction | | `efSearch` | 50 | Candidate list size during search | *** ## TF-IDF Variant (Zero Dependencies) For scenarios where ML model downloads aren't acceptable, use the TF-IDF variant: ```ts theme={"theme":{"light":"snazzy-light","dark":"dark-plus"}} import { TFIDFVectoria } from 'vectoriadb'; interface ToolDocument extends DocumentMetadata { toolName: string; category: string; } const db = new TFIDFVectoria({ defaultSimilarityThreshold: 0.0, defaultTopK: 10, }); // Add documents db.addDocument('tool1', 'User authentication tool', { id: 'tool1', toolName: 'auth', category: 'security' }); db.addDocument('tool2', 'User profile retrieval', { id: 'tool2', toolName: 'profile', category: 'user' }); // Reindex after adding documents (required for IDF update) db.reindex(); // Search const results = db.search('authentication', { topK: 5 }); ``` ### When to Use TF-IDF vs ML Embeddings | Feature | TFIDFVectoria | VectoriaDB | | ---------------------- | ------------------------------ | ------------------------------ | | Dependencies | Zero | transformers.js (\~22MB model) | | Initialization | Synchronous | Async (model download) | | Semantic understanding | Keyword-based | Full semantic | | Best for | Small corpora (under 10K docs) | Any size | | Reindex required | Yes, after changes | No | *** ## Storage Adapters ### File Adapter ```ts theme={"theme":{"light":"snazzy-light","dark":"dark-plus"}} import { VectoriaDB, FileStorageAdapter } from 'vectoriadb'; const toolIndex = new VectoriaDB({ storageAdapter: new FileStorageAdapter({ cacheDir: './.cache/vectoriadb', namespace: 'tool-index', }), }); ``` ### Redis Adapter For multi-pod environments, use Redis to share embeddings: ```ts theme={"theme":{"light":"snazzy-light","dark":"dark-plus"}} import { VectoriaDB, RedisStorageAdapter } from 'vectoriadb'; import Redis from 'ioredis'; const redisClient = new Redis(); const toolIndex = new VectoriaDB({ storageAdapter: new RedisStorageAdapter({ client: redisClient, namespace: 'tool-index', ttl: 86400, // 24 hours (default) keyPrefix: 'vectoriadb', }), }); ``` ### Memory Adapter (Default) No persistence - embeddings are lost on restart: ```ts theme={"theme":{"light":"snazzy-light","dark":"dark-plus"}} import { VectoriaDB, MemoryStorageAdapter } from 'vectoriadb'; const toolIndex = new VectoriaDB({ storageAdapter: new MemoryStorageAdapter({ namespace: 'tools' }), }); ``` *** ## Handle errors and monitor health All errors extend `VectoriaError` and ship with machine-readable `code` values so you can branch on them. ### Error Types | Error | Code | Description | | ----------------------------- | --------------------------- | -------------------------- | | `VectoriaNotInitializedError` | `NOT_INITIALIZED` | Call `initialize()` first | | `DocumentValidationError` | `DOCUMENT_VALIDATION_ERROR` | Invalid document data | | `DocumentNotFoundError` | `DOCUMENT_NOT_FOUND` | Document ID doesn't exist | | `DocumentExistsError` | `DOCUMENT_EXISTS` | Document ID already exists | | `DuplicateDocumentError` | `DUPLICATE_DOCUMENT` | Duplicate in batch | | `QueryValidationError` | `QUERY_VALIDATION_ERROR` | Invalid search query | | `EmbeddingError` | `EMBEDDING_ERROR` | Model embedding failed | | `StorageError` | `STORAGE_ERROR` | Storage operation failed | | `ConfigurationError` | `CONFIGURATION_ERROR` | Invalid config | ```ts theme={"theme":{"light":"snazzy-light","dark":"dark-plus"}} import { VectoriaError, VectoriaNotInitializedError, DocumentValidationError, DocumentNotFoundError, DocumentExistsError, DuplicateDocumentError, QueryValidationError, EmbeddingError, StorageError, ConfigurationError, } from 'vectoriadb'; try { await toolIndex.add(doc.id, doc.text, doc.metadata); } catch (error) { if (error instanceof VectoriaNotInitializedError) { await toolIndex.initialize(); } else if (error instanceof DocumentValidationError) { console.warn({ tool: error.documentId }, 'invalid document skipped'); } else if (error instanceof VectoriaError) { console.error({ code: error.code }, error.message); throw error; } else { throw error; } } ``` Use `toolIndex.getStats()` to feed dashboards or health endpoints: ```ts theme={"theme":{"light":"snazzy-light","dark":"dark-plus"}} const stats = toolIndex.getStats(); /* { totalEmbeddings: number; dimensions: number; estimatedMemoryBytes: number; modelName: string; } */ ``` Pair stats with `toolIndex.size()`, `toolIndex.clear()`, and `toolIndex.clearStorage()` to expose maintenance commands or admin tooling.