semantic-search

dsh-semantic-search

Local semantic code search for DeepSeek Harness (dsh) — and any Node script.

中文文档：README.zh.md

sema builds a fragment-level index of a workspace: source is tokenized by a language-aware tokenizer (camel/snake/kebab splitting, CJK n-grams), chunked with symbol-aware boundaries (functions/classes stay intact), and embedded into fixed dimension vectors — locally and dependency-free by default (feature-hashed lexical TF-IDF), or via any OpenAI-compatible embedding endpoint. Queries run a hybrid retrieval (vector cosine + BM25, fused with reciprocal-rank fusion) over the index, so meaning-based search works even when exact terms don't match.

Ships as a dsh plugin bundle — the sema_search, sema_reindex and sema_stats tools on the harness tool registry — plus a standalone sema CLI.

Highlights

Works offline by default — the built-in lexical provider needs no network,

no model download, no API key. Use it as a fast BM25-plus code search.

Symbol-aware chunking — boundaries from a conservative per-language table

(16 languages) keep functions/classes intact; a missed boundary degrades to a plain chunk rather than breaking.

CJK-aware tokenizer — n-gram tokenization (default bigrams) aligns Chinese

queries and documents without a segmentation library; full-width punctuation is folded, not treated as a hard break.

Hybrid retrieval with RRF — vector cosine + BM25 channels are fused by

reciprocal-rank fusion, so a document found by one channel still ranks.

Graceful degradation — if a configured remote provider is unreachable, the

index falls back to the local lexical provider (configurable via allowFallback).

Incremental refresh + file watching — sema_reindex diffs by size+mtime,

and an optional watcher keeps the index live.

Persistence — the index is saved to <root>/.sema atomically (JSON metadata
binary vectors), with staleness detection when the provider/dimension changes.
Deterministic — the same workspace and options produce the same index and

the same ranked answers.

Supported languages

TypeScript, JavaScript, Python, Go, Rust, Java, Kotlin, Scala, C, C++, C#, Objective-C, Ruby, PHP, Swift, Bash, plus common data/markup formats (JSON, YAML, TOML, Markdown, HTML, XML...).

Installation

As a dsh bundle

The package declares "dsh": { "bundle": { "patch": "./cordis.patch.yml" } }. The patch inserts one plugin row that mounts the bundle and registers sema_search / sema_reindex / sema_stats on ctx.tools.

# from npm (name reserved; publish pending access setup)
npm install -g dsh-semantic-search

# straight from this repository
dsh plugin --profile demo add github:JohnXu22786/semantic-search

# or from a local checkout
dsh plugin --profile demo add /path/to/semantic-search

As a standalone CLI

npm install -g dsh-semantic-search   # or: npm run build && node bin/sema.mjs
sema --help

CLI usage

sema index                build the full index from the workspace
sema reindex [--full]     incremental refresh (or full rebuild with --full)
sema search <query...>    hybrid vector + BM25 search, prints top hits
sema stats [--json]       index health, provider, and sizing numbers

Global options:

--root <dir>          workspace root (default: current directory)
--data-dir <dir>      index storage directory (default: <root>/.sema)
--provider <kind>     embedding provider: lexical | openai (default: lexical)
--dim <n>             embedding dimension (lexical default: 4096; openai 0 = auto)
--base-url <url>      OpenAI-compatible embeddings endpoint base URL
--model <name>        embeddings model name (openai only)
--api-key <key>       API key (openai only; env: SEMA_EMBEDDING_API_KEY)
--top <n>             hits to print for search (default: 20)
--json                machine-readable output where supported
--help                show this help

CLI exit codes

0 — success (including a search with zero hits and a --version/--help call).
1 — a runtime failure (config error, build/index/search error).
2 — a usage error: unknown command, unknown flag, or a missing query.

Configuration

The plugin is configurable through the bundle row's config (see cordis.patch.yml for an example), the CLI flags above, or defaults in code:

Option	Default	Meaning
`root`	cwd	workspace root to index
`dataDir`	`.sema`	index storage directory
`provider.kind`	`lexical`	`lexical` (offline) or `openai`
`provider.dimension`	`0` (lexical: 4096)	embedding dimension; `0` = auto-infer from the endpoint
`provider.baseUrl`	`https://api.openai.com/v1`	OpenAI-compatible endpoint root
`provider.model`	`text-embedding-3-small`	embedding model name
`provider.apiKeyEnv`	`SEMA_EMBEDDING_API_KEY`	env var holding the API key
`allowFallback`	`true`	fall back to the lexical provider when a remote one fails
`include` / `ignore`	defaults	glob sets of files to index / skip
`maxLinesPerChunk`	`80`	hard chunk size upper bound
`nGram`	`2`	CJK n-gram size (`1` disables n-gramming)
`topK`	`20`	hits returned by default
`rrfK`	`60`	RRF fusion constant
`vectorK`	`300`	candidates per channel before fusion
`autosave`	`true`	persist the index after builds
`autoIndex`	`true`	build lazily on first search
`watch`	`true`	watch the workspace for changes

Development

npm ci
npm test          # build + run the node:test suite (76 tests)
npm run typecheck
npm run build     # tsc -> lib/

專案介紹

專案資訊與收錄訊號

資料評分

安裝與使用

專案截圖

倉庫 README