dsh-testkit

DSH Testkit

The real-host release gate for DeepSeek Harness plugins.

简体中文 · Scenario reference · Architecture · Contributing

A plugin can compile, pass unit tests, and still fail after publication because files are missing from the tarball, its bundle does not register in DSH, or uninstall leaves the profile broken. DSH Testkit closes that gap: it tests the artifact a user actually installs against an exact, real DSH host and retains evidence a maintainer can review.

It makes no model call and needs no model API key.

At A Glance

Use DSH Testkit when you maintain a DSH plugin, review a release PR, operate a plugin template, or need a reproducible host-level bug report. It is intentionally a release gate, not another unit-test framework, static linter, model-output evaluator, or security certification.

Quick Start

Requirements: Node.js 22 or newer and Docker.

pnpm add -D dsh-testkit
pnpm dsh-test init
pnpm dsh-test

For a bundle below the repository root:

pnpm dsh-test init plugin/
pnpm dsh-test --config plugin/dsh-testkit.yaml

dsh-test init works offline, finds the nearest Git worktree, and creates three reviewable files:

• <plugin-root>/dsh-testkit.yaml with the exact DSH version and detected row expectations
• <repository-root>/.github/workflows/dsh-lifecycle.yml with a read-only default token contract and correct nested paths
• <repository-root>/.agents/skills/dsh-testkit/SKILL.md so compatible coding agents apply the same gate

For an exported tree without .git, pass --repo-root .. Generation is byte-idempotent and preflights every target; conflicts stop all writes unless --force is explicit. Review detected rows and add only services, tools, exercises, and update behavior that the plugin contract promises.

Docker is the default runner. Reports land under .dsh-testkit/runs/ as canonical report.json, CI-ready junit.xml, readable report.md, sanitized command logs, and bounded stage evidence.

Where It Fits

DSH quality needs several complementary checks:

A practical pipeline runs cheap static checks on every commit and DSH Testkit on release PRs and tags. Testkit exercises one subject plugin per isolated lifecycle; multi-plugin ownership and update ordering remain composition concerns.

The Lifecycle

resolve -> install-dsh -> package -> install-plugin -> assemble -> boot -> register
        -> exercise -> update? -> uninstall -> reboot -> recover? -> cleanup

DSH Testkit 0.4.3 accepts exact @deepseek-ai/dsh versions 0.1.1-rc.2 (default), 0.1.2-rc.1, 0.1.0-rc.8, 0.1.0-rc.7, and 0.1.0-rc.6. Use npm [email protected] or the corresponding Action release for this support matrix. An unknown version stops before runner creation with exit code 4, so host drift is not mislabeled as a plugin failure.

The disposable canary matrix follows exact npm artifacts and immutable upstream releases independently of formal support. As of 2026-09-08, unsupported published alpha versions include 0.1.2-alpha.2 through 0.1.2-alpha.5 and 0.1.3-alpha.2. Release Watch selects versions newer than the highest supported host: its current runnable candidate is 0.1.3-alpha.2, while 0.1.3-alpha.1 awaits a matching npm package. The older official 0.1.2-alpha.1 also has no matching npm artifact and is outside the current watch range. See Host Compatibility for evidence and promotion requirements.

What A Pass Means

• The same packed artifact identified in the report completed every required stage.
• Requested rows came from DSH --dump-config; services and tool schemas came from an in-process Cordis probe.
• Declared exercises ran through the real tool runtime without model selection.
• The same profile rebooted after removal without the subject bundle, capabilities, or attributable residue.
• Required observers were available. Missing required coverage is unsupported, never a synthetic pass.

A pass does not prove that arbitrary executable code is safe, that model output is good, or that unasserted behavior works.

Scenario As Code

dsh-test init produces a small starting scenario:

schemaVersion: 1
name: my-plugin-quick
subject:
  source: .
dsh:
  version: 0.1.1-rc.2
expect:
  boot: success
  rows: [tool-my-plugin]
  services: [myService]
  tools: [my_tool]
exercise:
  - tool: my_tool
    arguments:
      value: smoke
observers:
  filesystem: required
  process: preferred
  ports: preferred
  network: off
  canary: preferred

Local-directory subjects are mounted read-only, copied into the runner-owned writable root, and then packed. When prepare, prepack, or postpack is declared, Testkit restores dependencies in that copy using packageManager and its lockfile before npm pack; the original checkout is never modified.

For a public DSH web route, set profile: web and add a Docker-only assertion:

profile: web
http:
  routes:
    - id: health
      path: /health
      expect:
        status: 200
        json:
          status: ok
          version: $subject.packageVersion

The Scenario Reference covers http.routes, update targets, expected failure and recovery, stage reruns, observer policy, the attempt-wide watchdog, and the explicit dsh web TurnStatus browser smoke. HTTP and browser traffic stays on runner-owned 127.0.0.1. Missing Chromium is unsupported; a live but permanently unresponsive DSH web host or watchdog expiry is host/infrastructure, not a plugin failure.

Least-Privilege CI

The generated workflow uses a read-only token and makes that contract visible:

permissions:
  contents: read

steps:
  - uses: iiwish/dsh-testkit/.github/actions/dsh-test@v0
    with:
      plugin: .
      dsh-version: 0.1.1-rc.2
      config: dsh-testkit.yaml
      publish-junit-check: 'false'

This default writes JUnit annotations to the job and exposes the artifact ID, URL, digest, report path, and stable exit code. It does not call the Checks API. The source Action checks evidence before either publication outlet and uploads only a fresh staged copy with a hash manifest. Unsafe evidence fails the job and is not published. This guard is not included in the published v0 Action until a release includes it.

A trusted push or release workflow may opt into a named JUnit Check:

permissions:
  contents: read
  checks: write

steps:
  - uses: iiwish/dsh-testkit/.github/actions/dsh-test@v0
    with:
      plugin: .
      dsh-version: 0.1.1-rc.2
      publish-junit-check: 'true'

Do not enable that option for untrusted fork pull requests. The repository's external Actions are pinned to immutable commits; the moving v0 tag is the consumer compatibility channel. GitHub Enterprise Server and other CI systems can invoke the CLI directly.

Stable exits are 0 passed, 1 lifecycle failure, 2 invalid input, 3 infrastructure error, 4 unsupported, and 5 flaky. Published schemas are available at dsh-testkit/schemas/report-v1.json and dsh-testkit/schemas/scenario-v1.json.

Native And Agent Entry Points

The project-local Skill and the exported dsh-testkit/skills/dsh-testkit/SKILL.md teach compatible agents how to select coverage, interpret evidence, and preserve the Docker boundary. A Skill guides usage; it does not grant trust or replace review.

DSH Testkit also ships an optional, community-maintained DSH Profile Bundle:

dsh plugin --profile web add [email protected]
dsh --profile web --dump-config

It registers dsh_test, a confirmed, Docker-only adapter over the same engine. The external CLI or CI Action remains the independent recovery gate because an in-host tool cannot diagnose a host that fails before tool registration.

Safety And Trust

Plugins are executable code: package scripts and runtime code run during the lifecycle. Docker reduces the default blast radius with a read-only root filesystem and source mount, disposable writable state, dropped capabilities, resource limits, and bounded evidence. It is not a hardened malware sandbox.

Use disposable infrastructure for unknown code. Never use --runner local --unsafe-local for an untrusted plugin. The native tool requires Docker daemon access, and confirmation is a trust decision rather than certification. Private plugin source remains on the runner; DSH Testkit has no SaaS dependency and uploads nothing except the evidence configured by your CI workflow.

The Architecture documents trust boundaries. Use the Security Policy for private vulnerability reports.

Community

The community validation protocol defines credential-free, exact-version cohort runs and aggregate-only reporting. The dsh-shelf case study shows why install success alone is not proof of host registration. The design-partner follow-up gate records the immutable package baselines and prevents source-only fixes from being reported as package reruns.

Useful bug reports include the exact plugin and DSH versions, failing stage, report.json, and sanitized logs. Start with Contributing or join the official DeepSeek Harness Show & Tell discussion.

DSH Testkit is an independent, unofficial community project released under the MIT License.