DeepBase

Overview

DeepBase is a multi-driver persistence system for Node.js that provides a unified API across storage backends (JSON, SQLite, MongoDB, Redis, IndexedDB). It allows switching or combining backends without changing application code.

Use this skill when:

• Adding or integrating persistence into a Node.js project
• Setting up multi-backend storage with automatic failover
• Migrating data between storage drivers
• Working with nested object paths for data access
• Integrating any DeepBase driver package (deepbase, deepbase-sqlite, deepbase-mongodb, etc.)

Do NOT use this skill when:

• The project needs a full ORM or relational query builder (use Prisma, Drizzle, etc.)
• The project requires SQL queries, joins, or complex aggregations
• The task is about browser-only storage without DeepBase (use raw IndexedDB/localStorage)

Quick Start

Step 1: Install

# Core package (includes JSON driver by default)
npm install deepbase

# Optional drivers — install only what you need
npm install deepbase-sqlite       # SQLite (better-sqlite3)
npm install deepbase-mongodb      # MongoDB
npm install deepbase-redis        # Redis (vanilla)
npm install deepbase-redis-json   # Redis Stack (RedisJSON)
npm install deepbase-indexeddb    # Browser IndexedDB

Check package.json first — avoid reinstalling if deepbase is already listed.

Step 2: Create and use a store

import DeepBase from 'deepbase';

const db = new DeepBase({ path: './data', name: 'app' });
// No need to call connect() — lazy connect handles it automatically

await db.set('config', 'theme', 'dark');
const theme = await db.get('config', 'theme'); // 'dark'

await db.disconnect(); // Always disconnect on shutdown

Step 3: Go multi-driver (when resilience is needed)

import DeepBase from 'deepbase';
import { JsonDriver } from 'deepbase-json';
import { MongoDriver } from 'deepbase-mongodb';

const db = new DeepBase([
  new MongoDriver({ url: process.env.MONGO_URL, database: 'myapp', collection: 'data' }),
  new JsonDriver({ path: './backup', name: 'fallback' })
], {
  writeAll: true,            // Write to all drivers (default: true)
  readFirst: true,           // Read from first available (default: true)
  failOnPrimaryError: false  // Continue if primary fails
});

Imports

// ES Modules (recommended)
import DeepBase from 'deepbase';
import { DeepBase, DeepBaseDriver } from 'deepbase';

// CommonJS
const { DeepBase } = require('deepbase');

// Individual drivers
import { JsonDriver } from 'deepbase-json';
import { MongoDriver } from 'deepbase-mongodb';
import { SqliteDriver } from 'deepbase-sqlite'; // pragma + busyTimeoutMs/busyRetry; same-host multi-process safe
import { RedisDriver } from 'deepbase-redis';
import { RedisDriver as RedisJsonDriver } from 'deepbase-redis-json';
import { IndexedDBDriver } from 'deepbase-indexeddb';

API Reference

All methods are async. Path arguments are variadic strings representing nested keys.

Data operations

| Method | Signature | Description | |--------|-----------|-------------| | get | get(...path) | Get value at path. Returns null if not found. | | set | set(...path, value) | Set value at path. Last argument is the value. | | del | del(...path) | Delete value at path. | | inc | inc(...path, amount) | Increment numeric value. | | dec | dec(...path, amount) | Decrement numeric value. | | add | add(...path, value) | Add item with auto-generated ID. Returns full path array. | | upd | upd(...path, fn) | Atomic update — passes current value to fn, stores the return value. |

Query operations

| Method | Signature | Description | |--------|-----------|-------------| | keys | keys(...path) | Get keys at path (array of strings). | | values | values(...path) | Get values at path. | | entries | entries(...path) | Get [key, value] pairs at path. | | pop | pop(...path) | Remove and return the last item. | | shift | shift(...path) | Remove and return the first item. |

Connection

| Method | Description | |--------|-------------| | connect() | Connect all drivers. Returns { connected, total }. | | disconnect() | Disconnect all drivers. | | dispose(options) | Wait for pending writes, disconnect, and optionally clear memory/release singleton instances. |

Driver access

| Method | Description | |--------|-------------| | getDriver(index) | Get driver instance by index (default: 0). | | getDrivers() | Get array of all driver instances. |

Migration

// Migrate data from driver 0 to driver 1
await db.migrate(0, 1, {
  clear: true,       // Clear target first (default: true)
  batchSize: 100,    // Progress callback interval
  onProgress: ({ migrated, errors, current }) => console.log(`${migrated} items`)
});

// Sync primary (index 0) to all other drivers
await db.syncAll({ clear: true });

Constructor Options

new DeepBase(drivers, {
  writeAll: true,              // Write to all drivers
  readFirst: true,             // Read from first available driver in order
  failOnPrimaryError: true,    // Throw if primary driver (index 0) fails
  lazyConnect: true,           // Auto-connect on first operation
  timeout: 0,                  // Global timeout in ms (0 = disabled)
  readTimeout: 0,              // Override for read operations
  writeTimeout: 0,             // Override for write operations
  connectTimeout: 0            // Override for connect
});

Driver Configuration

| Driver | Key Options | |--------|-------------| | JsonDriver | path (directory), name (filename), stringify / parse (custom serialization) | | SqliteDriver | path (directory), name (database filename), pragma ('none' | 'safe' | 'balanced' | 'fast', default 'balanced'), busyTimeoutMs (native lock wait ms, default 5000), busyRetry ({ maxAttempts, baseDelayMs, maxDelayMs }) | | MongoDriver | url, database, collection | | RedisDriver | url, prefix | | RedisJsonDriver | url, prefix (requires Redis Stack with RedisJSON module) | | IndexedDBDriver | name, version |

Rules for Agents

Follow these rules when generating DeepBase code:

1. Check before installing. Verify package.json for existing deepbase dependency before running npm install.
2. Use the official API. Never manipulate driver internals or the underlying JSON/SQLite/Mongo storage directly. Always go through db.get(), db.set(), etc., or the documented driver methods reached via getDriver(index) (e.g. backup()).
3. Prefer lazy connect. Do not call db.connect() explicitly unless you need the { connected, total } result. Lazy connect handles it automatically.
4. Always disconnect() on shutdown. Especially important for MongoDB and Redis drivers to release connections.
5. Use add() for auto-IDs. Do not manually generate IDs with nanoid/uuid — add() returns the full path array including the generated ID.
6. Spread the path from add(). The return value is an array: use await db.get(...userPath) to retrieve the added item.
7. Use upd() for atomic changes. When modifying existing values based on their current state, use upd() instead of get() + set() to avoid race conditions.
8. Set failOnPrimaryError: false for resilient setups. When using multi-driver for fault tolerance, disable this so operations continue via fallback drivers.
9. Use environment variables for connection strings. Never hardcode MongoDB URLs or Redis URLs in source code.
10. Prefer deepbase-redis-json over deepbase-redis when working with Redis Stack, as it supports native JSON operations.
11. SQLite same-host writers are safe by default. Multiple processes may open the same .db file; writers wait/retry via BEGIN IMMEDIATE + busyTimeoutMs / busyRetry. Keep one writer process per host when contention is sustained.
12. Never put SQLite on NFS or share one file across hosts. WAL requires a local filesystem on a single host. For multi-host writers, use MongoDB/Redis/Postgres instead.
13. Stop old SQLite writers before upgrading. Do not run old in-memory-seq binaries and new SQL-seq binaries against the same file at once.
14. Never install better-sqlite3 alongside deepbase-sqlite. SQLite maintenance lives on the driver: backup(), checkIntegrity(), vacuum(), checkpoint(). A second copy of the native module invites a version mismatch with the driver's.

Examples

Example 1: Adding a simple persistent store to a new project User says: "I need to persist user settings in my Node.js app." Actions: Install deepbase, create a DeepBase instance with JsonDriver, use set/get for nested keys. Result: A ./data/settings.json file managed transparently via the DeepBase API.

Example 2: Multi-driver setup with MongoDB primary and JSON fallback User says: "I want MongoDB as my main database but with a local JSON backup in case it goes down." Actions: Create DeepBase with [MongoDriver, JsonDriver], set writeAll: true and failOnPrimaryError: false. Use process.env.MONGO_URL. Result: All writes go to both drivers; reads use MongoDB first and fall back to JSON silently.

Example 3: Migrating data from JSON to SQLite User says: "I want to move my existing JSON data into SQLite." Actions: Create DeepBase with both drivers loaded, call db.migrate(0, 1, { clear: true }). Result: All data is transferred from the JSON file to the SQLite database.

Example 4: Auto-ID collection User says: "I want to store multiple users with unique IDs automatically." Actions: Use db.add('users', { name, email }), capture the returned path array, use db.get(...path) to retrieve. Result: Each user gets a nanoid-based key under users.

Common Tasks

Store and retrieve nested data

await db.set('users', 'alice', { name: 'Alice', age: 30 });
await db.set('users', 'alice', 'email', 'alice@example.com');
const user = await db.get('users', 'alice');
// { name: 'Alice', age: 30, email: 'alice@example.com' }

Add items with auto-generated IDs

const userPath = await db.add('users', { name: 'Bob', email: 'bob@example.com' });
// userPath = ['users', 'aB3xK9mL2n']
const user = await db.get(...userPath);

Increment/decrement counters

await db.set('stats', 'views', 0);
await db.inc('stats', 'views');      // 1
await db.inc('stats', 'views', 10);  // 11
await db.dec('stats', 'views', 5);   // 6

Atomic update

await db.upd('user', 'name', name => name.toUpperCase());

Iterate over collections

const userKeys = await db.keys('users');
const userList = await db.values('users');
const userEntries = await db.entries('users'); // [[id, data], ...]

Pop/shift from collections

const last = await db.pop('queue');    // Remove and return last item
const first = await db.shift('queue'); // Remove and return first item

Custom JSON serialization (circular references)

import { JsonDriver } from 'deepbase-json';
import { stringify, parse } from 'flatted';

const db = new DeepBase(new JsonDriver({
  path: './data',
  name: 'circular',
  stringify,
  parse
}));

Transform sensitive fields in the JSON memory cache

JsonDriver accepts synchronous encodeForMemory(value, path) and decodeFromMemory(value, path) hooks. Recursively preserve object keys so non-sensitive paths remain queryable:

const secrets = new Set(['privateKey', 'mnemonic']);
const transform = (value, path, secretFn) => {
  if (secrets.has(path.at(-1))) return secretFn(value);
  if (Array.isArray(value)) return value.map((v, i) => transform(v, [...path, String(i)], secretFn));
  if (value && typeof value === 'object') return Object.fromEntries(
    Object.entries(value).map(([k, v]) => [k, transform(v, [...path, k], secretFn)])
  );
  return value;
};

const db = new DeepBase(new JsonDriver({
  encodeForMemory: (value, path) => transform(value, path, seal),
  decodeFromMemory: (value, path) => transform(value, path, unseal)
}));

await db.dispose({ clearMemory: true, releaseInstance: true });

Hooks receive defensive copies; throw on transform/authentication failure.

Three-tier architecture

const db = new DeepBase([
  new MongoDriver({ url: process.env.MONGO_URL, database: 'app' }),
  new JsonDriver({ path: './backup' }),
  new RedisDriver({ url: process.env.REDIS_URL })
], { writeAll: true, failOnPrimaryError: false });

Extend with a custom driver

import { DeepBaseDriver } from 'deepbase';

class MyDriver extends DeepBaseDriver {
  async connect() { /* ... */ this._connected = true; }
  async disconnect() { /* ... */ }
  async get(...args) { /* ... */ }
  async set(...args) { /* ... */ }
  async del(...args) { /* ... */ }
  async inc(...args) { /* ... */ }
  async dec(...args) { /* ... */ }
  async add(...args) { /* ... */ }
  async upd(...args) { /* ... */ }
}

All methods listed in DeepBaseDriver must be implemented. keys(), values(), entries() have default implementations that call get().

Troubleshooting

• All drivers must extend DeepBaseDriver — Ensure all drivers in the array are proper driver instances, not plain objects.
• Operation timed out — Increase timeout, readTimeout, or writeTimeout in the constructor options.
• MongoDB/Redis connection fails silently — Set failOnPrimaryError: true (default) to surface connection errors, or check connect() return value for { connected, total }.
• Data not synced across drivers — Ensure writeAll: true (default). For existing data, use db.migrate() or db.syncAll().
• Stale reads after failover — The fallback driver may have older data. Use db.syncAll() after the primary recovers.
• SQLITE_BUSY / database is locked — Another writer held the SQLite lock longer than the retry budget. Confirm one local filesystem (not NFS), look for long migrations/external SQLite tools, then raise busyTimeoutMs / busyRetry.maxAttempts only for transient contention. Sustained multi-writer load needs a client-server DB.
• DeepBase timeout did not stop a SQLite write — Promise.race() ignores the original promise; better-sqlite3 waits synchronously up to busyTimeoutMs and cannot be interrupted by writeTimeout. Bound lock waits on SqliteDriver, not only on DeepBase timeouts.

SqliteDriver

Pragma modes

SqliteDriver accepts a pragma option that controls performance vs. durability:

| Mode | When to use | |------|-------------| | none | Opening a database created by an older version of the driver (no WAL, no WITHOUT ROWID) | | safe | Apps where data integrity matters more than speed (WAL + synchronous=FULL) | | balanced (default) | Recommended for most apps — fast writes with WAL + synchronous=NORMAL | | fast | Maximum throughput — risk of data loss on OS crash (synchronous=OFF) |

Lock waiting is configured separately via busyTimeoutMs (also applies to pragma: 'none').

// Default (balanced) — just omit pragma
new SqliteDriver({ path: './data', name: 'app' })

// Explicit mode
new SqliteDriver({ path: './data', name: 'app', pragma: 'fast' })

// Tune transient same-host writer contention
new SqliteDriver({
  path: './data',
  name: 'app',
  busyTimeoutMs: 5000,
  busyRetry: { maxAttempts: 2, baseDelayMs: 25, maxDelayMs: 250 },
})

Defaults: busyTimeoutMs: 5000, busyRetry: { maxAttempts: 2, baseDelayMs: 25, maxDelayMs: 250 } (~two native waits plus short async backoff).

Multi-process concurrency (same host)

There is no object singleton anymore. Each SqliteDriver owns an independent connection and lifecycle; disconnecting one instance does not close another.

Same-host writers are coordinated by SQLite:

1. Writes use BEGIN IMMEDIATE (lock acquired before reads/callbacks).
2. Contending writers wait up to busyTimeoutMs per attempt.
3. Failed busy transactions retry with exponential backoff (busyRetry).
4. Exhausted budget surfaces SQLITE_BUSY.

Reads can proceed while another process writes (WAL). SQLite still allows only one writer at a time.

// Process A and Process B may both open the same file safely on one host
const db = new DeepBase(new SqliteDriver({ path: './data', name: 'app' }));
await db.set('counter', 0);
await db.inc('counter', 1); // waits/retries if the other process holds the write lock

Limits:

• Same host + local disk only — not NFS, not multi-host shared files.
• Native lock waits block that Node.js thread for up to busyTimeoutMs.
• For sustained write contention or multi-host writers, use MongoDB/Redis/Postgres.

Maintenance

SqliteDriver exposes maintenance on the driver itself, so never add better-sqlite3 to the app's dependencies — a second copy risks a version mismatch with the driver's native binding.

const driver = db.getDriver(0);

const status = await driver.checkIntegrity();  // 'ok', or SQLite's description of the damage
await driver.backup('./backups/app-2026-07-28.db');
await driver.vacuum();
await driver.checkpoint('TRUNCATE');           // PASSIVE (default) | FULL | RESTART | TRUNCATE

checkIntegrity() and backup() open the existing source with a temporary read-only connection (fileMustExist: true), so they never create, migrate, or change PRAGMAs on it. backup() creates destination parent directories, writes an online copy, verifies it with integrity_check, and resolves with the destination path. It rejects with code DEEPBASE_SQLITE_BACKUP_CORRUPT if the copy fails to verify, leaving the bad file on disk for inspection — so a restore routine must never pick a backup by timestamp alone. The copy is consistent with concurrent writes and does not block the driver's write queue.

vacuum() and checkpoint() take the write lock, queue behind the driver's own writes, and should run when the app is idle. checkpoint() rejects with DEEPBASE_SQLITE_NOT_WAL under pragma: 'none' (rollback journal, no WAL).

Retention, rotation and scheduling belong to the application, not the driver.

Schema / seq

seq is assigned in SQL (MAX(seq)+1) under the write lock and indexed for fast ordering. Schema setup is intentionally minimal: create the table, add seq when a legacy DB lacks it, and create a non-unique index. Existing seq values are never renumbered; historical ties remain stable through ORDER BY seq, key.

When upgrading from an older DeepBase that used an in-memory sequence counter: stop all old writer processes before starting the new version. Do not mix old and new allocators during a rolling deploy.

Benchmark gains of balanced vs none: +1772% write, +2187% batch write, 29% smaller disk (compacted). All modes pass the full test suite.

References

• Repository: https://github.com/clasen/DeepBase
• npm: https://www.npmjs.com/package/deepbase
• Examples: examples/ directory in the repository (01 through 12)
• Monorepo packages: packages/core, packages/driver-json, packages/driver-sqlite, packages/driver-mongodb, packages/driver-redis, packages/driver-redis-json, packages/driver-indexeddb