license: public domain CC0

Vision and design: Versioned filesystem for modern workloads

This is a design for a VMS‑style automatically versioned filesystem reimagined for:

SQLite and other page‑oriented databases
streaming and append‑heavy workloads
cloud‑local hybrid operation
semantic, provenance‑rich development environments

The core move: version at the block/page layer, not at the “whole file” layer, and expose a VMS‑like versioned namespace as a projection over an immutable, content‑addressable store.

1. Vision

1.1 What we want

A filesystem where:

Every file has a versioned history by default (like VMS foo.txt;17).
Versions are cheap to create and store, even for large, mutable files.
SQLite databases and similar workloads get transaction‑consistent snapshots without hacks.
Streaming and append‑heavy workloads don’t explode the version count.
The same underlying store can back:
- local development
- cloud sync
- semantic code storage
- time‑travel debugging and provenance

1.2 Core principles

Immutability at the block level: once written, blocks never change.
Copy‑on‑write structure: new versions share blocks with old ones.
Semantic version boundaries: versioning is aligned with meaningful events (fsync, close, WAL checkpoint, explicit commit), not every write.
Content addressability: blocks are identified by hash, enabling deduplication and integrity.
Namespace as projection: the “filesystem” is a view over a versioned object graph, not the primary storage primitive.

2. Goals and non‑goals

2.1 Goals

Automatic versioning: every file has a history without application changes.
SQLite‑friendly: support efficient, consistent snapshots of SQLite DBs and similar page‑oriented stores.
Streaming‑friendly: support long‑lived, append‑heavy files without pathological version growth.
Cloud‑ready: support remote block storage, local caching, and multi‑device sync.
Debuggable and explainable: versioning rules and boundaries are explicit and inspectable.

2.2 Non‑goals (for v1)

Not a full POSIX replacement: we target a FUSE‑style or OS‑integrated filesystem, but we can initially accept some edge‑case incompatibilities.
Not a distributed consensus system: multi‑writer concurrency is handled via version branching and merge semantics, not strong global transactions.
Not a full Git replacement: we can integrate with Git, but we’re not re‑implementing its UX.

3. Core concepts

3.1 Block store

Primitive: fixed‑size blocks (e.g., 4 KB or 8 KB).

ID: block_id = hash(block_contents)
Properties:
- immutable
- content‑addressed
- deduplicated
- stored in local cache + optional remote store

3.2 File version

A file version is a logical object:

FileVersion {
  file_id        // stable identity for the logical file
  version_id     // monotonically increasing or hash-based
  parent_version // optional, for history/branching
  path_at_time   // path in the namespace when this version was created
  block_list     // ordered list of block_ids
  size           // byte length
  metadata       // timestamps, permissions, etc.
  tags           // optional semantic labels (e.g., "checkpoint", "autosave")
}

Multiple FileVersions share blocks via block_list.

3.3 Namespace

The live filesystem view maps paths to a current version:

Path -> FileVersion(version = "latest")

Historical versions are accessible via extended syntax, e.g.:

foo.txt;17
foo.txt;latest
foo.txt;timestamp:2026-04-10T23:17Z
foo.txt;version:abc123

Internally, this is just a lookup into the version metadata store.

3.4 Version boundary

A version boundary is an event that causes a new FileVersion to be committed:

close()
fsync()
SQLite WAL checkpoint
explicit ioctl / API call
time‑based or size‑based thresholds (e.g., at most 1 version per second per file)

Between boundaries, writes mutate an uncommitted working state (in memory or temp structures) that is not yet a committed version.

4. Architecture

4.1 High‑level components

Block store
- Local block cache (disk)
- Optional remote block store (cloud)
- Content‑addressable, immutable
Metadata store
- File identities (file_id)
- File versions (FileVersion)
- Directory structure and path mapping
- Indices for lookup by path, time, tags
Versioned filesystem layer
- Implements POSIX‑like operations
- Maintains working state for open files
- Decides when to commit new versions
Sync and replication layer
- Push/pull blocks and metadata to/from remote
- Conflict detection and version branching
Semantic layer (optional, higher level)
- Tags versions with semantic events (e.g., “test passed”, “build succeeded”)
- Integrates with tools like Git, CI, editors, agents

4.2 Data flow: write path

Application opens foo.db.
Filesystem resolves file_id and current FileVersion.
Writes go to a working file state:
- In memory or temp file
- Tracked as a list of modified blocks
On version boundary (e.g., fsync, WAL checkpoint, close):
- Compute hashes for modified blocks
- Store new blocks in block store (if not already present)
- Construct new FileVersion with updated block_list
- Update namespace mapping for foo.db → new version

4.3 Data flow: read path

Application opens foo.db (or foo.db;17).
Filesystem resolves the requested FileVersion.
Reads map file offsets to block indices.
Blocks are fetched from:
- Local block cache if present
- Otherwise remote store, then cached locally

5. Versioning semantics by workload

5.1 Regular files (text, code, configs)

Default behavior:

New version on:
- close()
- fsync()
- explicit versioning call
Optional throttling:
- If a file is opened/closed rapidly, coalesce versions within a time window.

This gives a natural history of edits without overwhelming storage.

5.2 SQLite and page‑oriented databases

SQLite is special and important.

Key properties:

Writes in fixed‑size pages.
Uses WAL mode for concurrency and durability.
Checkpoints consolidate WAL into the main DB file.

Design:

Track SQLite DBs explicitly (by extension, magic bytes, or configuration).
Treat WAL checkpoints as version boundaries:
- Before checkpoint: DB is at version N.
- After checkpoint: DB is at version N+1.
Optionally, treat WAL segments themselves as versioned objects for finer‑grained time travel.

Benefits:

Each version is transaction‑consistent.
No version spam from individual page writes.
Snapshots are cheap: only changed pages create new blocks.

5.3 Streaming and append‑heavy files

Examples: logs, video recording, long‑running data streams.

Problems to avoid:

Creating a new version for every append.
Keeping infinite history for unbounded streams.

Design:

Maintain a live append version:
- Writes extend the current version’s block_list.
Version boundaries:
- On close()
- Periodic checkpoints (e.g., every N MB or N seconds)
Retention policy:
- Keep only the last K checkpoints or last T time window.
- Older versions can be garbage‑collected or compacted.

This keeps the version tree manageable while still enabling time‑bounded rewind.

6. Namespace and UX

6.1 Path and version syntax

Expose a VMS‑inspired syntax while remaining POSIX‑compatible.

Default path: foo.txt → latest version.
Explicit version: foo.txt;17
By timestamp: foo.txt;@2026-04-10T23:17Z
By label/tag: foo.txt;tag:checkpoint

Internally, these resolve to specific FileVersion objects.

6.2 Tools and introspection

Provide tools to explore history:

vls foo.txt → list versions with timestamps, sizes, tags.
vcat foo.txt;17 → print specific version.
vdiff foo.txt;17 foo.txt;23 → diff two versions.
vmeta foo.txt;17 → show metadata and block structure.

These tools make the system explainable and debuggable.

7. Consistency, concurrency, and branching

7.1 Single‑writer per file (v1 assumption)

For simplicity, assume:

At any moment, a given file_id has at most one active writer.
Concurrent writes from multiple processes are serialized by the filesystem layer.

This matches typical local FS semantics and avoids distributed locking.

7.2 Multi‑device / multi‑replica

When multiple devices modify the same logical file:

Each device creates its own FileVersion chain.
On sync, if two new versions share the same parent, we have a branch.
Branches can be:
- kept as parallel histories
- merged via higher‑level tools (e.g., text merge, DB merge)
- resolved by user choice

The filesystem itself remains neutral: it records divergent histories; it doesn’t auto‑merge semantics.

7.3 Atomicity and durability

A FileVersion is either fully committed (all blocks stored, metadata updated) or not visible.
Crash during commit:
- Blocks may be present, but metadata not updated → GC can reclaim or finalize.
Durability guarantees depend on:
- local fsync to block store
- remote sync policy

8. Storage, GC, and retention

8.1 Storage growth

Storage grows with:

number of unique blocks
number of versions
retention policy

Because blocks are content‑addressed and shared:

Repeated edits that reuse content are cheap.
Large files with small changes are efficient (only changed blocks are new).

8.2 Garbage collection

GC operates at the block and version levels.

Version GC:
- Apply retention policies:
  - keep last N versions
  - keep versions newer than T
  - keep tagged versions (e.g., “checkpoint”, “release”)
- Delete metadata for expired versions.
Block GC:
- Periodically scan for blocks not referenced by any remaining FileVersion.
- Delete unreferenced blocks from local and/or remote stores.

8.3 Tiered storage

Support multiple storage tiers:

Hot: local SSD cache for recent blocks and metadata.
Warm: remote object store (e.g., S3‑like).
Cold: archival storage or compressed packfiles.

Policies can move blocks between tiers based on age, access frequency, or tags.

9. Integration and migration

9.1 Integration with existing tools

Git:
- Map Git commits to snapshots of a directory tree.
- Store Git objects in the same block store for deduplication.
- Allow git checkout to be implemented as a cheap namespace projection.
Editors and IDEs:
- Provide an API to tag versions with semantic events (save, build, test).
- Allow time‑travel debugging by mapping source versions to runtime traces.
CI/CD:
- Pin builds to specific filesystem versions.
- Reproduce builds by re‑mounting the same versioned tree.

9.2 Migration path

Start as a user‑space filesystem (FUSE or equivalent).
Allow mounting a directory as versioned storage.
Existing applications can run unmodified:
- SQLite uses the filesystem as usual.
- Logs and streams write to regular files.

Over time, add:

OS‑level integration (kernel module).
Tooling and UI for history exploration.
Cloud sync and multi‑device support.

10. Open questions and extensions

10.1 Policy configuration

How configurable should version boundaries be per file/path?
Do we expose a policy language (e.g., “for *.db, version on WAL checkpoint; for logs/, checkpoint every 10 MB”)?

10.2 Semantic tagging

How deeply do we integrate with higher‑level semantics (tests, builds, deployments)?
Do we treat semantic events as first‑class objects linked to FileVersions?

10.3 Security and encryption

Per‑block encryption with keys managed per user or per project.
Integrity verification via hashes is already built‑in.

10.4 Observability

Expose metrics:
- version creation rate
- block deduplication ratio
- storage usage by tier
Provide tracing for:
- version commit paths
- sync operations
- GC cycles

Saturday, April 18, 2026

we make our own hell

Sunday, April 12, 2026

security is a four letter lie

Friday, April 10, 2026

vms homage