Skip to content

feat: introduce expiresInSeconds to .set() - #3

Merged
john-doherty merged 12 commits into
masterfrom
expiresInSeconds
May 12, 2026
Merged

feat: introduce expiresInSeconds to .set()#3
john-doherty merged 12 commits into
masterfrom
expiresInSeconds

Conversation

@john-doherty

@john-doherty john-doherty commented May 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds optional TTL support to stower and rewrites the disk I/O layer for multi-process safety. This is a semver-major release (v2.1.0 → v3.0.0) because the set() signature changes and the on-disk JSON format now includes a reserved __expires__ key.

What changed

TTL / expiry (expiresInSeconds)

  • set(key, value, expiresInSeconds) accepts an optional 3rd argument — a TTL in seconds after which the key becomes invisible to reads
  • Expired keys are automatically pruned from the JSON file on the next write cycle
  • Re-setting a key without a TTL clears any existing expiry
  • __expires__ is a reserved key in the JSON file used to persist TTL timestamps — blocked from user use
  • All read APIs (get, exists, keys, values) respect expiry

Multi-process safety overhaul

  • Replaced the hand-rolled .lock file mechanism with proper-lockfile for robust cross-process mutual exclusion
  • Introduced dirty-key tracking (_dirty map) — each process tracks which keys it changed since the last flush. On write, only dirty keys are merged on top of the current disk state, so concurrent writers don't clobber each other
  • load() now re-reads the file only when its mtime has changed and re-applies dirty keys on top of the freshly loaded state, preventing a reload from discarding pending in-memory writes
  • Per-process temp files (<file>.<pid>.tmp) replace the shared .tmp file to avoid cross-process collisions
  • Orphaned temp files from crashed processes are cleaned up on startup

persist() is now synchronous

  • The directory-readiness retry loop uses a blocking sleepSync() instead of setTimeout, so persist() completes before returning. This simplifies startup in Docker environments where volume mounts may appear slightly after process start.

SIGTERM handler

  • process.on('SIGTERM', ...) added alongside the existing SIGINT handler so docker stop flushes pending writes

Files changed

File Change
index.js Full rewrite — TTL state, dirty-key tracking, proper-lockfile, mtime-based reload, atomic per-PID temp files, mergeAndSerialize / commitToDisk split
tests/expires.spec.js New — 14 tests covering TTL lifecycle, edge cases (0, negative, re-set), __expires__ reservation, disk pruning
tests/multiprocess.spec.js New — 6 tests forking real child processes to verify lock contention, concurrent writes, clear-vs-write races, TTL merging, and 10-process stress
tests/helpers/worker.js New — child process worker used by multiprocess tests
tests/methods.spec.js Expanded — added tests for external-write pickup, merge behavior, dirty-key preservation, corrupt-file recovery, file-deleted resilience
README.md Updated usage example with TTL, updated method table, added multi-process safety note
package.json v3.0.0, added proper-lockfile as a production dependency
.gitignore Ignore runtime cache/*.json files from tests

How to review

  1. Start with the public API in index.js (set, get, remove, exists, keys, values, clear) — each is intentionally short
  2. Then read the disk I/O section — load(), reapplyDirtyKeys(), mergeAndSerialize(), commitToDisk(), write()
  3. Check tests/expires.spec.js for TTL behavior and tests/multiprocess.spec.js for concurrency guarantees

@john-doherty john-doherty self-assigned this May 12, 2026

This comment was marked as outdated.

Comment thread tests/helpers/worker.js

This comment was marked as outdated.

@orca-luke

Copy link
Copy Markdown
Contributor

One thing I've noticed reading this - just as an initial comment, is that this code is incredibly hard to follow. There are a lot of single-letter variable names. Reading this with absolutely zero context is incredibly difficult.

@Rick-Velthuis

Rick-Velthuis commented May 12, 2026

Copy link
Copy Markdown

Code read, I do agree that single char keys are making it slightly harder to read (my own code is usually more messy), but all in all the code is quite easy to follow if you ask me. I have ran the Jasmine tests a few times locally, every single time at least one or more tests fail, inconsistent result.

@orca-luke

orca-luke commented May 12, 2026

Copy link
Copy Markdown
Contributor

Stress tested stower for concurrent/container reliability. Found that when 8+ processes flush to disk at the same time (e.g. rolling deploys, container restarts), some writes are silently dropped. The lock retry budget (~1.5s) gets exhausted and write() just returns without saving - no error. This will only happen above ~5-7 concurrent disk writes/s, which if we're sure we'll hit this, means we will run into this. Consistently reproduced: 350/400 keys persisted with 8 workers.

Happy to rewrite the script I wrote into a jasmine test if you'd find this beneficial

@john-doherty
john-doherty marked this pull request as ready for review May 12, 2026 15:51
@john-doherty
john-doherty merged commit 5af3ae8 into master May 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants