Skip to content
Merged

V3 #23

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
41156cf
feat: removed not reliable dev deps, reorganized tests & moved to nod…
SerhiyGreench Jul 6, 2026
2f7ed21
fix:: added files
SerhiyGreench Jul 6, 2026
727410e
feat: refactored RedisQueue & fixed inspected issues
SerhiyGreench Jul 6, 2026
362d7f6
fix:: added files
SerhiyGreench Jul 6, 2026
430d91f
fix: issues after inspections in clustered queue implementation, type…
SerhiyGreench Jul 6, 2026
9dd7069
fix: formatting
SerhiyGreench Jul 6, 2026
5fc2a53
fix: test coverage report
SerhiyGreench Jul 6, 2026
0c8e73e
fix: tests
SerhiyGreench Jul 6, 2026
30f99f4
fix: github actions workflow
SerhiyGreench Jul 6, 2026
0edb13e
fix: improve tests coverage
SerhiyGreench Jul 6, 2026
99e710f
fix: added files
SerhiyGreench Jul 6, 2026
3f25ad4
chore: refreshed package-lock.json
SerhiyGreench Jul 6, 2026
325bcb6
fix: inspections on udp cluster manager
SerhiyGreench Jul 6, 2026
e8d4ad8
fix: added files
SerhiyGreench Jul 6, 2026
f49c058
fix: UDP CLuster Manager exit issues
SerhiyGreench Jul 6, 2026
bcc8785
fix: remove await writes option
SerhiyGreench Jul 6, 2026
aac74a7
chore: docs, coverage and build hygiene
SerhiyGreench Jul 7, 2026
3e5a0e5
feat: dual-mode @profile decorator and restore uuid() export
SerhiyGreench Jul 7, 2026
9c0ee71
fix: removed uuid export
SerhiyGreench Jul 7, 2026
088ef08
ci: run a plain test check, drop Coveralls upload
SerhiyGreench Jul 7, 2026
74e63ea
fix: aligned with PR comments
SerhiyGreench Jul 8, 2026
c9bd151
test: reach 100% coverage; node: imports; mockBuiltin; PR review fixes
SerhiyGreench Jul 8, 2026
f7c7290
test: fix UDPClusterManager spec CI timeout
SerhiyGreench Jul 8, 2026
beab476
3.0.0
SerhiyGreench Jul 8, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .codebeatignore

This file was deleted.

7 changes: 0 additions & 7 deletions .codebeatsettings

This file was deleted.

42 changes: 38 additions & 4 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,49 @@ name: Build

on:
push:
branches: [master]
tags: ['*']
pull_request:

permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
build:
quality:
name: Lint & format
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Use Node.js
uses: actions/setup-node@v4
with:
node-version: 24.x
cache: npm

- name: Install dependencies
run: npm ci

- name: Check formatting
run: npm run format:check

- name: Lint
run: npm run lint

test:
name: Test (Node ${{ matrix.node-version }})
runs-on: ubuntu-latest
timeout-minutes: 15
strategy:
fail-fast: false
matrix:
node-version: [lts/*]

node-version: [22.x, 24.x]
steps:
- name: Checkout repository
uses: actions/checkout@v4
Expand All @@ -20,9 +53,10 @@ jobs:
uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node-version }}
cache: npm

- name: Install dependencies
run: npm install
run: npm ci

- name: Run tests
run: npm test
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ tmp/
out-tsc/
build/
.nyc_output/
.agent-out/

node_modules/

Expand Down
9 changes: 9 additions & 0 deletions .oxfmtrc.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"singleQuote": true,
"tabWidth": 4,
"printWidth": 80,
"semi": true,
"trailingComma": "all",
"arrowParens": "avoid",
"bracketSpacing": true
}
20 changes: 20 additions & 0 deletions .oxlintrc.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"$schema": "./node_modules/oxlint/configuration_schema.json",
"categories": {
"correctness": "error"
},
"rules": {
"no-debugger": "error",
"no-console": ["warn", { "allow": ["warn", "error", "info"] }],
"no-unused-vars": "warn"
},
"ignorePatterns": [
"**/*.js",
"**/*.d.ts",
"docs/",
"coverage/",
".nyc_output/",
".agent-out/",
"node_modules/"
]
}
136 changes: 73 additions & 63 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,63 +2,63 @@

[![License](https://img.shields.io/badge/license-GPL-blue.svg)](https://rawgit.com/imqueue/core/master/LICENSE)

Simple JSON-based messaging queue for inter service communication
Simple JSON-based messaging queue for inter-service communication

**Related packages:**

- [@imqueue/rpc](https://github.com/imqueue/rpc) - RPC-like client/service
implementation over @imqueue/core.
- [@imqueue/cli](https://github.com/imqueue/cli) - Command Line Interface
for imqueue.
- [@imqueue/rpc](https://github.com/imqueue/rpc) - RPC-like client/service
implementation over @imqueue/core.
- [@imqueue/cli](https://github.com/imqueue/cli) - Command Line Interface
for imqueue.

# Features

With current implementation on RedisQueue:

- **Fast unwarranted** message delivery (if consumer grab the message and dies
message will be lost). Up to ~35-40k of 1Kb messages per second on i7 core by
benchmarks.
- **Fast warranted** message delivery (only 1.5-2 times slower than
unwarranted). If consumer grab a message and dies it will be re-scheduled
in queue. Up to ~20-25K of 1Kb messages per second on i7 core by benchmarks.
- **No timers or constant redis polling** used for implementation, as result -
no delays in delivery and low CPU usage on application workers. When idling
it does not consume resources!
- **Supports gzip compression for messages** (decrease traffic usage, but
slower).
- **Concurrent workers model supported**, the same queue can have multiple
consumers.
- **Delayed messages supported**, fast as ~10K of 1Kb messages per second on i7
core by benchmarks.
- **Safe predictable scaling of queues**. Scaling number of workers does not
influence traffic usage.
- **Round-robin message balancing between several redis instances**. This
allows easy messaging queue redis horizontal scaling.
- **TypeScript included!**
- **Fast, unreliable** message delivery (if a consumer grabs the message and dies,
the message will be lost). Up to ~35–40k of 1Kb messages per second on an i7 core
by benchmarks.
- **Fast, guaranteed** message delivery (only 1.5–2 times slower than
unreliable mode). If a consumer grabs a message and dies, it will be rescheduled
to the queue. Up to ~20–25k of 1Kb messages per second on an i7 core by benchmarks.
- **No timers or constant Redis polling** used for implementation, resulting in
no delays in delivery and low CPU usage on application workers. When idle,
it consumes no resources.
- **Supports Gzip compression for messages** (decreases traffic usage but is slower).
- **Concurrent workers model supported**, the same queue can have multiple
consumers.
- **Delayed messages supported**, fast as ~10K of 1Kb messages per second on i7
core by benchmarks.
- **Safe, predictable scaling of queues**. Scaling the number of workers does not
increase traffic usage.
- **Round-robin message balancing between multiple Redis instances**. This
allows easy horizontal scaling of the messaging queue across Redis instances.
- **TypeScript included!**

# Requirements

Currently this module have only one available adapter which is Redis server
related. So redis-server > 3.8+ is required.
Currently this module has only one available adapter, which is Redis. Redis server
6.2+ is required (the queue relies on `LMOVE`/`BLMOVE` commands for safe message
delivery).

If config command is disabled on redis it will be required to turn on manually
keyspace notification events (actual on use with ElasticCache on AWS), like:
If the config command is disabled on Redis, you must manually enable keyspace
notification events (particularly when using AWS ElasticCache), like this:

~~~
```
notify-keyspace-events Ex
~~~
```

Further, more adapters will be added... if needed.
More adapters will be added in the future as needed.

# Install

~~~bash
```bash
npm i --save @imqueue/core
~~~
```

# Usage

~~~typescript
```typescript
import IMQ, { IMessageQueue, IJson } from '@imqueue/core';

(async () => {
Expand Down Expand Up @@ -90,27 +90,27 @@ import IMQ, { IMessageQueue, IJson } from '@imqueue/core';
const delay = 1000;
await queueOne.send('QueueOne', { delay }, delay);
})();
~~~
```

# Benchmarking

First of all make sure redis-server is running on the localhost. Current
version of benchmark supposed to have redis running localhost because
it is going to measure it's CPU usage stats.
First, make sure redis-server is running on localhost. The current version of the
benchmark requires Redis to be running on localhost so it can measure its CPU
usage statistics.

All workers during benchmark test will have their dedicated CPU affinity
to make sure collected stats as accurate as possible.
All workers during the benchmark test will have dedicated CPU affinity to ensure
the collected statistics are as accurate as possible.

~~~bash
git clone git@github.com:Mikhus/core.git
cd imq
```bash
git clone git@github.com:imqueue/core.git
cd core
node benchmark -c 4 -m 10000
~~~
```

Other possible benchmark options:

~~~
node benchmark -h
```
node benchmark -h
Options:
--version Show version number [boolean]
-h, --help Show help [boolean]
Expand All @@ -128,30 +128,40 @@ Options:
-p, --port Redis server port to connect to.
-t, --message-multiply-times Increase sample message data given number of
times.
~~~
```

Number of child workers running message queues are limited to a max number
of CPU in the system -2. First one, which is CPU0 is reserved for OS tasks and
for stats collector process. Second, which is CPU1 is dedicated to redis
process running on a local machine, All others are safe to run queue workers.
The number of child workers running message queues is limited to the number of CPUs
in the system minus 2. The first CPU (CPU0) is reserved for OS tasks and the stats
collector process. The second CPU (CPU1) is dedicated to the local Redis process.
All others are available to run queue workers.

For example, if there is 8 cores on a machine it is safe to run up to 6 workers.
For 4-core machine this number is limited to 2.
If there is less cores results will not give good visibility of load.
For example, on an 8-core machine you can safely run up to 6 workers. On a 4-core
machine, this limit is 2 workers. If there are fewer cores, the results will not
provide good visibility of the load.

*NOTE: paragraphs above this note are Linux-only related! On MacOS
there is no good way to set process affinity to a CPU core, Windows support is
not tested and is missing for the moment in benchmarking. I does not mean
benchmark will not work on Mac & Win but the results won't be accurate and
predictable.*
**NOTE:** The paragraphs above apply to Linux only. On macOS there is no reliable way
to set process CPU affinity, and Windows support is not currently implemented for
benchmarking. This does not mean the benchmark won't work on macOS or Windows, but
the results will not be accurate or predictable on those platforms.

## Running Unit Tests

~~~bash
Tests run on the native Node.js test runner (`node:test`) with `node:assert` and
no external test framework, so a plain clone and install is all that is needed:

```bash
git clone git@github.com:imqueue/core.git
cd imq
cd core
npm install
npm test
~~~
```

To produce a coverage report use:

```bash
npm run test-coverage # prints coverage summary to the console
npm run test-lcov # writes coverage/lcov.info
```

## License

Expand Down
8 changes: 4 additions & 4 deletions benchmark/affinity.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,11 @@
* purchase a proprietary commercial license. Please contact us at
* <support@imqueue.com> to get commercial licensing options.
*/
import { execSync as exec } from 'child_process';
import * as os from 'os';
import { execSync as exec } from 'node:child_process';
import { platform } from 'node:os';

export function setAffinity(cpu: number) {
if (os.platform() === 'linux') {
if (platform() === 'linux') {
exec(`taskset -cp ${cpu} ${process.pid}`);
}
}
}
Loading
Loading