Skip to content

add json lib #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Easton97-Jens wants to merge 1 commit into
base: main
Choose a base branch
from
Open

add json lib #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
197 changes: 103 additions & 94 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,153 +5,162 @@ Welcome to the `jsonbench` documentation.

## Description

The purpose of this tool is to compare the performance of different JSON C libraries and to show what to expect from a possible alternative JSON processor. For more details, take a look at the tool's source. Currently, mod_security2 and libmodsecurity3 use YAJL for JSON processing. Unfortunately, this library is deprecated and is no longer supported by its maintainer.
The purpose of this tool is to compare the performance of different JSON C/C++ libraries and to show what to expect from a possible alternative JSON processor.

Note that mod_security2 processes JSON and creates keys and values. Keys are the names of the JSON keys and values are their values. If the JSON object is an array, the key will be the string `array`. For multidimensional arrays, the key is `array.array.....`.
Currently, mod_security2 and libmodsecurity3 use YAJL for JSON processing. Unfortunately, this library is deprecated and no longer maintained.

## Requirements

To compile the code you need the any of these JSON C libraries (hopefully this will be expand):
* yajl
* RapidJSON
* nlohmann JSON
## Supported JSON Engines

The tool currently supports the following JSON parsers:

To install them, try these commands:
* YAJL
* RAPIDJSON
* NLOHMANNJSON
* JSONC
* JANSSON
* CJSON
* JSONCPP
* JSONCONS
* SIMDJSON
* YYJSON
* GLAZE = It is only compatible with C++23 or newer and is therefore not entirely suitable for modsecurity as it is based on C++17.

You can list them via:

```bash
sudo apt install libyajl-dev libyajl2 rapidjson-dev nlohmann-json3.12-dev
src/jsonbench -h
```

Note, that RapidJSON and nlohmann JSON are "header-only" parsers, you need to install `rapidjson-dev` and `nlohmann-json3.12-dev`.
---

## Requirements

To compile the code, you need development packages (`-dev`) of the JSON libraries.

You also need `autoconf`, `automake`, `autotools-dev`, `make` and `gcc` or `clang` packages:
### Install required packages (Debian/Ubuntu)

```bash
sudo apt install autoconf automake autotools-dev make gcc
sudo apt install \
libyajl-dev \
rapidjson-dev \
nlohmann-json3-dev \
libjson-c-dev \
libjansson-dev \
libcjson-dev \
libjsoncpp-dev \
libsimdjson-dev \
libyyjson-dev \
libjsoncons-dev \
libglaze-dev
```

## Build
Note:

Download and unpack the source, and run these commands
* The `-dev` packages are required for compiling (headers, pkg-config, etc.)
* Runtime libraries (e.g. `libjsoncpp26`, `libsimdjson25`, etc.) are installed automatically

```bash
./autogen.sh
./configure
make
```
If you don't want to use any available JSON library, you can ignore it, eg:
```bash
./configure --wit-yajl=no
```
---

### Other `configure` flags:
For the complete list, please type `./configure --help`.
### Package availability note

Relevant flags:
```
--with-file-buffer-size=SIZE
Set buffer size (default: 524288 bytes (512kB))
--with-json-string-size=SIZE
Set buffer size for JSON strings (key, values)
(default: 2048 bytes (2kB))
```
I didn't want to use dynamic strings and use `malloc()`/`calloc()`, therefore I used static length buffer in every cases. With these options, you can set the buffer's size.
Some packages depend on your distribution version.

The `--with-file-buffer-size` describes how big file you want to process. The default value is 512kB.
Especially:

The `--with-json-string-size` describes how long can be a processed json key and value. The default value is 2kB. You can owervrite these values with these options. The values are in bytes.
* **libglaze-dev** is only available in newer Linux distributions
* Older systems may not provide it

## Run
You can check availability with:

To run the tool, type:
```bash
src/jsonbench -h
apt list | grep glaze
```

Or if you want to try a parser:
or

```bash
src/jsonbench -e YAJL
apt search libglaze-dev
```

or
---

### Header-only libraries

These parsers are header-only:

* RapidJSON (`rapidjson-dev`)
* nlohmann JSON (`nlohmann-json3-dev`)

They do not require linking against a library.

---

## Build

```bash
src/jsonbench -e RAPIDJSON
./autogen.sh
./configure
make
```
./configure optionally with --with-all-json=yes. This option requires all supported JSON libraries to be available and causes configuration to fail if any are missing.

Comment thread
Easton97-Jens marked this conversation as resolved.
If you want to disable a parser:

```bash
src/jsonbench -e NLOHMANNJSON
./configure --with-yajl=no
```

### Other command line flags

`src/jsonbench -h` gives a full list of available options. Here is a detailed review:
---

* `-e` sets the JSON engine (if there is any). Currently the tool supports YAJL and RAPIDJSON, so you can pass `-e YAJL` or `-e RAPIDJSON` (with capital letters).
* `-d` sets the maximum depth of the JSON strcture. The default value is 500. This option is the same as the [SecRequestBodyJsonDepthLimit](https://github.com/owasp-modsecurity/ModSecurity/wiki/Reference-Manual-(v2.x)#user-content-SecRequestBodyJsonDepthLimit) in ModSecurity. When the processor reached this limit with the processed depth, the tool terminates with error.
* `-a` sets the maximum size of arguments. The default value is 500. This option is the same as the [SecArgumentsLimit](https://github.com/owasp-modsecurity/ModSecurity/wiki/Reference-Manual-(v2.x)#secargumentslimit) in ModSecurity. When the processor reached this limit with the processed arguments, the tool terminates with error.
* `-s` keeps the tool in silence, and don't print out the processed data. This can be help to measure the performance more accurate.
## Run

### Example
Show help:

```bash
src/jsonbench -h
```
src/jsonbench -e YAJL -a 10000 -s tests/512KB.json

Time: 0.002567988 usec
```
This command processes the file `tests/512KB.json` and prints out the total time of JSON processing. Eg. this does not include the file processing.
Run with a specific parser:

```bash
src/jsonbench -e YAJL
```
src/jsonbench -e YAJL -a 10000 tests/test06.json
array.array.id: 2489651045
array.array.type: CreateEvent
array.array.actor.id: 665991
array.array.actor.login: petroav
array.array.actor.gravatar_id:
array.array.actor.url: https://api.github.com/users/petroav
array.array.actor.avatar_url: https://avatars.githubusercontent.com/u/665991?
array.array.repo.id: 28688495
array.array.repo.name: petroav/6.828
array.array.repo.url: https://api.github.com/repos/petroav/6.828
array.array.payload.ref: master
array.array.payload.ref_type: branch
array.array.payload.master_branch: master
...
array.array.public: true
array.array.created_at: 2015-01-01T15:00:01Z

Time: 0.000747715 usec
```
This command process the file `tests/test06.json` and prints out the processed keys and values. The syntax is the same as in case of mod_security2 JSON processing.

Here is a "quick" command to check the performance of three parsers:
Example:

```bash
$ src/jsonbench -s -e YAJL -a 10000 tests/test06.json; src/jsonbench -s -e RAPIDJSON -a 10000 tests/test06.json; src/jsonbench -s -e NLOHMANNJSON -a 10000 tests/test06.json
src/jsonbench -e SIMDJSON -s tests/test01.json
```

Time: 0.000092567 usec
---

## Command line options

* `-h` Show help
* `-e` Select JSON engine
* `-d` Max depth (default: 500)
* `-a` Max arguments (default: 500)
* `-s` Silent mode

Time: 0.000103500 usec
---

## Example

Time: 0.000196825 usec
```bash
src/jsonbench -s -e YAJL -a 10000 tests/test06.json
```

or if you want to see the result (note, that there is no `-s`):
Output:

```bash
$ src/jsonbench -e YAJL -a 10000 tests/test06.json; src/jsonbench -e RAPIDJSON -a 10000 tests/test06.json; src/jsonbench -e NLOHMANNJSON -a 10000 tests/test06.json
array.array.id: 2489651045
array.array.type: CreateEvent
array.array.actor.id: 665991
array.array.actor.login: petroav
array.array.actor.gravatar_id:
array.array.actor.url: https://api.github.com/users/petroav
array.array.actor.avatar_url: https://avatars.githubusercontent.com/u/665991?
array.array.repo.id: 28688495
```
Time: 0.000092567 usec
```

---

## TODO

Add more possible JSON processors.
* Improve documentation for all parsers
* Add more JSON processors

Loading