stringmatch-bench

Tools

This repository provides benchmark tools to compare the performance of data structures for string matching.

Rust

Directory rust provides benchmarks for the following libraries:

• aho_corasick::AhoCorasick
• daachorse::DoubleArrayAhoCorasick
• daachorse::CharwiseDoubleArrayAhoCorasick
• crawdad::Trie
• crawdad::MpTrie
• yada::DoubleArray
• fst::Map
• hashbrown::HashMap

After moving your current directory to rust, you can measure the memory usage of data structures built from your dataset with the following command:

$ cargo run --release --bin measure -- -k ../data/unidic/unidic

For base64-encoded datasets such as the tiktoken vocabularies, add the -b option:

$ cargo run --release --bin measure -- -k ../data/tiktoken/o200k_base.base64 -b

You can also measure build and search times with criterion.rs:

$ cargo bench

Python

Directory python provides benchmarks for the following libraries:

• ahocorasick_rs
• daachorse

After moving your current directory to python, you can measure search times with pytest-benchmark by running nox:

$ nox

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

The datasets contained in data are copied from third-party repositories. Follow the license terms of each dataset.