Skip to content
BranchesTags

Repository files navigation

Graph Database Cloud Benchmarking

A reproducible benchmark comparing CognoDB Cloud against four other graph databases on identical workloads, datasets, and resource constraints.

License: MIT Python 3.10+

Table of Contents

  1. Overview
  2. Databases Compared
  3. Dataset
  4. Methodology
  5. Environment
  6. Quick Start — Reproduce the Benchmark
  7. Results
  8. Charts
  9. Analysis
  10. Caveats & Fairness Disclosure
  11. Query Equivalence
  12. Project Structure
  13. License

Overview

This repository contains a fully automated benchmark suite that compares CognoDB Cloud against four other graph databases. The goal is to provide a fair, reproducible benchmark that honestly reports performance numbers, rather than trying to prove any specific database is the best.

What We Measure

Category Metrics
Data Loading Nodes/second, relationships/second, total wall-clock time
Traversals 1-hop, 2-hop, 3-hop query latency (p50, p95)
Lookups Point lookup and indexed/filtered lookup latency (p50, p95)
Aggregations COUNT and GROUP BY latency (p50, p95)
Mixed Workload Concurrent read/write throughput at 1, 10, 40 clients
Footprint Stored data size, memory usage (where observable)

Databases Compared

Database Tier Access Method vCPU RAM Storage Query Language
CognoDB Cloud Free c0 (managed) bolt+s:// 0.5 (burst) 256 MB 1 GB Cypher
Neo4j AuraDB Free (managed) neo4j+s:// Shared (unpublished) Unpublished Unpublished Cypher
Memgraph Cloud 14-day trial (managed) bolt+ssc:// Shared (trial) 1 GB In-memory Cypher
FalkorDB Docker (self-hosted) localhost:6379 0.5 (capped) 256 MB (capped) Container FS OpenCypher
ArangoDB Docker (self-hosted) localhost:8529 0.5 (capped) 256 MB (capped) Container FS AQL

Fairness note: Self-hosted databases (FalkorDB, ArangoDB) are capped via Docker resource limits (--cpus=0.5 --memory=256m) to match CognoDB's free tier. Network latency differences are documented — see Caveats.

Dataset

Source: SNAP soc-Pokec Social Network

Property Value
Original dataset 1,632,803 nodes, 30,622,564 edges
Subsample method Breadth-First Search (fixed seed for reproducibility)
Subsample size 20,000 nodes, 173,084 relationships
Format CSV (nodes.csv, edges.csv)
Node properties id, completion_percentage, gender, region, age
Relationship type FOLLOWS (directed)

The subsample is small enough to fit every free tier tested, while being large enough (~173K relationships) to produce meaningful benchmark results.

Reproducibility: Run python dataset/download_pokec.py --seed 42 to regenerate the exact same subgraph.

Methodology

Warm-Up Protocol

  • 10 warm-up iterations per workload (discarded from measurements)
  • 100 measured iterations per workload
  • Cold-start latency measured separately (first query after connection)

Statistical Reporting

  • p50 (median) and p95 latency for all read workloads
  • Mean, min, max, and p99 also recorded in raw JSON
  • Each traversal workload cycles through 100 randomly-selected start nodes (deterministic seed)

Mixed Workload

  • 80% reads (point lookups) + 20% writes (property updates)
  • Concurrency levels: 1, 10, 40 concurrent threads
  • Each level runs for 30 seconds
  • Reports: queries/second, average latency, p95 latency, error count

Ingest

  • Batch size: 500 nodes/edges per UNWIND batch
  • All databases use their native batch insertion mechanism
  • Nodes loaded first, then edges (edges require node MATCH)

Resource Parity

  • CognoDB free tier: 0.5 vCPU, 256 MB RAM, 1 GB disk
  • Docker-based databases: --cpus=0.5 --memory=256m
  • Managed services: smallest available free/trial tier (specs documented above)

Environment

Component Details
Client machine AMD64 Family 23 Model 24 (AMD Ryzen)
Python version 3.14.2
Neo4j driver 6.2.0
OS Windows 11 (10.0.26200)
Network Residential ISP; cloud databases accessed over public internet

Quick Start — Reproduce the Benchmark

Prerequisites

  • Python 3.10+
  • Docker (for FalkorDB and ArangoDB)
  • Free accounts on CognoDB Cloud, Neo4j AuraDB, Memgraph Cloud

Step 1: Clone and Install

git clone https://github.com/iinaa-eimrit/Graph-Database-Cloud-Benchmarking.git
cd Graph-Database-Cloud-Benchmarking
pip install -r requirements.txt

Step 2: Configure Credentials

cp .env.example .env
# Edit .env with your actual database credentials

Step 3: Download Dataset

python dataset/download_pokec.py

Step 4: Start Local Databases

# Linux/macOS
bash docker/start_local_dbs.sh

# Windows
powershell docker/start_local_dbs.ps1

Step 5: Run Benchmarks

# Run against all databases
python run_benchmark.py --databases all --workloads all

# Or target a specific database
python run_benchmark.py --databases cognodb --workloads traversal

Step 6: Generate Charts

python results/generate_charts.py

One-Command Run (Makefile)

make all  # install → dataset → local-dbs → run → charts

Results

Metrics show the median (p50) and 95th percentile (p95) over 100 runs, after 10 warm-up runs. FalkorDB couldn't finish the data load with the 0.5 vCPU limit, so it was excluded.

Data Loading (Ingest Throughput)

Database Nodes/sec Rels/sec Total Time (s) Load Method
ArangoDB 4,300 5,767 34.67 Driver batching (UNWIND/FOR)
CognoDB Cloud 1,235 511 354.95 Driver batching (UNWIND)
Memgraph Cloud 1,389 1,592 123.09 Driver batching (UNWIND)
Neo4j AuraDB 2,736 3,080 63.51 Driver batching (UNWIND)

Traversal Latency (ms)

Database 1-hop p50 1-hop p95 2-hop p50 2-hop p95 3-hop p50 3-hop p95
ArangoDB 47.672 51.794 n/a ¹ n/a ¹ n/a ¹ n/a ¹
CognoDB Cloud 314.433 393.187 321.278 661.057 421.872 1284.904
Memgraph Cloud 318.194 383.628 275.058 380.851 339.98 472.864
Neo4j AuraDB 108.765 120.79 113.519 204.288 153.304 331.621

¹ ArangoDB's 2-hop and 3-hop AQL queries returned empty results for the nodes we checked. This is just an issue with the sample data for ArangoDB's specific query direction, not an actual 0ms response time. See ANALYSIS.md for details.

Lookup Latency (ms)

Indexed properties on all platforms: User(id), User(age), User(region).

Database Point Lookup p50 Point Lookup p95 Indexed Lookup p50 Indexed Lookup p95
ArangoDB 47.46 51.861 48.009 49.54
CognoDB Cloud 321.904 481.095 318.213 435.745
Memgraph Cloud 268.077 374.46 276.802 368.541
Neo4j AuraDB 109.861 174.961 113.996 134.514

Aggregation Latency (ms)

Database Count p50 Count p95 Group-by p50 Group-by p95
ArangoDB 45.155 48.846 60.284 64.968
CognoDB Cloud 341.184 1029.163 339.265 448.544
Memgraph Cloud 271.53 373.343 314.058 362.465
Neo4j AuraDB 110.92 138.391 147.183 198.128

Mixed Workload (Concurrent Read/Write Throughput)

Database Concurrency Queries/sec Avg Latency (ms) p95 Latency (ms) Errors
ArangoDB 1 21.2 47.09 49.70 0
ArangoDB 10 181.5 54.99 74.60 1
ArangoDB 40 269.1 147.53 241.47 1
CognoDB Cloud 1 3.0 326.25 366.79 0
CognoDB Cloud 10 20.8 463.16 707.43 0
CognoDB Cloud 40 118.2 335.89 360.60 0
Memgraph Cloud 1 3.7 266.62 311.49 0
Memgraph Cloud 10 34.5 288.45 325.26 0
Memgraph Cloud 40 137.9 288.38 319.01 0
Neo4j AuraDB 1 8.6 116.33 139.64 0
Neo4j AuraDB 10 82.8 120.35 128.52 0
Neo4j AuraDB 40 316.5 125.84 145.68 0

Resource Footprint

Database Stored Data Size Memory Usage Instance Specs Notes
ArangoDB 20000 nodes, 173084 rels (19.75 MB) See docker stats (capped at 256 MB) Docker: --cpus=0.5 --memory=256m Self-hosted via Docker. Near-zero network latency (localhost).
CognoDB Cloud 20000 nodes, 173084 rels not observable (managed service) Free c0: 0.5 vCPU, 256 MB RAM, 1 GB disk CognoDB Cloud managed service
Memgraph Cloud 20000 nodes, 173084 rels not observable Memgraph Cloud trial: 1 GB RAM (in-memory) Memgraph is an in-memory graph DB (1 GB RAM trial)
Neo4j AuraDB 20000 nodes, 173084 rels not observable (managed service) AuraDB Free: shared resources Neo4j AuraDB Free — hard limit 200K nodes / 400K rels

Charts

Ingest Throughput

Ingest Throughput

Traversal Latency

Traversal Latency

Lookup Latency

Lookup Latency

Aggregation Latency

Aggregation Latency

Mixed Workload

Mixed Workload

Overall Comparison

Overall Comparison

Analysis

See ANALYSIS.md for a closer look at the results, including:

  • Why the databases performed differently
  • Thoughts on how different database setups (like in-memory vs on-disk) affect speed
  • How fair the free-tier comparisons actually are
  • Recommendations on which database to use when

Caveats & Fairness Disclosure

Caveat Impact Mitigation
Network delays ArangoDB runs locally so it's very fast over the network. Cloud databases add a ~250-300ms network delay per query Documented in all tables; compare cloud-vs-cloud for fairer signal
FalkorDB excluded Couldn't finish loading data with a 0.5 vCPU limit Code is still there, but it's not in the results
Memgraph RAM advantage Memgraph trial has 1 GB RAM vs. CognoDB's 256 MB Noted in specs table; Memgraph is in-memory by design
Neo4j AuraDB specs unknown Neo4j doesn't publish free-tier hardware Documented as "unpublished"; likely has more resources than CognoDB
AQL ≠ Cypher ArangoDB uses a different query language Queries are logically equivalent; side-by-side documented below
ArangoDB deep traversals empty 2-hop and 3-hop AQL queries returned empty results for sampled nodes Logged as n/a in results; 1-hop data is valid
Free-tier throttling Cloud services may throttle free tiers unpredictably p95 reported to capture tail latency and throttling effects
Single client location Client machine is a residential Windows PC, not a cloud VM Same client used for all databases; relative comparisons are valid
Small dataset 20K nodes / 173K edges is small relative to production Required to fit all free tiers

Query Equivalence

All databases execute logically identical workloads. Here are the exact queries used:

Workload Cypher (CognoDB / Neo4j / Memgraph / FalkorDB) AQL (ArangoDB)
1-hop MATCH (n:User {id: $id})-[:FOLLOWS]->(m) RETURN m.id FOR v IN 1..1 OUTBOUND "users/$id" follows RETURN v.id
2-hop MATCH (n:User {id: $id})-[:FOLLOWS*2]->(m) RETURN DISTINCT m.id FOR v IN 2..2 OUTBOUND "users/$id" follows OPTIONS {uniqueVertices: "global"} RETURN DISTINCT v.id
Point lookup MATCH (n:User {id: $id}) RETURN n FOR u IN users FILTER u.id == $id RETURN u
Indexed lookup MATCH (n:User) WHERE n.age >= $x AND n.age <= $y RETURN n LIMIT 100 FOR u IN users FILTER u.age >= $x AND u.age <= $y LIMIT 100 RETURN u
Count MATCH (n:User) RETURN count(n) RETURN LENGTH(users)
Group-by MATCH (n:User) RETURN n.region, count(n) ORDER BY count(n) DESC FOR u IN users COLLECT region = u.region WITH COUNT INTO cnt SORT cnt DESC RETURN {region, cnt}
Write MATCH (n:User {id: $id}) SET n.prop = $val UPDATE {_key: $id} WITH {prop: $val} IN users

Project Structure

cognodb-benchmark/
├── README.md                          # This file
├── ANALYSIS.md                        # Detailed results analysis
├── LICENSE                            # MIT License
├── Makefile                           # One-command automation
├── requirements.txt                   # Pinned Python dependencies
├── .env.example                       # Credentials template
├── .gitignore
│
├── dataset/
│   ├── download_pokec.py              # Download & BFS subsample script
│   ├── pokec_nodes.csv                # Generated (not committed)
│   └── pokec_edges.csv                # Generated (not committed)
│
├── benchmarks/
│   ├── __init__.py
│   ├── config.py                      # Env-based configuration
│   ├── stats.py                       # Percentile math & result types
│   ├── base_benchmark.py             # Abstract base class
│   ├── workloads.py                   # Query definitions & docs
│   ├── runner.py                      # Benchmark runner
│   └── drivers/
│       ├── __init__.py
│       ├── cognodb_driver.py          # CognoDB Cloud driver
│       ├── neo4j_driver.py            # Neo4j AuraDB driver
│       ├── memgraph_driver.py         # Memgraph Cloud driver
│       ├── falkordb_driver.py         # FalkorDB driver
│       └── arangodb_driver.py         # ArangoDB driver
│
├── docker/
│   ├── docker-compose.yml             # FalkorDB + ArangoDB with resource caps
│   ├── start_local_dbs.sh             # Linux/macOS startup script
│   └── start_local_dbs.ps1            # Windows startup script
│
├── results/
│   ├── generate_charts.py             # Chart generation script
│   ├── summary.md                     # Generated results summary
│   ├── summary.json                   # Generated combined JSON
│   ├── raw/                           # Per-database JSON results
│   └── charts/                        # Generated PNG charts
│
└── run_benchmark.py                   # CLI entry point

License

MIT License — see LICENSE.

About

Reproducible cross-database benchmark of cloud and self-hosted graph databases using controlled workloads and p50/p95 latency analysis.

Topics

benchmarkingcypherdata-engineeringdockergraph-databaseneo4jperformance-testingpython

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages