Read causality-network dynamics from Parquet via DuckDB

[jussty]

What & why

ecoli/analysis/causality_network/read_dynamics.py::convert_dynamics() was a stub:

# TODO: Convert to use DuckDB
raise NotImplementedError("Still need to convert to use DuckDB!")

so buildCausalityNetwork.py could not produce a seriesOut.zip from current vEcoli (Parquet) output. This ports the dynamics reader to the Parquet/DuckDB backend.

Split into two commits so the second is easy to drop:

1. Core port — the DuckDB dynamics reader + vEcoli-layout CLI options.
2. Optional downsampling (--max_timepoints) — separable; safe to leave out if you'd rather always read full resolution.

Commit 1 — core port

read_dynamics.py

• Read the required listener columns + bulk from the Parquet history dataset in one time-ordered pass via dataset_sql(); convert list columns with ndlist_to_ndarray(); build the nested timeseries dict the existing node readers expect.
• Replace the old config["state"][...]["_properties"]["metadata"] lookups (bulk-molecule ids, unprocessed-RNA ids) with field_metadata(conn, config_sql, ...).
• Keep the existing reshaping + seriesOut.zip writer unchanged.
• Remove array_timeseries() and its get_value_from_path import (only used by the removed DB path).

buildCausalityNetwork.py

• New CLI options for the vEcoli layout: --sim_data_path, --out_dir, --id (experiment id), --seriesout — replacing the hardcoded wcEcoli-era out/kb/simData.cPickle / out/seriesOut.

Commit 2 — optional downsampling

• Adds --max_timepoints (and a max_timepoints arg to convert_dynamics) to evenly downsample the dynamics read. Some listeners are 2-D per timestep (e.g. TF binding is cistrons × TFs), so full-resolution arrays for a multi-thousand-step run can be large; this keeps memory bounded on constrained machines. Kept as a separate commit so it can be dropped independently.

Testing

Ran a single-generation simulation, then:

python ecoli/analysis/causality_network/buildCausalityNetwork.py \
  --sim_data_path OUT/EXP/parca/kb/simData.cPickle \
  --out_dir OUT --id EXP --seriesout OUT/EXP/causality

Produces a valid seriesOut.zip (nodes.json, edges.json, series.json + per-node series/*.json).

Notes

• Fixes data generation only; the interactive viewer (CovertLab/causality) is a separate repo.
• Draft for maintainer review — happy to expose max_timepoints via config or wire this into runscripts/analysis.py if you'd prefer.