# CausalSieve — Complete Technical Reference (`llms-full.txt`) > Zero-runtime-dependency Python + C++20/AVX2+FMA streaming engine for online multivariate causal discovery, exogenous root-cause localization, counterfactual `do()` verification, multi-agent blackboard consensus, and 95–99.5% token-compressed lossless-recoverable stream digests. --- ## 1. Architecture Overview CausalSieve separates fast numerical streaming math (native C99 ABI shared library `libcausalsieve.so` compiled with AVX2+FMA or portable scalar fallback) from a zero-dependency Python layer (stdlib + `ctypes`, optional zero-copy NumPy `float64` buffer passing). | Layer | File | Responsibility | Runtime Dependencies | |---|---|---|---| | Native C99 ABI | `csrc/causalsieve.h` | Tier-1 `cs_engine_t` (`D <= 128`) and Tier-2 `cs_screened_t` (`D = 128..16,384+`) | C/C++ stdlib only | | Python Native Binding | `python/causalsieve/_native.py` | Exact `ctypes` ABI, zero-copy `ndarray`/`array('d')` ingest, `CausalSieve`, `ScreenedSieve`, `analyze_csv` | Python >= 3.10 stdlib only | | Causal Digest | `python/causalsieve/digest.py` | Caveman-style Markdown/JSON digests, victim-vs-root verdicts, adaptive Swing-Filter PLA knots, `slice://` exact recovery | Python stdlib only | | Multi-Agent Blackboard | `python/causalsieve/swarm.py` | Thread-safe monotonic-watermark `CausalBlackboard`, immutable `CausalSnapshot`, append-only scored hypotheses, Brier/log consensus | Python stdlib only | | Fast CLI | `python/causalsieve/cli.py` | Lazy-import sub-20 ms CLI (`rca`, `digest`, `slice`, `counterfactual`, `info`, `mcp`) | Python stdlib only | | MCP Server | `python/causalsieve/mcp_server.py` | Spec-compliant newline-delimited JSON-RPC 2.0 stdio server with path sandboxing and 6 tools | Python stdlib only | --- ## 2. Mathematical Specification ### 2.1 Online Ridge-VAR(p) with Optional Asymmetric Hinge Basis At tick $t$, observation $x_t \in \mathbb{R}^D$ is anchored to the initial baseline (shifted-anchor compensation for $10^{10}$ DC-offset stability) and optionally causally scaled (`normalize_scale=True`). The feature vector $\phi_t \in \mathbb{R}^K$ contains intercept + $p$ lagged vectors $x_{t-1}, \dots, x_{t-p}$ (and optional asymmetric hinge terms $\max(0, x_{t-\ell} - 1)$ when `nonlinear_hinge=True`). The engine maintains exponentially forgotten sufficient statistics with forgetting factor $\lambda \in (0, 1]$ and ridge $\alpha > 0$: - Regularized Gram matrix $G_t = \lambda G_{t-1} + w_t \phi_t \phi_t^\top$ (with $\alpha I$ ridge) - Shared inverse Gram $P_t = (G_t + \alpha I)^{-1}$ updated online via Sherman-Morrison - One-step-ahead innovation $\hat{u}_t = x_t - B_{t-1}^\top \phi_t$ (strictly causal: scored using state at $t-1$ before updating!) ### 2.2 Exact Block-Schur Conditional Granger Causality For every ordered pair $(i, j) \in \{0, \dots, D-1\}^2$, let $S_i$ be the index block of channel $i$'s lags in $\phi_t$. Removing channel $i$ from the full model increases channel $j$'s residual sum of squares by the exact Block-Schur complement: $$\Delta J_{i \to j} = B_{S_i, j}^\top P_{S_i, S_i}^{-1} B_{S_i, j}$$ This computes all $D^2$ conditional Granger $F$-statistics and partial $R^2$ values in $O(D p^3 + D^2 p^2)$ without refitting any reduced models, matching exact batch ridge-VAR reduced-model refits to $< 10^{-10}$. ### 2.3 Precision-Whitened Exogenous Attribution Instead of scoring raw magnitude $z^{\text{raw}}_{j,t}$ (which misfires on high-variance downstream victims), CausalSieve maintains the online innovation precision matrix $\Theta_t = (\hat{\Sigma}_{u,t} + \gamma I)^{-1}$ and computes precision-whitened exogenous shock scores: $$z^{\text{exo}}_{j,t} = \frac{(\Theta_{t-1} \hat{u}_t)_j}{\sqrt{\Theta_{jj,t-1}}}$$ Root-cause ranking (`cs_root_attribution`) combines exogenous shock intensity, earliest onset precedence, and causal graph outflow. ### 2.4 Online Contemporaneous Orientation (Hyvärinen-Smith LiNGAM) For pairs with significant innovation partial correlation $\rho_{ij} = -\Theta_{ij} / \sqrt{\Theta_{ii}\Theta_{jj}}$, the engine estimates non-Gaussian contemporaneous direction $R_{i \to j}$ (`cs_contemporaneous_orientation`). Positive $R_{i \to j} > R_{j \to i}$ indicates instantaneous causal direction $i \to j$ under skewed or heavy-tailed shocks. ### 2.5 Shock Shield, Drift Guard, and Anytime-Valid E-Process - **Huber/Mahalanobis shock shield**: computes $m_t^2 = \hat{u}_t^\top \Theta_{t-1} \hat{u}_t$ and down-weights severe faults via Huber weight $w_t \in [0, 1]$ so anomalies do not poison baseline VAR weights. - **Freivalds drift guard**: periodically verifies $\|G_{\text{reg}}(P v) - v\|_2 / \|v\|_2$ and triggers exact Cholesky refresh if Sherman-Morrison numerical drift accumulates. - **Anytime-valid Ville E-process**: maintains nonnegative supermartingale wealth $E_t$ raising an alarm when $E_t \ge 1/\alpha$ (default $\alpha = 0.01 \implies$ threshold $100$), valid under arbitrary optional stopping. ### 2.6 Swing-Filter $L_\infty$ PLA & Lossless FP64 Ring Buffer - Each channel retains its last `raw_ring_capacity` raw FP64 observations (default `8192`). - `cs_compress_channel_pla` emits piecewise linear knots guaranteeing $|\hat{x}(t) - x(t)| \le \varepsilon$ for every tick in `[start_tick, end_tick)`. - `cs_get_raw_slice` returns bit-exact IEEE-754 FP64 values (`0.0` reconstruction error) for any retained range. --- ## 3. Installation & Library Discovery ```sh make # compiles build/libcausalsieve.so pip install -e . # installs causalsieve & causalsieve-mcp entry points (zero dependencies) ``` `causalsieve._native.load_library()` searches in order: 1. Explicit path argument (`load_library(path)`). 2. `CAUSALSIEVE_LIB` environment variable. 3. Next to `_native.py` (`python/causalsieve/libcausalsieve.so` / `.dylib` / `.dll`). 4. Repository `build/libcausalsieve.so`. 5. `./build/libcausalsieve.so` in current working directory. 6. System dynamic linker (`ctypes.util.find_library("causalsieve")`). --- ## 4. Python API Reference ### 4.1 `CausalSieve` (Tier-1 Streaming Engine) ```python from causalsieve import CausalSieve with CausalSieve( dim=4, lag=2, channel_names=["db_cpu", "api_lat", "queue_depth", "err_rate"], forgetting=0.995, ridge=1e-2, precision_ridge=1e-2, shock_threshold=4.0, e_alpha=0.01, warmup_ticks=0, refresh_interval=0, nonlinear_hinge=False, raw_ring_capacity=8192, normalize_scale=True, ) as sieve: # Single tick (list, tuple, array('d'), or zero-copy float64 numpy ndarray) res = sieve.push([0.1, 0.2, 0.0, 0.0]) # Atomic batch push (all-or-nothing NaN/Inf validation in C) batch_results = sieve.push_batch(rows, return_results=True) # Exact Block-Schur Conditional Granger (D x D, [src][dst]) granger = sieve.granger_matrix() top_lagged = granger.top_edges(min_f=3.0, min_partial_r2=0.02, top_k=10) # Innovation precision matrix Theta and partial correlations prec = sieve.precision_matrix() # Online Hyvarinen-Smith LiNGAM orientation R (D x D; R[i][j] > 0 => i -> j) lingam_r = sieve.contemporaneous_orientation() # Causal root-cause attribution attr = sieve.root_attribution(lookback=0) print(attr.top_name, attr.items[0].peak_exo_z, attr.items[0].onset_tick) # Counterfactual do-intervention: do(db_cpu += 3.0) over 10 steps traj = sieve.counterfactual_do("db_cpu", delta=3.0, horizon=10) # Lossless exact FP64 slice recovery raw = sieve.get_raw_slice("db_cpu", start_tick=480, end_tick=520) # Certified L_inf Swing-Filter PLA knots pla = sieve.compress_pla("db_cpu", start_tick=480, end_tick=520, max_abs_err=0.05) # Internal VAR(p) weights B (K x D) and Freivalds inverse residual state = sieve.inspect_state() sieve.force_refresh() # Caveman-compressed Causal Digest (auto-registers stream_id for slice:// recovery) dig = sieve.digest(top_k=5, tokens_budget=400) print(dig.to_markdown()) ``` ### 4.2 `ScreenedSieve` (Tier-2 High-Dimensional Engine, `D = 128..16,384+`) ```python from causalsieve import ScreenedSieve with ScreenedSieve(dim=2048, active_m=32, lag=2, forgetting=0.995, ridge=1e-2) as ss: batch = ss.push_batch(X_2048, top_k=10, return_all_z=False) print(batch.top_names, batch.top_scores) graph = ss.active_graph() print(graph.active_count, graph.active_names) ``` ### 4.3 `analyze_csv` (Pure-Stdlib CSV Streaming) ```python from causalsieve import analyze_csv with analyze_csv("telemetry.csv", lag=2, nonlinear_hinge=False) as res: dig = res.digest(top_k=5, tokens_budget=400) print(dig.to_markdown()) ``` `analyze_csv` uses only Python's standard `csv` module, automatically detects headers, timestamp/index columns, and non-numeric columns, causally forward-fills isolated missing cells after the first valid row, and switches to `ScreenedSieve` automatically when `dim >= screened_threshold` (default `128`). ### 4.4 Causal Digest & Lossless Slice Recovery ```python from causalsieve import build_digest, recover_slice, recover_handle dig = build_digest(sieve, top_k=5, max_err=0.05, tokens_budget=400) md = dig.to_markdown() js = dig.to_json() # Exact FP64 recovery via stream_id + channel + tick range sl = recover_slice(dig.stream_id, "db_cpu", 480, 520) # Or directly via handle printed in the digest sl2 = recover_handle(dig.recovery_handles[0]) assert sl.values == sl2.values ``` ### 4.5 `CausalBlackboard` (Multi-Agent Shared Blackboard) ```python from causalsieve import CausalBlackboard bb = CausalBlackboard() bb.create_stream("prod-west", dim=4, lag=2, channel_names=["db", "cache", "api", "web"]) bb.push_batch("prod-west", rows) # Immutable versioned snapshot at current monotonic watermark snap = bb.snapshot("prod-west") # Retrieve historical snapshot at a recorded watermark tick same_snap = bb.snapshot("prod-west", as_of_tick=snap.as_of_tick) # Concurrent agents submit append-only hypotheses (no overwrites, objectively scored) h1 = bb.verify_hypothesis("prod-west", "agent-a", "db", delta=3.0, horizon=10, confidence=0.85) h2 = bb.verify_hypothesis("prod-west", "agent-b", "web", delta=3.0, horizon=10, confidence=0.95) # Proper-scoring-calibrated consensus resists overconfident false victim proposals report = bb.consensus("prod-west") print(report.winner_name, report.winner_probability, report.agent_weights) ``` --- ## 5. CLI Reference ```sh causalsieve info [--json] causalsieve rca [--lag 2] [--top-k 5] [--format md|json] [--hinge] [--max-err 0.05] [--forgetting 0.995] [--ridge 0.01] [--lookback 0] [--no-normalize] [--no-screened] causalsieve digest [--tokens-budget 400] [--lag 2] [--top-k 5] [--format md|json] [--hinge] [--max-err 0.05] causalsieve slice --channel --start --end [--format json|csv|table] causalsieve counterfactual --channel --delta [--horizon 10] [--lag 2] [--hinge] [--format md|json] causalsieve mcp [--allowed-root ] [--max-csv-bytes ] ``` Example Causal Digest output (`causalsieve rca incident.csv`): ```markdown # CausalDigest `a3f9c012e4b8` ticks=2000 | ch=8 | lag=2 | isa=avx2+fma | alarm=ACTIVE | E=412.50 (thr=100) | drift=OK (err=1.24e-13, refreshes=0) ## Root Causes (Exogenous vs Victim) | rank | channel | root_score | peak_exo_z | raw_z | onset_tick | verdict | |---:|---|---:|---:|---:|---:|---| | 1 | db_cpu | 8.412 | +7.92 | +6.10 | 1500 | EXOGENOUS_ROOT | | 2 | api_lat | 1.320 | +1.18 | +14.80 | 1501 | DOWNSTREAM_VICTIM | > Victim trap avoided: `api_lat` has highest |raw_z|=14.80 (rank #2, exo_z=+1.18), while exogenous root is `db_cpu` (exo_z=+7.92, onset=1500). ## Causal Graph - db_cpu --[lagged F=184.20, pR2=0.412]--> api_lat - api_lat --[contemp pcorr=+0.71, LiNGAM=+0.38]--> err_rate ## PLA Knots [1484..1996) - db_cpu: [(1484, 0.12), (1499, 0.14), (1500, 6.21), (1995, 5.98)] max_err<=0.05 ## Lossless Recovery - `slice://a3f9c012e4b8/db_cpu/1484..1996` tokens: raw_est=52000 | digest_est=245 | reduction=99.53% ``` --- ## 6. MCP Stdio Server Reference Launch via `causalsieve-mcp` or `causalsieve mcp`. Transport is newline-delimited JSON-RPC 2.0 over `stdin`/`stdout` (diagnostics only on `stderr`). ### Handshake Example Request: ```json {"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"agent","version":"1.0"}}} ``` Response: ```json {"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2024-11-05","capabilities":{"tools":{"listChanged":false}},"serverInfo":{"name":"causalsieve","version":"0.1.0"}}} ``` Notification (no response): ```json {"jsonrpc":"2.0","method":"notifications/initialized"} ``` ### Tool Input Schemas Summary 1. **`sieve_ingest_csv`**: `{"path": str, "stream_id"?: str, "lag"?: int(1..16), "top_k"?: int(1..64), "tokens_budget"?: int(120..8000), "nonlinear_hinge"?: bool, "max_err"?: float(>=0), "forgetting"?: float(0..1], "format"?: "md"|"json"}` 2. **`sieve_push_ticks`**: `{"stream_id": str, "rows": list[list[float]], "channel_names"?: list[str], "lag"?: int(1..16), "forgetting"?: float(0..1], "nonlinear_hinge"?: bool, "top_k"?: int(1..64)}` 3. **`sieve_causal_graph`**: `{"stream_id": str, "min_granger_f"?: float, "min_partial_r2"?: float, "min_pcorr"?: float, "max_edges"?: int}` 4. **`sieve_counterfactual`**: `{"stream_id": str, "channel": str|int, "delta": float, "horizon"?: int(1..500)}` 5. **`sieve_recover_slice`**: `{"handle": "slice:////.."}` OR `{"stream_id": str, "channel": str|int, "start_tick": int, "end_tick": int}` 6. **`sieve_swarm_propose`**: `{"stream_id": str, "agent_id": str, "candidate_channel": str|int, "delta"?: float, "horizon"?: int(1..500), "confidence"?: float(0..1), "rationale"?: str}` ### Security Controls - Path sandboxing: `--allowed-root ` or `CAUSALSIEVE_ALLOWED_ROOTS` restricts `sieve_ingest_csv` using `Path.resolve(strict=True)` and `is_relative_to()`. - Rejects non-regular files (directories, FIFOs, device nodes) and null bytes. - Enforces size limits (`--max-csv-bytes` / `CAUSALSIEVE_MAX_CSV_BYTES`, default 100 MiB) and slice point caps (`100,000` points per call). --- ## 7. C99 ABI Summary (`csrc/causalsieve.h`, `CAUSALSIEVE_ABI_VERSION = 1`) - Structs: `cs_config_t`, `cs_tick_result_t`, opaque `cs_engine_t`, opaque `cs_screened_t`. - Tier-1 functions: `cs_config_default`, `cs_create`, `cs_destroy`, `cs_reset`, `cs_simd_isa`, `cs_last_error`, `cs_tick_count`, `cs_dim`, `cs_lag`, `cs_feature_dim`, `cs_push`, `cs_push_batch`, `cs_granger_matrix`, `cs_precision_matrix`, `cs_contemporaneous_orientation`, `cs_root_attribution`, `cs_counterfactual_do`, `cs_get_raw_slice`, `cs_compress_channel_pla`, `cs_inspect_state`, `cs_force_refresh`. - Tier-2 functions: `cs_screened_create`, `cs_screened_destroy`, `cs_screened_push_batch`, `cs_screened_active_graph`. - Atomicity guarantee: `cs_push` and `cs_push_batch` reject any NaN/Inf input with status `-1` and zero state mutation. --- ## 8. Honest Limits & Failure Modes 1. **Sample size requirement**: reliable `VAR(p)` and precision matrix estimation needs warmup ticks exceeding `K = 1 + p * D` (or `1 + 2 * p * D` with `--hinge`). During early warmup, `top_root_channel` is `-1` and Granger F-stats are regularized toward zero. 2. **Ring buffer retention**: exact slice recovery (`cs_get_raw_slice`) covers only the most recent `raw_ring_capacity` ticks retained in memory (default `8192`). Older ticks are evicted in FIFO order. 3. **Contemporaneous Gaussian identifiability**: Hyvärinen-Smith LiNGAM orientation requires non-Gaussian innovations (skewness or excess kurtosis). Under purely Gaussian instantaneous shocks, partial correlation detects the undirected edge but orientation `R` approaches `0`. 4. **Unobserved confounders**: latent common causes outside the observed `D` channels can induce apparent Granger or partial-correlation edges between observed children. 5. **Non-monotone nonlinearities**: `VAR(p)` (with optional hinge `max(0, x-1)`) captures linear and saturation cascades, not arbitrary symmetric/periodic nonlinear interactions (e.g., `y_t = x_{t-1}^2` around zero mean). 6. **High-dimensional Tier-2 screening**: `ScreenedSieve` promotes the top `active_m` channels by scale-free pre-whitened activity; a subtle fault whose univariate/pre-whitened signal never enters the top `active_m` will not enter the Tier-1 active subgraph.