Every entry below comes from docs/limitations.md in the repo — a root-cause investigation, not a disclaimer. Where something was fixed, we say so. Where it wasn't, we say why we left it alone.
cmdi shares structural features — separators, binary names, redirection operators — with benign filesystem paths and path traversal, and isn't linearly separable at moderate tree depth. SMOTE was tried first: oversampling cmdi from 3,881 to 25,000 synthetic samples gained only +0.015 F1 at shallow depth and nothing at deeper trees, proving this was a separability problem, not a sample-count one. Fixed operationally with max_depth=25 — going shallower loses cmdi entirely. What actually broke the ceiling was real, diverse data: adding 2,470 genuinely novel cmdi payloads (0% near-duplicate via MinHash) gained +0.03 to +0.05 F1 at production depth, confirming the real gap was technique coverage, not volume.
ONNX's TreeEnsembleClassifier fully materializes trees natively — about 1.264 KB per node. The original model (367,655 nodes, 44MB on disk) expanded to 465MB RSS. The current model (180,022 nodes, 10.8MB on disk) expands to 122–147MB — a 70–74% reduction for only 0.005 macro F1 loss. This is how ONNX Runtime's RF implementation works, not something configurable away.
base64_like_count is a real evasion signal and a real false-positive risk accepted tradeoffUsed in 3,153 split nodes (~0.86% of splits, ranked #20/66 by importance), this feature counts base64-like sequences — a genuine evasion signal, but also true of legitimate JWTs, session hashes, and API keys in URLs. Traffic carrying bearer tokens or long tokens in GET params should expect a modest elevation in false positives.
A leave-one-source-out (LOSO) test found 0 of 9 data sources reach macro F1 ≥ 0.80 when held out entirely; only command_injection clears 0.60 (0.7762). Two distinct mechanisms are confirmed: volume starvation across 8 of the 9 sources (one or two "mega-sources" supply 85–99% of each class's signal), and a genuine style-driven failure in synthetic_nav — the smallest slice of benign data (0.9%) but by far the worst LOSO score (0.0027), independent of volume. In practice: a genuinely novel attack or benign style, unlike anything in these 9 sources, should be expected to degrade detection substantially.
An earlier Levenshtein-based dedup pass had two real flaws — a 100-character query cutoff that silently excluded 20.6% of rows, and a 10,000-pair cap giving near-zero coverage on large classes. A later MinHash/LSH investigation confirmed genuine, extensive near-duplication: xss at 61.2x saturation, cmdi at 52.3x, path_traversal at 32.4x (pairs-to-rows ratio); sqli and benign scans didn't finish at that scale. The decision was to flag, not deduplicate, this cycle — deduplicating now risks invalidating already-trained and validated models, so it's deferred as a separate retraining decision.
Several sources captured only query or body content, leaving method/path blank — impossible for a real HTTP request. Normalizing these to GET / / dropped the end-to-end benign false-positive rate from 21.0% to 2.0%.
Payloads encoded as <script> were missed (68.9% detection pre-fix) while raw or URL-encoded markup was reliably caught. Fixing this meant decoding HTML entities before pattern matching and recalibrating the threshold from 0.70 to 0.35 — moving overall detection from 77% to 94%. The same investigation fixed unicode-escape evasion (0% → 100%) and double-percent-encoding evasion (50% → 100%). Double HTML-entity evasion was checked and found not to be a practical gap: entity-decoding only happens during HTML parsing, never JS execution, so double-encoding the payload just breaks the exploit itself.
Command-injection payloads sent with minimal HTTP context — no user-agent, cookies, or referer, i.e. raw curl or script traffic rather than browser-driven — score measurably lower confidence. Bare recon commands (whoami, id, uname -a, env, ls -la /, ping) scored 0.233–0.336 in minimal context, below the 0.35 threshold, while overtly malicious targets (/etc/passwd, reverse shells) still cleared it. Of 33 distinct live cmdi techniques tested, 17 always blocked, 14 blocked inconsistently depending on formatting, and only 2 (6%) never blocked in any variant. Root cause: RF's feature set includes context signals like UA length and cookie presence that correlate with "legitimate traffic" in training — stripped-context requests lose that implicit confidence boost.
onnxruntime-node serializes concurrent inference calls within a single thread — fixed by splitting RF and IF into separate dedicated worker threads. A follow-on cold-start bug (0% real-world IF-capture despite 145/145 mocked tests passing) was fixed with a readiness handshake between the main thread and workers. An initial 5ms "grace window" fix was later found to cost ~1ms of latency on ~99% of requests just paying for IF's real inference time — it was replaced with the current design: RF resolves the response immediately, and a late IF reply patches the already-written log row asynchronously.
IF's pass_anomaly rate on pure benign traffic measured 94.6% in live Docker testing, against an offline-calibrated target of 5.3–5.7% — about 17x higher than expected. Root cause, confirmed by ablation: the benign training corpus is 99.6% missing User-Agent headers, so IF learned "no UA = normal," backwards from real traffic where a UA is essentially always present. Adding synthetic UA-bearing benign data at increasing scale regressed IF's recall on other attack classes by as much as −11.7pp (cmdi), −6.3pp (path_traversal), −5.5pp (sqli), −2.5pp (xss) — realistic benign traffic moves structurally closer to attack traffic. Decision: not pursued. IF has no blocking authority, so this is accepted as log/webhook noise, not a security regression — but if you wire pass_anomaly into an alerting pipeline, expect it to be loud.
A time-based blind technique like sleep(N) is syntactically identical between sqli and cmdi, and semicolon_count is a feature shared by both. At one point ~49% of cmdi payloads were mislabeled as sqli in reporting; the current minimal-context retrain narrowed this to ~35%. Blocking behavior is unaffected either way — only the triage label can be wrong.
A plain GET /posts from real Firefox, no payload at all, was blocked at 62% confidence as sqli — because ua_length is the single most important RF feature, and a real 78-character Firefox UA looked nothing like the short synthetic UAs in training.
The same benign query ("weekend hiking trip"), same session — only the User-Agent changed: curl/8.7.1 scored 0.367 (passed), Python-urllib/3.14 scored 0.400 (blocked as sqli), a custom corpus-runner UA scored 0.616 (blocked as sqli). Incidental client metadata, not just payload semantics, can flip a near-threshold verdict.
POST /posts with title=Hello&content=Just a normal note scores sqli at ~0.40 because sqli_operator_count matches ordinary key=value&key=value form syntax. This is not fixed: 37.4% of all sqli training rows rely on this exact feature as their only nonzero signal, so narrowing the regex would destroy detection for over a third of the sqli corpus. It's documented as an accepted model-boundary limitation, not an oversight.