We built a managed firehose where the producer is the network itself

A year ago we started Streaming Defense to solve what looked like a narrow problem: SOCs were drowning in late, lossy, log-shaped representations of what their networks were actually doing. The pipeline looked like this everywhere we went — packets get summarized into NetFlow, NetFlow gets shipped to a collector, the collector batches into syslog, syslog goes to a forwarder, the forwarder goes to a SIEM, the SIEM indexes, and eventually a detection engineer writes a search that runs against data that's 5 to 30 minutes old. By the time anything fires, the connection is closed and the evidence is partial.

The standard answer is "stream it" — put Kafka in front of the SIEM. We tried that. It works, technically. But it left us with a real architectural question: if you're going to stream network telemetry, why is the producer a log forwarder? And where, in the pipeline, is the right place to do the expensive work — the threat-intel matching, the geospatial lookup, the cumulative state, the scoring?

So we built it differently. The producer is a probe sitting on the wire. The probe captures, classifies, and ships per-flow records over UDP to a central collector (with local buffering for link outages — more on that below). The collector enriches the consolidated stream — once, in memory, for the entire network — and emits the result over a single WebSocket. Downstream consumers (dashboards, SIEMs, agents, custom apps) subscribe to that WebSocket and receive enriched events as they happen.

This post is the architecture, the trade-offs we made, and a code snippet you can run against our sandbox.

Why UDP from probe to collector

This is the first design choice that surprises people, so let's start with it.

The probe-to-collector path is UDP. No per-record acknowledgment, no TCP head-of-line blocking, no broker between them. We made this choice because the probe's job is to keep up with the wire, and TCP semantics actively get in the way of that — a slow collector or a momentarily saturated link should never block the probe's capture loop.

But UDP without a safety net would lose data, and we cared about that. So the probe buffers locally. If the collector is unreachable — link down, collector restart, network partition — records get written to a local store on the probe and replayed once connectivity returns. The probe never blocks on the network during normal operation, and records survive an outage. The only way you lose data is if the probe itself is destroyed before the link comes back.

This pattern — fast UDP transport with local durability on the producer — is well-trodden in network telemetry. It's how NetFlow, IPFIX, and sFlow have all approached the same problem. We just made the local buffering explicit and durable enough to survive real outages, not just transient blips.

Why enrich at the collector, not the probe

This is the more interesting design choice, and the one I'd most like critique on.

You could push enrichment onto the probe. Some products do. The argument is that probes scale horizontally — add more taps, get more enrichment capacity. That's true but it misses the cost: every probe has to maintain its own copy of the threat-intel tables, the GeoIP database, the vulnerability data, the cumulative counters, the customer entity context. Update those tables and you update them N times across N probes. Add a new TIP feed and you ship it everywhere. Compute cumulative pair-level state on a probe and you only see the flows that probe saw — east-west traffic across probe boundaries becomes invisible.

We took the opposite approach: probes do the work that has to happen on the wire (packet inspection, classification, per-flow records) and nothing else. The collector does the work that benefits from seeing the whole stream — geospatial lookup, threat-intel matching across all sources, vulnerability correlation against customer assets, cumulative counters at the IP-pair level across the entire network.

The collector keeps all of this in memory. There's no database lookup in the hot path. A flow record arrives, gets enriched against in-memory tables, gets scored, and gets emitted. We enrich once per record, for the entire network's view, in microseconds.

The downstream consequence is the part that surprised us most. Because enrichment is already on the event when a consumer receives it, the consumer never has to do a join. There's no "look this IP up in MaxMind," no "match this domain against my TIP feed," no "query my asset database to see if this destination is critical infrastructure." The GeoIP, the threat-intel match, the vulnerability context — they're all fields on the event. Detections that would normally require a streaming join, a sidecar lookup service, or a periodic batch enrichment job collapse to a stateless filter against fields that are already there. This is the part that genuinely changes how people write code against the firehose, and it took us a while to realize it was the main thing.

The trade-off is honest: the collector is a stateful single point. We run it hot-standby with state replication, but it's not a shared-nothing scale-out architecture. For most network sizes that's fine — a single collector handles enormous flow rates because the per-record work is small and in-memory. If you're enriching at hyperscale across geographically distributed taps, you'd want a different topology. We're not trying to be that.

Cumulative counters: putting query results on the event

The other thing the collector does in memory is maintain running counters: how many times has this source IP been seen, how many bytes have been sent to this destination, how long has this IP-pair been talking, how many flows total. Six counters per side (source and destination) plus four pair-level ones, all updated as records flow through.

These counters get attached to every event the collector emits.

The reason this matters isn't theoretical. It's that questions like "have we ever seen this IP before?" or "how much data have we transferred to this destination today?" — questions a detection engineer or SOC analyst asks constantly — normally require a query. A time-window scan over historical events, an aggregation against a state store, a join against a precomputed table. With the counters on the event, the answer is a field lookup. source_ip_cumulative_occurrences == 1 means first-time-seen. destination_ip_cumulative_bytes > threshold means high-volume destination. No query, no scan, no aggregation pipeline.

The cost on the collector is small — these are integer increments on in-memory hash tables. The savings downstream are huge. Most "stateful" network detections become stateless when the state is on the event.

What the firehose actually carries

Each event coming out of the collector is a JSON record with 154 fields. The shape covers identity (probe, organization, source IDs), transport (L3/L4 protocol, ports, packet and byte counts, flow state), addressing (v4 and v6, with public IP enrichment), geospatial enrichment (country, city, lat/lon, ASN, ISP), protocol classification (nDPI application protocol, category, breed), risk and detection (risk type, normalized risk score, severity, confidence), four families of threat-intel enrichment — IP, domain, JA4 fingerprint, and DGA — each with risk score, confidence, categories, and matched sources in basis points, plus the cumulative counters described above.

That's a lot of fields. We considered slimming it down. We didn't, because every field eliminates a downstream query, and the per-record overhead is small.

The WebSocket contract

The consumer-facing contract is intentionally boring. You open a WSS connection to the collector, you parse JSON, you route control frames (performanceMetrics, threatStream, allowlist.result) and treat everything else as a flow record. Anyone with a browser, curl, or 9 lines of Node can connect.

javascript
const ws = new WebSocket("wss://sandbox.signal-fabric.com/ws?token=...");

ws.onmessage = (event) => {
  const flow = JSON.parse(event.data);

  // First-time-seen high-risk destination — no joins, no queries
  if (flow.destination_ip_cumulative_occurrences === 1
      && flow.tip_hit && flow.risk_score_total > 100) {
    console.log("New malicious destination:",
                flow.destination_ip, flow.tip_best_value,
                "from", flow.source_ip_country);
  }
};

That's a real first-seen-suspicious-destination detection in five lines, with no state, no joins, and no queries. The cumulative counter, the TIP enrichment, the geo data — all already on the event.

Why this isn't just managed Kafka

I want to address this directly because we'll get the question.

Kafka is a broker. It assumes you have producers, you have consumers, and the hard problem is durable ordered delivery between them. If your producers already exist and are reliable, Kafka is excellent.

In our case, neither side was a given. The producer didn't exist — we had to build the probe. And the enrichment, which is the actual hard part, isn't broker work; it's stateful in-memory aggregation across the whole stream. A broker wouldn't do that; you'd put Flink or Kafka Streams downstream of the broker to do it, with TIP feeds and GeoIP tables loaded as side inputs, plus a state store for the cumulative counters, plus a join layer. And then your downstream consumers would either consume from the post-processing topic and trust the enrichment, or do their own joins anyway.

We collapsed that by making the collector the place where enrichment happens, by accepting UDP loss for in-flight records (with local buffering on the probe for outages) instead of TCP-and-broker overhead, and by putting the cumulative state directly on each event so consumers don't have to maintain it. The result is one stateful component instead of three, and the consumer-side code is dramatically simpler — most useful detections become stateless filters against fields that are already populated.

The trade-off is honest: we don't have Kafka's durability or replay-from-offset semantics. We hold a 24-hour replay window at the collector for detection engineers writing rules against history, but there's no exactly-once delivery. For real-time security, real-time observability, and real-time agent inputs, this trade-off is correct. For a financial transaction log, it isn't. Use Kafka.

What surprised us building this

Three things, in case they're useful to anyone building in this space.

First, the elimination of downstream joins was the architectural payoff we didn't fully appreciate at the start. We built the in-memory enrichment for collector-side performance reasons — to avoid query overhead in the hot path. The bigger win turned out to be on the consumer side: people writing detections, building dashboards, and feeding agents stopped writing join logic. The mental model becomes "filter the firehose by fields you care about" instead of "join the firehose against four sidecar services." That's a different developer experience.

Second, JA4 fingerprinting is more useful than JA3 ever was, and almost no SIEM surfaces it well. Computing the JA4 on the probe and matching it against threat-intel tables at the collector means consumers can do TLS-client identity matching without parsing handshakes themselves. It's become one of the most-used fields in our schema and we didn't predict that when we started.

Third, the UDP-with-local-buffering pattern works better than we expected in production. In-flight drop rates on normal links are well under 0.1%. Real link outages happen, and the local buffer absorbs them. The thing we were nervous about — losing data — turned out to be a non-issue with the buffering layer in place.

What we got wrong

A couple of things, in the spirit of being honest on HN.

We over-invested early in a custom UI for browsing the firehose. The right answer was to ship the WSS endpoint and let people build whatever they wanted on top. We have a reference dashboard now, but the firehose itself is the product; the UI is a sample.

We also under-invested in the schema documentation for a long time. 154 fields is a lot, and "go read the source" isn't a real answer for a serious consumer. The schema page on our docs site is now thorough, and we publish a JSON schema, but we should have done that on day one.

And we briefly tried to make the platform "agnostic" about what the enrichment meant — letting consumers attach their own scoring. That was a mistake. The collector's enrichment has semantic meaning (risk scores, severity, confidence are produced from a specific model) and pretending otherwise made the API harder to use, not easier. We ship opinionated enrichment now and let consumers ignore the fields they don't want.

The sandbox

If you want to try it, the sandbox is free and self-serve. You get a token, you connect to the WSS endpoint, and you immediately see a synthetic firehose with realistic network flows, TIP hits, JA4 anomalies, and the cumulative counters live. If you want real data from your own network, there's a one-line Docker command to start a probe — it ships to our collector and you see your own enriched events at the WSS endpoint within a few minutes.

Link to sandbox: signal-fabric.com/sandbox

Link to the schema: signal-fabric.com/schema

I'll be in the comments for the next several hours. Genuinely interested in critique on the architecture, especially the UDP-with-local-buffering choice and the in-memory enrichment-and-counters model. The two questions I expect and want to engage with: (1) how do you handle backpressure from collector to consumers, and (2) why not nDPI + Kafka + a stream processor. Happy to get into both.

— Derek Huyser, Co-Founder, Streaming Defense