Converting IPFIX flows to columnar format with all IEs - tool recommendations?

[Jonas Muecke via NANOG <nanog () lists nanog org>]

Hi,

I'm looking for recommendations on conversion and long-term storage of 
raw IPFIX flow data. Specifically, I need to convert IPFIX flows stored 
in pcap files into a columnar format like Parquet to enable easy and 
detailed historic analysis that isn't possible with aggregated data.

Requirements:
- Parse IPFIX from pcap files (including templates)
- Preserve ALL information elements, including custom IEs with 
enterprise PENs
- Output enterprise number + IE ID + data (detailed interpretation of 
the data not needed)
- Handle IP fragment reassembly or large IP packets

nfdump [1] gets close, but it skips custom IEs. Other tools require 
replaying the pcaps which risks overflowing buffers, so reading directly 
from pcap files would be preferred.

Has anyone had similar requirements and found a solution? I'm open to 
multi-step conversions (e.g., via JSON). Long-term I'll capture directly 
to a better format, but need to process existing pcap archives first.

Thanks,
Jonas

[1] https://github.com/phaag/nfdump

--
Jonas Muecke
Phd Student, TU Dresden

_______________________________________________
NANOG mailing list 
https://lists.nanog.org/archives/list/nanog () lists nanog org/message/U7ZR5BJFNCBWI4EBLRDUVPVEV45GHID6/