Re: Converting IPFIX flows to columnar format with all IEs - tool recommendations?

[Joe Loiacono via NANOG <nanog () lists nanog org>]

Take a look at SiLK. Very robust, very powerful, command line: 
https://tools.netsa.cert.org/silk/silk.html

FlowViewer sits on top and provides a full GUI interface: flowviewer.net

Joe

On 1/15/2026 12:57 PM, Jonas Muecke via NANOG wrote:

> Hi,
>
> I'm looking for recommendations on conversion and long-term storage of 
> raw IPFIX flow data. Specifically, I need to convert IPFIX flows 
> stored in pcap files into a columnar format like Parquet to enable 
> easy and detailed historic analysis that isn't possible with 
> aggregated data.
>
> Requirements:
> - Parse IPFIX from pcap files (including templates)
> - Preserve ALL information elements, including custom IEs with 
> enterprise PENs
> - Output enterprise number + IE ID + data (detailed interpretation of 
> the data not needed)
> - Handle IP fragment reassembly or large IP packets
>
> nfdump [1] gets close, but it skips custom IEs. Other tools require 
> replaying the pcaps which risks overflowing buffers, so reading 
> directly from pcap files would be preferred.
>
> Has anyone had similar requirements and found a solution? I'm open to 
> multi-step conversions (e.g., via JSON). Long-term I'll capture 
> directly to a better format, but need to process existing pcap 
> archives first.
>
> Thanks,
> Jonas
>
> [1] https://github.com/phaag/nfdump
>
> --
> Jonas Muecke
> Phd Student, TU Dresden
>
> _______________________________________________
> NANOG mailing list 
> https://lists.nanog.org/archives/list/nanog () lists nanog org/message/U7ZR5BJFNCBWI4EBLRDUVPVEV45GHID6/
_______________________________________________
NANOG mailing list 
https://lists.nanog.org/archives/list/nanog () lists nanog org/message/4YRD6RUYRNOYWYA2KGRP45PHLDYRBCZN/