Network update part 2: Monitoring the network with NetFlow, InfluxDB and Grafana2018-05-19 [ networking ]
After adding a fancy router (see here) to my setup I decided to monitor my network (now that it is possible). Having already a Grafana+InfluxDB setup for my standard monitoring I decided to (somehow) feed the network data into this setup.
Compile ipt_NETFLOW kernel module
To build ipt-netflow I needed to install some dependencies first:
dkms iptables-dev pkg-config build-essential git-core.
On the espressobin I also needed
linux-headers-next-mvebu64 instead of
build / install
git clone https://github.com/aabc/ipt-netflow.git cd ipt-netflow ./configure make all install depmod
test it out
As root (change your destination ip/port):
modprobe ipt_NETFLOW destination=192.168.2.12:2055 iptables -I FORWARD -j NETFLOW iptables -I INPUT -j NETFLOW iptables -I OUTPUT -j NETFLOW
Write some queries in grafana
SELECT sum("value") FROM "net_if" WHERE "scidr" = '192.168.2.0/24' AND "dcidr" = '0.0.0.0/0' AND "dport" != '80' AND "dport" != '443' AND "daddr" != '18.104.22.168' AND $timeFilter GROUP BY time(24h), "saddr", "daddr", "dport" fill(none)
You end up with something like this:
This allowed me to find many hosts doing:
- A LOT of DNS requests (1k+/min). As I didn't want to add a local DNS caching server (yet) for each of these (2) servers, I simply hard-coded the value in /etc/hosts.
- NTP and DNS requests going to the outside world. Now these are dropped and my router is advertised as the DNS/NTP server.
DNS(ACCEPT) $FW net DNS(REJECT) loc net NTP(ACCEPT) $FW net NTP(REJECT) loc net
I want to investigate using elasticsearch+kibana as it at least supports IPs as native datatypes, which would allow me to NOT have to decide on the
cidr myself (and maybe find a way to do automatic name resolution to help out with debugging).