Friday, February 16, 2018

Wireshark to the rescue !!!

So we had a massive increase in logging from our campus firewall.

Jan 25 2018 11:50:32: %ASA-1-106021: Deny UDP reverse path check from 172.16.160.29 to 172.16.70.1 on interface campus

What's the big deal? Well we send our logs to a centralized syslog server and we only pay for 20GB of volume per day and this increase was tipping us over that threshold which basically locks us out if we go over 5 times in a sliding 30 day window.  There were roughly 103,000 of these messages every minute, so in a day that equaled about 148,320,000 a day.  To put it in perspective, we normally log about 10GB a day and we were now logging over 40GB a day.  I need to figure out what was going on, and fast.

The source and destination always stayed the same, but the problem was those aren't private addresses that we used on our network anywhere. 


So to try to get a handle on what the traffic really was I figured a good old packet capture might help a bit.  So I started up a capture with the source address from above, and sure enough, the packets started filling up my capture.


So I still had no idea what the traffic really was, but hey at least now I know that it was some MPEG TS stream.  Hmm that would be really cool if you could have wireshark reassemble the packets into a visual stream for me to look at…  I mean I know you can do it with un-encrypted audio, and if you have SNMP v3 credentials you can dissect the traffic and actually read it.

I headed off to goo goo land and found the mpeg_dump.lua script !!! https://wiki.wireshark.org/mpeg_dump.lua

Basically install the script to your wireshark main directory and then edit some init.lua file and tell it to load the script.  See the above link for a more detailed description.  Once you restart your wireshark you will find it under Tools -> Dump MPEG TS Packets and then tell it where you want to dump the packets in their MPEG file.


After waiting, while it worked it's magic, I had to use VLC to actually see the traffic (no built in windows apps knew what to do with it), and there you have it… PBS stream.

"Hey what were those guys installing the other day in our data center for our TV system ?" I asked my boss.  "It was a PBS channel for our analog TV system" he said.

Well we now had a scapegoat, and after contacting their support they connected to their equipment and admitted to the newly installed device being misconfigured to send that out to the destination address from the log message.

PROBLEM SOLVED !!! Thanks Wireshark !!!