Network/Firewall Monitor

So I wanted to try to make something that involved live/real-time data and I figured that monitoring the firewall and security logs on my web server could serve as a good source since there is a lot of activity (the noisy brute force spamming thing that is the Internet). Ideally I would run some kind of message service locally on my server like RabbitMQ or just use Laravel job queue with redis or something (I like the challenge of the entire infrastructure). However I have extremely limited resources on my public web server (small scale VPS). I decided to use pusher.com instead of hosting the message infrastructure myself, they have a free tier and it works great! 

The process begins with some simple python code:

import time
import datetime
import requests
import hashlib
import json
import sys

def parseString(string):
    jsonencoded = json.dumps(string)
    return jsonencoded

def follow(thefile):
    thefile.seek(0,2)
    while True:
        line = thefile.readline()
        if not line:
            time.sleep(0.1)
            continue
        yield line

def post(type, data, reporter):
    s=requests.Session()
    url = ''
    token = ''
    now = time.time()
    datetimetoken = datetime.datetime.fromtimestamp(now).strftime('%Y%m%d%H')
    tokensend = hashlib.sha224(token+datetimetoken).hexdigest()
    postdata = dict(type=type, reporter=reporter, data=data, token=tokensend)
    response = s.post(url, data=postdata)

if len(sys.argv) < 2:
    sys.exit('Usage: %s filename type' % sys.argv[0])

filename = sys.argv[1]
sendtype = sys.argv[2]
reporter = sys.argv[3]

if __name__ == '__main__':
    logfile = open(filename,"r")
    loglines = follow(logfile)
    for line in loglines:
            datasend = parseString(line)
            post(sendtype, datasend, reporter)

The script takes 3 terminal variables from the command line: the log-filename and a type attribute (ex ssh, firewall rule etc) and the and a label/tag for the reporting server. This python code is basically an endless loop that looks at the end of the log file for changes, as soon as a new line is added, it parses that line to a JSON format and sends a http-request to a back-end Laravel instance on my VPS. 

Some php-logic parses the log-string and checks for some common identifiers (keywords like: BLOCK, BAN etc), it also double checks to see if the traffic might have been regular https 80/443 activity, if that is not the case the activity is considered suspicious (as 80/433 should be the only activity from regular users on external networks). A data model instance for the event is saved with additional geo-location data, based on the ip-address. In addition the ip address is reported to the abuseipdb.com database if it is a confirmed non-normal irregular web traffic or a brute-force ssh login attempt.

On the front end of my portfolio some javascript is listening to the pusher channel, new event id's are sent to the javascript thought the pusher service, then a jQuery/ajax call requests html code from the Laravel blade template. 

The end result is available at danielmellum.com/monitor