Machine Learning component part1

The design choices to be made while building the machine learning server were:
1. Train the model in real time
2. Train the model in batches beforehand and predict in real time
3. Keep samples of data for training. Train and predict in real time

The factors that help decide which approach to use are:
1. Volume of training data and How much time does it take to train the model?
           If the training takes too long, there is no point in real time model building. We would need to build the model apriori. Load the model in real time and use real time streams as test flows.
The machine learning algorithm chosen in this project takes less time to train(~2-5 seconds) , hence we can afford to learn, train and predict in real time.

2. Criticality of accuracy, how much error is reasonable?
           If the accuracy obtained is greater than 85%, its reasonable to go ahead and predict using the model. If not we will need to consider methods to improve the accuracy.

3. How do we improve accuracy of the model? Will feedback be considered?
           If  the training is done offline, we can train the model with the flows it predicted wrong. rebuild the model and load it.
           If the training is done in real time and the accuracy is good enough, we do not have to retrain the model. The real time flows can be used as test flows for prediction.
           If the training is done in real time and the accuracy is bad, then maybe use a different algorithm that fits the data better. The algorithm is time critical. If algorithm cant be changed then use different set of parameters to change the nature of the model.

Some suggestions for the model -
1. Rule based approach - Decision trees, K Nearest Neighbours
2. Neural network with real time feedback - One Shot Learning






Comments

Post a Comment

Popular posts from this blog

Day 12: Master Slave SDN Controller Architecture

In the previous post, I was struggling to figure out why all controllers can install flows in all switches irrespective of the code. Well, today the mystery is getting resolved. I found this book that explains beautifully the various roles that a controller can play. The three roles are: Equal : this happens by default all the equal controllers in a network that are connected to the switch can add and delete flows from it Slave: it has a read-only limitation to the switch openflow tables Master: very similar to the switch, but only 1 master can be connected to the switch at that point of time, and no other flows can intercept the communication between switch and master. I highly recommend you go through the above link to get a better picture. Since the default role of any controller is "Equal", we couldn't really differentiate between the three controller flows yesterday. But one good news from this is that we don't really need extensive coding to assi...
Read more

Day 50: Tcpreplay and tcpliveplay approach

Today we shall be exploring how to copy and redirect the OpenFlow packets to more than one controller from the load balancer. The load balancer is anyway sending the OpenFlow packets to one of the slave controllers - c1 or c2. The packets sent to these two controllers needs to be sent to master controller so the master controller can update the new flows added by c1 or c2 on the switches. This ensures synchronization between controllers. There are two implementation methods to achieve the same: Copy all packets generated from the load balancer to the controller side. Only the packets having destination IP address of c1 and c2 can be redirected to the master controller. Thus we can prevent the packets which already are reaching the master controller to be sent as duplicates Copy and redirect packets from the slave controllers - c1 and c2. We need not take care of duplicate packets reaching master controller. This will reduce the time taken to redirect packets. Since the second a...
Read more

Day 1: Understanding Ransomware and how to detect them?

Image
To start this journey, let’s begin at the basics and learn about what ransomwares are, what is the meaning of a ransomware attack, how have other people detected it so far? By learning the above, we will be in a position to analyze any ransomware attack and devise methods to detect, mitigate and prevent them. Definition: Ransomware is a type of malicious software from cryptovirology that threatens to publish the victim's data or perpetually block access to it unless a ransom is paid. To continue understanding and addressing the threats posed by ransomware, let us dig a little deeper. As the definition specifies, ransomware is a type of malicious software. Now, what is a malicious software? Definition: Malware (= Malicious + Software) is a software which is specifically designed to disrupt, damage, or gain authorized access to a computer system. These malwares (or simply dangerous content that can corrupt your system), can enter your system from various s...
Read more