Posts

Machine Learning component part1

The design choices to be made while building the machine learning server were: 1. Train the model in real time 2. Train the model in batches beforehand and predict in real time 3. Keep samples of data for training. Train and predict in real time The factors that help decide which approach to use are: 1. Volume of training data and How much time does it take to train the model?            If the training takes too long, there is no point in real time model building. We would need to build the model apriori. Load the model in real time and use real time streams as test flows. The machine learning algorithm chosen in this project takes less time to train(~2-5 seconds) , hence we can afford to learn, train and predict in real time. 2. Criticality of accuracy, how much error is reasonable?            If the accuracy obtained is greater than 85%, its reasonable to go ahead and predict using the model. If not we will need to consider methods to improve the accuracy. 3. How do we
Post a Comment
Read more

Setting up kafka bolt

Once storm topology was set up, there were two ways in which machine learning could be incorporated in the project 1. ML in storm  2. ML outside storm topology In order to choose the design for implementation the factors considered were: 1. Scalability of components            If the storm component is separated from machine learning component, then we can scale each component independently and individually. This factor gives us better flexibility for programming each component. 2. Ease of debugging            Errors that prop up in each individual component will be raised and identified better when the components are disaggregated. We can provide workarounds for the bugs and solve them more efficiently in the suggested setup. 3. Options and research scope that can be explored in each choice             Machine learning in storm has restrictions of its own. The types of built-in algorithms available are limited. If we want to explore different kinds of Machine
Post a Comment
Read more

Day 70 Part 2 : Getting MVP ready for ML part

Image
Todo for this week, > Getting a basic ML DDoS detection algorithm up and running The features extracted so far, 1. Total Inflow Packets 2. Average Number of inflow packets (Total/Number of flows) 3. Total Number of flows 4. Total Packet length 5. Average packet length (Total/Number of packets) Machine Learning algorithm at focus : K means clustering How will the ML algo fit into the project?
Post a Comment
Read more

Day 70: Demo

Image
The below video shows the DoS attack detection, syncing between controllers and how I have decided to mitigate these attacks. The video doesn't show mitigation process correctly, due to few recording issues. I shall upload another video only for the mitigation process shortly. Refer to previous and next posts here. Author: Shravanya Co-author: Swati
Post a Comment
Read more

Day 68: Syncing between master and slave controllers Part 2

We decided in our previous post that we need to keep our slave controller in sync. What is the scenario we are looking at? The master controller gets connected to all switches and contributed to installing flows in them, thus master controller knows all flows in all switches The slave controllers mostly get connected to only switches in their domain and thus whenever a switch migrates from one slave to another slave, the second slave controller has to spend bandwidth and time knowing about the flows installed in it To avoid the overhead that exists during switch migration, we need to ensure that all controllers are in sync with what flows exist in what switches. I had initially decided to follow the below approach: Since the master controller knows all flows at all times, the slave controller would contact the master controller just after switch migration. The master controller which is aware of the switch migration would give necessary details of the flows in the switch for
Post a Comment
Read more

Day 67: Syncing between master and slave controllers Part 1

Few assumptions stated in my previous posts have been wrongly assumed. The assumptions were: The flows installed in the switches are periodically refreshed or re-installed The flows in the switch are retained in the switch as long as it is connected to the controller After experimenting with the switches, trying to analyze if any timeouts exist on the flows, if the switch configurations are set properly, I realized the following: The flows once installed in the switch remain in it until the switch is turned off The flows remain in the switch even when the controller port is not connected to any controller So, why did I wrongly make the former assumptions? It so happens that my Zodiac FX switches (not all of them, few of them) automatically switch off and back on sometimes. And out of these, only few times the switches retain all flows, few other times they lose out on few or all flows. This appears as an error on my controller DPSET: Multiple connections from <switch
Post a Comment
Read more

Day 65: DoS detection from Ryu controller

Working on from where we left off in Day 63, we have to start off with writing the code to detect the DoS attack. I shall be considering only 1 feature - that is the bitrate of the packets that go in and out of the switch ports. This has been sufficient enough for a basic thresholding mechanism of identifying DoS attacks. According to my observation, I have set up a 200 bitrate threshold for identifying a DoS attack. This is a little higher threshold cosidering the average bitrate numbers. The reason I have setup a higher bitrate is to avoid false positives. At the same time, I am fine with few DoS attacks geting to my controller side as the controllers are designed to sustain the attacks in a round-robin sort of fashion. As of now, I have left my DoS attack prolong for 5 - 10 minutes without affecting the functionality of my controllers. The ML algorithm being built would take care of more critical details and better identification of DoS attacks. The one built on the Ryu contr
Post a Comment
Read more