Setting up kafka bolt

Once storm topology was set up, there were two ways in which machine learning could be incorporated in the project
1. ML in storm 
2. ML outside storm topology

In order to choose the design for implementation the factors considered were:

1. Scalability of components
           If the storm component is separated from machine learning component, then we can scale each component independently and individually. This factor gives us better flexibility for programming each component.

2. Ease of debugging
           Errors that prop up in each individual component will be raised and identified better when the components are disaggregated. We can provide workarounds for the bugs and solve them more efficiently in the suggested setup.

3. Options and research scope that can be explored in each choice
            Machine learning in storm has restrictions of its own. The types of built-in algorithms available are limited. If we want to explore different kinds of Machine learning algorithms we need to set it up outside the topology. 

Hence, the design choice is justified.

Now, we need to figure out a way to connect the storm topology to the machine learning server.
We could do it using a data messaging pipeline that we have already used - KAFKA! 

The idea is :
1. Set up storm topology
2. Stream/Publish the output tuples from bolt into a Kafka topic (aka KAFKA BOLT). 
  • Kafka bolt is within the topology.
  • The tuple has to be serialized while dumping it to the Kafka topic 

3. Subscribe to the data in Kafka topic, from machine learning server.
  • Since the tuple is serialized, if you directly consume it, garbage values will be dumped
  • We need a deserializer code before consuming the data
  • Machine learning server code can now consume the data, train the model and do some useful predictions.



   

Comments

Post a Comment

Popular posts from this blog

Day 12: Master Slave SDN Controller Architecture

In the previous post, I was struggling to figure out why all controllers can install flows in all switches irrespective of the code. Well, today the mystery is getting resolved. I found this book that explains beautifully the various roles that a controller can play. The three roles are: Equal : this happens by default all the equal controllers in a network that are connected to the switch can add and delete flows from it Slave: it has a read-only limitation to the switch openflow tables Master: very similar to the switch, but only 1 master can be connected to the switch at that point of time, and no other flows can intercept the communication between switch and master. I highly recommend you go through the above link to get a better picture. Since the default role of any controller is "Equal", we couldn't really differentiate between the three controller flows yesterday. But one good news from this is that we don't really need extensive coding to assi...
Read more

Day 50: Tcpreplay and tcpliveplay approach

Today we shall be exploring how to copy and redirect the OpenFlow packets to more than one controller from the load balancer. The load balancer is anyway sending the OpenFlow packets to one of the slave controllers - c1 or c2. The packets sent to these two controllers needs to be sent to master controller so the master controller can update the new flows added by c1 or c2 on the switches. This ensures synchronization between controllers. There are two implementation methods to achieve the same: Copy all packets generated from the load balancer to the controller side. Only the packets having destination IP address of c1 and c2 can be redirected to the master controller. Thus we can prevent the packets which already are reaching the master controller to be sent as duplicates Copy and redirect packets from the slave controllers - c1 and c2. We need not take care of duplicate packets reaching master controller. This will reduce the time taken to redirect packets. Since the second a...
Read more

Day 1: Understanding Ransomware and how to detect them?

Image
To start this journey, let’s begin at the basics and learn about what ransomwares are, what is the meaning of a ransomware attack, how have other people detected it so far? By learning the above, we will be in a position to analyze any ransomware attack and devise methods to detect, mitigate and prevent them. Definition: Ransomware is a type of malicious software from cryptovirology that threatens to publish the victim's data or perpetually block access to it unless a ransom is paid. To continue understanding and addressing the threats posed by ransomware, let us dig a little deeper. As the definition specifies, ransomware is a type of malicious software. Now, what is a malicious software? Definition: Malware (= Malicious + Software) is a software which is specifically designed to disrupt, damage, or gain authorized access to a computer system. These malwares (or simply dangerous content that can corrupt your system), can enter your system from various s...
Read more