Day 35: Implementation of Architecture Phase 3 - Adding Backup mechanisms
In the previous post we had discussed few problems we were facing with the working of the architecture. To be precise, it was regarding why the switch was not accepting packets from the backup controller even after traffic redirection from the load balancer. Well, since we have configured the controller IP address on the switch already, the switch expects to receive no packets from controllers bearing any other IP address. So, how do we deal with this problem?
We can introduce a NAT functionality on the load balancer that changes the IP address when the packets go from the switch subnet to the controller pool subnet. This way, even though the request may not be answered by a controller of a specific IP address, it can pose like it has that IP address. We shall be taking this up for implementation next week. Now, we shall first have to concentrate on securing our load balancer with a standby system that can act as backup when the load balancer goes down.
There is something called 'keepalived' that shall help us achieve the same. Keepalived is a routing software which is programmed in C. Its role in our architecture would be to detect the health of our load balancer and when the active load balacer is down, the passive one should be turned on. For this to happen, we shall be using a Virtual IP address which will be taken up by the load balancer whichever is active at that point of time. We do not have to worry about the intricate details as to how this happens since we shall let the Keepalived module do the work for us. There is an excellent article on how to secure HAProxy load balancers with keepalived module. We shall be implementing this tomorrow. I shall also be uploading a video once the implementation is successful for the better understanding of the architecture. The below diagram would give a better picture of how keepalived would work:
Few prerequisites we need to think of before implementing keepalived is if the architecture we have built right now would be sufficient or not? The architecture we have built right now has the master load balancer and router on the same system. This is the reason why all the traffic is forced to go to the load balancer and then either to the switch or controller side. If we introduce another standby load balancer, we need to take care of its placing so that whenever it is active, all traffic should pass through it. With the current set-up, we cannot add another standby system among one of the backend servers as the traffic would not necessarily go through the backup load balancer even when it is active. Thus, through the introduction of keepalived, some amount of disruption should happen wrt to our architecture. The new architecture would look something like this:
The load balancer and backup LB would be separate systems. They would also involve static routing tables in them for network traffic flow from Router-LB subnet to LB-Controller subnet. Thus each of my load balancer would again have router functionality in them. The reason being, the two interfaces the LB is connected to cannot be on the same subnet. If you can remember, we faced this problem in one of the previous implementation phases as well.
Next, we also need to address the functionality of the master controller in the architecture as well. In the architecture we have built currently, we are just ignoring the presence of the master controller as no packets are being sent to it, when infact, all packets should be sent to it. For implementing the same, we need to think of few ideas. One of them could be to just create copies of the packets and forward it to the master controller. But the diasdvantages associated with thiis are:
We can introduce a NAT functionality on the load balancer that changes the IP address when the packets go from the switch subnet to the controller pool subnet. This way, even though the request may not be answered by a controller of a specific IP address, it can pose like it has that IP address. We shall be taking this up for implementation next week. Now, we shall first have to concentrate on securing our load balancer with a standby system that can act as backup when the load balancer goes down.
There is something called 'keepalived' that shall help us achieve the same. Keepalived is a routing software which is programmed in C. Its role in our architecture would be to detect the health of our load balancer and when the active load balacer is down, the passive one should be turned on. For this to happen, we shall be using a Virtual IP address which will be taken up by the load balancer whichever is active at that point of time. We do not have to worry about the intricate details as to how this happens since we shall let the Keepalived module do the work for us. There is an excellent article on how to secure HAProxy load balancers with keepalived module. We shall be implementing this tomorrow. I shall also be uploading a video once the implementation is successful for the better understanding of the architecture. The below diagram would give a better picture of how keepalived would work:
Few prerequisites we need to think of before implementing keepalived is if the architecture we have built right now would be sufficient or not? The architecture we have built right now has the master load balancer and router on the same system. This is the reason why all the traffic is forced to go to the load balancer and then either to the switch or controller side. If we introduce another standby load balancer, we need to take care of its placing so that whenever it is active, all traffic should pass through it. With the current set-up, we cannot add another standby system among one of the backend servers as the traffic would not necessarily go through the backup load balancer even when it is active. Thus, through the introduction of keepalived, some amount of disruption should happen wrt to our architecture. The new architecture would look something like this:
The load balancer and backup LB would be separate systems. They would also involve static routing tables in them for network traffic flow from Router-LB subnet to LB-Controller subnet. Thus each of my load balancer would again have router functionality in them. The reason being, the two interfaces the LB is connected to cannot be on the same subnet. If you can remember, we faced this problem in one of the previous implementation phases as well.
Next, we also need to address the functionality of the master controller in the architecture as well. In the architecture we have built currently, we are just ignoring the presence of the master controller as no packets are being sent to it, when infact, all packets should be sent to it. For implementing the same, we need to think of few ideas. One of them could be to just create copies of the packets and forward it to the master controller. But the diasdvantages associated with thiis are:
- HAProxy does neither allows multicast options nor copying packets option
- the master controller will only receive packets even if we manage to send it copies, but won't be replying back since a legitimate TCP connection is not setup between the switch and itself
I did go through few tools that could help me achieve copying and sending packets to the master controller:
- Nginx has a mirroring option
- teeproxy is another tool which pretty much does what we want - copy packets and send it to two backend server. I'm yet to figure out the documentation details and how to integrate it with the exisitng HAProxy
- There are few other tools like agnoster, teecp - This is the StackOverflow answers given by people to a very similar question posed.
- Another option would be to use dpkt library and create TCP packets and send it to the master controller. This approach would be very good since we would have the ability to not only send packets, but also receive them as reply from master controller.
I'm yet to decide what is to be done with regards to the master controller and switch communications. Will keep you updated on the same after doing a comparative study on the above options and if any conclusive results are obtained from the same.
Author: Shravanya
Co-Author: Swati
Comments
Post a Comment