2017-10-29 22:15:50
good
backup containers for web and webssl
We have two Apache containers. One is handling SSL and is proxying to the second one, which is
running the web applications. We have talked before about the stop daemon. This one gates
all web access so we can restart internal services such as the reader, writer and even the database, without
disrupting the service.

Now, we needed a solution to restart the Apache containers without impacting the service as well. This
is normally done with the load balancer. You simply change the weight of the server you wish to maintain, setting
it to zero, in the load balancer. The traffic will shift to other servers in the cluster. You wait until all
connections are gone and you can perform your maintenance.

If you have just one server, which is our situation for now, you can't use that. But even if
you have a cluster, for performance
reason, you are trying to keep some affinity. You keep local traffic local. If you switch the
weight of the Apache server to zero, the traffic will move to another server, potentially
raising its load.

So we selected a slightly different solution. On all servers, we doubled the Apache containers. With Blackhole, we
have a load balancer between everything in the chain. At any given time, one pair of Apache containers is
taking the load while the other pair is idle. When we need to do some maintenance, we do the following:

*Update the backup/idle pair first (and make sure it runs)
*Switch the weight so the backup pair becomes active
*Update the normal pair
*Switch the load back to it.

With this strategy, we have almost no impact on performance and it works for single
server or a cluster.