mozilla

Implications of multiple redis instances (k8s pods, migrations)


#1

Hi,

We currently have Talk running on a number of AWS EC2 instances with a centralized Mongo (Atlas) and Redis (Elasticache).

We’re in the process of setting up (researching) Kubernetes, so at some point we want to move horizontally from our docker-on-EC2 based solution to K8S.

Now we’re looking into ‘what to do with Redis’. If possible, running Redis in a pod together with the application would be favorable because of simplicity. But what would be the effect of pod rescheduling?

To my understanding, Talk uses redis as task queue and (I assume) also for performance cache. So having multiple redis instances and the occasional rescheduling would mean:

  • Several parallel task queues sending e-mail, each with its own set of tasks (no duplicates).
  • Less efficient caching due to duplicate cache effort.
  • Short performance penalty when using new empty redis on spawning new pod.
  • Possible loss of some tasks if not completed before terminating redis.
  • No effect on JWT blacklisting as that is persisted in Mongo.

Are the above assumptions correct and would it therefore be possible to have multiple redis instances in use at a given moment?

This would allow us to keep K8S setup simple and also to move gradually from current to k8s based deployment via DNS load balancing.


#2

Thanks for the question @TBeijen!

It’s great that you’re looking into k8s for deploying and managing Talk, we have lots of newsrooms using it internally to manage small and large installations.

In regards to Redis, it is absolutely necessary to cluster your Redis instances. The most important task of Redis is to act as a Pub/Sub broker for our subscription system that pushes live updates to other users. If the Redis instances aren’t shared/clustered, then some updates may not reach all users, and for that reason, I would advise against the use of Redis inside the same pod, and instead suggest you keep your existing Elasitcache + MongoDB Atlas solution for persistence.

Please fell free to follow up if you have any more questions.


#3

Thx @wyattjoh. Clear.
VPC peering will be the way to go then. Should not be a problem as CIDR blocks don’t overlap.


#4

That would be C7k sir, not “Talk”. Kubernetes raises a number of known issues related to VMs. I was unaware there was a possibly better option. Thanks. So when is K8s preferable to Docker-on-Ec2? Is it a use case issue? I’ll post back any links I find.

I have been assuming that pub/sub and redis wouldn’t matter provided that a single instance of Redis and also Mongo served any given comments stream. Incorrect?