Use SASL/SCRAM for MSK if there are lot of clients
If you’re using MSK (Managed Kafka on AWS), using IAM is generally a good practice for authenticating your workload and accessing the cluster.
However, when number of clients scale, you might see connectings timing out.
Symptom
In MSK’s connection rate limit (with IAM) is 100/s 1. But what does it mean? The documentation does not clarify
But when apps are restarting / starting up en-masse, we saw frequent connection timeouts, which eventually recover. Recovery time is at the mercy of jitter
Debugging
Basics first:
- Resource constraints
When starting up, processes are CPU hungry. Resolving dependencies, intializing components, beans, connecting to database etc.. Was timeouts due to CPU starvation? Going by CPU throttling metrics, I ruled out this one as suspect.
What about MSK? Given that there are 100s of clients connecting at once, surely the problem was at server side I thought - we might probably be choking its network/CPU but quick glance over relevant cloudwatch indicators rules this one out.
- Network
Okay, but what about network? on inspecting packet drop counters, aws bandwidth allowance, conntrack limits etc, I ruled this one out as well. On server side (MSK) however, we dont have ethtools level visibility, but mere 100s of connections shouldnt choke that MSK server. they’re pretty resilient and can sustain GBs of data throughput.
Peeking into the process
Metrics only gets us so far - they can point us in general direction on where to look , but we need to dissect further and establish the mechanism on whats going on.
TLDR: do a high frequency threaddump on the process where kafka connections are timing out - we see that TCP connect is working fine but there is an iowait during SSL handshake.
So we know that MSK is not processing connections fast enough, but resources doesnt seem to be the constraint.
From aws support:
The 100 connections/second quota listed on the MSK quotas page is a connection acceptance rate limit — the maximum rate at which the broker will accept new TCP connections on the IAM listener. It is not a guaranteed handshake completion throughput.
Each IAM connection, once accepted, must complete a server-side identity validation process involving multiple internal round-trips. The effective completion throughput is bounded by the number of concurrent authentication requests the broker can process in parallel (fixed, not proportional to CPU) and the per-handshake latency of server-side validation.
Conclusion
Use username / password authentication with MSK if you have high connection churn