LDAP connection pooling instead of individual connections for all asks #3539

michaelalang · 2025-01-06T08:40:19Z

michaelalang
Jan 6, 2025

Using OpenTelemetry tracing and instrumenting the Quay code I identified that we utilize an LDAP connection for nearly everything we do instead of re-using the existing connection.

This type of behaviour has also been mentioned by Customers as "Quay hammering LDAP Servers".
In the authentication endpoint (/v2/auth) this means depending on the configuration we end up with:

one connection for searching the directory with the configured admin dn
one connection for verifying the credentials of the user (LDAP bind)
one connection if configured global_read_only_users
one connection if configured restricted_users
one connection if configured/monitored on end_to_end health endpoint (LDAP bind)

Even though, it is not possible to have one connection pool for all processes we fork (since the do not share objects), adding a ConnectionPool for LDAP improves the amount of connections being created and improves the duration of the LDAP calls since the TLS handshake does not need to happen each and everytime (worst case /v2/auth = 4)

I did some verification on the theory and implemented a LDAP ConnectionPool that also re-establishes the connections from the pool if the LDAP server closes or timeout occurs and the latency improvement is between 5-10 times (59ms vs 290-600ms)

On the concept of using one connection to search and authenticate a user, the ConnectionPool I did is using Context for providing LDAP to the methods and locks to authenticate a user and switching back to the config admin DN (also fallback on failed authentications).

Any input/suggestion/idea/issue from everyone is more than welcome.
I will also raise this question in the next Engineering and Community calls.

Performance difference on left Pooled connections vs Single instance connections

current Quay single instance spin of connections for all LDAPConnection objects in the code

michaelalang · 2025-01-07T18:18:10Z

michaelalang
Jan 7, 2025
Author

In numbers that means for example:

# for 100 failed logins we will currently have ~927 connections being issued to an LDAP Service
for x in $(seq 1 100) ; do podman login -u milang -p xxx quay.apps.example.com ; done
...
# on the Node running the Quay pod
conntrack -L -p tcp --dport=636 -d 192.168.192.201 -s 10.132.5.59 >/dev/null 
conntrack v1.4.7 (conntrack-tools): 933 flow entries have been shown.

# compared to pooled connections 
for x in $(seq 1 100) ; do podman login -u milang -p xxx quay.apps.example.com ; done
...
# on the Node running the Quay pod
conntrack -L -p tcp --dport=636 -d 192.168.192.201 -s 10.135.2.195 >/dev/null
conntrack v1.4.7 (conntrack-tools): 30 flow entries have been shown.

0 replies

eb4x · 2025-01-15T12:55:47Z

eb4x
Jan 15, 2025

This looks really promising, I recently started troubleshooting issues we had with connections getting dropped, quite consistently for queries in to /api/v1/superuser/users/, especially when the user count is getting in the high hundreds.

...
gunicorn-web stdout |   File "/app/lib/python3.9/site-packages/flask_principal.py", line 193, in can
gunicorn-web stdout |     return self.identity.can(self.permission)
gunicorn-web stdout |   File "/quay-registry/auth/permissions.py", line 253, in can
gunicorn-web stdout |     self._populate_superuser_provides(user_object)
gunicorn-web stdout |   File "/quay-registry/auth/permissions.py", line 201, in _populate_superuser_provides
gunicorn-web stdout |     ) and usermanager.is_superuser(user_object.username):
gunicorn-web stdout |   File "/quay-registry/data/users/__init__.py", line 412, in is_superuser
gunicorn-web stdout |     return self.state.is_superuser(username)
gunicorn-web stdout |   File "/quay-registry/data/users/__init__.py", line 429, in is_superuser
gunicorn-web stdout |     return self.federated_users.is_superuser(username) or super().is_superuser(username)
gunicorn-web stdout |   File "/quay-registry/data/users/__init__.py", line 367, in is_superuser
gunicorn-web stdout |     return self.state.is_superuser(username)
gunicorn-web stdout |   File "/quay-registry/data/users/externalldap.py", line 534, in is_superuser
gunicorn-web stdout |     (found_user, err_msg) = self._ldap_single_user_search(
gunicorn-web stdout |   File "/quay-registry/data/users/externalldap.py", line 322, in _ldap_single_user_search
gunicorn-web stdout |     with_dns, err_msg = self._ldap_user_search(
gunicorn-web stdout |   File "/quay-registry/data/users/externalldap.py", line 282, in _ldap_user_search
gunicorn-web stdout |     with self._ldap.get_connection():
gunicorn-web stdout |   File "/quay-registry/data/users/externalldap.py", line 96, in __enter__
gunicorn-web stdout |     self._conn.simple_bind_s(self._user_dn, self._user_pw)
gunicorn-web stdout |   File "/app/lib/python3.9/site-packages/ldap/ldapobject.py", line 248, in simple_bind_s
gunicorn-web stdout |     msgid = self.simple_bind(who,cred,serverctrls,clientctrls)
gunicorn-web stdout |   File "/app/lib/python3.9/site-packages/ldap/ldapobject.py", line 242, in simple_bind
gunicorn-web stdout |     return self._ldap_call(self._l.simple_bind,who,cred,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls))
gunicorn-web stdout |   File "/app/lib/python3.9/site-packages/ldap/ldapobject.py", line 128, in _ldap_call
gunicorn-web stdout |     result = func(*args,**kwargs)
gunicorn-web stdout | ldap.SERVER_DOWN: {'result': -1, 'desc': "Can't contact LDAP server", 'errno': 104, 'ctrls': [], 'info': 'Connection reset by peer'}
...

and on the ldap side there's just a short

Jan 15 13:51:44 ldap-test06 slapd[1381]: conn=8513629 fd=46 closed (TLS negotiation failure)

4 replies

michaelalang Jan 15, 2025
Author

@eb4x I also have two PR's submit for some time now which should handle LDAP exceptions more smoothly meaning not raising an exception and returning 500 but an error message.

I am not sure if pooling will solve the issue as it seems that the TLS session itself wasn't able to get established (cert expired?) and only if LDAP recovers within a certain period of time pooling would help in that regards.

Still thanks for your input and please keep more of that coming (also if possible, join the community meeting as it's always easier to get the pain when explained compared to written (no emotions))

eb4x Jan 15, 2025

@eb4x I also have two PR's submit for some time now which should handle LDAP exceptions more smoothly meaning not raising an exception and returning 500 but an error message.

#2761 would come in handy 👍

I am not sure if pooling will solve the issue as it seems that the TLS session itself wasn't able to get established (cert expired?) and only if LDAP recovers within a certain period of time pooling would help in that regards.

It made (and I'm not making this number up, and I'm not at all superstitious 😅) 666 individual tls connections for user queries within 4 seconds, before it crapped out. And subsequent queries seem to work just fine. It doesn't happen as much (if at all) to the two other endpoints I'm querying. (/api/v1/superuser/organizations/ and just plain /api/v1/organization/exampleorg)

Still thanks for your input and please keep more of that coming (also if possible, join the community meeting as it's always easier to get the pain when explained compared to written (no emotions))

I'll make a note in my calendar to join in on the one for February. (5th right?)

michaelalang Jan 15, 2025
Author

Yes, 5th February it should be :)
and another yes, that's one of the PR's I was mentioning as only with the second (customized healthcheck) Quay would not reconcile ...

michaelalang Jan 27, 2025
Author

add upstream PR for the python3-ldap(rpm-name) module python-ldap/python-ldap#582

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

LDAP connection pooling instead of individual connections for all asks #3539

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

LDAP connection pooling instead of individual connections for all asks #3539

Uh oh!

michaelalang Jan 6, 2025

Replies: 2 comments · 4 replies

Uh oh!

Uh oh!

michaelalang Jan 7, 2025 Author

Uh oh!

eb4x Jan 15, 2025

Uh oh!

michaelalang Jan 15, 2025 Author

Uh oh!

eb4x Jan 15, 2025

Uh oh!

michaelalang Jan 15, 2025 Author

Uh oh!

michaelalang Jan 27, 2025 Author

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

michaelalang
Jan 6, 2025

Replies: 2 comments 4 replies

michaelalang
Jan 7, 2025
Author

eb4x
Jan 15, 2025

michaelalang Jan 15, 2025
Author

michaelalang Jan 15, 2025
Author

michaelalang Jan 27, 2025
Author