Hi!
I know that it has passed some time since the topic had been created, but the problem described above is still actual and I would like to raise the issue again, adding some information.
I would like to remind you the problem: while I was load testing my company's system that uses FusionAuth as IdP, I've noticed that it is not possible to proceed more then several dozens (not even hundreds) logins per second using FusionAuth. When I say "login" I mean sequential calls to /oauth2/authorize and /oauth2/token. So I am looking for a way to increase the performance of the system.
Here are some details about the environment:
FusionAuth version is 1.22.2.
6 instances are deployed in kubernetes, each on its own node, no resource limits from kubernetes side.
FusiosionAuth is launched with the following config options:
FUSIONAUTH_APP_MEMORY: 2G
DATABASE_MAXIMUM_POOL_SIZE: 16
FUSIONAUTH_APP_RUNTIME_MODE: production
SEARCH_TYPE: database
The database is MySQL Percona Server 8.0.22-13, deployed on a separate virtual machine with the following params:
Model: Intel Core Processor (Broadwell, IBRS)
CPU Cores: 4
Clock: 2199 MHz
Load test emulates several concurrent users, each user performs the following steps sequentially:
- Send /oauth2/authorize request
- Send /oauth2/token request (exchange auth code to access token)
I ran several tests with various number of threads (users) and requests and could hardly reach throughput of 50 logins per second with 64 concurrent users (each login attempt consists of two requests /oauth2/authorize and /oauth2/token, so the total number of requests is x2). Increasing the number of concurrent users lowers the overall throughput and causes more errors.
I could see errors in logs reporing database connection problems:
2021-03-26 1:10:18.956 PM ERROR io.fusionauth.app.primeframework.error.ExceptionExceptionHandler - An unhandled exception was thrown
org.apache.ibatis.exceptions.PersistenceException:
### Error querying database. Cause: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 2090ms.
### The error may exist in io/fusionauth/api/domain/InstanceMapper.java (best guess)
### The error may involve io.fusionauth.api.domain.InstanceMapper.retrieve
### The error occurred while executing a query
### Cause: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 2090ms.
...
...
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 2090ms.
...
... 76 common frames omitted
I've investigated MySQL statistics and found out that even though the number of requests per second to FusionAuth is not high there are a lot of requests to MySQL.
As far as I can see, MySQL could be a possible bottleneck, but I wonder why the number of requests is so high and if there is a way to optimise FusionAuth communication with the database? Besides increasing MySQL's cpu/memory are there any other options?