Connection timeouts when using FusionAuth with GCP Cloud NAT

[FASupportBot]

When running FusionAuth in containers on Google Cloud Platform with Cloud NAT configured for static IP assignment, we're experiencing intermittent connection timeouts and failures when making outbound connections from our application to FusionAuth.

The issue appears to be related to network connectivity, but FusionAuth itself seems to be running correctly. The timeouts occur sporadically under load.

Our setup:

• FusionAuth running in containers on GCP
• Cloud NAT configured to assign static IP addresses to containers
• Intermittent connection failures to FusionAuth APIs

Has anyone encountered similar networking issues when using FusionAuth with GCP Cloud NAT?

[FASupportBot]

This issue is not actually related to FusionAuth itself, but rather to GCP Cloud NAT port allocation limits.

By default, Cloud NAT only allocates 64 TCP ports per VM for outbound connections. When your application makes many concurrent connections to FusionAuth (or any external service), you can quickly exhaust these ports, leading to connection timeouts.

Solution

Enable dynamic port allocation and increase the port range with this gcloud command:

gcloud compute routers nats update cloud-nat \
  --router=<ROUTER> \
  --region=<REGION> \
  --project=<PROJECT> \
  --enable-dynamic-port-allocation \
  --min-ports-per-vm=1024 \
  --max-ports-per-vm=32768

Replace <ROUTER>, <REGION>, and <PROJECT> with your actual GCP resource names.

This increases the available ports from 64 to between 1024-32768 per VM, which should resolve the connection timeout issues.

Additional Considerations

While this is primarily a GCP infrastructure issue, if you continue to experience connection problems after adjusting your NAT configuration, consider reviewing:

• Network latency between FusionAuth and your database - High latency or unstable network connectivity can cause database connection pool exhaustion, which may manifest as API timeouts
• Connection pooling settings - Ensure your application is properly reusing HTTP connections to FusionAuth rather than creating new connections for each request
• Load patterns - Monitor whether timeouts occur during specific load patterns that might indicate resource constraints

Related Documentation

• FusionAuth Networking Configuration - Configure how FusionAuth determines client IP addresses and network settings
• Troubleshooting Connection Issues - General guidance on troubleshooting API calls and connectivity
• Deploying FusionAuth on Google Kubernetes Engine - Best practices for running FusionAuth on GCP
• Google Cloud Platform with FusionAuth - Overview of deploying FusionAuth in GCP environments

External Resources

• GCP Cloud NAT port allocation documentation
• Consider monitoring your NAT port usage to right-size these settings for your workload