Kubernetes nginx ingress controller with FusionAuth Backend

kbi · 27 May 2021, 15:40

Hi,

Disclaimer: I am fairly new to FusionAuth, and OAuth2 in general.

TL; DR:
Trying: To configure kubernetes nginx ingress controller with FusionAuth backend.
Working: FusionAuth is running, authenticates properly, redirects properly, reachable.
Not working: ingress controller confuses FusionAuth's login page as "successful authentication"

Long Version:
I am trying to configure kubernetes nginx ingress with a FusionAuth backend to secure multiple services. The services themselves do not have any sign in requirement. The idea is that when the user hits an ingress endpoint, the ingress will direct the user to FusionAuth, and upon successful completion of authentication, the ingress will direct the user to the actual service endpoint.

FusionAuth is working, accounts are setup, and if I hit the oath ldp login url directly, FusionAuth properly authenticates and redirects to the desired endpoint.

I have been trying to utilize the auth-url annotation with the FusionAuth's oauth ldp login URL . However, ingress controller seems to interpret an HTTP response code 200 from auth-url as successful authentication. When the user is directed to FusionAuth without a valid session,FusionAuth presents the login page which comes with the response code of 200, and consequently, is treated as ingress controller as successful authentication.

This would indicate the need of a (reverse)proxy / middleman, I cannot find one that would work out of the box with kubernetes and fusionAuth.

So far I have checked these:

Any help in this regard will be appreciated.

joshua · 2 Jun 2021, 17:02

@kbi

I am less familiar with K8s but will try and offer some guidance.

I assume you have consulted the docs from the official site - but have linked below if not.
https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/

You may also want to check out the community-supported contribution repo below for advice. Linked below.
https://github.com/FusionAuth/fusionauth-contrib/tree/master/Reverse Proxy Configurations

Other related links
https://github.com/FusionAuth/fusionauth-containers
https://github.com/FusionAuth/fusionauth-issues/issues/470

After more tinkering, if you think this might be a bug with FusionAuth (based on your question, this doesn't seem to be the case) you can file an issue here.

I hope this helps!

Thanks,
Josh

samet · 24 Jun 2021, 21:23

Hello, I added the necessary path css js settings to nginx for the Oauth path.

I use this url for login : /oauth2/authorize?client_id={client_id}&redirect_uri={redirect_uri}&response_type=code&tenantId={tenantId}

I can see the Fusion auth login page with information such as client id redirect url etc.

But when I type the username and password and press the login button,

I get 403 error. Probably something is missing in my ngnix settings.

Do you have any idea about it?

Form submit response:

So far I have checked these:

Any help in this regard will be appreciated.

joshua · 29 Jun 2021, 00:06

@samet

Looks like the images cut out -- those might be helpful if diagnosing. Are you getting any errors from you log files?

I don't have any immediate solutions to try but will post back if something occurs to me.

Thanks,
Josh

samet · 29 Jun 2021, 23:26

Hi @joshua thanks for your answer.

I couldn't see an error log on the fusion auth side.

But we are using Nginx and I saw 403 error logs on that side,

/var/log/nginx/access.log: - [25/Jun/2021:12:37:24 -0400] "POST /oauth2/authorize HTTP/1.1" 403 0 "https://fusionauthcallbackurl.com/callbackmethod/"

as far as I understand Fusion Auth wants to do a redirect (for Oauth login callback), but nginx returns 403 and sometimes 404 error for this request. Do you have an opinion for this problem?

What can I do, so that Nginx doesn't block/change the redirect that fusion auth wants to do.

joshua · 30 Jun 2021, 15:00

Hi @samet,

I am certainly no k8's expert, but here are a few other things to try.

It could be that you are not setting up the proper headers in NGinx to all the proper processing of CSRF cookies. You could try researching and ensuring that those headers are set properly.

Regarding

I couldn't see an error log on the fusion auth side.

There should be a log created (more information here). In a K8s environment, you may have to install a logger to effectively capture any logs from FusionAuth.

Lastly, I assume that you are referencing or using the community maintained k8's repository, but just in case:
https://github.com/FusionAuth/fusionauth-containers/tree/master/kubernetes

Hopefully, this gives you some roads to travel. If you find yourself really stuck, and your business model supports it, we do offer support plans for additional troubleshooting as well.

Thanks,
Josh

joshua · 30 Jun 2021, 15:15

@samet

To add one other consideration:

If it is a CORS issue, then can enable debug on the CORS filter and get a really nice Debug Event Log to tell you what is happening (it's a super slick feature that our dev team released)

If it is not CORS, then it still may be our CSRF, however, that should be a 401. In reviewing with the dev team, most signs point to the fact that you have a CORS issue (either ours or yours).

Additional documentation can be found here
https://fusionauth.io/docs/v1/tech/reference/cors/

Thanks,
Josh

shaunladd · 2 Jul 2021, 14:59

Hello @joshua,
I enabled debug on the CORS filter and in the CORS event log we could see a mismatch of protocols http vs http. I then added the headers that RobotDan has mentioned in other posts and now when we login instead of getting a 403, we get a 404 from Nginx that /oauth2/authorize does not exist. Is getting a 404 at this point expected? With this configuration there is nothing logged so I am assuming the previous 403 was from a CORS error but now this 404 is unclear to us.

joshua · 7 Feb 2021, 18:28

@samet & @shaunladd

Getting a 404 could be related to how you set up the proxy. I will check to see if my colleagues have some thoughts and post back.

In the meantime, I posted some additional links to explore. If you hit a "safe/benign" link on FusionAuth (for instance / root) --- do you get a 404? If so, this would suggest FusionAuth/Proxy/OtherApps are not communicating with each other due to a proxy config issue (to my eyes, anyway)

Also, are you getting any errors in the error logs?

Thanks,
Josh

These posts seemed helpful/interesting to explore (but may or may not be fully relevant)

joshua · 2 Jul 2021, 20:07

@samet @kbi @shaunladd

After talking some more with colleagues, this error behavior related to your Nginx configuration and not related to FusionAuth directly.

You can check out two reverse proxy repos that are community maintained below for further guidance:

As I often have to remind myself, when deals with layers of abstraction, it is always best to start with the simplest base layer and build from there. So in this case, a good approach may be to expose the FusionAuth node directly and try to get it working without a proxy. Or try getting a proxy working with FusionAuth on a VM/docker locally (as opposed to the cloud/remote), just to remove variables.

I hope this helps!

Thanks,
Josh