We have a deployment of GoodData.CN on AWS that terminates TLS on a network load balancer that forwards traffic as HTTP to our pods running the ingress-nginx controller. We're experiencing an issue with the authentication callbacks from Auth0, which call back to our organization URL with the scheme set to HTTP rather than HTTPS - presumably because our ingresses are configured on port 80. But our load balancer is not configured to handle HTTP (only HTTPS) so the callbacks are failing. I'm not sure where the callback scheme is configured in the Helm chart (or if this is configurable), but we need the callback from Auth0 to use HTTPS so that it's properly handled by the load balancer. Is it possible to configure the scheme of the oauth callback URL's? If so, where / how is this done? Thanks so much.

Hi Pete, I’m assuming that the allowed callback URL/s are configured to use https protocol on Auth0 side, is that correct?

Yes, allowed callbacks are configured to use https

Hi @Pete Lorenz, it’s not possible to re-configure the scheme of oauth callbacks. And you’re right, the callback is built on Ingress port. It always takes the scheme which was used in the “authorization request” API URL processed by the Authorization Service (

authService

). This URL is just passed by Ingress there. The flow is following: • API client calls

<baseUrl>/oauth2/authorization/<domain>

• API client is redirected to OIDC IdP with the

redirect_uri

query parameter set to

<baseUrl>/login/oauth2/code/<domain>

As I understand, in your case, the API client is the network load balancer which uses

http

. The

<baseUrl>

is always computed from the API (authorization) request and it cannot be re-configured as well. I think that this will also apply to all other GD.CN APIs. If there’re any links (e.g. self/next used for paging) generated by any GD.CN microservice, they’ll contain the

http

scheme as well.

Am I understanding that the configuration we're trying to achieve is impossible?

As for the workaround - how does your ingress controller config look like? Does it contain annotations:

force-ssl-redirect: "true"
use-forwarded-headers: "true"

x-forwarded-* headers help to identify public url schema for redirect_uri generation. Can you review the following thread https://gooddataconnect.slack.com/archives/C01P3H2HTDL/p1680736799811699 if it could apply to your infrastructure?

Here's our ingress config

Hello @Pete Lorenz, just to clean some “fog”, from your outputs it seems like you are modifying Helm Chart templates instead modifying regular ConfigMap applied to Kubernetes. Would you please provide output from

kubectl get cm -n ingress-nginx ingress-nginx-controller -o yaml

so that I can see the actual ConfigMap in the cluster? Any changes should be modified in

values.yaml

and then installed to cluster using Helm. Modifications to Helm Chart (templates) are not recommended as your changes might be reverted during upgrades. Output should look something like:

$ kubectl get cm -n ingress-nginx ingress-nginx-controller -o yaml
apiVersion: v1
data:
  allow-snippet-annotations: "true"
  brotli-min-length: "1024"
  brotli-types: application/vnd.gooddata.api+json application/xml+rss application/atom+xml
    application/javascript application/x-javascript application/json application/rss+xml
    application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json
    application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon
    text/css text/javascript text/plain text/x-component
  client-body-buffer-size: 1m
  enable-brotli: "true"
  force-ssl-redirect: "true"
  gzip-min-length: "3072"
  gzip-types: application/vnd.gooddata.api+json application/atom+xml application/javascript
    application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject
    application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml
    application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component
  large-client-header-buffers: 4 32k
  proxy-buffer-size: 32k
  use-forwarded-headers: "true"
  use-gzip: "true"
kind: ConfigMap
metadata:
  [...]

Hello Martin, I've attached the configmap, not too much surprising, just the header we mentioned. use-forwarded-headers: "true". Agreed but for some reason the values were not getting picked up from the values.yaml so we changed the templates directly for our test.

Thank you for the outputs and the tests. Can you please provide also output for Service?

kubectl get service -n ingress-nginx ingress-nginx-controller -o yaml

so that we can review annotations present? (

<http://service.beta.kubernetes.io/aws-load-balancer-*|service.beta.kubernetes.io/aws-load-balancer-*>

). Is the NLB+TLS setup from k8s by AWS Load Balancer Controller or somehow manually/programatically without k8s? I am asking because in the next step I would like you to try this: • remove

use-forwarded-headers: "true"

from the config/ConfigMap • try annotation

<http://service.beta.kubernetes.io/aws-load-balancer-backend-protocol|service.beta.kubernetes.io/aws-load-balancer-backend-protocol>: tcp

if now it’s http (default) or vice-versa

Thanks Martin. Will try these suggestions. Attached is our service configuration. I have concerns about the second suggestion since AWS support has explicitly told us we must use "http" as the backend protocol because we are terminating TLS on the load balancer, but using "tcp" as the backend protocol will break this because the load balancer will operate in pass through mode.

Actually, can we try something a bit different? I got the idea from: https://github.com/kubernetes/ingress-nginx/issues/2724#issuecomment-661334310 (and I actually already saw some solution using Proxy v2 protocol somewhere) • can you try adding following annotation to

ingress-nginx-controller

service:

<http://service.beta.kubernetes.io/aws-load-balancer-proxy-protocol|service.beta.kubernetes.io/aws-load-balancer-proxy-protocol>: "*"

• remove

use-forwarded-headers

from ConfigMap • and set also

use-proxy-protocol: "true"

in the ConfigMap?

We can try, but AWS support has told us not to use proxy protocol. We initially had this in our configuration but AWS told us to remove it.

From your snippets I assume you (with AWS Support) were trying to reach solution similar to this: https://github.com/kubernetes/ingress-nginx/issues/2724#issuecomment-593769295

FYI AWS's solution is tested because we can add organizations using the metadata-api (using the organization's Authorization Bearer token)

Hi, I'm back 🙂 by any chance, did you accidentally changed the helm value

ingress.lbProtocol

from default

https

to

http

?

No,

lbProtocol

is https

I reviewed your ingress-ctl service configuration. I tried to figure out how to make your current configuration working without modifying LB setup, as we discussed. Findings: 1. Apps using Spring Security oauth2-client are using logic that can correctly discover original url using

X-Forwarded-Host

,

X-Forwarded-Port

,

X-Forwarded-Proto

,

X-Forwarded-Ssl

, and

X-Forwarded-Prefix

headers. These headers are NOT set by NLB, because it runs on Layer 4 (TCP) and is not aware of any Layer 7 (HTTP) headers. 2. As these headers are NOT present in requests passed from NLB, the ingress controller sets these headers based on available information - e.g.

X-Forwarded-Proto

is set based on

$scheme

value. 3. Your configuration terminates SSL on NLB and the request is passed to ingress-ctl on port 80 as plain HTTP, so

$scheme

is

http

. Header is set using

proxy_set_header X-Forwarded-Proto $pass_access_scheme

(initialized from

$scheme

). 4. There's no straightforward way how to later modify proxy headers that were already set by ingress-ctl template. If I attempt to add the same request header with different value, two headers will be set and it is wrong. One option is to write custom LUA plugin that would set

pass_access_scheme

variable to something else. But it is rather complex workaround. To sum it up: it's virtually impossible to properly deliver required

X-Forwarded-*

headers downstream. So I recommend updating your existing configuration of ingress controller with these changes. The other settings are OK and need not to be modified:

controller:
  service:
    annotations:
      # => Replace the current value "http" with "ssl".
      <http://service.beta.kubernetes.io/aws-load-balancer-backend-protocol|service.beta.kubernetes.io/aws-load-balancer-backend-protocol>: "ssl"
    targetPorts:
      http: http
      https: https   # Now you have "http"

You may consider adding annotation

<http://service.beta.kubernetes.io/aws-load-balancer-target-group-attributes|service.beta.kubernetes.io/aws-load-balancer-target-group-attributes>: "deregistration_delay.connection_termination.enabled=true,preserve_client_ip.enabled=true"

but it is not necessary.

As suggested above, I believe we will need a code update to allow explicit configuration of the OIDC connector to send HTTPS callbacks to our auth provider. Is this possible?

The

redirect_uri

is created automatically by Spring framework using

"{baseUrl}/{action}/oauth2/code/{registrationId}"

URI template. I can't estimate how much effort it will take to reimplement part of Spring Security framework to make the

scheme

(and possibly also host and port) configurable for this single use case. @Peter Plochan Any guess? (not sure where exactly it is, probably in org.springframework.security.config.oauth2.client)

We tried the above suggestion to set backend protocol to ssl and https target port to https. When we navigate to our GD org workspace, the auth provider is correctly receiving the registered callback with the https scheme and we are prompted to authenticate. However, we are now receiving "502 Bad Gateway" from nginx. Any suggestions as to what we can do to resolve this? @Robert Moucha

Glad to hear it's finally working. Such proxy buffer setting is surprising, as your global config shows

proxy-buffer-size: 32k

- it should be enough for the most of situations. Proxy-buffers-number is not explicitly set so it defaults to 4. It's possible your oauth2 tokens contain a lot of claims so as they are passed and encrypted to application cookies, the header size exceeds the given 32k limit.