We have a deployment of <GoodData.CN> on AWS that ...
# gooddata-cn
We have a deployment of GoodData.CN on AWS that terminates TLS on a network load balancer that forwards traffic as HTTP to our pods running the ingress-nginx controller. We're experiencing an issue with the authentication callbacks from Auth0, which call back to our organization URL with the scheme set to HTTP rather than HTTPS - presumably because our ingresses are configured on port 80. But our load balancer is not configured to handle HTTP (only HTTPS) so the callbacks are failing. I'm not sure where the callback scheme is configured in the Helm chart (or if this is configurable), but we need the callback from Auth0 to use HTTPS so that it's properly handled by the load balancer. Is it possible to configure the scheme of the oauth callback URL's? If so, where / how is this done? Thanks so much.
Does anyone have an answer? @Moises Morales @Boris @Marek Horvat @Martin Burian
Hi Pete, I’m assuming that the allowed callback URL/s are configured to use https protocol on Auth0 side, is that correct?
How does the issue manifests? What errors do you see?
Yes, allowed callbacks are configured to use https
Part of the issue is trying to figure out what parts of the callback URL are configured in GD and what comes from auth0
When I access the organization URL, the browser redirects to a URL that looks like this:
Copy code
I'm assuming (perhaps wrongly) that this URL originates in GD.CN. The part that isn't working is the "http" after redirect_uri. If I change this to "https" I get a login prompt. Otherwise, I see this error message:
Copy code
Callback URL mismatch.
The provided redirect_uri is not in the list of allowed callback URLs.
Please go to the Application Settings page and make sure you are sending a valid callback url from your application.
I think this means that the requested URL doesn't match because it's http but the URL registered with auth0 is https. Registered callback is:
Copy code
The part that has changed from our past configuration is that we are now terminating TLS at the load balancer not the nginx ingress (so our ingress is port 80 not 443). The callback seems to want to match the ingress port instead of the LB port.
Hi @Pete Lorenz, it’s not possible to re-configure the scheme of oauth callbacks. And you’re right, the callback is built on Ingress port. It always takes the scheme which was used in the “authorization request” API URL processed by the Authorization Service (
). This URL is just passed by Ingress there. The flow is following: • API client calls
• API client is redirected to OIDC IdP with the
query parameter set to
As I understand, in your case, the API client is the network load balancer which uses
. The
is always computed from the API (authorization) request and it cannot be re-configured as well. I think that this will also apply to all other GD.CN APIs. If there’re any links (e.g. self/next used for paging) generated by any GD.CN microservice, they’ll contain the
scheme as well.
Am I understanding that the configuration we're trying to achieve is impossible?
We're not able to terminate TLS on the ingress because we're using Amazon certs (this is a requirement) that have no integration with nginx and we cannot put cert private key in our cluster (security policy). So the above approach using an AWS LB to terminate TLS is our only option. We'll need a workaround to get the callbacks to work with our LB.
As for the workaround - how does your ingress controller config look like? Does it contain annotations:
Copy code
force-ssl-redirect: "true"
use-forwarded-headers: "true"
x-forwarded-* headers help to identify public url schema for redirect_uri generation. Can you review the following thread https://gooddataconnect.slack.com/archives/C01P3H2HTDL/p1680736799811699 if it could apply to your infrastructure?
Here's our ingress config
yes, the referenced thread seems to describe our symptoms exactly
We attempted the suggestions posted in the above thread: https://gooddataconnect.slack.com/archives/C01P3H2HTDL/p1680736799811699. When we connect to the organization URL, we're getting an ERR_TOO_MANY_REDIRECTS error. We already had the annotation use-forwarded-headers: "true". We added the annotation force-ssl-redirect: "true". This new annotation may be conflicting with two annotations we were instructed to add by AWS support:
Copy code
allow-snippet-annotations: "{{ .Values.controller.allowSnippetAnnotations }}"
  http-snippet: |
    server {
      listen 2443;
      return 308 https://$host$request_uri;
Attaching our ingress-nginx controller configmap
Will try again without the above annotations suggested by AWS support (we were experimenting with a proxy protocol but ended up not needed this).
What we tried: Experiment 1: we set the following annotations under "data:" in controller-configmap.yaml under ingress-nginx templates:
Copy code
use-forwarded-headers: "true"
force-ssl-redirect: "true"
Observed: browser returns an ERR_TOO_MANY_REDIRECTS error. Experiment 2: we set only the following annotation under "data:" in controller-configmap.yaml under ingress-nginx templates (which seems to be Robert's initial suggestion):
Copy code
use-forwarded-headers: "true"
Observed: this does not solve the problem as the callback URL scheme is still "http" instead of "https". The oauth provider fails with a validation error saying that our callback URL (with http) does not match the allowed callback (with https). We need guidance about how to configure the ingress-nginx controller for oauth with a load balancer that terminates TLS and forwards HTTP to the ingresses.
Hello @Pete Lorenz, just to clean some “fog”, from your outputs it seems like you are modifying Helm Chart templates instead modifying regular ConfigMap applied to Kubernetes. Would you please provide output from
kubectl get cm -n ingress-nginx ingress-nginx-controller -o yaml
so that I can see the actual ConfigMap in the cluster? Any changes should be modified in
and then installed to cluster using Helm. Modifications to Helm Chart (templates) are not recommended as your changes might be reverted during upgrades. Output should look something like:
Copy code
$ kubectl get cm -n ingress-nginx ingress-nginx-controller -o yaml
apiVersion: v1
  allow-snippet-annotations: "true"
  brotli-min-length: "1024"
  brotli-types: application/vnd.gooddata.api+json application/xml+rss application/atom+xml
    application/javascript application/x-javascript application/json application/rss+xml
    application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json
    application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon
    text/css text/javascript text/plain text/x-component
  client-body-buffer-size: 1m
  enable-brotli: "true"
  force-ssl-redirect: "true"
  gzip-min-length: "3072"
  gzip-types: application/vnd.gooddata.api+json application/atom+xml application/javascript
    application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject
    application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml
    application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component
  large-client-header-buffers: 4 32k
  proxy-buffer-size: 32k
  use-forwarded-headers: "true"
  use-gzip: "true"
kind: ConfigMap
Hello Martin, I've attached the configmap, not too much surprising, just the header we mentioned. use-forwarded-headers: "true". Agreed but for some reason the values were not getting picked up from the values.yaml so we changed the templates directly for our test.
Our data section looks a bit different from yours, let me try with the values you have in "data"
Reapplied the configmap and restarted the ingress-nginx-controller pod, waiting for the LB target group to reinitialize ...
now our configmap looks like this
The above settings result in ERR_TOO_MANY_REDIRECTS. As we observed yesterday, the combination of the headers
Copy code
use-forwarded-headers: "true"
force-ssl-redirect: "true"
results in a permanent redirect loop (308 status)
Redirect loop from Firefox Developer Tools:
Thank you for the outputs and the tests. Can you please provide also output for Service?
kubectl get service -n ingress-nginx ingress-nginx-controller -o yaml
so that we can review annotations present? (
). Is the NLB+TLS setup from k8s by AWS Load Balancer Controller or somehow manually/programatically without k8s? I am asking because in the next step I would like you to try this: • remove
use-forwarded-headers: "true"
from the config/ConfigMap • try annotation
<http://service.beta.kubernetes.io/aws-load-balancer-backend-protocol|service.beta.kubernetes.io/aws-load-balancer-backend-protocol>: tcp
if now it’s http (default) or vice-versa
👍 1
Thanks Martin. Will try these suggestions. Attached is our service configuration. I have concerns about the second suggestion since AWS support has explicitly told us we must use "http" as the backend protocol because we are terminating TLS on the load balancer, but using "tcp" as the backend protocol will break this because the load balancer will operate in pass through mode.
NLB+TLS setup is from AWS Load Balancer Controller version 2.4.3
👍 1
Actually, can we try something a bit different? I got the idea from: https://github.com/kubernetes/ingress-nginx/issues/2724#issuecomment-661334310 (and I actually already saw some solution using Proxy v2 protocol somewhere) • can you try adding following annotation to
Copy code
<http://service.beta.kubernetes.io/aws-load-balancer-proxy-protocol|service.beta.kubernetes.io/aws-load-balancer-proxy-protocol>: "*"
• remove
from ConfigMap • and set also
use-proxy-protocol: "true"
in the ConfigMap?
We can try, but AWS support has told us not to use proxy protocol. We initially had this in our configuration but AWS told us to remove it.
From your snippets I assume you (with AWS Support) were trying to reach solution similar to this: https://github.com/kubernetes/ingress-nginx/issues/2724#issuecomment-593769295
FYI AWS's solution is tested because we can add organizations using the metadata-api (using the organization's Authorization Bearer token)
It's just oauth that's a problem
Actually, the above example differs from our solution because the NLB is in pass-through mode (backend protocol = tcp) - not terminating TLS
We were not able to find a solution online and worked with AWS support directly using NLB's peculiar L7 functionality of terminating TLS and forwarding HTTP.
This solution is alluded to with a few snippets in online forums but we were not able to find a complete example publicly available online.
Rather than changing the LB configuration, I think we need a patch to the code that assumes that the scheme used by the ingress is the scheme to use in the oauth callback, since this assumption is not valid in use cases where TLS is terminated at a load balancer in front of nginx
When we manually change the scheme of redirect_uri in the query string to "https", the oauth request is correctly forwarded to the auth provider. So this is the only necessary change but it is a code change
Hi, I'm back 🙂 by any chance, did you accidentally changed the helm value
from default
(in gooddata-cn deployment)
is https
I reviewed your ingress-ctl service configuration. I tried to figure out how to make your current configuration working without modifying LB setup, as we discussed. Findings: 1. Apps using Spring Security oauth2-client are using logic that can correctly discover original url using
, and
headers. These headers are NOT set by NLB, because it runs on Layer 4 (TCP) and is not aware of any Layer 7 (HTTP) headers. 2. As these headers are NOT present in requests passed from NLB, the ingress controller sets these headers based on available information - e.g.
is set based on
value. 3. Your configuration terminates SSL on NLB and the request is passed to ingress-ctl on port 80 as plain HTTP, so
. Header is set using
proxy_set_header X-Forwarded-Proto $pass_access_scheme
(initialized from
). 4. There's no straightforward way how to later modify proxy headers that were already set by ingress-ctl template. If I attempt to add the same request header with different value, two headers will be set and it is wrong. One option is to write custom LUA plugin that would set
variable to something else. But it is rather complex workaround. To sum it up: it's virtually impossible to properly deliver required
headers downstream. So I recommend updating your existing configuration of ingress controller with these changes. The other settings are OK and need not to be modified:
Copy code
      # => Replace the current value "http" with "ssl".
      <http://service.beta.kubernetes.io/aws-load-balancer-backend-protocol|service.beta.kubernetes.io/aws-load-balancer-backend-protocol>: "ssl"
      http: http
      https: https   # Now you have "http"
You may consider adding annotation
<http://service.beta.kubernetes.io/aws-load-balancer-target-group-attributes|service.beta.kubernetes.io/aws-load-balancer-target-group-attributes>: "deregistration_delay.connection_termination.enabled=true,preserve_client_ip.enabled=true"
but it is not necessary.
As suggested above, I believe we will need a code update to allow explicit configuration of the OIDC connector to send HTTPS callbacks to our auth provider. Is this possible?
is created automatically by Spring framework using
URI template. I can't estimate how much effort it will take to reimplement part of Spring Security framework to make the
(and possibly also host and port) configurable for this single use case. @Peter Plochan Any guess? (not sure where exactly it is, probably in org.springframework.security.config.oauth2.client)
But it will be more effort than adding one annotation and updating target port in ingress-controller setup.
We tried the above suggestion to set backend protocol to ssl and https target port to https. When we navigate to our GD org workspace, the auth provider is correctly receiving the registered callback with the https scheme and we are prompted to authenticate. However, we are now receiving "502 Bad Gateway" from nginx. Any suggestions as to what we can do to resolve this? @Robert Moucha
Note that we only get the "Bad Gateway" error using the callback from our auth provider. We do not get this error when hitting our GD endpoints directly through Postman. Our GD.CN deployment is healthy with all services and pods in a Ready state.
This error from the ingress controller logs appears relevant:
Copy code
2023/08/16 21:37:10 [error] 814#814: *902209 upstream sent too big header while reading response header from upstream, client:, server: <http://gooddata-cn-yzaxytzlzt.gooddata-cn.uswe2.stg1.aws.cwan.io|gooddata-cn-yzaxytzlzt.gooddata-cn.uswe2.stg1.aws.cwan.io>, request: "GET /login/oauth2/code/gooddata-cn-yzaxytzlzt.gooddata-cn.uswe2.stg1.aws.cwan.io?code=TrvKiDLBbxrYzLsyCYYIzFuOWpH3ZzCDex298ce90FVD7&state=7z0qqcW6HLOOKO0QVycvjv14qovVjdu-7CHYrj7HrZuk HTTP/1.1", upstream: "<>", host: "<http://gooddata-cn-yzaxytzlzt.gooddata-cn.uswe2.stg1.aws.cwan.io|gooddata-cn-yzaxytzlzt.gooddata-cn.uswe2.stg1.aws.cwan.io>", referrer: "<https://clearwateranalytics.okta.com/>"
We resolved the issue, we needed the following annotations in our ingress template:
Copy code
<http://nginx.ingress.kubernetes.io/proxy-buffer-size|nginx.ingress.kubernetes.io/proxy-buffer-size>: "1024k"
        <http://nginx.ingress.kubernetes.io/proxy-buffers-number|nginx.ingress.kubernetes.io/proxy-buffers-number>: "8"
After adding these annotations and recreating our ingresses, we no longer get the "Bad Gateway" error and we can log in to our org workspace.
🙌 1
Glad to hear it's finally working. Such proxy buffer setting is surprising, as your global config shows
proxy-buffer-size: 32k
- it should be enough for the most of situations. Proxy-buffers-number is not explicitly set so it defaults to 4. It's possible your oauth2 tokens contain a lot of claims so as they are passed and encrypted to application cookies, the header size exceeds the given 32k limit.