Add support for having different URLs for user redirection and provider endpoints to Gatekeeper

[devopsix]

TL;DR With Gatekeeper, it is not possible to have users redirected to public URLs like https://www.example.com/auth/realms/test/protocol/openid-connect/auth?client_id=test&redirect_uri=... while using internal URLs like http://keycloak:8080/auth/realms/etlms/protocol/openid-connect/token for provider endpoints.

In a containerized environment I would like to use Keycloak Gatekeeper for securing a web service which has no OAuth2/OIDC support. The web service, Gatekeeper and Keycloak containers all are in one “scope” (same namespace in Kubernetes in my target setup; one docker-compose.yml file for demonstrating the issue). For reasons beyond my control I cannot point Gatekeeper to the “public Keycloak URL” (say https://www.example.com/auth/...) but have to use an “internal Keycloak URL” with a container/service name for the hostname part (e.g. http://keycloak:8080/auth/...).

URLs in the OIDC configuration retrieved from the discovery URL contain either the realm frontend URL (if configured) or URLs crafted from the request's Host header (if realm frontend URL not configured). But for my setup I need a mix of both in Gatekeeper and even after browsing the Gatekeeper sources I don't see how that would be possible.

I will try demonstrating the issue.

Given the attached docker-compose.yml file:

1. Run docker-compose up -d webservice
2. Open http://localhost:8081 in web browser and verify “Welcome to nginx!” page is shown.
3. Run docker-compose up -d keycloak
4. Open http://localhost:8080/auth/admin in web browser and log in as user “keycloak” with password “keycloak”.
5. Create realm “test”. Do not configure frontend URL.
6. Create client “test” (OIDC confidential).
7. Copy client secret to docker-compose.yml file.
8. Run docker-compose up -d gatekeeper
9. Open http://localhost:8082 in web browser.

Expected result: Login page loads (http://localhost:8080/auth/realms/test/protocol/openid-connect/auth?client_id=test&redirect_uri=...)

Actual result: Login page fails to load because user is redirected to http://keycloak:8080/auth/realms/test/protocol/openid-connect/auth?client_id=test&redirect_uri=... but hostname “keycloak” does not resolve for the user.

1. Set Frontend URL of test realm to http://localhost:8080.
2. Run docker-compose stop gatekeeper
3. Run docker-compose up gatekeeper
4. Open http://localhost:8082 in web browser.

Expected result: Login page loads (http://localhost:8080/auth/realms/test/protocol/openid-connect/auth?client_id=test&redirect_uri=...)

Actual result: Login page fails to load because Gatekeeper does not start due to {"error": "\"issuer\" in config (http://localhost:8080/realms/test) does not match provided issuer URL (http://keycloak:8080/auth/realms/test)"}.

http://localhost:8080/auth/... is the equivalent of the (user-resolvable) “public Keycloak URL” and http://keycloak:8080/auth/... is the equivalent of the (cluster private) “internal Keycloak URL” in my target setup.

docker-compose.yml and realm-export.json.zip

[stianst]

Gatekeeper points to discovery endpoint, the discovery endpoint then advertises the URLs that Gatekeeper should use. This is not something that should be configured separately in Gatekeeper. It's up to the IdP to advertise URLs to be used.

Keycloak supports using different URLs for frontend request, and backend request. Assuming Gatekeeper uses the backend URL for Keycloak to fetch discovery endpoint, only auth endpoint (used for user-agent redirects) will be advertised on the front-end URL, while backend requests will use internal URL.

So what is the issue? I'm completely lost in your long description..

[devopsix]

Keycloak supports using different URLs for frontend request, and backend request. Assuming Gatekeeper uses the backend URL for Keycloak to fetch discovery endpoint, only auth endpoint (used for user-agent redirects) will be advertised on the front-end URL, while backend requests will use internal URL.

That's what the second test case covers. In this case a frontend URL (http://localhost:8080/auth) is configured. But Gatekeeper fails to start.

curl http://keycloak:8080/auth/realms/test/.well-known/openid-configuration executed from within the gatekeeper container yields this configuration:

{
  "issuer": "http://localhost:8080/auth/realms/test",
  "authorization_endpoint": "http://localhost:8080/auth/realms/test/protocol/openid-connect/auth",
  "token_endpoint": "http://keycloak:8080/auth/realms/test/protocol/openid-connect/token",
  "token_introspection_endpoint": "http://keycloak:8080/auth/realms/test/protocol/openid-connect/token/introspect",
  "userinfo_endpoint": "http://keycloak:8080/auth/realms/test/protocol/openid-connect/userinfo",
  "end_session_endpoint": "http://localhost:8080/auth/realms/test/protocol/openid-connect/logout",
  "jwks_uri": "http://keycloak:8080/auth/realms/test/protocol/openid-connect/certs",
  "check_session_iframe": "http://localhost:8080/auth/realms/test/protocol/openid-connect/login-status-iframe.html",
  ...
  "introspection_endpoint": "http://keycloak:8080/auth/realms/test/protocol/openid-connect/token/introspect"
}

You are right: The auth endpoint is derived from the frontend URL.

But: Also the issuer URL seems to be derived from the frontend URL. And that seems to make Gatekeeper fail to start: “"issuer" in config (http://localhost:8080/auth/realms/test) does not match provided issuer URL (http://keycloak:8080/auth/realms/test)” (The actual error seems to come from coreos/go-oidc.)

I am not saying that the issuer URL as advertized by Keycloak is incorrect. For a public IdP I completely understand that. But in an containerized application that Keycloak and Gatekeeper are components of, making the config discovery request to a backend URL IMHO is a valid use case.

I should probably rephrase the issue title and focus the description to the second test case.

[stianst]

Gatekeeper should read issuer from discovery endpoint, so if it doesn't then that's a bug in Gatekeeper

[stianst]

issuer doesn't have to be a URL, it just happens to be that in most cases

[bard]

@abstractj this looks entirely distinct from #524. @devopsix I'm facing the same problem, did you find a workaround?

[bard]

In case someone stumbles on this, this is how I worked around the issue in docker-compose.

Let's say that the keycloak container is reachable from outside the stack as keycloak.localtest.me (because you reverse-proxied it with ngnix, traefik, etc). The following makes the keycloak container reachable on the same name from inside the stack, so gatekeeper doesn't run into the mismatch reported by OP.

version: '3.6'
services:

  # ...

  keycloak:
    image: quay.io/keycloak/keycloak:9.0.2
    command: -Djboss.http.port=80
    volumes:
      # ...
    environment:
      # ...
    labels:
      # ...
    networks:
      default:
        aliases:
          - keycloak.localtest.me
    sysctls:
      # necessary to allow keycloak to bind to port <1024 since it doesn't run as root
      - net.ipv4.ip_unprivileged_port_start=0

[abstractj]

@bard no problem, reopened so we can discuss more about your scenario.

[devopsix]

I'm facing the same problem, did you find a workaround?

@bard, I am using OAuth2 Proxy now. I had investigated similar work-arounds as yours for Kubernetes but with no success.

[bard]

Thanks @abstractj. My scenario is the same as OP's second case above, where gatekeeper and the browser access keycloak through different URLs, respectively the private (LAN) and public (Internet) one.

I think that gatekeeper doesn't support that scenario, because if the discovery URL is e.g. http://keycloak:8080/auth/realms/demo and authorization endpoint's host as returned by discovery is something other than keycloak:8080, gatekeeper will not start, with a "issuer in config does not match provided issuer URL" error.

The reason I hoped it would work was because I had read the following in keycloak's docs, configured keycloak accordingly (with -Dkeycloak.frontendUrl), and assumed that, if keycloak supported the scenario, gatekeeper would:

Frontend request do not have to have the same context-path as the Keycloak server. This means you can expose Keycloak on for example https://auth.example.org or https://example.org/keycloak while internally its URL could be https://10.0.0.10:8080/auth.

This makes it possible to have user-agents (browsers) send requests to ${project.name} through the public domain name, while internal clients can use an internal domain name or IP address. (https://www.keycloak.org/docs/latest/server_installation/#default-provider)

That said, they're different projects, so wrong assumption; also the reason why I was after this was for convenience in development only, and the workaround seems to be holding well.

[abstractj]

Hi @JoelSpeed I can be wrong, but I believe this may be one of the use cases you mentioned as core feature in oauth2-proxy. If that's not the case, let's create a new issue.

[JoelSpeed]

@abstractj I don't think it is no.

As for the topic of this conversation, Gatekeeper (or OAuth2 Proxy) needs an OIDC issuer URL to perform discovery. The way that OIDC works, this has to be a publicly accessible URL. OIDC mandates that the issuer is public so that the URLs it provides can be accessed by the end user as part of the authentication flow. Unfortunately, without allowing OIDC issuer verification to be skipped, there's not really a workaround for this apart from manually providing the token URLs etc yourself.

In OAuth2 Proxy, you have the option to skip issuer verification from our next major release, but this is a rare use case as far as I can tell (One of the Azure AD options requires it?). Normally I would expect GateKeeper or OAuth2 Proxy to hit the public endpoint for doing discovery.

If you want to play with that you may be able to using our test environments that were recently merged https://github.com/oauth2-proxy/oauth2-proxy/tree/master/contrib/local-environment

(I don't know GateKeeper well enough to give advice about it I'm afraid)

[cowlingj]

Rather than skipping the verification entirely, It could be nice to pass an issuer-url option and have issuer verification use that instead

[mreinli]

Gatekeeper does not start if the Issuer returned from the discovery endpoint is not matching the URL defined as discovery-url in the Gatekeeper config. As noted by OP, in Kubernetes environments you reach the discovery endpoint with a different URL than defined as Issuer (which is the public accessible URL = Frontend URL in Keycloak). Gatekeeper is not compatible with the Frontend URL feature of Keycloak. For me this is clearly a bug.

[cowlingj]

After more playing around with this, I found the combination of keycloak's frontendUrl option and both the discovery-url and openid-provider-proxy work.

1. Set the frontendUrl (for keycloak) to whatever the url is external to the cluster (this is our public issuer url) (e.g. http://localhost:3000/auth/)
2. Set the discovery-url as if keycloak was being accessed outside of the cluster (e.g. localhost:3000/auth/realms/master/
3. Set the openid-provider-proxy to the internal address to keycloak (e.g. http://keycloak:8080).

I think this may be the solution to this issue.

[devopsix]

@cowlingj I have updated my original example to louketo-proxy:1.0.0 and keycloak:10.0.2 and tried using the openid-provider-proxy option. It seems I get one step further now but I cannot seem to authenticate successfully.

Given the attached docker-compose.yml in your current working directory.

1. docker-compose up -d webservice keycloak
2. Open http://localhost:8080/auth/admin in web browser and log in as user “keycloak” with password “keycloak”.
3. Create realm “test”.
4. Set Frontend URL of test realm to http://localhost:8080/auth.
5. Create client “test” (OIDC confidential) and create a user in this realm.
6. Copy client secret to docker-compose.yml file.
7. docker-compose up louketo
8. Open http://localhost:8082 in web browser.
9. You will be taken to the login page. Enter valid credentials.
10. You will be redirect to http://localhost:8082/oauth/callback?state=... which returns HTTP status 403.

It seems that the token exchange fails because it is tried directly with the public provider URL without going through the “provider proxy”:

> docker-compose up louketo
Creating louketo-proxy-539_louketo_1 ... done
Attaching to louketo-proxy-539_louketo_1
louketo_1     | 2020-07-10T06:44:33.612Z        info    starting the service    {"prog": "louketo-proxy", "author": "Louketo", "version": "v2.3.0 (git+sha: 9eca196-dirty, built: 01-07-2020)"}
louketo_1     | 2020-07-10T06:44:33.612Z        info    attempting to retrieve configuration discovery url      {"url": "http://localhost:8080/auth/realms/test", "timeout": "30s"}
louketo_1     | 2020-07-10T06:44:33.628Z        info    successfully retrieved openid configuration from the discovery
louketo_1     | 2020-07-10T06:44:33.631Z        info    enabled reverse proxy mode, upstream url        {"url": "http://webservice:80"}
louketo_1     | 2020-07-10T06:44:33.631Z        info    using session cookies only for access and refresh tokens
louketo_1     | 2020-07-10T06:44:33.631Z        info    adding a default denial into the protected resources
louketo_1     | 2020-07-10T06:44:33.631Z        info    protecting resource     {"resource": "uri: /*, methods: DELETE,GET,HEAD,OPTIONS,PATCH,POST,PUT,TRACE, required: authentication only"}
louketo_1     | 2020-07-10T06:44:33.631Z        warn    no redirection url has been set, will use host headers
louketo_1     | 2020-07-10T06:44:33.632Z        info    Louketo proxy service starting  {"interface": ":8082"}
louketo_1     | 2020-07-10T06:46:44.934Z        error   no session found in request, redirecting for authorization      {"error": "authentication session not found"}
louketo_1     | 2020-07-10T06:46:48.272Z        error   unable to exchange code for access token        {"error": "Post \"http://localhost:8080/auth/realms/test/protocol/openid-connect/token\": dial tcp 127.0.0.1:8080: connect: connection refused"}

[cowlingj]

@devopsix what have you set openid-provider-proxy to? The error in the logs suggests that gatekeeper is trying to access the wrong url.

[devopsix]

I use --openid-provider-proxy=http://keycloak:8080.

Without that option louketo-proxy would fail to start as originally described:

louketo_1     | 2020-07-10T12:29:59.732Z        info    attempting to retrieve configuration discovery url      {"url": "http://localhost:8080/auth/realms/test", "timeout": "30s"}
louketo_1     | 2020-07-10T12:29:59.732Z        warn    failed to get provider configuration from discovery     {"error": "Get \"http://localhost:8080/auth/realms/test/.well-known/openid-configuration\": dial tcp 127.0.0.1:8080: connect: connection refused"}

The the attached docker-compose.yml shows all options:

version: "3.7"
services:
  webservice:
    image: nginx
    ports:
      - '8081:80'
  louketo:
    image: quay.io/louketo/louketo-proxy:1.0.0
    command: >-
      --listen=:8082
      --discovery-url=http://localhost:8080/auth/realms/test
      --client-id=test
      --client-secret=...
      --upstream-url=http://webservice:80
      --no-redirects=false
      --openid-provider-proxy=http://keycloak:8080
    ports:
      - '8082:8082'
  keycloak:
    image: jboss/keycloak:10.0.2
    environment:
      KEYCLOAK_USER: keycloak
      KEYCLOAK_PASSWORD: keycloak
      KEYCLOAK_LOGLEVEL: DEBUG
    ports:
      - "8080:8080"

[Morriz]

@devopsix docker compose is not kubernetes, it has no access to localhost, so you either have to provide the outside host replacement name (google it), or use internal service name. In case of the first just change http://localhost:8080/auth/realms/test to use docker.for.mac.host.internal in case of mac.

[devopsix]

docker compose is not kubernetes, it has no access to localhost

@Morriz, technically that's correct, no doubt.

But please note what @cowlingj suggested above:

2. Set the discovery-url as if keycloak was being accessed outside of the cluster (e.g. localhost:3000/auth/realms/master/)

As far as I understand, this is only expected to work in combination with using Keycloak as OpenID provider proxy to itself (--openid-provider-proxy=http://keycloak:8080).

Using the internal service name will result in the "issuer" in config (http://localhost:8080/realms/test) does not match provided issuer URL (http://keycloak:8080/auth/realms/test) error described in the original post.

[Morriz]

Yes, a proxy like leuketo or oauth2-proxy (it has better cookie domain support) solves that.