HTTP 404 always returned with load balancer in front of cloudflared tunnel hostname

[blazing lintel]

I'm attempting to move our API servers over to cloudflared tunnels, and so I've got an origin server with cloudflared running on it, and a public hostname that points towards the API server process on port 8080. I've also got an edge certificate made for this public hostname and it is deployed. (api1.lax2.domain.com) When I go to the public hostname URL, I can make API requests fine as expected.

The challenge comes when I put a CF load balancer in front of the public hostname. I've got health checks at /health/ready, and when I enable the pool, I see HTTP requests making it to the API server process. However, when I actually attempt to interact with my api (api.domain.com), I'm always getting an HTTP 404.

Doing a tcpdump on the lo interface for HTTP traffic shows the /health/ready HTTP requests being done, however I don't see any of my requests that I make towards api.domain.com. It seems cloudflare is arbitrarily returning a 404 status from their end?

[odd marsh]

are you able to share an example url? if not, could you dump the http response headers from a 404?
also check the cloudflared process logs for any errors

[blazing lintel]

I'll be honest, I'm testing this out on a production system lol. Let me try to set up a non-production example

I'm going to try using this cfargotunnel method

[blazing lintel]

the docs online make it sound so simple yet it absolutely is not

[odd marsh]

oh, i see

so the host header load balancer sends is the header for the load balancer hostname, you need a public hostname of api-test.
which if im remembering this right, the dash will prevent you adding because it thinks it knows better than you, and the solution is to use api or make use of the fallback origin

[blazing lintel]

Seems like the only way to use this cfargotunnel domain method is to use a custom config on the origin, but I can't seem to figure out the correct way to configure it

it seems like the config geeps getting overridden by cloudflare, and isn't using the local one in /usr/local/etc/cloudflared/config.yml

[blazing lintel]

cloudflared doesn't seem to read config files correctly if installed as a service 🙃

this all seems like it would be a pretty simply configuration. I don't get why it should be so difficult and buggy

oh it's because when you use cloudflared service install it makes a copy of your config and then uses the copy

that doesn't seem to be documented anywhere

it doesn't copy the credentials or anything though

Ok, looks like I got it working. None of the docs helped me figure this out, honestly. I had to create a locally managed tunnel (remotely managed tunnels will not work), then I had to use cloudflared service install to install a service, then I configured the file in /etc/cloudflared/config.yml to map all requests to my api process using the ingress section, then the load balancer was configured with the cfargotunnel.com domain (no host header). That finally routes requests correctly

I could possibly mess with hosts but honestly it seems super broken on its own in different ways, so I'll avoid that mess. These tunnels only have to serve one HTTP service each, so I can just ignore hosts

I still have no clue if edge certificates are necessary

this would all be significantly improved if there was better management of cloudflared via the zero trust portal, honestly. It's deceptive in that it makes you think you can do it all easily via portals, but running an api server behind a tunnel behind a load balancer is, as I said, apparently too niche to have UI for.

[odd marsh]

by the way if you wanted to avoid exposing each origin as its own subdomain you can set the load balancer origin directly to the <uuid>.cfargotunnel.com using the tunnel id

then the only way to access it will be via the LB

this is how my tunnel LBs are setup