Setting up a remote runner instance accessible from gitlab runners and hitting this error: error reading server preface: http2: failed reading the frame payload: http2: frame too large, note that the frame header looked like an HTTP/1.1 header... Requests being made with http2 but expecting http1.1? Should I be trying to ensure the request is http1.1 or configuring the engine a different way here?
#http2 / http1.1 error
brave tendon
karmic delta
Setting up a remote runner instance accessible from gitlab runners and hitting t...
👋 are you getting this from Dagger? first time I'm seeing this
brave tendon
Yes. Runners can reach the dagger engine but getting that error. It starts with "moby.buildkit.v1.Control/Info ERROR" though
karmic delta
do you access the engine directly? or do you have some sort of proxy / balancer in the middle?
brave tendon
Load balancer
karmic delta
ok, that's the issue then
I'd check there
try accesing the engine directly if you can test and that will probably work
brave tendon
The engine works fine from the instance (e.g. setting _EXPERIMENTAL... = tcp://localhost:<port>) so probably right... AWS ALBs support both http2 and http1.1 though, in struggling to see the cause of the above
They fall back to http1.1 if http2 support isn't present
karmic delta
@brave tendon do you have multiple engines behind the LB? or just a single one?
brave tendon
One, the LB listener forwards to a target group containing that instance so it's seamlessly updated when the instance is replaced .. and we get the URL which is preferable
karmic delta
One, the LB listener forwards to a target group containing that instance so it's...
gotcha. I'm not even sure if it will work behind a L7 balancer even with a single instance. Because even though the engine exposes graphQL API, there's some gRPC connection upgrade happening for some things. @latent jewel any thoughts here?
@brave tendon I'd try an NLB if possible.. I think that has better chances of working
brave tendon
Ok, ditched the ASG and switched the ALB for an NLB, will see how this goes
latent jewel
generally, my knowledge of load balancers with the kind of connections we use is that you need something with sticky sessions
every request in a session needs to end up at the same machine
brave tendon
There's only one machine here
latent jewel
mm, it's possible the load balancer is still attempting to create multiple http connections to the single machine
karmic delta
Ok, ditched the ASG and switched the ALB for an NLB, will see how this goes
IIRC you should be able to use NLB's with ASGs also
brave tendon
Yeah I've just read that, but in favour of just getting this deployed rather than figuring that out I've dropped the ASG temporarily
karmic delta
Yeah I've just read that, but in favour of just getting this deployed rather tha...
👍 let us know if the NLB works.. I think it might but not fully sure
brave tendon
Yeah @karmic delta that worked instantly. I put the ASG back in as well, turns out it's a one-liner to have your NLB listener forward to a target group. Been a while since I've set up an NLB, I had it in mind they couldn't forward to a TG.
Need to do some TLS stuff next but at least now it's running