#HTTP client err: read tcp connection reset by peer

19 messages · Page 1 of 1 (latest)

short latch
#

Hello fellow gophers!

I'm relatively new to Go, and found some nice opportunity to try it out for some k8s cronjobs.

Problem is that when running inside k8s (more specifically, AWS eks), all requests from HTTP client (using default one) are erroring with the same error: read tcp {ip}:{port}->{ip}:{port}: read: connection reset by peer.
Locally everything works fine, also in docker and also in locally created k8s cluster. Also, curl from the same pod in AWS to the same URL works fine!

After further debugging it seems that problem arrises when http client is trying to do TLS handshake.

I've searched for this problem a LOT, but unfortunatelly no luck, nor docs. A bit shame for golang for this. I've also tried to call the same url with the C# code (native http client from .net) and it worked well within same environment.

Can some of you help me with that? Thanks in advance!

Edit: Some further info for debug:

  • Tried with alpine and grc.io/distroless/static-debian12 docker images
  • Using default Go http.Client{}
  • Any other tools work fine with the same URL (tried curl, telnet, c# client...)
teal yacht
short latch
#

Hey @teal yacht thanks for reply, I've tried that just now and unfortunately it did not help. Still same error happens.

I also tried to simplify for the sake of testing and only try to dial directly via tls.Dial(...), and the same error is rising, so I'm now 100% sure it originates from TLS-negotiation part of HTTP request.

c, err := tls.Dial("tcp", "<my-host>:443", nil)
// error -> read tcp {ip}:{port}->{ip}:{port}: read: connection reset by peer

Also I don't think it is directly problem with docker or docker container itself (because it works on local docker instance and even in locally created k8s environment) - only AWS is cursed :/

eager ruin
#

Could be a firewall thing

#

Are all your security groups set up correctly?

short latch
#

At first I thought so that this is a firewall concern, however I've found 2 interesting facts:

  1. The host of API I'm trying to call is whitelisted in firewall
  2. It is working by calling e.g. curl from the same pod, which is running go app (only go app is not working well) - if it would be firewall issue, other tools would be blocked as well ofc, no connection would be allowed

edit: Also I've written a very simple c# program that calls the same API and it works fine, without any problems (dunno why golang cant connect in the same way :D)

teal yacht
#

make a debug image, that has apt or curl and does it actually work with curl in the exact same container?

short latch
#

only go http client has problem

#

nothing else

#

sadly, today Im forced to rewrite it I cant fix it fast :/. And a big shame for go

teal yacht
#

you mentioned same pod not necessarily same container. maybe you have some proxy settings, as it should not be connection reset

short latch
#

sry, I meant same container (exec into)

#

UPDATE - I SOLVED THE ISSUE!

You probably won't believe what is the solution: Downgrading Go to 1.22 from 1.23 !!!!!!!

Its like WTF?????

I've crawled issues of Go's official github repo to find some clues and found these from @gopherbot (yes, really, they test bot):

They all share the error "connection reset by peer"

SO I saw that they are relativelly new issues (Go 1.23) so I downgraded to 1.22. After downgrade, everything works fine !!!!!!

Not really good first impression for me about Go to be honest 😄

teal yacht
#

one of the many reasons to never use a .0 or .1 in prod yet… curious how you reproduce this

short latch
#

I have NO idea, because it works on literally every other environment I tried with 😀

#

just that corporate AWS clusters that have some kind of security shenanigans implemented surely

#

but, if they can reproduce those bugs in tests (which apparently they could, due to those issues above), contributors are not totally clueless like me 😂

warm marten
#

tbf those issues raised all appear to be on the riscv64 architecture, so it could be limited to just there and unrelated to your issue