Been doing some scraping with Go, but after an hour or two of running it I get this. Then it drops dead like a sack of potatoes. I restart it and it works again, I've been told this is due to my OS not permitting certain amount of idle connections or something, but I've raised that number as high as a thousand didn't make a difference.
#dial tcp 164.17.55.26:443: connect: cannot assign requested address
obtuse goblet
grand veldt
are you sure you're closing connections after you're done with them?
if you think you are, i'd run for a bit and check your established connections with, e.g., netstat to see what's taking up ports from OS' perspective
obtuse goblet
are you sure you're closing connections after you're done with them?
well im not sure what that means to be honest
i just use the defer resp.Body.Close function
which i dont think is that
grand veldt
that is closing, yes, but are you sure the function is actually exiting? defer waits until the function completes to execute, so if you're e.g. running an infinite loop, the close will never actually happen
obtuse goblet
yeah ive got a for loop
but it does exit with a return
so unless the for loop somehow screws it up.
i wonder
if i do continue
and the for loop resets
does that negate the defer?
cause then if it does the connection wouldnt be closed
grand veldt
it doesn't "negate" it, but it still doesn't occur until the return
obtuse goblet
body, err := io.ReadAll(resp.Body)
if err != nil {
log.Println("Retrying, Error occured: ", err)
continue
}```so i guess this would just leave an open connection
grand veldt
it would remain open until the current function returns
obtuse goblet
well ive plenty of these statements
and i can have several errors thats why i have a for loop so it retries until it succeeds. but with many requests at once i guess this can create the issue.
am i able to just call resp.Body.Close() directly in that before continue
grand veldt
sure
and yeah, it's good practice to be wary when you start using defer inside a loop
it isn't always disastrous, but you should really think about what's happening
obtuse goblet
what happens to the defer then, or should i just make it so when it returns it just calls resp.Body.Close()
without it?
grand veldt
good point, yeah you should only end up calling it once
obtuse goblet
well i'd have resp.Body.Close() before every continue, and then when i return finally when it succeeds
grand veldt
i would probably ditch the defers here, personally
obtuse goblet
yeah thats what i was gonna ask cause it seems unnecessary
well thats interesting. gonna test see if this resolves it.
grand veldt
while you’re waiting to get to the point where it had issues in the past i’d still monitor your socket usage from the OS
could tip you off to the problem still being present without having to wait all that time
elder lynx
you ran out of ephemeral ports. there aren’t that many in particular on macos