I work on a large Laravel project, and I have a healthcheck endpoint that does not call the database (confirmed when I enabled profiling), and I have another service that periodically calls the healthcheck endpoint to check that my main Laravel project is up, and then records the time it takes for the healthcheck endpoint to response.
Normally it takes about 0.02 seconds for the endpoint to respond.
But rarely it randomly takes >5 seconds to respond, even though it's confirmed that there's barely any traffic to the site.
I can confirm that it's not the connection that takes long to establish, but waiting for the response to complete that takes long, because my other PHP monitoring application shows a curl "operation timed out" error (I have a 5 second limit set).
My primary question is is it somewhat normal for requests to take long to respond rarely i.e. am I being too strict with the 5 second limit, and should probably be a lot more generous with the limit?
Maybe there are DNS issues with some middleware involving I/O e.g. the Sentry or ElasticSearch logger.