#Disable Opentelemetry on K8s Pod

13 messages · Page 1 of 1 (latest)

kind girder
#

We're using Dagger via helm with Gitlab K8s runner to build container images. On successful builds, we have no issues but when the build errors, the job goes into an endless try to send metrics/traces which eventually dies when the Gitlab job timeout is reached.

16  : [13.8s] | WARNING [opentelemetry.exporter.otlp.proto.http.trace_exporter]: Transient error Service Unavailable encountered while exporting span batch, retrying in 2.10s.
16  : [15.9s] | WARNING [opentelemetry.exporter.otlp.proto.http.trace_exporter]: Transient error Service Unavailable encountered while exporting span batch, retrying in 3.96s.
16  : [17.2s] | WARNING [opentelemetry.exporter.otlp.proto.http._log_exporter]: Transient error Service Unavailable encountered while exporting logs batch, retrying in 0.91s.

I tried to set the following env vars via the Helm Chart and Gitlab CI but it doesn't get reflected in the runtime env vars of our custom build module.

DAGGER_NO_NAG=1
DO_NOT_TRACK=1
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=""
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=""
#

print(os.environ.items()) OTEL_* values during the job:

16  : [2.5s] | ItemsView(environ({
'OTEL_EXPORTER_OTLP_ENDPOINT': 'http://127.0.0.1:38697',
'OTEL_EXPORTER_OTLP_INSECURE': 'true',
'OTEL_EXPORTER_OTLP_LOGS_ENDPOINT': 'http://127.0.0.1:38697/v1/logs',
'OTEL_EXPORTER_OTLP_LOGS_INSECURE': 'true',
'OTEL_EXPORTER_OTLP_LOGS_PROTOCOL': 'http/protobuf',
'OTEL_EXPORTER_OTLP_METRICS_ENDPOINT': 'http://127.0.0.1:38697/v1/metrics',
'OTEL_EXPORTER_OTLP_METRICS_INSECURE': 'true',
'OTEL_EXPORTER_OTLP_METRICS_PROTOCOL': 'http/protobuf',
'OTEL_EXPORTER_OTLP_PROTOCOL': 'http/protobuf',
'OTEL_EXPORTER_OTLP_TRACES_ENDPOINT': 'http://127.0.0.1:38697/v1/traces',
'OTEL_EXPORTER_OTLP_TRACES_INSECURE': 'true'
'OTEL_EXPORTER_OTLP_TRACES_LIVE': '1',
'OTEL_EXPORTER_OTLP_TRACES_PROTOCOL': 'http/protobuf',
'OTEL_LOGS_EXPORTER': 'otlp',
'OTEL_METRICS_EXPORTER': 'otlp',
'OTEL_PYTHON_LOG_CORRELATION': 'true',
'OTEL_PYTHON_LOG_FORMAT': '%(levelname)s [%(name)s]: %(message)s',
'OTEL_PYTHON_LOG_LEVEL': 'warning',
'OTEL_PYTHON_LOGGER_PROVIDER': 'sdk_logger_provider',
'OTEL_PYTHON_TRACER_PROVIDER': 'sdk_tracer_provider',
'OTEL_SERVICE_NAME': 'dagger-python-sdk',
'OTEL_TRACES_EXPORTER': 'otlp',
}))
modern flax
kind girder
#

The error was related to NO_PROXY setting needed for 127.0.0.1

modern flax
#

hmm interesting.. seems strange that connections to 127.0.0.1 might go through a proxy at all 🤔

sand plover
#

If you set http_proxy/https_proxy everything that's not on no_proxy goes through proxy by design.

modern flax
sand plover
#

"typically"... So yeah, s/everything/some things/g.

modern flax
#

looks like in most cases localhost doesn't use the proxy

#

which is what I assumed initially

sand plover
#

Depends on the implementation...

#

Same case with uppercase lowercase. Proxy is a mess. :/

modern flax