#RAILWAY_REPLICA_ID and RAILWAY_REGION have empty values

9 messages · Page 1 of 1 (latest)

hollow river
#

Hey, I'm currently battling elevated DPM in Grafana, and pinpointed the reason to the fact that we have a few services deployed in multiple regions, each exporting metrics every 60s with the same OTEL_RESOURCE_ATTRIBUTES. We use standard OTEL libraries that are configured entirely through environment variables across 6 different services.

The challenge now is that we need to ensure metrics from multi-instance services are exported with unique instance IDs. So I went looking for that and found RAILWAY_REPLICA_ID and RAILWAY_REGION. I thought I could just set OTEL_SERVICE_ATTRIBUTES to e.g.

service.name=filecoin-app,service.namespace=app,deployment.environment=production,service.instance.id=${{RAILWAY_REPLICA_ID}},cloud.region=${{RAILWAY_REGION}}

but unfortunately those substitutions are empty. I assume that those two env vars are only available at run / build-time and not available for substitution within other env vars, and this is probably expected behavior?

But then, the question remains - how can I solve this problem? 🤔

#

RAILWAY_REPLICA_ID and RAILWAY_REGION have empty values

chrome folio
#

That is expected behavior, those variables are dynamically injected at runtime.

hollow river
#

any chance someone had this problem before and you're maybe aware of some workaround? not really sure what the best approach is... with our node services i could relatively easily mess with the OTEL sdk and configure these values at runtime, but with Deno OTEL as far as I'm aware, the only way to set these attributes is through the environment... 😵‍💫

chrome folio
#

Then set them through the environment at runtime?

hollow river
#

duh - trying that

hollow river
#

ended up simply setting the start command to append the runtime vars:

sh -c 'OTEL_RESOURCE_ATTRIBUTES="${OTEL_RESOURCE_ATTRIBUTES},service.instance.id=${RAILWAY_REPLICA_ID},cloud.region=${RAILWAY_REPLICA_REGION}" node --require ./build/telemetry.cjs build'

works perfectly 🎉 thanks @chrome folio, no idea why i didn't think of that 🙂

chrome folio
#

No problem!