Azure DevOps Pipeline Build Failing | Backstage | Page 1

jagged lantern May 18, 2023, 3:16 PM

#

We have a deployment of backstage via ADO that is failing when we try to use environment variables in the app-config.yaml.

The specific error:
Loaded config from app-config.yaml

Error: Configuration does not match schema

Config must have required property 'clientToken' { missingProperty=clientToken } at /app/datadogRum
Config must have required property 'applicationId' { missingProperty=applicationId } at /app/datadogRum

app-config.yaml:

app:
title: Backstage Title
baseUrl: baseurl.com
datadogRum:
clientToken: ${DATA_DOG_RUM_CLIENT_TOKEN}
applicationId: ${DATA_DOG_RUM_APPLICATION_ID}

In our azure-pipelines.yaml in a pre-build step we download all the keyvault secrets and grab the other values from the ADO library. The issue as it appears is that those variable values are not available when the yarn build is executed in the build step. I have read some information on the --lax option but this seems more like an ADO issue and less backstage but It would be great if someone else has run into this with ADO and has some insights.

agile lagoon May 18, 2023, 5:25 PM

#

Hey! I have some experience with Azure DevOps, can you maybe provide your azure-pipelines.yml without sensitive data?

jagged lantern May 18, 2023, 6:08 PM

#

@agile lagoon sure thing!

#

@agile lagoon here it is

📎 azure-pipelines.yaml

agile lagoon May 18, 2023, 6:24 PM

#

Thanks for sharing ! Are you running the build in a job container? Maybe sharing the involve part of the /pipelines/pipeline-v3.0.yaml@dev template would help to understand better your flow.

jagged lantern May 18, 2023, 6:40 PM

#

@agile lagoon I am grabbing the template and redacting any info standby

jagged lantern May 18, 2023, 7:09 PM

#

@agile lagoon here is the template and yes it runs in a job container

📎 pipeline-v3.0.yaml

agile lagoon May 18, 2023, 8:38 PM

#

Thanks for taking the effort!

That looks quite complex flow! Seems like you have another layer of template for the build but no worry, let me just share my thoughts and hope that helps you.

Usually either you are running a build with a step script: yarn build or building directly the docker image you should not need the environment variables with the secret for that

Typically you inject the secrets while deploying (I think common way is injecting in app-config.production.yaml)

Based on some of the parts of your pipeline, is there a reason you need the DATADOG secret while building?

jagged lantern May 19, 2023, 1:05 PM

#

@agile lagoon You are correct we have a dockerfile that issues the yarn build. For some reason when we inject them in the deploy step the docker build fails giving us the error I posted. So we wanted to verify that they were available prior to the deploy because the error is pretty clear that there is no value present for the required field under app:. I am not the author of the ADO flow. I am the engineer responsible for the backstage application. I am trying to help out devOps folks puzzle through this. I will take a look and see if we can do the injection in the app-config-production.yaml

agile lagoon May 19, 2023, 3:40 PM

#

Reviewing your template I saw this runAsPreJob. Are you running different steps in different jobs? If so are you sharing does environment variable through all the jobs?

agile lagoon May 19, 2023, 4:11 PM

#

Not sure if this will help you but this is how we use the production one

#

we mix both files

#

and then we inject the values passing the environment variables from the deployment pipeline through terraform and then ends in our nomad file (which is our orquestrator)

#

This can be done directly writing down the content of the file and replacing the values or having a templated file with placeholders and pass the values to the template.

I guess you use kubernetes so not 100% sure how will look ike there

#

jagged lantern May 22, 2023, 6:32 PM

#

@agile lagoon I noticed that your env variables are lowercase that shouldn't matter?...

agile lagoon May 22, 2023, 6:46 PM

#

agile lagoon

If you mean in this picture (e.g.: ${azure_devops_token}) this is just the placeholder for the environment variable we inject through terraform to the nomad file, usually the format/case you use should not matter at all.

jagged lantern May 22, 2023, 9:26 PM

#

@agile lagoon that has been my experience also, it's just so bizarre the variable values print out in a bash echo on the container but are not available when yarn builds

agile lagoon May 22, 2023, 10:06 PM

#

Thats indeed a weird behaviour, sorry that I cannot help you more.

Good luck 😄

jagged lantern May 24, 2023, 5:19 PM

#

@agile lagoon in case you were wondering I was able to establish that the host was not getting any of the ENV VARS at build time, but they are available at deploy, what is preventing us at this point is backstage's required configuration fields at build time.

agile lagoon May 24, 2023, 6:29 PM

#

Hi @jagged lantern thanks for sharing your findings!

To be honest I still confuse about why your docker build process requires any values in the configuration, was working fine for me even removing some required values.

Maybe @inner meteor or @past wave can help you more than me 😅

jagged lantern May 24, 2023, 10:30 PM

#

@agile lagoon From my understanding of the backstage documentation required configuration items like baseUrl will cause an exception and the build to fail. The --lax option is not available for the cli build command. So our pipeline fails before it gets to the runtime step where the ENV vars would be injected. This may be my own ignorance, but if we set app-config.yaml with hardcoded values and use the app-config.production.yaml to include the env var substitutions at runtime the production yaml will have values which should supersede the app-config.yaml when the app runs. If this is standard practice then great I just assumed that the production yaml since it existed would just cause the same issue.

inner meteor May 25, 2023, 12:11 PM

#

I've been reading along (more lurking) and honestly I'm stumped. This is for sure one of those cases where I need to just try it out all myself and play around with things. Sorry, not a great answer

inner meteor May 25, 2023, 12:43 PM

#

But just for sake of being complete any chance you can share the Dockerfile used to build app/compass/catalog/backstage-backend, please?

jagged lantern May 25, 2023, 4:47 PM

#

@inner meteor the previous devs abandoned the generated Dockerfile and created a new Dockerfile in the root of the backstage project. They attempted to implement the multistage build outlined in the Backstage.io documentation. So diagnostically I added some RUN backstage-cli config:print stanzas to the file as well as RUN printenv to output the environment VARS and came to the conclusion that there are 0 variables available at build time in each step. Since we don't want to pass secrets, tokens, keys, etc. as a build argument to docker to get around the validation at build time I used the $${escape} syntax in the app-config.yaml and also included an app-config.production.yaml that contains the correct ${SUBSTITUTION} syntax and my working theory is that at runtime those values should be available in the production config and take precedent over the app-config.yaml definition.

And just as an FYI I added a "lax" build script item to the package.json that calls the cli config:print --lax command.

#

Here's the contents of the docker file

📎 message.txt

inner meteor May 26, 2023, 11:57 AM

#

Thanks @jagged lantern, I'll take a look over the next few days, have a big To Do list right now

jagged lantern May 26, 2023, 1:18 PM

#

@inner meteor thanks, it seems like the To Do list never gets smaller.

agile lagoon May 26, 2023, 7:42 PM

#

Hey @jagged lantern just wanted to tell you that I tried your image with our backstage instance.

I got the same error locally after add the datadog without passing values.

Can you maybe show us the very low level of the build template? (the exactly steps running, the way you are passing the environment variables to the job container y then to the docker build context)

If I;m not wrong this template

#

Anyways if you wanna a short fast solution you could just pass to the docker build thoes variables as arguments emptys (without any value) and declare them as ENVs in the image before run the build command.

I mean is just a ugly workaround that covers the real issue (My feeling is something regarding how you are passing the ENV to the job container and then to the docker context) but not sure and that could work In guess.

jagged lantern May 31, 2023, 11:12 PM

#

@inner meteor and @agile lagoon so going into this I did not know that backstage was being deployed on Azure App Services, so the reason the env variables were not available at run time is that configuration has to be done on the App Service and since these were key vault secrets they had to be integrated there. Also the dockerfile was multi stage and the build args weren't assigned in the other stages. Where I currently am is still stuck with the second config yaml not being honored at runtime, so I think I need to change the CMD to ENTRYPOINT.

agile lagoon Jun 1, 2023, 1:31 AM

#

Glad to hear you did some progress 😄

jagged lantern Jun 1, 2023, 12:34 PM

#

@agile lagoon I always wonder why I picked this profession at times like these...

jagged lantern Jun 12, 2023, 12:07 PM

#

@inner meteor and @agile lagoon I would love to get your help again. I did a version check on the backstage version we had deployed and it was 1.3.1 which is many moons from 1.14.0. I upgraded our development instance in Azure and had a few issues I had to sort out. The team I inherited this project from had not setup local development and worked in two codebases a vanilla create-app and the codebase actually used to deploy. I have created a docker compose implementation as well as stripped down the app-config.yaml down to it's very base. If you recall my last issue was our app-config.production.yaml not overriding the values in app-config.yaml. We are using a bundled deployment of backstage. Currently I specify the host and ports for the app and backend in the production config file but when I run the docker container the frontend only responds on the backend port. Which causes problems with auth/sso. here is my docker compose:

#

📎 docker-compose.yaml

agile lagoon Jun 15, 2023, 1:42 PM

#

Hey @jagged lantern sorry I'm quite bussy these days😅

sacred notch Sep 20, 2024, 12:28 PM

#

Hey guys!

#

sorry to join bit late

#

but wanted to check if I can get help in integrating Azure DevOps Pipeline UI in backstage

#

Error
401: AuthenticationError

Message
Cannot read properties of undefined (reading 'getBaseUrl')

Stack Trace
AuthenticationError: Cannot read properties of undefined (reading 'getBaseUrl')
at DefaultIdentityClient.getIdentity (/Users/shujaahmed/Documents/Code/onyx/onyx-idp/node_modules/@backstage/plugin-auth-node/src/identity/DefaultIdentityClient.ts:90:13)
at AuthCompat.authenticate (/Users/shujaahmed/Documents/Code/onyx/onyx-idp/node_modules/@backstage-community/plugin-azure-devops-backend/node_modules/@backstage/backend-common/src/compat/auth/createLegacyAuthAdapters.ts:89:24)
at HttpAuthCompat.credentials (/Users/shujaahmed/Documents/Code/onyx/onyx-idp/node_modules/@backstage-community/plugin-azure-devops-backend/node_modules/@backstage/backend-common/src/compat/auth/createLegacyAuthAdapters.ts:222:25)
at <anonymous> (/Users/shujaahmed/Documents/Code/onyx/onyx-idp/node_modules/@backstage-community/plugin-azure-devops-backend/src/service/router.ts:319:24)

Request
GET /builds/shujaahmed?entityRef=component%3Adefault%2Fnothing-new-project-new&repoName=nothing-new-project-new&top=25

#Azure DevOps Pipeline Build Failing