formal cliff
Hi Guys,
I wanted to know how to read static configuration from app-config.yaml file into my catalog.ts file
Whole task:
read values of secrets from .env into app-config.yaml
then read values from app-config.yaml in the project i.e. for auth of api ..etc
formal cliff
@flint dust is there a doc or example for this?
flint dust
Maybe that works to enable the .env bit. We don't have that by default
The config already exist in the env that is given to each plugin, so using that is trivial as such
formal cliff
i am currently using this to get secrets from .env file in root to app-config.yaml
what i want to do is rather than using process.env.SECRET and getting it from .env file, i want to get it from my app-config.yaml as that will already have these values ,i.e. things like auth tokens etc
[am a beginner, so dont have much idea on the best of doing this]
flint dust
So in your app-config you can set values to ${SECRET}
formal cliff
yeah, currently in app-config i am setting using${} and getting these values from .env
flint dust
Thing is, the code that loads the config will then inject that env var into that value
formal cliff
but lets say i want to use its value in catalog.ts or an entityProvider file
formal cliff
Thing is, the code that loads the config will then inject that env var into that...
integrations:
bitbucketCloud:
- host: bitbucket.org
username: ${BITBUCKET_USERNAME}
appPassword: ${BITBUCKET_APP_PASSWORD}
lets take this. these values i.e. BITBUCKET_USERNAME and BITBUCKET_APP_PASSWORD are coming from .env
how do i use these in catalog.ts from the app-config.ts
flint dust
So you have env.config here https://github.com/backstage/backstage/blob/master/packages/backend/src/plugins/catalog.ts#L28
And the docs on config are actually very comprehensive
This is one of the pages about that
formal cliff
I'll check it out. thanks a lot 🙏
formal cliff
@flint dust reg. the passing env variables using app-config, I tried but am getting an error
const respositoryEntityProvider = new BitbucketIncrementalEntityProvider(env.config);
I am passing the env.config to the entityProvider and am trying to access the values using
const bitbucketConfig = this.config.getOptionalConfig('integrations.bitbucketCloud');
const bitbucketUsername = bitbucketConfig?.getOptionalString('username') ?? '';
const bitbucketAppPassword = bitbucketConfig?.getOptionalString('appPassword') ?? '';
error:
const bitbucketConfig = this.config.getOptionalConfig('integrations.bitbucketCloud');
^
TypeError: Cannot read properties of undefined (reading 'getOptionalConfig')
is this not the way to fetch values from app-config.yaml using config?
flint dust
that just means you didn't pass in the config somehow
this.config ended up being undefined, instead of the actual config, for some reason - you'll have to look into why that is
some bug in your code somewhere
formal cliff
@flint dust if i console.log my env in /plugins/catalog.ts, i get
Debug : ObservableConfigProxy {
parent: undefined,
parentKey: undefined,
config: ConfigReader {
data: {
app: [Object],
organization: [Object],
backend: [Object],
integrations: [Object],
proxy: [Object],
techdocs: [Object],
auth: [Object],
catalog: [Object]
},
context: 'app-config.local.yaml',
fallback: ConfigReader {
data: [Object],
context: 'app-config.yaml',
fallback: undefined,
prefix: '',
notifiedFilteredKeys: Set(0) {},
filteredKeys: undefined
},
prefix: '',
notifiedFilteredKeys: Set(0) {},
filteredKeys: undefined
},
subscribers: [ [Function (anonymous)] ]
}
then i pass this into the entityProvider like
const respositoryEntityProvider = new BitbucketIncrementalEntityProvider(env.config);
then use it to get configs like
const bitbucketConfig = this.config.getOptionalConfig('integrations.bitbucketCloud');
const bitbucketUsername = bitbucketConfig?.getOptionalString('username') ?? '';
const bitbucketAppPassword = bitbucketConfig?.getOptionalString('appPassword') ?? '';
is this not the method to do this? also how to verify that the values from my .env file are going into app-config.yaml
formal cliff
btw this is not for a separate plugin, its for a entityProvider
path of catalog.ts /packages/backend/plugins
path of entityProvider /packages/backend/catalog/entity provider file
formal cliff
@flint dust How to allow User entities to be owner of Component entities. by default, backstage allows only group
flint dust
group is only the default if you don't specify a kind
you can say owner: user:DevCooper
or owner: user:othernamespace/someusername
ultimately, it's a string-form entity ref where you can sometimes (depending on the context) leave out some of the parts https://backstage.io/docs/features/software-catalog/references
formal cliff
got it, thanks. will try this
also had question reg. system property? how and where do companies maintain this?
i.e. if i ingest from lets say bitbucket or github, where do companies usually store the mapping that this repository or service is being used by these services?
asking as i am ingesting data using entityProviders rather than keeping catalog_info.yaml files in individual repositories
flint dust
I don't think what you're doing is necessarily the most common way ... it's gonna be up to you then, to solve
Not sure about the wording "this repository or service is being used by these services"
And not sure how you wanted to be working with systems either - everybody has different ideas about how systems are constructed and arranged
formal cliff
so i am ingesting data by making bitbucket api calls -> get data from workspace and ingest by mapping into the entity schema
in the entity schema, we have a property system
flint dust
Most make a system for groups of apis and components that belong together in a bounded context
And that's a human construct, not particularly automatable
So they'd probably fill in a kind: System entity together with their other entity data in their repo, and make relations to it by hand as they see fit
formal cliff
ok, so they maintain a mapping of system {apis, components} somewhere and ingest from that
flint dust
No what I'm saying is more like
yaml files are the standard flow
at spotify it's what we've always done, for like a decade now
far before the open sourcing of backstage
formal cliff
so each repository has a catalog-info.yaml and individual owners are responsible for maintaining the data for the entity?
flint dust
and as we scaffold things using the software templates (scaffolder), that results in automatic yaml files in those new repos that get automatically registered in the catalog without extra user interaction
yes
it's just like all of the other metadata files in your repo
formal cliff
as initially i was doing a POC, I thought of writing entityProvider to ingest data
flint dust
we also for example typically have a kubernetes file with the objects specific to the deployment, for services ... it's "just another file" and we want to keep things close to the source code, just like with the philosophy with the techdocs plugin
formal cliff
but maintaining catalog-info.yaml seems more scalable
flint dust
yeah the thing with the catalog-info files is ... how shall i put this
the catalog is a bit special
it is not meant necessarily to catalog "the universe"
it's heavily focused on human concepts
formal cliff
but then how does the platform handle role base access?
as in anyone can make changes in the repository and it will be reflected in the catalog
flint dust
curated data, metadata that you really care about and that often is of such a nature that it doesn't have a good place anywhere else
like the owner field - that can be really subtle and not even perfectly match what the repo settings might make it look like - so you set it by hand, to the perfect value
as you note, this does place some restrictions on the sources used - and we're leaning into that, rather than shying away from it
for example
as we source files out of git, those files are placed near the source code that they represent
so
if you are worried about someone being able to change the owner field in that file, you might consider being way more afraid of someone introducing a keylogger in the source itself
ie we leverage the fact that these source systems already care about access control
and have it implemented, probably really well
git also has change tracking, so you can revert changes
and traceability
all of those concepts, the catalog would have to implement internally itself, if we weren't able to lean into git
formal cliff
ok got it. Also how are cloud based resources like s3 buckets, ecs clusters ingested?
flint dust
you can regard the catalog almost like a cache of external truths
you should be able to ideally completely wipe the catalog db, and then it just reconstructs itself from sources again
(in an ideal world; this may not always be 100% true depending on your providers)
resources are interesting because those start to touch the edges of what the catalog does best - people are tempted to mass-ingest every s3 bucket in the universe and that can turn out to be prretty ugly
formal cliff
you should be able to ideally completely wipe the catalog db, and then it just r...
yeah, I usually do that after making changes to the entityProvider i.e. remove all the preexisting entities from db.
flint dust
like, every single temporary piece of crap, making like search difficult because it gets overwhelmed with noise
so
formal cliff
resources are interesting because those start to touch the edges of what the cat...
but to represent a service fully , we would need the cloud resources used by it too right?
flint dust
it may actually turn out to be a good idea to treat things like pets, not cattle - maybe you actually declare the resource you are using, in your catalog-info file, along with careful comments and relations and stuff
that's at least one way of doing it
formal cliff
ok, so we should declare the cloud services in the catalog-info files too i.e. only resources used by that system
flint dust
maybe you have a CRD that declares the resource, in your source repo? could it make sense to make a provider or processor that looks for those files and creates entities based on those?
there's different ways of doing these things and they may all have benefits and drawbacks
some put more pressure on individual teams to maintain yaml, some put more pressure on infra teams to try to automate things
the power / flexibility of backstage, blessing and curse 🙂
in all honesty we do batch ingest a fair amount of resources too, and data pipelines / datasets are one of the large ones that are being pulled in in ever larger numbers using automation, and it can be a bit frustrating because inevitably the metadata quality levels tend to get diluted pretty quickly and the size of the catalog grows very quickly then
so it's a balancing act
the team that owns the catalog instance needs to ask themselves "do we really WANT this in the catalog, especially in its current state, or is there some other way that we could better serve end users?"
sometimes the answer is to actually leave things out of there, and serving them in a dedicated fashion, from a dedicated separate system
like our deployment and build related information - that all gets pulled in on demand from plugin frontend views, instead of trying to shoehorn it into the catalog itself
and that's perfectly fine
probably superior, even
the catalog excels at slow moving, human centric, pet-style things
can do other things too, but excels at that
then you hook rich ecosystems of plugins off of those
with the mindset of "oh my engineer customers only need to specify this one field with a key into other systems, and a whole world of functionality becomes available to them"
we for example have a kind of application id concept internally
such that all workloads are labeled with that, and builds are tagged with it, etc
so setting that one application id in your yaml file gives you access to info out of a ton of systems
formal cliff
got it, thanks a lot for these insights. They will really help in the actual implementation (currently doing a POC). Are there case studies of how companies have implemented catalog in IDP (other than the backstage documentation)?
flint dust
you might want to ask that as a separate question in #support or so to see what people's experiences are
formal cliff
yeah, will do that. btw thanks, these insights were helpful 🙂