Emit Relations from EntityProvider

[pastel ingot]

In https://github.com/backstage/backstage/issues/1329 I see an example of using an EntityProcessor to emit relationships for multiple owners of a component.

I have the same use case but have an EntityProvider for creating catalog entities from an existing system which provides me with multiple owners for a component.

Single owner is a non-starter for us and I’m trying to figure out the correct way to do this. Adding an EntityProcessor seems like a wasteful way to do this as I’d be processing the same input data twice.

It doesn’t seem I can add relations to a ComponentEntity as it fails Policy check. Error: Malformed envelope, /relations/0 must have required property 'target' (using EntityRelation)

Is there a full Entity type I could emit?

[soft ingot]

@pastel ingot I think you're encountering an issue that's been discussed recently which is that Entity Providers do not allow emiting relationships. We had to work around this by storing relationships in spec , emmiting them in an processor and then deleting the property from the spec.

[pastel ingot]

I was about to ask if that’s the best workaround after digging through the source some. Sounds like it

Where did you end up storing the extra information in the component?

[soft ingot]

yeah, just in each entity. Pretty much anywhere in the entity is fine cause you're going to delete it after

[pastel ingot]

Lack of Multi ownership is basically a non starter for us to adopt backstage to replace our existing system.

[soft ingot]

Does this solve the problem for you or do you need more?

[pastel ingot]

i think it probably does -- going to go ahead and try it out.

[pastel ingot]

just following up to confirm I was able to get this working with an entity processor -- curious if there's any strong argument for pre vs post processing for this case. also debating just leaving the extra owners in the entity as it doesnt seem to cause any issues, just some duplication (as does the spec.owner) field

[soft ingot]

@runic basin ^^ any thoughts on this?

[dapper estuary]

I’m interested in this topic as well… I have a feeling that many orgs will care about this, particularly if they are oriented towards collective code ownership rather than strong (single team) ownership. Is there a good example of this or can you share how you went about this in more detail @pastel ingot @soft ingot ?

[runic basin]

Hey all, just saw this. Here's how HP did it: We rewrote our entity providers to include any data that would be used to emit relations by a catalog processor right in the entity itself, as part of the spec property. This is ugly, but super efficient, because it means we were able to remove all asynchronous operations from our catalog processors and let them do just a few basic things: Validate entities, emit any relations, and then remove the unnecessary data from spec. Running the catalog processors became very, very fast.

[pastel ingot]

^^^ this is what I just did except I used the metadata field. If there’s some consensus on why spec would be better I would change it. We are in a PoC phase atm so things are flexible. Considering replacing our in house service catalog with backstage.

Currently have about 20,000 entities in the catalog but that is a small subset of what we would have in a fully baked set up.

The way our current system is structured we allow multiple teams to own a service as well as individual users as supplemental owners.

We have an existing yaml based metadata file similar to the backstage one so I’m using an EntityProcessor to ingest that format instead and do some mapping

One aspect is that I’m loading it from a single index file another existing tool generates as we want to avoid crawling GHE

[runic basin]

In our case, it's because spec is malleable without a need to extend the entity spec schema. We didn't want to extend the metadata schema.

[pastel ingot]

Ah I didn’t realize that. Tbh I did not extend anything and it still worked

[runic basin]

@pastel ingot We're probably doing a lot more customization of entity schemas than most companies, so we're extending things left and right.

[amber spruce]

Regarding pre vs post - it's probably a good idea to do mutative things in pre (which happens before the full validation step), and then emitting things in post once all of the mutations - which in theory might have ended up pushing more things onto your owners field - are complete

And leaving things lingering in the spec is harmless in theory, but of course adds to the "noise" for a human reader of the resulting output entities, as well as adding to the API payload size

Full transparency:

We've actually toyed with the thought of forcibly deleting the spec (or at least relation-centric parts of it) as well. This would address the problem of consumer reading from the spec instead of from relations as they should.

And as an extension of those thoughts, this is one more factor that plays into the need for a clearer input vs output model for the entities. As in, the typescript types should probably not be the same on the input side (where stuff like metadata.uid and relations actually are forbidden) vs the output side (where we might want to eventually strip out things from the input)

[pastel ingot]

i'm experiencing the inverse of this -- i was able to freely put whatever i wanted in metadata but type check fails if i add to spec.

[pastel ingot]

(had to use as ComponentEntityV1alpha1['spec'])

[pastel ingot]

@runic basin curious how you're dealing with fields added to spec not being part of the type? starting to seem like making your own component type would be simpler.

[runic basin]

That's odd, because spec is just defined as a JSON object. You should be able to have pretty much anything you want in there.

[pastel ingot]

the problems i run into are having to throw a lot of as <some type> everywhere just to pass compiler typechecks

i suppose some of the type of information i'd like to store could be put in the labels section of the metadata

easier to reason about though if its more structured though

things like compliances and data sensitivity that apply to an entity can be in tags but then there's not a good way to filter them from other types of tags

feels weird to store them as flags in labels (compliance:sox = true) since labels are not a list.

[pastel ingot]

hey got it working using spec just fine :p PEBKAC

[buoyant panther]

Hi @amber spruce @heady ruin is there a way to load the additional metadata in the inspect entity view? We are trying to store extended metadata for a group like operational details etc which we want to load under the metadata and make them visible under the entity inspector view. Please suggest.

[amber spruce]

That view shows the full raw entity body. Should already be there

Oh you mean on an external store, NOT on the entity?