Hybrid Event Sourcing for Legacy System

[junior kraken]

I’m currently working on ye big ball of mud. Implementing new features in this 20+ year code base is near impossible. The company has a hard requirement to remove some database related dependencies (thought that never happened ), and a requirement to support an auditable history for specific areas of the system. Overall the system is a monitoring application standing up a realtime support center. Event sourcing is a no brainer and Marten seems like a place my team start.

We currently have tons of configuration stored in the database used to drive the monitoring for each system. Could we be successful using event sourcing to standup all monitoring while maintaining our CRUD system for configuration? I have a few concerns.

(1) Knowing we have existing data that would need to used to create aggregates with their current state and configuration does it make sense to apply an initial event to the new event sourced aggregate?SensorImportedEvent RoomImportedEvent etc.

(2) knowing that the existing UX for configuration and creation would still be used to modify the configuration (not state) does it make sense to apply modified events to the aggregates?SensorConfigurationEdited

(3) Do we apply this initial event and no longer utilize the existing data from the existing tables. This data is so ingraining into so many business tools and processes that I’m not sure I can ever break down those tables. Obviously once it’s inside the event history I can start adding events and projecting state. I’m concerned I would end with multiple sources of truth and inconsistent data if the old data is not nuked.

Thanks in advance for any insightful tips or information. The company is also open to hiring a consultant once we get a little further into this process to help guide us a bit.

[graceful glacier]

A few shots from the hip here:

Ad 1) It might give some redundancy where you have to handle both "SensorImportedEvent" and "SensorCreated" with near identical code with only slight differences. Sometimes a one time translation into a series of events can reduce this, but at other times it crucial to know what is important and what isn't and there might be something that the Imported one can magically do that you don't want in the new system etc... are you planning to threat the imported ones differently from the ones created in the system going forward?

This can turn quite complex is SensorImportedEvent is a "fat" event which in the new system will be multiple events then you start building very divergent codepaths in the application instead of handling this in "throw-away" event migration code..

Ad 2) If traceability of the configuration is not a direct concern you can just load it from a non event sourced storage when needed in the beginning. Sounds like bringing this into your aggregates has little business value.

Ad 3) "This data is so ingraining into so many business tools and processes that I’m not sure I can ever break down those tables. O" <-- Is it modified or only read?

If only read then a succesful approach could be to have first goal to capture what comes in there from your application and event source that and rebuild this data as a projection that these tools and processes can keep using as today.

One option here is to hook up CDC to the "old table" and transform into Events and play into event sourced system and rebuild the readmodel that everyone relies on from that and in that way replace, then you can start limit CDC and generate events directly slowly.

And when you get to hiring the consultant then I can recommend this company https://jasperfx.net/ 🙂

[junior kraken]

Ideally the imported devices and systems would be treated exactly as if it originated from events. However we’re in the process of documenting everything. I believe I could get away with locking the old data as read only we just need to really figure out everything that changes state. That’s easier said than done for some components. I’m liking your last comment as that’s sort of what I always imagined. I suppose if I focus this a bit backwards and ensure I have enough read model projections to actually keep the business moving i could just lift it out from under them with events and no one would be the wiser. I’m still left with the fact that there’s hardly enough time in the day to rebuild all our UI tools and I’ll probably need some garbage fat events to those big giant save buttons working.

[graceful glacier]

Unfortunately i can't help you with more time in the day 😄 But i know the feeling and in my experience getting the infrastructure up so you have a flow of "integration events" out of the old system that you can translate into a clean event sourced system is a very important step, that means that you can start building new functionality in your new set up and integrate it into the existing application on the UI level or replace existing functionality partly by replacing existing tables with event sourced projections. This is more time efficient than having to still do a lot of work in the old system because you are sort of in between systems.

I can see you are in the Event Modeling discord too, there is a short section on the event modelling website describing an approach similar to this:
https://eventmodeling.org/posts/what-is-event-modeling/#legacy-systems

Its of course a bit naive example compared to yours I would guess, but starting to draw up:

1. The events you want out of the old systems
2. The translation you need to do on those to get it into proper Events
3. How you can build any planned new roadmap functionality in this way "on the side"

Can be extremely beneficial. The process of drawing this usually surfaces a lot of details that you can iron out before you write a line of code. And it can be a fun ad-hoc team activity, reserve X time a day for moving the whiteboard model foreward.

It can maybe also help you identify a good first POC where you can get a 100% event sourced UI functionality embedded into the legacy UI so you can demonstrate some value of this approach to your business which might make it easier for you to get that extra time in the day if you can demonstrate that this could speed up delivery in the future.