#Automatic Audit/archive/cold storage/persistence functionality?

14 messages · Page 1 of 1 (latest)

molten compass
#

Right now i mostly use convex where realtime state occurs. Most of my tables therefore contain fairly ephemeral records that after use I get rid of.

However, it'd be really helpful to be able to persist them in case i later need them for auditing purposes. For now I do this by pumping them to postgres and putting them in some storage tables. This is a good solution but I was wondering if convex provided any out of hte box functionality to do this. Right now the options I see to do this are

  1. Define backup tables in convex (ie if you have a table articles you could have another table archived_articles).
  2. Push to your own db layer (postgres, mysql, etc)

However, option 1 makes the schema start to look really polluted. If convex had some functionality for this that could auto handle it that would be amazing (seems like a good candidate for the pro plan!). Basically i'm imagining convex has implicit schemas for all tables that are considered backup tables, and maybe some other functionality like in your schema you could define a ttl with a persistOnDelete option. I dunno though, maybe that's too messy. Just food for thought

placid torrentBOT
#

Thanks for posting in #1088161997662724167.
Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets.

    - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.)
    - Use [search.convex.dev](https://search.convex.dev) to search Docs, Stack, and Discord all at once.
    - Additionally, you can post your questions in the Convex Community's #1228095053885476985 channel to receive a response from AI.
    - Avoid tagging staff unless specifically instructed.

    Thank you!
tired marlin
#

hi! so I think I'm trying to better grasp your use cases. in 99% of projects, convex is the persistent store, so I'm not sure why you would default to its state being ephemeral

#

ah, this is sort of an incremental adoption case where it's living next to an existing app in postgres?

#

and most of the data is there?

molten compass
#

Yes exactly sorry for not explaining more clearly....though that's actually quite interesting to hear cause I guess I thought of convex as more for temporary stateful data that needs to be realtime and the amount of data in the tables should be kept relatively small. Regardless,

  1. I'm worried about my tables getting too large and performance being impacted, though I have no experience with convex with larger tables.
  2. I have indexes on my tables, but without partitioning will they hold up well over time? My thinking was that generally I only keep performant realtime data in active tables, then move old data off to archived tables where it's less needed and performance isn't a concern
#

It's interesting to think of convex as a fulltime backend replacement, postgres included. I hadn't been mentally conceiving of it that way though so maybe that will change my approach...

#

I guess if convex supported automatic partitioning that could also be a solution. But maybe i'm doing premature optimization and convex will hold up fine for realtime data even when the tables get quite large?

#

I just read the convex bot. I have a pro account so just realized i should have opened there first. I'll do that so I can also give you a lot more context about my app

tired marlin
#

replied there

timid panther
#

Had a similar question. @molten compass would you be ok with posting the response/parts of it? I'm in a similar situation where for a particular document that will be heavily used and get big over time, pre-emptively adding a bigtable_completed and bigtable_archived document. Things would move to completed on close out, archived based on a scheduler.

molten compass
# timid panther Had a similar question. <@419567637266300928> would you be ok with posting the r...

Sure thing. The short of it is basically convex will handle it... very unlikely you will need to worry about performance problems. That is IF you remember to utilizie indexes! Of course it may still be useful to pipe stuff off to a data warehouse or somethign if you want to just be sure to retain the data, but from a performance perspective the advice was to not worry about it as indexes will be fully sufficient

#

And i guess furhtermore there's no built in support for an automated persistence/cold storage/archived (whatever you want to call it), because it's really not needed. Best just to leave your data be and convex should be fine

timid panther
#

Amazing thank you!