#Best way to perform "Transactions" / Rollbacks on a storage bucket from the server?

1 messages Β· Page 1 of 1 (latest)

left sparrow
#

Hi everyone, I have a system where a user has an in browser code editor. When they autosave / delete files / update content etc, I am updating the metadata (file_name, content_hash) etc in the DB, but the file contents themselves are updated in a storage bucket (e.g. S3, Google Cloud Storage, Local Filesystem).

In spring with @Transactional, the transaction rolls back if something goes wrong. But As far as i'm aware there is not a similar thing with storage buckets. I have so far been collecting the ids/paths of the uploaded stuff and in case it goes wrong i manually try delete / reverse the changes, but then theres the issue that maybe the delete fails too, etc.

I was wondering what the best way to do something like this is?

If it's at all helpful, i've been doing something like this so far (in Kotlin):

override fun uploadList(req: StoragePutRequestList): UploadedPaths {
    val uploaded = mutableListOf<String>()
    try {
        req.requests.forEach { req ->
            uploadData(bucketName, req)
            uploaded.add(req.path)
        }
    } catch (exception: Exception) {
        rollbackAdditions(bucketName, uploaded)
        throw exception
    }

    return UploadedPaths(uploaded)
}

private fun uploadData (bucketName: String, request: StoragePutRequest) {
    val blobId = BlobId.of(bucketName, request.path)
    val blobInfo = BlobInfo.newBuilder(blobId)
        .setContentType("text/plain")
        .build()
    storage.create(blobInfo, request.content.toByteArray(Charsets.UTF_8))
}

private fun rollbackAdditions (bucket: String, uploaded: List<String>) {
    uploaded.forEach { path -> storage.delete(bucket, path) }
}
floral bisonBOT
#

<@&1008423204219531294> please have a look, thanks.

echo topaz
#

do you send only the diff ?

#

You could just make a list of all diff, then apply or remove them

floral bisonBOT
#

@left sparrow

Your question has been closed due to inactivity.

If it was not resolved yet, feel free to just post a message below
to reopen it, or create a new thread.

Note that usually the reason for nobody calling back is that your
question may have been not well asked and hence no one felt confident
enough answering.

When you reopen the thread, try to use your time to improve the quality
of the question by elaborating, providing details, context, all relevant code
snippets, any errors you are getting, concrete examples and perhaps also some
screenshots. Share your attempt, explain the expected results and compare
them to the current results.

Also try to make the information easily accessible by sharing code
or assignment descriptions directly on Discord, not behind a link or
PDF-file; provide some guidance for long code snippets and ensure
the code is well formatted and has syntax highlighting. Kindly read through
https://stackoverflow.com/help/how-to-ask for more.

With enough info, someone knows the answer for sure πŸ‘

sullen sphinx
#

Kind of a non-java way to approach it, but if its solely text content (just based on your blobInfo type) why not just rely on git to manage this for you? I think you would get all the features you want with built in protections for conflicts. You could use branches to handle retries/failures etc. I suppose you could use JGit as a Java layer πŸ™‚

warm barn
#

Or doing periodic prunes of unreferenced static files

#

Or even just accepting the cost of the extra files

floral bisonBOT
#

@left sparrow

Your question has been closed due to inactivity.

If it was not resolved yet, feel free to just post a message below
to reopen it, or create a new thread.

Note that usually the reason for nobody calling back is that your
question may have been not well asked and hence no one felt confident
enough answering.

When you reopen the thread, try to use your time to improve the quality
of the question by elaborating, providing details, context, all relevant code
snippets, any errors you are getting, concrete examples and perhaps also some
screenshots. Share your attempt, explain the expected results and compare
them to the current results.

Also try to make the information easily accessible by sharing code
or assignment descriptions directly on Discord, not behind a link or
PDF-file; provide some guidance for long code snippets and ensure
the code is well formatted and has syntax highlighting. Kindly read through
https://stackoverflow.com/help/how-to-ask for more.

With enough info, someone knows the answer for sure πŸ‘

left sparrow
#

Its not really a full blown text editor though, just a basic place to make files and run code in a playground for beginners.

echo topaz
#

Maybe just use the browser storage.

#

Do you really need to keep it in your server?

left sparrow
#

local storage makes sense though if they clear their browser then it'd be gone right

echo topaz
#

Well if they come back to the same browser it is persistent

#

Make a export import feature

#

It will save you so much space and pain in the long term.

left sparrow
#

Thank you for the suggestion though, its an interesting approach with the diff / local storage

echo topaz
#

Well then only sending change would be the best

#

Diff / server would pretty much work the same

#

You just sync the local storage with the server

#

:D

left sparrow
echo topaz
#

I meant it as the command diff

#

There is surely library/package that offer the same use

left sparrow
#

Currently looks like

private fun hasFileChanged (incomingFile: ProjectFileSnapshot, existingFile: ProjectFile) : Boolean {
    if (incomingFile.path != existingFile.filePath) return true
    val incomingHash = sha256(incomingFile.content)
    return incomingHash != existingFile.contentHash
}
floral bisonBOT
left sparrow
#

Then it collects it into arrays of toAdd, toDelete, toUpdate, and applies changes on only those (so not the whole project, just changed files)

left sparrow
echo topaz
#

Java-diff-utils

#

And jsdiff on js

#

It just work so well that people keep reusing it

#

Another thing you can do is cheat. And only have the complete file in http cache and browser cache

#

That way your server will not have to build it every time but only on change or when the cache is empty

left sparrow
echo topaz
#

Yes. The browser will ask the server to sync, then get the latest one.

#

Don't touch caching until you need it, it was just something I use that can be nice to know

warm barn
#

Don't give an AI to a beginner dude. Keep them the fuck away from it

#

Shake you like a baby

#

In the small chance this is a corporate product I just need to shame you

echo topaz
#

You got eagle eyes ethan

left sparrow
#

but i dont expect it to be commercial or used. More just to send it to an existing company to showcase skills

left sparrow
echo topaz
#

You can just use jsonl

#

And put it in a column in postgres

#

It should be pretty fast to implement and low effort

warm barn