#Uploading 1000s of files instantly to a Storage Container

1 messages · Page 1 of 1 (latest)

hoary marsh
#

Hi there, I am trying to figure out the best way to go about tackling this problem. I have 10s of thousands of really small (~1kb) files which I would like to be uploaded to my Azure storage container instantly.

I will be regularly replacing these files with new ones every hour or so. Users will be able to access these files via my CDN.

The reason why I need the files to be replaced/uploaded instantly is so that users don't get a miss match of files. Say the upload process takes 5mins and a user fetches the files during this upload period, then there's the potential for them to have part of the files out of sync with the other part.

I understand that it's near impossible for all these files to be uploaded instantly which is fine for my solution. I was thinking of ways to try and upload the files to a different container and then replace the container with the proper one after everything has been uploaded, this would mean the user can only ever fetch data from one upload and not multiple. But I understand you can't do this.

Any alternative solutions are welcome here. Maybe there is a way we can freeze the container until all the files have been uploaded or some other method I haven't thought of.

Thanks in advance!

full sedge
#

Have you looked at blob versioning?
https://learn.microsoft.com/en-us/azure/storage/blobs/versioning-overview
Each upload of a new file would generate a new version id and leave the previous one in tact. Once you have uploaded all of your files, you would then generate a new "index" file that contains a list of all the files and the new current versions. When someone goes to download files, they would use that index and download those files and specific versions. Even if you are half way through uploading new files, it would still download the specific versions you specify.

Blob versioning - Azure Storage

Blob storage versioning automatically maintains previous versions of an object and identifies them with timestamps. You can restore a previous version of a blob to recover your data if it's erroneously modified or deleted.

astral tangle
#

You cannot achieve this in a single atomic operation like with a symlink on a linux filesystem. Closest you probably can get is upload the blobs with a version or hour prefix and when done change the cdn alias to point to the latest version. You also need to make sure that your cdn is not caching the files (what is the purpose of the cdn?).

hoary marsh
# astral tangle You cannot achieve this in a single atomic operation like with a symlink on a li...

The purpose of the cdn would be for users to quickly fetch the files. I thought it was better than just using a URL to the storage container? I've never heard of blob versioning but would I be able to only keep two versions of the data at once? I don't need to keep track of old data really? And then would I be able to update the cdn alias in code too? Ideally the user should be able to just call the same URL without knowing which version to use.

astral tangle
#

The CDN only helps with performance if you allow the CDN to cache. However, if you allow the CDN to cache you increase the chance of inconsistent files as some cached files may be from the old version and others from the new one. I don't think versioning will help with what you are trying to achieve.

full sedge
# hoary marsh The purpose of the cdn would be for users to quickly fetch the files. I thought ...

It will keep all previous versions until you purge the old ones. If you do use versioning, you would also want to setup a lifecycle policy to limit how many days worth of versions you retain. Using a link to a blob/file without the version specified will always retrieve the current version, so if you have a user doing a download while you are uploading, they could grab half of the previous version files, and half of the new version files. This is why you need to create an "index" file after you complete your upload that tells users which versions are part of the same set. https://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview#manage-previous-versions

That said though, a better option would be to compress your files into a single archive like a 7z or zip and then only upload that single file. The operation will be much faster than uploading 10's of thousands of files, and then you can also ensure that a user always retrieves a matching set. With thousands of small files, your TCP packet overhead for setting up the transfer of each individual file will actually take longer than the upload itself and consume more bandwidth. Downloading a 100kb zip will be instant. Downloading 100,000 1kb files will not be.

Optimize costs by automatically managing the data lifecycle - Azure...

Use Azure Blob Storage lifecycle management policies to create automated rules for moving data between hot, cool, cold, and archive tiers.

hoary marsh
hoary marsh
#

@full sedge @astral tangle don't suppose either of you know the upload limitations on a storage account? I'm running a function app to do the upload and it does the first few hundreds really fast and quick, then for whatever reason the upload logs just stop appearing, no clue why. I'm guessing it's due to a limit that I'm hitting or something?

Was going to try and publish the function app in azure to see if it plays nicely but was wondering what the limits are as I couldn't find any information on upload limits.

full sedge
hoary marsh
#

From what I'm reading it's based on the size not the number of requests. Which is why I said I couldn't find any information on that.

full sedge
#

There is both, but you're most likely running into API or request limits, not throughput limits. All of Azure's APIs have limits. As mentioned in the article, you can contact Azure Support to have your limit increased, but there is no guarantee that they will accommodate. Generally speaking, if you are being rate limited/throttled you will get a message from Azure in the response to your API calls that indicate such. Some APIs will return a Retry-After value in the response header to indicate how long you will be throttled for.

hoary marsh
#

Ok I'll take a look at it tomorrow thanks.

#

I'm honestly thinking about spinning up a file server to store the files instead. That way I could easily upload a zip of all the files on there and extract them into the correct folder. I could also do stuff like deleting the old folder and replacing it with the new one too, that way all the data will be available at the same time. It would solve a lot of my problems. I just have no idea if this is something that you can do or how you would go about doing it.

full sedge
#

Look at Logic Apps. You have the Storage Account trigger the worklow when a new zip file is uploaded and then extract to your needed folder and then delete the zip:

image.png
#

Logic Apps would likely be far cheaper than spinning up a server and dealing with the maintenance

echo meteor
#

I think you might get murdered on IO costs in blob/storage though. This feels kind of fraught. @full sedge for an app of this scale, I'd probably want to recreate that logic in a function, which is doable.

full sedge
#

I think logic app or function could be equally suited for this task depending on how comfortable they are with code. From a cost standpoint, that's going to ultimately come down to how many people per day are going to download all of the files. There will be a break even point where it'll be cheaper to run a VM and host the files on there along with a web server.

Another option could be to leverage a container app and just build a new docker image with the files and then swap the images. Once a user connects and starts to interact, they'll continue to interact with that version of the container and get the specific version of files that it's hosting. When a new user connects, they would get the new image/files. Once the old container doesn't have any active connections it could be killed off and the cycle repeated. This approach could also have better scaling options as well depending on their needs.

hoary marsh
#

I thought that you couldn't extract zip files in a blob?

#

Or that you had to extract it and upload all the files?

#

If I were to use that logic app action would it actually be somewhat quick?

#

I do like your idea of the container but I'm not experienced in what I'd need to do to set that up

#

I did also have a look into storing the files using durable entities which I could then fetch using a http trigger but I wasn't able to get the state to store.

hoary marsh
#

Just to let you know I tried to logic app action and it doesn't work if your archive has more than 100 files in or maybe it was folders but still it's not going to work for me.