problem with encode neural network

[silent garden]

good morning

i have strange problem

im using djl library, im generating neural network, and then save it to file (only avaible method). Then im getting bytes from file, encode it to base64 and save to db. Saving works almost fine, i have data, but it a lot of chars are diferent.

Original: DJL@
test FLOAT32


decoded value from db DJL@���
�test�FLOAT32���
�

gpt says the problem may be caused by float32

function that create file and copy content:

  public static byte[] modelToBlob(Model model) throws IOException {
        Path tempDir = Files.createTempDirectory("temp-model");
        model.save(tempDir, "test");
        Path modelFile = tempDir.resolve(model.getName() + "-0000.params");

        return Files.readAllBytes(Paths.get(modelFile.toUri()));
    }

create network

    public void createEmptyNeuralNetwork() {
        Block block = new Mlp(6, 2, new int[]{128, 64});
        NDManager manager = NDManager.newBaseManager();
        NDArray x = manager.randomUniform(0, 1, new Shape(2, 6));
        block.initialize(manager, DataType.FLOAT32, x.getShape());
        try (Model model = Model.newInstance("test")) {
            model.setBlock(block);
            byte[] modelBytes = modelToBlob(model);
            String serializedModel = Base64.getEncoder().withoutPadding().encodeToString(modelBytes);
            NeuralNetwork nn = new NeuralNetwork();
            nn.setUuid(UUID.randomUUID());
            nn.setName("test");
            nn.setSerializedNeuralNetwork(serializedModel);
        }
    }

additional info: https://base64.guru/converter/encode/file - here encode and decode (from source file) working if im using just file

https://www.base64decode.org/ here i tried encode and decode value of file, but here i have a lot of differences

[dusk goblet]

<@&987246399047479336> please have a look, thanks.

Here is an AI assisted attempt to answer your question 🤖. Maybe it helps! In any case, a human is on the way 👍. To continue talking to the AI, you can use </chatgpt:1108714622413963314>.

One potential problem with encoding a neural network file to Base64 online and embedding it into a text document is the size limitation. Base64 encoding increases the file size by approximately 33% due to the encoding process. If the neural network file is already large, this increase in size can become a significant issue when embedding it into text documents.

Another problem is that embedding the encoded neural network file directly into HTML, JSON, or XML can make the document more complex and harder to read. This can be problematic when sharing or transferring the document between different systems or platforms.

Additionally, decoding from Base64 format or encoding into it with various advanced options may require specific libraries or tools that are not readily available in all programming languages or environments. This could limit the compatibility and ease of use for developers who want to work with these encoded files.

Lastly, if security is a concern, embedding sensitive neural network files directly into text documents may pose a risk. Base64 encoding does not provide encryption, so anyone with access to the document can easily decode and extract the original file.

To address these problems, alternative approaches such as hosting the neural network file separately and referencing it in the text document through URLs or using encryption techniques for secure transmission should be considered.

[silent garden]

(bump)

[dusk goblet]

@silent garden

Your question has been closed due to inactivity.

If it was not resolved yet, feel free to just post a message below
to reopen it, or create a new thread.

Note that usually the reason for nobody calling back is that your
question may have been not well asked and hence no one felt confident
enough answering.

When you reopen the thread, try to use your time to improve the quality
of the question by elaborating, providing details, context, all relevant code
snippets, any errors you are getting, concrete examples and perhaps also some
screenshots. Share your attempt, explain the expected results and compare
them to the current results.

Also try to make the information easily accessible by sharing code
or assignment descriptions directly on Discord, not behind a link or
PDF-file; provide some guidance for long code snippets and ensure
the code is well formatted and has syntax highlighting. Kindly read through
https://stackoverflow.com/help/how-to-ask for more.

With enough info, someone knows the answer for sure 👍

[silent garden]

.

I will check by copy file bytes to another file, maybe problem is with db

[south lodge]

is everything using the same encoding?

[silent garden]

For now i did only base64 encode and tried to decode it using online decoder

I copied encoded value directly from db

Oh, i forgot, when i wasnt know where is problem i tried to decode using base64 decoder in java and load model

[south lodge]

Sorry, as in UTF-8, UF-16, ISO-8859-1

[silent garden]

I tried everything from them, same settings from encode and decode