🔒 Trying to implement simple arithmetic encoding, got stuck

[twin walrus]

I am not really good in compression algorithms. But I have an assignment where I have to benchmark existing librarys and my own implementation. I implemented a basic lz78 compressor and a basic arithmetic encoder and combined the two but the arithmetic encoder doesn't seem to be of any use as my probability table is almost as big as the original data, can anyone help who is a little more experienced in things like these?

[night zenith]

@twin walrus

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[night zenith]

Click here to see this code in our pastebin.

[royal island]

@twin walrus so what is your question really?

[twin walrus]

that's a good question

[twin walrus]

Can I compress the probability table to be smaller with some kind of an algorithm? Or what other approach should I do? Should I scrap the whole arithmetic thing?

[royal island]

you should explain what you're doing, we're python developers not mathematicians

i dont have the slightest idea of what you actually want to solve here

[twin walrus]

so first of all, I compress a text with lz78

Original size: 410000
Lz78 Encoded: 73784
Encoded data size in bytes: 11002
Probability tables size in bytes: 92156
Chunk size table size in bytes: 132
Compressed prob table: 49399
Original size: 410000
Decode successful? True

then I take the output of that and put it into my arithmetic compression which sounded cool when I started to implement but didn't really get me anywhere as its actually bigger then the lz78 data I put into it

I think the problem is with the way I implemented arithmetic coding, but can't really do any better

Arithmetic coding takes the data and a probability table of each character then turns the data into a single float (double) call it whatever

[royal island]

so is your problem with the encoding itself or the comparing/benchmarking

[twin walrus]

With the encoding

[royal island]

okay, is there an error?

[twin walrus]

No there is no error, I came to the python server hoping someone understands the concept and can look at my code and point me in a direction on what to change.

[twin walrus]

I will rephrase, my problem is that I need to use mpmath to get more accurate floats, and chunk my text into little bits, (mpmath 100 dps works for chunks of 50 chars) but if I chunk my probabilty table the table will be too big, if I use a fixed probabilty table for all ascii chars for example, my encoded data will be too big so If there is anyone on this server who understands the concept of arithmetic encoding and can look at my code and point me in a direction on what to change, it would be appreciated

[night zenith]

Python help channel closed for inactivity

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.