🔒 Wrong Gzip file decompression

[queen pewter]

I'm trying to design an algorithm that decompresses a file that can be in gzip or tar formats as a solution to this CTF challenge. The whole game is designed to be solved with Linux commands but I decided to go my way with Python, so the challenge solution was tested before. Back to the problem though, the starting file has a 1F 8B 08 file signature that I confirmed by reading it with a short script in Python REPL which means that I ought to use Python gzip to decompress it, however calling gzip.decompress(my_file_contents) returns an empty by string b''. I also tried to follow this Stack Overflow answer but it didn't work. What confuses me about gzip.open is that reading the contents shows a different string to the original one and the signature changes to something that's not listed as a file signature. What am I doing wrong? Does Python file decompression work in a specific way I should learn?

REPL

>>> import gzip
>>> with open("decompress/start", "rb") as r:
...     print(r.read())
...
b'\x1f\x8b\x08\x08\xdf\xcd\xebf\x02\x03data2.bin\x00\x01>\x02\xc1\xfdBZh91AY&SY\xca\x83\xb2\xc1\x00\x00\x17\x7f\xff\xdf\xf3\xf4\xa7\xfc\x9f\xfe\xfe\xf2\xf3\xcf\xfe\xf5\xff\
>>> with gzip.open("decompress/start", "rb") as r:
...     print(r.read())
...
b'BZh91AY&SY\xca\x83\xb2\xc1\x00\x00\x17\x7f\xff\xdf\xf3\xf4\xa7\xfc\x9f\xfe\xfe\xf2\xf3\xcf\xfe\xf5\xff\xff\xdd\xbf~[\xfe\xfa\xff\xdf\xbe\x97\xaao\xff\xf0\xde\xed\xf7\xb0\
>>>

I would've liked to give you guys a minimal reproducible code snippet but this is a file stored in the CTF server that needs a password, if I give you that password I would give you a spoiler.

[dense fjord]

@queen pewter

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[queen pewter]

This is the decompress function in my algorithm by the way:

def decompress(path: Path):
    try:
        with gzip.open(path, "rb") as r:
            with open(Path.cwd() / "decompressed" / random.choices(string.ascii_letters, k=6), "wb") as f:
                content = gzip.decompress(r.read())
                f.write(content)
    except gzip.BadGzipFile:
        with tarfile.open(path, "r") as r:
            r.extractall()

And I've just noticed that r.read() returns string not byte

That might be part of the problem

gzip.decompress docs says that I have to pass a byte type through it as a paramater.

[bleak idol]

What are you actually running? Not seeing where you decompress, is that after reading?

[queen pewter]

Hmm... the debugger tells me that r.read() does return a bytes type.

[queen pewter]

Hi! Thanks for taking time to read my question

Could you explain your question a bit for me please?

[bleak idol]

I'm questioning if you've already read the file without realising what that does

[queen pewter]

Sorry I still don't get it

[bleak idol]

Once you read() on the file, you're effectively at the end of the file. ie it's been read, so subsequent reads would be empty, without reopening the file or seeking to the start

[maiden drum]

Also, it looks like gzip.open returns decompressed data?

Or at least returns a file object that produces decompressed data

[queen pewter]

Yeah that's a bit confusing cause the resulting file signature doesn't match anything listed here

[maiden drum]

I think you might have mixed up gzip.open and open.

[queen pewter]

Alright guys I think we could check this Stack overflow answer together shall we?

As @maiden drum said I shouldn't use read() cause gzip.open already works with decompressed data of the file. That's cool

But I still wonder why the decompressed version doesn't have a compressed file signature

with gzip.open("decompress/start", "rb") as r:
     print(r.read())

output

b'BZh91AY&SY\xca\x83\xb2\xc1\x00\x00\x17\x7f\xff\xdf\xf3\xf4\xa7\xfc\x9f\xfe\xfe\xf2\xf3\xcf\xfe\xf5\xff\xff\xdd\xbf~[\xfe\xfa\xff\xdf\xbe\x97\xaao\xff\xf0\xde\xed\xf7\xb0\

[vague saddle]

u sure it's still compressed?

[queen pewter]

Oh hang on

[maiden drum]

How many times are you decompressing?

[queen pewter]

Should I decode the string? Why then? I don't understand

[queen pewter]

First time from the original

[maiden drum]

I'm not familiar with the compression algorithm, but are you sure the decompressed data will start with the file signature?

The file on the disk may, but the data may not.

[queen pewter]

Hmmm that's a good question

Interesting

So file signature might not be at the beginning of the file right?

Let me verify that with the REPL, I'd like to be sure

[maiden drum]

Oh actually, it looks like it does

[bleak idol]

It's bz2 yea

[queen pewter]

How did you find that out?

[bleak idol]

Well search auto completed when typing bzh9 for the full header you were getting

[queen pewter]

Alright! So there's a way. I have to check file signature properly, cool to know that it's possible to work that out with Python.

Python has a bz2 module for that compression format

[mild sinew]

if you can install modules where you are working with this you can do pip install python-magic and (or if it's already installed) then import magic and use that module to identify file types

[queen pewter]

Thanks for the suggestion! That could be a solution if my code was run locally but it has to be executed on the server and I don't have any permissions to do that. But yeah I've got to find out file signatures with re module or by catching exceptions that modules such as gzip or bzip can throw.

I think I've got enough hints to work from thanks to you guys 🤘

I'd like to try them on my own for now but I'll wait just a couple of minutes to let everyone add anything before I close the question.

[mild sinew]

and it's not already available on the server?
otherwise you could run the external file command, unless you only want to use pure python solutions

[queen pewter]

It is not on the server I'm afraid buy yeah I'm going for pure python as my first option. Had to use it to run an external command to get the first compressed file from a hexdump after trying a lot and failing

[mild sinew]

you should be able to do that as well in pure python if you really want to

[queen pewter]

Yeah, that was a cool challenge but I couldn't, spent lots on time on that and failed. I think that was beyond my knowledge and in spite of doing lots of research I still couldn't do it. That's something I want to try again in the future though 😉 when I have more experience with the language. For now I'd like to move on with this CTF level 🤘

Alright, thanks everybody again for taking time to read my question an share some knowledge here!

I'm gonna close it now

!close

[dense fjord]

Python help channel closed with !close

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.