Encoding binary data in url

[jagged marlin]

I have binary hash (sha1) I need to use in get requests, I can't use base64 because thats how the protocol works instead it expects to escape non ascii characters (other than A-Z 0-9 etc) by % and write the hex value.

I can convert this to String and pass to URLEncode because this is not valid utf8 so the data would be lost when converting to string.

Is there such function or do I have to write it myself? I looked around at spring boot but could find it too but these seems rather common thing to do?

[proven silo]

<@&1004656351647117403> please have a look, thanks.

Here is an AI assisted attempt to answer your question 🤖. Maybe it helps! In any case, a human is on the way 👍. To continue talking to the AI, you can use </chatgpt:1108714622413963314>.

TJ-Bot

I have binary hash (sha1) I need to use in get requests, I can't use base64 because thats how the protocol works instead it expects to escape non ascii characters (other than A-Z 0-9 etc) by % and write the hex value. I can convert this to String and pass

an a-zA-z0-9) using %. How can I encode the binary data in a URL-friendly format using Java?

You can use the java.net.URLEncoder class to encode binary data in a URL-friendly format. Here's an example of how you can do this:

import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;

public class UrlEncoderExample {
    public static void main(String[] args) throws Exception {
        String binaryData = "your binary data here";
        
        byte[] bytes = binaryData.getBytes(StandardCharsets.UTF_8);
        String encodedData = URLEncoder.encode(new String(bytes), StandardCharsets.UTF_8.toString());
        
        System.out.println("Encoded data: " + encodedData);
    }
}

In this example, we first convert the binary data to a byte array using UTF-8 encoding. Then, we use the URLEncoder.encode() method to encode the byte array as a string with non-ASCII characters escaped using '%'. Finally, we print out the encoded data.

Remember to replace "your binary data here" with your actual binary hash (sha1) before running the code. This should help you encode your binary data in a URL-friendly format for use in GET requests.

- AI generated response

[jagged marlin]

stupid bot stupid llm

[wispy sierra]

I can't use base64 because thats how the protocol works
I'm sorry, I don't understand why you can't use base64. Because of what protocol? Why does it prohibit using base64?

it expects to escape non ascii characters (other than A-Z 0-9 etc) by % and write the hex value
Okay, that seems like the fix to what you're trying to do, no?

I can convert this to String and pass to URLEncode because this is not valid utf8
You "can convert" because it's "not valid"? That doesn't sound right.

[jagged marlin]

can't

can't convert (typing mistake)

I'm sorry, I don't understand why you can't use base64. Because of what protocol? Why does it prohibit using base64?

the protocol expects it like that I did not invent this

whatever im gonna do it by hand

[wispy sierra]

You mean the convention that you need to encode non-ascii characters with %<hexcode>? That's really not a big deal

[jagged marlin]

no big deal, but can't find this in stdlib

[wispy sierra]

Yeah, such a thing doesn't exist pre-built

[jagged marlin]

ok got it 👍

[wispy sierra]

@jagged marlin I'm very proud of this solution 🐒 :

import java.util.function.Predicate;
import java.util.stream.Collectors;
import java.util.stream.Stream;

public class Main {
    static String encodeHash(final byte[] bytes, final Predicate<Character> excludeFromConversion) {
        return Stream.of(bytes)
                .<Character>mapMulti((bytes1, consumer) -> {
                    for (byte b : bytes1)
                        consumer.accept((char) Byte.toUnsignedInt(b));
                })
                .map(c -> {
                    if (excludeFromConversion.test(c))
                        return String.valueOf(c);

                    return String.format("%%%02X", (int) c);
                })
                .collect(Collectors.joining());
    }

    public static void main(String[] args) {
        final byte[] hash = {
                0x12, 0x34, 0x56, 0x78, (byte) 0x9a, (byte) 0xbc, (byte) 0xde, (byte) 0xf1,
                0x23, 0x45, 0x67, (byte) 0x89, (byte) 0xab, (byte) 0xcd, (byte) 0xef, (byte) 0x12,
                (byte) 0x34, (byte) 0x56, (byte) 0x78, (byte) 0x9a
        };

        System.out.println("encodeHash(hash, b -> Character.isLetterOrDigit(b) || b == '.' || b == '-' || b == '_' || b == '~') = "
                + encodeHash(hash, b -> Character.isLetterOrDigit(b) || b == '.' || b == '-' || b == '_' || b == '~'));
        System.out.println("encodeHash(hash, b -> String.valueOf(b).matches(\"[0-9a-zA-Z.\\\\-_~]\")) = "
                + encodeHash(hash, b -> String.valueOf(b).matches("[0-9a-zA-Z.\\-_~]")));
    }
}

Unfortunately it appears as if the first version, i.e. this one:

encodeHash(hash, b -> Character.isLetterOrDigit(b) || b == '.' || b == '-' || b == '_' || b == '~')
```has a bug where it excludes too many characters

Probably isLetter is wrong

Ah okay, yeah. I fixed it:

encodeHash(hash, b -> Character.isAlphabetic(b) || Character.isDigit(b) || b == '.' || b == '-' || b == '_' || b == '~')

[fervent root]

Some encode spaces with + instead of %20. Try to replace %20 with + after encoding.

Common with GET requests

It is super friggin annoying

If you are integrating two systems, it sounds like one has its own bastardized system for processing that common base64 string

[jagged marlin]

cool, though I don't need army swiss knife, sniper rifle is enough

  public String info_hash_url() {
    var url = new StringBuilder();
    byte[] hash = info_hash();
    for (int i = 0; i < 20; i++) {
      byte ch = hash[i];
      if (Character.isLetterOrDigit(ch) || "._~-".indexOf(ch) != -1) {
        url.append((char) ch);
        continue;
      }
      url.append(String.format("%%%02x", ch));
    }
    return url.toString();
  }

boring code easy to read

[empty hamlet]

https://github.com/gbevin/urlencoder