Erlang FFI options in httpc

[dusky escarp]

I am trying to work on this issue:

https://github.com/gleam-lang/httpc/issues/34#issuecomment-3548386644

However, I'm not entirely sure how the Gleam options are passed to the Erlang httpc library through the FFI. I have added:

type ErlOption {
  BodyFormat(BodyFormat)
  SocketOpts(List(SocketOpt))
  Ipv6HostWithBrackets(Bool)
}

To src/gleam/httpc.gleam, and then:

  let erl_options = [
    BodyFormat(Binary),
    SocketOpts([Ipfamily(Inet6fb4)]),
    Ipv6HostWithBrackets(True),
  ]

in pub fn dispatch_bits()

I've verified that my test is calling the correct code in my forked version of the library, but I'm still getting the runtime error.

Any clues? As far as I can tell, the PascalCase in gleam should be translated to ipv6_host_with_brackets in erlang automatically, but since this is unfamiliar territory I'm not entirely sure if the problem is that it isn't getting passed through, or if there is an issue in the underlying erlang library.

[sick shuttle]

For passing data to Erlang like that, you should make a custom type in Gleam that represents the options and translate it into the Erlang representation using Erlang code

[dusky escarp]

Well, I think that already exists? I'm using an existing library (written by @magic delta ) and just trying to add a single new option.

Here's the existing set of custom options:

https://github.com/gleam-lang/httpc/blob/main/src/gleam/httpc.gleam#L41-L44

And the current code that uses it:

https://github.com/gleam-lang/httpc/blob/main/src/gleam/httpc.gleam#L122

I can't see anything obvious in the custom FFI:

https://github.com/gleam-lang/httpc/blob/main/src/gleam_httpc_ffi.erl

So I have to imagine that this works already?

[magic delta]

Check the generated Erlang to see what it becomes

[dusky escarp]

I looked for a way to check that, but I couldn't find it? I expected a flag on the gleam build command but didn't see one

[jovial jetty]

you can read the erlang artifact files under build

I don't get the error mentioned in the ticket even without the option, instead I get nxdomain:

8> httpc:request(get, {"http://[2600:1406:bc00:53::b81e:94c8]", []}, [], []).
{error,{failed_connect,[{to_address,{"2600:1406:bc00:53::b81e:94c8",
                                     80}},
                        {inet,[inet],nxdomain}]}}

[dusky escarp]

ah, ok. I see:

-type erl_option() :: {body_format, body_format()} |
    {socket_opts, list(socket_opt())} |
    {ipv6_host_with_brackets, boolean()}.

And then:

    Erl_options = [{body_format, binary},
        {socket_opts, [{ipfamily, inet6fb4}]},
        {ipv6_host_with_brackets, true}],

Which looks about right? This is my first time really looking at Erlang code so forgive me if it isn't.

If you're doing this with straight erlang you may need another IPv6 config

checking...

I think you need socketopts inet6fb4

[jovial jetty]

red herring

[dusky escarp]

ok

sorry?

[jovial jetty]

Gleam's request.to_uri formats the URI as http://2600:1406:bc00:53::b81e:94c8/, explains why I didn't get the same results when calling straight from the Erlang shell

[dusky escarp]

ah

[jovial jetty]

I don't think that's valid

A valid host string must be a valid domain string, a valid IPv4-address string, or: U+005B ([), followed by a valid IPv6-address string, followed by U+005D (]).

seems to be a bug in gleam/uri to me

echo uri.parse("http://[123:456::789]:8000/com")
// Ok(Uri(Some("http"), None, Some("123:456::789"), Some(8000), "/com", None, None))

compare to original report in the ticket

// Ok(Uri(scheme: Some("http"), userinfo: None, host: Some("[123:456::789]"), port: Some(8000), path: "/com", query: None, fragment: None))

the host no longer contains the brackets. maybe an issue should be filed on the stdlib before adding any options to httpc.

[dusky escarp]

Where are you quoting from?

[jovial jetty]

the whatwg URL standard (https://url.spec.whatwg.org/#host-writing)

RFC3986 that the documentation states it follows also says

A host identified by an Internet Protocol literal address, version 6
[RFC3513] or later, is distinguished by enclosing the IP literal
within square brackets ("[" and "]"). This is the only place where
square bracket characters are allowed in the URI syntax.

problem comes from Erlang side already

1> uri_string:parse("http://[123:456::789]:8000/com").
#{port => 8000,scheme => "http",path => "/com",
  host => "123:456::789"}

they have some special code paths to detect ipv6 addresses when stringifying with uri_string:recompose, Gleam does not

they do classification of the host here upon stringifying, for reference: https://github.com/erlang/otp/blob/45ccff7a876854a5d650e0c3499b7466cabfa1d3/lib/stdlib/src/uri_string.erl#L2108

so I'd say this should be changed in the stdlib's uri module and see if that fixes httpc by itself already

(or the OTP implementation should be changed not to remove the brackets)

[dusky escarp]

I found this statement in the inets 5.8 release notes:

https://erlang.org/documentation/doc-5.10/lib/inets-5.9.3/doc/html/notes.html#id65645

"[httpc] Wrong Host header in IPv6 HTTP requests. When a URI with a IPv6 host is parsed, the brackets that encapsulates the address part is removed. This value is then supplied as the host header. This can cause problems with some servers. A workaround for this is to use headers_as_is and provide the host header with the requst call. To solve this a new option has been added, ipv6_host_with_brackets. This option specifies if the host value of the host header shall include the brackets or not. By default, it does not (as before)."

If I'm reading this correctly, then the URI needs the brackets, but the Host http header does not?

(Sorry for disappearing... a toddler needed lunch.)

[jovial jetty]

we probably could both have support for the option and fix gleam/uri's to_string

[dusky escarp]

Actually, looking at this:

https://datatracker.ietf.org/doc/html/rfc3986#section-3.2.2

It seems that the Host header should be in brackets anyway. So the correct thing to do is to change the gleam stdlib URI handler to always encapsulate IPv6 addresses in square brackets.

Which is to say that here:

https://github.com/gleam-lang/stdlib/blob/main/src/gleam/uri.gleam#L238

host: Some(uri_string)

Should be something like:

host: Some("[" <> uri_string <> "]")

and possibly the same here?

https://github.com/gleam-lang/stdlib/blob/main/src/gleam/uri.gleam#L244

There's probably a cleverer way to do it that doesn't involve re-appending brackets though...

[jovial jetty]

note that different parsing code is used for Erlang

[dusky escarp]

oh

[jovial jetty]

I think the first order of business would be to file an issue on the stdlib repo about how uri.to_string should not remove the brackets for IPv6 addresses. maybe link to this discussion. then the discussion can be continued there

notes about JS: http://[::1]/wibble works correctly: https://playground.gleam.run/#N4IgbgpgTgzglgewHYgFwEYA0IDGyAuES+aIcAtgA4JT4AEA5gDYQCG5A9AK5RwA6SCtVqMW7DlAgwuTfAIGUuAIzoAzJHXKs4SABQBKOsAF06EHAAsEdHnAB0lVrAi6+IC/nyVUHDgG1UDABdDgB3OCUlFjdDAB8APjpJaVk7LUpdWzt8BAB9GHxeJAZ9AQBfARAyoA

but "http://[2600:1406:bc00:53::b81e:94c8]" does not even parse: https://playground.gleam.run/#N4IgbgpgTgzglgewHYgFwEYA0IDGyAuES+aIcAtgA4JT4AEA5gDYQCG5A9AK5RwA6SCtVqMW7DlAgwuTfAIGUuAIzoAzJHXKs4SABQBKOsAF06EHAAsEdHnAB0lVrAi6+IC/nyVUHDgG0AJgA2AAYQjAAWEKDUJRww1ABWAGZUWIAOdAhUAE4InHSAXTdDAB8APjpJaVk7LUpdWzt8BAB9GHxeJAZ9AQBfARA+oA

"http://[123:456::789]:8000/com" works: https://playground.gleam.run/#N4IgbgpgTgzglgewHYgFwEYA0IDGyAuES+aIcAtgA4JT4AEA5gDYQCG5A9AK5RwA6SCtVqMW7DlAgwuTfAIGUuAIzoAzJHXKs4SABQBKOsAF06EHAAsEdHnAB0lVrAi6+IC/nyVUHDgG10ACYAZlQAFgBWADZUVAB2AA4ATgBdVASABiyOPHI3QwAfAD46SWlZOy1KXVs7fAQAfRh8XiQGfQEAXwEQTqA===

[dusky escarp]

https://github.com/gleam-lang/stdlib/issues/881

Feel free to add anything else