produce PackeByteArray from Array[Variant]?

[haughty grove]

Say I have an array with values of any type in it (well, a limited subset. No strings. Nor any nesting with other arrays or dicts), and yes, they can be mixed ([42, 3.14]).

How would I create a PackedByteArray from this array? Specifically such that it follows the same rules as the Packed*Array ones (like how PackedInt32Array gives each int 4 bytes, and has no delimiting bytes), resulting in similar bytes?

I have tried using var_to_bytes by mapping var_to_bytes(v).slice(4) over the array. But the issue is that it uses 64 bits for some types (i.e floats) - likely because I got a 64-bit system.
But I need it to be fooled into thinking I have a 32-bit system during the conversion, as the array I produce need to have 4 bytes for floats (and 8 for doubles). This is because I will be using it to create a buffer using RenderingDevice, to send the data to shaders (where float is 4 bytes, and doubles are 8).

I considered making my own serializing method/class, but I absolutely do not want to do that, for something that is obviously going to have some simply std-way to do it.

[median berry]

var_to_bytes uses this function:
https://github.com/godotengine/godot/blob/4364ed6ccd001cbfe7cb781d074100695c878d90/core/io/marshalls.cpp#L1361

Here's a condition for int to be encoded into 64 bits (happens when it's out of int32 range):
https://github.com/godotengine/godot/blob/4364ed6ccd001cbfe7cb781d074100695c878d90/core/io/marshalls.cpp#L1372-L1374

Here's a condition for float to be encoded into 64 bits (happens when the Godot's 64-bit float can't be represented exactly as 32-bit float):
https://github.com/godotengine/godot/blob/4364ed6ccd001cbfe7cb781d074100695c878d90/core/io/marshalls.cpp#L1376-L1381

All these types are encoded using floats (32-bit) by default, you'd need a custom Godot build with REAL_T_IS_DOUBLE flag for them to use doubles:
https://github.com/godotengine/godot/blob/4364ed6ccd001cbfe7cb781d074100695c878d90/core/io/marshalls.cpp#L1409-L1422

Color always is encoded as 4 floats (if you want specific format then you should be able to use Image class (create/setup and then get_data())):
https://github.com/godotengine/godot/blob/4364ed6ccd001cbfe7cb781d074100695c878d90/core/io/marshalls.cpp#L1720-L1727

So if:

• you're not using a custom Godot build,
• these are the only types you're using,
• you're using ints in int32 range,
  then you'd only need a workaround for too-precise-for-float32 Godot-float64s. And that's not hard, you can use e.g. the fact that Vector2 by default uses float32:

func as_f32(f: float) -> float:
    return Vector2(f, 0).x

This function would make it internally be converted into float32 (potentially loosing some precision), and then converted back into double/Godot-float64 and returned back. Such float64 will represent exactly the same value after converting to float32 again, and hence will be encoded into 32 bits by var_to_bytes.

So you could fill your array like [42, as_f32(3.14)].

[haughty grove]

huh. That actually works, thanks!

unsure why this one matters:

• you're using ints in int32 range,
  considering I was asking about floats, though? Unless you just meant that var_to_bytes is going to switch to encoding as int64 if my ints in the array are large enough. Because in that case yeah, that would make sense, and actually probably be what I want.

I doubt I would need it, but any good idea how I could likewise force a float32 to be encoded as a float64?
Actually, that turned out to be simple, and my naive guess based on what you had said (that it increases to 64 bits whenever not doing so would lose precision) was correct: enforce a lossy calculation and then revert it. i.e var_to_bytes(1.1 + vec.x - 1.1)

[haughty grove]

I have been working a bit on doing the conversion (aka forcefully lose the precision so no precision is lost if converted to f32) through a/the byte-representation, and now I wonder if I can make it use this as fallback if REAL_T_IS_DOUBLE is set (rather than being for custom builds, it is for in case it because the default in some future godot version)?

as in, can I somehow check the flag?

[median berry]

You've shown an int (42) within the array, so though you care about ints as well. And I meant exactly that: var_to_bytes(0x7f_ff_ff_ff) will return 32 bits (+4 for type/flags of course) but var_to_bytes(0x80_00_00_00) will return 64 bits (even though it fits into 4 bytes...), just something to be aware of if you're doing stuff like this.

Regarding var_to_bytes(1.1 + vec.x - 1.1), it won't work in general case, e.g. for vec.x = 0.0. You'd probably need somthing akin to: https://en.cppreference.com/w/cpp/numeric/math/nextafter, probably manually modifying the bits, not sure.

[haughty grove]

yeah I care about ints, was just confused as the context made it sound like you referenced it in regards to floats

[median berry]

You're asking if you could make the as_f32 function a fallback? If REAL_T_IS_DOUBLE is enabled then Vector2 etc. all use doubles, so of course the workaround used there would no longer work.

For sure REAL_T_IS_DOUBLE won't be a default in 4.x, if anything there would be separate releases using doubles.

[median berry]

You can check relevant feature tag, see: https://docs.godotengine.org/en/4.3/tutorials/export/feature_tags.html#default-features

So OS.has_feature("double")

[haughty grove]

no, I was asking about adding a fallback to me as_f32 method, for when the flag is set so the vector-method wont work

static func as_f32(f: float) -> Variant:
    var REAL_T_IS_DOUBLE = true
    if not REAL_T_IS_DOUBLE:
        return Vector2(f, 0).x
    else:
        var as_bytes = var_to_bytes(f)
        #var metadata = as_bytes.slice(0,4) # 3,0,1,0 ; magic number tells bytes_to_var() to convert double-precision to float
        var metadata = PackedByteArray([3, 0, 0, 0]) # magic number tells bytes_to_var() to convert single-precision to float
        as_bytes = as_bytes.slice(4) # strip metadata
        var single_precision = f64_bytes_to_f32_bytes(as_bytes)
        if single_precision.is_ok():
            return bytes_to_var(metadata + single_precision.unwrap())
        return Result.Err("Failed to convert value %s" % f) # error-state. Would be surprised if this ever happens```
like how this would work, if it could check the flag

[haughty grove]

yeah liek this. is double the same as T_IS_DOUBLE flag?

[median berry]

This should do regardless of the build:

func as_f32(f: float) -> float:
    var bytes := PackedByteArray()
    bytes.resize(4)
    bytes.encode_float(0, f)
    return bytes.decode_float(0)

[median berry]

Yeah: https://github.com/godotengine/godot/blob/4364ed6ccd001cbfe7cb781d074100695c878d90/editor/export/project_export.cpp#L446-L450

https://github.com/godotengine/godot/blob/4364ed6ccd001cbfe7cb781d074100695c878d90/core/os/os.cpp#L428-L436

[haughty grove]

oh yeah, I could have always gone through Packed*Array in the first place, lol. You are right

like even if I didn't want to use encode/decode, I could have done PackedFloat32Array([float64])[0]

Thanks for the help anyway :)

[median berry]

It's funny how so much more complicated solutions come up first.

[haughty grove]

yup

and how I spent a lot of time implementing the complciated one, then hours trying to make it readable despite being bitwise-heavy

instead of thinking of the obvious route :p

I were even converting from having a PackedFloat32Array using .to_byte_array to trying to have some ints too, so that answer was staring me right in the face all this time :d

all I really had to ask myself was "how did I get it to work in the first place? Can't I use that one as a conversion-method?" 😅