#Distance between to spans

20 messages · Page 1 of 1 (latest)

ionic storm
#

I am trying to write code that serializes objects into a std::vector<std::byte> to then write that into a file. Each section of the file gets a small header which contains uint64_ts that represent byte offsets into the start of the file at which data is located. As I'm using C++20 I want to use std::span as much as possible. I currently have a std::spanstd::byte on which I frequently call subspan to advance the bytes. Though now my issue is how do I calculate the difference between two spans without having a bunch of reinterpret_cast and pointer arithmetic in my own code. This is what the code looks like currently:

    auto* fileHeader = reinterpret_cast<ShaderFileHeader*>(output.data());
    fileHeader->magic = headerMagic;
    fileHeader->shaderCount = inputCount;

    // Create a view of the section containing shader headers and write the basic header info.
    std::span<ShaderDescription> shaders = { reinterpret_cast<ShaderDescription*>(output.data() + sizeof(ShaderFileHeader)), inputCount };
    for (auto i = 0U; i < inputCount; ++i) {
        auto& input = inputs[i];
        shaders[i].byteSize = input.shaderBytes.size();
        shaders[i].stage = input.stage;
        shaders[i].lang = input.lang;
    }

    auto offset = sizeof(ShaderFileHeader) + shaders.size_bytes();
    for (auto i = 0U; i < inputCount; ++i) {
        auto& input = inputs[i];

        // Write the name to the buffer.
        auto size = input.name.size() + 1; // For the null terminator.
        shaders[i].nameByteOffset = offset;
        std::memcpy(&output[offset], input.name.data(), size);
        offset += size;

        // Write the shader binaries to the buffer.
        shaders[i].byteOffset = offset;
        std::memcpy(&output[offset], input.shaderBytes.data(), input.shaderBytes.size());
        offset += input.shaderBytes.size();
    }
noble thunderBOT
#

When your question is answered use !solved to mark the question as resolved.

Remember to ask specific questions, provide necessary details, and reduce your question to its simplest form. For more information use !howto ask.

loud yarrow
#

This feels super sus and UB, and I'm not entirely sure what your aim is here anyway

#

In particular, you can't reinterpret_cast<ShaderDescription*> if there isn't actually a ShaderDescription object there for you to refer to, otherwise you're violating strict aliasing

loud yarrow
#

My attempt at rewriting your code without actually knowing what you're doing: ```cpp
void serialise(std::spanstd::byte output_buffer, std::span<const Shader> input) {
// really we want an output iterator, not an output range
// so there's no need to be subspanning, because we don't care about its size;
// we'll also use this helper function to factor out the "memcpy and step"
// that we'd otherwise be repeating over and over
auto write = [output = output_buffer.data()](const void* data, std::size_t size) mutable {
std::memcpy(output, data, size);
output += size;
};

// first write the file header
ShaderFileHeader header {
.magic = headerMagic,
.shaderCount = input.size()
};
write(&header, sizeof header);

// then we calculate byte offsets in one pass as we first write the section headers
auto data_offset = sizeof(ShaderDescription) * input.size() + sizeof hdr;
for (const auto& shader : input) {
// (you'll need to reorder these fields to match your ShaderDescription definition)
ShaderDescription description {
.nameByteOffset = data_offset,
.byteOffset = data_offset + shader.name.size(),
.byteSize = shader.shaderBytes.size(),
.stage = shader.stage,
.lang = shader.lang
};
write(&description, sizeof description);
data_offset = description.byteOffset + description.byteSize;
}

// and finally we write the data as we go
for (const auto& shader : input) {
write(shader.name.data(), shader.name.size());
write(shader.shaderBytes.data(), shader.shaderBytes.size());
}
}

#

(also I got rid of the "include null terminator" because serializing null terminated strings is silly if you already know their size)

ionic storm
#

it's like allocating a new char[sizeof(ShaderDescription)] and then casting that to a ShaderDescription.

ionic storm
ionic storm
#

I've also improved the deserialization based on your code

    auto read = [output = bytes.data()](void* data, std::size_t size) mutable {
        std::memcpy(data, output, size);
        output += size;
    };

    ShaderFileHeader header;
    read(&header, sizeof header);

    ShaderLibrary library;
    library.functionNames.resize(header.shaderCount);
    library.binaries.resize(header.shaderCount);

    // We first read all the shader descriptions
    for (auto i = 0U; i < header.shaderCount; ++i) {
        ShaderDescription description;
        read(&description, sizeof description);
        library.binaries[i].desc = std::move(description);
    }

    // Then read all names and binaries.
    for (auto i = 0U; i < header.shaderCount; ++i) {
        auto& binary = library.binaries[i];

        // We read the name first and determine its size by looking where the shader bytes begin.
        binary.name.resize(binary.desc.byteOffset - binary.desc.nameByteOffset);
        read(binary.name.data(), binary.name.size());

        binary.bytes.resize(binary.desc.byteSize);
        read(binary.bytes.data(), binary.bytes.size());

        library.functionNames[i] = binary.name;
    }
loud yarrow
#

There's some exceptions for certain functions and implicit-lifetime types, but "pull data out of raw memory" isn't one of them

#

You would need to use new(output) ShaderDescription or something to create the ShaderDescription object in raw memory for that kind of thing to work, but we may as well just memcpy (that all compilers will be able to optimise equivalently anyway)

ionic storm
#

oh ok thanks

#

well I think this solved it

#

!solved

noble thunderBOT
#

Thank you and let us know if you have any more questions!

#

[SOLVED] Distance between to spans

noble thunderBOT
#

@ionic storm

This question thread is being automatically marked as solved.