#Relative custom pointer types in an IPC library

21 messages · Page 1 of 1 (latest)

unborn willow
#

As some of you may know, I have been developing a library with custom allocators that provide custom pointers - the goal fo this has been to absolutely minimize the memory footprint of the datastructures themselves.
The primary mechanism to achieve this reduction have been custom pointer types, such types that only store the bare minimum information that they then use together with the information stored in their static storage member to resolve into a full real pointer.
An example to help you imagine would be the offset_ptr : directly stores the offset as a member variable, usually in a very narrow type like the uint8_t, and then when the time comes and they need to resolve into a real ponter, they access the static storage, look up the base address for the memory region, and just do offset arithmetic to find out the real pointer.

The trick, and what is doing the heavy lifting here is the static_storage itself - it's a special static member, whose type is parametrized on a lambda.
The lambdas type is captures in the allocators type, then passed down to the pointer_type- so the allocator is always responsible for the creation and the provision of the custom pointer type.
Since the static storage member is in both the pointer and allocator is parametrized on the same unique lambda type, it's the same static storage, and therefore both the pointer and the allocator can used is as a sort of a "shared memory" region.

The advantage of this is very light pointer size.
The main disadvantage is the fact that it forces the allocators and the pointers to be unique types (this makes them more difficult to work with and make it impossible to move them accoss TU boundaries).

I now wish to create an IPC library from my queue & allocators. I like the idea of light pointers, but in the current state they are too cumbersome to use.
Can I somehow refactor/rearchitect, so that the custom pointers will be a great addition in the IPC library?

vagrant siloBOT
#

When your question is answered use !solved or the button below to mark the question as resolved.

Remember to ask specific questions, provide necessary details, and reduce your question to its simplest form. For tips on how to ask a good question use !howto ask.

floral stream
#

Can you make the information obtained by the lambda explicit via template parameters? That way the types would no longer depend on the lambda and allow explicit instantiation. You can keep the lambda for deduction purposes.

unborn willow
floral stream
#

Ah, you mean you might have 2 int arrays of the same size and need the lambda type to figure out which static data members to use?

unborn willow
#

but essentially the static there is just for communication
the pointer types themselves must share the same static storage, so that the size cost of the storage is amortized, but the pointer and the allocator may not necesarily share the same static storage, the allocator may not have any static storage, but then they will have to communicate in a different way

floral stream
#

I guess giving it a number or something would work, but it's probably a pain to remember what is what.

unborn willow
floral stream
#

Anyway, for an IPC I don't see much hope for your small pointers. Usually you only pass arrays and indexes anyway, pointers don't work well between processes.

unborn willow
#

which these kindof are

floral stream
#

Yeah, so, indexes. Which is what everyone already uses.

#

I like the added type safety of making the offset only work for the right container, I think, assuming that's what you're doing.

#

What is the core feature you wish to preserve from the lambda approach that regular arrays and indexes don't have?

unborn willow
#

well the I assume that by indexes you mean that the pointer stores just an index into an array that is the memory controlled by the allocator
so an offset essentially

this is one of the pointers I use, and it's the much simpler one
then I have segmented_ptr, that works with a discontigous allocator, and this discontigous allocator manages groups of segments, and only the segments themselves are contigous

so the resolution strategy here is not so simple, and requires storing more information in the static storage, like a base pointer (for type erasure, to erase away the unique type) to the allocator, and only the allocator itself can properly resolve the pointer accross the groups and segments

#

my main motivation is that in most cases, using full 8 byte pointers is just wasteful, and actually need a lot less information to encode the "position" of the object in the memory

#

the question now is what is the best way so encode it

floral stream
#

Hmm. Allocators don't do so well across processes and non-arrays aren't great to pass through networks.

#

The thing is that the other side needs to allocate anyway, may as well just send an array and convert pointers to indexes.

#

I can't think of anything better than "just use arrays and indexes".

#

Which should be fairly easy to do, just treat your pointers and allocators as implementation details that are not part of the protocol.