thoughts and advice on handling dynamic set of columns in an NxM grid

[minor kelp]

I'm looking for some input on Model design and Form handling. The client can specify a set of TestPoint that they will add Trials to. The number of Trial that should be captured for every TestPoint depends on a Source.total_trials field.

class Source(Model):
    total_trials = IntegerField()

class TestPoint(Model):
    source = FK(Source)
    value = FloatField()

class Trial(Model):
    testpoint = FK(TestPoint)
    value = FloatField(null=True)

The end result is an NxM matrix where the first column depicts all the TestPoint.value and every subsequent column is the Trial "Number" and its value. See this representation of how it currently looks:

https://ibb.co/CPjMW59

# Model Related Question

The first glaring problem in the above model representation is a missing Trial number field, if TestPoint1 Trial4 was set and the previous ones werent, we need to make sure the persistence captures which trial this is. The second issue I'm noticing is how tiny the Trial model is, unlike the other two models, the Trial model is really that small, it only captures a "Value".

Question - I'm contemplating dropping the Trial model altogether and making trials a mere Array field on the TestPoint model with a default value of [0.0, 0.0, 0.0, 0.0, 0.0] this will tackle the ordering issue and drop an unnecessary model that will grow indefinitely and not carry anything more than a value. Thoughts? Any other idea?

# Form Related Question

The way the form is being built assigns text input names according to what they represent:

test points = tp_1, tp_2, tp_3, ...
trials = tp_1_1, tp_1_2, ... tp_1_3 ...

and the html display is via a CSS grid that is created to suspiciously look like a inline django form.

Question - Since the number of trials is dynamic, would you use a django formset for this, instead of handling it manually the way I'm doing?

thoughts and advice on handling dynamic set of columns in an NxM grid

[glossy schooner]

I was going to suggest either an array field or a JSON field on TestPoint to represent the trials. In your example models is Trial missing a FK to TestPoint? At the moment it's just an entirely isolated "value" which you commented on but surely that can't be the case?

[minor kelp]

Edit - added the foreign key to test point, thanks.

[glossy schooner]

As for the forms, I wouldn't necessarily use a formset, but that's only because I've never used them and don't know what they can do 😆 I've done this in the past by compiling a list of Form objects each with a unique prefix. That may be exactly what a formset does anyway 🤷‍♂️

[minor kelp]

That makes two of us, I've used them before but they never clicked for me.

[glossy schooner]

I'll see if I can find the code I used previously

[minor kelp]

So your advice is, move to a json/array field, have a form with a dynamic aspect to it for the trials and make it prefixed accordingly. prefix-tp-N

[glossy schooner]

Yeah

[minor kelp]

that sounds fair, I'll give that a go, thanks

[glossy schooner]

Ah, I do that in the form itself apparently. This is the CBV get_forms method: ```py
def get_forms(self, template):
forms = []
if not template:
return forms
for section in template.sections.order_by("name"):
if self.request.method == "POST":
form = ProgressionSectionForm(data=self.request.POST, template=section)
else:
form = ProgressionSectionForm(template=section)
forms.append(form)
return forms

class ProgressionSectionForm(forms.Form):
    section_name = forms.CharField(required=True)
    target_bpm = forms.IntegerField(required=True, initial=120, label="Target BPM")

    def __init__(self, *args, **kwargs):
        self.template = kwargs.pop("template")
        kwargs["prefix"] = f"section{self.template.pk}"
        super().__init__(*args, **kwargs)

[minor kelp]

I'm assuming you're calling that get_forms manually, it's not part of the lifecycle of your CBV's base class [can't find it anywhere in the docs]

[glossy schooner]

Oh lol. Yeah it's get_form normally isn't it!

[minor kelp]

Yup ^.^ for a second there I thought you were hiding some voodoo 3rd party package from us. First thing I did was check if that existed in: https://django-brackets.readthedocs.io/en/latest/mixins/form_views/

I remember i read something about "Multiple Forms" in that package.

[glossy schooner]

def get_context_data(self, *args, **kwargs):
        context = super().get_context_data(*args, **kwargs)

        chosen_template = self.template or self.get_object()

        context["chosen_template"] = chosen_template
        context["forms"] = self.forms or self.get_forms(chosen_template)
        context["name_form"] = self.name_form or self.get_name_form()

        return context

    def post(self, *args, **kwargs):
        self.template = self.get_object()
        self.forms = self.get_forms(self.template)
        self.name_form = self.get_name_form()

        cleaned_forms = []
        for form in self.forms:
            if form.is_valid():
                cleaned_forms.append(form)

        if len(cleaned_forms) != len(self.forms) or not self.name_form.is_valid():
            return self.get(*args, **kwargs)

        template_data = [f.section_info() for f in cleaned_forms]
        new_progression = Progression.create(
            self.request.user,
            self.name_form.cleaned_data["name"],
            template_data,
        )

        return redirect(reverse("gold:index"))```

[minor kelp]

yup sounds about right!

That'll atleast keep the logic isolated in the form, per testpoint.

Thanks Ben!

[glossy schooner]

No worries! I think formsets are probably the "correct" answer but like you I was turned off them a while back and never investigated more

[minor kelp]

Biting the bullet. It's high time to accept that someone already did the heavy lifting with formsets and we just need to adapt it. Wish me luck! ^.^

[glossy schooner]

Good luck! I will be interested to see how much wrangling they take

[minor kelp]

I'll post an update if and when this works.

[minor kelp]

I'm getting my ass handed to me by formsets. 😂 ||well, formsets with dynamic set of fields||

[glossy schooner]

Not going well? 😅

[minor kelp]

I think it's fine so far, I put some effort into it few days back:

• I implemented the sample outside the scope of the project I'm working on such that nothing affects the implementation
• Got it to work with a JSONField as a TextInput just fine.
• Got it to generate [and load] the Trial fields dynamically.
• Now it's just a matter of getting it to save correctly. Hit a speed bump then I stopped, i'll get back to it soon.

When this works, I'd really like to cleanly plug in htmx, but that will come after the former works fine with formsets.

[glossy schooner]

That’s looking really nice

What CSS framework do you use?

Bootstrap?

[minor kelp]

that's bootstrap 5

I'm keeping it as few dependencies as possible so I can share the solution in the end, so it's just sqlite and a couple of django dependencies.

[glossy schooner]

Say what you will about it, it does allow for very rapid getting to a “looking pretty good” stage

[minor kelp]

What's even more suprising is that this is via crispy-forms as well.

[glossy schooner]

Even better

[minor kelp]

        self.helper.layout = Layout(
            'id',
            'trials',
            Row(
                Column(FloatingField('value')),
                *[
                    Column(FloatingField(f'trial_{i+1}'))
                    for i in range(total_trials)
                ]
            )
        )

[glossy schooner]

Oh that’s nice

[minor kelp]

Here we go. It's rather chaotic and unpolished right now granted I was also testing some business logic, but should be easily traceable. There's one primary missing scenario, which I highlighted in forms.py.

You can ignore any "Bootstrapping" logic.

Sorry for the zip file, I simply can't put this publicly for now ^.^