ManyToMany field using custom "through" table, makemigrations wants to delete. | Django | Page 1

void locust May 14, 2024, 8:10 PM

#

Hi everyone,

Posting a new thread here as the old one might be stale. The other post was about displaying something on the front-end, and this is more about the migrations.

I've read and followed these sources:

I have an existing many-to-many relationship in an app called stem:

class Collection(models.Model):
    datasets = models.ManyToManyField(
        Dataset,
        through="CollectionDataset",
        verbose_name='Datasets',
        related_name='collections')

Django creates the table stem_collection_datasets automatically when I define a ManyToMany relationship this way. So this is now an existing table with existing records that holds these relationships.

Now I want to add an additional field to this relationship, order, so that I could order the m2m relationships. To do this, I have to actually create a through table myself and specify this new field. So I can no longer use the through table that Django created automatically:

class Collection(models.Model):
    datasets = models.ManyToManyField(
        Dataset, 
        verbose_name='Datasets', 
        related_name='collections')


class CollectionDataset(models.Model):
    class Meta:
        ordering = ('order',)
        auto_created = True
        unique_together = ['dataset', 'collection'] 
    dataset = models.ForeignKey(Dataset, on_delete=models.CASCADE)
    collection = models.ForeignKey(Collection, on_delete=models.CASCADE)
    order = models.PositiveIntegerField(null=True, blank=True, default=0)

The instructions linked above indicated that if I just create a new through table manually, then create a migration, it will delete the old through table and I will lose the existing relationships. After following all of the instructions, I've ended up with two migration scripts. I'll post them as a comment because I'm out of characters for the initial post.

#

My migrations looks like this:


# migration #0050

from django.db import migrations, models
import django.db.models.deletion


class Migration(migrations.Migration):

    dependencies = [
        ('stem', '0049_alter_userrole_user_with_role'),
    ]

    operations = [
        migrations.SeparateDatabaseAndState(
            database_operations=[
                # Old table name from checking with sqlmigrate, new table
                # name from CollectionDataset._meta.db_table.
                migrations.RunSQL(
                    sql='ALTER TABLE stem_collection_datasets RENAME TO stem_collectiondataset',
                    reverse_sql='ALTER TABLE stem_collectiondataset RENAME TO stem_collection_datasets',
                ),
            ],
            state_operations=[
                migrations.CreateModel(
                    name='CollectionDataset',
                    fields=[
                        (
                            'id',
                            models.AutoField(
                                auto_created=True,
                                primary_key=True,
                                serialize=False,
                                verbose_name='ID',
                            ),
                        ),
                        (
                            'dataset',
                            models.ForeignKey(
                                on_delete=django.db.models.deletion.DO_NOTHING,
                                to='stem.Dataset',
                            ),
                        ),
                        (
                            'collection',
                            models.ForeignKey(
                                on_delete=django.db.models.deletion.DO_NOTHING,
                                to='stem.Collection',
                            ),
                        ),
                    ],
                ),
                migrations.AlterField(
                    model_name='collection',
                    name='datasets',
                    field=models.ManyToManyField(
                        to='stem.Dataset',
                        through='stem.CollectionDataset',
                    ),
                ),
            ],
        ),
        migrations.AddField(
            model_name='collectiondataset',
            name='order',
            field=models.PositiveIntegerField(null=True, blank=True, default=0),
        ),
    ]


# migration #0051

from django.db import migrations, models
import django.db.models.deletion


class Migration(migrations.Migration):

    dependencies = [
        ('stem', '0050_auto_20240426_1157'),
    ]

    operations = [
        migrations.AlterModelOptions(
            name='collectiondataset',
            options={
                'ordering': ('order',),
                'auto_created': True,
            },
        ),
        migrations.AlterField(
            model_name='collection',
            name='datasets',
            field=models.ManyToManyField(related_name='collections', through='stem.CollectionDataset', to='stem.Dataset', verbose_name='Datasets'),
        ),
        migrations.AlterField(
            model_name='collectiondataset',
            name='collection',
            field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='stem.collection'),
        ),
        migrations.AlterField(
            model_name='collectiondataset',
            name='dataset',
            field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='stem.dataset'),
        ),
        migrations.AlterField(
            model_name='collectiondataset',
            name='id',
            field=models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID'),
        ),
    ]

humble shard May 14, 2024, 8:13 PM

#

Have you tried setting the db_table meta field to the same name as the current table?

void locust May 14, 2024, 8:14 PM

#

I have and I haven't seen a difference. I've tried with and without the app name prefix (in my case, stem_collection_datasets or collection_datasets or collectiondataset etc.). Thanks for the help. I need to drop offline for a couple hours. I'll be back online later.

humble shard May 14, 2024, 8:16 PM

#

I can check this against my own work from a while ago tomorrow.

sonic garnet May 15, 2024, 7:03 AM

#

The issue is that django cannot see in its history, that the through model ever existed. Short of going the unmanaged route, it is possible to do this and keep it managed. Here's a rough example. Given a Workshop model as follows:

class Workshop(models.Model):
    name = models.CharField(max_length=100)
    attendees = models.ManyToManyField(User)

Create the migrations, apply them, then add data to attendees.
The backing table is now core_workshop_attendees.
We make the changes needed to the models:

class Workshop(models.Model):
    name = models.CharField(max_length=100)
+    attendees = models.ManyToManyField(User, through="WorkshopUser")


+class WorkshopUser(models.Model):
+    workshop = models.ForeignKey(Workshop, on_delete=models.CASCADE)
+    user = models.ForeignKey(User, on_delete=models.CASCADE)

+    class Meta:
+        db_table = 'core_workshop_attendees'

Even doing this, will not rid us from the changes being detected, because the registry doesn't contain an explicit model for WorkshopUser.
We now have two options. Either update a previous migrations file with the relevant model

        migrations.CreateModel(
            name='WorkshopUser',
            fields=[
                ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
                ('user', models.ForeignKey(on_delete=models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)),
                ('workshop', models.ForeignKey(on_delete=models.deletion.CASCADE, to='core.workshop')),
            ],
            options={
                'db_table': 'core_workshop_attendees',
            },
        ),

This will trick django into thinking the model exists. OR - create a new migration that contains that very same code and instead of asking django to migrate, add the migration yourself to the django_migrations table. Both options are dirty, but CreateModel doesn't have any skip option.

#

Now if i test via the shell, everything is working fine. I can now add a new field to the WorkshopUser through table and run make migrations and it will detect the changes successfully.

$ python manage.py makemigrations
Migrations for 'core':
  core/migrations/0003_workshopuser_date.py
    - Add field date to workshopuser

P.S. You will also need to create the content type in a migration, something along the lines of:

    ContentType = apps.get_model('contenttypes', 'ContentType')
    ContentType.objects.create(app_label='core', model='workshopuser')

#

The simplest option you have, is to create the new model, copy the data from the old table then delete the old table. (which I believe should not present any issues, considering there are no direct references to your existing m2m table records.)

humble shard May 15, 2024, 7:15 AM

#

Yeah, I think we did that too, create a new table with FK's to both (don't change the m2m) migrate data from the old table to the new one then change the m2m and it should all work.

void locust May 15, 2024, 12:59 PM

#

Thanks I will read through this. It sounds like you're saying I can do this all in migrations, moving the data from the old table to the new table before it deletes. After this call I'm on I will read back through carefully and see if I can make sense of it

#

Here's what I think you are saying:

Assuming the migration already exists that created the m2m relationship originally, without my custom through table
Create the through table but don't "hook it up" yet. Keep the existing model using the general m2m relationship and not the custom through table. Make a new migration that will create the new through table
Create another migration that copies all of the data from the old table to the new table
Update the code so that the field references the new through table
Create another migration that will delete the old table and keep the new table

If this is what you are thinking, it sounds like it might work except I'm not sure whether I still need to have a _meta.db_table in this instance or not.

sonic garnet May 15, 2024, 1:18 PM

#

Correct, you don't need the db_table

void locust May 15, 2024, 2:00 PM

#

For some reason it's telling me "No changes detected." I'm using sqlite, and I have a backup database from before any migrations. So now I have the model using the old m2m relationship, and I have created the new through table but I haven't hooked it up yet. So now I tried the makemigrations, and it says no changes detected. Will I need to create a migration manually?

sonic garnet May 15, 2024, 2:12 PM

#

No, you should get a migration.

void locust May 15, 2024, 3:38 PM

#

Okay, so after all of this, I still have the problem where a new migration wants to delete the model. What am I missing? I have three migrations:


# migration 0050

class Migration(migrations.Migration):

    dependencies = [
        ('stem', '0049_alter_userrole_user_with_role'),
    ]

    operations = [
        migrations.CreateModel(
            name='CollectionDataset',
            fields=[
                ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
                ('order', models.PositiveIntegerField(blank=True, default=0, null=True)),
                ('collection', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='stem.collection')),
                ('dataset', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='stem.dataset')),
            ],
            options={
                'ordering': ('order',),
                'unique_together': {('dataset', 'collection')},
                'auto_created': True,
            },
        ),
    ]


# migration 0051
class Migration(migrations.Migration):

    dependencies = [
        ('stem', '0050_collectiondataset'),
    ]

    operations = [
        migrations.RunSQL(
            sql=[('INSERT INTO stem_collectiondataset (id, `order`, dataset_id, collection_id)  SELECT id, 0, dataset_id, collection_id FROM stem_collection_datasets;', [])],
            reverse_sql=[('INSERT INTO stem_collection_datasets (id, dataset_id, collection_id)  SELECT id, dataset_id, collection_id FROM stem_collectiondataset;', [])],
        ),
    ]


# migration 0052
class Migration(migrations.Migration):

    dependencies = [
        ('stem', '0051_collectiondataset_sql'),
    ]

    operations = [
        migrations.RunSQL(
            sql=[("DROP TABLE stem_collection_datasets;", [])],
            reverse_sql=[('CREATE TABLE "stem_collection_datasets" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "collection_id" char(32) NOT NULL REFERENCES "stem_collection" ("id") DEFERRABLE INITIALLY DEFERRED, "dataset_id" char(32) NOT NULL REFERENCES "stem_dataset" ("id") DEFERRABLE INITIALLY DEFERRED)', [])],
        ),
    ]

You mentioned about adding ContentType above. Could that be why this is happening?

sonic garnet May 15, 2024, 4:01 PM

#

Migration #1 - Auto - Create model B (you already have this)
Migration #2 - Data - Copy the content from A to B
Migration #3 - Auto - Add through field to B in existing many to many field. This will delete model A.

void locust May 15, 2024, 4:52 PM

#

I think my Migrate #2 matches what you wrote too, right? So it's just the third that I've gotten wrong?

#

I thought perhaps it was because the names were too similar. So I renamed my through table and I'm still stuck on #3. It wants to delete my through table.

humble shard May 15, 2024, 5:38 PM

#

The third migration drops the old table right? If the data is migrated that's fine

void locust May 15, 2024, 5:46 PM

#

It's not, it's dropping the new table

#

Even when I change the name to make sure there is no confusion about that, it keeps wanting to delete my new table. I don't get it.

sonic garnet May 15, 2024, 5:54 PM

#

I'm sorry i don't have a machine handy, i'll run a quick test tomorrow.

void locust May 15, 2024, 5:55 PM

#

Thanks, I really appreciate it. Hate being totally stuck and really appreciate the help.

void locust May 15, 2024, 6:44 PM

#

I may have figured out why the new table isn't getting detected when running makemigrations, i.e. no createmodel statement is being inserted. I had the meta property auto_created = True. Without this statement, I can't run migrations because I get the admin.E013 error, which I've read is because Django doesn't want you to include m2m fields in a Django admin field list. Even though it worked before, until I created my custom through table. From what I read, adding the auto_created = True statement fixes that error. And it does, but I think it also then doesn't detect it. So then, when I create the database manually, the next time I run makemigrations, that's probably why it's deleting it (because it sees it in the database, but doesn't find code for it with that field set).

void locust May 15, 2024, 8:09 PM

#

Ok so I think I've figured it out. At least, I've hacked my way around it. The above is true... with auto_created= True, Django doesn't recognize that the model is in the code, so it tries to delete it. Without that, I can't include the m2m field in field lists in the admin modules. So what I had to do is let it create an additional migration to delete that model, but then override the apply and unapply functions so that it doesn't do anything. After running migrate on that, when I run makemigrations again, it's no longer trying to delete it. So while it's hackish, I think I've actually got it working. And reverting the migration too. Thanks for the help!

humble shard May 15, 2024, 8:19 PM

#

Glad you figured it out, I totally missed the auto_created on your model. But that explains it

#ManyToMany field using custom "through" table, makemigrations wants to delete.