Cargo failing to store data

[wraith sage]

Hello, I am an editor for the Project SEKAI wiki Sekaipedia. We're currently having issues with multiple of our Cargo tables inconsistently storing data in certain columns.

For example, the table virtual_lives is defined by the template {{Infobox virtual live}}. All columns are stored correctly except for song id, which is blank across multiple rows. You can check the source on the pages and see they're filled in correctly there. Recreating the table did not fix the issue.

This is occurring across multiple tables, but seems to be the same column(s) each time. I can't really find a pattern to which columns work consistently and which don't, but it tends to be integers and lists of any type.

For lists, issues seem to appear more often when storing more than one list item. For example, these two pages were made at the same time, and the page with one ID stored but the one with multiple did not. However, there are other pages in this table that store multiple values just fine.
https://www.sekaipedia.org/wiki/Special:CargoTables/gacha_rate_up

Any advice to go about fixing this issue would be greatly appreciated 🙏

[surreal tinsel]

https://meta.miraheze.org/wiki/Infobox_virtual_live?action=edit&redlink=1

[nimble gust]

Created a phorge task for you https://issue-tracker.miraheze.org/T14728

[wraith sage]

Thank you!!

[nimble gust]

(copying my Phorge response here for reference)

I thought about this a bit more. It seems that this issue can be circumvented by making this field a list of strings instead of a list of integers. This wiki uses Lua to perform queries, so strings can be easily converted to integers.

Even for database operations, this shouldn't matter much since mariadb automatically casts strings to integers when performing comparisons against an integer (e.g. during a join). The only downside is a bit more storage overhead.

Also, the https://www.sekaipedia.org/wiki/List_of_Virtual_Lives page takes around 5 seconds to parse right now. Please try to use joins instead of performing a cargo query of song ids for each live and see if performance improves. Cargo query is fairly cheap but doing it hundreds of times will lead to performance problems when this Cargo issue is resolved. See https://stellasora.miraheze.org/wiki/Banner_List for an example 3-way join. Maybe I should do this myself...

And as an unrelated side note, the list of virtual lives page seems like a good use case for https://dev.miraheze.org/wiki/Template:FilterTable. The table could benefit from some filter buttons.

[wraith sage]

Thank you for looking into it! I'm willing to try storing the integer lists as strings, though there are two places I'm aware of where we're having difficulty storing lists of strings. It only shows up with character lists containing more than approx. 20 items. It might be a separate issue, but there are other rows in the same table that store 30 characters fine, so I assume its related.
https://www.sekaipedia.org/wiki/Special:CargoTables/connect_lives
https://www.sekaipedia.org/wiki/At_This_Festival_Tinged_With_Twilight/Story, compare with https://www.sekaipedia.org/wiki/Link_the_Beats!/Story

Thank you for letting me know about the joins, though! I'll definitely be implementing those when I get a chance. And I'll chat with the other editors about the filter as well, its a great suggestion.

[nimble gust]

Hmm. String has a length limit of 300 characters according to https://www.mediawiki.org/wiki/Extension:Cargo/Storing_data#Declaring_a_table, and in my tests this is enforced the list of strings as well. In a template call like in the picture, t4 fails but t5 succeeds because list of text doesn't have a length limit.
Switching to text may have performance implications due to it being unindexed, but I won't be too concerned about performance when correctness is affected.

I made a proof-of-concept for joins in https://www.sekaipedia.org/wiki/User:PetraMagna/sandbox?oldid=256925
which uses these two pages as templates
https://www.sekaipedia.org/wiki/User:PetraMagna/Template:Virtual_live_year
https://www.sekaipedia.org/wiki/User:PetraMagna/Template:Virtual_live_row
It still has a few issues but I think it's not too complicated to have them ironed out.

The Characters column needs to have a Lua module split the string by ; and then call the function that adds profile pictures, which shouldn't be too complicated. The Songs column is untested for lives with multiple songs, so hopefully GROUP_CONCAT works as expected on those. It currently ignores lives without song ids (which it shouldn't: it's a left outer join), but that'll go away once the cargo_store issue is resolved.