gilded plover
Hi guys, I raised this question quite some time ago but noticed today that my models use old data for fine-tuning.
Can we get an ETA when this is fixed? Many tournament participants would prefer the most up-to-date data, especially due to retraining and fine-tuning on weekends.
Thanks.
cc @wintry jackal @ripe nymph
robust nova
Of course "making it consistent" would likely involve holding onto newer data for an extra week to make sure it is always available for the weekend update. The inconsistency comes from using the latest data they've got at the time the data file needs to be locked. Unless they are locking that file willy-nilly at different times each weekend.
gilded plover
I don't think they are locking it "willy-nilly", I think it's just like @wintry jackal said: the historic data generation pipeline is currently independent of the targets generation pipeline.
All the data is already there, the only (occasionally) missing part is targets from 4(5?) weeks before. Which is independent of data providers and appears to be computed on the same weekends anyway (@wintry jackal please correct me if I'm wrong).
It just needs to be made a pipeline dependency for the historic data generation.