The reasoning for this is fairly simple, mostly that storing all scrobbles for everyone would take too much resources. It's manageable on a small scale which is why it together with the features that depend on it are suppporter-only.
When I decided this I actually did the math on it, but should have written it down. Anyway, here it is again (with european number formatting sorry):
So, some baseline numbers:
100 million rows of plays = ~15gb
Average scrobbles per user: 31.978
Total users: 372.356
Total scrobbles to be stored: 11.907.200.168
Storage required for all .fmbot users: 1,79tb
Does that mean its impossible to store all scrobbles for everyone? No. Just look at Last.fm, they're doing it for a lot more users. But it would come with some significant technical challenges. For example you will have to invest a lot of time into performance optimizations at this point. And adding onto that we get around ~200k new users yearly, which adds up quickly.
Also, the scrobble cache limit amount for non-supporters has not been decided yet. I recently rewrote a large part of how it works, but the cache clearing has not been added yet. I'd like for it to be at least a year, but that depends on how it will perform. It will be at least 3 months though.