FSRS Megathread

[tepid spoke]

that takes away a lot of flexibility from FSRS, compared to if it'd be looking at something else for initial stability

[unique salmon]

But what though? There is no other info, other than the grade

A neural net could look at deck/preset/siblings/whatever, but that's a different story

[tepid spoke]

Could for example calculate the mean of all learning phase reviews, and then from that number derive it by averging the two parameters around it

[unique salmon]

No, like
When you first review a card, FSRS has to come up with the next interval. How would it do that?

[tepid spoke]

It hasn't though, at that point in time the learning steps still do

[sick moth]

Most of the people who make money consistently:

• Are the smartest people you'll ever meet
• Have years of experience
• Have near unlimited resources
• Can dedicate their lives full time to the grind

I'm not saying you can't make money, but the odds are against you

This is a choice by FSRS, no?

[unique salmon]

No

[tepid spoke]

Obvious problem with my "averaging" idea is that "1333" would not be very different to "1331333", even though it probably should be

[sick moth]

Anki can't handle it well, yet, but any other dependencies on the library have the same constraints transitively

... maybe I misread, 1 second

[tepid spoke]

In theory FSRS could forever keep a card in the learning phase, and would then be able to set second-accuracy timestamps forever

[unique salmon]

I'll call you in 10 years when I have a million dollars 😎
In all seriousness though, making trading strategies is by no means easy (again, hence why I recommend just buying ETFs to most people), but I'm quite positive that I can do it

[tepid spoke]

My ETFs have averaged out to exactly +-0 after 5 years now...

used to be +20% not too long ago

[sick moth]

Whats the starting salary at HRT? 😉

[tepid spoke]

@unique salmon remember that PR I made? It was an attempt of dealing with an actual learning learning phase. It hid all first revies from FSRS until the first "arbitrary longer interval".

[unique salmon]

https://en.wikipedia.org/wiki/Hudson_River_Trading

It differs from stereotypical high-frequency trading firms in several important ways: it holds about 25% of its trading capital overnight (unlike most high-frequency trading firms that hold almost nothing overnight), its average holding time is about five minutes as opposed to the sub-second times observed for some high-frequency trading firms
My average holding time is measured in weeks 🤣

[tepid spoke]

Cause looking at my at that point 16k+ cards showed that that strategy lead to a much wider spread of initial stabilities, than just "95% Again"

[unique salmon]

Nope, remind me

[sick moth]

Not my point. What's the starting salary at Jump?

[tepid spoke]

https://github.com/ankitects/anki/pull/3734

[sick moth]

I wouldn't be surprised if those guys LITERALLY move mountains

[unique salmon]

Has it been tested on the 10k dataset though?

[tepid spoke]

nope, I sent it as an RFC for the Idea, and it was just "noped"

[ashen light]

they're pretty real. like say I download a random deck off ankiweb and just start going, the first time I see a card its gonna inhernetly be a learn and not a review

[unique salmon]

No idea

[tepid spoke]

Like, my implementation just has a hard coded time limit of "first interval longer than 2h", that's obviously not ideal and only fits my specific deck

It'd either need to be a setting in the Deck/Anki, or become another FSRS parameter

[unique salmon]

Propose it in the benchmark repo

[ashen light]

you forgot

• endless amounts of cocaine

[tepid spoke]

btw., I bit myself a bit with splitting my WaniKani deck into subdecks... it messed with the order, cause apparently Anki picks cards in Alphabetic order of the decks first, and only THEN based on the "New #" shown in the Browser...

so I suddenly got Kanji whose Radicals I had never seen

Luckily "Kanji" and "Vocab" are in the right order :D

[sick moth]

It's geeks with spreadsheets these days.

Ferraris and cocaine are on the way out. Geeks make more

[ashen light]

my bad, geeks are on straight amphetamines

[unique salmon]

Call me insane if you want, but I think I have a higher chance of 10xing my money in 7 years than getting a gf within the same period of time
Completely unironically, without a shred of irony

[sick moth]

https://tenor.com/view/danarco-sbf-sb-jailed-gif-27289502

[tepid spoke]

My University busted an Adrall-Syndicate last year

so it's real

[ashen light]

@unique salmon hey with that kind of money you can just buy a gf

[sick moth]

A million doesn't go as far as you think

[ashen light]

dude I knew a guy at cmu who said the entier engineering dept was microdosing actual meth

not even adderal or whatev

[tepid spoke]

I'm at Uni since close to 15 years now, and I never got into contact with anything or anyone related to drugs

Well, except of course alcohol and weed

[ashen light]

those hardly count nowadays

[tepid spoke]

both are legal here

[robust hill]

you can do whatever you want as long as you put your efforts to it

[ashen light]

yeah same no drugs here either, but I went to an irrelevant school in the middle of nowehre, no one needed to do hard drugs to do the workload

[tepid spoke]

@unique salmon does the 10k dataset contain the deck settings? Specifically the learning steps.

[sick moth]

Mostly due to that attitude

[ashen light]

he needs a mil in the bank before he thinks a woman will want to interact with him

[tepid spoke]

You know how exepensive a girlfriend is? And I don't mean buying one.

And no I'm not serious :D

[sick moth]

You don't want to interact with the type of people who'll only interact with you if you have a mil

[tepid spoke]

If you perceive your gf as expensive, maaaybe reconsider your relationship

[ashen light]

we've already established he wants to be abused so he actually might!

[unique salmon]

Yes

[tepid spoke]

sweet, then I can actually implement my idea

[unique salmon]

Not really

[ashen light]

two mil then, wow

[sick moth]

lol, you can find that with 0 in your bank account

[ashen light]

yeah but its not the sameeeeeeee

[ashen light]

expertium just wants a vtuber gf

[tepid spoke]

which would be to derive that "cut off length" from the learning steps. I.e. last learning step minus learn-ahead time minus max fuzz time would be the cut-off time

I'm interested how doing that affects FSRS quality on the set

[sick moth]

'Rich' these days is ~30MM

[tepid spoke]

I'll look into Benchmarking it

[unique salmon]

Go make an issue so that Jarrett may test it a year later 🤣

[unique salmon]

bruh

[tepid spoke]

I can just implement and test it myself

[sick moth]

I feel I'm missing something here

https://github.com/open-spaced-repetition/fsrs-rs/blob/1c2bd3d9fe83592a49056b14394b4ed299526777/examples/schedule.rs#L86-L90

[unique salmon]

wait what the shit is MM

Millions of millions?

[tepid spoke]

million million

10^12

[ashen light]

lodsofemone

[tepid spoke]

That might be more than all money in the world?

[robust hill]

mm means 1 million

[unique salmon]

Why MM and not M?

[robust hill]

MM in roman numerals is 2000 tho

[sick moth]

When you're working with people who MAY use M as 1,000, MM is unambiguous

It's incorrect, but it's a standard

[unique salmon]

Anyway, 30 millions is not "rich", that's "UBER MEGA GIGA RICH"

[tepid spoke]

I've seen 1k, 1kk and 1kkk used a lot as well

[sick moth]

Private bank minimum rich

[sick moth]

I haven't. FT uses Mn these days I believe

[robust hill]

and what is 1 billion

[tepid spoke]

Russians use chains of k a lot, and there's a lot of russians in EVE Online

[unique salmon]

"Too rich to even gaze upon the plebs"

[robust hill]

elon musk

looks at the plebs

[sick moth]

People can choose not to exist if they have a billion

https://www.mirror.co.uk/news/uk-news/mystery-billionaire-who-doesnt-even-29064059

[unique salmon]

Freakin' americans, man
Here where I live if you make 1000 dollars per month, you are a king 🤣

If you want to make 3000 per month, you only have 3 options:

1. Be a senior software dev
2. Be a striptease dancer
3. Sell drugs
   I'm not even joking

Actually, idk if striptease dancers make that much

[ashen light]

4. win at the stock market 🍃

[robust hill]

how rich are u if you make 1000 rubles a month

[unique salmon]

They definitely make 1000, but idk if it goes as high as 3000

[unique salmon]

You can buy enough groceries to last you 3-6 days

Lel

1000 rubles is like 12 bucks

[robust hill]

yeah exactly

[unique salmon]

Man, now I want to post yearly updates on how my trading strategies are doing just to flex on David 🤣

Maybe next year

[sick moth]

good luck.

[unique salmon]

This one sentence perfectly encapsulates how I feel whenever someone thinks I'm being pessimistic about my romantic prospects

[ashen light]

how disfigured are you

are you like sloth from the goonies (is this too american of a reference?)

[bold terrace]

YOu're not from Germany ?

[unique salmon]

wut

Why did you think I'm from Germany?

[bold terrace]

I think I remember you said it but I might be wrong

[unique salmon]

I have a friend in Germany, I guess

[bold terrace]

Oh Russia !

[tepid spoke]

Can one actually run the benchmark stuff on a GPU?

There is only some side-remarks to it in some issues/prs on the repo

[unique salmon]

You don't need a GPU for FSRS specifically

But yes

For the neural nets you can use a GPU

[tepid spoke]

Well, but does it make it faster? :D

[unique salmon]

No

[tepid spoke]

dang

[unique salmon]

It makes neural nets faster

[tepid spoke]

gonna get out the big guns then, cause I cba to have my PC be loud for an hour :D

Rented a 128 core server

[tepid spoke]

Is there anything else to it other than python3 ./other.py --processes=64 --algo FSRS-5 --short and wait for results?

[unique salmon]

https://forums.ankiweb.net/t/desired-retention-ui-overhaul/57678/32

A new idea

[tepid spoke]

I don't see any info on the learning steps in the dataset btw. :(

there's pretty much only the revlog. Zero deck-settings

[unique salmon]

Where are you looking?

[tepid spoke]

in the dataset itself, the various parquet files

[cursive badge]

I guess the idea was that it shouldn't matter for predicting R 🤷‍♂️

[unique salmon]

Tbh, I don't really know the data processing part of the benchmark. But Jarrett was able to somehow make fractional intervals, so it's possible

You should make an issue

In the benchmark repo

[tepid spoke]

The only way I see to do it, unless I'm missing something, would be to guess the steps from the revlog

which should be possible with reasonable accuracy

ignore "2", reset at every "1", then look at all same-day "3", and you got the learning steps

if there's a 4, ignore the whole card

[cursive badge]

Anki stores the timestamp in seconds when a review takes place (which is reflected in the dataset)

[unique salmon]

@quasi shadow maybe a dumb question, but can't we use the custom scheduling field to set a timer and periodically optimize FSRS that way?
I mean, it's not ideal by any means, but it's better than nothing
EDIT: ok, if this isn't possible, how about adding it to the Helper add-on? So that you can enable AO via the add-on

Assuming the optimizer part is "reachable" via custom scheduling

Maybe it's not, idk

[cursive badge]

I guess a problem would have been that Anki does not store historical Preset settings. The dataset would not be able to tell you if learning steps changed mid-revlog

[unique salmon]

For that matter, are there any guides on writing custom scheduling code?

I don't think I've seen a guide anywhere

[tepid spoke]

There is a very loose documentation

but nothing akin to a guide

it's also very limited in what it can do

[unique salmon]

The crux is whether you can run the optimizer by using the custom scheduling field

So that nerds like us can have AO that way

[tepid spoke]

don't think so, it only has access to a pre-made set of JS functions

it can't just freely run Python code or anything

[unique salmon]

god damn it

[cursive badge]

You could do it as part of an addon.

[tepid spoke]

Yeah, it's probably a few lines of an addon

[cursive badge]

It wouldn't work on mobile though

[unique salmon]

Helper add-on AO?

[tepid spoke]

I think it's so insanely limited so that Apple would allow it

[unique salmon]

There is a problem: fuzz

1. Display a range, like 14-18 days: “Why is there a range of intervals in deck options, but only a single interval in reality?”

2. Just add fuzz to intervals, like ivl + fuzz_factor: “Why do intervals in deck options randomly jump around?”

3. Don’t show fuzz: “Why do intervals in deck options do not jump around, but in reality they do?”

[tepid spoke]

3 seems already to be the case?

I've just implemented a VERY crude approach for now, and will look at result.
Threw in a ("elapsed_seconds", ">=", 7200) filter into the revlogs query

straight up cut out all reviews of the first 2h.

[hasty fractal]

expertium talking about the same thing in two different channels lol

[tepid spoke]

hm, this is kinda weird. Throwing out all reviews of the first 2h leads to A LOT of users having no reviews in their decks

i.e. they never studied a card for more than 2h oO

[unique salmon]

https://forums.ankiweb.net/t/desired-retention-ui-overhaul/57678/39?u=expertium
Poll time (regarding fuzz)

[ashen light]

I think we should all go on strike until @unique salmon delivers a full feature to anki

[sick moth]

Ok, I'm striking. What do I need to do?

[tepid spoke]

hm, the more i look at the 10k dataset, the more dubious I am about it. There's a lot of users which look like they crammed for one day, and then never touched their cards again.

Granted, I only looked at a fraction of the users, since I ain't manually looking at all 10k

But at best half of the random ones I clicked on looked like sensible review data

[sick moth]

Any good metrics to capture outliers?

[ashen light]

heckle @unique salmon till he actually learns to code and implements a feature instead of nothing getting done cause hes the only one who cares

[sick moth]

That sounds really hard. Can't I just be on strike?

[tepid spoke]

The most obvious criteria would be "not a single card was reviewed on another day than the one it started at"

[unique salmon]

For the record, the average number of times a card has been reviewed (across it's entire lifetime) in the 10k dataset is 5.7

Median=4

Unless I somehow screwed up my counting code, but I don't think so

[tepid spoke]

no, that does sound reasonable

[bold terrace]

How crazy can you be to want that

Even the code is not clear enough to understand how the range is computed

[unique salmon]

Maybe you misunderstood. I meant "Don't show fuzz in answer buttons in deck options", not "Don't show fuzz in practice"

[bold terrace]

Ah !

LIke 2+[-1,+1] or something ?

[ashen light]

like "4-6 days" instead of "5 days"

[bold terrace]

Would be neat

Let's make a poll : Disable any kind of fuzz

My filtered deck could rest in peace like that

[ashen light]

don't think dae would ever sign off on that

[bold terrace]

Strange no ?

[ashen light]

hes firmly in the pro-fuzz camp

[bold terrace]

I mean, I don't understand why you'd force it down the throat to people

sure maybe by default

having to disable it with obscure config, sure

[ashen light]

because like 3 people complain about it

[bold terrace]

but being full totalitarian and saying NO is strange

yeah

But that was before FSRS

with FSRS it's way more visible

[tepid spoke]

It's probably the argument of "too many options"

[ashen light]

its visible if you grab a microscope

[bold terrace]

Don't know

I remember the first thing I did when I set my DR to 80% back then was to check what was my retention for that day

Never reached 80%, didn't understood why

I'm sure plenty people have the same frustration

Btw I'm curious how many people enabled FSRS

[unique salmon]

Btw, since I fine-tuned the formulas for the load balancer, maybe you will notice an improvement with the next release

So when the next Anki version comes out, don't use prop: search query stuff and welcome LB into your heart

[bold terrace]

We'll see haha

[tepid spoke]

Did you analyze how long people study on average and in the extremes in the dataset?

[unique salmon]

Then again, according to simulations it only makes retention like 0.3% closer to desired

[tepid spoke]

Cause it feels like most users only studied a relatively short amount of time

[unique salmon]

Or 0.2%

[ashen light]

@bold terrace just accept sub-optimality into your life

[unique salmon]

Not really, I guess
The highest number of reviews is like 4 million

[bold terrace]

Right now my DR is 90% but life has been good

[tepid spoke]

welp, my DB viewer just crashed, cause it exhausted all my RAM

[bold terrace]

avg future retention per day is >90% but my filtered deck make sure no one get shifted

[tepid spoke]

Guess loading all revlogs is not its thing

[unique salmon]

2 million without same-day reviews. That's one guy

Some omega power user

[ashen light]

my drs 87% and I couldn't care about those 3%

[tepid spoke]

over what time-frame? oO

[unique salmon]

Idk

[tepid spoke]

Cause it could just be someone using Anki since a decade or so

[unique salmon]

ID in the 10k dataset: 6810

In case you want to study this specimen, lel

[tepid spoke]

yeah, the file is slightly bigger than most :D

250029476 seconds is the longest elapsed_seconds

[bold terrace]

in the next weeks once I finished the graph I'm doing (The Lapse/Introduced Duration), I'd like to try also neural networks but not for predicting R, but to predict some kind of D for card not known based on cards known

[tepid spoke]

4557 days offset

[bold terrace]

So I could learn them by increasing D

Base on their Kanjis, Readings, Review Time, current D, current S...

[tepid spoke]

Meaning that person has been at it for at least 2893 days

[cursive badge]

My meme is relevant again!

[tepid spoke]

so, that's 8 years of studying in Anki.

[ashen light]

this is completely what I had in mind when I said "pro-fuzz"

[tepid spoke]

I honestly wonder if it's even at all possible to predict a retention curve for anything in the long term

Cause there will inevitably be outside influences that are impossible to predict

[bold terrace]

Also something to remember is that you could also predict shorter interval, being less accurate in prediction, but the user doesn't want necessarly good prediction, he just might want to learn, even if the workload is +20%

That's why when I see the benchmark I'm thinking : Maybe other areas of what make a SRS interesting, might interesting to discover

The one that interest me the most right now is predicting complexity

[unique salmon]

Step 1: Make a super-accurate spaced repetition algo
Step 2: It gives overly long intervals
Step 3: Fudge DR so that it's actually higher than what users see without documenting it anywhere or notifying the user in any way

Alex be like

[tepid spoke]

I'm also wondering if I should set the max interval for my deck to less than 5 years

[polar maple]

no that's me making fun of your idea about purposefully lying about R in order to give shorter intervals, which is what anki does right now

why is it that if FSRS predicts R = 0.5 i can translate it into R = 0.6 for an automatically better prediction most of the time?

[tepid spoke]

even something crazy like 180 days would in theory just be 100 cards per day 🤔

[polar maple]

0.5 FSRS prediction corresponds to 0.6 more often in reality

[unique salmon]

Oh my goooooood duuuuuuuude
How many times are we going to have this conversation...

[polar maple]

and the reason you don't want to fix this is that you don't want longer intervals

[polar maple]

i mean we have a glaring issue about FSRS predictions, why not fix it? I don't get it lol

[cursive badge]

It could be worse. It could be p-zombies again.

[unique salmon]

AND we aren't sure that this isn't some sort of artifact of optimization
AND we aren't sure that it generalizes across different retention levels well, AND we can't test it because we can't just take the revlog of one user and somehow pretend that he has a different retention

Like take a revlog of somebody with 90% overall retention and turn it into 70%

I mean, I guess we kinda can if we delete some specific reviews non-randomly

[polar maple]

yeah but changing decay provably improves generalization on 90% of users, but you are concerned about the possible existence of users where this goes wrong, on an issue that we cannot even test because we don't have people going from 90% DR to 70% DR in the dataset

90% is huge

and we don't even know if the issue actually exists in the first place because we don't have the data..

imaginary issue (which is probably exists btw) but we don't have data for it

[unique salmon]

Actually, maybe we can settle this.
What if we take some users and then delete histories of some cards in a way that maximizes the difference in retention before and after, and benchmark how well parameters optimized at one level of retention generalize to a different level of retention for the same user?

[bold terrace]

90% got from ? You seem to extrapolate your own anecdotes. [Take that back]

[polar maple]

no lol

[unique salmon]

Why not?

I don't see any other way to test it

[polar maple]

similar to what dyzur and sound were doing by splitting into different D subdecks, it just doesn't work that way

[unique salmon]

https://tenor.com/view/why-isnt-it-posssible-its-just-not-american-psycho-gif-26353597

[polar maple]

@bold terrace I made a little transform to fix the calibration curve a bit

@unique salmon read this for some idea as to why

[unique salmon]

Alright
Still, I have a feeling that something about FSRS is fucky, based on how many posts there are about true retention not matching desired retention

And not by 1-2%

And this is the kind of change that doesn't add extra flexibility and just strictly makes intervals longer, so I'm extra sceptical

Changing decay to another fixed value, I mean

I could ask people whose TR doesn't match DR to submit their decks, but I doubt I will get a lot of participants

[unique salmon]

If for most users FSRS underestimates probability of recall, we wouldn't see reports of TR<DR, we would see reports of TR>DR

That's the part that gets to me

I can make a list of several dozens of posts about TR<DR

It shouldn't be like this if FSRS is erring on the underestimation side

And if I'm right and FSRS is bad at generalizing from one level of retention to another, we're kinda screwed

And making the absolute value of decay smaller -> making intervals longer is absolutely not going to help with that

Look at this: https://www.reddit.com/r/Anki/comments/1ip9ov9/why_does_my_true_retention_deviate_from_my/
90% vs 71%
Again, assuming that for most users FSRS underestimates R, how the hell does this happen?

I feel cases like this is what we should be investigating

Like, make it high priority

...ok, maybe not this one

https://www.reddit.com/r/Anki/comments/1ifgwki/fsrs_low_retention_rate_for_mature_cards_help/
This one
96% vs 86%, with optimized parameters

@polar maple

[ashen light]

maybe some people just suck at anki

[unique salmon]

I "cold messaged" 12 Reddit users who reported having low TR and asked them to submit their decks using Jarrett's google forms link

Idk if anything useful will come out of it

I'm kinda worried I will trigger Reddit's spam filters

[tepid spoke]

I got banned by Discord in the past for something like that

[sick moth]

Some things are worth being banned for

[tepid spoke]

Was co-organizing an event, and sent near identical messages about the event to 50+ participants

Asking them to confirm their timeslot and them being set up and ready.

[sick moth]

Is there a limit on username mentions, rather than PMs?

[unique salmon]

I was just leaving comments, not DMing

And I have no clue how Reddit filters work

[tepid spoke]

I am pretty sure if I would fully trust FSRS and not manually patch the parameters, my TR would also be MASSIVELY below DR

[sick moth]

Ah, should be fine, I see so many ReplyGuys these days

[tepid spoke]

Even with quite aggressively patched initial stability, my TR is still not reaching DR, but only by a little bit

[unique salmon]

Tomorrow I'll make a post specifically asking people to submit their decks/collections

People with TR<DR

[tepid spoke]

It'd be cool if there was an opt-in for people on Ankiweb to share their decks for science

[unique salmon]

All decks that are synced to AnkiWeb are available for science

There is actually a tiny clause written in some place that nobody ever looks

[tepid spoke]

But only to the admin :D

[unique salmon]

Well, yes, only to Dae

[tepid spoke]

I still don't know what I did to get my deck into the state it's in.

If I Optimize Parameters, it does it twice for some reason. And then still gives too high initial stabilities for anything but Again

[unique salmon]

I'm surprised that you are surprised

Double optimization has been around for a few months

[tepid spoke]

I have not hit that button for a few months.

[unique salmon]

Maybe we need to pin that one message

Then again, nobody reads pins

[tepid spoke]

My parameters right now largely work, so I ain't touching them

The new ones that come out are also nearly identical except for the initial stabilities anyway

[unique salmon]

Basically, it's to prevent S from growing during same-day reviews too much

Right now the last two params are just set to 0 in some cases, with the next update they will have dynamic limits instead

Their max values will be adjusted

[tepid spoke]

yeah, I end up with 1.9829, 11.0519, 15.8118, 21.4527, 6.0335, 0.8959, 3.8060, 0.0010, 1.4089, 0.2178, 1.0391, 2.2297, 0.0511, 0.4080, 2.8324, 0.0002, 3.0366, 0.0000, 0.0000

[unique salmon]

DerIshmaelite will finally be able to sleep at night

[tepid spoke]

I'm still baffled by those numbers lol

What was it again? Several thousand reviews a day?

[unique salmon]

Yeah, he does like 2k reviews at 2 seconds/review or something

Tbf, I probably wouldn't have asked Jarrett to improve how the last two params are handled if not for DerIshmaelite

[tepid spoke]

What do they even control?

[unique salmon]

So he indirectly contributed to making FSRS better via complaining, lol

[tepid spoke]

They're both just labeled "short-term stability (exponent)"

[unique salmon]

Increase of S during same-day reviews

[tepid spoke]

hm, that should mostly be a non-issue for me I think

[unique salmon]

Maybe we need more people complaining to figure out which aspects of FSRS to improve

[tepid spoke]

I wish I could clearly pinpoint what the issue is. Right now I can only confidently say that something is off

And make a few guesses of course

[polar maple]

bad assumption, people who have FSRS overestimating R are less likely to make reddit posts complaining about the issue

or is it the other way around, idk idc

either way there is bias going on if you only look at reddit posts

[unique salmon]

It's literally the same assumption that underlies your transformation

[polar maple]

then it's the other way around

[unique salmon]

That in average FSRS under- and not overestimates R

[polar maple]

and look i don't care about n = 1 anecdotes that much so i'm not even gonna read that post

even in jarrett's FSRS 6 candidate I can likely find users in the 10k that does worse

[polar maple]

anyways my original point with this is that you want to purposefully keep a forgetting curve that can easily be 'corrected' with a translation

so i suggest if you want the original behavior then we can fix the curve still, but un-do the translation when scheduling

in the perfect world we would just have a new better forgetting curve and don't need this mess

its just boring to work on FSRS if the predictor model is so coupled with the scheduler

I just want a good memory model, that's all

I'm sure that w[1] is also correlated with average retention in the 10k set yet we're fine with it being a learnable value

its just strange that our standard for decay is so much higher than for every other parameter

[clever cargo]

https://github.com/kuroahna/anki_srs_kai is a good example

[quasi shadow]

Fine. I will change the decay to 0.2 in FSRS-6. https://github.com/open-spaced-repetition/fsrs-optimizer/pull/169

And have a retirement

[quasi shadow]

Practice is the Sole Criterion for Testing Truth

Just let's use DECAY=-0.2 in Anki's next release and see what would happen.

[quasi shadow]

😎 Nobody could complain that the improvement of FSRS-6 is minor now.

[lapis hearth]

= )

I just want the stability to be lowered to 0.001 or have FSRS-sec. Could I complain about that @polar maple

[bold terrace]

Nothing ground breaking but still :

• We saw Higher D comes (in general) from Higher Lapses
• Repetitions and Lapse, also have a high correlation (which is to be expected if FSRS does its job).

Thus, detection a leech can be as simple as looking at cards with high repetitions number.

I mean, it might sound so obvious that it's not worth saying, but since we were a bit tinkering about detecting leeches in a smart way, it's good to remember that there are very very straightforward and dumb way to find those.

It's even reinforced by the fact that the average stability by reps is depressingly decreasing

Meaning, doing a lot of reps is not something that will help you remember it, it's more a marker that show you you can't remember it

Thus, you don't remember something better by only SRS-ing it. Or maybe "SRS-ing things works better for things that are ready to be SRS-ed"

[lapis hearth]

I feel outed here, I am in bad need of this. I argue an in-review leech detector is as helpful as making the short tem scheduler go lower than 13 min

The problem is that I know that I have leech cards which i try to brute-force memorize, the question is how do I objectively identify at what point is a card considered leech

It would have to be a dynamic threshold

But whether leech cards exist, they definitely do exist, and I have a lot of prominent examples

---

Not only would the leech detector itself be helpful but also some indicator which shows whether the "leechiness" of the leech card is increasing or decreasing

[bold terrace]

I plotted the repetition/lapse average load (graph in the middle), it's also a good way to see that the average load by card is increasing with reps/lapses

If someone wants to check for themselves

Ofc, since the distribution is decreasing more violently than the average, the total load is fortunately also decreasing, but still, it shows that basically, you can suspend cards at a certain lapse/repetition and give you an insight of how many cards you would lose vs how many load you regain

[cursive badge]

I feel like if you need an interval less than 13mins you are probably outside of the realm of SRS being useful.
You probably need to try learning it in another way outside of Anki to make it stick enough for SRS to make sense.

[lapis hearth]

I dont know what to say

I keep repeating myself

[bold terrace]

By sacrificing 16% of my cards, I could for example regain 32% of my load, based on a threshold at reps=34
And by cutting at threshold lapse >=6, then it's at 15.6% sacrifice for 31%.

Basically, using reps or threshold to know what to cut, won't change how the cut will be made 🙂

[lapis hearth]

This is what my steps stats look like

[bold terrace]

Trying to convincing us you can't do without Anki won't solve your problem you know... We really say this to try to help you, but you have to try to help you, yourselves

You're spamming reviews every 2sec, of course your stats will be completely fucked up

Sometimes I feel you're waiting for someone else to fix your issues

[lapis hearth]

I am not spamming just for the sake of it. I am capable of recalling fast enough

[bold terrace]

If you can't remember it 10min later, yeah, you can recall something you saw 5sec earlier, guess what, everyone can. Fix your problem instead of hiding behind "I can recall fast what I saw 5sec earlier"

[lapis hearth]

My brain works definitely it seems

I am not abusing the again button. If I fail to recall something, I just press again

If I do succeed in recalling, I press good

And I am very strict about it. No half-assed recalling.

[bold terrace]

Listen, it seem you made your idea, so I'll let you advocate for minimum intervals of 3sec, but don't complain you'll keep repeating yourself to everyone, because if everyone tell you something, there's probably a bit of truth in it

[robust hill]

can i have thiss

this graph pleasee

i want ittt

:)

[bold terrace]

It's not ready to merge but it's a local build with it

So you can play around with it

[robust hill]

love you so much

[bold terrace]

I'll improve a bit the tooltips etc before opening a PR for @cosmic hedge , but I also see some test failing so not sure if it can be merged quickly

[robust hill]

wow i did not see this

💀

[bold terrace]

I think that one had a few bugs so take the last one

[cursive badge]

I'm sorry if you have explained your reasoning earlier and I've forgotten/missed it.
I know FSRS helper is giving you those steps, but sub 30s is firmly within short-term memory.
Repeatedly forgetting at 13m intervals suggests that it never got properly encoded into long-term memory to begin with.
SRS is for reinforcing long-term memory, it cannot help you if you card never even made it to long-term memory to begin with.
That's why I'm saying it sounds like you need to change something outside of Anki.

[robust hill]

hey where do i find avg reps per lapse

ah

[lapis hearth]

No, sometimes people dont realize stuff. You do realize that different things have different complexities. There is stuff that is just hard to remember, there is no way about it other than just repeating it over and over again. Something like a god-awful pharmacological, anatomical or pathological name could get as dry a 10 year old bread crumb under your bed. These things you just forget right away and my brain has just trouble with such things. This is a problem a lot of medical students suffer from, and when you have oral examinations where recalling things have to be perfect, you have to be a perfectionist.

SO it is inevitable you press again a lot of times, and this problem is further aggravated by brain's inability to be at basic human level of intelligence.

[robust hill]

@lapis hearth when you are doing anki

do you do anything else

or just look at the card and press the button

and thats it

[lapis hearth]

Yes I wrote a message earlier about how I am afraid that this is a problem I am overseeing. The problem is that I have already tried. I am already 3 years in this mess.

[lapis hearth]

No. I read the question, reset the timer and then try to recall the answer within 4-5 seconds, if it gets too long, then i press again

@bold terrace this is the reason why my time seems a bit off

I have the LifeDrain addon and the toggle back and front addon, which resets the answer time. I only want Anki to count the time I spend on trying to answer - actively trying to recall the answer.

So when Anki says I spent 5-6 hours Anki time, I am in fact spending much more than that

[robust hill]

okay so

what if

are you able to speak when you do your reviews?

speak out what you are doing when you do your reviews + write it down or atleast at the bareminimum the answer

this is what i do and it makes my retention in young cards jump up

[lapis hearth]

This is how I do it

1-Read the Question
2-Understand it
3-Press reset answer time to start over
4-try to recall the answer
5-SPEAK the answer by moving the tongue, doesnt have to be audible
6-I press reveal answer
7-Rate the card (and pressing again if there is any slight deviance)

My long term retention is actually doing great!

I am quite satisfied with it

[cursive badge]

@bold terrace On the topic of graphs we have not shared with Luc yet.
What if Future Due, but heatmap? 😮

[robust hill]

okay so

#5 is good

now try with writing

[lapis hearth]

My DR for this deck is 95%

[robust hill]

it will help

wow 300k cards is astonishing work

good job

[lapis hearth]

This is a deck which i started 6 months ago (post FSRS 5 release)

So I am more or less satisfied with my long term recall.

The problem is short term recall makes me cry blood

It is a pain in the ass trying to learn with Anki when you are trying to learn the driest of things

It is about making the short term learning easier for me, not making long term recall better. I am already good in that department

My mature cards perform better than young cards

Which is quite weird

But I am not complaining.

[robust hill]

I told you the solution

now do it

[bold terrace]

Scroll a bit ?

[robust hill]

i see

rep load rep avg load rep distribution rep total

and no Avg Repetitions per Lapse

surely im not stupid or blind

could be

[lapis hearth]

Brother, I have tried all sorts of ways, ways which you haven't even heard of.

I know where my problem is and it is that I am asking Anki to do something which it cannot do (yet, hopefully). My learning is completely encapsulated inside Anki for convenience, but it is a program not originally designated for learning and encoding stuff - i am just forcing it to be, and while it is not doing a horrendous job at it, it is certainly not optimal either.

[bold terrace]

Quite nice ! But the problem is I think it won’t change , vary much right ?

[bold terrace]

Aah no this one I messed up a bit I need to redo it later

Sorry

[robust hill]

noooo

please i need it

that is the one i need

thats going to solve my lifes problems

[cursive badge]

It will change very slowly as the deck gets older and you start to see more stable mature cards.
It might be interesting to have a log-log graph so it can show all time, not just the next month.
I just did it for fun, I don't know if I will ever clean it up and make a PR.

[robust hill]

my girl thinks im crazy

i have tried to put her on anki

apparently i was sleep talking about the lapse review ratio

[cursive badge]

Oh no, too much Anki for dyzur.

It's invading their dreams ;p

[robust hill]

yk

if i spent more time studying and less time trying to optimize

maybe id have better retention

[unique salmon]

Alright, since this horrendous change made it into FSRS and we need to make sure people actually use FSRS and don't run away from long intervals, how about increasing minimum DR to 80%?

I looked at the graph of R and with the new decay intervals at 75-77% DR are about as long as they were before at 70%

I still expect that complaints about FSRS being inaccurate will skyrocket, but at least the longest intervals people can have will remain more or less the same

Oh, and @quasi shadow please do the things I recommended in the neural D pull request.
You don't want to do everything? Fine. But at least do two things so I can benchmark it on my own

https://github.com/open-spaced-repetition/srs-benchmark/pull/199#issuecomment-2779315938

[quasi shadow]

I will do it in the next week.

[unique salmon]

Ok. Please also change minimum DR in Anki

And in CMRR, etc.

To either 75% or even better - to 80%

[unique salmon]

https://github.com/ankitects/anki/pull/3898
I made a PR for it

Why does ci fail...
I literally just changed a few constants 😭

Oh, it's not even related to my changes, it's just something about unmaintained dependencies

...I think

Man, I can't even make a PR where I change A FEW CONSTANTS without something failing

https://tenor.com/view/the-simpsons-homer-simpson-bake-baking-cook-gif-4710706

[bold terrace]

I'm tryin to make a nice visualization to show load in ratio and cumulative. Making progress but now I'll apply it to avg(load)/card to find more easily a cut out point where it's too much workload compared to the amount of card

[bold terrace]

Got it 🙂

In non cumulative

[bold terrace]

Quite useful ! I loaded my deck from ~2 weeks ago, without D split, and without any suspending. You can see in "Repetition Average Load" that around ~70 reps in. So I can quickly with the Pie, see that sacrificing the 3.5% cards (~131 cards over 3600) with the higher reps, would lead me to a reduction of 10% load (~26reps per day saved)

[unique salmon]

https://github.com/ankitects/anki/pull/3898#issuecomment-2780733883
I summarized the new curve change here

[severe storm]

What addon is this?

[unique salmon]

FSRS Helper

[unique salmon]

where rwkv

[polar maple]

when i feel like cleaning up my code

@unique salmon remember that the shape of the forgetting curve is a statement of how R changes when you go outside of the region near DR, so it's not right to say that most people will experience increased intervals since optimization would still just stretch the curve to still have the right DR

for people who do anki on time all the time, their intervals should be the same as before

[unique salmon]

If you mean my Github comment, I meant "for the same S"

Like, if we hold S fixed

[polar maple]

yep. anyways if you really care about scheduling you can just revert the change when scheduling only

if you want to lie to your users, sure go for it

[unique salmon]

Nah. But I hope Jarrett and Dae won't object to raising minimum DR

[polar maple]

@unique salmon much more messy here but here is log(w[3]) vs x

they still seem to be correlated even if not perfectly linear

what makes w[3] different than decay?

if w[3] instead of log(w[3]) on fewer data points

so why is correlation not a huge problem in all of these parameters

i'd expect every parameter is correlated with R based on the fact that the 10k set is generated with SM-2

the resulting R is based on the difficulty distribution of the data

and even if the 10k set was generated with a certain DR, the difficulty distribution of the data could still affect what DR the user would select

[unique salmon]

I want to raise minimum DR purely to make sure that people don't complain that hard

About intervals lengths, I mean

We (probably) can't prevent people from complaining about TR<DR, but at least we can prevent people from complaining about intervals

Kinda

Hopefully

https://www.reddit.com/r/Anki/comments/1jrqbey/anki_for_the_mnemonically_challenged/
btw, here's a post where a person literally admits that the only reason he didn't use Anki before was that the cards weren't being shown often enough

And another guy says he has two copies of each card just to see them twice as often

...honestly, this kind of stuff makes me wonder "What's even the point?"

Like, of ever recommending Anki to anyone, or putting any amount of effort into spreading the word about Anki

Anyway, putting algorithmic accuracy aside, making intervals shorter would be good because users care about it WAY more than about how good the algorithm is theoretically

To the point that some people straight up don't use Anki because of that

[polar maple]

idk maybe this has to do more with the default DR rather than minimum DR, i'd expect users who set DR to a very low value should know what they're getting themselves into

[lapis hearth]

I was about to do this

[lapis hearth]

Perhaps you are referring to me, I want to have the smallest interval scheduled be less than 13 min 😅

I mean seriously this is a drug name I have to memorize, Aminoimidazolecarboxamido­ribonucleotid

[severe storm]

Thanks, but this doesn't work 🤔

[lapis hearth]

Use shift t to get the stats

U have to enable Steps stats in the configuration first

[severe storm]

I already did that, it shows me everything but those steps table

[lapis hearth]

.

[severe storm]

ohhh

mb cuh

[lapis hearth]

Dont worry fellow brainrot

I have been there

[severe storm]

Ayy thanks mate, it works now (also thank you expertium 🙏 ) I dont think I understand any of it, but it is nice to have.

[unique salmon]

I've proposed moving default DR to 95%: https://forums.ankiweb.net/t/thoughts-on-changing-desired-retention-to-95/58273?u=expertium

[unique salmon]

Just look only at the bottom part

[severe storm]

why are you censoring your data haha , But I am fine with my current (re)learning steps (default one's)

[unique salmon]

It's not mine, it's DerIshmaelite's. And it's to illustrate my "just look at the bottom part" point

[severe storm]

ah okay

[sick moth]

Note: Anki problem, not a FSRS problem

[bold terrace]

LOL, I did the %load/%card for each repetition range, and how silly I looked when I realized it gives the same curve than "Avg Load by Card" basically

So good news @cosmic hedge , only +4 graphs instead of +6

😄

+2 instead of +4 sorry

[cosmic hedge]

Hey the new graph count seems to be decreasing by the second! 😄

[cosmic hedge]

I'll make the categories collapsable at some point if im able, that way we can add all the obscure stats we could ever want.

[robust hill]

why is ur color blue

mines auburn

is this concerning btw

[quasi shadow]

https://github.com/open-spaced-repetition/fitting-forgetting-curve/blob/main/analysis.ipynb

I fit three kind of forgetting curve on first 100 collections of the 10K dataset and analyzed the parameters.

It's interesting that the distribution of decay is not normal.

It's even not unimodal.

It has two peaks.

@polar maple you would be interested in that.

[polar maple]

Good work, i'll take a closer look later

[polar maple]

btw what if you remove the [-1, 0] clip on decay?

[quasi shadow]

let me try.

/Users/jarrettye/Codes/open-spaced-repetition/fitting-forgetting-curve/script.py:79: RuntimeWarning: invalid value encountered in power
return (1 + factor * t / s) ** decay

😂

ok, the decay should be negative at least

now the code runs well

wtf?

The new distribution after I set the clip [-2, -0.01]

[unique salmon]

@quasi shadow I give up, I have no clue how I am going to combine revlogs. The code from pretrain.py doesn't tell me how to

1. Take data from ...\anki-revlogs-10k\decks and ...\anki-revlogs-10k\revlogs
2. Save them as .parquet files
   It doesn't even use anything from decks as far as I can tell

So I'm not even sure what purpose ...\anki-revlogs-10k\decks serves

Other than making my life harder

[unique salmon]

I've tried a bunch of solutions made by Claude 3.7 and whenever I try to run FSRS on the combined revlog, I get pyarrow.lib.ArrowTypeError: Unable to merge: Field user_id has incompatible types: int64 vs dictionary<values=int32, indices=int32, ordered=0>. Got it three times with three different solutions

https://tenor.com/view/pepe-gif-8493382194019209370

Screw it, I'm just gonna use The Big Dude with 2 mil reviews

AttributeError: 'Namespace' object has no attribute 'model'
Never works on the first try, man. Not even once 🤣
It's actually hilarious. I'm not even exaggerating at this point, I actually, genuinely have never managed to make the benchmark code run on the first try
Time to make a new folder, copy-paste all the benchmark code there and don't forget to update the optimizer code as well

Here comes my fifth benchmark folder, yay...

No, wait, my 6th benchmark folder

[cosmic hedge]

if i understand correctly and you just want to load several users at once you can just change the filters

df = pd.read_parquet(
    "../anki-revlogs-10k/revlogs", filters=[("user_id", "<", 150), # Note the "less than" we're loading 149 users here
)

or i think you can do

df = pd.read_parquet(
    "../anki-revlogs-10k/revlogs", filters=[("user_id", "in", ["1", "5", "25"]),
)

Example from me:
https://github.com/Luc-Mcgrady/anki-10k-notebooks/blob/master/card-average-duration.ipynb

[unique salmon]

I need to take .parquet revlog files from several users and combine them into one big .parquet file

Idk about deck data

It doesn't seem like it's needed (?)

pretrain.py doesn't do anything with deck data, it seems

Card data from anki-revlogs-10k\cards is 100% not needed

Huh. I occasionally thought "I wonder if having a name like FSRS when there is already an abbreviation SRS is confusing", but this is the first time I see someone actually being confused

[cosmic hedge]

why do you need to do that when you could load them selectively from the full dataset?

[unique salmon]

I need the neural net for D to have only one set of parameters, rather than different parameters for different users

[cosmic hedge]

this does load them as one big block

like as one user i guess (they still have different user_id's)

[bold terrace]

Got me checking the meaning of FSRS

Free Spaced Repetition Scheduler

I would have though "System" instead of "Scheduler"

And "Free" is also quite an interesting qualifier. Sure it's free, but like other alternatives I guess

[unique salmon]

Also, I've seen people butchering the abbreviation to FRSS, FSFR, FRS, etc.

Now I want to go back in time and discuss the name with Jarrett 🤣

I feel like it has to either be an abbreviation that consists of only 2 letters, to make it easy to remember, or some short word instead of an abbreviation

[robust hill]

sound

if i got 1750 cards and the exams in 60 days

and i would like to pass through all of them in 40 days, so 20 before the exam

should i start with like 80% retention and once theyre all not new, start to up it?

[bold terrace]

How much time do you have per day 😄 ?

For that particular exam

What I mean is : You could start right away with 90%, because in the end it depends on how much time you have per day for this 🙂

With interval of ~4d, 1750 cards means ~437 review/day for example. If you have only 30min/day, and cards take in avg 10 sec/card, it means you have a max workload of ~180 review/day.

So it depends

• how easy those intervals could be bigger (so material specific)
• how well you already have acquired that material (through your class)
• how much time/day you have for this specific exam (so cursus/side jobs/errands dependent)

IMO, if it's a class you're attending right now, I'd after each lesson add as new all cards related to that lesson

[quasi shadow]

I don't know why you want to combine them.

[bold terrace]

And if it's too difficult time-wise, drop DR only then

[quasi shadow]

So I don't know how to code it.

Do you need the tensor column?

The benchmark data-processing cannot deal with multi-user dataset.

[unique salmon]

So that the neural net learns from multiple users

[unique salmon]

Then how does pretrain work?

I just want to take a bunch of .parquet files from different users and smash them together into one .parquet file

[quasi shadow]

It reads the collection one by one, and extract the feature for each one. Then it concats all processed dataset into one.

[unique salmon]

Can you do that and then save it as .parquet and send me? 👀

[quasi shadow]

The .parquet file cannot store tensor.

[unique salmon]

Tbf, it's probably fine to run it on only one user

So just to be clear, I can't combine .parquet files of different users?

[quasi shadow]

Yep.

Otherwise, the combined dataset will have duplicated card_id.

The review histories will be mixed.

[unique salmon]

Meanwhile the NN is learning really well

This is new_d

[quasi shadow]

Should it be 10?

[unique salmon]

It should be something inbetween that isn't just the same value repeated over and over 🤣

[quasi shadow]

In my previous test, the d output of the NN is ~9.

[unique salmon]

Idk, maybe this is just the first epoch, so it hasn't updated params yet

Actually, no, even then there should be a difference after the first few batches

I'll try this instead:
self.difficulty_nn = nn.Sequential( nn.Linear(3, self.hidden_dim), # Input: retrievability, difficulty, rating nn.Mish(), nn.Linear(self.hidden_dim, self.hidden_dim), nn.Mish(), nn.Linear(self.hidden_dim, 1), nn.Sigmoid() # Output: new difficulty )
Without clamping

[quasi shadow]

Sigmoid?

[unique salmon]

Yep

Maybe it magically fixes the problem

[quasi shadow]

Then the output will always fall into (-1, 1)

But the range of D should be 1~10.

[unique salmon]

0 to 1. And that's not an issue anyway, I'm using a new function for D

For extra flexibility

I mean, in the main SInc formula

[quasi shadow]

It's an issue because the SInc is based on D.

[unique salmon]

I changed that part

[hasty fractal]

these people don't know SRS

[quasi shadow]

So it's not merely neural D.

[unique salmon]

Yeah, it's neural D + new f(D) in SInc

I always thought that 11-D is too inflexible, but trying different functions didn't help

welp, nevermind

https://tenor.com/view/finding-nemo-bags-floating-stuck-now-what-gif-5473087

No sigmoid - everything is always 1
Sigmoid - gradients die for some reason

[quasi shadow]

I guess it's cause by S_fail

D should not be negative.

[unique salmon]

Oh, yeah, I'm dumb. I forgot about PLS

[unique salmon]

It's not negative. But yeah, if it's 0, this is likely the cause of nans

Now every value of D is around 0.5. Progress! 🤣

[polar maple]

what are you using now?

[unique salmon]

transformed_d = torch.pow(1 - torch.pow(normalized_d, self.w[19]), self.w[20]) # (1 - norm_d ^ w[19]) ^ w[20]
normalized_d is the output of
self.difficulty_nn = nn.Sequential( nn.Linear(3, self.hidden_dim), # Input: retrievability, difficulty, rating nn.Mish(), nn.Linear(self.hidden_dim, self.hidden_dim), nn.Mish(), nn.Linear(self.hidden_dim, 1), nn.Sigmoid() # Output: new difficulty )

New S after Hard/Good/Easy
new_s = state[:, 0] * ( 1 + torch.exp(self.w[8]) * transformed_d * torch.pow(state[:, 0], -self.w[9]) * (torch.exp((1 - r) * self.w[10]) - 1) * hard_penalty * easy_bonus )

PLS
new_s = ( self.w[11] * transformed_d * (torch.pow(old_s + 1, self.w[13]) - 1) * torch.exp((1 - r) * self.w[14]) )

[robust hill]

i dont know

i have to do other things too

[bold terrace]

Hmmm so I'd say :
Start with the usual DR you'd like to have (90 for ex). Add the normal amount of new card/day you want to do to be sure you cover everything ~20d before the test, and if the workload becomes too hard, either put more time on it, either drop DR at that moment

That's how I'd do

[robust hill]

alr

and what do you think

is the most optimal dr

for its time x workload

90?

[bold terrace]

Optimal is a word we should treat with carefully haha

I'd stick to a healthy DR of 90% and cruise with it

I think 80% can be healthy too, but below that I think it's start to be a bit muddy

[unique salmon]

May Compute Minimum Recommended Retention be with you

[bold terrace]

Also, I don't think Anki is not the most* important thing in your study 😄

[unique salmon]

Actually, on second thought, if you are preparing for an exam, CMRR doesn't matter much, since you are probably much more worried about memorizing as much as possible than about efficiency

[robust hill]

Yeah but also

Don’t wanna get to a point where I memorize and lose it

[robust hill]

Indeed it is

I learn everything with anki

[bold terrace]

The most important part of my sentence was probably the not lol

Depends also on what you study I guess

It'd be fun to see someone learning programming with Anki

"What is a function ?"
"What does if do ?"
"What does a for loop do ?"

Good, now go debug 20+ year legacy app

But yeah 🙂 Don't overthink it, keep your good old 90%

You already have your parameters optimized for that one 🤷‍♂️ Better rely on good prediction than a bad forgetting curve 👼

[robust hill]

i study everything

when i make a new deck with new subject/forms of content

i should NOT clone it right and optimize the old fsrs from some other deck to the new deck

right

[robust hill]

medical school, at the moment
biocheimstry, anatomy, physiology, microbiology

[polar maple]

I think the sizes of the dfs are just too small which is why the most common decay is one at either extreme [-1, -0.05]. For example this decay = -1 isn't insightful for this small sample

[quasi shadow]

But not all sample sizes are too small.

[polar maple]

not all but it seems that most of them are if you print them out

I think it needs a filters like the number of unique elapsed days is at least 4 and the average y cannot be above 0.995

also len(df) > 50

[quasi shadow]

like this?

[polar maple]

yeah that's a start

[polar maple]

but I think there will still be many samples where all the values are in just one day and the remaining days have little data

[quasi shadow]

It passed the filter.

[polar maple]

yeah I mean this doesn't mean that it's enough data

we need stricter filters or relax the constraint that all the r_hist is the same

bc while r_hist does make it more equal it reduces the sample size by a lot probably

and you are still throwing out information like elapsed_seconds either way

[quasi shadow]

but if the r_hist is different, it means the user has different learning steps.

it also affects the forgetting curve

I pushed the code.

It doesn't change the distribution.

[polar maple]

but you don't know elapsed_seconds so it's still an approximation, it might be a price to pay to get a larger sample size

anyways I'll revisit a sum of weighted exponential curves in rwkv

[quasi shadow]

weighted_loss_exp: 0.38323390507506705
weighted_loss_pow: 0.3552603130824736
weighted_loss_exp_with_intercept: 0.35515668780535625

Notice that the exp forgetting curve with intercept also performs well.

[polar maple]

yea on GRU I tried something like that but it wasn't very good

but who knows maybe with enough curves a nn will figure it out

gru is a tiny model after all

[quasi shadow]

😂 The workload graph with flat forgetting curve.

@unique salmon If we clamp the minimum recommend retention in [0.8, 0.99], the output will always be 0.8.

[robust hill]

is there a way to see

workload and desired retention

by my own parameters

like can i make my own graph

[quasi shadow]

https://github.com/open-spaced-repetition/fsrs-optimizer/blob/main/src/fsrs_optimizer/fsrs_simulator.py

run this script

[robust hill]

where

must i change anything?

[quasi shadow]

Oops, I forget that it doesn't support personal use...

So you need add some code for yourself.

[robust hill]

😭

damn

its alright nvm then i dont know how to code anymore

[unique salmon]

Welp...
Idk man, I'm still in favor of raising minimum DR (to 75% btw, not 80%). Plus, maybe for some users it's different

[quasi shadow]

https://github.com/open-spaced-repetition/fsrs4anki/blob/main/fsrs4anki_optimizer.ipynb

I forget that it included the workload graph😅

[sick moth]

Please no. I use 0.72

[unique salmon]

@quasi shadow https://github.com/ankitects/anki/pull/3898
I'd like you to

1. Approve/disapprove this change
2. Check the code changes and help me make ci happy (idk why checks fail, I don't think it's related to my changes)

[hasty fractal]

@unique salmon can u update me on what happened? how did we come to FSRS-6 from "FSRS-5.5 will be the last version".

well good thing this happened, now we dont need to say "we use FSRS-5.5" in the manual.

That sounds weird.

[hasty fractal]

ah, okay so there is no 5.5 now. great.

[unique salmon]

It's...a long story

[lapis hearth]

Jarrett is adding a new param which makes the coming FSRS incompatible with Anki in its current state (I am not sure if that is the exact reason). He is also making a drastic change in something which makes a significant change in RMSE (i am also not sure)

[unique salmon]

So originally Jarrett added one new parameter and wanted to call the new version FSRS-5.5. Then he decided to call it FSRS-6 because adding a new parameter makes it incompatible with previous versions, hence it counts as a new major version rather than a slightly changed previous version. Then he also changed the shape of the forgetting curve, making it even flatter than before.

[lapis hearth]

When will we have some preliminary version of an actual leech detector

[unique salmon]

https://tenor.com/view/troll-trollge-reverse-troll-sad-troll-reverse-gif-19858090

[lapis hearth]

Why are you acting sus all of the sudden

[unique salmon]

Whenever someone decides to make a fully functioning leech detector. Which is not going to be Jarrett, he doesn't want to work on the Helper add-on anymore. And it won't be @cursive badge either, he just made a prototype.
So..I'd say 2045 is a good estimate 🤣

By that time we will have AI advanced enough that you can just ask it to code whatever you want, lol

Reminder that Easy Days could've been implemented natively, like, 7 or so years ago

Or whenever the original Easy Days add-on was made

It's not like there were any technical difficulties with implementing it natively

We don't get new features when it becomes technically feasible. We get new features when one random guy finally decides to implement them.

[lapis hearth]

I am asking this because I also suspect a weird behavior FSRS is doing. Ever since FSRS 5 has been out, my true retention has been on average around 95%, my DR is 95% which is really great.

However, I am noticing that my performance on my cards which I deem to be leechy (cards that I seem to forget everytime I see the next day after having thought that I finally got them inside my head the day before) has more or less not changed.

I am suspecting that FSRS is overcompensating for those leechy cards by scheduling easier cards so that the true retention percentage remains around 95%

So if I want to know for sure, I want to see how am I performing on my leech cards today

Which I have no way of doing

[unique salmon]

Heck, FSRS itself could've been invented ages ago, it's just that due to slower hardware optimization and benchmarking would be a pain

If FSRS was developed in, like, 2006

[lapis hearth]

So for example If you have 100 cards in total, 5 of which are leech and you inevitably fail them again, you have 95% true retention.

Eventually you run out of easy cards to review (say you have 20 cards to review now) and you end up with the same 5 leech cards, now you have 75% true retention.

FSRS counteracts this by scheduling easy cards as "reviewing fodder" - filler cards that just inflate your true retention to make it seem that your true retention is as high as you want.

And that your performance on the leech cards hasnt changed

Do you get what I am saying❓

Or am I speculating too much❓ Because why does it seem to me that despite my true retention performance my performance on my leech cards is as abysmal as it ever was

[unique salmon]

No, I get it. I think it's possible

[lapis hearth]

So how do we know for sure. Is there a way❓

@quasi shadow

Sorry for pinging you

[quasi shadow]

I have no idea.

[ashen light]

imagine it was you who made the new leech detector

and I thought you and jarrett were specing something out to show to dae?

[unique salmon]

I thought Jarrett wants to make it a feature in the Helper add-on, but then he said nope

[ashen light]

rip

well write down a spec and I'll totally get to it

[unique salmon]

Do you want a long Word document? 😅

[ashen light]

sure

why not

[unique salmon]

Alright, I'll make it later

[ashen light]

something to convince dae it is even a reasonable feature

[unique salmon]

I really wish it was in an add-on first though, to make testing easier

So that we can test it before implementing it in Anki

[ashen light]

can just change it

[ashen light]

whats a .docx

[unique salmon]

It's like .doc, but with an extra x

[ashen light]

sorry I only understand plain text

[cosmic hedge]

I already added his leech detector to SSE.
it worked I thought? https://forums.ankiweb.net/t/automated-leech-detection/56887/11?u=a_blokee

[unique salmon]

"the less likely the cards are to have that number of leeches"
You probably meant lapses

[cosmic hedge]

oh yeah XD

[unique salmon]

Wanna implement it in Anki based on my spec above?

[cosmic hedge]

not particulaly XD

[unique salmon]

Somehow I'm not surprised 🤣

I really hope these words won't end up being prophetic...

[ashen light]

re: poisson_binomial_pmf is that a common enough function that it'll be in some rust library somewhere

[unique salmon]

I doubt it

Then again, it shouldn't be super difficult to code
Here, have some AI slop:

fn fast_poisson_binomial_pmf(p: &[f64]) -> Vec<f64> {
    // Validate input
    for &prob in p {
        if prob < 0.0 || prob > 1.0 {
            panic!("All probabilities must be between 0 and 1");
        }
    }
    
    let n = p.len();
    
    // Handle trivial case
    if n == 0 {
        return vec![1.0];
    }
    
    // Initialize the PMF - we'll use a dynamic programming approach
    // pmf[j] will represent P(X = j) after considering the first i trials
    let mut pmf = vec![0.0; n + 1];
    pmf[0] = 1.0;  // Base case: probability of 0 successes with 0 trials is 1
    
    // Process each probability one at a time
    for &prob in p {
        // For each new Bernoulli trial, we update the entire PMF
        // We do this in reverse order to avoid overwriting values we still need
        for j in (1..=n).rev() {
            pmf[j] = pmf[j] * (1.0 - prob) + pmf[j-1] * prob;
        }
        // Update the probability of 0 successes
        pmf[0] *= 1.0 - prob;
    }
    
    pmf
}

fn main() {
    // Example usage
    let probs = vec![0.1, 0.3, 0.7, 0.9];
    
    // Calculate full PMF
    let pmf = fast_poisson_binomial_pmf(&probs);
    
    println!("Full PMF:");
    for (k, &prob) in pmf.iter().enumerate() {
        println!("P(X = {}) = {:.6}", k, prob);
    }
    
    // Verify sum of probabilities is 1
    let sum: f64 = pmf.iter().sum();
    println!("\nSum of all probabilities: {:.10}", sum);
    
    // Get specific value from the PMF
    let k = 2;
    println!("\nP(X = {}) = {:.6}", k, pmf[k]);
}

[tepid spoke]

you want tripple-ticks :D

[unique salmon]

?

[tepid spoke]

For that code

[unique salmon]

Ah, yeah, I keep forgetting that's a thing

[robust hill]

im going to try

in the easter break

but probably it will be chatgpt coded too 💀

is this normal

[unique salmon]

Nah, ain't no way you can make an LLM write a PR to Anki that adds a bunch of new functionality and it actually succeeds with 0 bugs on the first try

We're not at that level yet, sadly

[robust hill]

no im just gonna make a leech detector

a rudimentary one

my idea is like this

divide lapses by review, get a ratio

find a way to exclude cards with low reps

exclude outliers
and then yeah, tag it with a leech tag (different than "leech"), and make it so the user can change the ratio

[unique salmon]

Nah, I already have an idea for a cooler detector

[unique salmon]

Here

Math isn't the hard part, it's the opposite - math is the easy part

The hard part is integrating that math into Anki. The really hard part is adding the functionality to remove the leech tag automatically, which currently doesn't exist.

[robust hill]

?

mines cooler trust

also

[robust hill]

how to fix

[unique salmon]

No idea

I was gonna say "Submit a bug report", but someone already did: https://github.com/open-spaced-repetition/fsrs4anki-helper/issues/544

[robust hill]

yea i updated and kaboom

[tepid spoke]

Having LLMs write your applications.
How it started: https://x.com/leojr94_/status/1900767509621674109

leo (@leojr94_) on X

my saas was built with Cursor, zero hand written code

AI is no longer just an assistant, it’s also the builder

Now, you can continue to whine about it or start building.

P.S. Yes, people pay for it

X · Mar 15, 2025, 4:34 AM

How it went: https://x.com/leojr94_/status/1901560276488511759

leo (@leojr94_) on X

guys, i'm under attack

ever since I started to share how I built my SaaS using Cursor

random thing are happening, maxed out usage on api keys, people bypassing the subscription, creating random shit on db

as you know, I'm not technical so this is taking me longer that usual to

X · Mar 17, 2025, 9:04 AM

[quiet saddle]

I saw those tweets but thought they were forged, I couldn't imagine someone is silly enough to sell an app entirely made by an AI.. do you think that guy really did it?

[tepid spoke]

From what I heard it's real

Cursor exists and is a real product people buy

[quiet saddle]

yeah but.. but..

we need an label for software made only by humans, surely "organic" would work for software too

my app is 95% organic, yeah!!

[tepid spoke]

I don't know, I think if that guy hand-wrote it, it'd have gone simiarily :D

[quiet saddle]

yeah, the code made by an IA is probably better than the one he would have wrote.. but the a code he would have wrote couldn't pass as a real app, so nobody would have put their data in it.

[tepid spoke]

i see nothing wrong with using an LLM to generate boilerplate

but it's a problem the moment nobody understands what the code does anymore

[quiet saddle]

yeah, for maybe boilerplate, although frameworks are there for that and more reliable.. only issue is you need to read the documentation and to know what you're doing.

[sick moth]

I pay for Cursor

[quasi shadow]

me too

60% code of the first version of fsrs-rs was written by Claude because I didn't know Rust.

[quiet saddle]

so it was 40% organic 😅

[quasi shadow]

The main work of Claude is to translate python code into rust.

[quiet saddle]

A little bit more if the code was read and validated by humans after generation, but I don't know how to take that into account for my "organic software" label yet.

[cursive badge]

Go full FSA traffic light label 😂 :

Each serving (10k LOC) contains:

• 6k LOC AI generated (HIGH) 60%
• 1k LOC AI unchecked by humans (MED) 10%
• 0.1k LOC no test coverage (LOW) 1%
  of an adult's reference intake.

Typical values (as sold) per 1k LOC: 600 AI LOC

Enjoy as part of a balanced software diet.

[quiet saddle]

I may have to write a semi-joking blog article about that, it's a fun idea. Because if 100% of the code is AI generated, but those parts were hand picked and ordered by a human, it's almost organic. So it's a difficult question, maybe I should pitch an AI to write that article 🤔

[fluid lagoon]

If I am working with a case of hard-misuse, it seems I basically have two options for correcting it:

1. In the FSRS optimize section, set "Ignore reviews before" to the date where I started using "hard" correctly.

OR

2. Download the fsrs4anki-helper add-on, and use the "Remedy hard misuse" functionality.

I'm wondering which is my better option here. It feels like just using "Ignore reviews before" is simpler, since I'll soon have quite a lot of reviews in order to perform optimizations.

However, is there any negative affect by having a bunch of incorrect hard reviews even though I'm using "Ignore reviews before"? Or, are optimizations likely to be much higher quality if I instead just use "Remedy hard misuse"?

I guess at some point I may just have to read through the scheduling and optimization code...

[cursive badge]

N.B. it was renamed "Ignore cards reviewed before" in newer versions of Anki because it will ignore all reviews on cards you reviewed before that date, including newer reviews after that date.
Only cards that you reviewed for the first time after that date will contribute to optimising FSRS parameters. This can be an issue if you already learnt most/all of your deck before that date and are not adding new cards.

[fluid lagoon]

Well that's news to me!

Thank you, that's very good information. I guess the fsrs4anki-helper add-on, and "Remedy hard misuse" is the way to go then.

it will ignore all reviews on cards you reviewed before that date, including newer reviews after that date

It feels like this information should be in giant big neon letters 🥵

[quasi shadow]

😅 I don't know how the human memory works now.

[lapis hearth]

What is odd about this, for the uninitiated❓

[quasi shadow]

The data shows that the user retention increases between 3 days to 5 days.

@polar maple the forgetting curve is sharper when the first rating is again.

When I try to fit the next forgetting curve for cards relearned in the fist time, the shape becomes sharp, too.

[unique salmon]

That's evidence in favor of "flatter curves are better for easier material"

[quasi shadow]

I think it also shows the degree of heterogeneity.

[unique salmon]

Honestly, yeah

[quasi shadow]

I did a simulation and found that the heterogeneity of first_grade = 1 is the least.

[unique salmon]

That's why we gotta use a large dataset, like the 10k dataset, to draw conclusions. This stuff is messy af

[quasi shadow]

😅 Too much messy.

[quasi shadow]

Is it hard?

https://github.com/ankitects/anki/blob/ccab18b7ba624d888f3d881e14f04c830e3eaa44/pylib/anki/notes.py#L175-L188

Anki provides the API to add/remove tag.

But I guess you need flag instead of tag.

Because tag is note-level.

[unique salmon]

Good point, I forgot about that

Actually, maybe it's worth ditching leechs tags and adding a new leech flag for both the new detector and the old one

To prevent siblings from being tagged as leeches

[quasi shadow]

😅Oops, a card can only have one flag...

so it's still problematic

https://forums.ankiweb.net/t/feature-request-multiple-flag-colours-for-one-card/33421

https://github.com/ankitects/anki/issues/1806

[lapis hearth]

Crap

[cursive badge]

Time to stuff more fields into the card custom data!

[lapis hearth]

Could you introduce 3 leech flags. A neutral colour to say that the leechiness of the card has not increased since last time, a red colour to say it has worsened, a green colour of some sort to say it has gotten better

I think this would be a good idea to implement

A flag should be searchable on the browser

[cursive badge]

That wouldn't work the way you want. At least with the Poisson binomial idea that Expertium is championing.

[lapis hearth]

And instead of a flag, maybe a UI colour hue to make recognition easier

[lapis hearth]

You mean that the detector wont be able to compare the leechiness of the card between now and last time seen

❓

[cursive badge]

It would always become less leechy if you got it right and more leechy if you got it wrong and never stay the same.

[lapis hearth]

So how would you be able to determine if the card has not increased or decreased in leechiness

[unique salmon]

just look at number

Speaking of which, do we want to show p(leech) as a stat?

[lapis hearth]

what number

[cursive badge]

I think I said before that it would be cool to have a prop:leech-p (1 - p)

[unique salmon]

number

[lapis hearth]

tf

[unique salmon]

The detector outputs the probability that the card is a leech

[lapis hearth]

Oh

that makes sense

Okay maybe p(leech) as a stat is not so bad after all

But a colour indicator inside the reviewer for good recognition is best

To show if the number increased or decreased from last time

[unique salmon]

As ross said, p(leech) will always increase if you press Again

[lapis hearth]

yes but it is about the overall number of agains and overall number of goods you press

or does it not work like that

[cursive badge]

A colour gradient for how leechy a card would be possible, but a trend would be pretty useless with the Poisson Binomial proposal.

[unique salmon]

I'm saying that if you want a color indication of whether p(leech) has increased or decreased, it would be equivalent to just "This color if I pressed Again, the other color otherwise"

[lapis hearth]

again again again good is not the same as again good again good

I dont know if I am speaking like a caveman

[lapis hearth]

oh right

[cursive badge]

I can see what you mean, it just unfortunately does not work like that for this method.

[lapis hearth]

so the moment you press again, it becomes red, the moment you press good it becomes green

Isnt there any other way to show trend

🙁

[cursive badge]

Not on a per-answer basis.

[lapis hearth]

it doesnt have to be a per answer basis

It is like a graph

a graph can fluctuate, but on the grand scale of things you can sketch a trend line

[cursive badge]

You can look at how p(leech) changes over a longer period

[lapis hearth]

Maybe, record p(leech) on a graph

And make a trendline

[lapis hearth]

Yes exactly

[cursive badge]

It would be possible to put something in the card info like the forgetting curve

[lapis hearth]

yes please!

A positive gradient meaning red in colour

A negative gradient meaning green in colour

But wont that be too extreme

you have to set some threshold which if the gradient surpasses it is considered significantly worse

So what does significantly mean in this contexx

[crude raptor]

excuse can someone checkout my help thing i posted please

i need help can someone check it out, its the most recent one

[unique salmon]

No user-defined thresholds for anything, the detector will be a black box with one toggle. But you will be able to search for cards based on p(leech) in the browser

[lapis hearth]

I have made a godawful sketch of what I mean

@cursive badge

then use a colour gradient

[unique salmon]

Man, just let Anki mark cards as either "leech" or "not a leech", relax and live a peaceful life

Keep it simple

[lapis hearth]

I want to know if the trend is increasing or not

It should give me an indicator that what ever I was doing to tackle the leech is helping or not and if I should take more measures

So this is the colour gradient I was talking about

I dont know which colour comes after which but you should get the idea

[quiet saddle]

Another issue with the leech flag idea is that it would conflict with the intended usage of flags.
If the user flagged a card for another reason, either the card can't be tagged by the leech detector or the previous state of the flag is lost.

[cursive badge]

The actual data looks more like this
N.B. this is p(normal) i.e. 1-p(leech):

[fluid lagoon]

Reading through the "Remedy hard misuse" code, it looks like it works on the whole collection, but I only have one deck where I was misusing Hard.

So I whipped up a little sql to set the ease in the revlog for only cards that come from a certain deck:

UPDATE revlog
SET ease = 0
WHERE id IN
    (SELECT revlog.id
     FROM revlog,
          cards,
          notes,
          notetypes
     WHERE revlog.cid == cards.id
       AND cards.nid == notes.id
       AND notetypes.id == notes.mid
       AND notetypes.name LIKE 'MyNoteType'
       AND revlog.ease == 2
       AND revlog.id < 1742310000000);

Hopefully this doesn't cause any problems

[lapis hearth]

You get the idea 😅

[unique salmon]

Can someone help me find the code in the Anki repo that does this stuff?

[clever cargo]

are you looking to add a new warning or change an existing one

[unique salmon]

I want to add a new one to "Ignore cards reviewed before"

[clever cargo]

you can grep for <Warning to see how its currently used

[cosmic hedge]

https://github.com/ankitects/anki/pull/3847 you can use this as an example if you want

[unique salmon]

https://github.com/ankitects/anki/pull/3907
I did a thing

I gotta say though, looking through the repo makes me want to give up on ever trying to understand programming, or at least the "Big Important Project Made By Lots Of Big Brained People" kind of programming
deck-config-too-short-maximum-interval is defined once and then never used anywhere else, at least according to Github search, yet somehow it is displayed
HOW?!
HOOOOOOOOOOW?!

This isn't

print(text)```
This is more like
```text='blah'
literal magic
(somehow the text is displayed)

I can't think of any logical explanation for this, other than Github search not working properly and not showing me usages of this thing

Or take deck-config-learning-step-above-graduating-interval. Again, same thing. Just defined once and then never used again, seemingly

HOW?!

deck-config-save-to-all-subdecks - same

deck-config-easy-days-sunday same

Everything in ftl/core/deck-config.ftl is just some shit that is seemingly never used, except it is

But the usages don't exist

But it's being used

https://tenor.com/view/futurama-scream-afraid-gif-15549420

• Bro, just learn programming, it's easier than you think! It makes sense!
  Programming:

[polar maple]

i think the samples are just too small and unreliable, think about how much variance goes in each data point itself (binomial distributed variable, variance of np(1-p)), and the fact that the exact position of the point heavily determines the shape of the resulting fitted curve. I think if you want a more sound approach you can try a bayesian perspective, set a prior on the parameters decay and stability and compute a posterior distribution given the data. This way the possible variance in the data itself would be considered.

[quiet saddle]

I don't know the translation tool used in Anki, but those variable names are probably transformed somewhere before being used. Translation tools often add some complexity to the code base.

[quiet saddle]

Anki code isn't the best one to learn coding, like learning English from rap lyrics or something

[unique salmon]

https://github.com/ankitects/anki/pull/3847/commits/783d477949a29667e0250e03ece2b64c36ab85bf
Look at this PR
There is nothing that indicates that deck-config-too-short-maximum-interval is in any way used in
$: maxIntervalWarningClass = $config.maximumReviewInterval < 50 ? "alert-danger" : "alert-warning"; $: maxIntervalWarning = $config.maximumReviewInterval < 365 ? tr.deckConfigTooShortMaximumInterval() : "";

hoOoOoOoOoOoOoOoOoOoOoOw

[sick moth]

Fluent is a bit of a nightmare in that sense. Blame Mozilla

[quiet saddle]

deck-config-too-short-maximum-interval <-> deckConfigTooShortMaximumInterval that's the transformation you're looking for

[sick moth]

Getting into a project is just one "oh... that's how it works" after another

[unique salmon]

https://tenor.com/view/question-mario-god-filthy-frank-idubbbz-gif-21508154

[cosmic hedge]

Its not really that hard to deal with when your ide can auto complete it for you

[clever cargo]

this

[sick moth]

It's kinda painful though. Ctrl+F should list something

[clever cargo]

also, "display the warning only if the chosen date is after the date of the first ever review the user did in Anki" is potentially going to make it very annoying

"first review that this preset covers" would be more palatable imo

[unique salmon]

I'm fine with a constant warning. Idk of Dae and others are, though

[cosmic hedge]

Yeah i'd just do it if the values not the default

[sick moth]

It /should/ be in our GeneratedTranslations.kt Javadocs

[clever cargo]

the default is 1970

so that's the same thing

[unique salmon]

I (think?) made it display a constant working, assuming I got the code right

And it doesn't break

[unique salmon]

Feel free to do that

[bold terrace]

• I will never understand that codebase
• I totally get that codebase now !
• [Introduce a regression] oh shit
• Repeat

[clever cargo]

please don't @cosmic hedge

[unique salmon]

Or feel free to tell me how to do that 🤣

Can you edit my PR?

Is that a thing?

[clever cargo]

there's so many warnings as it is

[cosmic hedge]

I can not

Id have to pr ur pr

[bold terrace]

Fork-the-Fork