knotty token
If you cannot reproduce the results from a dl paper, is it considered to be academic dishonesty from the author?
Im not reading the latest papers so some papers might be like 5 years ago. I think nowadays the larger the model the easier it is to reproduce the results? I dont really know since Im not an expert in deep learning and it's just my guess.
I dont really mean a specific paper, just a general question
west elm
@rough python @dim swift @unique ginkgo @pale tangle @weak sequoia@spice vine @inland cypress @olive bane@plush thistle What are your thoughts?
fervent sage
peer review in ML DL is usually super scuffed
i once had a reviewer rate me 3/5 on every criteria and quote part of my abstract verbatim as "review"
they often dont ask for code or data (unless u wanna specifically provide it to them)
missing a few lines of maths or some hyperparams really is considered no biggie
hey, found runoob here too lol
unique ginkgo
<@888209450211606589> <@165248929733148672> <@367803030738960396> <@332556475572...
I think it's a big problem. For example, I was unable to replicate a paper I was basing a project off of, either because they lied or because they didn't have a good explaination of their subset (even though I was using their code as a baseline)
fervent sage
I think it's a big problem. For example, I was unable to replicate a paper I was...
the worst is when reviewers get mad at u, because u cant reproduce some other paper for some results to compare against ur model, cuz their paper is written poorly 😂
unique ginkgo
exactly
rough python
Things are changing. In other fields failure to reproduce a result is a scientific contribution on itself. We should get there in ML.
dim swift
Definitely a big problem as it is in fact not even expected that you provide code. Just an optional thing.
It's fine for very basic stuff, e.g. modifying relu for another activation function or something, but for a whole new mechanism then it should be required.
I also believe we will get there as DrDub said. Things are just slower than we want them to be haha.
fervent sage
Definitely a big problem as it is in fact not even expected that you provide cod...
exactly why Louis keeps asking: "What's AI?"
rough python
Definitely a big problem as it is in fact not even expected that you provide cod...
Today I found a new fraud in the field "our code is in github: link to code " but the github repo just has an empty README. I guess they do it to appear in paperswithcode? Sometimes I feel more than a research field we're a band of highway robbers 🤣
fervent sage
Today I found a new fraud in the field "our code is in github: _link to code_ " ...
that happens all too often 😂
unique ginkgo
Things are changing. In other fields failure to reproduce a result is a scientif...
ML isn't the only one with this issue, but I am very glad to see the start of us returning to this mentality in the broader scientific community, even if we aren't experiencing it yet ourselves.
rough python
ML isn't the only one with this issue, but I am very glad to see the start of us...
Yeah, people talk about a "reproducible science crisis" and it sounds like that
unique ginkgo
it started in psychology, or at least psychology was the first one where it was broadly recognized as systemically problematic, and now even the 'hard' sciences are facing these issues consistently.
rough python
then psychology had its revenge https://journals.sagepub.com/doi/full/10.1177/2515245917747646