`repo-slopscore`: Analyzing git repos for signs of AI involvement

[lethal pond]

Link to deployment | Source code

"Repo slopscore" is a CLI + optional webapp which collects evidence of AI assistance in git repositories; specifically, right now, "git repositories" means "git https:// url".

It goes through the entire commit history of a repository (upper limit is 5000 commits currently, but I have ideas to change this in the future) and detects visible signs of AI/LLM tool usage in the commit history and the source tree. The tool uses partial cloning to reduce data usage (and potentially forge server load) by skipping blobs/blob contents entirely.

Additionally, aggressive caching is used to ensure that a repo that has been analyzed before does not need to get fully cloned again. Instead, the tool remembers the precise point in time (+commit hash) of the most up-to-date commit to the repo and uses this knowledge to only clone from that point onward the next time analysis is requested.

Go to the source code page I linked to find out more about how this works internally, and feel free to throw as many git repos as you want (...within a reasonable limit) into the tool! :)

[brazen gale]

This is pretty cool

While the false negative rate is maybe a little high due to lack of heuristic analysis (a decent amount of AI projects disable commit trailers and don't commit their agent configs either), by the same token no heuristic analysis means you can't have false positives, which is actually quite a beneficial property to have

It also runs pretty dang fast (tried via the website linked on the repo)

This is gonna save people a lot of future git log / git greping ^^

I am curious, how is the score field on the website calculated, and does it hold any particular meaning?

I saw a repo with ~13 signals had a score of 62 (C), a repo with 0 signals got 100 (S) (obvious), and a repo with 117 signals got 0 (F)

Also, I think it would be really interesting to add some histograms to the scan data

Such as %age of AI-assisted commits per month over time

[mystic ore]

looks like it starts at 100, and it's decremented for each signal

impl Default for PenaltyWeights {
    fn default() -> Self {
        Self {
            ai_author: 4,
            ai_coauthor: 3,
            ai_tool_trailer: 3,
            ai_generation_phrase: 3,
            source_tree_hit: 5,
        }
    }
}

and there are some false positives
https://github.com/meithecatte/enumflags2/blob/master/AGENTS.md
https://slopscan.ava.pet/repo/https%3A%2F%2Fgithub.com%2Fmeithecatte%2Fenumflags2

[brazen gale]

heh

[supple pilot]

I mean some false positives are sadly unavoidable but I really like the idea of the project. Also bonus points for codeberg ❤️

[stuck sigil]

this is a gem ❤️

[low eagle]

we need it here badly

[stuck sigil]

@lethal pond can you make an endpoint for the hosted version that performs a slopscore from a query param or something and then shows it in discord embed

[lethal pond]

Oh my I didn't even see that this got this many responses! I'll try to get back to everyone tomorrow :)

[lethal pond]

Thank you!!

[lethal pond]

Yeah :) I want to minimize false positives, in the sense that this tool is supposed to be able to stay as close to the facts as possible in detecting things. The idea is to make it easy to point out whether a repository has been made by someone who uses one of the various available AI tools, many of which would be sort of painful to check for manually

[lethal pond]

I am glad you say that! I definitely do know of at least some areas where I could make things significantly faster and also less resource intensive; but also, really, this is good enough for now, and I am happy about that

[lethal pond]

The "score" is just one metric that might help in assessing how much a project has been tainted with AI usage. Each detection signal type (commit co-authored by ai, commit authored by ai, ...) has a numeric weight attached to it, and every signal deducts $weight from the perfect 100 score

[lethal pond]

Yes, that would be really cool! The very basic solution for showing the age of the signals themselves is just to print the commit date in the corresponding row, which is already better than nothing, but pretty visuals would be sweet :>

[lethal pond]

It's true, yeah; the mere existence of an AGENTS.md file will already deduct points from the score. But I kept this as-is for now since, well, the existence of that file is hinting to the possibility that AI might have been involved in the making of a certain project at some point

I mean, heck, I myself have indulged in writing silly commit trailers to, on purpose, trigger the tool: https://slopscan.ava.pet/repo/https%3A%2F%2Fcodeberg.org%2Fpolyphony%2Fpolyproto-rs

[lethal pond]

Oh that would be fun. I believe discord creates embed based on what an URL returns when it is queries with the HEAD method? But I am not sure, I'd have to look into that

[stuck sigil]

yeah

it would be really funy to post that in showcase channel for obviously sloperated projects

[low eagle]

can you try it on my github repos?