#Chrome extension for AI chat during game analysis

51 messages · Page 1 of 1 (latest)

kind canyon
#

https://github.com/tombedor/checkmate

Basic idea is set up. There's a lot of hallucination with respect to making up positions on the board, but I like how the concept is working overall

strong sky
#

I'm curious if this just relies on chat GPT's chess knowledge? GPT-4o has an ELO of 1428: https://openlm.ai/chatbot-arena/

Or does this augment the LLM with Stockfish as we discussed?

#

Aha! I see a function called extractStockfishEvaluations, which is a good sign!

#

It looks like it calls the Lichess.org analysis engine, which returns the list of best moves, and then sends that analysis to the LLM along with the system prompt:

You are a helpful chess tutor assisting a student in analyzing a chess game. You should provide educational insights about chess strategy, tactics, and gameplay.

And:

The user's message will include context about which specific move or position they are currently viewing. Please provide helpful, educational responses about chess concepts, the current position, tactics, and strategy. Keep responses concise but informative.
strong sky
#

The analysis text is a bit concise and hard to understand. Did you try sending variations of it to the LLM? Perhaps checking changes to the best moves as the game progresses to identify blunders and especially good moves?

#

I've also been thinking about adding a "strategy" overlay. It might do some board analysis for backwards pawns, open columns, and such and include that along with the best moves.

kind canyon
#

Or does this augment the LLM with Stockfish as we discussed?

Intent is to use the stockfish analysis from LiChess. I figure the stockfish annotations would be useful in augmenting

The analysis text is a bit concise and hard to understand. Did you try sending variations of it to the LLM? Perhaps checking changes to the best moves as the game progresses to identify blunders and especially good moves?

Yeah I haven't tweaked the prompts too much yet, I'd be curious for feedback on the prompts.

I've also been thinking about adding a "strategy" overlay.

Curious if you know of any existing (non-AI) tools that can do this, I agree strategic callouts is the big gap that I think something like this could fill

#

Thanks for checking it out!

#

Looks like maybe actually the study screen might be a better tool thing to use for this, it inlines the blunder, inaccuracy etc info

#

Ui is tricky, it just spitting out chess notation isn’t easy to read

strong sky
#

I thought I had seen blunder somewhere on Lichess...

#

I think that takes the best of the possible next moves and compares its centipawn value against the last best move value?

kind canyon
#

yeah something like that I thnk

#

just pushed an update, this screen is working a lot better

#

though still hallucinating some positions

#

strategy reference is the other thing I think, e.g. one of the comments the ai made referenced the greek gift, which i wasn't familiar with.. I found this study: https://lichess.org/study/LvIG3kF5 but not sure if there's a database of these somewhere

strong sky
#

Is it hallucinating positions or invalid moves? Stockfish can give you a list of legal moves using perft, but perhaps you can't send UCI commands to the Lichess analysis engine?

#

You could also consider adding more instruction to your system prompt so that it only chooses moves from the list of preferred moves generated by the analysis?

kind canyon
#

The recommended lines are there, it hallucinates mostly positions.

#

I am sending the stockfish recommendations along with it

#

Most of the actual questions I end up ask are, why did the engine recommend this

#

Maybe the better name for this is stockfish-explainer

jovial fossil
#

If it can do that, that is valuable in it's own right.

strong sky
jovial fossil
#

explain StockFish in a way humans can understand. A StockFish explainer.

strong sky
jovial fossil
kind canyon
#

I think it does ok when given stock fish annotated games

#

The tricky bit for me is the ui

#

Should it be an extension for lichess, a dedicated site, etc

#

Not grand master level certainly but, good

jovial fossil
#

Yes, and I think that is great, especially for amateur learners. I think it might have a struggle with giving analysis fit for a GM.

But, there are a tone of non GM's that could benefit.

#

How tied into the chrome ecosystem is it? Like, how hard would it be to port it?

strong sky
#

It is pretty straightforward to port. Just check out the code in the github repo. It only takes a few minutes.

kind canyon
#

Which browser are you interested in?

#

I wrote the extension itself in like 5 mins with ai so happy to add to others

jovial fossil
#

Awesome. I would love to try it out on Firefox!

#

I do not know much about the ecosystem or I woud try it myself

kind canyon
#

It also only works with OpenAI right now jfyi

#

But if that’s ok for now I can port it

strong sky
#

...I wonder if GPT-5 is significantly better, with a higher ELO rating?

jovial fossil
#

Defiantly, what does that entail?

Do I require a subscription?

kind canyon
#

But that would be subject to a rate limit

strong sky
#

I am torn between using the big frontier models and requiring people to enter their API key every time they start using the AI, or using a much less powerful client side LLM that doesn't require any payment. If the app had a server, we could store the API key securely, but without a server in order to be secure you need to enter it every single time you run the app.

kind canyon
#

Yeah my general approach is try with frontier models first, if that doesn’t work well then the smaller models probably won’t either

#

Fwiw this extension doesn’t store anything except local to the browser, at least in chrome that goes to a chrome storage location

#

But like, probably not the most secure