Api: multiple eval bars for broadcast | lichess.org | Page 1

south plover Jul 29, 2023, 2:51 PM

#

hey i was working on a project that gives you mutiple eval bars , i was extracting the fen from a link like https://lichess.org/study/jvfVgeHr/EbyG8zDv.pgn . it runs locally but when i hosted it i got cors error . here is my code https://github.com/Bot-Rakshit/eval-bar

#

can someone help me fix it please 🥺

#

Access to fetch at 'https://lichess.org/study/rwNUxCe5/R7suc96F.pgn' from origin 'https://golden-pegasus-1b5591.netlify.app' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled. . this is the error

green narwhal Jul 29, 2023, 3:30 PM

#

Instead of using https://lichess.org/study/...-urls,
you could try using Lichess API urls.

In the screenshot,
the green circle shows the Broadcast Tournament ID,
and the blue circle shows the Round ID.

Here's an endpoint for the green circle / Broadcast Tournament ID - https://lichess.org/api#tag/Broadcasts/operation/broadcastAllRoundsPgn
and here's an endpoint for the Round ID - https://lichess.org/api#tag/Broadcasts/operation/broadcastRoundPgn

Hope they will help! And cool eval bars!

south plover Jul 29, 2023, 5:47 PM

#

@green narwhal so for the broadcast round pgn how will the endpoint look like

#

got it nvm

green narwhal Jul 29, 2023, 6:02 PM

#

Cool cool,
for reference - https://lichess.org/api/broadcast/round/R6UwwMir.pgn

(Found it and what header to use by enabling logging with chariot):

[tors@xps ~]$ jshell --class-path chariot-0.0.71.jar
|  Welcome to JShell -- Version 20.0.2
|  For an introduction type: /help intro

jshell> var client = chariot.Client.basic()
client ==> chariot.Client@4232c52b

jshell> client.logging(l -> l.request().all())

jshell> var pgns = client.broadcasts().exportOneRoundPgn("R6UwwMir").stream().toList()
Jul 29, 2023 7:51:01 PM chariot.internal.InternalClient request
INFO: ### Request: GET https://lichess.org/api/broadcast/round/R6UwwMir.pgn
Headers:
accept: application/x-chess-pgn
user-agent: Java/20.0.2 chariot/0.0.71
Body:
<no body>
pgns ==> [[Event "Round 3: Takayasu Melody - Sakai Azumi"] ... . Bd3+ { [%eval #8] } 1-0]

jshell>

south plover Jul 29, 2023, 6:45 PM

#

thanks a lot

#

managed to shift to api's

#

im just very new to building real world projects 🙂

green narwhal Jul 29, 2023, 6:46 PM

#

Niiice!

south plover Jul 29, 2023, 7:08 PM

#

thanks a lot man @green narwhal

south plover Jul 29, 2023, 7:29 PM

#

@green narwhalhttp://139.59.79.118:3000/ just made it live 🙂

#

you have to enter the round url and the board number

#

all thanks to you 🙂

green narwhal Jul 29, 2023, 7:33 PM

#

Coool, it works!
Haha, you would have surely figured it out by yourself, given some extra time.
You fixed that fast! "very new to building real world projects" didn't stop you on this one!

south plover Jul 29, 2023, 7:46 PM

#

thank you 🙂

south plover Jul 31, 2023, 1:20 PM

#

@green narwhal so many times the lichess api doesnt respond is there any way to work around that 😦

green narwhal Jul 31, 2023, 1:27 PM

#

Maybe you are getting rate limited - there's a section in the documentation - https://lichess.org/api#section/Introduction/Rate-limiting

So check all the response codes to see if you get any 429 instead of 200, to verify if that is the case.

If you get a 429, the documentation says you should wait a minute before making more requests.
If you continue sending requests without waiting, I think the API won't respond for a while.

south plover Jul 31, 2023, 1:28 PM

#

since my code essentially pulls the pgn every 10 seconds to check for new moves, it is bound to get rate limited

#

after it runs for 5-10 minutes i start getting runtime errors, but the app continues to run properly though

green narwhal Jul 31, 2023, 1:29 PM

#

Maybe you can use the streaming endoint instead of pulling,
I.e you only make one request which continiously receives updates.
https://lichess.org/api#tag/Broadcasts/operation/broadcastStreamRoundPgn

south plover Jul 31, 2023, 1:31 PM

#

oh yes

#

this makes so much sense thanks

#

man i should have read the documentation properly

#

this will solve everything 🙂

green narwhal Jul 31, 2023, 1:46 PM

#

Hope it works out with redesigning the fetching of the broadcasts!

south plover Jul 31, 2023, 2:11 PM

#

yea finding it hard to understand how to deal with this different response

south plover Jul 31, 2023, 2:18 PM

#

green narwhal Maybe you can use the streaming endoint instead of pulling, I.e you only make on...

the responses are not updated right

green narwhal Jul 31, 2023, 2:45 PM

#

Oh, in what way aren't they updated right?

south plover Jul 31, 2023, 2:47 PM

#

like i dont know how to deal with a open connection

#

and how is the response of this api different

green narwhal Jul 31, 2023, 2:52 PM

#

Ah, yeah - it is a bit different to stream the answer than to poll it.
You are using python?
Are you using some library? (i.e berserk), or are you using just requests or something?
I think I've seen some python code where someone added some parameter stream=True somewhere... 😅

I just tried it out with the chariot library (not python, just checking if the endpoint seems to work),
by creating a queue to put the pgn:s in,
and then starting a separate thread which reads from the endpoint continiously and puts each pgn in the queue,
and then I can check how the queue grows - seems to work... Here's a "recording" https://asciinema.org/a/KQOKS2Hsjq9uor10YqcM1dAld

south plover Jul 31, 2023, 2:53 PM

#

im using javascript

green narwhal Jul 31, 2023, 2:54 PM

#

Oh cool!
Maybe there are some examples in the officiail Lichess API demo 🤔

south plover Jul 31, 2023, 2:54 PM

#

i'll look it up

green narwhal Jul 31, 2023, 2:55 PM

#

https://github.com/lichess-org/api-demo
Deployed at https://lichess-org.github.io/api-demo/

Maybe the "Watch TV" is "streaming" stuff,
https://lichess-org.github.io/api-demo/#!/tv

south plover Jul 31, 2023, 2:56 PM

#

thankss!

#

i think i'll have to give up on this , this is too tough to implement for me .

#

i have already pushed my limits with this project

#

thanks for all your help

green narwhal Jul 31, 2023, 3:14 PM

#

Aww 😦
Well, you could still use your original pulling solution I guess, and maybe add some delay in case Lichess responds with status code 429...
And maybe there exists more Javascript examples out there - I only know about the official Lichess API demo (but I haven't tried it)

south plover Jul 31, 2023, 3:14 PM

#

yea thats what i'll do

south plover Jul 31, 2023, 3:15 PM

#

green narwhal Ah, yeah - it is a bit different to stream the answer than to poll it. You are u...

how are you able to come up with solutions so quickly

green narwhal Jul 31, 2023, 3:20 PM

#

The example I wrote, you mean? Hmm, I have much experience with the chariot library, so it gets easier to do things quickly when one already knows how to do it sort of 😅
It would take me a long long long time to attempt to implement a Javascript solution - because I don't know how to do it. But it is great that there exists examples one can use as a base of origin when trying to learn and understand what is happening!

south plover Jul 31, 2023, 3:25 PM

#

oh nice, its been like few months since i'm coding. the project i made until now was a push for me. I spent close to 70-80 hours while someone more expiernced would do it quickly. Now to understand how to deal with the incoming pgn's is tough. Especially that i was asking the user to enter the game number and lets say they enter 5 , i would count the spaces between the pgn to reach that game number . but now if i use stream that logic wont work. i'll have to look at something more unique . I learnt so much by making a single project, depolying it on a ubuntu server was cool and fun though, to have a server was only coool lol. I never imagined you could ssh to other machine and literally own that machine and host anything you want. maybe i'll take a break and then try and implement this stream api , because until then i wont get peace of mind

south plover Jul 31, 2023, 5:07 PM

#

isn't there any other api other than lichess

green narwhal Jul 31, 2023, 5:16 PM

#

I don't know if any other sites provides APIs to chess broadcasts.
Lichess gets their moves from the people hosting the tournament, I guess. Something like Lichess sends an email to the tournament hosts and ask them for a link to a live feed of moves, and then the Lichess software can show the games to viewers - and as an extra service Lichess also provides an API which one can use to read the moves from Lichess. Cool stuff!
Maybe other chess sites do too, I haven't asked.

south plover Jul 31, 2023, 5:16 PM

#

ah i see

#

i just so want it to work properly because i have setup the frontend, i have the flask server that returns the eval when u send fen's

#

http://139.59.79.118:5000/evaluate?fen=r2qkb1r/pp3ppp/2n1pn2/2pp4/3P1Bb1/2P1PN2/PP1N1PPP/R2QKB1R w KQkq - 0 7 - flask server

green narwhal Jul 31, 2023, 5:20 PM

#

One idea,
In case it is the Javascript programming which is stopping you,
would be to use berserk (https://github.com/lichess-org/berserk/blob/master/berserk/clients/broadcasts.py#L147) on your serverside to stream the moves,
and then you can change your frontend to query your serverside instead somehow? 🤔

south plover Jul 31, 2023, 5:22 PM

#

hmm

#

but then i can't tell user to search by game number

#

because there will be new PGN that are appended , so no way to find out the board numbers

#

http://139.59.79.118:3000/ right now i take broadcast link and the game number as input

green narwhal Jul 31, 2023, 5:26 PM

#

My thinking was that if you feel more comfortable with Python (you mentioned Flask), you could use the streaming endpoint from your Python code and parse the data into some data structure which you feel comfortable with. And that datastructure you could somehow serve to you Javascript client code, coming up with your own API of how to access the Python datastructure.
But maybe that would be to overcomplicate things... 😅

south plover Jul 31, 2023, 5:27 PM

#

only if i was that skilled , this is maybe something i could do but would take a lot of time

#

by the way lets say using chariot and berserk , lets say you wanted to follow Magnus's game and everytime a new move is there in magnus's game the moves gets updated for you . could u do that using stream broadcasts api ?

#

because at the end of the day i just require it for the chessbase india broadcasts - they require live eval bars for all the indian players, so i know the player names , so if i understand how you can do the above its simple for me

#

once i can get updated pgn's for a specific game of the broadcast

green narwhal Jul 31, 2023, 5:48 PM

#

I'm not sure what you mean...
All the data is in the data you get via the broadcast api - you get all the pgns as they update - the players' names are in the pgns.

south plover Jul 31, 2023, 5:52 PM

#

okay

#

thank you for your help

green narwhal Jul 31, 2023, 6:05 PM

#

You're welcome!

south plover Jul 31, 2023, 11:09 PM

#

@green narwhal python berserk client doesnt have the streambroadcast? couldnt find it in docs

#

client.broadcasts.get_round_pgns can i use this itself is there any rate limit or is this slow?

green narwhal Jul 31, 2023, 11:10 PM

#

green narwhal One idea, In case it is the Javascript programming which is stopping you, would ...

Looks like it ^ ?

#

I don't know if the berserk client has implemented automatic throttling, or if you need to do it manually

south plover Jul 31, 2023, 11:11 PM

#

by throttling you mean?

green narwhal Jul 31, 2023, 11:13 PM

#

If Lichess API answers with a status code 429, one shouldn't make any more requests for the next 60 seconds (according to the Lichess API documentation),
and one shouldn't send parallel requests,
and not faster than 1 request per second.

I don't know how the berserk client does - maybe it helps to uphold these limits - or maybe it relies on the user of the library to uphold those limits.

south plover Jul 31, 2023, 11:13 PM

#

ok fine i'll try and find out 🙂

green narwhal Jul 31, 2023, 11:14 PM

#

Yes, testing is the way to go!

south plover Jul 31, 2023, 11:14 PM

#

and from now on i'll read the documentation more and try to be self dependent

#

and maybe letter on contibute to the docs myself ! 🙂

south plover Jul 31, 2023, 11:30 PM

#

sorry @green narwhal disturbing you for the last time , i was looking through the berserk documentation got nothing on lient.broadcasts.get_round_pgns. and when i tried to run it Pgn_list = client.broadcasts.get_round_pgns(broadcast_id="nSlEoh8w")
AttributeError: 'Broadcasts' object has no attribute 'get_round_pgns' i got this error

green narwhal Jul 31, 2023, 11:34 PM

#

Oh, maybe you have the wrong version of berserk.
There have been like three different maintainers of berserk - it was recently embraced by Lichess (here's a link about that move - https://discord.com/channels/280713822073913354/1093269351697756293 ).
I don't know how to verify which version one uses and how to change the version should it be wrong version.
I guess there should be info in the readme-file in the berserk repository...

south plover Jul 31, 2023, 11:38 PM

#

got it thanks

south plover Jul 31, 2023, 11:55 PM

#

so i tested berserk and 🤡 Concurrency Level: 1
Time taken for tests: 59.467 seconds
Complete requests: 50
Failed requests: 46
(Connect: 0, Receive: 0, Length: 46, Exceptions: 0)
Non-2xx responses: 17
Total transferred: 706305 bytes
HTML transferred: 697480 bytes
Requests per second: 0.84 [#/sec] (mean)
Time per request: 1189.338 [ms] (mean)
Time per request: 1189.338 [ms] (mean, across all concurrent requests)
Transfer rate: 11.60 [Kbytes/sec] received

#

this was when the concurrency was 1

#

so i'll go back to js then

green narwhal Aug 1, 2023, 8:08 AM

#

With the Python suggestion I was thinking that you would try to implement the streaming endpoint,
and then redirect your Javascript polling code to poll your Python server instead of polling Lichess.

Here's an example of such a solution using chariot - https://asciinema.org/a/xXF0VxIl6gXEVYsPw44DMsUny

The chariot script starts a thread which uses the streaming endpoint of Lichess,
and stores the PGNs in memory in a Map.
It also starts a HTTP server which listens for requests (I was thinking that your Javascript could be querying such a server instead of querying Lichess directly).
So when a request comes to the HTTP server, it parses the URL for an index (i.e something like http://someaddress:port/5 -> index 5),
and looks up that index in the Map.
The Javascript can send how many requests it wants to this HTTP server as there is no rate limiting there. It acts as a cache for the Lichess server.

#

The example script,

import chariot.Client;
import java.net.*;
import java.util.*;
import java.util.concurrent.*;
import com.sun.net.httpserver.*;

String broadcastRoundId = "Ij6y5h9H";

Map<String, Integer> tagToIndex = new ConcurrentHashMap<>();
Map<Integer, String> indexToPgn = new ConcurrentHashMap<>();

new Thread( () -> Client.basic().broadcasts().streamBroadcast(broadcastRoundId).stream()
        .forEach(pgn -> {
            Integer index = tagToIndex.computeIfAbsent(
                    pgn.tagMap().get("Site"),
                    key -> 1 + tagToIndex.values().stream().max(Integer::compare).orElse(0));
            indexToPgn.put(index, pgn.toString());
        })).start();

var httpServer = HttpServer.create(new InetSocketAddress(InetAddress.getByName("0.0.0.0"), 8080), 0);

httpServer.createContext("/", exchange -> {
    try {
        String path = exchange.getRequestURI().getPath().substring(1); // "/1" -> "1"
        int pgnIndex = Integer.parseInt(path);

        String responseBody = indexToPgn.getOrDefault(pgnIndex, "No pgn for index " + pgnIndex);
        int status = responseBody.startsWith("No pgn") ? 404 : 200;

        byte[] bytes = responseBody.getBytes();
        exchange.getResponseHeaders().put("content-type", List.of("text/html"));
        exchange.sendResponseHeaders(status, bytes.length);
        exchange.getResponseBody().write(bytes);
    } finally {
        exchange.close();
    }
});
httpServer.start();
System.out.println("Listening to " + httpServer.getAddress());

south plover Aug 1, 2023, 11:01 AM

#

green narwhal With the Python suggestion I was thinking that you would try to implement the st...

got it , but python doesn't have a streaming endpoint

#

or am i wrong?

#

i'll try to use java and chariot then, idk java but i'll learn it

green narwhal Aug 1, 2023, 11:05 AM

#

This looks like the Python code for the streaming endpoint - https://github.com/lichess-org/berserk/blob/master/berserk/clients/broadcasts.py#L147-L154

    def get_round_pgns(self, broadcast_round_id: str) -> Iterator[str]:
        """Get all games of a single round of a broadcast in pgn format.

        :param broadcast_round_id: broadcast round ID
        :return: iterator over all games of the broadcast round in PGN format
        """
        path = f"/api/broadcast/round/{broadcast_round_id}.pgn"
        yield from self._r.get(path, fmt=PGN, stream=True)

But maybe it isn't available in a deployed version of berserk then... 🤔

#

Oooh, maybe it isn't!

south plover Aug 1, 2023, 11:06 AM

#

this isn't the streaming endpoint i have used this itself first

green narwhal Aug 1, 2023, 11:07 AM

#

You are right!
I just checked the chariot code and there's a /api/stream/...-prefix for the streaming endpoint - sorry!

    public static EPMany<Pgn> streamBroadcast =
        Endpoint.of(Pgn.class).endpoint("/api/stream/broadcast/round/%s.pgn")
        .streamMapper(Util::toPgnStream)
        .accept(chesspgn).toMany();

    public static EPMany<Pgn> exportBroadcastOneRoundPgn =
        Endpoint.of(Pgn.class).endpoint("/api/broadcast/round/%s.pgn")
        .streamMapper(Util::toPgnStream)
        .accept(chesspgn).toMany();

south plover Aug 1, 2023, 11:08 AM

#

yes yesterday i went through both chariot and berserk. And chariot has way more endpoints than berserk!

south plover Aug 1, 2023, 11:08 AM

#

green narwhal The example script, ```java import chariot.Client; import java.net.*; import jav...

i'll work on this

#

thank you so so so much! For going that extra mile to help me !

green narwhal Aug 1, 2023, 11:12 AM

#

I was thinking that since you already have a Flask server,
it would be "easy" to add similar code as the chariot example but using berserk...
I didn't realize the endpoint wasn't implemented yet.

Hmm, introducing a Java process might be a big thing though? 😅

south plover Aug 1, 2023, 11:17 AM

#

green narwhal I was thinking that since you already have a Flask server, it would be "easy" to...

yes its a big thing but i have time, and i think i'll learn things and try to implement

#

how you deal with things that you are not familiar with and don't know , is what makes a goodf software dev i guess !

#

by the way is lichess.org down? or did i get a ip ban for sending too many requests 😢

green narwhal Aug 1, 2023, 11:21 AM

#

😬 It is up for me... So you might need to wait for a while then...

south plover Aug 1, 2023, 11:21 AM

#

lichess.org is only working with VPN 😂

green narwhal Aug 1, 2023, 11:30 AM

#

Hmm,
the architecture of your solution would probably be simpler if you managed to get the Javascript version of the streaming endpoint to work.
Maybe there exists more examples of open source projects which use Javascript to stream data from Lichess (in case the official Lichess api-demo was complicated).
I think I would try to spend some time on that, instead of trying to involve yet another HTTP server, and needing to learn yet another languange.. 😅
But sure, trying out the chariot example will include some learning experience which I guess is never bad 😅

south plover Aug 1, 2023, 11:34 AM

#

green narwhal Hmm, the architecture of your solution would probably be simpler if you managed ...

i have some expierence with java and idk feels easier to implement than js! By some experience i mean only data strctures

south plover Aug 1, 2023, 12:04 PM

#

lichess is still down for me sadKEK

green narwhal Aug 1, 2023, 12:08 PM

#

Doh!
The blocking was orignally set up to grow in duration exponentially if one continued to make requests after receiving a 429 status code, I think,
but "recently" it has been capped to some duration, i.e not exponentially growing, if I recall correctly,
but maybe the limits have been updated again since the recent DDoS attacks on Lichess.
Best thing to do is to not send requests if one gets a 429 😅

south plover Aug 1, 2023, 12:11 PM

#

yes, now it is up

south plover Aug 1, 2023, 1:48 PM

#

hey @green narwhal i managed to implement it through js finally but im facing a small little hurdle, sometimes the user wants 2 eval bars from one broadcast, and 2 eval bars from other broadcast. so what is happening is since the connection is live, and u cant make 2 requests at a time , i get 429 error if i try to make another stream api call with another broadcast id

green narwhal Aug 1, 2023, 1:56 PM

#

Cool cool! 💪
Oh, two broadcasts.. Maybe you could open an issue (similar to this one maybe https://github.com/lichess-org/lila/issues/12346),
and explain your use case, to see if it is considered too much or if the limits can be changed 🤷

south plover Aug 1, 2023, 1:57 PM

#

i'll do that thanks @green narwhal !

green narwhal Aug 1, 2023, 2:00 PM

#

Another angle to investigate could be if you can create your own broadcast and add the games of interest to that broadcast (I don't know how to do that, or if that is even possible though)... 🤔

south plover Aug 1, 2023, 2:02 PM

#

yea true

green narwhal Aug 1, 2023, 2:05 PM

#

I've seen sometime when clicking around in the web browser on some broadcast creation page that one could add "Lichess game ids" or something to a broadcast - but maybe those are only games being played between Lichess accounts - a broadcast tournament is typically not played on the Lichess server by Lichess accounts...

lunar vessel Aug 1, 2023, 2:20 PM

#

Api: multiple eval bars for broadcast

#Api: multiple eval bars for broadcast